1. Introduction

The unique properties of two-dimensional materials (2D materials) for light emission, detection, modulation, and strong quantum confinement make them ideal for a wide range of applications in electronics, photonics, and optoelectronics devices [1–3]. Graphene and other 2D layered materials, for example, transition metal dichalcogenides (TMDCs), hexagonal boron nitride (h-BN) and black phosphorus have attracted increasing attention because of their novel electric and optical properties [3–5]. Furthermore, they are easy to integrate with photonic structures such as waveguides and cavities, as they can just be transferred on top of such structures [6]. These characteristics make 2D materials suitable for photodetectors [7–9], excitonic light-emitting devices (LEDs) [5,10], lasers [11,12], optical generation of spin–valley currents [6], single photon emission [13], and modulators [14–17]. However, despite the current technological advances, identifying if a film is a monolayer remains a challenging process. The complexity of monolayer detection processes limits the optimization and the development of new devices based on 2D materials. Monolayer films of two-dimensional materials can be readily produced via chemical vapor deposition (CVD) or mechanical exfoliation (ME) from a bulk crystal [18]. Monolayers produced by ME contain fewer defects in their atomic structure which is crucial for their performance and applications [19]. Regardless of how the monolayer films are produced, identifying if the films are indeed monolayers is a time-consuming process [20] that typically takes a skilled person several hours. The detection process consists of manually scanning a sample using an optical microscope to find monolayer candidates and then applying a characterization technique such as atomic force microscopy (AFM), Raman [21], or photoluminescence (PL) spectroscopy to confirm they are monolayers. The manual detection process is a complex task that requires human interaction, which is subjective and prone to produce erroneous detections. The monolayer degrades during the time it takes to complete the detection and characterization processes.

An automated process for detection of two-dimensional materials based on low magnification imaging and without the need of human interaction would significantly reduce the time needed to identify monolayers. Previous works have demonstrated that it is possible to train deep learning (DL) image recognition of monolayers [22], bilayers, and multilayers but the use of high magnification objectives to capture the images limits the system to find small areas (25 $\mathrm{\mu} {m^2}$ – 225 $\mathrm{\mu} {m^2}$) [23–26]. These algorithms analyze between 1,000–12,000 images with a small field of view to cover an area up to 1.5 cm x 1.5 cm, but the total detection time is not significantly reduced (∼1hr) [27]. These works use high magnification objectives to image the sample. Their training processes takes many computation hours to be completed (∼31 hrs.), in addition to the time needed to prepare the training dataset. Despite these strong methodologies, their accuracy rate oscillates between 67% and 93%. Several works have used digital image processing (DIP) to optimize optical microscopy (OM) as a non-destructive characterization process [19,28] but the low monolayer image quality at high magnifications limits the optimization for detection systems. These factors limit the application of current algorithms for industrial and academic purposes. Others have used image segmentation to separate the monolayer colors of graphene and $Mo{S_2}$ from the background using the k-means algorithm [29,30]. K-means separates the color of the area of interest from the rest of the image based on pixel similarities or differences, but the use of low contrast and blurry images limits its accuracy. Other efforts have demonstrated that through the analysis of total color difference (TCD) of the RGB values between the two-dimensional material image and different substrate thicknesses (background) it is possible to detect monolayers using a specific color channel [28,29,31,32]. The use of different substrate thicknesses enables a better visualization of the monolayer, but the detection accuracy depends on subjective criteria (i.e. human evaluation) using high magnification objectives.

Here, we show an algorithm based on digital image processing (DIP) and deep learning (DL) to demonstrate autonomous detection of transition metal dichalcogenides (TMDC’s) and hexagonal boron nitride (h-BN) monolayers on an area of 1 cm x 1 cm with a total processing time of nine minutes. We developed a standardized image capture methodology to reduce the total detection time for monolayers larger than 20 µm x 20 µm with a high accuracy rate (Fig. 1(a)). The sample is autonomously imaged by scanning the total sample area of 1 cm x 1 cm with a 5X magnification objective adapted to a motorized X,Y setup. Then we apply a series of filters to reduce the noise in the image and the monolayers colors are separated from the rest of the image [33,34]. We combine the filtered image with the original one to create a well-defined monolayer image. Finally, we run the deep learning (DL) algorithms to detect the monolayers. The DL algorithms take six minutes to run on the full 1 cm x 1 cm area (100 images), and the measured accuracy rate is 99.9%. The h-BN and TMDC’s samples are generated by exfoliation on a PDMS substrate [35]. All the images are taken with the same illumination conditions making it possible to apply the same detection process. The proposed methodology avoids human interaction in the detection process and any additional characterization tests such as Photoluminescence or Raman Spectroscopy.

 

Fig. 1. (a) The detection process consists of 5 steps where the application of digital image processing (steps 1–4) enhances the image quality facilitating the information extraction for the two deep learning algorithms (step 5). This process allows us to develop an autonomous detection process. (b) The monolayer color image shows its RGB color values at an intersection point between the monolayer and the background. (c) In the color image histogram, we observe that the pixel color values are oscillating mainly between 80–146, where the monolayer average pixel values (blue box) are oscillating between 126–136, and the background pixel values (red box) are oscillating between 125–129. These groups of pixels have overlapping pixel values between the background and the monolayer, making the monolayer segmentation (separation from the background) difficult. (d) This image shows the same monolayer image taken in its negative format after the white balance correction. (e) The histogram analysis shows the same pixel region as in the image in b), but the difference between the pixel values is larger (10 pixels). Through experimentation, we noticed that a minimum difference of 10 in each color channel of a pixel is needed to produce a color difference between the monolayer and the background. The negative format makes it possible to differentiate the monolayer from the background. We observe in the histogram that the pixel colors values oscillate mainly between 120–160, where the Monolayer average pixel values (blue box) oscillate between 122–128, and the background pixel values (red box) oscillate between 133–136. These groups of values are separated enough to allow a better differentiation of the monolayer colors from the rest of the image.

Download Full Size | PDF

2. Digital image processing

The digital image processing (DIP) algorithm optimizes the image properties for accurate detection in the deep learning stage by maximizing the contrast between the background and the monolayer through a series of filters. The random variations in the contrast of the image background are similar to the contrast difference from the presence of the monolayer. Due to this similarity, it is challenging to identify the monolayer using numerical, automated methods. We develop a standardized image capture methodology with four steps to optimize the detection process. First, we image the sample in the negative format with white balance correction using an automated X,Y scanning system integrated with a 5X objective. Then we reduce the noise, perform a color segmentation thresholding, and combine the filtered image with the negative image to create a new enhanced image that is optimized for the detection of monolayers.

3. Digital image capture

We enhance the contrast between the monolayer and the image background by taking the negative of the original image and applying a white balance algorithm. The image capture typically produces a low contrast image. To maximize the contrast, first we optimize the exposure settings to maximize the difference between the monolayer and the background in the RGB format under white illumination. Then we apply a negative filter to avoid the brightness saturated points in the RGB format. This simple analysis enables a better interpretation of the RGB chromatic values in the image.

Neg = Negative channel

i = Color channel

MPV = Maximum pixel value

APV = Actual pixel value

We apply a white balance correction on the negative image to enhance the real color perception and increase the difference between the pixel values of the monolayer and the background. During the image capture at the microscale the external light sources illuminating the sample cause color irregularities in the image. This color imbalance is almost imperceptible at microscales and appears in the image as a shift color (e.g. reddish or bluish) that does not represent the real color of the object. This color shift occurs across the whole image and is caused by the illumination conditions in the capture process. To identify 2D monolayers, the accuracy in the real color is critical. Using the camera settings, we apply a white balance correction (WBC) process to the negative image. WBC multiplies each pixel in the image by a gain factor following the gray world algorithm [36]. This correction proportionally adjusts the pixel values in the image assuming that the image is illuminated by a single white light source. This process fixes the color imbalance in the image and does not affect the real color of the pixel in the image avoiding losing key information [37].

The negative image and white balance algorithms increase the difference between the background and monolayer by a factor of two to five depending on the color channel. To analyze the monolayer contrast optimization in the image, we plot the histogram of the color and negative image pixel values and compare the original image with the processed one. The original image contrast was manually optimized by adjusting the exposure settings. We choose a pixel at the boundary between the monolayer and the background (Fig. 1(b), red marker) to analyze the pixel value differences between them. The pixel in the original image shows a minimal difference (between 2 and 7 counts) across the RGB color channels with respect to the background. In the histogram, the pixel values corresponding to the background and the monolayer overlap (Fig. 1(c)). The same pixel on the processed image has a two- to five-fold larger pixel count difference (between 11 and 18) across the RGB color channels; and the pixel counts for the monolayer and the background are now clearly separated in the histogram of the processed image (Fig. 1(d),(e)). The enhanced contrast allows us to separate the monolayer colors from the rest of the image.

4. Noise reduction

To reduce the noise, we apply an average filter to the negative image that homogenizes the colors. Due to the low contrast difference between the monolayer and the background, variations in the pixel values of the image (i.e. noise) make it challenging to numerically identify the monolayer. The average filter extracts the RGB values of a group of pixels and applies the RGB average value in the group [30]. We empirically tested different vector sizes in a set of monolayer images and found an optimum vector size of 15 × 15 pixels to perform the filter. Using this vector size (optimum kernel) the noise in the image is effectively reduced without affecting the monolayer information. By applying smaller vector sizes the noise in the background is not significantly reduced. Using larger vector sizes, the background pixel values started encroaching in the monolayer. This average calculation is repeated throughout the whole image to homogenize the colors. Figure 2 a - b shows the image before and after the average filter. At first glance, the image lost its sharp boundaries after the average calculation (blurry image); however, the algorithm is not using the sharp boundaries as the key characteristic of the monolayer. The average calculation reduces the background pixels that have similar values to the monolayer pixels. By applying this filter, the image loses definition for human perception, but we obtain a cleaner image (numerically) that is better interpreted by the deep learning algorithm.

 

Fig. 2. (a) We show a monolayer image in the negative format where a vector of pixels is evaluated to analyze the differences in the noise reduction (red numbers) after the average filter. (b) The image shows the same vector of pixels after the average filter. This process is performed by taking the dominant pixel value from the vector and averaging the pixel values. This process is repeated in the entire image to reduce the noise. After the average filter, the image is blurry for human perception, but it exhibits an enhanced color definition of the monolayer. (c) We show a negative image after applying the average filter. (d) We create a threshold segmentation mask using the L*a*b color space. The threshold mask separates the monolayer colors from the rest of the image. Boolean operators are used to generate the mask, where “IF” the color levels are different than the threshold mask “THEN” the color will be 0 (belonging to the black color) otherwise a value of 1 will be placed (belongs to a different color than black). The gray dots seen around the monolayer constitute noise that contain pixel values similar to the monolayer but aren’t part of a monolayer. (e) Selected L* a* b* values for the threshold mask. The blue squares determine the range of values that represent the monolayer.

Download Full Size | PDF

5. Color segmentation–creating the threshold mask

We map the negative image into the L*a*b* color space to extract the real color of the monolayer and evaluate and separate the monolayer from the rest of the image (Fig. 2). The information about the monolayer is encoded in the color of the image. In the RGB color space, the color and brightness (luminosity) of the image are entangled in the relationship between the color channels rather than the channel values. Color spaces are mathematical methods for interpreting the color of an object by numerical notation (pixel values) within different channels. The L*a*b* color space, developed by The Commission Internationale de E'clairage (CIE), [38] separates the luminosity from the color into a distinct channel (L*) and the chromatic values into two channels (a*, b*). The L* channel contains the luminosity of the image with higher values representing higher brightness (close to white) and zero representing black. The a* channel represents the difference between red and green colors and the b* channel represents the difference between yellow and blue colors.

Within the L*a*b* color space we create a threshold segmentation to separate the monolayer colors from the rest of the image [34]. We performed 1200 tests of an assortment of different $Mo{S_2}$, and h-BN monolayer images to determine the baseline color values of the monolayers in the L*a*b* color space. $Mo{S_2}$, $WS{e_2}$ and h-BN are 2D materials with different physical and optical properties (e.g. density, refractive index, atom size) [39–42]. However, these differences are not visually noticeable in the image taken with a 5X magnification objective. Therefore, monolayers of these materials exfoliated on the same substrate (PDMS films) produces a similar baseline under the same illumination conditions. This baseline is used in the identification of the pixel color regions contained in the monolayer (Fig. 2(c)-(e)). The monolayers are represented by narrow ranges in the L*, a*, and b* channels, which allows us to better discriminate the monolayer from the background. The threshold filter mask effectively separates the monolayer area values of the three materials from their background.

TM = Threshold mask

g = L*a*b* channel values

APV = Actual pixel values

6. Enhanced image

We combine the original negative image with the filtered image to create a new image where the monolayer, background, and bulk material are separated by different colors (Fig. 3(a)-(c)) allowing for a better monolayer identification. The filtered image separates the monolayer colors from the rest of the image effectively, but it is challenging to identify an isolated monolayer from other regions that pass the threshold segmentation. To optimize the monolayer contrast and provide context for the monolayer, we combine the original negative image and the filtered image (Fig. 3 a-b). The combination enhances the monolayer visualization into the image and allows us to use the bulk material as a reference point to identify the monolayer location in the sample (Fig. 3 c).

 

Fig. 3. Deep learning training process. (a) Negative image capture. (b) The same negative image after the threshold segmentation mask. (c) To improve the monolayer detection, we combine these two images (a and b) and create a new filtered image. This new image separates the monolayer location (green color) from the bulk crystal (black color) and the background (purple color) making it easier to classify it within the image. (d) We show the complete detection methodology using the two convolutional neural networks (CNNs). First, the sample (1 cm x 1 cm) is imaged autonomously taking 100 images. This batch of images is passed through the digital image filters. After the deep learning algorithm, the system detects the monolayer characteristics (pattern) compared with its training data identifying just the monolayer images with 99.9% of accuracy. The complete process (image acquisition through detection) requires nine minutes to be completed.

Download Full Size | PDF

7. Deep learning detection

To optimize the total detection time, we train a pre-trained deep convolutional neural network, AlexNet, to recognize $Mo{S_2}$, $WS{e_2}$, and h-BN monolayers error-free without human interaction. Deep learning is based on convolutional neural networks (CNN) finding a pattern from a database through the interaction of a few or hundreds of hidden layers able to learn and detect different features in an image [24]. The training is performed by a target identification process where the CNN extracts the target characteristics from a data set, then identifies the same target in a new input image with high accuracy. We use a pre-trained network to extract the monolayer’s characteristics through a new training process from a group of images. Pre-trained convolutional neural networks (CNNs) require significantly less training [43]. AlexNet is a pre-trained network that was trained with 1.2 million high resolution images and can extract information of thousands of different object categories [44]. This algorithm performs the information extraction faster than training from scratch, which would require thousands of images to train a new network for image recognition. The network structure consists of eight learned layers, 650,000 neurons with five convolutional layers followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax [45].

We use two convolutional neural networks to effectively identify the monolayers from each sample with a high accuracy rate. The complexity in monolayer detection stems from the noise that is present in image captures and the fact that monolayers do not have a specific size or shape. We collected the training dataset by running the digital image filters (previously described) in an assortment of 600 samples of MoS₂, WSe₂, and h-BN to highlight potential monolayer images. Each sample is autonomously scanned with an X,Y stage and imaged using a 5X magnification objective in three minutes. We collect 100 images (per sample) to fully cover the sample area of 1 $c{m^2}$ using a 5X magnification objective with a field of view of approximately 1.1 mm x 1.3 mm. The resulting 60,000 images were visually analyzed by a person previously trained to detect monolayers (expertise). Not all images have monolayers, but after the DIP filters the monolayers are highlighted and can be identified quickly. Once a potential monolayer is detected, the operator carefully analyzes the filtered and negative image to confirm the visual detection. Filtered images with high presence of noise (difficult to judge) are analyzed with Optical Microscopy (OM). After the confirmation, the image is added to the training dataset. Collecting the 580 monolayer images to train the first CNN takes approximately 200 hours. Across the training dataset, the monolayers are located on different positions in the image with different degrees of noise. This database is used to train the first deep learning (DL) algorithm and identify the monolayers’ characteristics. To extract the monolayer (target) characteristics from each image in the dataset during the training process, we apply a ground-truth label map process to the images. This process consists in manually drawing a rectangle enclosing the monolayer to separate it from the rest of the image. At this point the images in the training dataset contain monolayers that are easy to identify by the operator. The label mapping process (i.e. enclosing the monolayer in a rectangle) takes around two human-hours with our operational conditions and level of expertise. We feed the X,Y coordinates of the rectangle to the deep learning algorithm, which extracts the features of the specific area. We apply this process to all the 580 images of 2592 × 1944 pixels and perform the training process [46]. The first CNN correctly identifies monolayers larger than 20 µm x 20 µm but misidentifies sections of bulk material as possible monolayers about 5% of the time (an average of five images out of one hundred). The error occurs because the shape and size of the bulk material are similar to that of a monolayer. To filter the bulk material images, we train a second DL algorithm that identifies the bulk material images and separates them from the monolayer images. For the second algorithm, we perform the same label mapping process to 623 bulk material images (miss-detections) produced by the first algorithm. The average time to perform this activity is approximately 3 human-hours. Since the second CNN runs only on the small subset of images identified by the first CNN, it only takes a few seconds to complete. We run the two CNN’s in a batch of 3000 new images that were not included in the training process and the system reduced the misidentification error to 0.133%, (an average of one image out of one thousand). The total deep learning algorithm with both CNNs runs in six minutes having an effectivity rate of 99.9%. We use the MatLab environment with an Nvidia GTX 1650 GPU to perform the training process and implement the deep convolutional neural network. The total computational training time for the two networks was 23 minutes. The training time could be vary depending on the computer specifications.

8. Results of the ML detection process using DIP and DL

We demonstrate a total detection time per sample of 1cm x 1cm of nine minutes with a 99.9% accuracy rate. We test our monolayer identification algorithm with a total of 140 exfoliated samples: 50 samples of $WS{e_2}$, 50 samples of $Mo{S_2}$, and 40 samples of h-BN. The test samples are not included in the training images. Then we run the DL algorithms using a computer with an Intel Core i7-9750H CPU 2.60 GHz to 4.50 GHz with an Nvidia GTX 1650 GPU with 6 cores. The two deep learning algorithms process one sample, that consists of a batch of 100 images, in six minutes. From the 140 exfoliated samples tested, the system detects 92 monolayers in total: 31 monolayers of $Mo{S_2}$, 35 monolayers of $WS{e_2}$, and 26 monolayers of h-BN. All the identified monolayers are larger than 20µm x 20µm. Figure 4 shows three monolayer images detected by the system, one for each material. The first column shows one of the 100 acquired images for MoS₂, WSe₂, and h-BN in the negative format (Fig. 4(a)-(c), respectively). The second column shows the same image after the digital image processing (Fig. 4(d)-(f)), and in the third column (Fig. 4(g)-(i)) we can see a digital zoom of the detected monolayer after the DL algorithms, which shows a correlation value over 99.5% with the training data. Even in the presence of significant debris (i.e. noise,) the algorithm successfully identifies the monolayer (Fig. 4(j)). Further experimentation confirms that by adjusting the illumination conditions in the capture process, the same Deep Learning algorithm can be used for small monolayer detection using high magnification objectives. When using a larger magnification objective (e.g. 10X) in the system, the brightness of the image increases. By adjusting the illumination conditions to produce the same baseline smaller monolayers can be detected. The tradeoff is that a larger number of images is needed to cover the same sample area, and the total detection time will increase. This algorithm can be used for the autonomous detection of new materials by producing the same baseline used in this work.

 

Fig. 4. We designed the detection algorithm to automatically highlight the monolayer by drawing a yellow rectangle showing its position in the image. Additionally, a text box with the confidence percentage is displayed. Here we show examples of images displayed by the software after running the algorithms. In the first column (a - c), we can see the negative image capture of the sample (step 1). In the second column (b - f), we see the images after the DIP (steps 2 through 4). In the third column (g - i), we see the monolayer detection image (step 5) with a confidence value of 99.5% with the target sought. The confidence values are determined by the deep learning algorithm comparing the new image characteristics with the training information. (j) $Mo{S_2}$ image in its negative format before the detection process. In k) we see at first glance that a high presence of noise is noticeable after the digital image processing filters. l) Image after the detection process. Here, the CNNs search for monolayer patterns according to their training process identifying the monolayers in the image. The system can identify more than one monolayer in an image if it matches with the monolayer characteristics even if the monolayers are isolated.

Download Full Size | PDF

To confirm that the detected areas by the DL algorithms are indeed monolayers, we randomly select 12 samples of $Mo{S_2}$ and 11 samples of $WS{e_2}\; $ from the 92 detected monolayers and perform photoluminescence characterization. The PL results show the typical emission spectrum of a monolayer (Fig. 5(b), c) and confirm that all the detected areas are monolayers. We fit the PL spectrum to a Voigt function [47,48] that is a convolution of Lorentzian and Gaussian functions (Fig. 5(a)). We use optical microscopy (100 X objective) in 7 samples of h-BN to verify that the detected areas are monolayers.

 

Fig. 5. Monolayer photoluminescence results that confirm the detected regions are monolayers. (a) Photoluminescence data for a $WS{e_2}$ fit to a Voigt function. (b) and (c) Measured PL spectra of WSe₂ and MoS₂.

Download Full Size | PDF

9. Conclusion

Deep learning combined with digital image processing is a powerful platform for image recognition of two-dimensional materials. Pre-trained convolutional neural networks require significantly fewer training instances compared to ones developed for a specific application. Using a pre-trained convolutional neural network and digital image processing, we developed an automated detection system for two-dimensional materials that significantly reduces the time needed to identify monolayers. The automated system enables a low-cost, high-speed monolayer identification process with an accuracy of 99.9% without the need of advanced characterization techniques such as Raman, PL, or AFM. Our demonstration paves the way for automated production of monolayer materials for use in research and industrial settings by greatly reducing the processing time.