1. Introduction

Leakage radiation refers to a leaky wave of surface plasmon polariton (SPP) modes through a dielectric substrate and has drawn interests due to the possibility of direct visualization and analysis of SPP propagation. Since the first observation of leakage radiation of scattered light in rough surfaces [1], extensive research has been performed on its physical properties [2,3]. Scattering by small defects on a flat film or grating coupling can act as a source of evanescent momentum that is sufficient for the light waves to match with the SPP dispersion relation. The SPP waves are subsequently emitted back into the glass substrate as leakage radiation and form ring-like distribution in the narrow angular range of the imaging back focal plane. The intensity of a leakage radiation image can be used to monitor SPP waves [4].

Various methods have been proposed to produce leakage radiation in addition to introducing small defects. For example, NSOM tips have been used to launch SPPs on a metal film [5] and to excite SPP waves to induce luminescence of quantum dots [6]. Single and periodic nanostructures were fabricated by an electron-beam to be scattered by propagating SPP waves into leakage radiation modes for observation in the far-field [7,8].

The use of leakage radiation in practical monitoring and sensing applications has been limited largely due to the difficulty that arises from the uncertainty in the relationship between excitation and propagation of SPP waves in the near-field and leakage radiation measured in the far-field. This is exacerbated by the optical complexity, not to mention the bulky set-up needed to acquire leakage radiation images. In this work, we explore the possibility of predicting leakage radiation from the images of defects that excite and scatter SPP waves. Attention in particular has been drawn to leakage radiation by random nanoislands. Nanoscale islands of a size smaller than the diffraction-limit can be fabricated without any lithographic techniques and therefore have been explored widely in many applications, e.g., in biological sensing [9–12], Raman signal enhancement [13–19], spontaneous light emission [20], resonant energy transfer [21,22], solar cells [23,24], fluorescence and photoluminescence switching [25–29], localization microscopy [30,31], light-emitting diodes [32], and broadband photodetectors [33]. Nanocomposite substrates were also used for enhanced surface plasmon resonance (SPR) detection [34–45] and to investigate its theoretical sensitivity limit [46]. Moreover, near-field localization was induced by random gold-glass nanocomposites [47,48] and applied to plasmonic heating in microfluidics [49]. Morphologies of nanocomposite islands have been scrutinized with analytical tools such as AFM and TEM [50,51].

Previous works demonstrated that leakage radiation patterns formed in the Fourier space can be analyzed using the Green’s function method and coupled-mode theory [52–55]. In order to perform theoretical analysis, additional information regarding nanostructured surface such as height, size, and distribution would be needed. Because the acquisition of such information demands expensive and time-consuming morphological measurements, for example, with AFM [48], this study instead explores the feasibility of machine learning-based prediction of leaky momentum with bright-field (BF) images of plasmonic random nanosubstrates. Understanding and predicting leakage radiation characteristics by machine learning from BF data are much simpler and require less sophisticated equipment for measurement. Improved accuracy by machine learning in many microscopic image analysis and design applications has been well-documented [56–60]. The prediction of leakage radiation would make it available for a wider range of applications.

2. Methods

2.1 Optical set-up for bright-field and leakage radiation microscopy

The set-up to acquire optical images of BF and leakage radiation of nanoislands in the image and back focal plane (BFP) from an objective lens is illustrated in Fig. 1(a). Two light sources were used, one for BF images (OSL2, Thorlabs, Inc. USA) and the other for leakage radiation excitation (MRL-FN-721-30 mW, $\lambda$ = 721 nm, CNI, China). After passing through optical components for polarization and beam profile manipulation, incident light impinges on a nanoisland substrate through an objective (UAPON 100XOTIRF, NA = 1.49, Olympus, Tokyo, Japan) at the resonance angle. The resonance angle was determined as the incident angle at which the intensity of a reflected beam was minimized. An objective lens on the top (LMPlanFLN, NA = 0.5, Olympus, Tokyo, Japan) and a bandpass filter (590/43, Edmund Scientific Corporation, USA) were used to obtain BF images of nanoislands. A BFP image of leakage radiation from the nanoisland substrate was recorded by a scientific CMOS (sCMOS) camera (Zyla, Andor Technology Ltd., UK) which was relayed by a series of lenses through a spatial filter with a bandpass filter (ET720/60m, Chroma Technology Corporation, USA). A spatial filter blocked the reflected beam from the nanoisland substrate whose angle corresponded to the resonance angle. An electron-multiplying charge-coupled device (EMCCD) camera (imagEM, Hamamatsu Photonics, Japan) captured BF images of nanoisland substrate through a bandpass filter (ET590/50m, Chroma Technology Corporation, USA) and a neutral density filter. The observation area was adjusted by an iris which was omitted in the optical schematic of Fig. 1(a).

 

Fig. 1. (a) Schematic illustration of optical set-up (WL: white light, F; filter, OBJ: objective, IO: immersion oil, BFP: back focal plane, LR: leakage radiation, DBS: dichroic beam splitter, L: lens, NDF: neutral-density filter, PMF: polarization-maintaining fiber, M: mirror, and SF: spatial filter). An inset represents a SEM image of a gold nanoisland substrate (BF: bright-field, viewing angle = 55$^{\circ }$). (b) Dataset of post-processed BF and BFP images. (c) ANnet based Machine learning algorithm for BFP image prediction.

Download Full Size | PDF

2.2 Nanoisland sample fabrication

For observation of leakage radiation of SPP with randomly distributed nanoislands, a BK7 substrate was cleaned in acetone, isopropyl alcohol, and distilled (DI) water with sonication, after which a 25-nm thick gold film was formed by thermal deposition. Gold random nanoislands were produced by annealing process after keeping the thin film sample on a hot plate at 350 $^{\circ }$C for 4 hours. After annealing, additional layers of chrome and gold with 2 and 50-nm thickness were deposited for uniformity between randomly distributed gold nanoislands. An inset of Fig. 1(a) represents an SEM image of the nanoisland substrate with a viewing angle of 55$^{\circ }$.

2.3 Post-processed raw images

BF and BFP images were post-processed to clarify the relation between leakage radiation characteristics and morphology of nanoislands. We have applied image flattening along the x- and y-axis to the BF images acquired from the optical setup in Fig. 1(a). The pixel values of BF images can be categorized into three regions: nanoislands, gold bare film inside optical iris pupil and the iris. The pixel value of a flattened image was normalized so that these three regions can be differentiated in the image histogram. In the case of BFP images, only the image normalization was conducted for the measurement dataset by eliminating the pixel values under the noise level of 600 in arbitrary unit. The post-processed raw images are shown in Fig. 1(b).

2.4 Input image processing for noise removal

After having obtained the input nanosubstrate images and their corresponding target leaky momentum images, an image processing step is applied to the input nanosubstrate image for noise removal. A bilateral filter [61] which can effectively reduce the image noise while preserving the edge features has been adopted for the image processing. The implementation of the bilateral filtering has utilized the Matlab’s library function [62] where all the parameters have been set to their default values except for the degree of smoothing which is empirically set at 20. Figure 2 shows some sample input-output images (shown in the top and the bottom rows) from the optical set-up together with the corresponding bilaterally filtered input images (shown in the middle row).

 

Fig. 2. This figure shows five sample images captured at different field-of-views (FOV) for predicting the leakage momentum of plasmonic random nanosubstrate. Row (a) shows the experimentally obtained raw images of bright-field images (BFI). Row (b) shows the histogram equalized and bilaterally filtered image of BFI. Row (c) shows the back focal plane images (BFPI) of leakage radiation after post-processing.

Download Full Size | PDF

2.5 ANnet for image-to-image prediction

Prediction of the leaky momentum of plasmonic nanosubstrate is a challenging task in view of the difficulty in collecting a representative set of data for good learning. In this work, the ANnet model [63] has been adopted for this image learning task in view of its good kernel and range mapping property without needing a large set of training data. Comparing with many deep learning models [64,65], the ANnet model trains a multilayer network by means of gradient-free learning.

Denote each input image of width $w$ and height $h$ of the nanosubstrate by $\mathbf {I}\in \mathbb {R}^{w\times h}$. A vectorized form of this image can be written as $\mathbf {I}'\in \mathbb {R}^{1\times d}$ where $d=wh$. Suppose we have $m$ samples of these images to be used as input for learning. These input samples for training can be stacked as $\mathbf {X}\in \mathbb {R}^{m\times d}$. Correspondingly, we have $m$ samples of target leaky momentum images, each being $\mathbf {L}\in \mathbb {R}^{u\times v}$ which can be vectorized as $\mathbf {L}'\in \mathbb {R}^{1\times p}$ where $p=uv$. The stacked matrix of the output target is written as $\mathbf {Y}\in \mathbb {R}^{m\times p}$.

A two-layer ANnet has been adopted to learn the target $\mathbf {Y}$ with corresponding input $\mathbf {X}$ based on

Due to the linear output mapping between $\sigma (\mathbf {X}\mathbf {W}_1)$ and the real output weight values $\mathbf {W}_2$ in 1, the predicted output of the network might contain negative intensity values. Since we are focusing on the bright part of the image which represents the momentum of the leakage radiation, a ReLU activation has been included at the prediction output. This activation process removes the noise caused by the negative output values hence enhances the prediction performance without affecting our observing target. Figure 1(c) provides an overview of the machine learning process.

3. Results

3.1 Data set and evaluation protocol

A data set containing 1099 pairs of the nanosubstrate and the leaky momentum images, with each image having a resolution of $160\times 160$ and $300\times 300$ respectively, has been collected for experimentation. Among these image pairs, 1000 pairs have been randomly selected as training data and the remaining 99 pairs have been used for test evaluation. In the experiment, this process of random pick to form the training and test sets is repeated 10 times in order to form strong statistical testing evidence. The results will be reported in terms of the average and standard deviation of these 10 trials. In order to cope with the large running memory utilizing the 1000 image pairs for training, both the nanoisland substrate and its leaky momentum images are resized to $100\times 100$ in grayscale based on the bicubic interpolation.

The training and testing evaluations have been performed on an i9 desktop (3.10GHz) with 32GB RAM and a RTX 2080Ti GPU. In order to evaluate the prediction performance, the mean-squared-error (MSE) and the mean-absolute-error (MAE) between the predicted output and the target image are adopted as performance metrics. Apart from quantitative evaluation, samples of predicted output and the target image are reported for qualitative visual inspection.

In terms of comparison with competing state-of-the-art methods, two convolutional neural network based U-net [64] architectures, which are well-known for its effectiveness in image segmentation [66], image reconstruction [67] and image generation tasks [68], are implemented. The first adopted conventional U-net utilizes 4 convolutional layers for both the contracting path and the expanding path with the $tanh$ activation function. The batch normalization and dropout with 5% rate have also been deployed during U-net training. As the loss function, the MSE has been adopted for learning based on gradient backpropagation.

Apart from the original U-net, a specialized version called the Holo-Unet [69], which was tailored for denoising digital holograms, has been implemented for comparison. Although this network was not designed directly for leakage radiation characteristics, its architecture had taken advantage of the Fourier domain and phase consideration relevant to our application. When implementing the Holo-Unet, we follow strictly the given architecture and loss function according to [69] except for the input size in order to match with our training images. In order to pass through 6 encoding layers with down samplings, both the nanoisland substrate and its leaky momentum images are resized to $128\times 128$ in grayscale. Based on an empirical tuning observation, we report the results of U-net and Holo-Unet utilizing a training of 50 epochs with a batch size of 50 samples. During the tuning stage, 20% of the training samples have been utilized as the validation set.

3.2 Prediction performance over training sample size

In view of the difficulty in collecting a large set of the plasmonic nanosubstrate images, an evaluation of the impact of training sample size would be meaningful. In this experiment, the sample size of the training set has been varied to observed its impact on the prediction MSE and MAE results. Among the 1099 image pairs, the testing set contains 99 image pairs while the training set has been varied from 100 to 1000 image pairs. For all the U-net, Holo-Unet and the ANnet models, their prediction outputs have been put through a min-max normalization to keep the output values within the interval [0,1] before computation of the MSE and MAE.

Figure 3 shows the average values over 10 trials for the test MSE and MAE of U-net, Holo-Unet and ANnet being plotted over different training sample sizes. From this figure, both the MSE and the MAE show an improvement of having a lower prediction error over increment of training size for U-net where the best performance using 1000 training samples is 0.0244 in terms of MSE and 0.0741 in terms of MAE. For Holo-Unet, both the MSE and the MAE show a similar trend of having a lower prediction error over increment of training size with the best prediction of 0.01871 for MSE and 0.04868 for MAE at 800 training size. Attributed to its domain specific design, the Holo-Unet shows better performance than the original U-net over all settings. For ANnet, the MSE and MAE errors are also seen to reduce with respect to the increment of training size. However, these reduction of MSE and MAE prediction errors are seen to saturate from around 600 training samples onwards with little improvement thereafter. The MSE and MAE values respectively fall within the ranges of 0.0046 to 0.0028 and from 0.0412 to 0.0271 for ANnet without bilateral filtering. With the use of bilateral filtering for input image pre-processing, the performance of ANnet shows a similar consistent trend of improvement for all settings of training sizes. The performance of ANnet with bilateral filtering also shows a saturation trend after 600 training samples. The best prediction performance of ANnet with bilateral filtering is found to be 0.00258 for MSE and 0.02262 for MAE. This shows a clear advantage of ANnet over U-net based models with preference to inclusion of a bilateral filter in terms of the prediction accuracy.

 

Fig. 3. The observed (a) MSE and (b) MAE of test prediction being plotted over different number of training samples for the compared models.

Download Full Size | PDF

Table 1 lists the best observed (among training sample sizes from 100 to 1000) averaged test MSE and MAE together with their standard deviations based on the 10 training and test trials. For ANnet without the bilateral filter, the best results are obtained from the model which was trained with 700 samples. ANnet with bilateral filter shows the best results from the model which was trained with 900 samples. For U-net and Holo-Unet, the best results are respectively obtained by models trained with 1000 and 800 samples. Apart from having a lower prediction error, the ANnet model with and without bilateral filtering also shows low standard deviation which can be translated as stable prediction performance. The U-net models show much higher MSE and MAE values with high standard deviations comparing with that of ANnet.

Table 1. The averaged (from 10 runs of random data partitioning) MSE, MAE and their standard deviations of the implemented models.

View Table

3.3 Qualitative evaluation

In terms of qualitative evaluation, some sample results of the best and the worst performed ANnet, U-net and Holo-Unet are shown in Fig. 4 for visual inspection. Five sample target images for learning are shown in row (a) of Fig. 4. The selected ANnet predictions which had utilized 100 and 900 training samples are shown respectively in row (b) and row (c) of Fig. 4. The U-net and Holo-Unet had respectively utilized 1000 samples and 800 samples for training based on their best averaged performance. These sample images are selected from the 99 test images based on their best and worst MSE performance according to ANnet. Since the difference between those input images with and without bilateral filtering cannot be distinguished visually, only the bilateral filtered images for ANnet are shown.

 

Fig. 4. Results of predicted leaky momentum images by ANet, U-net and Holo-Unet. Row (a) shows the experimentally obtained target images acquired from different FOVs. Rows (b) and (c) show respectively the predicted image from ANnet based on 100 and 900 training samples. Row (d) and (e) show respectively the predicted images from U-net and Holo-Unet based on 1000 and 800 training samples. Results in shaded boxes show sample predicted images with high (blue) and low (yellow) performances. Columns one and two are chosen based on the best performed ANnet. Column three is chosen based on the best performed U-net. Columns four and five are respectively chosen based on the worst performed ANnet and U-net.

Download Full Size | PDF

As shown in the best predicted samples (the first three columns of Fig. 4(b) and (c)), the predicted images using the ANnet hits the k-space momentum of surface plasmon leakage radiation as shown in the target image. When the model utilizes 900 training samples with 7 million network weight parameters, it matches most of the brightest k-space momentum of surface plasmon leakage radiation. Also, the two layer ANnet appears to predict relatively well when only 100 training samples with 1 million network weight parameters have been utilized for learning. The predicted images with high MSE values (see the two columns on the right in Fig. 4(b) and (c)) tend to fail in predicting the k-space momentum by plotting either too few or too much k-space momentum. The U-net, which utilizes 1,885,057 trainable parameters and 2,560 non trainable parameters with its predicted images, shows only the white circular shape of the target image and failed to hit the correct k-space momentum of leakage radiation. The Holo-Unet, which utilizes 31,471,297 trainable parameters and 12,032 non trainable parameters, shows a spotted ring shape, also failed to hit the correct k-space momentum ring of leakage radiation.

Next, we plot the average of all 99 predicted images in order to compare the prediction outcomes from learning with 700 and 1000 samples. Figure 5 shows the mean image of (a) the predicted LR images from ANnet trained with 700 samples, (b) the target LR images and (c) the predicted LR images from ANnet trained with 1000 samples. As shown in Fig. 5(d) and (f), the ANnet trained with 1000 samples shows brighter prediction difference (higher error) than that of ANnet trained with 700 samples along the k-space momentum ring. However, according to Fig. 5(e) and (g) which respectively show the histogram equalized images of Fig. 5(d) and (f), the ANnet trained with 700 samples shows much noises beside the inner ring region. This observation suggests a trade-off between the lower sample size of 700 with much inner ring noises and the higher sample size of 1000 with higher prediction difference on the ring along the saturation region depicted in Fig. 3.

 

Fig. 5. The Averaged image of (a) predicted LR images from the ANnet trained with 700 samples, (b) target LR images, (c) predicted LR images from the ANnet trained with 1000 samples and their difference maps (d), (f). (e) and (g) respectively shows the histogram equalized images of (d) and (f) to observe the intermediate values between the bright k-space momentum ring and the dark background.

Download Full Size | PDF

3.4 Training and test processing times

Figure 6 shows (a) the training time and (b) the testing time plotted over training sample sizes. For the training time, we measure the total elapsed time to train the network from scratch using the given number of training samples. In terms of training effort, all the compared methods show an increasing training time over the training sample size. The training time for U-net (utilizing RTX 2080Ti GPU) ranges from 15 seconds to 132 seconds. The training time for Holo-Unet is much longer than that of Unet (ranges from 73 seconds to 515 seconds) since it utilizes a deeper structure and more complex loss function. The ANnet with bilateral filtering shows a longer training time (utilizing only CPU) than one without bilateral filtering. The training time for ANnet ranges from 0.03 seconds to 3 over seconds. This result shows an average of 140 times faster training speed of ANnet than that of U-net considering all the training sample sizes. With the bilateral filtering, ANnet shows an average of 44 times faster training speed over that of U-net. The Holo-Unet shows a further 4 times of longer training time than that of U-net.

 

Fig. 6. (a) training time and (b) testing time, plotted over the number of training samples. The training time records the total elapsed time to train the network from scratch using the given number of training samples and the test time records the total elapsed time for predicting all 99 test samples.

Download Full Size | PDF

Since the implemented methods take very short time for predicting each sample, the test processing time can be sensitive to background processes. Thus, we measure the total test processing time for predicting all the 99 test samples. As shown in Fig. 6(b), all methods show below 0.37 seconds of total test processing time. Due to the increase in network size according to the training matrix, the ANnet without bilateral filtering shows an increasing test processing time with respect to the training size. The average value of test times over all training sizes is 0.0126 seconds. This is much lower than that with the bilateral filter which consumes an average test processing time of 0.2169 seconds. For ANnet with bilateral filtering, the relatively constant test processing times over the training sizes show that the time consuming bilateral filtering had masked the relatively small incremental trend of the ANnet testing speed. Finally, the testing times of U-net and Holo-Unet are respectively 0.2289 and 0.3558 seconds.

4. Conclusion

We have explored machine learning-based prediction of leakage radiation characteristics that arise from surface plasmon polariton from BF images of nanoislands. A two-layer ANnet model has been deployed to learn the relationship between the leaky momentum and the plasmonic random nanosubstrate. The model has shown to learn well with a relatively small number of training samples at the cost of utilizing a relatively large number of network weight parameters. In terms of the MSE and MAE, the proposed framework has shown promising leaky image prediction with much shorter training and testing times, when compared with that of the popular U-net based models, which are well known for its image-to-image prediction. Based on the qualitative observations, the ANnet model was found to produce fewer failure cases in predicting the correct momentum of leakage radiation. The proposed framework is expected to significantly simplify the process of constructing the leaky radiation images without needing sophisticated equipment.