1. INTRODUCTION

Structural color filters can display various colors by selectively transmitting or reflecting a specific wavelength by varying structural parameters rather than material components in the visible region [1]. They have received enormous interest recently due to their potential applications in chromatic display [2,3], color printing [4,5], optical encryption [6,7], solar cells [8], and so on. They largely exceed the conventional colorant-pigment-based filters in multiple aspects including nontoxicity, great scalability and durability, high resolution, and easy tenability [9]. Generation of structural color normally employs resonances (e.g., plasmonics, Mie scattering, guided mode resonance) from subwavelength patterns [10–13]. However, all these require nanoscale patterning, which involves complicated fabrication steps and can be cost prohibitive for high-volume and large-area applications.

Planar thin-film structures based on Fabry–Perot (F-P) cavity resonances are an alternative structural color filter technology based on the thin-film interference [14]. A typical F-P resonator consists of a lossless dielectric that is sandwiched between two reflective metal layers. The function of color filtering is achieved by multiple round-trip phase delays of electromagnetic waves in the F-P resonator [15]. The resonant peak location can therefore be controlled by varying the dielectric layer thickness to obtain different colors. The full width at half-maximum (FWHM) of the peak, which relates to the color purity and brightness, can be tuned through the two metal layer thicknesses. Compared with the pattern-based filters, F-P-cavity-based structural color offers a much lower-costing and higher-scalability way of structural color manufacturing while offering larger color gamut and high color purity and contrast [16]. The high-index materials of phase compensation overlayers can also achieve great angle insensitivity for both transverse-magnetic (TM) and transverse-electric (TE) polarizations [17]. There are several works published on the aspect of improving the performance of F-P cavity-based structural color in both transmissive and reflective modes [18–20]. In addition, the F-P-cavity-based color filter also demonstrates high lateral resolution. Wang et al. reported that its color crosstalk is comparable to those of plasmonic filters and pigmented filters with the same sizes [4]. A minimum pixel dimension of 500 nm (ca. 50,800 dpi) has also been successfully demonstrated, showing a great potential for F-P-cavity-based color filters to be applied in high-resolution colorization.

An important aspect for structural color is the ability to design a structure that can accurately display the desired color [21]. This conventional design task is realized by a trial-and-error method in which an initial random design is converged to the desired design through iterative optimization. Identification of a reasonable design therefore often requires an experienced designer with prior knowledge of the problem and a significant amount of calculations or simulations, which could be prohibitively slow as the complexity of the structure increases. However, this paradigm has recently been changed by the unprecedented development of deep learning techniques [21–23].

Deep learning is an important subset of machine learning in which multilayered artificial neural networks are utilized to achieve high-accuracy predictions and classifications [24]. Several works have been reported recently in which neural networks were used to inversely design the nanostructure and material to achieve desired optical responses [25–31]. Before the networks can perform the intended functions, a training process needs to take place in which a dataset of structural parameters to color relations is required. This dataset normally involves a large amount of relations, and it needs to be generated by theoretical calculation or simulation. However, this is a one-time investment, and no more computational resources will be consumed once the network is properly trained. Pioneering works have also been reported on deep-learning-aided structural color design [32–36]. Hemmatyar et al. designed and optimized the hafnia array based all-dielectric metasurfaces via neural network to generate a wide color gamut [32]. Gao et al. reported a structural color inverse design by employing a bidirectional neural network [35], which was first proposed by Liu et al. [37]. The model consists of a fully connected inverse network that directly connects to a pretrained forward network. The inverse network can produce the design of the structural parameters for desired colors, while the forward network predicts the colors from the structural parameter inputs. The training of the inverse network is conducted by feeding the inversely designed structures directly into the pretrained forward network, and the network is optimized by minimizing the difference between the predicted color and the input color through backpropagation. This model has been widely used in different nanophotonic applications, as it overcomes the issue of nonuniqueness [31,38,39].

Although previous works have achieved remarkable progress in the structural color inverse design, one remaining challenge is the relatively low design accuracy (large color differences between design and target) even when extremely low validation loss is achieved. One key reason for that is the nonuniformity of the color space from which the loss function was defined. The loss function measures the difference between the network predicted value and the true value in the dataset. It is crucial for neural networks, as the training is a process of minimizing the loss function [40]. An unsuitable selection of loss function may lead to a very low loss function but relatively higher prediction error [41]. Defining loss function in a nonuniform color space such as CIE 1931-XYZ means the same Euclidean distance among the $\mathbf{X}\mathbf{Y}\mathbf{Z}$ vectors may signify different color differences, resulting in biased optimization against some colors [42].

Here we report, to the best of our knowledge, the first inverse design of F-P cavity structure using deep learning through a bidirectional artificial neural network. This structure enables the production of a significantly wider color gamut that is over 215% of sRGB color space, which is essential for real display applications. By defining the training loss function in the uniform CIE 1976-Lab color space where the same Euclidean distance indicates same color difference, our network is able to achieve a much higher design accuracy with average color difference $\mathrm{\Delta }{E}_{2000}$ below 1.2. The high design efficiency of the network is also evaluated by comparison to the evolutionary algorithm, which shows over 100,000 times savings of computational resources. The demonstration of this work offers exciting prospects for deep learning techniques to be used to achieve accurate and efficient structural color designs for a wide range of different applications.

2. DATASET GENERATION

The schematic diagram of the transmissive F-P-cavity-based color filter employed in this work is illustrated in Fig. 1(a). A transmissive-type of color filter was chosen because of its wide application in spectrometers, CMOS image sensors, and liquid crystal displays [43–45]. Compared with reflective filters, transmissive filters enjoy the advantages of single-mode operation, which could lead to a higher color purity. However, it is worth mentioning that a similar approach can also be used on the reflective type. The system has a trilayer metal-insulator-metal (MIM) films stack on the quartz substrate in which a ${\mathrm{SiO}}_2$ dielectric layer is sandwiched between two Ag metal layers. Optical interference occurs when white light enters the cavity which filters out the wavelengths that are not matching the resonant wavelength of this multilayer system. With the materials of the dielectric and metal layers fixed, the most critical parameters here affecting the resonant wavelengths are the thickness of each layer, which are represented as ${\mathit{d}}_1$, ${\mathit{d}}_2$, and ${\mathit{d}}_3$ as shown in Fig. 1(a). In this work, the ranges of ${\mathit{d}}_1$, ${\mathit{d}}_2$, and ${\mathit{d}}_3$ are set to be 0 to 50 nm, 0 to 1000 nm, and 0 to 50 nm to allow a large color gamut coverage. Only integer values are selected in this work to ensure the compatibility with fabrication techniques. A total of 101,000 parameter combinations are randomly generated, and the corresponding transmissive spectra from 380 to 780 nm are computed by the multiple beam interference formulas [16]. However, color is not a property of electromagnetic radiation but a subjective perception of an observer. Color-matching functions are then required to convert the transmissive spectra into corresponding color coordinates in CIE 1931-XYZ color space to generate a dataset containing 101,000 parameters $\mathbf{D}$ $(d_1,d_2,d_3)$ to color $\mathbf{X}\mathbf{Y}\mathbf{Z}$ $(X,Y,Z)$ relations. This dataset is divided into three groups for training (90,000), validation (10,000), and testing (1000) purposes. All colors generated in the training and validation dataset are plotted in the CIE 1931-xy chromaticity diagram in Fig. 1(b). It is clear that our Ag-${\mathrm{SiO}}_2$-Ag F-P-cavity-based color filter in this work can achieve a substantially larger (ca. 215%) color gamut than the sRGB color space (plotted by red lines for reference). The large gamut of coverage is one of the advantages of F-P-cavity-based structural color and is particularly beneficial for real applications such as display and full-color nanoprinting [4,19]. By varying the thickness of the dielectric layer (${\mathit{d}}_2$), the transmission peak could be swept across the whole color in the visible light range. The modification of the Ag layer thicknesses (${\mathit{d}}_1$ and ${\mathit{d}}_3$) serve to further tune the FWHM of the transmittance peak, resulting in a large gamut of coverage. The testing dataset is plotted in Fig. 1(c) (referred to as the testing set below). To further test the robustness of our networks, we also generated an additional testing set with 7000 colors that are uniformly distributed on the CIE 1931-xy chromaticity diagram [shown in Fig. 1(d) and referred to as the uniform testing set below].

 

Fig. 1. (a) Schematic illustration of a transmissive F-P-cavity-based color filter with an MIM structure. (b) The training and validation dataset generated by the F-P cavity plotted in a CIE 1931-xy chromaticity diagram. The red triangle depicts the boundary of sRGB color space for a guide of the eye. (c) The original 1000 testing data and (d) the 7000 uniformly distributed testing data plotted in the CIE 1931-xy chromaticity diagram.

Download Full Size | PDF

3. FORWARD NEURAL NETWORK CONSTRUCTION

A forward neural network (FNN) was first trained to obtain accurate prediction of colors based on the layer thicknesses. Prior to training, a loss function needs to be established. Although our original dataset converts the spectrum into a CIE 1931-XYZ tristimulus vector, it is not a suitable output for loss function definition due to its nonuniformity [42]. sRGB color space is also not ideal, as the conversion between $\mathbf{X}\mathbf{Y}\mathbf{Z}$ and $\mathbf{s}\mathbf{R}\mathbf{G}\mathbf{B}$ is not reversible when the color is outside of the sRGB color space. On the other hand, the CIE 1976-Lab color space has a one-to-one correspondence to the CIE 1931-XYZ but with much better uniformity, rendering it a more suitable color space for accurate color difference identification. In fact, the color difference function CIE $\mathrm{\Delta }{E}_{1976}$ is defined by the Euclidean distance of two Lab vectors $(L,a,b)$. This property is particularly beneficial in neural network training. By defining the loss function to be mean squared error (MSE) between the predicted and original $\mathbf{L}\mathbf{a}\mathbf{b}$ values, it can be directly converted to the actual color difference ($\mathrm{\Delta }{E}_{1976}$) and enable higher accuracy. Figure 2(a) shows the summary of the dataset preparation process for the FNN. After obtaining $\mathbf{X}\mathbf{Y}\mathbf{Z}$ from the spectrum, it was converted to $\mathbf{L}\mathbf{a}\mathbf{b}$, which was then used to construct the loss function and identify color difference. The architecture of the FNN is illustrated in Fig. 2(b) and is composed of a fully connected neural network (NN) including one input layer, one output layer, and several hidden layers. It takes the parameter $\mathbf{D}$ as input and outputs the $\mathbf{L}\mathbf{a}\mathbf{b}$, which can be converted to other color vectors such as $\mathbf{X}\mathbf{Y}\mathbf{Z}$ and $\mathbf{s}\mathbf{R}\mathbf{G}\mathbf{B}$ for different applications.

 

Fig. 2. Forward neural network for predicting F-P cavity structural colors. (a) The relationships of different parameters in the dataset. The single arrow means the transformation is unidirectional, the double arrow means the transformation is reversible. (b) The architecture of the forward neural network with input layer of geometric parameter D, hidden layers, and output layers of Lab color values.

Download Full Size | PDF

During the deep learning process, the training data group is fed to the FNN to continuously adjust the weight and bias of each connection with every batch of data in epochs. This is achieved by backpropagation of the loss function. The selection of the hyperparameters (i.e., number of hidden layers and neurons per layer) is crucial to the performance of the network [21]. A systematic study was therefore conducted to investigate the impact of hyperparameters for this FNN. The CIE $\mathrm{\Delta }{E}_{2000}$ color difference was chosen here to provide a better quantification of the FNN performance. Similar to the $\mathrm{\Delta }{E}_{1976}$, the $\mathrm{\Delta }{E}_{2000}$ color difference is also a function of the two $\mathbf{L}\mathbf{a}\mathbf{b}$ values but corresponds better with the way in which human observers perceive small color differences and hence is used as the metric for design accuracy in this work [46]. The $\mathrm{\Delta }{E}_{2000}$ can be classified into five groups: 1) $\mathrm{\Delta }{E}_{2000}<1$, it can be considered no color difference; 2) $1<\mathrm{\Delta }{E}_{2000}<2$, the difference can be observed by experienced persons; 3) $2<\mathrm{\Delta }{E}_{2000}<3.5$, the difference can be observed by unexperienced persons; 4) $3.5<\mathrm{\Delta }{E}_{2000}<5$, a clear difference can be noticed; and 5) $5<\mathrm{\Delta }{E}_{2000}$, two different colors are observed [47]. The distribution of $\mathrm{\Delta }{E}_{2000}$ from the testing set of each FNN is plotted in Fig. 3(a) as a function of layer number. It provides a clear indication on the color prediction performance of each FNN. The average $\mathrm{\Delta }{E}_{2000}$ values (blue squares) can therefore be used to provide a quantified metric for the performance. It can be observed that the average $\mathrm{\Delta }{E}_{2000}$ in the testing set decreases from 1.87 with two hidden layers to 0.44 with seven hidden layers, indicating that the FNN with seven hidden layers has the best performance in our optimization range. The impact of number of neurons per layer was also investigated while the number of layers was fixed at seven.

 

Fig. 3. Forward neural network training for predicting F-P cavity structural colors. The histogram of the probability and average values of $\mathrm{\Delta }{E}_{2000}$ of the FNNs with different (a) hidden layer number, (b) neuron number per layer, and (c) the FNN with seven hidden layers and 250 neurons in each hidden layer for different dataset size. (d) The training loss curves for defining the loss function in CIE 1931-XYZ and CIE 1976-Lab color spaces. (e) The probability histogram and average values of $\mathrm{\Delta }{E}_{2000}$ comparisons while the loss function is defined in CIE 1931-XYZ color space and CIE 1976-Lab color space. (f) The selected colors in the CIE 1931-xy chromaticity diagram with the boundary of each ellipse representing the colors that have a $\mathrm{\Delta }{E}_{1976}$ of 6 to the selected color.

Download Full Size | PDF

Similar to the optimization of hidden layer numbers, the $\mathrm{\Delta }{E}_{2000}$ distribution and average values are plotted in Fig. 3(b).The average $\mathrm{\Delta }{E}_{2000}$ in the testing set plunged sharply from 3.05 to 0.44 with the increase of neuron number from 10 to 50 and subsequently bottomed out at 0.38 for 250 neurons. Therefore, the architecture of the FNN is optimized with seven hidden layers and 250 neurons in each hidden layer. The influence of the dataset size was also investigated as shown in Fig. 3(c). The results suggest that sufficient dataset size (over 50,000 in our case) is required for the network to achieve a good performance of an average $\mathrm{\Delta }E$ value below 0.5 in our case.

The high accuracy of our FNN is further confirmed by evaluation using the uniform testing set, resulting in a $\mathrm{\Delta }{E}_{2000}$ of 0.35. We believe this high accuracy can be ascribed to the use of uniform CIE 1976-Lab color space for the network training. An FNN with the same network architecture using $\mathbf{X}\mathbf{Y}\mathbf{Z}$ as the output was also trained for comparison as shown in Figs. 3(d) and 3(e). Although a lower MSE (ca. ${10}^{-6}$) was achieved by the FNN with $\mathbf{X}\mathbf{Y}\mathbf{Z}$ output, the $\mathrm{\Delta }{E}_{2000}$ distribution is significantly poorer than the one with $\mathbf{L}\mathbf{a}\mathbf{b}$ output with a decrease of $\mathrm{\Delta }{E}_{2000}$ values observed from 1.03 to 0.38. To better explain the advantage of our approach, Fig. 3(f) presents a selection of colors in the CIE 1931-xy chromaticity diagram with the boundary of each ellipse representing the colors that have a $\mathrm{\Delta }{E}_{1976}$ of 6 to the selected color. It is clear that the ellipses in the green and red regions are larger than that in the blue region due to the nonuniformity of the CIE 1931-XYZ color space. One could obtain a very small $\mathrm{\Delta }\mathbf{X}\mathbf{Y}\mathbf{Z}$ in the green region, but the actual reduction of color difference might be limited. This suggests that minimizing $\mathrm{\Delta }\mathbf{X}\mathbf{Y}\mathbf{Z}$ values, especially in the green and red regions, will not be as effective in reducing color difference as the approach suggested in our work. This proves that the correct selection of loss function is critical to the performance of the neural network.

4. INVERSE NEURAL NETWORK CONSTRUCTION

The training of the inverse neural network (INN) is more challenging due to the nonuniqueness nature that one color can be formed by different F-P cavity structures. This multisolution property could lead to the adjustment of weight being pulled to different local or global minima during the training process, making the training difficult to converge. The bidirectional neural network architecture and tandem training strategy were therefore employed in this work in which the output parameter ${\mathbf{D}}^{\prime }$$(d_1^{\prime },d_2^{\prime },d_3^{\prime })$ from the INN was directly fed into our pretrained FNN to generate predicted $\mathbf{L}\mathbf{a}{\mathbf{b}}^{\prime }$$(L^{\prime },a^{\prime },b^{\prime })$ as shown in Fig. 4(a). The parameters of FNN are fixed during the INN training. The loss function can be defined as the MSE between the original and predicted Lab vectors instead of parameter $\mathbf{D}$. This means the INN will be optimized to match the desired color rather than the structure, avoiding the nonuniqueness problem in the training process. Here we adopted a recently reported penalized tandem training strategy to further enhance the robustness of the network in the initial 100 epochs [39], which indicates that the inverse error (the MSE between the original $\mathbf{D}$ and predicted ${\mathbf{D}}^{\prime }$) was included in the loss function for the first 100 epochs to avoid the predictions violating the ground truth of $\mathbf{D}$.

 

Fig. 4. Inverse neural network for predicting F-P cavity structural colors. (a) The bidirectional architecture with input layer of Lab values and output layer of geometric parameter D and connected to the pretrained forward neural network. (b) The schematic of different weights’ initialization positions. (c) The loss surface schematics of nonuniqueness (left) and uniqueness (right) problems, respectively. The MSEs after 200 epochs as a function of random seed, the training loss curves, the histogram of the distribution, and average values of $\mathrm{\Delta }{E}_{2000}$ of the INNs with (d)–(f) different numbers of hidden layers and (g)–(i) different numbers of neurons per layer.

Download Full Size | PDF

Another important factor in INN training is the selection of a random seed. The initial weights are obtained from random sampling using determined distributions (e.g., Xavier or Kaiming initialization). The training is therefore repeatable once the selection of a random seed was fixed. Different random seeds could place the INN at different starting positions in the loss plane [Fig. 4(b)]. In the case of multiple solutions, the loss plane is very complicated due to the interference of different global or local minima [Fig. 4(c), left]. Different starting positions could cause the INN to converge into a different local/global minimum, or in the case of a poor initialization, struggle to even converge. Selection of the random seed therefore plays a vital role in the INN training process. In this work, each INN underwent a process of random seed selection before hyperparameter optimization took place. The summary of the MSEs after 200 training epochs for INNs with varying hidden layers is presented in Fig. 4(d). It is obvious that the training progress differs significantly due to the selection of different random seeds, and different INN architectures prefer different random seed groups for optimized performance. This is particularly important for multisolution questions with the existence of a large number of local and global minima. It is less critical for single-solution questions (e.g., the FNN in this work), as the loss plane is relatively simple for the network to converge [Fig. 4(c), right] regardless of the initial starting position.

Figures 4(e) and 4(f) present the optimization of hidden layer numbers for INN. A drop of MSE at the 100th epoch in each training is due to the removal of the inverse error term in the loss function as mentioned above. Unlike the FNN where higher network complexity always leads to a better performance in the training set, a reduction of performance was observed for the INN with more than five layers. This inferior performance for more complicated INNs may be caused by the increased dimensions in the loss plane and number of existing global minima, which causes the network convergence to be more difficult. Similar behavior was also observed in the process of neuron number optimization as shown in Figs. 4(g)–4(i). A final INN with five hidden layers with 100 neurons was identified to be the best INN architecture to be used in this work. This network enables the average $\mathrm{\Delta }{E}_{2000}$ of 1.16 in the nonuniform testing set and 1.18 in the uniform testing set, which are superior to most of the commercial display equipment. For example, the high-end Dell UltraSharp 32 PremierColor UltraHD 8K Monitor is factory calibrated at 100% sRGB coverage to an accuracy of $\mathrm{\Delta }{E}_{2000}$ less than 2.

5. PERFORMANCE EVALUATION

To better evaluate the performance of our INN, we randomly selected 6 F-P cavity structures (D values) that the INN has never seen before, as listed in Table 1. These structures were converted to the corresponding spectra and subsequently $\mathbf{L}\mathbf{a}\mathbf{b}$ values as discussed previously. The target $\mathbf{L}\mathbf{a}\mathbf{b}$ values were then fed into our INN to obtain the designed structures ${\mathbf{D}}^{\prime }$. Further tests by converting the designed structures to designed colors through the theoretical calculation have resulted in very close matches with the targeted colors. The color differences, represented by the $\mathrm{\Delta }{E}_{2000}$ values, are below 1 in all six cases, suggesting that human eyes are not able to distinguish their differences. More importantly, in five of the six cases, our INN has produced designed structures that are significantly different from the original to the targeted structures while obtaining small color difference. This demonstrates the existence of nonuniqueness solutions for INN. More importantly, it highlights the ability of a neural network to discover solutions outside of the boundaries of the training data. Nanophotonic research has become more computation intensive due to the large spatial degrees of freedom and wide choice of materials [22]. Such ability to identify new design structures that can never be found through a conventional forward design technique would be extremely beneficial for discovering novel finding in nanophotonics.

Table 1. Comparison of the Six Randomly Selected Target Structure D, Lab, and sRGB Color Values with the Corresponding Values Inversely Designed by the Inverse Neural Network

View Table

The excellent performance of our network is further supported by designing the F-P cavity structures to reproduce the painting of “Haystacks, end of Summer” by Claude Monet. This was done by extracting the $\mathbf{s}\mathbf{R}\mathbf{G}\mathbf{B}$ values of all $2000\times 1176$ (2,352,000) color pixels from the original painting [shown in Fig. 5(a)] and inputting them into the bidirectional network. The INN outputs the designed geometric parameters for all 2,352,000 pixels, and the reconstructed paintings were subsequently generated and plotted in Fig. 5(b).An extremely high accuracy can be clearly observed, as the difference between the two images is almost undistinguishable. This proves the robustness and accuracy of our network in designing colors over a wide gamut.

 

Fig. 5. Actual application of the INN-based structural color design. (a) The origin and (b) the reproduction via INN of the painting “Haystacks, end of Summer” by Claude Monet. Reproduction of “Haystacks, end of Summer” is permitted by the Musée d’Orsay, Paris (RF 1975 3).

Download Full Size | PDF

We will now provide a detailed analysis of the network performance by evaluating the spectra of the blue (sRGB 0, 0, 102), green (sRGB 0, 102, 0), and red (sRGB 102, 0, 0) color filters designed by our INN. The designed F-P cavity for blue color has a dielectric thickness (${\mathrm{d}}_2$) of 424 nm, and its transmissive spectrum is characterized by a main peak at 465 nm with a secondary peak at 690 nm as shown in Fig. 6(a-ii). The contribution from each CIE 1931-RGB spectral tristimulus is demonstrated by the size of the shades underneath the spectrum. It is obvious that the designed blue color consists of a majority of blue stimuli with a small proportion of green and red stimuli, the integrals of which are extremely close to the definition of blue color in the sRGB color space [shown by the dashed line in Fig. 6(b-i), and the corresponding CIE 1931-RGB tristimulus values can be obtained by converting the CIE 1931-XYZ tristimulus]. Almost identical values are obtained with a $\mathrm{\Delta }{E}_{2000}$ of 0.07. An unoptimized design with a dielectric layer 10 nm thinner will cause a blueshift of the spectrum [Fig. 6(a-i)]. This results in an increased contribution from the red stimuli but a reduced contribution from the green stimuli, driving the designed color away from the blue definition with a $\mathrm{\Delta }{E}_{2000}$ of 6.16. Similarly, a 10 nm thicker dielectric layer induces a redshift with more contribution from green stimuli and less contribution from red stimuli, resulting in a $\mathrm{\Delta }{E}_{2000}$ of 11.83. Similar behavior is also observed on the design of green color as shown in Fig. 6(a-iii). Our design with a dielectric thickness (${\mathit{d}}_2$) of 326 nm results in a $\mathrm{\Delta }{E}_{2000}$ of 0.35 [Figs. 6(a-v) and 6(b-ii)], whereas a $\pm 10\mathrm{nm}$ change of the thickness alters the contribution from the CIE 1931-RGB tristimulus and results in inferior color design with $\mathrm{\Delta }{E}_{2000}$ of 7.72 ($d_2-10\mathrm{nm}$) or 11.83 ($d_2+10\mathrm{nm}$) as shown in Fig. 6(a-iv/vi). It is clear that our network is able to identify the optimized thickness that places the spectrum in the right position to achieve minimum difference from the desired color.

 

Fig. 6. Transmissive spectra and corresponding CIE 1931-RGB tristimulus values for the designed colors. The transmissive spectra (black line) and the contribution from the three stimuli (shades underneath the line) for the (i, ii, iii) blue, (iv, v, vi) green, and (vii, viii, ix) red color designed by the INN in this work. Within each figure, the middle row figure presents the original design, while the top and bottom figures represent the spectra from a 10 nm thinner layer and a 10 nm thicker dielectric layer. (b) The CIE 1931-RGB tristimulus values as a function of dielectric layer thickness for the blue, green, and red colors, respectively. The CIE 1931-RGB tristimulus values of the targeted colors are also included (dotted lines) for comparison.

Download Full Size | PDF

It is also worth pointing out the limitation of our network, which is manifested in the design of the red color. The network selects a design with ${\mathit{d}}_2$ of 369 nm, which uses the second-order peak at 595 nm, resulting in a $\mathrm{\Delta }{E}_{2000}$ of 8.93 [Fig. 6(a-viii)]. Although it outperforms the other two scenarios with $\pm 10\mathrm{nm}$ of ${\mathit{d}}_2$ [$\mathrm{\Delta }{E}_{2000}$ of 20.21 and 19.64, respectively, shown in Fig. 6(a-vii/ix)], it fails to select the first-order peak, which could realize a much better design by using a smaller thickness ($d_2\sim 160\mathrm{nm}$). This suggests that the network is not flexible enough when a large change of thickness is required. This may be attributed to two factors. The first factor is the uneven color distribution of the training dataset caused by the nonlinear relation among thickness, spectrum, and color. The resonant cavity lengths in the F-P cavity are close to the positive integer times of $\lambda /4n$, where $n$ is the refractive index of the dielectric layer and $\lambda$ is the light wavelength in free space. The wavelengths of blue ($\sim 490\mathrm{nm}$) and green ($\sim 550\mathrm{nm}$) are shorter than that of the red ($\sim 610\mathrm{nm}$). Higher-order blue and green peaks therefore appear more frequently than red within the same dielectric layer range. Hence, colors near the red region are significantly under-represented. This leads to the network being trained in favor towards optimization of colors in the blue and green regions. The second factor lies in the inherent limitation with tandem network architecture, which suffers from mode collapse [23]. The high-quality first-order red peak was abandoned by the network during the training process to achieve a high overall accuracy for all colors. This results in the red colors being predicted to the higher orders, leading to limited quality. These two factors are believed to contribute to the difficulty of designing colors in the red region, and further improvements in both the network design and training process are required to tackle this challenge.

Finally, we evaluate the computational efficiency of our network by comparing it with an evolutionary algorithm—a popular metaheuristic optimization technique [48]. Compared with a neural network, an evolutionary algorithm performs better in the design of red color (details of EA can be found in Section 7). This forward design process was able to find the first-order red peak, achieving a $\mathrm{\Delta }{E}_{2000}$ of 0.73. The designs of green and blue colors are not as good as our INN with $\mathrm{\Delta }{E}_{2000}$ of 1.36 and 1.11, respectively. Moreover, this method demands much more computational resources than INN and is unpractical in real applications. For example, the time required to design the painting of “Haystacks, end of Summer” ($500\times 297$, 148,500 pixels) via our INN was 0.17 s. It took the evolutionary algorithm 4.8 h to design the same number of pixels under the same computational environment (see Section 7, Method). This translates to over 100,000 times of savings in time and computational resources for our network. In addition, the overall design accuracy for those 148,500 colors obtained from the network is significantly better ($\mathrm{\Delta }{E}_{2000}$ of 0.78) than that designed by EA ($\mathrm{\Delta }{E}_{2000}$ of 1.18), further proving the superior performance in both computational efficiency and design accuracy of our network.

6. CONCLUSION

In conclusion, we demonstrate the use of a bidirectional neural network to inversely design the geometric structures of F-P cavity color filters. This work leads to a gamut coverage that is 215% of the sRGB color space. By selecting the uniform CIE 1976-Lab color space over the conventional CIE 1931-XYZ color space as the representation of color, the bidirectional network has shown a superior accuracy for color design with an average $\mathrm{\Delta }{E}_{2000}$ value below 1.2 in the testing set. This excellent performance is also verified by comparison with the gradient-free evolutionary algorithm in which our network demonstrates a 100,000 times design efficiency improvement with higher accuracy when designing the famous painting “Haystacks, end of Summer” by Claude Monet. The challenges in designing colors at longer wavelength due to uneven dataset distribution and the continuous gradient descent nature of artificial neural networks are also discussed. This proposed model will contribute to the establishment of standard procedure for future design of nanostructured color filters with deep learning technology.

7. METHOD

Data Processing. Before the neural networks are trained, the datasets are normalized from 0 to 1. By doing this, the effects of unit, magnitude, and dimension of the dataset are waived, which is able to improve the performance and probability of convergence. This process was done by the open-source machine learning library Scikit-Learn.

Color Conversion. The color conversions, which include the conversions between different color spaces (e.g., sRGB to CIE 1931-XYZ and CIE 1931-XYZ to CIE 1976-Lab) and color difference ($\mathrm{\Delta }{E}_{2000}$), were performed by the open-source library Colour-Science.

Deep Learning. All the deep learning models and training were developed and performed on the open-source deep learning framework PyTorch.

Training Hyperparameters. The training hyperparameters are listed as follows. Epochs: 2000; batch size: 64; activation function: ReLU; loss function: mean squared error (MSE); optimizer: Adam; learning rate: 0.001; learning rate scheduler: MultiStepLR; $\mathrm{milestones}=[\mathrm{1800,1900}]$ (forward training) and [1,1800,1900] (inverse training); $\mathrm{gamma}=0.1$. The Kaiming uniform initialization method was adopted in this work to investigate the impact of random seed selection [49].

Loss Functions. In the training of the forward neural network, the loss function is defined as ${\mathrm{Loss}}_{\mathrm{forward}}=\mathrm{MSE}({\mathrm{Lab}}_{\mathrm{predicted}},{\mathrm{Lab}}_{\mathrm{truth}})$. In the training of the inverse neural network, a penalty term in the loss function has been introduced for the first 100 epochs to ensure ${\mathit{D}}_{\mathrm{predicted}}$ is not too far away from the input dimension ${\mathit{D}}_{\mathrm{truth}}$ (i.e., reduce the chance of generating negative values). The penalized loss function is defined as ${\mathrm{Loss}}_{\mathrm{inverse}}=\mathrm{MSE}({\mathrm{Lab}}_{\mathrm{predicted}},{\mathrm{Lab}}_{\mathrm{truth}})+0.2\times \mathrm{MSE}(D_{\mathrm{predicted}},D_{\mathrm{truth}})$. The second term in the equation is a penalty term, which is used to control the inverse output error compared with the ground truth of the geometric parameters. The penalty coefficient $\lambda$ is set to be 0.2 in our case. After the 100 epochs, the loss function is defined as ${\mathrm{Loss}}_{\mathrm{inverse}}=\mathrm{MSE}({\mathrm{Lab}}_{\mathrm{predicted}},{\mathrm{Lab}}_{\mathrm{truth}})$.

Evolutionary Algorithm. The evolutionary algorithm (EA) is one type of gradient-free optimization method, and it is an appealing option for solving this optimization problem. It uses a refined iterative process in which an elite percentage of the individuals are retained through each iteration, allowing the samples to genetically evolve until the best option has been identified. The evolutionary algorithm in this work was realized by open-source EA library $\mathit{D}\mathit{E}\mathit{A}\mathit{P}$. The EA generation and population applied in this work are 100 and 50, respectively. The EA designed geometrical parameters (d₁, d₂, d₃) are (39 nm, 163 nm, 46 nm) for red (102, 0, 0); (33 nm, 138 nm, 50 nm) for green (0, 102, 0); and (21 nm, 417 nm, 49 nm) for blue (0, 0, 102). The ΔE₂₀₀₀ are 0.73, 1.36 and 1.11, respectively.

Computational Environment. CPU: Intel Core i9-9900K; GPU: nVIDIA RTX 2070; RAM: 48 GB; OS: Windows 10 Pro; Python version: 3.7. The same computational environment was used for the ANN and EA methods in this work.