In this paper, we propose a framework of starting points generation for freeform reflective triplet using back-propagation neural network based deep-learning. The network is trained using various system specifications and the corresponding surface data obtained by system evolution as the data set. Good starting points of specific system specifications for further optimization can be generated immediately using the obtained network in general. The feasibility of this design process is validated by designing the Wetherell-configuration freeform off-axis reflective triplet. The amount of time and human effort as well as the dependence on advanced design skills are significantly reduced. These results highlight the powerful ability of deep learning in the field of freeform imaging optical design.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Using freeform optical surface in an imaging system is a revolution in the field of optical design , because freeform surfaces can effectively improve system performance while decreasing the system size and mass, and number of elements [2–11]. However, in fact, designing a freeform imaging system is a difficult task because such systems are generally highly non-symmetric, and advanced system specifications are generally required. The traditional design method for imaging optics is to first find a starting point from patents or other available systems and then perform optimization. However, for freeform system design, proper starting points with similar system specifications and special nonsymmetric configurations are very rare, which greatly increases the possibility of using extensive human effort or even failure. To overcome this problem, some direct or point-by-point design methods for freeform imaging system have been developed in recent decades, such as the partial differential equations (PDEs) method , the Simultaneous Multiple Surface (SMS) method (and the related analytical methods) [13–15] and the Construction-Iteration (CI) method [16,17]. Given the system specification, the required freeform surfaces are calculated and the corresponding system can be taken as the starting point for further optimization. However, some of these methods require a complex mathematical derivation, which makes them difficult for designers to use; some methods can only be applied to a design by considering a limited number of fields or coaxial configurations. Additionally, all these methods only offer a specific solution for a single system design task. The design process does not provide any insight into designing other related systems; that is, a specific design task can hardly benefit from former designs, and the design framework is not “intelligent”. For different design tasks, designers have to repeat the entire design process, and significant amount of human effort and time are still required. In some cases, it is difficult for beginners or non-specialists in optical design to complete design tasks.
In recent years, artificial intelligence and deep learning have been applied to many areas of scientific research and engineering, including the field of optics [18–21]. Modern deep-learning uses a multi-layered artificial neural network for data analysis, feature extraction and decision making. The artificial neural network are computing systems inspired by the biological neural networks and astrocytes that constitute animal brains . It is an interconnected group of nodes called artificial neurons, and these “neurons” inside the networks compute output values based on inputs. A typical feature of deep learning is that it can solve complex and difficult problems with little human work using pre-knowledge learned from existing results. Machine learning (or deep learning) has been successfully introduced into the freeform illumination design area. Gannon and Liang present a method for improving the efficiency and user experience of freeform illumination design with machine learning . By using an artificial neural network which can output orthogonal polynomial coefficients, they generalize relationships between performance parameters and lens shape. They successfully created a network to generate uniform squares or rectangles on the target plane. If we introduce deep learning into imaging optical design, or freeform imaging system design discussed in this paper, it is possible for the design tool to generate the required design results quickly and easily using a network trained by former design results. Thus, human effort will reduce significantly, the difficulty of freeform imaging system design will decrease, and even beginners in optical design will be able to perform design tasks. Using deep learning in imaging system design could be another revolution in the field of geometric optics.
In this paper, we propose a framework for generating starting points for freeform off-axis reflective three-mirror imaging system using neural network based deep-learning. A number of base systems for data set acquisition are obtained through system evolution, starting from one initial base system, which is designed easily and quickly using various methods. Then, a data set that consists of system data as the input and surface data as the output can be obtained. A feed-forward back-propagation (BP) network is trained using the data set. Using this network, good starting points with various system specifications for further optimization can be generated immediately in general. Designers do not have to manage the starting point exploration or analytical/numerical design process. The amount of time and human effort as well as the dependence on advanced design skills reduce significantly. Beginners in optical design can also generate a good starting point using the obtained network. We validate the feasibility of this design process by designing the Wetherell-configuration freeform off-axis reflective triplet. The proposed design approach opens up new possibilities for designing complex and high-performance imaging optical systems using artificial intelligence.
The diagram of the design framework is shown in Fig. 1. For a given system folding geometry, the entire design framework consists of four steps. The first step is to generate a number of base systems that have the same configuration within a given range of system specifications. The system data of these systems and the corresponding surface locations and coefficients are taken as the input and output parts respectively in the data set. Then, the feed-forward BP network can be trained using the obtained data set. When the network is ready, for a specific design task, the given system data can simply be input into the network. Then the surface locations and coefficients can be output directly. The corresponding system can be taken as a good starting point for further optimization. Detailed descriptions are provided in the following sections.
2.1 Data set generation
An important aspect of framework is to find a series of base systems with good imaging performance. The basic configuration or folding geometry of these base systems are the same as the design requirement. The system data and corresponding surface parameters are recorded as the data set for training. In our design framework, the field-of-view (FOV) of the system is rectangular and the system is symmetric about the YOZ (meridional) plane, which is the most common case for a freeform nonsymmetric imaging system. When the central field is given, the full FOV of the system is determined using the half FOV in the x direction (Half-XFOV) and half FOV in the y direction (Half-YFOV). Generally, central field (CenFLDx, CenFLDy) = (0°, 0°), which means that no biased FOV is used in the design. Two other key specifications of an imaging system are the effective focal length (EFL) and system F-number (F#). The entrance pupil diameter of the system is calculated as EFL/F#. Therefore, in our design framework, four parameters (i.e., Half-XFOV, Half-YFOV, EFL and F#) are used to describe the system specification, and they are taken as the input for a single system in the data set. We use vector SSP = [Half-XFOV, Half-YFOV, EFL, F#] to denote the system specification of the system.
The output part of a single system in the data set are the surface parameters, which include the global surface locations and surface coefficients. The global location of a surface in a YOZ plane-symmetric system includes the global y-decenter, global z-decenter and global α-tilt angle, where “global” indicates that the y-decenter and z-decenter are global decenter values of the surface vertices in the y and z directions with respect to a global coordinate system in space, and α-tilt is the angle of the local surface z-axis with respect to the global z-axis. The surface coefficients are the parameters that describe the freeform surface shape. Typical freeform surface types includes XY polynomials and Zernike polynomials. The base sphere or conic is not used in our design framework for simplicity, which means that the freeform surfaces are only described by the polynomials if XY polynomial or Zernike polynomial surface is used. We assume that a total of Ω different surface parameters are recorded for each system as the output surface data, which are recorded in output vector OSD.
The next step is to determine the range of each system parameter by defining its lower and upper limit based on actual need
For each SSPi (1≤i≤N), the corresponding base system BaseSysi with good imaging performance has to be generated to obtain the surface data. Therefore, each base system has to be optimized. However, the optimization should be performed carefully. Traditional optical design generally begins from a starting point whose system specification is not far from the current design. Otherwise the optimization will be much more difficult and fall easily into bad local optimums or even other unexpected results. In this paper, the base systems should be designed under the following principle: Each base system BaseSysi is optimized starting from the specific system that has the most similar specifications to BaseSysi among the base systems that have already been obtained. Here a special system evolution approach following this principle is used to obtain all the base systems.
- (1) First, an initial base system BaseSys1 has to be generated. We can simply choose the SSP that corresponds a non-advanced specification as the initial SSP1. This system is easy to optimize and can be designed by non-specialists or beginners with no advanced design skills and little experiences. Although there are many approaches available to achieve the design of BaseSys1, we recommended that the design is achieved in two steps. In the first step, an “initial configuration” is generated using a point-by-point design method (e.g., PDEs method, SMS method, and CI method) based on SSP1 and the given folding geometry. Then the system is optimized to achieve good imaging performance. The constraints during the optimization should control both the image size (distortion) and light obscuration. The system is constrained to use the central area of each freeform surface; off-axis aperture is not used. Other specific constraints can be added based on actual need. The above constraints are used for the subsequent optimization of each base system. Note that all the constraints used here should not be too strict to make the overall optimization easier, and this is adequate for a starting point design using loose constraints. In fact, we can choose any SSP as the initial SSP1. Using an initial base system with a more advanced SSP may improve the quality of the data set for network training. However, this requires advanced design skills and experience, which is not the aim of this design framework.
- (2) When the ith base system BaseSysi has been obtained, the N−i weighted distances WD between SSPi and the N−i remaining SSP are calculated, where WD is a distance that describes the similarity of two system specifications and is calculated as ||w⊗(SSPi−SSP*j)||2, where SSP*j (1≤j≤N−i) denote the SSP which have not been used, ||·||2 is the 2-norm. w = [wHalf-XFOV, wHalf-YFOV, wEFL, wF#] is the weight vector that determines the individual weight of each system parameter. The four scalar weight elements should be chosen to balance the contribution that results from the same amount of data change for each system parameter to the overall optical design difficulty. ⊗ represents element-wise vector multiplication, which means that the operation multiplies two vectors element by element. For example, if A = [a1, a2, a3, a4] and B = [b1, b2, b3, b4], then A⊗B = [a1b1, a2b2, a3b3, a4b4] and it is a vector. Find the smallest WD and the corresponding system specification among the N−i remaining SSP is defined as SSPi+1.
- (3) Before generating BaseSysi+1, note that SSPi may be not the most similar system specification to SSPi+1. Hence, the next base system BaseSysi+1 may not be evolved from BaseSysi. The total i weighted distance WD between SSPi+1 and the total i SSPk (1≤k≤i) that have been used already should be calculated. Find the smallest WD and the corresponding base system BaseSysq (1≤q≤i). This base system is that from which BaseSysi+1 should be evolved.
- (4) The optimization is conducted starting from BaseSysq to obtain BaseSysi+1. Note that the constraints for controlling the image size should be modified according to the specific FOV and EFL values in SSPi+1. However, if Half-XFOV and Half-YFOV of two systems BaseSysq and BaseSysi+1 are the same, then it is better to scale the system BaseSysq by a scale factor EFLi+1/EFLq before optimization.
- (5) Steps (2)-(4) are repeated until all the base systems have been generated and all the surface data have been recorded.
To increase the working efficiency, the entire data set generation process is implemented into a program (except for the generation of the initial base system BaseSys1). The system model establishment and optimization are achieved using optical design software CODE V. MATLAB is used for data processing and the detailed algorithm implementation. Because the CODE V application programming interface (API) uses the Microsoft Windows standard Component Object Model (COM) interface, users can execute CODE V commands using MATLAB which supports Windows COM architecture. The final MATLAB program enables automatic data set generation.
2.2 Network training
When we have obtained the data set that consists of system data and surface data, the next step is to train the network. In this study, the feed-forward back-propagation (BP) network is used. This type of multi-layer network is a widely used artificial neural network. It is trained using an error back propagation algorithm, and can learn and store a large number of mapping relations of input-output model. Here, the “feed-forward” means that the connections between the nodes do not form a cycle. The information moves in only the forward direction: from the input nodes, through the hidden nodes and to the output nodes. There are no cycles or loops in the network . The “back-propagation” refers to the algorithms used to efficiently train artificial neural networks following a gradient descent approach. The gradient of the loss function (representing the difference between the network output and its expected output) is calculated and it is then used to update the weight value of the nodes to minimize the loss function . The topology of the BP network consists of the input layer, hidden layers (may be more than one), and output layer. Generally speaking, the input layer is consisted of many nodes which are used to receive the nonlinear input information. The output layer is consisted of nodes which correspond to the outputs generated by the network. The hidden layers are between the input layer and the output layer, where artificial neurons take in weighted inputs and produce an output through a transfer function. The nonlinearity and robustness of a network increase with the growing number of nodes in the hidden layers. In our design framework, the system data constitute the input layer and the surface data constitute the output layer. Two hidden layers are used to improve learning performance. As shown in Fig. 2, the input layer has four nodes. The jth element in a four-element SSP vector is the input of the jth node of the input layer (1≤j≤4). The first and second hidden layers have H1 and H2 nodes respectively. b1m is the threshold of the mth node (1≤m≤H1) in the first hidden layer and b2n is the threshold of the nth node (1≤n≤H2) in the second hidden layer. w1jm is the connection weight from the jth node in the input layer to the mth node in the first hidden layer. w2mn is the connection weight from the mth node in the first hidden layer to the nth node in the second hidden layer. wonp is the connection weight from the nth node in the second hidden layer to the pth node in the output layer. bop is the threshold of the pth node (1≤p≤Ω) in the output layer. The pth node of the output layer corresponds to the pth element in the Ω-element OSD vector. The transfer function of the nodes in the network may be a linear or a nonlinear function and it is chosen to satisfy some specification of a problem that a neuron is attempting to solve . The output of a single node is determined by its transfer function and the weight input. For example, for the mth node in the first hidden layer shown in Fig. 2, the net weighted input to this node is
3. Example demonstration
In this section, we use the design of the Wetherell-configuration freeform off-axis reflective triplet to validate our proposed design framework. The Wetherell unobscured configuration is a traditional type of off-axis three mirror system and has no intermediate image inside the system . We set the range of the system parameters as follows: 2°≤Half-XFOV≤4.5°, 2°≤Half-YFOV≤4.5°, 80mm≤EFL≤120mm, and 1.5≤F#≤4. The system uses (0°, 0°) as the central field and the secondary mirror (M2) is taken as the aperture stop. The spectral band is long-wave-infrared (LWIR). For data set generation, NHalf-XFOV = 6 different Half-XFOVs, NHalf-YFOV = 6 different Half-YFOVs, NEFL = 9 different EFLs and NF# = 12 different F#s are sampled within the above range with equal intervals. After full combinations, N = NHalf-XFOV × NHalf-YFOV × NEFL × NF# = 3888 different system specifications (SSP) are used. If we focus on the performance of the network after training, N = 3888 is not the best sampling of the system specification. Increasing the number of different system specifications will lead to more training data and therefore better performance of the network after training in general. However, it will also increase the time duration for the data set generation and the network training. So here N = 3888 is just a balance of the network performance and time duration, and this is not the only choice for this design task.
The next step is to generate all the base systems and obtain the corresponding surface data. The initial base system BaseSys1 uses SSP1 = [2°, 2°, 120mm, 4], which means that the smallest FOV and largest F# are used, which leads to an easily designed system. To generate this system, we firstly use the CI method  to generate an “initial configuration”. The CI method is a point-by-point design method of freeform imaging systems that uses the light rays from multiple fields and different pupil coordinates. An initial planar system is first established as the start point. The decentered and tilted planes in the initial system should be located at approximately the same places as the final freeform surfaces and form the same type of folding geometry. As the system is symmetric about the YOZ plane, only half of the full FOV need to be considered during the design. We use six typical sample fields in half FOV during the design process: (0°, 0°), (0°, 2°), (0°, −2°), (2°, 0°), (2°, 2°), and (2°, −2°). A similar field sampling method is used in the optimization of other base systems. The initial planar system is shown in Fig. 3(a). Then the freeform surfaces are generated point-by-point through the preliminary construction stage and iteration stage based on the given system specification SSP1. The result of the design is shown in Fig. 3(b), and this system is used for the subsequent optimization of BaseSys1. The constraints used here should control the image size and light obscuration. For image size control, the ideal image height is calculated based on EFL and field angles of specific fields. The actual image heights are obtained using real ray trace data. Here the constraints are loose: the allowable errors of the full image height in the y-direction and the half-full image height in the x-direction are ± 0.5mm and ± 0.3mm respectively. This is adequate for a starting point design and significantly reduces the design difficulty. The smile distortion is constrained by controlling the difference of the image locations of the (0°, 0°) and (2°, 0°) fields in the y-direction and the allowable error is ± 0.1mm. Proper structure constraints have to be used during optimization. The marked distances L1 to L5 have to be controlled to eliminate light obscuration or avoid surface interference, as shown in Fig. 3(b). These distances are controlled to be larger than 10mm. Additionally, to eliminate the strange structures of the overall system, the global α-tilt of the image plane and the incident angle of the marginal ray onto M2 are controlled so that they are not too large. The freeform surface type used here is XY polynomials up to the fourth order with no base conic. Because the optical system is symmetric about the YOZ plane, only the even items of x in the XY polynomials are used
After we have obtained the data set containing the system data and the surface data of the base systems, the next step is the network training. In this design, the BP network have two hidden layers and they have 30 and 40 nodes respectively. Tansig is taken as the sigmoid-type transfer function for the hidden layers and purelin is taken as the linear transfer function for the output layer. The root mean squared error (RMSE) of the network after training is 0.00129 and the correlation coefficient R is 0.99702. The network training is conducted in MATLAB on the same computer given above and the training time is 2.5 hours. When we have obtained this network, for a specific SSR input with in the given range of the system data, the corresponding surface data of a system with acceptable imaging performance can be output immediately and for most cases the corresponding system can be taken as a very good starting point for further optimization.
To analyze the effect of the obtained network, we randomly sample 1000 different system specifications within the whole range and output the corresponding surface data. The corresponding output systems are then established and the imaging performance are analyzed. Here the average RMS spot diameter of the six sample fields and maximum absolute distortion among the sample fields of each system are calculated, as shown in Fig. 4. The absolute distortion of a field point refers to the distance between the ideal image point and the actual image point. It can be seen that for most of the systems (984 out of 1000 systems), the average RMS spot diameter is within the range of 0mm to 0.8mm, and the maximum distortion is below 1.5mm. Although these 984 systems maybe cannot be used as the final system for working and maybe a little amount of light obscuration will appear in some systems, they can be taken as good starting points for further optimization. Output systems of several typical system specifications generated by the network are plotted in Fig. 5 as examples. For the other 16 systems whose average RMS spot diameter is outside the range of 0mm to 0.8mm, or the maximum distortion is larger than 1.5mm, these systems may be problematic. Here a detailed analysis of these systems are performed. Ten systems (Nos. 226, 295, 311, 446, 606, 656, 657, 754, 912 and 974) are truly abnormal as the system is not valid and/or ray tracing error is encountered. This may be due to the imperfection of the network at the corresponding inputs or some other complex reasons in the software. These systems cannot be used for further optimization. Note that the CODE V macro function RMSSPOT is used to calculate the RMS spot diameter of a field. The function returns −1 if there are ray trace errors with the reference ray trace, or if no rays get through the system . Therefore the calculated RMS spot diameter may be below 0. For the other six systems (Nos. 43, 159, 218, 629, 875 and 977) out of the 16 systems, although the spot diameter and/or the distortion are relatively larger than the 984 systems mentioned above, these are still acceptable for the starting point design. After all, although several systems are problematic, nevertheless, 99% of the systems generated by the network can be taken as good starting points for further optimization. This validate the feasibility of the proposed design framework. If a user want to obtain a starting point of Wetherell-configuration freeform triplet within the system specification range, he can input the system specification into the network and obtain the surface data immediately in general.
We demonstrate a novel generation method of starting points for freeform reflective imaging triplet using neural network based deep-learning. The data set consists of the system data as the input and the surface data obtained through system evolution as the output. A feed-forward BP network is trained using the data set. Good starting points with various system specifications for further optimization can be generated immediately in general. The feasibility of this design framework is validated by designing the Wetherell-configuration freeform off-axis reflective triplet. The amount of time and human effort as well as the dependence on advanced design skills can be significantly reduced. The difficulty of freeform imaging system design is reduced and the design tasks can handled by beginners of optical design. Future work will focus on the deep-learning-based design framework enabling system generation of wider system specifications and automated folding geometry selection.
National Key R&D Program of China (2017YFA0701200); National Natural Science Foundation of China (61805012, 61727808); Beijing Institute of Technology Research Fund Program for Young Scholars.
We thank Synopsys for the educational license of CODE V, and we thank Maxine Garcia, PhD, for helping improving the English text of this paper
1. K. P. Thompson and J. P. Rolland, “Freeform optical surfaces: a revolution in imaging optical design,” Opt. Photonics News 23(6), 30–35 (2012). [CrossRef]
3. C. Liu, C. Straif, T. Flügel-Paul, U. D. Zeitner, and H. Gross, “Comparison of hyperspectral imaging spectrometer designs and the improvement of system performance with freeform surfaces,” Appl. Opt. 56(24), 6894–6901 (2017). [CrossRef] [PubMed]
4. Q. Meng, H. Wang, K. Wang, Y. Wang, Z. Ji, and D. Wang, “Off-axis three-mirror freeform telescope with a large linear field of view based on an integration mirror,” Appl. Opt. 55(32), 8962–8970 (2016). [CrossRef] [PubMed]
5. J. Zhu, W. Hou, X. Zhang, and G. Jin, “Design of a low F-number freeform off-axis three-mirror system with rectangular field-of-view,” J. Opt. 17(1), 015605 (2015). [CrossRef]
8. F. Fang, X. Zhang, A. Weckenmann, G. Zhang, and C. Evans, “Manufacturing and measurement of freeform optics,” CIRP Ann. 62(2), 823–846 (2013). [CrossRef]
9. D. Reshidko and J. Sasian, “Method for the design of nonaxially symmetric optical systems using free-form surfaces,” Opt. Eng. 57(10), 1 (2018). [CrossRef]
10. M. Beier, J. Hartung, T. Peschel, C. Damm, A. Gebhardt, S. Scheiding, D. Stumpf, U. D. Zeitner, S. Risse, R. Eberhardt, and A. Tünnermann, “Development, fabrication, and testing of an anamorphic imaging snap-together freeform telescope,” Appl. Opt. 54(12), 3530–3542 (2015). [CrossRef]
12. D. Cheng, Y. Wang, and H. Hua, “Free form optical system design with differential equations,” Proc. SPIE 7849, 78490Q (2010). [CrossRef]
14. F. Duerr, P. Benítez, J. C. Miñano, Y. Meuret, and H. Thienpont, “Analytic design method for optimal imaging: coupling three ray sets using two free-form lens profiles,” Opt. Express 20(5), 5576–5585 (2012). [CrossRef] [PubMed]
15. Y. Nie, H. Thienpont, and F. Duerr, “Multi-fields direct design approach in 3D: calculating a two-surface freeform lens with an entrance pupil for line imaging systems,” Opt. Express 23(26), 34042–34054 (2015). [CrossRef] [PubMed]
16. T. Yang, J. Zhu, X. Wu, and G. Jin, “Direct design of freeform surfaces and freeform imaging systems with a point-by-point three-dimensional construction-iteration method,” Opt. Express 23(8), 10233–10246 (2015). [CrossRef] [PubMed]
19. Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light Sci. Appl. 7(2), 17141 (2018). [CrossRef] [PubMed]
20. M. T. McCann, E. Froustey, M. Unser, M. Unser, and Kyong Hwan Jin, “Deep convolutional neural network for inverse problems in imaging,” IEEE Trans. Image Process. 26(9), 4509–4522 (2017). [CrossRef] [PubMed]
21. Z. Cao, N. Guo, M. Li, K. Yu, and K. Gao, “Back propagation neutral network based signal acquisition for Brillouin distributed optical fiber sensors,” Opt. Express 27(4), 4549–4561 (2019). [CrossRef] [PubMed]
23. C. Gannon and R. Liang, “Using machine learning to create high-efficiency freeform illumination design tools,” arXiv preprint arXiv: 1903.11166v1 (2018).
26. M. T. Hagan, H. B. Demuth, M. H. Beale, and Neural Network Design, (PWS Publishing, 1996).
27. W. B. Wetherell and D. A. Womble, “All-reflective three element objective,” U.S. Patent 4,240,707 (23 December, 1980).
28. CODE V Documentation Library, Synopsys Inc. (2018).