Open Access
Add to BibSonomy
Abstract
Snapshot microlens array microscopic hyperspectral imaging systems do not require a scanning process and obtain (x,y,λ) three-dimensional data cubes in one shot. Currently, the three-dimensional spectra image data are interleaved on a charge-coupled device detector, which increases subsequent data processing difficulty. The optical design software cannot simulate actual engineering installation and adjustment results accurately and the tracking results cannot guide precise rapid online calibration of the snapshot microlens array microscopic hyperspectral imaging system. To solve these problems, we propose an accurate spectral image reconstruction model based on optical tracing, derive spatial dispersion equations for the prisms and gratings, establish an algorithm model for the correspondence between the microlens array’s surface dispersion spectral distribution and its imaging position, and propose a three-dimensional spectral image reconstruction algorithm. Experimental results show that this algorithm’s actual spectral calibration error is better than 0.2 nm. This meets the image processing requirements of snapshot microlens array microscopic hyperspectral systems.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
With the continuous development of exogenous biosensor markers such as fluorescent proteins, quantum dots, and organic fluorophores in the biomedicine field [1–3], the spectral information introduced through these markers adds further observation dimensions to the time-series biochemical reactions of living cells. This information plays an indispensable role in research in immunology and molecular biology [4–7]. To achieve non-single-point observation of living cells, hyperspectral imaging technology that combines optical imaging with spectral analysis has been applied in the microscopy field, and a data cube that contains both the image information and the spectral information for the target under observation is obtained through either spatial scanning or wavelength scanning [8–12]. However, collection of full-field spectral information from the target is realized by scanning, which cannot meet the existing demand for time-sensitive spectral collection of moving cells with high timeliness. In contrast, the snapshot hyperspectral imager can obtain all the spatial spectrum information about the target from the detector through a single imaging process [12]. The integral field-of-view hyperspectral imager is one type of snapshot hyperspectral imager and was constructed by introducing the “integral field-of-view unit”. After the acquired two-dimensional image information is divided and rearranged, it is dispersed via the hyperspectral imager, and the two-dimensional image information and one-dimensional spectral information of the target under observation can then be obtained simultaneously to realize hyperspectral imaging data acquisition without scanning [12–15]. The integral field-of-view snapshot hyperspectral imager offers advantages that include high spectral reconstruction efficiency, a wide working wavelength range, and high spectral resolution. This imager has been used widely in the fields of astronomy and remote sensing [16–17] and continues to expand its applications into other fields. It provides an effective technique to realize time series acquisition of moving cell spectra.
The microscopic hyperspectral imager, which is constructed in the form of a microlens array, is a type of snapshot hyperspectral imager that divides and reconstructs a microscopic image through its microlens array, thus enabling its charge-coupled device (CCD) detector to record the two-dimensional spatial image and spectral dispersion information simultaneously [18]. As shown in Fig. 1, the microlens array consists of an array of lenses with transparent apertures and a micron-scale relief depth. This array offers the basic focusing and imaging functions of traditional lenses, is contained within a small unit size, is highly integrated, has a relatively simple structure, and demonstrates high device maturity.
The snapshot microlens array microscopic hyperspectral imaging system does not require to perform scanning, and can obtain three-dimensional (x,y,λ) data cubes in a single shot. As shown in Fig. 2, the three-dimensional spectrogram data are all interlaced on the CCD detector simultaneously, which adds to the subsequent data processing difficulty. Therefore, it is necessary to study the processing algorithm for the three-dimensional spectrum data collected via the CCD detector to realize reconstruction of the three-dimensional spectrum. At present, the spectral image reconstruction method for traditional hyperspectral imaging systems mainly uses optical design software intended for ray tracing applications [19]. This method requires a full-spectrum channel simulation for a large number of microlens units, which can take a long time. In addition, during the use of a hyperspectral imager with a transmission structure, the nonlinear relationship between the optical elements and the transmission characteristics of the light will change with changes in the spatial position of the microhole array. In actual engineering applications, to avoid any overlap between the spectral bands of the microlens array after dispersion, the microlens array must be rotated around the optical axis [20]. In addition, during the assembly and adjustment process, the microlens array will deviate from its ideal position; the hyperspectral imager also has an object plane tilt [20], which will form an oblique angle with the microlens array when in use; and lens processing errors and instrument installation and adjustment errors are present throughout the entire system. Because the optical design software cannot simulate the actual engineering installation and adjustment results accurately, the results traced using the optical design software cannot guide rapid online calibration of the snapshot microlens array micro-hyperspectral imaging system accurately.
Therefore, a method to reconstruct the three-dimensional spectral image from the target surface of the detector is urgently required. To solve this problem, we follow the optical path of the transmission structure, study the surface dispersion of the hyperspectral imager, and then apply the dispersion law to the prism and the transmission grating. A three-dimensional spectral image restoration method that uses a prism-grating as a dispersive element is established, and the corresponding online spectral image reconstruction algorithm is given through the development of a model. Experiments show that this algorithm can be applied successfully to image reconstruction of spectral data, with calculation accuracy that can reach the sub-pixel level.
2. Establishment of the spectral image reconstruction algorithm
Spectral image reconstruction is the process of converting the digital signals collected via the CCD detector unit of the hyperspectral imager into wavelength and position information.
The CCD detector outputs a digital matrix in which the horizontal and vertical coordinates represent the horizontal and vertical pixel numbers of the detector, and the value at each coordinate represents the light intensity value at the corresponding pixel. Optical tracing can be used to calculate the position of any microlens unit in the CCD detector after the units are dispersed by the hyperspectral imager. Depending on the size of the microlens unit and the number of microlens arrays used, it becomes necessary to construct the corresponding relationship between the dispersed spectral strip array and the sample information when magnified via the microscope objective lens. This correspondence represents the three-dimensional spectral image distribution model, which then forms the basis of spectral image information processing for the snapshot microscopic hyperspectral system.
The prism and the grating used in the optical system will produce opposite spectral line bending phenomena during the dispersion process, which is beneficial for spectral line bending correction of the hyperspectral imager. Therefore, a combined prism-grating optical path structure is used in the design, as illustrated in Fig. 3.
Fig. 3. Optical system model of the transmission hyperspectral imager using the prism-grating dispersion method.
A pixel located at a specific detector position corresponds to the specific wavelength of the specific microlens unit at that pixel. The spectral image reconstruction algorithm establishes the relationship between the detector pixel position and the three-dimensional spectrum and then calculates the wavelength contained in the microlens array unit based on the spot position.
For convenience of analysis, the front lens group and the rear lens group of the hyperspectral imager are regarded as ideal optical systems here. At this time, the equivalent spatial angle of incidence θ1 of the principal light ray from any microlens array unit that is incident on the front lens group of the prism-grating module is equal to the actual spatial angle of incidence θ0 on the front surface of the prism. Using the chief ray from any microlens array unit as an example in the prism, a spatial coordinate system is established in the y-x-z direction as follows:
Figure 4 shows the chief ray of the micro-aperture formed by any microlens unit in the prism module at M1 from point M0 as M0M1, where OM1O’ is the optical axis of the prism-grating dispersion hyperspectral imaging system. The angle between the optical axis OM1O’ and the normal HM1H’ is the prism apex angle β. Suppose here that the spatial angle of incidence of the M0O light on the prism is θ; its projection on the arc vector is then M0xM1 and its projection on the meridian surface is M0yM1 because ΔM0yHM1 is located on the meridian surface. Here, nα represents the refractive index of each medium, dα is the thickness of each medium, and Mα is the intersection of the chief ray from the micro-hole formed by the microlens and each refractive surface. ωy is the angle between the projection of the chief ray from the micro-aperture formed by the microlens and the optical axis after it passes through the prism-grating transmission spectrometer’s front mirror group on the meridian surface. The spatial dispersion equation of the prism can then be written here as:
Fig. 4. Schematic diagram of the principal ray of the micro-aperture as formed by any microlens unit in the prism module.
As shown in Fig. 5, with the grating surface acting as a diaphragm, the prism-grating light dividing module’s output is then projected into the meridian surface:
Fig. 5. Schematic diagram of the optical path of any chief ray of the micro-aperture in the meridian surface in the prism-grating spectroscopic module.
Let the coordinates of the chief ray of any micro-aperture in the microlens array be (x, y). After it passes through the front mirror group of the prism-grating (P-G) hyperspectral imager, this ray first intersects with the prism surface at ${M_1}$, and then intersects with the grating surface acting as a diaphragm. ${M_6}$ is located at the intersection of the optical axis with the grating and its meridian surface coordinates and sagittal surface coordinates represent the origins with the coordinates (0, 0); the coordinate system used under meridian surface projection conditions is as shown in Fig. 5, where
The coordinate system under the sagittal surface projection conditions is as shown in Fig. 6. It can be seen from the figure that:
Fig. 6. Schematic diagram of the optical path of any chief ray of the micro-aperture in the sagittal surface in the P-G spectroscopic module.
Figure 7 shows a schematic diagram of the optical path of any chief ray from the micro-aperture at the grating, where the grating equation for the transmission grating is as shown here:
Fig. 7. Schematic diagram of the optical path of any chief ray of the micro-aperture at the grating.
Therefore,
Here, θy6 is the angle between the projection of the chief ray from the micro-aperture onto the main section of the transmission grating and the grating normal; in addition, θy6’ is the angle between the projection of the chief ray from the micro-aperture when emitted from the principal section of the transmission grating and the grating normal. α is the angle between the projection of the chief ray from the microlens unit when incident on the transmission grating’s x-o-y-plane and the main grating cross-section and α’ is the corresponding angle between the projection of the chief ray of the microlens unit on exiting the transmission grating on the x-o-y plane and the main transmission grating cross-section. In addition, m is the diffraction spectrum order of the transmission grating, and d is the transmission grating constant.
From the geometric relationship, we find that:
If the angle of inclination between the rear mirror group of the prism-grating transmission spectrometer and the original optical axis is ε, then the rotation angle of the CCD detector relative to this rear mirror group is η. After image formation through the rear mirror, the image point coordinates on the CCD detector are then given by:
Therefore, a three-dimensional spectral image reconstruction algorithm can be established for the segmented field of view of the microlens unit. This algorithm can calculate the corresponding pixel position for each spectral line on the detector after the dispersion of the micro-hole matrix (X, Y) based on the position of the micro-hole matrix, which then provides theoretical support for the assembly and adjustment of the prototype unit.
3. Results and discussion
The three-dimensional spectral image reconstruction algorithm described above was established during the process of establishing the appropriate front and rear mirror groups for the hyperspectral imager to form the ideal lens. It will therefore be necessary to modify the spectral image reconstruction algorithm parameters for the front and rear mirrors when the prototype is installed and adjusted.
Consider the microlens array snapshot hyperspectral microscopy system that we designed earlier as an example here. A microlens array was obtained from SUSS Co. The array material is SiO2, size 10mm*10 mm and the center distance of the microlens unit is 250 µm. The grating constant d=300 l/mm, the diffraction order is −1, and the prism vertex angle β=10.4°. The filter range is 500 nm–600 nm, the prism material is K9 glass from CDGM Glass, with a thickness of 5 mm, and the transmission grating and filter are composed of B270 glass from Schott, with a thickness of 3 mm. The optical layout of the snapshot microlens array hyperspectral microscope system is shown in Fig. 8.
The three-dimensional spectrum of the mercury lamp within the 500 nm–600 nm spectral range is shown in Fig. 9. Using a centroid algorithm, and by performing three-dimensional spectral image calibration of the three wavelengths of the mercury lamp at 546.074 nm, 576.961 nm, and 579.067 nm, the precise positions for the three spot matrices of 546.074 nm, 576.961 nm and 579.067 nm are found.
As a result of the installation errors that occurred during the system installation process and the aberrations of the optical system, the three-dimensional spectral bands are different for the different microholes, and the three-dimensional spectral position traced using the optical design software thus cannot reflect the true state of the system. The precise positions of the three spot matrices of the mercury lamp at 546.074 nm, 576.961 nm, and 579.067 nm are therefore brought into the three-dimensional spectrum reconstruction algorithm model. By traversing the parameters of the three-dimensional spectral distribution model, the prism vertex angle error and the grating line number error introduced by the processing of the system are corrected. In addition, the rotation angle error between the prism and the grating introduced during the installation and adjustment process, the installation inclination error of the prism-grating module, and the optical axis deviation error of the microlens array are also corrected. The model can then find the 3D spectrum reconstruction algorithm model that is most suitable for the precise positions of the three spot matrices of 546.074 nm, 576.961 nm, and 579.067 nm automatically, and the system structure parameters in the model state are recorded.
The precise positions for the mercury lamp at 546.074 nm, 576.961 nm, and 579.067 nm are then compared with the predicted positions of the three spot matrices at 546.074 nm, 576.961 nm, and 579.067 nm that were calculated via the optimal three-dimensional spectrum reconstruction algorithm model. The optimal displacement △[X, Y] of the x-pixel direction and the y-direction on the target surface of the CCD detector can then be found for the most suitable three-dimensional spectrogram reconstruction algorithm model.
This optimal displacement is brought into the three-dimensional spectral image reconstruction algorithm model, the positions for all wavelengths in the 500 nm–600 nm spectral range on the target surface of the CCD detector are calculated, and full-spectrum calibration in the 500 nm–600 nm spectral range is achieved. The intensity profile of the 500 nm–600 nm spectral band of the single-lens mercury lamp that was extracted after calibration is shown in Fig. 10. The mercury lamp has a half-maximum width of 1 pixel, and the spectral resolution is 0.56 nm.
After correction, we then verify the actual application of the online spectral image reconstruction algorithm above. The snapshot microlens array microscopic hyperspectral imaging system described in this work was used to capture the 520.0 nm spectrum of a cerium lamp and the 589.0 nm spectrum of a sodium lamp after the light passed through a monochromator. Through extraction of the two relevant spots, the 520.0 nm spectrum of the cerium lamp and the 589.0 nm spectrum of the sodium lamp were obtained. The deviations between the predicted spectra and the 520.0 nm and 589.0 nm spectra after calibration of the three-dimensional spectral image reconstruction algorithm model were determined by comparison; these deviations are as shown in Fig. 11.
Fig. 11. Spectral deviation scatter diagrams for deviations between the spectral images predicted by the online spectral image reconstruction algorithm and the spectral images of actual element lamps extracted via the centroid extraction algorithm.
The verification process demonstrates that the measurement error between the theoretical spectral coordinate position of Cerium lamp at 520.0 nm and Sodium lamp at 589.0 nm and the real spectral coordinate position of A predicted by our online spectrum reconstruction algorithm is less than 0.2 nm. This error level is within the acceptable range. These errors are caused by a combination of the cumulative error of the assignment calculation process and the deviations in the incident and exit angles caused by the actual fx, fy, fx’and fy’ values when the principal ray from the microlens array passes through the front and rear lens groups because these parameter values are not equal to the theoretical values for the on-axis point.
As shown in Fig. 12, we conducted imaging experiments on gastric cancer samples and passed the resulting images through our online spectral image reconstruction algorithm. The single-wavelength spectral image of the gastric cancer sample acquired after the field of view was cut by using the microlens array was extracted from a real-time three-dimensional spectrum image of the gastric cancer sample. The images in the figure show that the proposed online spectrogram reconstruction algorithm can extract the full spectrum of the surface field of view at the detector’s frame rate (which, depending on the detector, ranges up to 20.6 Hz in this work) and provides high spatial positioning accuracy and high spectral extraction accuracy.
4. Conclusions
In a snapshot microlens array hyperspectral microscopy system, the existing optical design software cannot track the dispersion spectra of nearly 1000 microlens units in an array simultaneously, and there is also a problem with the deviations between the actual implementation and the theoretical design of a microlens array-based optical system in practical engineering applications. To solve this problem, we propose an online spectral image reconstruction algorithm for use with a snapshot microlens array hyperspectral microscopy system. The algorithm constructs a mathematical model of the correspondence relationship between the wavelengths of the spectral band and the microhole array through ray tracing. Based on the spectral calibration correction model parameters of the mercury lamp, the best correspondence relationship model is selected, which can then be selected directly when performing spectral image restoration, thus shortening the information processing time for array restoration and spectral image restoration. The wavelength calibration error of the proposed spectral image reconstruction algorithm is less than 0.2 nm and the algorithm offers high spectral calibration accuracy, high spatial positioning accuracy, and high spectral extraction accuracy. This approach solves the problem that the spectral strip array is interlaced on the detector surface, which makes data extraction difficult, and can also meet the spectral image reconstruction requirements of snapshot hyperspectral microscope systems based on microlens arrays.
Funding
Jilin Provincial Development and Reform Commission's Funds for Innovation Ability Construction – High Tech Industry (2019C038-4); Jilin Province Science & Technology Development Program Project in China (20180201049SF, 20190201304JC, 20200403009SF); National Natural Science Foundation of China (61805240); Changchun Science and Technology Bureau (18DY003).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. Ulf Maeder, Kay Marquardt, Sebastian Beer, Thorsten Bergmann, Thomas Schmidts, Johannes T. Heverhagen MD, Klemens Zink, Frank Runkel, and Martin Fiebich, “Evaluation and quantification of spectral information in tissue by confocal microscopy,” J. Biomed. Opt. 17(10), 1060111 (2012). [CrossRef]
2. Jin Li, Hanyang Li, Haifeng Hu, Yong Zhao, and Qi Wang, “Preparation and spectral characteristics of silver nano-sphere doped quartz micro-fiber,” Opt. Laser Technol.. 68, 79–83 (2015). [CrossRef]
3. Isabel Egea, Wanping Bian, Cristina Barsan, Alain Jauneau, Jean-Claude Pech, Alain Latché, Zhengguo Li, and Christian Chervin, “Chloroplast to chromoplast transition in tomato fruit: spectral confocal microscopy analyses of carotenoids and chlorophylls in isolated plastids and time-lapse recording on intact live tissue,” Ann. Bot. 108(2), 291–297 (2011). [CrossRef]
4. J Tao, “FTIR and Raman studies of structure and bonding in mineral and organic-mineral composites,” Methods Enzymol. 532, 533–556 (2013). [CrossRef]
5. Alexandra Guedes, Manuel Algarra, A. Carmelo Prieto, VanesaHortelano Bruno Valentim, Selma Neto, Rafael Algarra, and Fernando Noronha, “Raman Microspectroscopy of Genuine and Fake Euro Banknotes,” Spectrosc. Lett. 46(8), 569–576 (2013). [CrossRef]
6. M. Cristina Pedroso, Michael B. Sinclair, Howland D. T. Jones, and David M. Haaland, “Hyperspectral Confocal Fluorescence Microscope: A New Look into the Cell,” Microsc. Microanal. 15(S2), 880–881 (2009). [CrossRef]
7. Ci Sun, Mingjia Wang, Jicheng Cui, Xuefeng Yao, and Jianjun Chen, “Comparison and analysis of wavelength calibration methods for prism - Grating imaging spectrometer,” Results Phys. 12, 143–146 (2019). [CrossRef]
8. Karel J. Zuzak, Robert P. Francis, Eleanor F. Wehnerm, Jack Smith, Maritoni Litorja, David W. Allen, Chad Tracy, Jeffrey Cadeddu, and Edward Livingston, “DLP Hyperspectral Imaging for Surgical and Clinical Utility,” Proc. SPIE 7210(1), 721006 (2009). [CrossRef]
9. Paul M. Kasili and Tuan Vo-Dinh, “Hyperspectral Imaging System Using Acousto-Optic Tunable Filter for Flow Cytometry Applications,” Cytometry, Part A 69A(8), 835–841 (2006). [CrossRef]
10. Rusty Lansford, Gregory Bearman, and Scott E. Fraser, “Resolution of multiple green fluorescent protein color variants and dyes using two-photon microscopy and imaging spectroscopy,” J. Biomed. Opt. 6(3), 311–318 (2001). [CrossRef]
11. Sparks William and Ford Holland, “Imaging Spectroscopy for Extrasolar Planet Detection,” Astrophys. J. 578(1), 543–564 (2002). [CrossRef]
12. Jason G. Dwight and Tomasz S. Tkaczyk, “Lenslet array tunable snapshot imaging spectrometer (LATIS) for hyperspectral fluorescence microscopy,” Biomed. Opt. Express 8(3), 1950–1964 (2017). [CrossRef]
13. Ye Wang, John Gawedzinski, Pawlowski Michal E, and Tomasz S Tkaczyk, “3D printed fiber optic faceplates by custom controlled fused deposition modeling,” Opt. Express 26(12), 15362–15376 (2018). [CrossRef]
14. Yuri Murakami, Keiichiro Nakazaki, and Masahiro Yamaguchi, “Hybrid-resolution spectral video system using low-resolution spectral sensor,” Opt. Express 22(17), 20311–20325 (2014). [CrossRef]
15. Liang Gao, Robert Kester, and Tomasz Tkaczyk, “Optical design of a snapshot high-sampling Image Mapping Spectrometer (IMS) for hyperspectral microscopy,” Proc. SPIE 7570, 75700Z (2010). [CrossRef]
16. M McElwain, J Larkin, S Metchev, and B Zuckerman, “High-Contrast Imaging with Keck Adaptive Optics and OSIRIS,” Proc. SPIE 7015, 70151A (2008). [CrossRef]
17. Gregg Vane, Robert O Green, Thomas G Chrien, Harry T Enmark, Earl G Hansen, and Wallace M Porter, “The airborne visible/infrared imaging spectrometer (AVIRIS),” Remote Sens. Environ. 44(2-3), 127–143 (1993). [CrossRef]
18. Miran Bürmen, Franjo Pernuš, and Boštjan Likar, “A practical approach to spectral calibration of short wavelength infrared hyper-spectral imaging systems,” Proc. SPIE 7556(14), 755614 (2010). [CrossRef]
19. Ai Zhenyi, Yuan Yan, Su Lijuan, Ding Xiaoming, and Wang Wanyue, “A spectral calibration approach for snapshot image mapping spectrometer (IMS),” 9th International Symposium on Advanced Optical Manufacturing and Testing Technologies (AOMATT) - Optical Test, Measurement Technology, and Equipment 108339, 27 (2019). [CrossRef]
20. Changeben Yu, Jin Yang, Nan Song, Ci Sun, Mingjia Wang, and Shulong Feng, “Microlens array snapshot hyperspectral microscopy system for the biomedical domain,” Appl. Opt. 60(7), 1896–1902 (2021). [CrossRef]
