Abstract
Accurate and rapid autofocus technology plays a crucial role in various fields, including automatic optical inspection technology, bio-chips scanning, and semiconductor manufacturing. The current photoelectric autofocus methods have limitations because of detecting the focal plane solely at the center of the microscope field of view. In the application of Stereo-seq the risk of autofocus errors will be increased, which have reduced the robustness of the system, like when the surface of the tested samples are wrinkling and inconsistent thickness, or the detection spot is at the edge of the sample. To enhance the robustness of the autofocus system and mitigate the constraints of the photoelectric autofocus methods, the laser-based arrayed spots photoelectric autofocus method has been proposed. To achieve the uniform light splitting, a 2D-Dammann grating is incorporated into the optical path of the autofocus system, resulting in the formation of an n × n arrayed spots on the surface of the sample. Through experimental verification, it has been demonstrated that this method can achieve the autofocus range of ±100μm and the autofocus accuracy of ±1/4 DOF when applied to a microscope equipped with a 10× objective lens, thereby satisfying the requirements for microscopic focusing. The arrayed light autofocus method devised in this study presents what we believe is a novel research concept for active autofocus detection and holds significant application value.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
In recent years, autofocus technology has played an increasingly significant role in various application fields [1], including automatic optical inspection, TFT arrays scanning, semiconductor manufacturing, and biological imaging [2,3]. In the fields such as bio-chips scanning and semiconductor manufacturing, the size of the sample area is generally larger than the microscope's field of view. Therefore, scanning imaging technology is commonly employed, which involves completing the exposure imaging of a sample area and then moving the sample step by step through the stage to bring the adjacent area into the imaging field of view for another exposure. However, during this process, defocusing may occur due to mechanical errors, platform movement, or sample flatness errors. Relying solely on manual focus not only slows down the scanning speed but also introduces human errors. To address these challenges, an autofocus system is utilized. This system performs real-time scanning to detect defocus errors of the sample during the step scanning movement of the motion-stage. It then controls the movement of the motion-stage to compensate for the defocus amount in real time. This ensures that the sample remains in the optimal focus of the microscope objective throughout the imaging process.
Autofocus solutions can be broadly categorized into two methods: image-based autofocus detection and photoelectric-based autofocus detection. The image-based autofocus detective method determines the focus position by analyzing the sharpness or the spatial frequency function of the overall image within the microscope field of view to obtain focus information [4–10]. This method is capable of detecting surface undulations of complex samples, enabling automatic focusing on different positions within the same field of view. However, due to the requirement of image acquisition and processing, this method has low autofocused efficiency which is not suitable for high-speed scanning imaging technology.
Another autofocus method is based on photoelectric detection technology [1,11–16]. This method typically employs laser or LED light sources to certain the defocus distance and direction of the sample in order to rectify the focal plane variation so that we can achieve rapid and precise focal plane positioning. Nevertheless, autofocus methods based on photoelectric detection are often limited to single-point autofocus, with the focal point situated at the center of the field of view on the sample surface. Therefore, this method can only determine the focal plane of the central spot position within the field of view.
Stereo-seq utilizes microscopic scanning imaging to identify the anatomical structure of tissue sections, facilitating the production of high-definition, comprehensive transcriptome atlases. It is capable of capturing tissue section areas up to a few square centimeters [17]. The photoelectric autofocus system is critical in the swift imaging scans of Stereo-seq. Nonetheless, the preparation of large tissue sections frequently results in issues like wrinkling or inconsistent thickness, which leads to errors in focus judgment with conventional single-point autofocus systems. Moreover, the mapping of Stereo-seq necessitates the scanning of edges of samples, yet single-point autofocus systems tend to detect the focal planes of areas lacking biological tissue, leading to the failures of the autofocus feature.
In the past, various solutions have been proposed to address the issue. One approach suggested was the incorporation of an adjustable lens (typically the liquid lens) to alter the position of the focal plane and focus on different planes. Alternatively, we could adjust the optical axis of the relay lens in the autofocus optical system [18–20]. However, these solutions would either increase the complexity of the autofocus system or necessitate modifications to the microscope itself. Another solution involved programming a scan pre-modeling process to handle focus anomalies [21]. Nevertheless, this method required manual intervention and was comparatively less efficient.
Consequently, we have developed an innovative technique for arrayed spots autofocus. This research enables the detection of defocus at any position within the microscopic field by generating the n × n autofocus detection light array on the surface of the sample. Utilizing this method, we have effectively resolved the issues of focus failure due to sample fabrication defects or scanning positions, substantially improving the robustness of conventional photoelectric autofocus techniques. This method ensures both the efficiency and extensive range of photoelectric autofocus without adding to the complexity of the detection system, offering a viable solution for industrial applications.
Moreover, aberrations are an unavoidable aspect between the autofocus system and the microscopy system in practical scenarios. These aberrations affect the shape of the autofocus spots near the focal surface, which in turn impacts the accuracy of the autofocus system and the linearity of the autofocus curve. To address this issue, we added an appropriate amount of defocus to the autofocus system. Under these circumstances, the defocus has a significant effect on the autofocus spots, which overshadows the degradation of the spots caused by system aberrations. Consequently, the CCD captures a relatively standard semi-circular spot pattern. Experimental results confirm that this method effectively improves the linearity of the autofocus curve and the precision of autofocus system.
2. Arrayed spots autofocus system for microscope
Laser-based autofocus technology has been extensively employed in the autofocus of microscopy systems [22–25]. This technique leverages a CCD sensor to capture an image of a sample's reflected laser beam and generates a signal based on the center-of-mass position within the image.
The schematic diagram of the optical path principle of the proposed automatic focus detection system is depicted in Fig. 1(a) [26]. According to the principles of geometric optics, when the laser irradiates point A on the sample surface, which is located on the focal plane of the objective lens, it is reflected and then falls on point A’ of the focusing image sensor. At this particular point, we are able to obtain the following information:
The expression for AA1 can be derived based on the given information. Let α denote the semi-aperture angle of the microscopic objective lens, which represents the maximum angle after the beam parallel to the optical axis is refracted by the objective lens. Let d represent the spot radius of the collimated laser beam, and let f1 represent the focal length of the microscopic objective lens.
When the sample is positioned at a distance of ±δ from the focal plane, the laser beam is directed at point C on the surface of the sample. The reflected beam intersects with A1 at the focal plane and then falls onto point A1’ of the focus-checking image sensor. At this particular configuration, AA1 can be expressed as:
The displacement of the point at the focus-checking image sensor is:
where k represents the total magnification of the achromatic lens and the microscope objective. This value is determined by the ratio of the focal length f2 of the achromatic lens to the focal length f1 of the microscope objective.The simultaneous Eqs. (1)–(4) ultimately yield a linear relationship between defocus amount and displacement of light on the image sensor:
The focus-detection measurement value refers to the displacement of light on the focus-detecting image sensor. In order to enhance autofocus efficiency, it is necessary for the focus-detection measurement value to have a linear relationship with the defocus amount. Geometric optics suggests several commonly used measurement values for focusing, including centroid of spot, distance between geometrical center and centroid of spot, and spot radius.
Figure 1(b) illustrates the morphology of the laser spot on the CCD sensor under varying conditions, as well as the efficacy of the focus-detection measurement value on the spot. In the Fig. 1(b), Δ represents the laser spot radius. The positions of the geometric center and the centroid of the image captured from the CCD sensor are denoted as (Xc, Yc) and (Xcentroid, Ycentroid) respectively. The geometric image center (Xc, Yc) remains unchanged, whereas the centroid of the image (Xcentroid, Ycentroid) can be mathematically expressed as:
In Eqs. (6) and (7), the variables i and j represent the row number and column number of the CCD sensor respectively. The variable Pij represents the gray value of the image corresponding to the pixel with coordinates (x, y). Assuming uniform gray value throughout the entire image, there is a linear correlation between the centroid of the image and the defocus amount δ on the CCD sensor. Therefore, it can be inferred that the centroid or radius of the reflected laser beam's image, as captured by the image sensor, exhibits a linear variation with the defocus amount of the microscope. This variation can be used as a suitable feedback signal for the closed-loop position control system.
The paper intends to utilize the spot radius as the measured value, as the superposition calculation of the center of mass in Eqs. (6) and (7) requires some time. On the other hand, the spot radius value only necessitates recording the intensity of luminous distribution of the CCD detector. Hence, this method not only guarantees autofocus accuracy but also achieves high autofocus speed.
The basic structure of the arrayed spots autofocus system of microscope proposed in this paper is illustrated in Fig. 2(a). The system utilizes a semiconductor laser with a wavelength of 660 nm as its light source. The visible light is employed for the autofocus light source to facilitate the maintenance and observation of the autofocus module. To address the issues concerning the beam quality and light intensity uniformity of semiconductor lasers, we employs a spatial filter positioned after the laser. This spatial filter helps generate spots with uniform light intensity. The incident beam undergoes a transformation process through the beam expander and diaphragm, resulting in the formation of a semicircular beam. Subsequently, the beam is divided by spatial light modulator (the paper employs a two-dimensional Dammann grating to divide a beam of light into the 3 × 3 array). Two beam splitters are then utilized to guide the laser light emitted from the second splitter through the objective lens and onto the sample surface, thereby generating an array of illumination spots. Upon reflection from the sample, these beams are once again directed through the objective lens and two beam splitters before reaching the imaging lens. Ultimately, they are captured by the CCD sensor. It is worth noting that the shape of the laser spots on the CCD detector is also semi-circular.
After the signal processing module processes the light spots, the arrayed light spots acquire the defocus amount at different positions of the sample. Subsequently, the defocus signal is transmitted to the worktable motor, which drives the sample to move up and down along the optical axis direction in order to achieve focusing.
When the sample being measured is in close proximity to the focal plane, the light spot detected by the CCD sensor is influenced by aberration, resulting in significant distortion. Moreover, the deformation of the autofocus light spot near the focal plane is minimal. Both of these factors lead to errors in determining the defocus distance and direction. The research proposes moving the CCD detector towards the imaging lens, causing the autofocus light reflected by the objective lens's focal surface to deviate from the focus surface of the imaging lens. This method eliminates the impact of aberration on the focus detection system when the sample is near the microscope objective focal plane with improving the system's focus accuracy and reducing auto-focusing time.
Figure 2(b) shows the optical model of the autofocus system constructed using optical design software. The simulation light source employs a 660 nm Gaussian beam with a beam diameter of 3 mm. The microscope objective is a 20× objective lens with an approximate focal length of 10 mm. The imaging lens has a focal length of 100 mm, while the CCD sensor is positioned 80 mm behind the imaging lens. Figure 3 displays the simulation of footprint diagram imaged on the CCD sensor and sample surface, respectively under varying defocus amounts.
By constructing a laboratory prototype, depicted in Fig. 4, the effect of the simulated autofocus microscope system has been verified in this study. The prototype employs a 660 nm low-noise semiconductor laser as its light source, with a spatial filter consisting of a small 2mm-diameter aperture. Furthermore, a 3× beam expander from Thorlabs is utilized, along with a mirror serving as the sample. The spatial light modulator utilized in this study is a two-dimensional Dammann grating produced by HOLO/OR, a leading manufacturer in the field. The grating can generate a 3 × 3 array of beams, with the splitting angle of 0.79°×0.79°. The grating exhibits an impressive total efficiency of 72% and maintains a contrast ratio below 15%. To assess the suitability of the designed autofocus system, microscope objectives with magnifications of 10×and 20× are integrated into the infinity corrected optical system. To ensure a more pronounced variation in the autofocus spots on the CCD detector, the focal length of the imaging lens is extended to 180 mm, while maintaining a distance of approximately 160 mm between the imaging lens and the CCD sensor. The CCD features a unit pixel size of 3.45µm and a sensor size of 14.2 × 10.4 mm, guaranteeing a sufficiently high level of focus accuracy and range.
The defocus of the prototype is adjusted by the movement of the objective lens. The objective lens shift-stage used is P725.4CA from PI Company, and image processing is carried out using XIMEA CAMTOOl. Spot analysis and extraction of focus measurement values are achieved through Matlab algorithm.
Figures 5(a) and 5(c) illustrate the notable enhancement in quality of autofocus spot image following the addition of a spatial filter subsequent to the laser source. At the same time, Figs. 5(b) and 5(d) contrast the autofocus spot from a standard single-point autofocus system with the arrayed spots resulting from the incorporation of the two-dimensional grating.
Considering the reliance on autofocus spot for acquiring focus detection measurements, the shape of the autofocus spot substantially influences the precision of autofocus as well as the linearity of the autofocus curves. Because of aberrations between the autofocus system and the microscopy system in practical scenarios, autofocus spots near the focal plane undergo pronounced deformations, posing a considerable challenge to the precise retrieval of focus detection measurements, as depicted in Figs. 6(a)-(d). In an effort to improve the linearity of the autofocus curves and the rate of auto-focusing, the CCD in the autofocus system was advanced by a certain distance, effectively mitigating the detrimental influences of diffraction and aberrations on the spot's shape, as illustrated in Figs. 6(e)-(h). A comparison of the two sets of images reveals that our approach normalizes the spot shape, which in turn significantly increases the accuracy of focus detection measurement and the linearity of the autofocus curves. This technique not only refines the multi-spot autofocus technology but also optimizes conventional autofocus strategies.
Figure 7 shows the experimental image captured by the CCD sensor, portraying the spots images under varying magnification objective lenses (10× and 20×) and differing degrees of defocus. Figure 8 exemplifies the utilization of Matlab for processing the array of focus detection spots. The x-axis signifies the distance of defocus, while the y-axis represents the autofocus spot diameter (measured in pixels), which is employed to construct the focus detection curve.
The experimental results indicate that there is a linear functional relationship between the focusing curves of objective lenses with different magnifications within their focusing range, which satisfies the requirements for autofocus of microscopy. Furthermore, the experimental focusing range and accuracy meet the demands of the industry. Among them, the 10× objective lens achieves a minimum focusing accuracy of 1µm with the focusing range of ±100µm, the 20× objective lens achieves 0.5µm with the focusing range of ±20µm, all of which meet the focus accuracy requirement of ±1/4 objective lens depth of field.
3. Analysis of different positions of light spots
The paper presents the utilization of a two-dimensional Dammann grating to generate 3 × 3 arrayed spots. The sample used in this study is a reflection mirror, which is positioned perpendicular to the optical axis. The spots at various sample positions were examined under the same magnification objective lens. The semi-circular spots captured by the CCD detector were fitted into circles using the least squares method. The resulting auto-focusing spots were labeled as c1-c9 respectively and the changes in radius of the spots were analyzed for different defocus amounts using 10× and 20× objective lenses. The results were then plotted as focus curves, with the defocus amount represented on the horizontal axis and the pixel value of the spot radius represented on the vertical axis. The curves demonstrate that each spot exhibits linear characteristics, thus meeting the auto-focusing requirements. The specific curves can be seen in Figs. 9(a)-(b).
As shown in Fig. 9, the autofocus spots at each sample location have a consistent slope of the autofocus curve, in agreement with theoretical results. However, these autofocus curves show performance variations, meaning the curves do not completely coincide. To investigate the cause of this phenomenon, we conducted simulation experiments. We constructed an aberration-free autofocus system model, and set up both a horizontal sample surface and a tilted sample surface (X tangent = 0.01, Y tangent = 0.01), and calculated the corresponding autofocus curves. Subsequently, we introduced aberrations into the model, obtaining the autofocus curves for the non-tilted sample surfaces. The experimental results are displayed in Fig. 10.
We ultimately identified two primary reasons for the performance disparity in autofocus curves. The first reason is the difficulty in ensuring the sample's complete perpendicularity to the optical axis. During the experiments, we employed a high-precision angular adjustment stage for meticulous alignment to minimize the impact of sample tilt on the focus detection curves. Second, aberrations in the autofocus system and the microscopy optical path may cause variations in the autofocus curves at different sample positions. Therefore, we entered all the autofocus curve data from spots at different positions into the program, ensuring that these autofocus spots do not interfere with each other, to achieve multi-position autofocus functionality.
4. Sample verification
To verify the efficacy of arrayed spots autofocus in resolving focus failure issues, we employed this system to perform a focus test on real biological analysis chip. We specifically analyzed the scenario where the periphery of sample was positioned at the microscope's field of view.
Figure 11(a) shows a microscopic image of the sample and substrate boundary of the bio-analytical chip. The bio-analytical chip consists of three regions: region A represents the sample part, region B represents the substrate part, and region C is the boundary between the two. Figures 11 (b), (c), and (d) display the light spot obtained by the autofocus CCD when the autofocus light is directed towards the sample part, substrate part, and boundary part, respectively. The spots obtained from the reflection on the sample surface appear more regular and exhibits some interference fringes. However, the light spots reflected back from the base surface are distributed in a point-like pattern rather than a regular semicircular spot due to the rough surface of the base material. Therefore, it is difficult to obtain corresponding focusing measurement values. Autofocusing failure is likely to occur when the single-point autofocus system detects light irradiation on the boundary part. In order to address this issue, we propose a method that utilizes an array of beams. As shown in Fig. 11(d), when the light is irradiated on the boundary part, it can be observed that some spots can obtain defocus values normally. This approach reduces the possibility of autofocus failure and enhances the robustness of the autofocus system.
5. Conclusion
This study presents a novel and accurate laser-based arrayed spots autofocus system of microscope, aiming to address the autofocus failures caused by sample problems or scanning processes and improve the robustness of photoelectric-based autofocus methods for sample detection. The proposed method utilizes the feedback signal of the reflected laser spot radius, captured by the CCD sensor, to determine the defocus amount. Subsequently, it drives the motion-stage of the sample placement to move incrementally along the optical axis direction to achieve autofocus. The main reliance of the arrayed spot in this study is on the spatial light modulator inserted in the system. Furthermore, by providing a pre-defocus amount to the CCD sensor in the autofocus system, the shape of the spots can be optimized, and the influence of aberrations on the autofocus spots can be eliminated. As a result, the system accuracy is improved and the response time is shortened.
Furthermore, this study has established an optical system within the laboratory and employed a reflective mirror as a specimen to showcase the viability of the suggested arrayed spots autofocus system. The experimental findings indicate that the aforementioned autofocus system not only possesses the capability to automatically focus on samples with substantial planar errors at various focal planes, but also exhibits a wider focusing range, enhanced focusing accuracy, and improved focusing speed. Consequently, the proposed system presents an optimal resolution for a diverse array of automated detection applications in the domains of electronics, bio-medicine, and industry with high development prospects. Future work will focus on system integration and development of optical instrumentation.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. W.-Y. Hsu, C.-S. Lee, P.-J. Chen, et al., “Development of the fast astigmatic auto-focus microscope system,” Meas. Sci. Technol. 20(4), 045902 (2009). [CrossRef]
2. L. Han, X. Wei, C. Liu, et al., “Cell transcriptomic atlas of the non-human primate Macaca fascicularis,” Nature 604(7907), 723–731 (2022). [CrossRef]
3. B. J. Jung, H. J. Kong, B. G. Jeon, et al., “Autofocusing method using fluorescence detection for precise two-photon nanofabrication,” Opt. Express 19(23), 22659 (2011). [CrossRef]
4. X. Zhou, P. Xiong, D. Chi, et al., “Fast autofocusing based on pixel difference with the Tanimoto coefficient between images,” Opt. Lett. 47(15), 3752–3755 (2022). [CrossRef]
5. Z. Hua, X. Zhang, and D. Tu, “Autofocus methods based on laser illumination,” Opt. Express 31(18), 29465–29479 (2023). [CrossRef]
6. Z. Hua, X. Zhang, D. Tu, et al., “Learning to high-performance autofocus microscopy with laser illumination,” Measurement 216, 112964 (2023). [CrossRef]
7. S. Yazdanfar, K. B. Kenny, K. Tasimi, et al., “Simple and robust image-based autofocusing for digital microscopy,” Opt. Express 16(12), 8670–8677 (2008). [CrossRef]
8. X. Zhang, Z. Liu, M. Jiang, et al., “Fast and accurate auto-focusing algorithm based on the combination of depth from focus and improved depth from defocus,” Opt. Express 22(25), 31237–31247 (2014). [CrossRef]
9. D. Hu, Y. Liang, Y. Chen, et al., “Autofocusing Airy beam STED microscopy with long focal distance,” Opt. Commun. 404, 196–202 (2017). [CrossRef]
10. C. Guo, Z. Ma, X. Guo, et al., “Fast auto-focusing search algorithm for a high-speed and high-resolution camera based on the image histogram feature function,” Appl. Opt. 57(34), F44–F49 (2018). [CrossRef]
11. K. Du, D. Zhou, S. Zhou, et al., “High-accuracy differential autofocus system with an electrically tunable lens,” Opt. Lett. 48(11), 2789–2792 (2023). [CrossRef]
12. K.-H. Kim, S.-Y. Lee, S. Kim, et al., “DNA microarray scanner with a DVD pick-up head,” Curr. Appl. Phys. 8(6), 687–691 (2008). [CrossRef]
13. J. Yang, L. Qiu, W. Zhao, et al., “Laser differential reflection-confocal focal-length measurement,” Opt. Express 20(23), 26027–26036 (2012). [CrossRef]
14. H. Wu, J. Yang, L. Qiu, et al., “Measuring the lens focal length by laser confocal technique,” in Sixth International Symposium on Precision Mechanical Measurements (SPIE), 8916, pp. 373–382 (2013).
15. K. Guo, J. Liao, Z. Bian, et al., “InstantScope: a low-cost whole slide imaging system with instant focal plane detection,” Biomed. Opt. Express 6(9), 3210–3216 (2015). [CrossRef]
16. X. Zhang, F. Zeng, Y. Li, et al., “Improvement in focusing accuracy of DNA sequencing microscope with multi-position laser differential confocal autofocus method,” Opt. Express 26(2), 887–896 (2018). [CrossRef]
17. A Chen, Y Sun, Y. Lei, et al., “Single-cell spatial transcriptome reveals cell-type organization in the macaque cortex,” Cell 186(17), 3726–3743 (2023). [CrossRef]
18. S. J. Abdullah, M. M. Ratnam, and Z. Samad, “Error-based autofocus system using image feedback in a liquid-filled diaphragm lens,” Opt. Eng. 48(12), 123602 (2009). [CrossRef]
19. H. Qu and Yang, “Optical microscopy with flexible axial capabilities using a vari-focus liquid lens,” J. Microsc. 258(3), 212–222 (2015). [CrossRef]
20. M. Bathe-Peters, P. Annibale, and M. J. Lohse, “All-optical microscope autofocus based on an electrically tunable lens and a totally internally reflected IR laser,” (2018). [CrossRef]
21. W.-Y. Hsu, “Automatic Compensation for Defects of Laser Reflective Patterns in Optics-Based Auto-Focusing Microscopes,” IEEE Sens. J. 20(4), 2034–2044 (2020). [CrossRef]
22. S. H. Wang, C. J. Tay, C. Quan, et al., “Laser integrated measurement of surface roughness and micro-displacement,” Meas. Sci. Technol. 11(5), 454–458 (2000). [CrossRef]
23. Q. Li, L. Bai, S. Xue, et al., “Autofocus system for microscope,” Opt. Eng. 41(6), 1289–1294 (2002). [CrossRef]
24. C. -S. Liu, P. -H. Hu, Y. -H. Wang, et al., “Novel fast laser-based auto-focusing microscope,” in Sensors, (IEEE, 2010) pp. 481–485.
25. C.-S. Liu, Y.-C. Lin, and P.-H. Hu, “Design and characterization of precise laser-based autofocusing microscope with reduced geometrical fluctuations,” Microsyst. Technol. 19(11), 1717–1724 (2013). [CrossRef]
26. C.-S. Liu, P.-H. Hu, and Y.-C. Lin, “Design and experimental validation of novel optics-based autofocusing microscope,” Appl. Phys. B 109(2), 259–268 (2012). [CrossRef]