Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging

Young-Gil Cha; Young-Gil Cha; Jiwoong Na; Hyun-Kyung Kim; Hyun-Kyung Kim; Jae-Myeong Kwon; Jae-Myeong Kwon; Seok-Haeng Huh; Seung-Un Jo; Chang-Hwan Kim; Min H. Kim; Ki-Hun Jeong; Ki-Hun Jeong

doi:10.1364/OE.498763

1. Introduction

High dynamic range (HDR) imaging surpassed the contrast ratio of conventional digital cameras [1,2]. A single CMOS image sensor captured low dynamic range (LDR) images that falls short in capturing the full range of lighting information found in the real world. HDR imaging provided a broader range of brightness levels, resulting in more vivid and realistic images [3,4]. Conventional HDR techniques rely on the combination of multiple LDR images captured at different exposure times. However, these methods had some technical limitations in capturing dynamic scenes, resulting in motion artifacts such as ghosting or blurring [5–7]. In addition, they still required high hardware cost such as multiple optical elements [8–10], non-conventional image sensor fabrication [11], spatial light modulators (DMD or LCD) [12,13] or even high computational power for learning algorithms [14–16].

Microlens array (MLA) cameras have actively provided a promising avenue for low-cost compact imaging solutions [17–20]. They reduced the total track length (TTL) of cameras with large depth of field and small optical aberration [21,22]. Consequently, the MLA cameras were actively utilized for diverse applications such as wide field-of-view imaging [23], contact imaging [24], multifocal imaging [25], 3D imaging [26,27], and portable point of care testing device [28]. In addition, the MLA camera incorporated either anti-reflection structures [29,30] to improve the image brightness or light absorber structures [31–33] to enhance the image resolution and contrast by reducing optical crosstalk between microlenses. However, the captured images still showed some technical limitations in increasing the dynamic range of the MLA camera.

Here we report microlens array camera with variable apertures (MACVA) for single-shot HDR imaging. The MACVA comprises aperture arrays with various hole sizes, formed by a patterned chromium (Cr) thin film, microlens arrays, and packaged with CMOS image sensor arrays (Sony IMX 477, pixel size: 1.55 µm × 1.55 µm). The unique configuration allowed each microlens to collect different amount of illuminating light, producing slightly different LDR image array with various exposure levels and small visual disparities when objects were located in a far-field plane (Fig. 1). The camera finally captures a single HDR image reconstructed from different exposure bracketing LDR array images under single-shot exposure.

Fig. 1. Microlens array camera with variable apertures (MACVA) for high-dynamic-range (HDR) imaging. The MACVA features a variable aperture arrays (AAs), microlens arrays, gap spacers, and a CMOS image sensor. Objects situated in the far-field plane are imaged uniformly across each channel, because of the minimal visual disparity of individual channels. The variable apertures collect different amount of light to form array images under single-shot exposure so that array images of multiple exposure bracketing are simultaneously acquired via microlens arrays. The HDR image reconstructed by LDR images with HDR imaging and tone mapping algorithms.

Download Full Size | PDF

2. Microfabrication of MACVA

The MACVA involves wafer-level microfabrication techniques microlens arrays (MLAs), various aperture arrays (AAs), and the compact integration of a CMOS image sensor (Fig. 2(a)). Lift-off photoresist (DNR L-300, Dong-jin Semichem, Co., Ltd, Korea) was spin-coated onto a 4-inch borosilicate wafer substrate and then photolithographically patterned to create micropatterns with various aperture diameters using an MA-6 mask aligner. An electron beam was then used to evaporate a 100 nm thick Cr layer onto the patterns and the remaining resist was removed by using a resist stripper (DPS-7300, Dong-jin Semichem. Co., Ltd, Korea) for Cr lift-off process. A 22 µm thick negative-tone photoresist (DNR L-4615, Dong-jin Semichem, Co., Ltd, Korea) was photolithographically defined on the patterned metal layer. Microlens arrays were uniformly formed in a convection oven at 140°C for 30 minutes using a resist reflow process. The microfabricated microlens has f/1.4, a focal length of 420 µm, and a radius of curvature of 284 µm. The MLAs were precisely attached using a flip-chip bonder, with 420 µm tall alumina spacers on a single CMOS image sensor. The microscope images show the microfabricated microlens plate with variable apertures and microlens, where the Cr apertures completely block visible light (Fig. 2(b) and 2(c)). The captured images display a perspective and the side view of fully packaged MACVA, confirming that image sensor is accurately placed at the focal length of MLAs (Figs. 2(d) and 2(e)). The total track length (TTL) of the camera including the window glass is 920 µm.

Fig. 2. Microfabrication and configuration of the MACVA. (a) The microfabrication steps of MACVA. The patterned Cr layer was formed by using metal lift-off process. After photolithographic patterning, the microlens arrays were created by using thermal reflow. The microlens plate was fully packaged with gap-spacers on a single CMOS image sensor. (b) A transmission microscope image of microlens plate. (c) A captured image of microlens plate. (d) Optically captured side view of MACVA after packaging. (e) A photograph of a fully packaged MACVA.

Download Full Size | PDF

3. MACVA characterization and HDR imaging

The MACVA demonstrates single-shot array imaging with different exposure values (Fig. 3(a)). A confocal laser scanning microscope (CLSM) displays light adjusting properties of the microlens with aperture via a collimated 532 nm laser beam. The MLAs focus a laser beam on the same focal plane, and the amount of light entering the lens clearly decreases with the aperture size. The modulation transfer function (MTF) of the captured images was also analyzed by using the slanted-edge method with the ISO 12233 target. Depending on the aperture size, each image was captured under a constant exposure value by adjusting the shutter speed. The MTF50 value was measured at different aperture sizes, with a decrease in aperture size. (Figure 3(b)). The MTF50 value shows the maximum at 156.4 cycle/mm for 2-stop (f/2.8), which is 1.97 times higher than the MTF50 value at 0-stop (f/1.4). The MTF50 value initially increases due to the decrease in lens aberration as the aperture size decreases. However, the MTF50 value decreases after f/4 due to the increase of aperture diffraction. The MACVA captures array images of a printed ‘Lena’ image 10 cm far from the camera under single-shot exposure, showing the multiple LDR array images at different exposure values (Fig. 3(c)). The image histogram exhibits the distribution of pixel intensities depending on the f-stop (Fig. 3(d)). 2 and 3 stops show the pixel intensity distributions of bell-shape, while 1 or fewer stops and 4 or more stops lead to skewed distributions, indicating overexposure or underexposure respectively. As a result, the MACVA allows single-shot array imaging with different exposure levels, selecting a set of the proper f-stops for HDR imaging.

Fig. 3. MTF and image histogram measurement depending on the f-stops. (a) Optically sectioned beam profiles through microlenses (scale bar: 100 µm) and captured ISO 12233 target images with different f-numbers. (b) Measured MTF50 at constant exposure value and luminance mean under single-shot exposure for the f-stops. (c) “Lena” array image captured under single-shot exposure. (d) The intensity histogram exhibits the distribution of pixel intensity depending on the f-stop.

Download Full Size | PDF

HDR images were finally reconstructed by developing an HDR imaging algorithm. LDR images depending on different f-stops were used as inputs and combined for a single HDR image through weighted average of the pixel intensity values. The radiance value (E_i) at each pixel of the HDR image was determined using the following equation.

lo{g_2}{E_i} = \frac{{\mathop \sum \nolimits_{j = 1}^P [{lo{g_2}({{Z_{ij}}} )+ 2lo{g_2}({{F_j}} )} ]w({{Z_{ij}}} )}}{{\mathop \sum \nolimits_{j = 1}^P w({{Z_{ij}}} )}}\; \; \; \;,

where Z_ij is the pixel intensity at the i-th pixel of the j-th image, and F_j is the f-stop of j-th image. The weighting factor $\textrm{w}(z )$ as determined by using a normalized pyramid function, where $\textrm{w}(z )$ takes the value of ($z - {Z_{min}}$) for $z \le \frac{1}{2}\; ({{Z_{min}} + {Z_{max}}} )\; $ and $({Z_{max}} - z$) for $z \ge \frac{1}{2}\; ({{Z_{min}} + {Z_{max}}} )$. Tone mapping was finally employed to spatially render HDR images on LDR formats [14]. The MTF curves were evaluated for HDR images formed by reconstructing LDR images captured at three different sets of f-stops, as well as a single f-stop (f/4) image, all at optimal exposure of 3-stop. (Fig. 4(a)). The MTF50 values for HDR1 (from three LDR images for f/2, f/4, and f/8, i.e., 2-stop difference) and HDR2 (from three f/2.8, f/4, and f/5.6, i.e., 1-stop difference) are comparable with a single f-stop (f/4) image at 153.2 cycles/mm. However, the MTF50 value for HDR3 (from three LDR images for f/1.4, f/4, and f/11.2, i.e., 3-stop difference) is reduced to 127.8 cycles/mm. This decrease is attributed to the low MTF50 values of the individual LDR images captured at f/1.4 and f/11.2, as depicted in Fig. 3(b). Consequently, a set of LDR images with 2-stop difference provides both a broad dynamic range and high resolution for HDR imaging. Single-shot HDR imaging was experimentally demonstrated and compared with multi-shot HDR imaging (Fig. 4(b)). The single-shot HDR images were reconstructed from three LDR images for f/2, f/4, and f/8. The MACVA captures array images of a chip-on-board at a distance of 5 cm. The reconstructed HDR image clearly exhibits a broad dynamic range without data loss unlike the LDR images. In addition, the MACVA captures a rotating color disc (40 rpm) and a lighting lamp at a distance of 20 cm, allowing for a comparison between single-shot and multi-shot exposures. For the single-shot exposure, LDR images at different exposure values were simultaneously captured with a shutter speed of 16 ms. On the other hand, LDR images for f/2 were obtained through multi-shot exposures with shutter speeds of 16 ms, 4 ms, and 1 ms. The reconstructed single-shot HDR image clearly exhibits minimal motion artifacts, unlike the multi-shot HDR image [34], yet both HDR images provide a broad dynamic range.

Fig. 4. HDR imaging through the MACVA. (a) The measured MTF curves of tone-mapped HDR images, and f/4 LDR image. HDR1 image was reconstructed by using f/1.4, f/4, and f/11.2 LDR images, HDR2 image was reconstructed by using f/2, f/4, and f/8 LDR images, and HDR3 image was reconstructed by using f/2.8, f/4, and f/5.6 LDR images. (b) Comparison of single-shot and multi-shot HDR imaging. The single-shot HDR images reconstructed by f/2, f/4, and f/8 LDR images with a shutter speed of 16 ms and the multi-shot HDR image is created by LDR images for f/2 with shutter speeds of 16 ms, 4 ms, and 1 ms.

Download Full Size | PDF

4. Conclusions

In summary, this work has successfully demonstrated an ultrathin arrayed camera for high dynamic range imaging. The MACVA features microlens arrays with variable apertures, which capture LDR images with different brightness levels under single-shot exposure. The reconstructed HDR images clearly display not only expanded dynamic ranges surpassing LDR images, but also high resolution comparable to the maximum MTF50 value among the LDR images. Unlike conventional HDR techniques, the MACVA minimize motion artifacts effectively. Furthermore, the potential for future developments, such as adjusting the MACVA's fill factor, suggests the capability not only for high dynamic range but also high-resolution imaging. This camera provides a new platform for diverse compact and affordable imaging solution in machine vision or mobile devices.

Funding

Defense Acquisition Program Administration (UC190002D); National Research Foundation of Korea (2021R1A2B5B03002428, 2022M3H4A4085645).

Acknowledgment

This work was performed based on the cooperation with the Defense Acquisition Program Administration and Defense Rapid Acquisition Technology Research Institute's Critical Technology R&D program (UC190002D) and also financially supported by the National Research Foundation of Korea (NRF) grant funded by Korea government (MSIT) (2021R1A2B5B03002428, 2022M3H4A4085645).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. S. Mann and R. Picard, “Beingundigital’with digital cameras,” MIT Media Lab Perceptual 1, 2 (1994).

2. S. K. Nayar and T. Mitsunaga, “High dynamic range imaging: Spatially varying pixel exposures,” in Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2000), pp. 472–479.

3. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in ACM SIGGRAPH 2008 classes (2008), pp. 1–10. [CrossRef]

4. T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion,” in 15th Pacific Conference on Computer Graphics and Applications (PG'07) (2007).

5. A. Srikantha and D. Sidibe, “Ghost detection and removal for high dynamic range images: Recent advances,” Signal Process-Image 27(6), 650–662 (2012). [CrossRef]

6. J. H. Zheng, Z. G. Li, Z. J. Zhu, S. Q. Wu, and S. Rahardja, “Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects,” IEEE Trans. on Image Process. 22(12), 5190–5201 (2013). [CrossRef]

7. Z. G. Li, J. H. Zheng, Z. J. Zhu, and S. Q. Wu, “Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects,” IEEE Trans. on Image Process. 23(10), 4372–4382 (2014). [CrossRef]

8. S. K. Nayar and V. Branzoi, “Adaptive dynamic range imaging: Optical control of pixel exposures over space and time,” in Proceedings Ninth IEEE International Conference on Computer Vision (ICCV) (2003), pp. 1168–1175.

9. M. McGuire, W. Matusik, H. Pfister, B. Chen, J. F. Hughes, and S. K. Nayar, “Optical splitting trees for high-precision monocular imaging,” IEEE Comput. Grap. Appl. 27(2), 32–42 (2007). [CrossRef]

10. M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, “A Versatile HDR Video Production System,” ACM Trans. Graph. 30(4), 1–10 (2011). [CrossRef]

11. A. Morimitsu, I. Hirota, S. Yokogawa, I. Ohdaira, M. Matsumura, H. Takahashi, T. Yamazaki, H. Oyaizu, Y. Incesu, M. Atif, and Y. Nitta, “A 4 M pixel full-PDAF CMOS image sensor with 1.58 µm 2× 1 On-Chip Micro-Split-Lens technology,” in ITE Technical Report39.35 (2015), pp. 5–8.

12. W. Feng, F. M. Zhang, W. J. Wang, W. Xing, and X. H. Qu, “Digital micromirror device camera with per-pixel coded exposure for high dynamic range imaging,” Appl. Opt. 56(13), 3831–3840 (2017). [CrossRef]

13. B. Niu, X. H. Qu, X. M. Guan, and F. M. Zhang, “Fast HDR image generation method from a single snapshot image based on frequency division multiplexing technology,” Opt. Express 29(17), 27562–27572 (2021). [CrossRef]

14. N. K. Kalantari and R. Ramamoorthi, “Deep High Dynamic Range Imaging of Dynamic Scenes,” ACM Trans. Graph. 36(4), 1–12 (2017). [CrossRef]

15. Q. S. Yan, D. Gong, Q. F. Shi, A. van den Hengel, C. H. Shen, I. Reid, and Y. N. Zhang, “Attention-guided Network for Ghost-free High Dynamic Range Imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) pp. 1751–1760.

16. B. Liang, D. D. Weng, Y. H. Bao, Z. Q. Tu, and L. Luo, “Method for reconstructing a high dynamic range image based on a single-shot filtered low dynamic range image,” Opt. Express 28(21), 31057–31075 (2020). [CrossRef]

17. L. Li and A. Y. Yi, “Design and fabrication of a freeform microlens array for a compact large-field-of-view compound-eye camera,” Appl. Opt. 51(12), 1843–1852 (2012). [CrossRef]

18. Y. M. Song, Y. Xie, V. Malyarchuk, J. Xiao, I. Jung, K.-J. Choi, Z. Liu, H. Park, C. Lu, R.-H. Kim, R. Li, K. B. Crozier, Y. Huang, and J. A. Rodgers, “Digital cameras with designs inspired by the arthropod eye,” Nature 497(7447), 95–99 (2013). [CrossRef]

19. W. Yuan, L.-H. Li, W.-B. Lee, and C.-Y. Chan, “Fabrication of microlens array and its application: a review,” Chin. J. Mech. Eng. 31(1), 16 (2018). [CrossRef]

20. M. Ma, Y. Zhang, H. Deng, X. Gao, L. Gu, Q. Sun, Y. Su, and X. Zhong, “Super-resolution and super-robust single-pixel superposition compound eye,” Optics and Lasers in Engineering 146, 106699 (2021). [CrossRef]

21. D. Keum, K.-W. Jang, D. S. Jeon, C. S. Hwang, E. K. Buschbeck, M. H. Kim, and K.-H. Jeong, “Xenos peckii vision inspires an ultrathin digital camera,” Light: Sci. Appl. 7(1), 80 (2018). [CrossRef]

22. K. Kim, K.-W. Jang, S.-I. Bae, H.-K. Kim, Y. Cha, J.-K. Ryu, Y.-J. Jo, and K.-H. Jeong, “Ultrathin arrayed camera for high-contrast near-infrared imaging,” Opt. Express 29(2), 1333–1339 (2021). [CrossRef]

23. H. Deng, X. Gao, M. Ma, Y. Li, H. Li, J. Zhang, and X. Zhong, “Catadioptric planar compound eye with large field of view,” Opt. Express 26(10), 12455–12468 (2018). [CrossRef]

24. K. W. Jang, K. Kim, S. I. Bae, and K. H. Jeong, “Biologically Inspired Ultrathin Contact Imager for High-Resolution Imaging of Epidermal Ridges on Human Finger,” Adv. Mater. Technol. 6(8), 2100090 (2021). [CrossRef]

25. S.-I. Bae, K. Kim, S. Yang, K.-W. Jang, and K.-H. Jeong, “Multifocal microlens arrays using multilayer photolithography,” Opt. Express 28(7), 9082–9088 (2020). [CrossRef]

26. A. Lumsdaine and T. Georgiev, “The focused plenoptic camera,” in 2009 IEEE International Conference on Computational Photography (ICCP) (2009), pp. 1–8.

27. K. Kim, K.-W. Jang, S.-I. Bae, and K.-H. Jeong, “Multi-functional imaging inspired by insect stereopsis,” Commun Eng 1(1), 39 (2022). [CrossRef]

28. B.-H. Kang, K.-W. Jang, E.-S. Yu, H. Na, Y.-J. Lee, W.-Y. Ko, N. Bae, D. Rho, and K.-H. Jeong, “Ultrafast Plasmonic Nucleic Acid Amplification and Real-Time Quantification for Decentralized Molecular Diagnostics,” ACS nano 17(7), 6507–6518 (2023). [CrossRef]

29. D.-H. Ko, J. R. Tumbleston, K. J. Henderson, L. E. Euliss, J. M. DeSimone, R. Lopez, and E. T. Samulski, “Biomimetic microlens array with antireflective “moth-eye” surface,” Soft Matter 7(14), 6404–6407 (2011). [CrossRef]

30. H. Jung and K.-H. Jeong, “Monolithic polymer microlens arrays with antireflective nanostructures,” Appl. Phys. Lett. 101(20), 203102 (2012). [CrossRef]

31. K. Kim, K.-W. Jang, J.-K. Ryu, and K.-H. Jeong, “Biologically inspired ultrathin arrayed camera for high-contrast and high-resolution imaging,” Light: Sci. Appl. 9(1), 28 (2020). [CrossRef]

32. S. I. Bae, K. Kim, K. W. Jang, H. K. Kim, and K. H. Jeong, “High contrast ultrathin light-field camera using inverted microlens arrays with metal–insulator–metal optical absorber,” Adv. Optical Mater. 9(6), 2001657 (2021). [CrossRef]

33. Y. Zhai, J. Niu, Y. Liu, X. Chen, J. Liu, and B. Yang, “Biologically Inspired, Optical Waveguide with Isolation Layer Integrated Microlens Array for High-Contrast Imaging,” Adv. Opt. Mater. 11(14), 2300020 (2023). [CrossRef]

34. M. H. Kim, “High-fidelity colour reproduction for high-dynamic-range imaging,” University of London (2010).

Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging

Abstract

1. Introduction

2. Microfabrication of MACVA

3. MACVA characterization and HDR imaging

4. Conclusions

Funding

Acknowledgment

Disclosures

Data availability

References

Data availability

Cited By

Figures (4)

Equations (1)

Optics Express