1. Introduction

Integral imaging microscopy (IIM) is a practical three-dimensional (3D) microscopy system that acquires parallax and depth information from a specimen using a micro lens array (MLA) with a single capture [1–5]. Thus, full-parallax, full-color, and continuous-viewing features are achieved in the integral imaging technique [6–9]. The reconstruction provides multi-view representations of the specimen, depending on the viewing direction of the elemental lenses. An elemental image array (EIA) captured using an MLA includes both the parallax and depth-of-field (DoF) information of the specimen. Y.-T. Lim et al. reported an MLA shifting method in IIM in which the DoF of the IIM system can be significantly extended compared with conventional two-dimensional (2D) microscopies [10]. However, the DoF of the representation of the specimen from an IIM system might not be sufficiently deep, particularly for an objective lens with a high magnification, because each MLA has its own DoF range through which the 3D image can be reconstructed clearly and this range depends on the focal length and most MLAs have short focal lengths. Note that, there is a trade-off between the magnification and DoF that the magnification of an objective lens is proportional to the numerical aperture of an objective lens, but inversely proportional to the DoF.

Several methods have been proposed to improve the DoF of integral imaging displays using a non-uniform lens array (with non-uniform aperture sizes and focal lengths) or a varifocal liquid lens array [11–14]. Furthermore, integral imaging displays that consist of double or multilayered display devices have been reported recently [15–18]. However, these methods are complicated in practical applications and are inappropriate and impossible to implement in IIM systems.

In this paper, an IIM system is proposed and implemented using the spatial multiplexing method. The proposed spatial multiplexing method separates the optical path reflected from the specimen through the objective lens into two channels using a beamsplitter, and it provides the opportunity to acquire two EIAs simultaneously via the two MLAs and two cameras. Here, the MLAs focus on different DoF planes in the specimen. The experimental results demonstrate that the total DoF of the proposed IIM is extended by a factor of more than twofold compared with the conventional IIM. The analysis and experimental results for this issue are described and presented in the next sections.

2. Depth-of-field in integral imaging microscopy

Figure 1 presents the basic structure of the conventional IIM, which consists of the objective and tube lenses, an MLA, and an image sensor. When the visualization of the specimen is imaged onto the intermediate plane, the EIAs are captured through the MLA. The captured EIA includes not only the parallax information but also the depth information. In the reconstruction, the orthographic view images and the depth slices for the specimen are reconstructed. Here, the DoF of the IIM is comprised of various image slices with different depths within a certain depth range of the IIM. The depth slices are reconstructed using a computational integral imaging reconstruction (CIIR) algorithm [19, 20].

 

Fig. 1 Basic structure of the IIM.

Download Full Size | PDF

The potential DoF of the conventional optical microscopy considers both the geometrical and wave optical theories, which are given as follows:

With the IIM, the total DoF depends on the DoF range of the MLA and the magnification of the objective lens. The allowable DoF range of an MLA, which is the range in which an MLA can reconstruct clear images, is given as follows:

The DoF of the IIM was analyzed using a numerical simulation according to Eqs. (1)-(3). Here, all specifications of the optical devices, e.g. focal lengths of the objective lens and MLA, NA_LA, and pixel pitch of image sensor, are constant; the magnification of the objective lens is set in two conditions ( × 5 and × 10); and z is increased from 3 mm to 10 mm. Furthermore, g varies along the z transformation from 12 mm to approximately 3.1 mm according to the Gaussian lens law. Figure 2 illustrates the correlation between the allowable DoF of the MLA and the z parameter. Here, the DoF of the MLA is directly proportional to the separation between the MLA and intermediate plane, z, according to the integral imaging theory.

 

Fig. 2 Correlation between the DoF of MLA and z.

Download Full Size | PDF

As discussed earlier, the DoF of IIM depends on the D_LA and M, as depicted in Fig. 3. Here, it can be seen that the DoF of the conventional optical microscopy and IIM are inversely proportional to the magnification of the objective lens, M. For example, when the magnification is set to × 5 and NA = 0.14 mm, D_con is continual approximately 35.26 µm and D_IIM is measured to be approximately 33.15 µm to 60 µm, where z changes from 3 mm to 10 mm, as depicted in Fig. 3(a). In Fig. 3(b), D_con is degraded to approximately 8.81 µm and D_IIM is approximately 8.3 µm for z = 3 mm and 15 µm for z = 10 mm, where M = 10 and NA = 0.28 mm.

 

Fig. 3 The dependence of D_IIM on z, where (a) M = 5 and NA = 0.14 mm, and (b) M = 10 and NA = 0.28 mm.

Download Full Size | PDF

Furthermore, if a user wants to obtain 3D images for a specimen with a more enhanced DoF than the conventional 2D microscopy, z parameter must be more than 1.5 times the value of f_LA. If z < 1.5 f_LA, then D_IIM has a lower value than D_con; if z = 1.5 f_LA, D_IIM is similar to D_con. In the numerical simulation, when z becomes 3.6 mm, D_IIM is the same as D_con, which is approximately 8.3 µm and 35.26 µm according to the magnification of the objective lens, and further results are more than D_con when the z parameter is z > 1.5 f_LA, where f_LA is 2.4 mm. However, this DoF range of the 3D reconstruction in the IIM does not have sufficient depth: if the specimen size is greater than the DoF of the IIM system, it is difficult to determine the specimen size.

3. Proposed method

In order to solve the narrow DoF problem in the IIM, a spatial multiplexing system with two MLAs and two cameras is proposed. In addition, through placing a beamsplitter on the intermediate plane, the optical axis can be separated into two directions, where each of the optical axes obtains the EIAs of different focal positions with both MLAs appropriately fixed. Note that the separation between the lens array and the intermediate plane depends on the focal length of the MLA.

Figure 4 presents a schematic of the configuration of the proposed DoF-enhanced IIM system. The light rays illuminating the specimen pass through the objective and tube lenses, and the dual cameras simultaneously detect the 3D parallax and depth information of the specimen via the corresponding MLA. The MLAs focus on different depth planes of the specimen near the intermediate plane, and different depth data is captured by both EIAs. The CIIR method was used in the display system in order to reconstruct the depth-sliced images using the acquired depth data. The focused depth-sliced images for both EIAs are selected, and the entire specimen can be observed.

 

Fig. 4 Schematic of the proposed IIM system with dual MLAs and dual cameras.

Download Full Size | PDF

With the proposed IIM system, the three types of conditions can appear:

In the proposed IIM system, the primary analyses are discussed for the first and second conditions; however, the third condition is not considered because it is inconsistent with the conventional and proposed IIM features, which have a depth-sliced reconstruction.

The calculation of the DoF enhancement is accomplished when the two MLAs have the same elemental lens pitch but have different focal lengths, or two MLAs have identical specifications and the z parameters are transformed. The DoFs acquired from each of the EIAs using the dual MLAs can be expressed using Eq. (3), as follows:

Figure 5(a) illustrates the difference in the sampling effects among the elemental lenses with different focal lengths. As depicted in Fig. 5(b), the CIIR algorithm is executed twice for each EIA; i.e. the depth-sliced images reconstructed from the first EIA captured using MLA1 and camera1 represent the front-to-middle parts of the resulting image, and the depth-sliced images reconstructed from the second EIA represent the middle-to-back parts of the resulting image. The entire DoF of the specimen, i.e. D_tot, can be observed using the two MLAs, where a single MLA cannot reconstruct at the entire DoF of the specimen.

 

Fig. 5 Schematic of the DoF enhancement in the proposed IIM using two MLAs: (a) pickup and (b) reconstruction.

Download Full Size | PDF

As discussed earlier, two conditions of DoF ranges were considered; however, both ranges are fully separated. In general, i.e. for the first condition, the total DoF for the reconstructed depth-sliced images from both EIAs, i.e. D_{tot_gen}, is determined using each DoF range formed by two MLAs (D_IIM1 and D_IIM2), the difference between the z₁ and z₂ parameters, and the overlapped portion of both DoF ranges (D_ov), as depicted in Fig. 6(a).

 

Fig. 6 Schematic for the (a) first and (b) second conditions.

Download Full Size | PDF

Here, D_{tot_gen} can be calculated as follows:

For the second condition, when the back-and-forth marginal planes of D_IIM1 and D_IIM2 are coincident, as depicted in Fig. 6(b), D_tot has its maximum value, which is the sum of both DoF ranges, as follows:

Furthermore, the separations between each MLA and its corresponding camera can be made equal (i.e. g₁ = g₂), even where the MLAs have different focal lengths. Conversely, if the MLAs have identical focal lengths, then g₁ > g₂ is required in order to acquire images at different depths. Because z₁ and z₂ are determined by the Gaussian lens law considering the g and f_LA parameters, extensions for the depth-of-field are not detected when the z, g, and f_LA are identical.

Using this analysis, another numerical simulation was conducted in order to verify the DoF enhancement using the proposed IIM system. Here, the numerical simulation is conducted for two different cases: in the first case, the DoF is analyzed for different f_LA and z parameters with identical g parameters and, in the second case, it is analyzed for different g and z parameters with identical f_LA parameters. The other specifications of the components are fixed as follows: λ = 532 nm, M = 10 × , NA = 0.28 mm, P_S = 6 µm, and P_EL = 125 µm. In the first case, the focal lengths of the MLAs are set as f_LA1 = 2.4 mm and f_LA2 = 2.7 mm, g₁ and g₂ are identical, and z₁ and z₂ are from 3 mm to 10 mm in increments of 0.5 mm. In the second case, f_LA1 and f_LA2 are 2.4 mm, but g₂ is 1.25 times less than g₁. The simulation results for these two cases are depicted in Figs. 7(a) and 7(b), respectively. Note that D_tot is given as the maximum value (D_{tot_max}) for simplified calculations.

 

Fig. 7 Numerical simulation results: (a) the first case where f_LA1 ≠ f_LA2 and g₁ = g₂, and (b) the second case where f_LA1 = f_LA2 and g₁ > g₂.

Download Full Size | PDF

As seen in Fig. 7(a), the DoF is directly proportional to the focal length distance of the MLA, and the proposed method has a significantly improved DoF range compared with the conventional IIM and 2D optical microscopic systems. For example, when z₁ and z₂ are 7 mm, g₁ and g₂ are approximately 3.65 mm, D_IIM1 and D_IIM2 are approximately 12.26 µm and 14.9 µm, respectively, and D_tot is approximately 27.16 µm where D_con is 8.93 µm.

In the second case in the numerical simulation, as seen in Fig. 7(b), it was verified that different g parameters create different DoF ranges in the 3D reconstruction and the DoF difference will be greater according to the increments of the z parameter. Here, the focal lengths of the MLAs, i.e. f_LA1 and f_LA2, are fixed at 2.4 mm and g₂ is 1.25 times less than g₁, where z₁ and z₂ are from 3 mm to 10 mm with 0.5 mm increments. In this case, the DoF in the proposed IIM system is also significantly improved compared with the conventional optical microscopy and IIM using a single MLA. For example, when z₁ = z₂ = 7 mm, g₁ and g₂ are approximately 3.65 mm and 2.92 mm, respectively, D_IIM1 and D_IIM2 are approximately 12.26 µm and 13.37 µm, respectively, and D_tot is approximately 25.63 µm, where D_con is 8.93 µm.

From these numerical simulations, two MLAs with different focal lengths and two identical MLAs with different separations from the image sensor successfully enhance the DoF range of the IIM more than twofold that of the conventional case. This indicates that the proposed method is very efficient when it is applied in IIM. Note that a slight difference can occur between the two cases according to the focal lengths of the MLAs or the difference between the g₁ and g₂ parameters.

4. Experimental results

In order to verify the DoF enhancement method, a prototype of the proposed IIM system is constructed and implemented using two MLAs and two cameras; the prototype is depicted in Fig. 8. In the experiment, the objective and tube lenses have a magnification of 10 × ; telecentric reflective illumination, a 50/50 beamsplitter, and dual cameras with 1” CMOS sensors and 2048 × 2048 RGB pixels are also utilized. The MLAs are identical, with a focal length of 2.4 mm and 100 × 100 micro lenses with elemental lens pitches of 125 µm.

 

Fig. 8 A photograph of the prototype DoF-enhanced IIM system.

Download Full Size | PDF

There are three types of micro objects are used as specimens in the experiment: a surface-mounted resistor, a stamen and pistil from a chrysanthemum flower, and the eyes of a fruit fly. Figure 9 presents the visualizations of the experimental objects and the corresponding EIAs captured through the two MLAs and cameras, and the generated orthographic-view images. For each object, the separation between the parts captured using the MLAs is approximately 200 µm. The six EIAs for the three objects consisted of 2020 × 2020 pixels on average, where each elemental image included 23 × 23 pixels. Here, the regions-of-interest, which are the central portions of each EIA, are selected for all EIAs, because a lens distortion issue occurrs in the real capturing process. In particular, the distortion became more intense toward the outer boundary portions of the captured EIA. Finally, the generated orthographic-view images consisted of 23 × 23 directional-view images.

 

Fig. 9 (a) The 2D images for top (left) and bottom (right) sides of a surface-mounted resistor, the corresponding EIAs, and reconstructed orthographic-view images; (b) images of the stamen (left) and pistil (right) of a chrysanthemum flower and their corresponding EIAs, and the reconstructed orthographic-view images; and (c) images of the left eye (left) and right eye (right) of a fruit fly, their corresponding EIAs, and orthographic-view images.

Download Full Size | PDF

Object 1 is a surface-mounted resistor, and the MLAs focus on the top (via MLA1, left) and bottom (via MLA2, right) planes, as depicted in Fig. 9(a). For object 2, the MLAs focus on different parts inside the chrysanthemum flower, i.e. the stamen (via MLA1, left) and pistil (via MLA2, right), as seen in Fig. 9(b). As depicted in Fig. 9(c), the MLAs focus on the left (via MLA1, left) and right (via MLA2, right) eyes of a fruit fly for object 3.

Due to the identical MLAs used in the experiment, the dual MLAs are located at different distances from the intermediate plane and the center of the beamsplitter, such that z₁ ≈ 7.2 mm and z₂ ≈ 9.2 mm, through setting g₁ ≈ 3.6 mm and g₂ ≈ 3.25 mm, respectively. The reconstructed depth slice images for each object and EIA through the CIIR algorithm are presented in Fig. 10: Fig. 10(a) presents the depth slices reconstructed from the EIAs for a surface-mounted resistor; Fig. 10(b) is the depth slices for the stamen and pistil of the chrysanthemum flower; and Fig. 10(c) is the depth slices for the eyes of a fruit fly. Here, some of the reconstructed depth slices are focused and some are unfocused. From Fig. 10, it is not possible to observe the overall appearances of the three specimens using the IIM system with a single MLA that leads to the DoF of a single MLA-based IIM being smaller than the entire depth of the specimen. In the experiment using the proposed method, the well-focused images are selected from the reconstructed depth slices. The well-focused images are indicated using a dashed red line in Fig. 10.

 

Fig. 10 The reconstructed depth slices for each EIA captured from the different depth planes of the specimens: the depth slices for corresponding EIA1 (left) and EIA2 (right) of (a) a surface-mounted resistor (Visualization 1), (b) a chrysanthemum flower (Visualization 2), and (c) a fruit fly (Visualization 3).

Download Full Size | PDF

The well-focused depth slice image selection process depends on the corresponding power spectral density (PSD) value for each depth slice, where the threshold is 5.3 dB. In the other hands, the depth slice images which with PSD values more than 5.3 dB are picked as the well-focused images. Note that, 5.3 dB is most efficient value for the PSD measurements of reconstructed depth-slices in the experimental condition of proposed IIM system.

Figure 11 illustrates the measured PSD values of the depth-sliced reconstructions for the corresponding EIAs. The PSD values for the depth slices selected as the well-focused images for the surface-mounted resistor were approximately 5.73 dB at ∆z_2RT ≈30 µm and 5.94 dB at ∆z_3RT ≈60 µm from EIA1, and 6.04 dB at ∆z_2RB ≈130 µm and 5.63 dB at ∆z_2RB ≈160 µm from EIA2 [Fig. 11(a)]; for the stamen and pistil of the chrysanthemum flower, the PSD values were approximately 5.6 dB at ∆z_2CS ≈30 µm and 5.93 dB at ∆z_3CS ≈60 µm from EIA1, and 5.65 dB at ∆z_2CP ≈130 µm mm and 5.85 dB at ∆z_3CP ≈160 µm from EIA2 [Fig. 11(b)]; and for the eyes of the fruit fly, the PSD values were approximately 6.07 dB at ∆z_2FL ≈30 µm and 6.06 dB at ∆z_3FL ≈60 µm from EIA1, and 6.0 dB at ∆z_2FR ≈130 µm and 6.01 dB at ∆z_3FR ≈160 µm from EIA2 [Fig. 11(c)].

 

Fig. 11 The PSD values for each of the depth-sliced reconstructions from EIA1 and EIA2: (a) the surface-mounted resistor, (b) the chrysanthemum flower, and (c) the fruit fly.

Download Full Size | PDF

The selected well-focused depth slices were combined according to the PSD values and are depicted in Fig. 12. The experimental results demonstrate that the proposed IIM using the spatial multiplexing method successfully enhanced the DoF of the IIM system, and focused images of each depth-sliced reconstruction are obtained for the different depth planes in the specimen. From Figs. 12(a)-12(c), the overall appearances of the three objects are successfully represented when an objective lens was configured with 10 × magnification.

 

Fig. 12 The overall visualizations of the three objects with combined the well-focused depth slices: (a) a surface-mounted resistor (Visualization 4), (b) a chrysanthemum flower (Visualization 5), and (c) a fruit fly (Visualization 6).

Download Full Size | PDF

5. Conclusion

It can be concluded that the proposed IIM spatial multiplexing method using dual MLAs enables an increase in the DoF of a typical single channel IIM system. The two lens arrays focus on different depth planes of the specimen, and two corresponding cameras capture the different EIAs simultaneously; this is the general implementation of proposed IIM system. The dual lens arrays can have either different or identical focal lengths. The proposed system can extend the depth-of-field maximum of DoF1 + DoF2. The different focal lengths of MLAs do not affect the magnification of the specimen. Further work will be conducted in order to identify more effective DoF-enhancement methods and real-time implementations of the IIM.

Acknowledgments

This work was supported by “The Cross-Ministry Giga KOREA Project” grant from the Ministry of Science, ICT and Future Planning, Korea, and was supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2015-R0992-15-1008) supervised by the IITP (Institute for Information & Communications Technology Promotion).