Open Access
Add to BibSonomyAbstract
Due to the limitations of micro lens arrays and camera sensors, images on display devices through the integral imaging microscope systems have been suffering for a low-resolution. In this paper, a resolution-enhanced orthographic-view image display method for integral imaging microscopy is proposed and demonstrated. Iterative intermediate-view reconstructions are performed based on bilinear interpolation using neighborhood elemental image information, and a graphics processing unit parallel processing algorithm is applied for fast image processing. The proposed method is verified experimentally and the effective results are presented in this paper.
© 2015 Optical Society of America
1. Introduction
An optical microscope is a piece of equipment that enlarges very small objects for viewing with the human eye, but the observer can see only a two-dimensional (2D) representation of the specimen, not a three-dimensional (3D) [1]. Microscopic systems based on stereoscopic and confocal techniques, etc., obtain 3D depth information of the specimen and are widely used in medical operations and inspection processes in hospitals and medical research institutes [2]. However, parallax information cannot be acquired, and a real-time display is not provided, especially in a confocal microscope system.
The first microscope system using a micro lens array (MLA), which is a light field microscope, was reported by M. Levoy et al. [3, 4]. A MLA provides a simultaneous acquisition of 3D depth and parallax information of a specimen through a single shot, based on the integral imaging technique. Integral imaging is one of the rapidly developing 3D autostereoscopic displays / image processing techniques that display full-parallax, full-color and continuous-viewing 3D images, within a certain viewing region with insufficient viewing characteristics such as low resolution, small depth range and narrow viewing angle due to the limitations of lens array and camera sensor [5–8]. An integral imaging microscope (IIM) also captures the specimen through an MLA, but the concept is different that a single point of the specimen can be captured through multiple elemental lenses, whereas a single point of the specimen passes through only a single corresponding elemental lens in light field microscopy [9]. Therefore, the IIM can acquire more accurate 3D information of the specimen. A digital reconstruction process is utilized in the display to reveal the directional-view images without using a MLA. In accordance with the development of digital image processing technology, processing time from capturing to display is getting faster; however, the drawbacks of the integral imaging technique, especially the resolution, remain a problem in an IIM.
Many studies have been reported up to now on how to improve the resolution of an IIM [10–13]. Most of them are based on mechanical movement of the optical device, such as a 3D camera system employing a scanning MLA and a stationary pinhole array [10], synchronously moving the lens array [11], sensing and recognition of the 3D object using time multiplexed computational integral imaging [12], and simple shifting of the MLA [13]. However, these methods require a long time for processing due to the mechanical movement of the MLA, and accuracy is also needed.
Intermediate-view reconstruction (IVR) is an in-between image based on disparity information between two neighboring images, and the number of IVRs can be synthesized digitally, where the disparity values between neighboring elemental images should be separated exactly [14–16]. Nevertheless, it is a little difficult to generate the IVRs in an elemental image array (EIA) captured through an IIM system. The disparity value between the elemental images is exceedingly small, because the size of each elemental lens is only few micrometers.
Recently, K.-C. Kwon et al. presented a compound IIM system that generates intermediate-view images and orthographic-view images, and displays the directional-view images according to user interactivity in real-time [17]. The IVRs for the disparity information enhance the resolution of the captured EIA by using an interpolation method in horizontal, vertical and diagonal directions. Based on the newly generated resolution-enhanced EIA, the orthographic-view image is generated digitally, and directional-view images are displayed on the display device. The observer can replace different viewpoints of the directional-view images through an interactive technique. All of these complex processes are run through graphics processing unit (GPU) parallel processing, and a real-time display is provided [18–20]. However, the main goal of this method is real-time display for the IIM by using GPU parallel processing and just verified the possibility of resolution enhancement by generating the IVRs; therefore, in-depth analysis of cases using an interpolation method was not included. If a bilinear interpolation process is proceed more than two times, the resolution of the directional-view images naturally improve; however, calculation of the IVR becomes incredibly complicated and requires a very long processing time.
In this study, we propose a high-speed directional-view image display method with considerable improved resolution in the IIM. To properly ensure the resolution-enhancement, three times of iterations of the IVR generation processes are conducted based on the disparity information between neighboring elemental images, and they are applied with GPU parallel processing for fast computation. Also, several in-depth analyses have been conducted for analyzing the performances of proposed IIM system such as the displayed directional-view image quality, significant improvement for the resolution of EIA, the functional analysis for the how much of enhancement of the EIA resolution can be completed within the best possible short-term, which are presented in this paper. The implemented IIM display presents resolution-enhanced directional-view images with a 513 × 513 pixel size by using bilinear interpolation, where the initially captured EIA has elemental images at 65 × 65 pixels.
2. System configuration of the IIM
First, we explain the basic principle of the IIM, as illustrated in Fig. 1. The optical structure of the IIM using an infinity corrected optical system consists of the objective lens, tube lens, micro lens array, and image sensor [13, 17]. The specimen is imaged via objective lens and tube lens onto the intermediate plane, and the EIA is captured through an MLA. If the microscope is focused correctly, the distance from the specimen to the first principal plane of the objective lens, and from its second principal plane to the telecentric stop, are both equal to the focal length of the objective lens (fOL). The distance from the first principal plane (the telecentric stop) to the tube lens and the distance from the tube lens to second principal plane of the tube lens (the intermediate plane) are equal to the focal length of the tube lens (fTL). Also, the intermediate plane locates at the focal length of the MLA (fLA), where the gap between the MLA and sensor is fixed as g.
An EIA consists of the elemental images captured by the corresponding elemental lenses. Figure 2 shows a schematic diagram for the geometry of the integral imaging pickup system. An object point with (y, z) coordinate is captured by a corresponding q-th elemental lens, and the corresponding elemental image is given with respect to the center of the q-th elemental lens by:
where P is the elemental lens pitch. The disparity value Δyq between the captured elemental image points as illustrated by q1 and q2 neighboring elemental images, is given by Eq. (2):A MLA placed at form by multiple elemental images which are captured on the sensor plane, where the captured elemental image array includes the depth and coordinate information of the specimen, as shown in Fig. 2. When elemental images for the multiple objects or object points are captured through the MLA, the gaps between pixels become different (farther away or closer) according to the viewpoint. Based on the acquired disparity information, the orthographic-view image is generated and observed by the user on the display device. Here, the orthographic-view image includes the many sub-images (directional-view images), which are different view directions horizontally and vertically, and the observer can choose the viewpoint that he or she wants to see.
3. Considerable resolution-enhancement by iterative intermediate-view reconstruction
The resolution of the IIM system strongly depends on the number of elemental images. The high-resolution 3D image display of the IIM requires sufficient overlapping of elemental images. In this paper, an iterative IVR technique based on a bilinear interpolation method is employed to obtain an orthographic view with sufficient resolution based on the captured elemental images without requiring additional optical devices. The flowchart of the proposed resolution-enhancement method for the IIM is shown in Fig. 3. When the user defines the number of elemental images (NEI), the IVR-based bilinear interpolation process iterates until the resolution is sufficiently enhanced. In other words, the algorithm is designed as the bilinear interpolation process is repeated until it attains the user-defined EIA size, or number of elemental images.
First, the realignment process, which includes an adjustment of image and the selection of effective area, is run based on the captured EIA by using the IIM during preprocessing. The next process is generating the IVRs, which are adapted to the bilinear interpolation method several times for the selected effective area in the aligned EIA. The bilinear interpolation method calculates and generates the intermediate information between each four original neighboring elemental images. The generation process of IVRs using the bilinear interpolation method performed by the GPU parallel processing is illustrated in Fig. 4.
First of all, the threads are created in the GPU as the same number of elemental images which will be increased, and all of the IVRs are generated simultaneously through the GPU parallel processing for each neighboring four elemental images. For each of the four elemental images, firstly the two IVRs are determined between the neighboring elemental images in the horizontal direction. Then, other two IVRs between the neighboring elemental images are calculated for the original elemental images in vertical direction and the central IVR located at relative distance between the original four elemental images is calculated by performing the interpolation for two newly determined IVRs horizontally, in same time; so at least five IVRs are generated from among the four original elemental images (two horizontally and three vertically). EIIVT and EIIVB, the IVRs generated along the horizontal direction for each of the selected four elemental images EITL, EITR, EIBL and EIBR, are calculated with Eq. (3):
Then, EIIVL and EIIVR, the IVRs generated along the vertical direction, and EIIVC, central IVR between the original elemental images, are generated by using Eq. (4):where α and β are the weight values to generate IVRs in the horizontal and vertical directions.After iterative generation of the IVR, an orthographic-view image is reconstructed based on the resolution-enhanced EIA, and the directional-view image (a sub-image from the user-defined viewpoint) is presented on the display device. Here, another GPU parallel processing is utilized. Because, for the EIAs with very high-resolution of the implemented IIM system, the generation process of the orthographic-view image requires a very long computation time, where the directional-view images are constructed with a combination of the pixels with same coordinates from each elemental image. The directional-view image reconstruction process is illustrated in Fig. 5.
Fig. 5 The reconstruction process for each directional-view image based on the resolution-enhanced EIA.
Here, each directional-view image is directly generated based on the resolution-enhanced EIAs and displayed in high-speed, instead of generating the entire orthographic-view image and displaying its directional-view images one by one.
Figure 6 shows an example of resolution-enhancement for a real specimen. The skin of a bronchial tube, captured without an MLA, is shown in Fig. 6(a), while Fig. 6(b) and Fig. 6(c) show the initially grabbed EIA from using the implemented IIM and the resolution-enhanced EIA from iterative IVR, respectively. By realigning and selecting the effective area of the obtained EIA during the preprocessing, the corrected EIA can be obtained. The directional-view image generated from the resolution-enhanced EIA at 513 × 513 pixels after three IVR iterations, while the directional-view image based on the corrected EIA at 65 × 65 pixels. Figures 6(d) and 6(e) present the reconstructed orthographic-view images based on Figs. 6(b) and 6(c).
Fig. 6 Example of resolution enhancement in the IIM by the proposed method. (a) The skin of a bronchial tube imaged through a conventional microscope, (b) the original EIA captured by the IIM, (c) a resolution-enhanced EIA via three times of iterative IVR, (d) the orthographic-view image based on the initially captured EIA, and (e) the orthographic-view image based on the resolution-enhanced EIA. Note that the directional-view image at 65 × 65 pixels in the (d) and at 513 × 513 pixels in the (e).
Note that the directional-view images can be displayed in real-time when the IVRs are generated just once with the bilinear interpolation method. However, if the IVR generation process iterates more than two times, there might not be enough of memory in the PC.
4. Experimental results
Based on the analysis and simulation as explained above, the IIM system and corresponding image processing software were implemented. The prototype comprises an Olympus BX41 microscope, an MLA with 125 µm 100 × 100 micro lenses, and a 4.1 megapixel CCD camera with a Nikon 105 mm macro lens. Figure 7(a) shows an experimental prototype of the proposed IIM system, Fig. 7(b) shows the initially captured and resolution-enhanced EIAs after iterative IVRs for the eye of a fruit fly, and the corresponding orthographic-view images are presented in Fig. 7(c).
Fig. 7 (a) Prototype IIM for the proposed resolution-enhancing method consisting of the IIM and camera, a PC and an LCD. (b) The initially captured EIA (smallest image, bottom left) and resolution-enhanced EIAs after iterative interpolation processes, and (c) reconstructed orthographic-view images based on the corresponding EIAs.
The specifications of the optical devices required in the experiment are listed in the Table 1.
Table 1. Specifications for the proposed prototype IIM system.
The proposed method was verified with a real experiment. Specimens chosen for the experiment were a fruit fly, a dayfly, and skin of bronchial tube. By using the implemented IIM system, the color EIA at 2048 × 2048 pixels (90 fps) was captured through the MLA with 100 × 100 micro lenses. The size of the selected effective area of the initially captured EIA after the realignment process was 1625 × 1625 pixels with 65 × 65 elemental images where a single elemental image was composed of 25 × 25 pixels. After the three iterations of bilinear interpolation processes, the resolution of the EIA was enhanced to 12825 × 12825 pixels, and directional-view images were generated and displayed for the resolution-enhanced EIAs. Images in each part of Fig. 8 (from top to bottom) show the captured EIA, the resolution-enhanced EIA and an example of the reconstructed directional-view image for the corresponding specimens.
Fig. 8 Experimental results for evaluation of the IIM and the displayed directional-view images: (a) dayfly, (b) fruit fly, (c) the eye of a fruit fly, (d) the skin of a bronchial tube, and (e) an enlarged area of skin from the bronchial tube.
Measurement of the processing time was run for IVR generation and directional-view image reconstruction. GPU parallel processing is required in the implemented IIM system, and we confirmed that the directional-view images at 513 × 513 pixels can be displayed in lessthan one second, while the CPU-based case requires a lot of processing time, more than four hundred seconds, as illustrated in the graphs of Fig. 9. Due to so dramatically enhancement of processing data, the processing time cannot satisfy real-time display through the common devices used in the experiment where IVR generation process iterated three times. But it can deal with quite a fast time (less than a second). Note that when the IVRs are generated once, the directional-view images are displayed in real-time.
Evaluation of directional-view image quality using the proposed method was performed by peak-signal-to-noise ratio (PSNR) and image fidelity (IF) [21] calculations between the original and resolution-enhanced EIAs. Table 2 shows the PSNR and IF values for the resolution-enhanced directional-view images for the corresponding specimens. The PSNR and IF values for the five test images were obtained in ranges from 25.7 to 32.89 dB (PSNR) and from 0.8933 to 0.9717 units (IF), respectively. This may be a weak result compared with conventional virtual object–based systems, but the test images in the existing methods were the simple object images, which clearly separated the background (black) from the object. Test targets were reconstructed by using the proposed IVR method for II obtained through the implemented IIM from real specimens.
Table 2. PSNR and IF value for the proposed resolution-enhanced IIM.
Figure 10 shows a graph of the quality assessment for displayed directional-view image that comparing with the original image of the specimen, after five iterations of the IVR generation process. The PSNR value decreases by approximately 1.8 dB constantly in each iteration of the bilinear interpolation process in the experiment. Here, the objects are presented in sequence as listed in the Table 2, such that Obj.1 is the dayfly, Obj.2 is the fruit fly, Obj.3 is an eye if fruit fly, Obj.4 is the skin of bronchial tube, and Obj.5 is the enlarged skin of bronchial tube. Note that, the specimen is real 3D object, therefore we captured the image of the specimen without lens array, and measured the image quality of the displayed central directional-view image compared with captured 2D image.
Fig. 10 The PSNR value for the objects listed in the Table 2 up to 5 iterations of the bilinear interpolation process.
When four iterations of the bilinear interpolation process proceed through IIM system, the original resolution of the directional-view image, 65 × 65 pixels (EIA at 1625 × 1625 pixels), can be improved to 1025 × 1025 pixels (EIA at 25625 × 25625 pixels). However, the processing time is also getting increased according to the multiple iterations. In the experiment, the three iterations of the interpolation process was the most efficient case that obtain the directional-view images with 513 × 513 pixels, from the EIA with 12825 × 12825 pixels, within one second, where the average PSNR value is over 27 dB.
To verify the above experimental results, we prepared videos of the displayed directional-view images in the original format (the resolution of the EIA did not been enhanced) and in resolution-enhanced format after 3 iterations of the IVR generation process. Figure 11 shows a screen shot of a comparison video.
Fig. 11 The example of the comparison of the directional-view image display between the original and the proposed resolution-enhancing method (Media 1: Dayfly; Media 2: Fruit fly; Media 3: The eye of a fruit fly; Media 4: The skin of a bronchial tube: Media 5: An enlarged area of skin from the bronchial tube).
5. Conclusion
An iterative IVR-based resolution-enhancement method for the IIM was analyzed and implemented in a real IIM system. The proposed resolution-enhancing method augments IIM based orthographic-view image display by using IVRs obtained through bilinear interpolation processes. From the experiment, it seems like a weak result was obtained, when compared with conventional interpolation-based methods, but the conventional systems used a simple input image that was fully separated from a black background. For the experimental results, the proposed method successfully and sufficiently enhances resolution for the IIM system when using complex real scenes through. When the bilinear interpolation method is iterated three times, the most efficient directional-view image is generated and displayed in a short processing time (less than 1 second) and good image quality (more than 27dB), where resolution-enhanced EIA at 12825 × 12825 pixels and each corresponding directional-view image at 513 × 513 pixels. The proposed system can be useful in a high-resolution display that presents directional-view images or videos based on an optical 3D IIM. In further research of proposed IIM system, another methods such as bicubic interpolation method etc. will be applied for the enhancement of EIA resolution through the better equipment.
Acknowledgment
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. NRF-2014R1A2A2A01003934) (No. 2011-0030079). The authors’ special thank goes to Prof. S.-S. Kang, from Dept. of Biology Education, Chungbuk Nat’l. Univ., for providing the specimen samples.
References and links
1. B. Herman and J. J. Lemasters, Optical microscopy: Emerging methods and applications (Academic, 1985).
2. K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010). [CrossRef]
3. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM SIGGRAPH 25(3), 924–934 (2006). [CrossRef]
4. M. Levoy, Z. Zhang, and I. McDowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235(2), 144–162 (2009). [CrossRef] [PubMed]
5. G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).
6. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009). [CrossRef] [PubMed]
7. N. Kim, A.-H. Phan, M.-U. Erdenebat, M. A. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. 1, 73–95 (2014).
8. N. Kim, M. A. Alam, L. T. Bang, A.-P. Phan, M.-L. Piao, and M.-U. Erdenebat, “Advances in the light field displays based on integral imaging and holographic techniques,” Chin. Opt. Lett. 12, 060005 (2014). [CrossRef]
9. J.-S. Jang and B. Javidi, “Three-dimensional integral imaging of micro-objects,” Opt. Lett. 29(11), 1230–1232 (2004). [CrossRef] [PubMed]
10. L. Erdmann and K. J. Gabriel, “High-resolution digital integral photography by use of a scanning microlens array,” Appl. Opt. 40(31), 5592–5599 (2001). [CrossRef] [PubMed]
11. J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27(5), 324–326 (2002). [CrossRef] [PubMed]
12. S. Kishk and B. Javidi, “Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging,” Opt. Express 11(26), 3528–3541 (2003). [CrossRef] [PubMed]
13. Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17(21), 19253–19263 (2009). [CrossRef] [PubMed]
14. K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42(6), 1778–1786 (2003). [CrossRef]
15. D.-C. Hwang, J.-S. Park, S.-C. Kim, D.-H. Shin, and E.-S. Kim, “Magnification of 3D reconstructed images in integral imaging using an intermediate-view reconstruction technique,” Appl. Opt. 45(19), 4631–4637 (2006). [CrossRef] [PubMed]
16. S. Zhu and Y. Yu, “Intermediate view synthesis based on disparity estimation and image interpolation,” in Proceedings of IEEE International Conference on Computer Science and Information Processing (IEEE, 2012), pp. 1035–1038.
17. K.-C. Kwon, J.-S. Jeong, M.-U. Erdenebat, Y.-T. Lim, K.-H. Yoo, and N. Kim, “Real-time interactive display for integral imaging microscopy,” Appl. Opt. 53(20), 4450–4459 (2014). [CrossRef] [PubMed]
18. K.-C. Kwon, C. Park, M.-U. Erdenebat, J.-S. Jeong, J.-H. Choi, N. Kim, J.-H. Park, Y.-T. Lim, and K.-H. Yoo, “High speed image space parallel processing for computer-generated integral imaging system,” Opt. Express 20(2), 732–740 (2012). [CrossRef] [PubMed]
19. D.-H. Kim, M.-U. Erdenebat, K.-C. Kwon, J.-S. Jeong, J.-W. Lee, K.-A. Kim, N. Kim, and K.-H. Yoo, “Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array,” Appl. Opt. 52(34), 8411–8418 (2013). [CrossRef] [PubMed]
20. J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53(1), 015103 (2014). [CrossRef]
21. A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” IEEE Trans. Commun. 43(12), 2959–2965 (1995). [CrossRef]
