Abstract

A real-time interactive orthographic-view image display of integral imaging (II) microscopy that includes the generation of intermediate-view elemental images (IVEIs) for resolution enhancement is proposed. Unlike the conventional II microscopes, parallel processing through a graphics processing unit is required for real-time display that generates the IVEIs and interactive orthographic-view images in high speed, according to the user interactive input. The real-time directional-view display for the specimen for which 3D information is acquired through II microscopy is successfully demonstrated by using resolution-enhanced elemental image arrays. A user interactive feature is also satisfied in the proposed real-time interactive display for II microscopy.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).
  2. G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).
  3. J.-H. Park, K.-H. Hong, and B.-H. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48, H77–H94 (2009).
    [CrossRef]
  4. G. Li, K.-C. Kwon, G.-H. Shin, J.-S. Jeong, K.-H. Yoo, and N. Kim, “Simplified integral imaging pickup method for real objects using a depth camera,” J. Opt. Soc. Korea 16, 381–385 (2012).
    [CrossRef]
  5. N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).
  6. J. Kim, J.-H. Jung, C. Jang, and B. Lee, “Real-time capturing and 3D visualization method based on integral imaging,” Opt. Express 21, 18742–18753 (2013).
    [CrossRef]
  7. L. Erdmann and K. J. Gabriel, “High-resolution digital integral photography by use of a scanning microlens array,” Appl. Opt. 40, 5592–5599 (2001).
    [CrossRef]
  8. J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27, 324–326 (2002).
    [CrossRef]
  9. S. Kishk and B. Javidi, “Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging,” Opt. Express 11, 3528–3541 (2003).
    [CrossRef]
  10. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).
  11. M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).
    [CrossRef]
  12. Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17, 19253–19263 (2009).
    [CrossRef]
  13. B. Lee and J. Kim, “Real-time 3D capturing-visualization conversion for light field microscopy,” Proc. SPIE 8769, 876908 (2013).
    [CrossRef]
  14. NVIDIA, “OpenCL programming guide for the CUDA architecture,” Version 2.3 (2009).
  15. NVIDIA, “CUDA C programming guide,” Version 3.1.1 (2010).
  16. D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” Pattern Recogn. 13, 111–122 (1981).
    [CrossRef]
  17. K.-C. Kwon, C. Park, M.-U. Erdenebat, J.-S. Jeong, J.-H. Choi, N. Kim, J.-H. Park, Y.-T. Lim, and K.-H. Yoo, “High speed image space parallel processing for computer generated integral imaging system,” Opt. Express 20, 732–740 (2012).
    [CrossRef]
  18. D.-H. Kim, M.-U. Erdenebat, K.-C. Kwon, J.-S. Jeong, J.-W. Lee, K.-A. Kim, N. Kim, and K.-H. Yoo, “Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array,” Appl. Opt. 52, 8411–8418 (2013).
    [CrossRef]
  19. J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
    [CrossRef]
  20. K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42, 1778–1786 (2003).
    [CrossRef]
  21. D.-C. Hwang, J.-S. Park, S.-C. Kim, D.-H. Shin, and E.-S. Kim, “Magnification of 3D reconstructed images in integral imaging using an intermediate-view reconstruction technique,” Appl. Opt. 45, 4631–4637 (2006).
    [CrossRef]
  22. S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

2014

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

2013

2012

2011

S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

2010

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

2009

2006

2003

K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42, 1778–1786 (2003).
[CrossRef]

S. Kishk and B. Javidi, “Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging,” Opt. Express 11, 3528–3541 (2003).
[CrossRef]

2002

2001

1981

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” Pattern Recogn. 13, 111–122 (1981).
[CrossRef]

1908

G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).

Adams, A.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Alam, A. M.

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

Bae, K.-H.

K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42, 1778–1786 (2003).
[CrossRef]

Ballard, D. H.

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” Pattern Recogn. 13, 111–122 (1981).
[CrossRef]

Choi, J.-H.

Erdenebat, M.-U.

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

D.-H. Kim, M.-U. Erdenebat, K.-C. Kwon, J.-S. Jeong, J.-W. Lee, K.-A. Kim, N. Kim, and K.-H. Yoo, “Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array,” Appl. Opt. 52, 8411–8418 (2013).
[CrossRef]

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

K.-C. Kwon, C. Park, M.-U. Erdenebat, J.-S. Jeong, J.-H. Choi, N. Kim, J.-H. Park, Y.-T. Lim, and K.-H. Yoo, “High speed image space parallel processing for computer generated integral imaging system,” Opt. Express 20, 732–740 (2012).
[CrossRef]

Erdmann, L.

Footer, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Gabriel, K. J.

Hong, J.-M.

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

Hong, K.-H.

Horowitz, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Hwang, D.-C.

Jang, C.

Jang, J.-S.

Javidi, B.

Jeong, J.-S.

Jung, J.-H.

Kim, C.-K.

S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

Kim, D.-H.

Kim, E.-S.

S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

D.-C. Hwang, J.-S. Park, S.-C. Kim, D.-H. Shin, and E.-S. Kim, “Magnification of 3D reconstructed images in integral imaging using an intermediate-view reconstruction technique,” Appl. Opt. 45, 4631–4637 (2006).
[CrossRef]

K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42, 1778–1786 (2003).
[CrossRef]

Kim, J.

B. Lee and J. Kim, “Real-time 3D capturing-visualization conversion for light field microscopy,” Proc. SPIE 8769, 876908 (2013).
[CrossRef]

J. Kim, J.-H. Jung, C. Jang, and B. Lee, “Real-time capturing and 3D visualization method based on integral imaging,” Opt. Express 21, 18742–18753 (2013).
[CrossRef]

Kim, K.-A.

Kim, N.

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

D.-H. Kim, M.-U. Erdenebat, K.-C. Kwon, J.-S. Jeong, J.-W. Lee, K.-A. Kim, N. Kim, and K.-H. Yoo, “Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array,” Appl. Opt. 52, 8411–8418 (2013).
[CrossRef]

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

G. Li, K.-C. Kwon, G.-H. Shin, J.-S. Jeong, K.-H. Yoo, and N. Kim, “Simplified integral imaging pickup method for real objects using a depth camera,” J. Opt. Soc. Korea 16, 381–385 (2012).
[CrossRef]

K.-C. Kwon, C. Park, M.-U. Erdenebat, J.-S. Jeong, J.-H. Choi, N. Kim, J.-H. Park, Y.-T. Lim, and K.-H. Yoo, “High speed image space parallel processing for computer generated integral imaging system,” Opt. Express 20, 732–740 (2012).
[CrossRef]

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17, 19253–19263 (2009).
[CrossRef]

Kim, S.-C.

S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

D.-C. Hwang, J.-S. Park, S.-C. Kim, D.-H. Shin, and E.-S. Kim, “Magnification of 3D reconstructed images in integral imaging using an intermediate-view reconstruction technique,” Appl. Opt. 45, 4631–4637 (2006).
[CrossRef]

Kishk, S.

Kwon, K.-C.

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

D.-H. Kim, M.-U. Erdenebat, K.-C. Kwon, J.-S. Jeong, J.-W. Lee, K.-A. Kim, N. Kim, and K.-H. Yoo, “Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array,” Appl. Opt. 52, 8411–8418 (2013).
[CrossRef]

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

G. Li, K.-C. Kwon, G.-H. Shin, J.-S. Jeong, K.-H. Yoo, and N. Kim, “Simplified integral imaging pickup method for real objects using a depth camera,” J. Opt. Soc. Korea 16, 381–385 (2012).
[CrossRef]

K.-C. Kwon, C. Park, M.-U. Erdenebat, J.-S. Jeong, J.-H. Choi, N. Kim, J.-H. Park, Y.-T. Lim, and K.-H. Yoo, “High speed image space parallel processing for computer generated integral imaging system,” Opt. Express 20, 732–740 (2012).
[CrossRef]

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17, 19253–19263 (2009).
[CrossRef]

Lee, B.

B. Lee and J. Kim, “Real-time 3D capturing-visualization conversion for light field microscopy,” Proc. SPIE 8769, 876908 (2013).
[CrossRef]

J. Kim, J.-H. Jung, C. Jang, and B. Lee, “Real-time capturing and 3D visualization method based on integral imaging,” Opt. Express 21, 18742–18753 (2013).
[CrossRef]

Lee, B.-H.

Lee, J.-H.

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

Lee, J.-W.

Levoy, M.

M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).
[CrossRef]

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Li, G.

Lim, Y.-T.

Lippmann, G.

G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).

Mcdowall, I.

M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).
[CrossRef]

Ng, R.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Park, C.

Park, G.-C.

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

Park, J.-H.

Park, J.-S.

Phan, A.-H.

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

Piao, M.-L.

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

Piao, Y.

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

Shin, D.-H.

Shin, G.-H.

Yoo, K.-H.

Zhang, Z.

M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).
[CrossRef]

3D Res.

S.-C. Kim, C.-K. Kim, and E.-S. Kim, “Depth-of-focus and resolution-enhanced three-dimensional integral imaging with non-uniform lenslets and intermediate-view reconstruction technique,” 3D Res. 2, 1–9 (2011).

Appl. Opt.

C. R. Acad. Sci.

G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).

Disp. Imag. Technol.

N. Kim, A.-H. Phan, M.-U. Erdenebat, A. M. Alam, K.-C. Kwon, M.-L. Piao, and J.-H. Lee, “3D display technology,” Disp. Imag. Technol. 1, 73–95 (2013).

EURASIP J. Image Video Process.

K.-C. Kwon, Y.-T. Lim, N. Kim, K.-H. Yoo, J.-M. Hong, and G.-C. Park, “High-definition 3D stereoscopic microscope display system for biomedical applications,” EURASIP J. Image Video Process. 2010, 1–8 (2010).

J. Microsc.

M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).
[CrossRef]

J. Opt. Soc. Korea

Opt. Eng.

J.-S. Jeong, K.-C. Kwon, M.-U. Erdenebat, Y. Piao, N. Kim, and K.-H. Yoo, “Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera,” Opt. Eng. 53, 015103 (2014).
[CrossRef]

K.-H. Bae and E.-S. Kim, “New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction,” Opt. Eng. 42, 1778–1786 (2003).
[CrossRef]

Opt. Express

Opt. Lett.

Pattern Recogn.

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” Pattern Recogn. 13, 111–122 (1981).
[CrossRef]

Proc. SPIE

B. Lee and J. Kim, “Real-time 3D capturing-visualization conversion for light field microscopy,” Proc. SPIE 8769, 876908 (2013).
[CrossRef]

Trans. Graph.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” Trans. Graph. 25, 924–934 (2006).

Other

NVIDIA, “OpenCL programming guide for the CUDA architecture,” Version 2.3 (2009).

NVIDIA, “CUDA C programming guide,” Version 3.1.1 (2010).

Supplementary Material (2)

» Media 1: MP4 (1085 KB)     
» Media 2: MP4 (2620 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1.
Fig. 1.

(a) Basic optical structure of the IIM and (b) concept of the orthographic views.

Fig. 2.
Fig. 2.

Schematic diagram of the proposed real-time IIM display system: acquisition, calculation of the IVEIs and orthographic-view images, and display.

Fig. 3.
Fig. 3.

Examples of preprocessing (a) an acquired image before realignment, (b) the realigned EIs by using the Hough transform, and (c) the ROI setting and image extracted from the ROI.

Fig. 4.
Fig. 4.

Example of the IVEI generation for (upper) captured images, and (lower) generated images.

Fig. 5.
Fig. 5.

Schematic diagram to generate IVEIs.

Fig. 6.
Fig. 6.

Schematic diagram for generating and displaying the orthographic-view images. (a) For the previously generated EEIAs, data from every pixel are stored in the GPU memory, and the threads are created for each pixel of EEIA to generate the orthographic-view images. The generated directional-view images for the corresponding detected user view directions are displayed on the display device in real time. (b) The generating and displaying processes of the orthographic-view images are illustrated in detail, inside the GPU computation.

Fig. 7.
Fig. 7.

Experimental setup: (a) hardware setup and (b) image processing software.

Fig. 8.
Fig. 8.

Experiment results: (a) a 3D real object on bonding area of the CMOS sensor chip, (b) the initially acquired EIA via the MLA, (c) the EEIA that included the generated IVEIs for the initially captured EIs, and (d) the reconstructed orthographic-view images. Each image has 119×119 pixels and the number of the images is 25×25.

Fig. 9.
Fig. 9.

Real-time interactive orthographic-view image display for the proposed prototype IIM. (a) For the given object, (b) the initial EIs and directly reconstructed 3D images without resolution enhancement from multiple viewing direction (Media 1), and (c) EEIA with reconstructed images from multiple viewing directions (Media 2). The display refresh rate was 250 Hz, and 95 Hz in the case of (c).

Tables (2)

Tables Icon

Table 1. Specifications for the Proposed Prototype IIM System

Tables Icon

Table 2. Processing Speed Measurements of the Proposed Method and Conventional Method

Equations (2)

Equations on this page are rendered with MathJax. Learn more.

{IVEIh(i,j)=EI(i,j)+EI(i+1,j)2IVEIv(i,j)=EI(i,j)+EI(i,j+1)2IVEId(i,j)=EI(i,j)+EI(i+1,j)+EI(i,j+1)+EI(i+1,j+1)4,
Vx=MPx×EEIw2ELw1;Vy=MPy×EEIh2ELh1,

Metrics