Abstract

We propose a novel approach to optically refocus three-dimensional (3-D) objects on their real depth from the captured elemental image array (EIA) by using a sifting property of the periodic δ-function array (PDFA) in integral-imaging. By convolving the PDFAs whose spatial periods correspond to each object’s depth with the sub-image array (SIA) transformed from the EIA, a set of spatially filtered-SIAs (SF-SIAs) for each object’s depth can be extracted. These SF-SIAs are then inverse-transformed into the corresponding versions of the EIAs, and from these, 3-D objects with their own perspectives can be reconstructed to be refocused on their depth in the space. The feasibility of the proposed method has been confirmed through optical experiments as well as ray-optical analysis.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).
  2. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light Field Photography with a Hand-Held Plenoptic Camera,” Technical Report CTSR 2005–02, Dept. of Computer Science, Stanford Univ., 2005.
  3. M. Levoy, P. Hanrahan, “Light Field Rendering,” Proc. ACM SIGGRAPH, (1996), pp. 31–42.
  4. R. Ng, “Digital light field photography,” Ph.D. dissertation (Stanford University, Stanford, CA, USA, 2006).
  5. T. Bishop, S. Zanetti, P. Favaro, “Light Field Superresolution,” Proc. International Conference on Computational Photography, (2009), pp. 1–9.
  6. A. Lumsdaine, T. Georgiev, “The focused plenoptic camera,” Proc. International Conference on Computational Photography, (2009), pp. 1–8.
  7. R. Raskar and A.-K. Agrawal, “4D light field cameras,” US patent 772423 (September 2010).
  8. S.-H. Hong, J.-S. Jang, B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004).
    [CrossRef] [PubMed]
  9. D.-H. Shin, B. Lee, E.-S. Kim, “Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model,” ETRI Journal 28(4), 521–524 (2006).
    [CrossRef]
  10. J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
    [CrossRef]
  11. H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
    [CrossRef]
  12. A. Stern, B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
    [CrossRef]
  13. F. Okano, H. Hoshino, J. Arai, I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997).
    [CrossRef] [PubMed]
  14. J.-S. Jang, B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27(5), 324–326 (2002).
    [CrossRef] [PubMed]
  15. Y. Kim, K. Hong, and B. Lee, “Recent researches based on integral imaging display method,” 3D Research, vol. 1, 17–27 (2010).
  16. B. Lee, S. Jung, J.-H. Park, “Viewing-angle-enhanced integral imaging by lens switching,” Opt. Lett. 27(10), 818–820 (2002).
    [CrossRef] [PubMed]
  17. M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43(31), 5806–5813 (2004).
    [CrossRef] [PubMed]
  18. J.-H. Park, J. Kim, Y. Kim, B. Lee, “Resolution-enhanced three-dimension / two-dimension convertible display based on integral imaging,” Opt. Express 13(6), 1875–1884 (2005).
    [CrossRef] [PubMed]
  19. D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
    [CrossRef]
  20. B. Javidi, R. Ponce-Díaz, S.-H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. 31(8), 1106–1108 (2006).
    [CrossRef] [PubMed]
  21. J.-H. Park, J. Kim, B. Lee, “Three-dimensional optical correlator using a sub-image array,” Opt. Express 13(13), 5116–5126 (2005).
    [CrossRef] [PubMed]
  22. J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
    [CrossRef]
  23. M. Zhang, Y. Piao, E.-S. Kim, “Occlusion-removed scheme using depth-reversed method in computational integral imaging,” Appl. Opt. 49(14), 2571–2580 (2010).
    [CrossRef]
  24. D. Shin, B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett. 37(9), 1394–1396 (2012).
    [CrossRef] [PubMed]
  25. B.-G. Lee, H.-H. Kang, E.-S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research, 1:2 (2010).
  26. J.-Y. Jang, J.-I. Ser, S. Cha, S.-H. Shin, “Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging,” Appl. Opt. 51(16), 3279–3286 (2012).
    [CrossRef] [PubMed]
  27. J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
    [CrossRef]

2014 (1)

J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
[CrossRef]

2012 (2)

2010 (1)

2008 (1)

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

2007 (2)

D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

2006 (5)

A. Stern, B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[CrossRef]

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

D.-H. Shin, B. Lee, E.-S. Kim, “Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model,” ETRI Journal 28(4), 521–524 (2006).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

B. Javidi, R. Ponce-Díaz, S.-H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. 31(8), 1106–1108 (2006).
[CrossRef] [PubMed]

2005 (2)

2004 (2)

2002 (2)

1997 (1)

Adams, A.

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

Arai, J.

Bishop, T.

T. Bishop, S. Zanetti, P. Favaro, “Light Field Superresolution,” Proc. International Conference on Computational Photography, (2009), pp. 1–9.

Cha, S.

Favaro, P.

T. Bishop, S. Zanetti, P. Favaro, “Light Field Superresolution,” Proc. International Conference on Computational Photography, (2009), pp. 1–9.

Footer, M.

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

Georgiev, T.

A. Lumsdaine, T. Georgiev, “The focused plenoptic camera,” Proc. International Conference on Computational Photography, (2009), pp. 1–8.

Hanrahan, P.

M. Levoy, P. Hanrahan, “Light Field Rendering,” Proc. ACM SIGGRAPH, (1996), pp. 31–42.

Hong, S.-H.

Horowitz, M.

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

Hoshino, H.

Hwang, D.-C.

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

Jang, J.-S.

Jang, J.-Y.

J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
[CrossRef]

J.-Y. Jang, J.-I. Ser, S. Cha, S.-H. Shin, “Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging,” Appl. Opt. 51(16), 3279–3286 (2012).
[CrossRef] [PubMed]

Javidi, B.

Jung, S.

Kim, E.-S.

J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
[CrossRef]

M. Zhang, Y. Piao, E.-S. Kim, “Occlusion-removed scheme using depth-reversed method in computational integral imaging,” Appl. Opt. 49(14), 2571–2580 (2010).
[CrossRef]

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

D.-H. Shin, B. Lee, E.-S. Kim, “Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model,” ETRI Journal 28(4), 521–524 (2006).
[CrossRef]

Kim, J.

Kim, Y.

Lee, B.

Lee, H.-J.

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

Lee, J.-J.

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

Lee, S.-H.

D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
[CrossRef]

Levoy, M.

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

M. Levoy, P. Hanrahan, “Light Field Rendering,” Proc. ACM SIGGRAPH, (1996), pp. 31–42.

Lumsdaine, A.

A. Lumsdaine, T. Georgiev, “The focused plenoptic camera,” Proc. International Conference on Computational Photography, (2009), pp. 1–8.

Martínez-Corral, M.

Martínez-Cuenca, R.

Ng, R.

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

Okano, F.

Park, J.-H.

Park, J.-S.

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

Piao, Y.

Ponce-Díaz, R.

Saavedra, G.

Ser, J.-I.

Shin, D.

J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
[CrossRef]

D. Shin, B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett. 37(9), 1394–1396 (2012).
[CrossRef] [PubMed]

Shin, D.-H.

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

D.-H. Shin, B. Lee, E.-S. Kim, “Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model,” ETRI Journal 28(4), 521–524 (2006).
[CrossRef]

Shin, S.-H.

Stern, A.

A. Stern, B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[CrossRef]

Yoo, H.

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

Yuyama, I.

Zanetti, S.

T. Bishop, S. Zanetti, P. Favaro, “Light Field Superresolution,” Proc. International Conference on Computational Photography, (2009), pp. 1–9.

Zhang, M.

ACM Trans. Graph. (1)

M. Levoy, R. Ng, A. Adams, M. Footer, M. Horowitz, “Light Field Microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006).

Appl. Opt. (4)

ETRI Journal (1)

D.-H. Shin, B. Lee, E.-S. Kim, “Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model,” ETRI Journal 28(4), 521–524 (2006).
[CrossRef]

Opt. Commun. (3)

H.-J. Lee, D.-H. Shin, H. Yoo, J.-J. Lee, E.-S. Kim, “Computational integral imaging reconstruction scheme of far 3D objects by additional use of an imaging lens,” Opt. Commun. 281(8), 2026–2032 (2008).
[CrossRef]

D.-H. Shin, S.-H. Lee, E.-S. Kim, “Optical display of ture 3D objects in depth-priority integral imaging using an active sensor,” Opt. Commun. 275(2), 330–334 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Resolution-enhanced 3D image correlator using computationally reconstructed integral images,” Opt. Commun. 276(1), 72–79 (2007).
[CrossRef]

Opt. Eng. (1)

J.-S. Park, D.-C. Hwang, D.-H. Shin, E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006).
[CrossRef]

Opt. Express (3)

Opt. Lasers Eng. (1)

J.-Y. Jang, D. Shin, E.-S. Kim, “Improved 3-D image reconstruction using the convolution property of periodic functions in curved integral-imaging,” Opt. Lasers Eng. 54, 14–20 (2014).
[CrossRef]

Opt. Lett. (4)

Proc. IEEE (1)

A. Stern, B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[CrossRef]

Other (8)

Y. Kim, K. Hong, and B. Lee, “Recent researches based on integral imaging display method,” 3D Research, vol. 1, 17–27 (2010).

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light Field Photography with a Hand-Held Plenoptic Camera,” Technical Report CTSR 2005–02, Dept. of Computer Science, Stanford Univ., 2005.

M. Levoy, P. Hanrahan, “Light Field Rendering,” Proc. ACM SIGGRAPH, (1996), pp. 31–42.

R. Ng, “Digital light field photography,” Ph.D. dissertation (Stanford University, Stanford, CA, USA, 2006).

T. Bishop, S. Zanetti, P. Favaro, “Light Field Superresolution,” Proc. International Conference on Computational Photography, (2009), pp. 1–9.

A. Lumsdaine, T. Georgiev, “The focused plenoptic camera,” Proc. International Conference on Computational Photography, (2009), pp. 1–8.

R. Raskar and A.-K. Agrawal, “4D light field cameras,” US patent 772423 (September 2010).

B.-G. Lee, H.-H. Kang, E.-S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research, 1:2 (2010).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (14)

Fig. 1
Fig. 1

Block-diagram of the proposed system: (a) Capturing process, (b) EIA-to-SIA transformation and sifting processes, (c) Optical 3-D refocusing process.

Fig. 2
Fig. 2

Geometrical relation between a point object and its corresponding imaging points in the lens array-based integral-imaging system.

Fig. 3
Fig. 3

Effective-capture-zone (ECZ), two kinds of EIAs captured from 2 and 3 objects locating within the ECZ and their SF-EIAs obtained with the PDFAs: (a) ECZ, (b) Captured EIAs, (c) SF-EIAs.

Fig. 4
Fig. 4

Two kinds of EIAs captured from 2 and 3 objects locating outside the ECZ and their SF-EIAs obtained with the PDFAs: (a) Captured EIAs, (b) SF-EIAs.

Fig. 5
Fig. 5

Conceptual block-diagram of an EIA-to-SIA transformation process.

Fig. 6
Fig. 6

Two kinds of EIAs and transformed SIAs for two or three object cases of Fig. 4(a): (a) Captured EIAs, (b) Transformed SIAs.

Fig. 7
Fig. 7

Pixel correspondence between the EIA and its transformed SIA.

Fig. 8
Fig. 8

Three kinds of SF-SIAs extracted the transformed SIA with the PDFAs having the spatial periods corresponding to each depth of ‘K’, ‘W’ and ‘U’.

Fig. 9
Fig. 9

SF-EIAs inverse-transformed from the SF-SIAs of Fig. 8 and reconstructed object images to be refocused on their depth: (a) SF-EIAs, (b) Reconstructed object images.

Fig. 10
Fig. 10

Two 3-D volumetric objects of ‘Car’ and ‘Plant, and the EIA captured from them: (a) Test 3-D objects, (b) Captured EIA.

Fig. 11
Fig. 11

Optically reconstructed 3-D objects of ‘Car’ and ‘Plant’ from the whole captured EIA and observed object images at three view-points of the left, center and right along the horizontal direction.

Fig. 12
Fig. 12

Three kinds of SF-EIAs obtained from the captured EIA of Fig. 10(b) on each depth of + 35, −40 and −60mm, and their optically reconstructed 3-D objects: (a) SF-EIAs, (b) Optically reconstructed 3-D objects.

Fig. 13
Fig. 13

SIA transformed from the captured EIA and the SF-SIAs obtained from the SIA with the PDFAs: (a) Sampled 1x3 sub-images of the transformed SIA, (b) Sampled 6 subsets of 1x3 SF-SIAs, (c) 6 SF-EIAs inverse-transformed from the corresponding SF-SIAs.

Fig. 14
Fig. 14

Optically reconstructed 3-D object images to be refocused on their depth.

Equations (13)

Equations on this page are rendered with MathJax. Learn more.

x Ek = x O + z O z O +f [ ( k 1 2 )P x O ].
z E = z O f ( z O +f ) .
x CEk =ceil[ x Ek × N P k max ],
X z O =| x CEi x CE( i1 ) |.
g( x E ) | z O =f( x E ) | z O k=1 k max δ( x E x Ek | z O ) ,
{ x O P 2 f z O + ( k max 1 2 ) P x O P 2 f z O + P 2 z O < 0
S( N ny , N nx )= E ¯ ( p y r y + q y t y , p x r y + q x t x )
m = x C E k ( k 1 ) n ,
( k , m ) c = 1 ( j , η ) = ( m , k ) .
s j ( η ) = ( j 1 ) ξ + η ,
s( x CEk )=( x CEk ( k1 )n1 ) k max +k.
X sub =| s( x CEi )s( x CE( i1 ) ) |=| ( x CEi x CE( i1 ) n ) k max +1 |,
g sub ( s( x CEk ) ) | z O =f( x CEk ) | z O j=0 j max 1 δ[ s( x CEk )( s 1 ( η )+j X sub | z O ) ] .

Metrics