Abstract

In this paper, we propose an occlusion removal method using sub-image block matching for improved recognition of partially occluded 3D objects in computational integral imaging (CII). When 3D plane images are reconstructed in CII, occlusion degrades the resolution of reconstructed images. To overcome this problem, we apply the sub-image transform to elemental image array (EIA) and these sub-images are employed using block matching method for depth estimation. Based on the estimated depth information, we remove the unknown occlusion. After completing the occlusion removal for all sub-images, we obtain the modified EIA without occlusion information through the inverse sub-image transform. Finally, the 3D plane images are reconstructed by using a computational integral imaging reconstruction method with the modified EIA. The proposed method can provide a substantial gain in terms of the visual quality of 3D reconstructed images. To show the usefulness of the proposed method we carry out some experiments and the results are presented.

© 2008 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. Lippmann, "La photographic integrale," C.R. Acad. Sci. 146, 446-451 (1908).
  2. A. Stern and B. Javidi, "Three dimensional image sensing, visualization, and processing using integral imaging," Proc. of IEEE 94, 591-607 (2006).
    [CrossRef]
  3. B. Lee, S.-Y. Jung, S.-W. Min, and J.-H. Park, "Three-dimensional display by use of integral photography with dynamically variable image planes," Opt. Lett. 26, 1481-1482 (2001).
    [CrossRef]
  4. J.-S. Jang and B. Javidi, "Improved viewing resolution of three- dimensional integral imaging by use of nonstationary micro-optics," Opt. Lett. 27, 324-326 (2002).
    [CrossRef]
  5. M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, "Multifacet structure of observed reconstructed integral images," J. Opt. Soc. Am. A 22, 597-603 (2005).
    [CrossRef]
  6. D.-H. Shin, B. Lee, and E.-S. Kim, "Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens," Appl. Opt. 45, 7375-7381 (2006).
    [CrossRef] [PubMed]
  7. D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
    [CrossRef]
  8. H. Arimoto and B. Javidi, "Integral three-dimensional imaging with digital reconstruction," Opt. Lett. 26, 157-159 (2001).
    [CrossRef]
  9. Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002).
    [CrossRef] [PubMed]
  10. S.-H. Hong, J.-S. Jang, and B. Javidi, "Three-dimensional volumetric object reconstruction using computational integral imaging," Opt. Express 12, 483-491 (2004).
    [CrossRef] [PubMed]
  11. D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
    [CrossRef]
  12. S.-H. Hong and B. Javidi, "Improved resolution 3D object reconstruction using computational integral imaging with time multiplexing," Opt. Express 12, 4579-4588 (2004).
    [CrossRef] [PubMed]
  13. S.-H. Hong and B. Javidi, "Distortion-tolerant 3D recognition of occluded objects using computational integral imaging," Opt. Express 14, 12085-12095 (2006).
    [CrossRef] [PubMed]
  14. D.-H. Shin and H. Yoo, "Image quality enhancement in 3D computational integral imaging by use of interpolation methods," Opt. Express 15, 12039-12049 (2007).
    [CrossRef] [PubMed]
  15. H. Yoo and D.-H. Shin "Improved analysis on the signal property of computational integral imaging system," Opt. Express 15, 14107-14114 (2007).
    [CrossRef] [PubMed]
  16. B. Javidi, R. Ponce-Díaz, and S.-H. Hong, "Three-dimensional recognition of occluded objects by using computational integral imaging," Opt. Lett. 31, 1106-1108 (2006).
    [CrossRef] [PubMed]
  17. J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
    [CrossRef]
  18. J.-H. Park, J. Kim, and B. Lee, "Three-dimensional optical correlator using a sub-image array," Opt. Express 13, 5116-5126 (2005).
    [CrossRef] [PubMed]
  19. D.-H. Shin, B. Lee, and E.-S. Kim, "Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model," ETRI J. 28, 521-524 (2006).
    [CrossRef]
  20. M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
    [CrossRef]
  21. J.-S. Lee, J.-H. Ko, and E.-S. Kim, "Real-time stereo object tracking system by using block matching algorithm and optical binary phase extraction joint transform correlator," Opt. Commun. 191, 191-202 (2001).
    [CrossRef]
  22. S.-H. Hong and B. Javidi, "Three-dimensional visualization of partially occluded objects using integral imaging," J. Display Technol. 1, 354-359 (2005).
    [CrossRef]
  23. J. -S. Jang and B. Javidi, "Depth and lateral size control of three-dimensional images in projection integral imaging," Opt. Express 12, 3778-3790 (2004).
    [CrossRef] [PubMed]

2007 (4)

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

D.-H. Shin and H. Yoo, "Image quality enhancement in 3D computational integral imaging by use of interpolation methods," Opt. Express 15, 12039-12049 (2007).
[CrossRef] [PubMed]

H. Yoo and D.-H. Shin "Improved analysis on the signal property of computational integral imaging system," Opt. Express 15, 14107-14114 (2007).
[CrossRef] [PubMed]

2006 (5)

2005 (4)

2004 (3)

2003 (1)

M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
[CrossRef]

2002 (2)

2001 (3)

1908 (1)

G. Lippmann, "La photographic integrale," C.R. Acad. Sci. 146, 446-451 (1908).

Arimoto, H.

Brown, M. Z.

M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
[CrossRef]

Burschka, D.

M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
[CrossRef]

Frauel, Y.

Hager, S. D.

M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
[CrossRef]

Hong, S.-H.

Hwang, D.-C.

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

Jang, J. -S.

Jang, J.-S.

Javidi, B.

S.-H. Hong and B. Javidi, "Distortion-tolerant 3D recognition of occluded objects using computational integral imaging," Opt. Express 14, 12085-12095 (2006).
[CrossRef] [PubMed]

A. Stern and B. Javidi, "Three dimensional image sensing, visualization, and processing using integral imaging," Proc. of IEEE 94, 591-607 (2006).
[CrossRef]

B. Javidi, R. Ponce-Díaz, and S.-H. Hong, "Three-dimensional recognition of occluded objects by using computational integral imaging," Opt. Lett. 31, 1106-1108 (2006).
[CrossRef] [PubMed]

M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, "Multifacet structure of observed reconstructed integral images," J. Opt. Soc. Am. A 22, 597-603 (2005).
[CrossRef]

S.-H. Hong and B. Javidi, "Three-dimensional visualization of partially occluded objects using integral imaging," J. Display Technol. 1, 354-359 (2005).
[CrossRef]

J. -S. Jang and B. Javidi, "Depth and lateral size control of three-dimensional images in projection integral imaging," Opt. Express 12, 3778-3790 (2004).
[CrossRef] [PubMed]

S.-H. Hong and B. Javidi, "Improved resolution 3D object reconstruction using computational integral imaging with time multiplexing," Opt. Express 12, 4579-4588 (2004).
[CrossRef] [PubMed]

S.-H. Hong, J.-S. Jang, and B. Javidi, "Three-dimensional volumetric object reconstruction using computational integral imaging," Opt. Express 12, 483-491 (2004).
[CrossRef] [PubMed]

J.-S. Jang and B. Javidi, "Improved viewing resolution of three- dimensional integral imaging by use of nonstationary micro-optics," Opt. Lett. 27, 324-326 (2002).
[CrossRef]

Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002).
[CrossRef] [PubMed]

H. Arimoto and B. Javidi, "Integral three-dimensional imaging with digital reconstruction," Opt. Lett. 26, 157-159 (2001).
[CrossRef]

Jung, S.-Y.

Kim, E.-S.

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

D.-H. Shin, B. Lee, and E.-S. Kim, "Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model," ETRI J. 28, 521-524 (2006).
[CrossRef]

D.-H. Shin, B. Lee, and E.-S. Kim, "Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens," Appl. Opt. 45, 7375-7381 (2006).
[CrossRef] [PubMed]

D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

J.-S. Lee, J.-H. Ko, and E.-S. Kim, "Real-time stereo object tracking system by using block matching algorithm and optical binary phase extraction joint transform correlator," Opt. Commun. 191, 191-202 (2001).
[CrossRef]

Kim, J.

Ko, J.-H.

J.-S. Lee, J.-H. Ko, and E.-S. Kim, "Real-time stereo object tracking system by using block matching algorithm and optical binary phase extraction joint transform correlator," Opt. Commun. 191, 191-202 (2001).
[CrossRef]

Lee, B.

D.-H. Shin, B. Lee, and E.-S. Kim, "Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens," Appl. Opt. 45, 7375-7381 (2006).
[CrossRef] [PubMed]

D.-H. Shin, B. Lee, and E.-S. Kim, "Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model," ETRI J. 28, 521-524 (2006).
[CrossRef]

D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

J.-H. Park, J. Kim, and B. Lee, "Three-dimensional optical correlator using a sub-image array," Opt. Express 13, 5116-5126 (2005).
[CrossRef] [PubMed]

B. Lee, S.-Y. Jung, S.-W. Min, and J.-H. Park, "Three-dimensional display by use of integral photography with dynamically variable image planes," Opt. Lett. 26, 1481-1482 (2001).
[CrossRef]

Lee, J.-S.

J.-S. Lee, J.-H. Ko, and E.-S. Kim, "Real-time stereo object tracking system by using block matching algorithm and optical binary phase extraction joint transform correlator," Opt. Commun. 191, 191-202 (2001).
[CrossRef]

Lee, S.-H.

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

Lippmann, G.

G. Lippmann, "La photographic integrale," C.R. Acad. Sci. 146, 446-451 (1908).

Martínez-Corral, M.

Martínez-Cuenca, R.

Min, S.-W.

Park, J.-H.

Park, J.-S.

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

Ponce-Díaz, R.

Saavedra, G.

Shin, D.-H.

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

H. Yoo and D.-H. Shin "Improved analysis on the signal property of computational integral imaging system," Opt. Express 15, 14107-14114 (2007).
[CrossRef] [PubMed]

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

D.-H. Shin and H. Yoo, "Image quality enhancement in 3D computational integral imaging by use of interpolation methods," Opt. Express 15, 12039-12049 (2007).
[CrossRef] [PubMed]

D.-H. Shin, B. Lee, and E.-S. Kim, "Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens," Appl. Opt. 45, 7375-7381 (2006).
[CrossRef] [PubMed]

D.-H. Shin, B. Lee, and E.-S. Kim, "Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model," ETRI J. 28, 521-524 (2006).
[CrossRef]

D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

Stern, A.

A. Stern and B. Javidi, "Three dimensional image sensing, visualization, and processing using integral imaging," Proc. of IEEE 94, 591-607 (2006).
[CrossRef]

Yoo, H.

Appl. Opt. (2)

C.R. Acad. Sci. (1)

G. Lippmann, "La photographic integrale," C.R. Acad. Sci. 146, 446-451 (1908).

ETRI J. (1)

D.-H. Shin, B. Lee, and E.-S. Kim, "Improved viewing quality of 3-D images in computational integral imaging reconstruction based on lenslet array model," ETRI J. 28, 521-524 (2006).
[CrossRef]

IEEE Trans. Pattern Analysis and Machine Intelligence (1)

M. Z. Brown, D. Burschka, and S. D. Hager, "Advances in computational stereo," IEEE Trans. Pattern Analysis and Machine Intelligence 25, 993-1008 (2003).
[CrossRef]

J. Display Technol. (1)

J. Opt. Soc. Am. A (1)

Jpn. J. Appl. Phys. (1)

D.-H. Shin, E.-S. Kim, and B. Lee, "Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

Opt. Commun. (3)

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of ture 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

J.-S. Lee, J.-H. Ko, and E.-S. Kim, "Real-time stereo object tracking system by using block matching algorithm and optical binary phase extraction joint transform correlator," Opt. Commun. 191, 191-202 (2001).
[CrossRef]

J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, "Resolution-enhanced 3D image correlator using computationally reconstructed integral images," Opt. Commun. 276, 72-79 (2007).
[CrossRef]

Opt. Express (7)

Opt. Lett. (4)

Proc. of IEEE (1)

A. Stern and B. Javidi, "Three dimensional image sensing, visualization, and processing using integral imaging," Proc. of IEEE 94, 591-607 (2006).
[CrossRef]

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1.
Fig. 1.

Concept of integral imaging (a) Pickup (b) Computational reconstruction.

Fig. 2.
Fig. 2.

Principle of CII (a) Generation of template (b) Recognition of partially occluded object.

Fig. 3.
Fig. 3.

Conceptual diagram for proposed occlusion removal method

Fig. 4.
Fig. 4.

Experimental setup for pickup process

Fig. 5.
Fig. 5.

(a) EIA (b) Sub-image array based on multiple pixel extraction (c) Sub-image array after occlusion removal (d) The modified EIA

Fig. 6.
Fig. 6.

ES transform based on multiple pixel extraction (a) EIA (b) Sub-image array.

Fig. 7.
Fig. 7.

Ray analysis for ES transform

Fig. 8.
Fig. 8.

Occlusion removal process in the sub-image array

Fig. 9.
Fig. 9.

Reconstructed images by using VCR process (a) Conventional method (b) Proposed method.

Fig. 10.
Fig. 10.

Reconstructed images for three kinds of test images (a) Cow (b) Toy (c) Car

Tables (1)

Tables Icon

Table 1. PSNR results for three test images

Equations (6)

Equations on this page are rendered with MathJax. Learn more.

S ( i , j ) = E ( t x s x + q x m + r x , t y s y + q y m + r y )
m max = ( m max , x = g z n x = n x M , m max , y = g z n y = n y M )
SAD ( x , y ) ( u , v ) = i = 1 B j = 1 B I L ( x + i , y + j ) I R ( x + u + i , y + v + j ) .
( u , v ) = arg min SAD ( x , y ) ( u , v ) .
PSNR ( O , R ) = 10 log 10 ( 255 2 MSE ( O , R ) )
MSE = 1 PQ x = 0 P 1 y = 0 Q 1 [ O ( x , y ) R ( x , y ) ] 2

Metrics