Abstract

A method for the reconstruction of 3D shape and texture from integral photography (IP) images is presented. Sharing the same principles with stereoscopic-based object reconstruction, it offers increased robustness to noise and occlusions due to the unique characteristics of IP images. A coarse-to-fine approach is used, employing what we believe to be a novel grid refinement step in order to increase the quality of the reconstructed objects. The proposed method's properties include configurable depth accuracy and direct and seamless triangulation. We evaluate our method using synthetic data from a computer-simulated IP setup as well as real data from a simple yet effective digital IP setup. Experiments show reconstructed objects of high-quality indicating that IP can be a competitive modality for 3D object reconstruction.

© 2007 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. Lippmann, "La photographie integrale," C. R. Acad. Sci. 146, 446-455 (1908).
  2. J. Son and B. Javidi, "Three-dimensional imaging methods based on multiview images," in IEEE/OSA Display Technology (IEEE/OSA, 2005).
    [CrossRef]
  3. S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
    [CrossRef] [PubMed]
  4. J. Jang and B. Javidi, "Two-step integral imaging for orthoscopic three-dimensional imaging with improved viewing resolution," Opt. Eng. 41, 2568-2571 (2002).
    [CrossRef]
  5. J. Jang and B. Javidi, "Formation of orthoscopic three dimensional real images in direct pickup one-step integral imaging," Opt. Eng. 42, 1869-1870 (2003).
    [CrossRef]
  6. M. Corral, B. Javidi, R. Cuenca, and G. Saavedra, "Formation of real orthoscopic integral images by smart pixel mapping," Opt. Express 13, 9175-9180 (2005).
    [CrossRef]
  7. H. Kim, "A 3D modelling system using multiple stereo cameras," Ph.D dissertation (Yonsei University, 2005).
  8. J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).
  9. S. Kishk and B. Javidi, "Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging," Opt. Express 11, 3528-3541 (2003).
    [CrossRef] [PubMed]
  10. Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002).
    [CrossRef] [PubMed]
  11. J. Park, Y. Kim, J. Kim, S. Min, and B. Lee, "Three-dimensional display scheme based on integral imaging with three-dimensional information processing," Opt. Express 12, 6020-6032 (2004).
    [CrossRef] [PubMed]
  12. J. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
    [CrossRef] [PubMed]
  13. D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
    [CrossRef]
  14. J. Park, S. Min, S. Jung, and B. Lee, "New stereovision scheme using a camera and a lens array," Proc. SPIE 4471, 73-80 (2001).
  15. O. Faugeras, Three-Dimensial Computer Vision: A Geometric Viewpoint (MIT Press, 1993).
  16. N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
    [CrossRef] [PubMed]
  17. J. Son, V. Bobrinev, and K. Kim, "Depth resolution and displayable depth of a scene in three-dimensional images," J. Opt. Soc. Am. A 22, 1739-1745 (2005).
    [CrossRef]
  18. S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
    [CrossRef]

2006 (2)

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

2005 (5)

J. Son, V. Bobrinev, and K. Kim, "Depth resolution and displayable depth of a scene in three-dimensional images," J. Opt. Soc. Am. A 22, 1739-1745 (2005).
[CrossRef]

D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

J. Son and B. Javidi, "Three-dimensional imaging methods based on multiview images," in IEEE/OSA Display Technology (IEEE/OSA, 2005).
[CrossRef]

M. Corral, B. Javidi, R. Cuenca, and G. Saavedra, "Formation of real orthoscopic integral images by smart pixel mapping," Opt. Express 13, 9175-9180 (2005).
[CrossRef]

H. Kim, "A 3D modelling system using multiple stereo cameras," Ph.D dissertation (Yonsei University, 2005).

2004 (2)

2003 (3)

J. Jang and B. Javidi, "Formation of orthoscopic three dimensional real images in direct pickup one-step integral imaging," Opt. Eng. 42, 1869-1870 (2003).
[CrossRef]

J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).

S. Kishk and B. Javidi, "Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging," Opt. Express 11, 3528-3541 (2003).
[CrossRef] [PubMed]

2002 (2)

Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002).
[CrossRef] [PubMed]

J. Jang and B. Javidi, "Two-step integral imaging for orthoscopic three-dimensional imaging with improved viewing resolution," Opt. Eng. 41, 2568-2571 (2002).
[CrossRef]

2001 (2)

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

J. Park, S. Min, S. Jung, and B. Lee, "New stereovision scheme using a camera and a lens array," Proc. SPIE 4471, 73-80 (2001).

1993 (1)

O. Faugeras, Three-Dimensial Computer Vision: A Geometric Viewpoint (MIT Press, 1993).

1908 (1)

G. Lippmann, "La photographie integrale," C. R. Acad. Sci. 146, 446-455 (1908).

Athineos, S.

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Bobrinev, V.

Choi, H.

J. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
[CrossRef] [PubMed]

J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).

Corral, M.

Cuenca, R.

Dohi, T.

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

Faugeras, O.

O. Faugeras, Three-Dimensial Computer Vision: A Geometric Viewpoint (MIT Press, 1993).

Frauel, Y.

Jang, J.

J. Jang and B. Javidi, "Formation of orthoscopic three dimensional real images in direct pickup one-step integral imaging," Opt. Eng. 42, 1869-1870 (2003).
[CrossRef]

J. Jang and B. Javidi, "Two-step integral imaging for orthoscopic three-dimensional imaging with improved viewing resolution," Opt. Eng. 41, 2568-2571 (2002).
[CrossRef]

Javidi, B.

J. Son and B. Javidi, "Three-dimensional imaging methods based on multiview images," in IEEE/OSA Display Technology (IEEE/OSA, 2005).
[CrossRef]

M. Corral, B. Javidi, R. Cuenca, and G. Saavedra, "Formation of real orthoscopic integral images by smart pixel mapping," Opt. Express 13, 9175-9180 (2005).
[CrossRef]

J. Jang and B. Javidi, "Formation of orthoscopic three dimensional real images in direct pickup one-step integral imaging," Opt. Eng. 42, 1869-1870 (2003).
[CrossRef]

S. Kishk and B. Javidi, "Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging," Opt. Express 11, 3528-3541 (2003).
[CrossRef] [PubMed]

Y. Frauel and B. Javidi, "Digital three-dimensional image correlation by use of computer-reconstructed integral imaging," Appl. Opt. 41, 5488-5496 (2002).
[CrossRef] [PubMed]

J. Jang and B. Javidi, "Two-step integral imaging for orthoscopic three-dimensional imaging with improved viewing resolution," Opt. Eng. 41, 2568-2571 (2002).
[CrossRef]

Jung, S.

Kim, E.

D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

Kim, H.

H. Kim, "A 3D modelling system using multiple stereo cameras," Ph.D dissertation (Yonsei University, 2005).

Kim, J.

Kim, K.

Kim, Y.

Kishk, S.

Lee, B.

D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

J. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
[CrossRef] [PubMed]

J. Park, Y. Kim, J. Kim, S. Min, and B. Lee, "Three-dimensional display scheme based on integral imaging with three-dimensional information processing," Opt. Express 12, 6020-6032 (2004).
[CrossRef] [PubMed]

J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).

J. Park, S. Min, S. Jung, and B. Lee, "New stereovision scheme using a camera and a lens array," Proc. SPIE 4471, 73-80 (2001).

Lippmann, G.

G. Lippmann, "La photographie integrale," C. R. Acad. Sci. 146, 446-455 (1908).

Maroulis, D.

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Masamune, K.

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

Min, S.

Nakajima, S.

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

Nakamura, K.

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

Papageorgas, P.

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Park, J.

J. Park, Y. Kim, J. Kim, S. Min, and B. Lee, "Three-dimensional display scheme based on integral imaging with three-dimensional information processing," Opt. Express 12, 6020-6032 (2004).
[CrossRef] [PubMed]

J. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
[CrossRef] [PubMed]

J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).

J. Park, S. Min, S. Jung, and B. Lee, "New stereovision scheme using a camera and a lens array," Proc. SPIE 4471, 73-80 (2001).

Saavedra, G.

Sakuma, I.

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

Sangriotis, M.

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Sgouros, N.

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Shin, D.

D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

Son, J.

J. Son and B. Javidi, "Three-dimensional imaging methods based on multiview images," in IEEE/OSA Display Technology (IEEE/OSA, 2005).
[CrossRef]

J. Son, V. Bobrinev, and K. Kim, "Depth resolution and displayable depth of a scene in three-dimensional images," J. Opt. Soc. Am. A 22, 1739-1745 (2005).
[CrossRef]

Theofanous, N.

N. Sgouros, S. Athineos, M. Sangriotis, P. Papageorgas, and N. Theofanous, "Accurate lattice extraction in integral images," Opt. Express 14, 10403-10409 (2006).
[CrossRef] [PubMed]

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

Appl. Opt. (2)

C. R. Acad. Sci. (1)

G. Lippmann, "La photographie integrale," C. R. Acad. Sci. 146, 446-455 (1908).

Comput. Med. Imaging Graph. (1)

S. Nakajima, K. Nakamura, K. Masamune, I. Sakuma, and T. Dohi, "Three-dimensional medical imaging display with computer-generated integral photography," Comput. Med. Imaging Graph. 25, 235-241 (2001).
[CrossRef] [PubMed]

J. Electron. Imaging (1)

S. Athineos, N. Sgouros, P. Papageorgas, D. Maroulis, M. Sangriotis, and N. Theofanous, "Photorealistic integral photography using a ray traced model of capturing optics," J. Electron. Imaging 15, 043007-043014 (2006).
[CrossRef]

J. Opt. Soc. Am. A (1)

Jpn. J. Appl. Phys. (1)

D. Shin, E. Kim, and B. Lee, "Computational reconstruction of three-dimensional objects in integral imaging using lenslet array," Jpn. J. Appl. Phys. 44, 8016-8018 (2005).
[CrossRef]

Opt. Eng. (2)

J. Jang and B. Javidi, "Two-step integral imaging for orthoscopic three-dimensional imaging with improved viewing resolution," Opt. Eng. 41, 2568-2571 (2002).
[CrossRef]

J. Jang and B. Javidi, "Formation of orthoscopic three dimensional real images in direct pickup one-step integral imaging," Opt. Eng. 42, 1869-1870 (2003).
[CrossRef]

Opt. Express (4)

Other (5)

J. Park, S. Min, S. Jung, and B. Lee, "New stereovision scheme using a camera and a lens array," Proc. SPIE 4471, 73-80 (2001).

O. Faugeras, Three-Dimensial Computer Vision: A Geometric Viewpoint (MIT Press, 1993).

J. Son and B. Javidi, "Three-dimensional imaging methods based on multiview images," in IEEE/OSA Display Technology (IEEE/OSA, 2005).
[CrossRef]

H. Kim, "A 3D modelling system using multiple stereo cameras," Ph.D dissertation (Yonsei University, 2005).

J. Park, H. Choi, and B. Lee, "Comparative study of integral imaging and multi-camera pickup for acquisition of three-dimensional information," Proc. SPIE 5202, 158-167 (2003).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1

Operating principles: (a) acquisition and (b) reproduction of IP images.

Fig. 2
Fig. 2

(Color online) Reconstruction of a die: (a) Input IP image ( 52 × 52 pixels∕lens, f = 3.3   mm ) and (b), (c) reconstructed 3D object rendered with triangulation superimposed.

Fig. 3
Fig. 3

(Color online) Hardware setup of the proposed approach.

Fig. 4
Fig. 4

Point P is projected through two lenses on their image planes at p 1 and p 2 . The distance of the optical center from the image plane is equal to f.

Fig. 5
Fig. 5

(Color online) IP image of a die using 124 × 124 pixels per elemental image, f = 3.3   mm and N = 3 : A pixel (marked yellow) and its correspondences (marked red). The colored boxes denote the window area ( W = 8 ) .

Fig. 6
Fig. 6

(Color online) Elemental images from a real-life object with 98 × 98 pixels per lens and f = 3.3   mm : (a) 3 × 3 neighborhood with central pixels marked yellow. The correspondences of the central pixels in the bottom-left elemental image are marked red. (b) Triangulation of the same neighborhood using only central pixels, projected onto R 2 and superimposed over the IP image.

Fig. 7
Fig. 7

(Color online) Computational requirements of reconstruction: (a) varying W with S = 1 , N = 3 ; (b) varying N with S = 1 , W = 4 ; and (c) varying S with N = 1 , W = 1 .

Fig. 8
Fig. 8

(Color online) Accuracy distribution for variable focal length f (from 3.2 to 11.2), using R = 32 × 32 . Each plot shows graphs utilizing from N = 1 to 8 neighbors (color coded from bright red to black, respectively).

Fig. 9
Fig. 9

(Color online) Synthetic fish used in the virtual lens array experiments.

Fig. 10
Fig. 10

(Color online) Reconstruction of a synthetic fish (Fig. 9) for variable focal lengths (3, 6, and 9   mm from top to bottom) using 64 × 64 pixels per lens, 64 × 64 lenses (elemental images), and 1   mm pitch: (a)–(c) Input IP images and (d)–(f) corresponding renderings of reconstructed 3D object with the surface triangulation superimposed.

Fig. 11
Fig. 11

(Color online) Shape and texture reconstruction of a real-life object: (a) Input IP image using 98 × 98 pixels per lens and f = 3.3   mm ; (b), (c) renderings of reconstructed 3D object with the surface triangulation superimposed (using S = 1 ); (d)–(f) renderings using more subdivision steps (S = 2, 3, and 4, respectively).

Tables (1)

Tables Icon

Table 1 Lens Array Specifications Used in the Acquisition Setup

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

[ U V H ] = [ f 0 0 0 0 f 0 0 0 0 1 0 ] [ x y z 1 ] ,
M = [ 1 0 0 p i t c h 0 1 0 0 0 0 1 0 0 0 0 1 ] .
u 2 = f x + p i t c h z ,
D ( p 1 , p 2 ) = j = W W i = W W | E 1 ( u 1 + i , v 1 + j ) E 2 ( u 2 + i , v 2 + j ) | ,
D t o t a l = j = N N i = N N D ( p k , l , p k + i , l + j ) ,

Metrics