Abstract

A novel technique generating arbitrary view images in perspective and orthographic geometry based on integral imaging is proposed. After capturing three-dimensional object using a lens array, disparity estimation is performed for the pixels at the selected position of each elemental image. According to the estimated disparity, appropriate parts of elemental images are mapped to synthesize new view images in perspective or orthographic geometry. As a result, the proposed method is capable of generating new view images at arbitrary positions with high resolution and wide field of view.

© 2008 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. W. J. Matusik and H. Pfister, "3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes," ACM Trans. Graphics 23, 814-824 (2004).
    [CrossRef]
  2. H. Nakanuma, H. Kamei, and Y. Takaki, "Natural 3D display with 128 directional images used for human-engineering evaluation," in Stereoscopic Displays and Virtual Reality Systems XII, A. J. Woods, M. T. Bolas, J. O. Merritt, and I. E. McDowall, eds., Proc. SPIE 5664, 28-35 (2005).
    [CrossRef]
  3. J. Rosen, "Three-dimensional joint transform correlator," Appl. Opt. 37, 7438-7544 (1998).
    [CrossRef]
  4. J.-H. Park, J. Kim, and B. Lee, "Three-dimensional optical correlator using a sub-image array," Opt. Express 13, 5116-5126 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-13-5116.
    [CrossRef] [PubMed]
  5. D. Abookasis and J. Rosen, "Computer-generated holograms of three-dimensional objects synthesized from their multiple angular viewpoints," J. Opt. Soc. Am. A 20, 1537-1545 (2003).
    [CrossRef]
  6. L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
    [CrossRef]
  7. B. Lee, S. Jung, and J.-H. Park, "Viewing-angle-enhanced integral imaging by lens switching," Opt. Lett. 27, 818-820 (2002).
    [CrossRef]
  8. M. Okui, J. Arai, Y. Nojiri, and F. Okano, "Optical screen for direct projection of integral imaging," Appl. Opt. 45, 9132-9139 (2006).
    [CrossRef] [PubMed]
  9. D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
    [CrossRef]
  10. G. Passalis, N. Sgouros, S. Athineos, and T. Theoharis, "Enhanced reconstruction of three-dimensional shape and texture from integral photography images," Appl. Opt. 46, 5311-5320 (2007).
    [CrossRef] [PubMed]
  11. A. Stern and B. Javidi, "Three-Dimensional Image Sensing and Reconstruction with Time-Division Multiplexed Computational Integral Imaging," Appl. Opt. 42, 7036-7042 (2003).
    [CrossRef] [PubMed]
  12. M. Okutomi and T. Kanade, "A multiple-baseline stereo," IEEE Trans. Pattern Anal. Mach. Intell. 15, 353-363 (1993).
    [CrossRef]
  13. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (2nd ed., Cambridge university press, Cambridge, 2000).
  14. B. Lee, J.-H. Park, and S.-W. Min, "Three-dimensional display and information processing based on integral imaging," in Digital Holography and Three-Dimensional Display, T.-C. Poon, eds. (Springer, New York, USA, 2006) Chapter 12.
  15. J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
    [CrossRef] [PubMed]
  16. R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
    [CrossRef]
  17. R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
    [CrossRef]
  18. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, "Real-time pickup method for a three-dimensional image based on integral photography," Appl. Opt. 36, 1598-1603 (1997).
    [CrossRef] [PubMed]

2007 (2)

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

G. Passalis, N. Sgouros, S. Athineos, and T. Theoharis, "Enhanced reconstruction of three-dimensional shape and texture from integral photography images," Appl. Opt. 46, 5311-5320 (2007).
[CrossRef] [PubMed]

2006 (2)

M. Okui, J. Arai, Y. Nojiri, and F. Okano, "Optical screen for direct projection of integral imaging," Appl. Opt. 45, 9132-9139 (2006).
[CrossRef] [PubMed]

L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
[CrossRef]

2005 (1)

2004 (2)

J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, "Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification," Appl. Opt. 43, 4882-4895 (2004).
[CrossRef] [PubMed]

W. J. Matusik and H. Pfister, "3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes," ACM Trans. Graphics 23, 814-824 (2004).
[CrossRef]

2003 (2)

2002 (1)

1998 (1)

J. Rosen, "Three-dimensional joint transform correlator," Appl. Opt. 37, 7438-7544 (1998).
[CrossRef]

1997 (2)

F. Okano, H. Hoshino, J. Arai, and I. Yuyama, "Real-time pickup method for a three-dimensional image based on integral photography," Appl. Opt. 36, 1598-1603 (1997).
[CrossRef] [PubMed]

R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
[CrossRef]

1993 (1)

M. Okutomi and T. Kanade, "A multiple-baseline stereo," IEEE Trans. Pattern Anal. Mach. Intell. 15, 353-363 (1993).
[CrossRef]

1987 (1)

R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
[CrossRef]

Abookasis, D.

Arai, J.

Armstrong, E.

R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
[CrossRef]

Athineos, S.

Barnard, K.

R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
[CrossRef]

Carpenter, L.

R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
[CrossRef]

Catmull, E.

R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
[CrossRef]

Choi, H.

Cook, R. L.

R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
[CrossRef]

Hardie, R.

R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
[CrossRef]

Hoshino, H.

Javidi, B.

Jung, S.

Kanade, T.

M. Okutomi and T. Kanade, "A multiple-baseline stereo," IEEE Trans. Pattern Anal. Mach. Intell. 15, 353-363 (1993).
[CrossRef]

Kim, E.-S.

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

Kim, J.

Kim, Y.

Lee, B.

Lee, S.-H.

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

Matusik, W. J.

W. J. Matusik and H. Pfister, "3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes," ACM Trans. Graphics 23, 814-824 (2004).
[CrossRef]

Nojiri, Y.

Okano, F.

Okui, M.

Okutomi, M.

M. Okutomi and T. Kanade, "A multiple-baseline stereo," IEEE Trans. Pattern Anal. Mach. Intell. 15, 353-363 (1993).
[CrossRef]

Park, J.-H.

Passalis, G.

Pfister, H.

W. J. Matusik and H. Pfister, "3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes," ACM Trans. Graphics 23, 814-824 (2004).
[CrossRef]

Rosen, J.

Sgouros, N.

Shin, D.-H.

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

Stern, A.

Theoharis, T.

Vincent, A.

L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
[CrossRef]

Wang, D.

L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
[CrossRef]

Yuyama, I.

Zhang, L.

L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
[CrossRef]

ACM SIGGRAPH Computer Graphics (1)

R. L. Cook, L. Carpenter, and E. Catmull, "The Reyes image rendering architecture," ACM SIGGRAPH Computer Graphics 21, 95-102 (1987).
[CrossRef]

ACM Trans. Graphics (1)

W. J. Matusik and H. Pfister, "3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes," ACM Trans. Graphics 23, 814-824 (2004).
[CrossRef]

Appl. Opt. (6)

IEEE Trans. Circuits Syst. Video Technol. (1)

L. Zhang, D. Wang, and A. Vincent, "Adaptive reconstruction of intermediate views from stereoscopic images," IEEE Trans. Circuits Syst. Video Technol. 16, 102-113 (2006).
[CrossRef]

IEEE Trans. Image Processing (1)

R. Hardie, K. Barnard, and E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Processing 6, 1621-1633 (1997).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

M. Okutomi and T. Kanade, "A multiple-baseline stereo," IEEE Trans. Pattern Anal. Mach. Intell. 15, 353-363 (1993).
[CrossRef]

J. Opt. Soc. Am. A (1)

Opt. Commun. (1)

D.-H. Shin, S.-H. Lee, and E.-S. Kim, "Optical display of true 3D objects in depth-priority integral imaging using an active sensor," Opt. Commun. 275, 330-334 (2007).
[CrossRef]

Opt. Express (1)

Opt. Lett. (1)

Other (3)

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (2nd ed., Cambridge university press, Cambridge, 2000).

B. Lee, J.-H. Park, and S.-W. Min, "Three-dimensional display and information processing based on integral imaging," in Digital Holography and Three-Dimensional Display, T.-C. Poon, eds. (Springer, New York, USA, 2006) Chapter 12.

H. Nakanuma, H. Kamei, and Y. Takaki, "Natural 3D display with 128 directional images used for human-engineering evaluation," in Stereoscopic Displays and Virtual Reality Systems XII, A. J. Woods, M. T. Bolas, J. O. Merritt, and I. E. McDowall, eds., Proc. SPIE 5664, 28-35 (2005).
[CrossRef]

Supplementary Material (2)

» Media 1: AVI (1728 KB)     
» Media 2: AVI (2206 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (15)

Fig. 1.
Fig. 1.

Concept of the proposed method

Fig. 2.
Fig. 2.

Depth from disparity

Fig. 3.
Fig. 3.

Projection geometry (a). Perspective projection geometry (b). Orthographic projection geometry

Fig. 4.
Fig. 4.

Synthesis of view image in perspective geometry (a) depth detection for central pixels (b) central pixel projection to view image plane (c) object area corresponding to the view image area (d) elemental image areas corresponding to the view image area

Fig. 5.
Fig. 5.

2D case of elemental image mapping in perspective projection geometry

Fig. 6.
Fig. 6.

Synthesis of view image in orthographic projection geometry (a) directional pixels identification (b) depth detection and projection to the view image plane for the directional pixels (c) object area corresponding to the view image area (d) elemental image area corresponding to the view image area

Fig. 7.
Fig. 7.

2D case of elemental image mapping in orthographic projection geometry

Fig. 8.
Fig. 8.

Effect of the disparity error on the generated view (a) perspective projection case (b) orthographic projection case

Fig. 9.
Fig. 9.

Object and the elemental images used in simulation (a) object (b) elemental images

Fig. 10.
Fig. 10.

Simulation results (a) detected disparity map for central pixels (b) detected disparity map for directional pixel at (-6.9°, -6.9°) (c) generated view images in perspective projection geometry (d) generated view images in orthographic projection geometry

Fig. 11.
Fig. 11.

Simulation results in comparison with elemental or sub-image (a) FOV enhancement in perspective projection geometry (b) resolution enhancement in orthographic projection geometry

Fig. 12.
Fig. 12.

Experimental setup

Fig. 13.
Fig. 13.

Experimental results: (a) captured elemental images (b) detected disparity map of the central pixels (c) detected disparity map of the directional pixels at (5°,5°)

Fig. 14.
Fig. 14.

Experimental results: (a) generated view images in perspective projection geometry (b) generated view images in orthographic projection geometry

Fig. 15.
Fig. 15.

(Movie) Experimental results (a) generated view images in perspective projection geometry [1.69MB] (b) generated view images in orthographic projection geometry [2.15MB] [Media 1][Media 2]

Tables (1)

Tables Icon

Table 1. Specifications of experimental setup

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

SSSD ( d ) = p 2 , q 2 N i , j W [ E p 1 , q 1 ( u + i , ν + j ) E p 2 , q 2 ( u + i + ( p 2 p 1 ) d , ν + j + ( q 2 q 1 ) d ) ] 2 ,
Z = f sd φ ,
V m , n V = a V m , n V m + 1 , n + b V m , n V m , n + 1 , 0 < a + b 1 ,
C m , n E = a C m , n C m + 1 , n + b C m , n C m , n + 1 ,
tan 1 φ 2 f < θ x , θ y < tan 1 φ 2 f .

Metrics