Abstract

There are many different three-dimensional (3D) techniques to capture and deliver autostereoscopic 3D content. A promising technique that provides two-dimensional parallax as well as high-quality, full-color 3D content is integral imaging (InI). Misalignments between the lens arrays (LAs) and the camera charged coupled device, however, introduce geometric distortions in the acquired image that propagate through the different image processing stages and deteriorate the 3D effect. Here, we propose a method to accurately rectify the perspective distortion of integral images (InIms) generated using circular lenses. Using an edge-linking approach, we extracted elliptically shaped contours of elemental images in the perspectively distorted InIm. To calculate the rectification matrix, we used the images of the circular points. Subsequently, we applied a triangulation scheme followed by a statistical approach to accurately estimate the grid structure of the LA. Finally, we provided experimental results over a wide range of InIms to evaluate the robustness and accuracy of the proposed method using objective metrics.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. Lippmann, “La photographie integràle,” Comptes-Rendus Academie des Sciences 146, 446–451 (1908).
  2. J.-Y. Son and B. Javidi, “Three-dimensional imaging methods based on multiview images,” J. Display Technol. 1, 125 (2005).
    [CrossRef]
  3. J.-H. Park, Y. Kim, J. Kim, S.-W. Min, and B. Lee, “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express 12, 6020–6032 (2004).
    [CrossRef]
  4. J.-S. Jang and B. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42, 1869–1870 (2003).
    [CrossRef]
  5. Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17, 19253–19263 (2009).
    [CrossRef]
  6. G. Passalis, N. Sgouros, S. Athineos, and T. Theoharis, “Enhanced reconstruction of three-dimensional shape and texture from integral photography images,” Appl. Opt. 46, 5311–5320 (2007).
    [CrossRef]
  7. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48, H77–H94 (2009).
    [CrossRef]
  8. N. Sgouros, I. Kontaxakis, and M. Sangriotis, “Effect of different traversal schemes in integral image coding,” Appl. Opt. 47, D28–D37 (2008).
    [CrossRef]
  9. D. Liebowitz and A. Zisserman, “Metric rectification for perspective images of planes,” in Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231) (IEEE, 1998), pp. 482–488.
  10. K. Hong, J. Hong, J.-H. Jung, J.-H. Park, and B. Lee, “Rectification of elemental image set and extraction of lens lattice by projective image transformation in integral imaging,” Opt. Express 18, 12002–12016 (2010).
    [CrossRef]
  11. E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Robust integral image rectification framework using perspective transformation supported by statistical line segment clustering,” Appl. Opt. 50, H265–H277 (2011).
    [CrossRef]
  12. E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Perspective rectification of integral images produced using hexagonal lens arrays,” in IIH-MSP 2012: The Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2012), pp. 75–78.
  13. R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University, 2004).
  14. H. Ip and Y. Chen, “Planar rectification by solving the intersection of two circles under 2D homography,” Pattern Recogn. 38, 1117–1120 (2005).
    [CrossRef]
  15. M. Lourakis, “Plane metric rectification from a single view of multiple coplanar circles,” in Proceedings of the International Conference on Image Processing (ICIP) (IEEE, 2009), pp. 509–512.
  16. N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).
    [CrossRef]
  17. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. (Addison-Wesley, 2001).
  18. G. Bradski, “The OpenCV Library,” Dr. Dobb’s J. Software Tools 25 120, 122–125 (2000).
  19. P. Kovesi, “MATLAB and octave functions for computer vision and image processing,” http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/index.html .
  20. S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
    [CrossRef]

2011 (1)

2010 (1)

2009 (2)

2008 (1)

2007 (1)

2006 (1)

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

2005 (2)

J.-Y. Son and B. Javidi, “Three-dimensional imaging methods based on multiview images,” J. Display Technol. 1, 125 (2005).
[CrossRef]

H. Ip and Y. Chen, “Planar rectification by solving the intersection of two circles under 2D homography,” Pattern Recogn. 38, 1117–1120 (2005).
[CrossRef]

2004 (1)

2003 (1)

J.-S. Jang and B. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42, 1869–1870 (2003).
[CrossRef]

2000 (1)

G. Bradski, “The OpenCV Library,” Dr. Dobb’s J. Software Tools 25 120, 122–125 (2000).

1979 (1)

N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).
[CrossRef]

1908 (1)

G. Lippmann, “La photographie integràle,” Comptes-Rendus Academie des Sciences 146, 446–451 (1908).

Athineos, S.

Athineos, S. S.

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

Bradski, G.

G. Bradski, “The OpenCV Library,” Dr. Dobb’s J. Software Tools 25 120, 122–125 (2000).

Chen, Y.

H. Ip and Y. Chen, “Planar rectification by solving the intersection of two circles under 2D homography,” Pattern Recogn. 38, 1117–1120 (2005).
[CrossRef]

Gonzalez, R. C.

R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. (Addison-Wesley, 2001).

Hartley, R. I.

R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University, 2004).

Hong, J.

Hong, K.

Ip, H.

H. Ip and Y. Chen, “Planar rectification by solving the intersection of two circles under 2D homography,” Pattern Recogn. 38, 1117–1120 (2005).
[CrossRef]

Jang, J.-S.

J.-S. Jang and B. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42, 1869–1870 (2003).
[CrossRef]

Javidi, B.

J.-Y. Son and B. Javidi, “Three-dimensional imaging methods based on multiview images,” J. Display Technol. 1, 125 (2005).
[CrossRef]

J.-S. Jang and B. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42, 1869–1870 (2003).
[CrossRef]

Jung, J.-H.

Kim, J.

Kim, N.

Kim, Y.

Kontaxakis, I.

Koufogiannis, E. T.

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Robust integral image rectification framework using perspective transformation supported by statistical line segment clustering,” Appl. Opt. 50, H265–H277 (2011).
[CrossRef]

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Perspective rectification of integral images produced using hexagonal lens arrays,” in IIH-MSP 2012: The Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2012), pp. 75–78.

Kwon, K.-C.

Lee, B.

Liebowitz, D.

D. Liebowitz and A. Zisserman, “Metric rectification for perspective images of planes,” in Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231) (IEEE, 1998), pp. 482–488.

Lim, Y.-T.

Lippmann, G.

G. Lippmann, “La photographie integràle,” Comptes-Rendus Academie des Sciences 146, 446–451 (1908).

Lourakis, M.

M. Lourakis, “Plane metric rectification from a single view of multiple coplanar circles,” in Proceedings of the International Conference on Image Processing (ICIP) (IEEE, 2009), pp. 509–512.

Maroulis, D. E.

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

Min, S.-W.

Otsu, N.

N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).
[CrossRef]

Papageorgas, P. G.

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

Park, J.-H.

Passalis, G.

Sangriotis, M.

Sangriotis, M. S.

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Robust integral image rectification framework using perspective transformation supported by statistical line segment clustering,” Appl. Opt. 50, H265–H277 (2011).
[CrossRef]

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Perspective rectification of integral images produced using hexagonal lens arrays,” in IIH-MSP 2012: The Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2012), pp. 75–78.

Sgouros, N.

Sgouros, N. P.

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Robust integral image rectification framework using perspective transformation supported by statistical line segment clustering,” Appl. Opt. 50, H265–H277 (2011).
[CrossRef]

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Perspective rectification of integral images produced using hexagonal lens arrays,” in IIH-MSP 2012: The Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2012), pp. 75–78.

Son, J.-Y.

Theofanous, N. G.

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

Theoharis, T.

Woods, R. E.

R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. (Addison-Wesley, 2001).

Zisserman, A.

R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University, 2004).

D. Liebowitz and A. Zisserman, “Metric rectification for perspective images of planes,” in Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231) (IEEE, 1998), pp. 482–488.

Appl. Opt. (4)

Comptes-Rendus Academie des Sciences (1)

G. Lippmann, “La photographie integràle,” Comptes-Rendus Academie des Sciences 146, 446–451 (1908).

Dr. Dobb’s J. Software Tools (1)

G. Bradski, “The OpenCV Library,” Dr. Dobb’s J. Software Tools 25 120, 122–125 (2000).

IEEE Trans. Syst. Man Cybern. (1)

N. Otsu, “A threshold selection method from gray-level histograms,” IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).
[CrossRef]

J. Display Technol. (1)

J. Electron. Imaging (1)

S. S. Athineos, N. P. Sgouros, P. G. Papageorgas, D. E. Maroulis, M. S. Sangriotis, and N. G. Theofanous, “Photorealistic integral photography using a ray-traced model of capturing optics,” J. Electron. Imaging 15, 043007 (2006).
[CrossRef]

Opt. Eng. (1)

J.-S. Jang and B. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42, 1869–1870 (2003).
[CrossRef]

Opt. Express (3)

Pattern Recogn. (1)

H. Ip and Y. Chen, “Planar rectification by solving the intersection of two circles under 2D homography,” Pattern Recogn. 38, 1117–1120 (2005).
[CrossRef]

Other (6)

M. Lourakis, “Plane metric rectification from a single view of multiple coplanar circles,” in Proceedings of the International Conference on Image Processing (ICIP) (IEEE, 2009), pp. 509–512.

R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. (Addison-Wesley, 2001).

E. T. Koufogiannis, N. P. Sgouros, and M. S. Sangriotis, “Perspective rectification of integral images produced using hexagonal lens arrays,” in IIH-MSP 2012: The Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2012), pp. 75–78.

R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University, 2004).

D. Liebowitz and A. Zisserman, “Metric rectification for perspective images of planes,” in Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231) (IEEE, 1998), pp. 482–488.

P. Kovesi, “MATLAB and octave functions for computer vision and image processing,” http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/index.html .

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1.
Fig. 1.

(a) InI acquisition and (b) InI display setup.

Fig. 2.
Fig. 2.

Various LAs consisting of (a) circular, (b) square, and (c) hexagonal and triangular lenses.

Fig. 3.
Fig. 3.

(a) Undistorted coplanar circular EIs, (b) perspectively distorted EIs, (c) a conventional photograph of a 3D dice, (d) acquired InIm of the dice without distortion, and (e) perspectively distorted acquired InIm. The borders in (d) and (e) are shown for illustration purposes.

Fig. 4.
Fig. 4.

(a) Set of coplanar ellipses corresponding to perspectively distorted circles, (b) the affine plane of ellipses resulting after applying Hp, (c) the metric-corrected ellipses transformed to circles after applying Ha, and (d) the final and correctly rotated and scaled set of coplanar circles after applying the similarity transformation Hs.

Fig. 5.
Fig. 5.

(a) Any pair of coplanar circles verifies the CPs shown in (b) under the perspective distortion H1, the pair of ellipses in (c) corresponding to the circles in (a) verifies the ICPs shown in (d).

Fig. 6.
Fig. 6.

(a) Distorted acquired InIm, (b) edge image with fitted ellipses overlaid, (c) rectified image and Delaunay triangulation on the rectified circle centers, and (d) correctly rotated image with the registered grid superimposed.

Fig. 7.
Fig. 7.

Histogram of candidate vanishing line distances from the image origin. The lobe corresponds to correct ICPs.

Fig. 8.
Fig. 8.

Angle histogram of the segments connecting the circle centers after rectification.

Fig. 9.
Fig. 9.

Rectification on a raytraced and an optically acquired image: (a) A “3D Objects” scene, (b) an optically acquired “Toy,” (c),(d) the corresponding acquired and distorted InIms, and (e),(f) rectified InIms using the proposed method. The borders in (c),(f) are shown for illustration purposes.

Fig. 10.
Fig. 10.

(a),(b) Side view subimages rendered from the undistorted “3D Objects” and “Toy” InIms, (c),(d) side view subimages of the “3D Objects” and “Toy” rendered using the perspectively distorted InIms in Figs. 9(c) and 9(d), (e),(f) side view subimages of the “3D Objects” and “Toy” rendered using the rectified InIms of Figs. 9(e) and 9(f).

Fig. 11.
Fig. 11.

Mean value and standard deviation of the PSNR sequence for the rectified EIs of Fig. 9(c). The horizontal axis denotes the input InIm quality, while the vertical axis denotes the evaluated PSNR sequence of the EIs in the geometrically corrected InIm.

Tables (6)

Tables Icon

Table 1. Mean Relative Error Percentages of the Rectification Parameters for the Entire Raytraced InIm Set

Tables Icon

Table 2. Cumulative Geometric Consistency Results for the Raytraced Inim Set

Tables Icon

Table 3. Relative Error Percentages of the Rectification Parameters for “3D Objects”

Tables Icon

Table 4. Geometric Consistency Results for “3D Objects”

Tables Icon

Table 5. Cumulative Geometric Consistency Results for All InIms of the Optically Acquired Set

Tables Icon

Table 6. Geometric Consistency Results for the “Toy”

Equations (32)

Equations on this page are rendered with MathJax. Learn more.

x⃗=Hx⃗,
H=HsHaHp.
Hp=(100010l1l2l3),
Ha=(1βαβ0010001),
Hs=(Rx0y0001)(c000c0001),
R=(cosθsinθsinθcosθ),
(XmZ)2+(YnZ)2=r2Z2.
C=(ab/2d/2b/2ce/2d/2e/2f).
HTCH1=C.
ICPs=(αl3il3β,l3,l2l1α±il1β)T.
ax2+bxy+cy2+dx+ey+f=0,
{a1x2+b1xy+c1y2+d1x+e1y+f1=0a2x2+b2xy+c2y2+d2x+e2y+f2=0.
{(xs0,ys0),(xs0¯,ys0¯)(xs1,ys1),(xs1¯,ys1¯).
(l1,l2,l3)T=(xc,yc,1)T=(xc¯,yc¯,1)T.
{α=Real(l2xcl3+l1xc)=Real(l3+l2ycl1yc)β=|Imag(l2xcl3+l1xc)|=|Imag(l3+l2ycl1yc)|.
Au=0,
A=(x12x1y1y12x1y11x12x2y2y22x2y21xM2xMyMyM2xMyM1),
u=(a,b,c,d,e,f)T.
A=UDVT.
MSE=1MAu^.
P={(xs0,ys0),(xs0¯,y0¯)(xs1,ys1),(xs1¯,ys1¯),}.
Ii=(xsi,ysi)T×(xsi¯,ysi¯)T.
xh=mean(Δx{Sh})+mean(Δy{Sv})2,
yh=mean(Δy{Sh})+mean(Δx{Sv})2,
R=1xh2+yh2(xhyhyhxh).
(HaHp)1(1,±i,0)T=ICP1,2.
ICP1,2=(αl3il3β,l3,l2,l1α±il1β)T.
ICP1=(xc,yc),ICP2=(xc¯,yc¯).
(l1,l2,l3)T=(xc,yc,1)T×(xc¯,yc¯,1)T.
{(xc,yc,1)=(αl3il3βl2l1α+il1β,l3l2l1α+il1β,1)(xc¯,yc¯,1)=(αl3+il3βl2l1αil1β,l3l2l1α+il1β,1),
αiβ=l2xcl3+l1xc=l3+l2ycl1yc,
{α=Real(l2xcl3+l1xc)=Real(l3+l2ycl1yc)β=|Imag(l2xcl3+l1xc)|=|Imag(l3+l2ycl1yc)|.

Metrics