Abstract

Plenoptic imaging systems are often used for applications like refocusing, multimodal imaging, and multiview imaging. However, their resolution is limited to the number of lenslets. In this paper we investigate paraxial, incoherent, plenoptic image formation, and develop a method to recover some of the resolution for the case of a two-dimensional (2D) in-focus object. This enables the recovery of a conventional-resolution, 2D image from the data captured in a plenoptic system. We show simulation results for a plenoptic system with a known response and Gaussian sensor noise.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. T. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
    [CrossRef]
  2. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).
  3. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
    [CrossRef]
  4. C. Perwass and L. Wietzke, “Single-lens 3D camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012).
    [CrossRef]
  5. R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).
  6. T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).
  7. Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” IEEE International Conference on Computational Photography (IEEE, 2009).
  8. D. J. Brady and D. L. Marks, “Coding for compressive focal tomography,” Appl. Opt. 50, 4436–4449 (2011).
    [CrossRef]
  9. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1986).
  10. S. A. Shroff and K. Berkner, “Defocus analysis for a coherent plenoptic system,” in Frontiers in Optics, OSA Technical Digest (Optical Society of America, 2011), paper FThR6.
  11. S. A. Shroff and K. Berkner, “High Resolution image reconstruction for plenoptic imaging systems using system response,” in Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America, 2012), paper CM2B.2.
  12. S. A. Shroff and K. Berkner, “Wave analysis of a plenoptic system and its applications,” Proc. SPIE 8667, 86671L (2013).
  13. The Mathworks, Natick, MA, http://www.mathworks.com .
  14. J. Goodman, “Assessing a new imaging modality,” in Optical Sensors, OSA Technical Digest (Optical Society of America, 2012), paper JM1A.1.
  15. J. R. Fienup and C. C. Wackerman, “Phase-retrieval stagnation problems and solutions,” J. Opt. Soc. Am. A 3, 1897–1907 (1986).
    [CrossRef]
  16. K. Berkner and S. A. Shroff, “Optimization of spectrally coded mask for multi-modal plenoptic camera,” in Computational Optical Sensing and Imaging/Information Photonics (Optical Society of America, 2011), paper CMD4.

2013 (1)

S. A. Shroff and K. Berkner, “Wave analysis of a plenoptic system and its applications,” Proc. SPIE 8667, 86671L (2013).

2012 (1)

C. Perwass and L. Wietzke, “Single-lens 3D camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012).
[CrossRef]

2011 (1)

2006 (1)

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

1992 (1)

T. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
[CrossRef]

1986 (1)

Adams, A.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

Adelson, T.

T. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
[CrossRef]

Athale, R.

R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Berkner, K.

S. A. Shroff and K. Berkner, “Wave analysis of a plenoptic system and its applications,” Proc. SPIE 8667, 86671L (2013).

S. A. Shroff and K. Berkner, “Defocus analysis for a coherent plenoptic system,” in Frontiers in Optics, OSA Technical Digest (Optical Society of America, 2011), paper FThR6.

S. A. Shroff and K. Berkner, “High Resolution image reconstruction for plenoptic imaging systems using system response,” in Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America, 2012), paper CM2B.2.

K. Berkner and S. A. Shroff, “Optimization of spectrally coded mask for multi-modal plenoptic camera,” in Computational Optical Sensing and Imaging/Information Photonics (Optical Society of America, 2011), paper CMD4.

Bishop, T. E.

T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Brady, D. J.

Brédif, M.

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

Duval, G.

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

Euliss, G.

R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Favaro, P.

T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Fienup, J. R.

Footer, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

Goodman, J.

J. Goodman, “Assessing a new imaging modality,” in Optical Sensors, OSA Technical Digest (Optical Society of America, 2012), paper JM1A.1.

Goodman, J. W.

J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1986).

Hanrahan, P.

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

Horowitz, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

Horstmeyer, R.

R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Levoy, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” IEEE International Conference on Computational Photography (IEEE, 2009).

Marks, D. L.

Ng, R.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

Perwass, C.

C. Perwass and L. Wietzke, “Single-lens 3D camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012).
[CrossRef]

Shroff, S. A.

S. A. Shroff and K. Berkner, “Wave analysis of a plenoptic system and its applications,” Proc. SPIE 8667, 86671L (2013).

S. A. Shroff and K. Berkner, “High Resolution image reconstruction for plenoptic imaging systems using system response,” in Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America, 2012), paper CM2B.2.

S. A. Shroff and K. Berkner, “Defocus analysis for a coherent plenoptic system,” in Frontiers in Optics, OSA Technical Digest (Optical Society of America, 2011), paper FThR6.

K. Berkner and S. A. Shroff, “Optimization of spectrally coded mask for multi-modal plenoptic camera,” in Computational Optical Sensing and Imaging/Information Photonics (Optical Society of America, 2011), paper CMD4.

Wackerman, C. C.

Wang, J.

T. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
[CrossRef]

Wietzke, L.

C. Perwass and L. Wietzke, “Single-lens 3D camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012).
[CrossRef]

Zanetti, S.

T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Zhang, Z.

Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” IEEE International Conference on Computational Photography (IEEE, 2009).

ACM Trans. Graph. (1)

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25, 924–934 (2006).
[CrossRef]

Appl. Opt. (1)

IEEE Trans. Pattern Anal. Mach. Intell. (1)

T. Adelson and J. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992).
[CrossRef]

J. Opt. Soc. Am. A (1)

Proc. SPIE (2)

S. A. Shroff and K. Berkner, “Wave analysis of a plenoptic system and its applications,” Proc. SPIE 8667, 86671L (2013).

C. Perwass and L. Wietzke, “Single-lens 3D camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012).
[CrossRef]

Other (10)

R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” Proceedings of the IEEE International Conference on Computational Photography (IEEE, 2009).

Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” IEEE International Conference on Computational Photography (IEEE, 2009).

J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1986).

S. A. Shroff and K. Berkner, “Defocus analysis for a coherent plenoptic system,” in Frontiers in Optics, OSA Technical Digest (Optical Society of America, 2011), paper FThR6.

S. A. Shroff and K. Berkner, “High Resolution image reconstruction for plenoptic imaging systems using system response,” in Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America, 2012), paper CM2B.2.

The Mathworks, Natick, MA, http://www.mathworks.com .

J. Goodman, “Assessing a new imaging modality,” in Optical Sensors, OSA Technical Digest (Optical Society of America, 2012), paper JM1A.1.

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. (Stanford University, 2005).

K. Berkner and S. A. Shroff, “Optimization of spectrally coded mask for multi-modal plenoptic camera,” in Computational Optical Sensing and Imaging/Information Photonics (Optical Society of America, 2011), paper CMD4.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1.
Fig. 1.

Schematic layout of a plenoptic imaging system, with the first subsystem containing the main lens (solid gray), and the second subsystem containing the microlens array (dotted gray) [10,11,12].

Fig. 2.
Fig. 2.

(a), (c), and (e) show the PSF responses on a stretched scale for impulses that are laterally shifted 0%, 50%, and 85% of the lenslet radius away from the optical axis, respectively, cropped by the extent of the on-axis lenslet. (b), (d), and (f) show the overall PIF responses of the system for the same impulses.

Fig. 3.
Fig. 3.

Simulation of incoherent plenoptic image for a single lenslet. (a) shows the pristine object, (b) is the sensor data when the object plane is focused at the plane of the lenslet array, and (c)–(f) are the sensor data when the object is defocused by + 2 , + 5 , 2 , and 5 mm , respectively.

Fig. 4.
Fig. 4.

Schematic layout of a plenoptic imaging system when the object is moved closer to the main lens.

Fig. 5.
Fig. 5.

Schematic layout of a plenoptic imaging system when the object is moved farther from the main lens.

Fig. 6.
Fig. 6.

Simulation for a single lenslet. (a) Conventional image. (b) Deconvolved conventional image. (c) Reconstruction with a twin-image error, obtained using a circle shaped lenslet shown in (d), and (e) is a reconstruction without the twin-image error, obtained using a lenslet with a noncentrosymmetric shape, shown in (f).

Fig. 7.
Fig. 7.

Noncentrosymmetry in shape as well as amplitude improved image reconstruction. (a) and (b) show lenslets with noncentrosymetric shape and amplitude, respectively. Alternatively, phase aberrations in lenslets were also useful. (c) and (d) illustrate the real and imaginary parts of noncentrosymmetric phase in a lenslet.

Fig. 8.
Fig. 8.

Simulation for a single lenslet. (a) Pristine object used in simulation. (b) Conventional image, no noise. (c) Deconvolved noiseless conventional image. (d) Reconstruction with pseudoinverse for noiseless data. (e) and (f) are reconstructions using linear least squares and nonlinear iterative algorithms for noisy data, SNR = 40 dB .

Fig. 9.
Fig. 9.

Simulation for an 11 × 11 array of lenslets. (a) Pristine object used in simulation. (b) Plenoptic sensor data. (c) Traditional single plenoptic image obtained by binning light behind each lenslet into 1 pixel. (d) Reconstruction with pseudoinverse for noiseless data. (e) and (f) are reconstructions using linear least squares and nonlinear iterative algorithms for noisy data, SNR = 40 dB for the case where the object is in focus at the lenslet array. (g) and (h) are the same when the object is defocused by + 2 mm .

Equations (13)

Equations on this page are rendered with MathJax. Learn more.

h 1 ( u , v ; ξ , η ) = e j k z 1 e j k z 2 λ 2 z 1 z 2 exp [ j k 2 z 2 ( u 2 + v 2 ) ] exp [ j k 2 z 1 ( ξ 2 + η 2 ) ] d x d y P 1 ( x , y ) exp [ j k 2 ( 1 z 1 + 1 z 2 1 f 1 ) ( x 2 + y 2 ) ] exp { j k z 2 [ ( u M ξ ) x + ( v M η ) y ] } ,
h 1 ( u , v ; ξ , η ) = e j k z 1 e j k z 2 exp [ j k 2 z 2 ( u 2 + v 2 ) ] exp [ j k 2 z 1 ( ξ 2 + η 2 ) ] M d x d y P 1 ( x λ z 2 , y λ z 2 ) exp { j k 2 ( 1 z 1 + 1 z 2 1 f 1 ) [ ( x λ z 2 ) 2 + ( y λ z 2 ) 2 ] } exp { j 2 π [ ( u M ξ ) x + ( v M η ) y ] } .
h 1 ( u , v ; ξ , η ) = M e j k z 1 e j k z 2 exp [ j k 2 z 2 ( u 2 + v 2 ) ] exp [ j k 2 z 1 ( ξ 2 + η 2 ) ] h 1 ( u M ξ , v M η ) .
U i ( u , v ) = e j k z 1 e j k z 2 M exp [ j k 2 z 2 ( u 2 + v 2 ) ] d ξ d η U o ( ξ M , η M ) exp { j k 2 z 1 [ ( ξ M ) 2 + ( η M ) 2 ] } h 1 ( u ξ , v η ) = e j k z 1 e j k z 2 M exp [ j k 2 z 2 ( u 2 + v 2 ) ] { [ U o ( u M , v M ) exp { j k 2 z 1 [ ( u M ) 2 + ( v M ) 2 ] } ] * h 1 ( u , v ) } ,
U i ( u , v ) = U i ( u , v ) m n P 2 ( u m D 2 , v n D 2 ) exp { j k 2 f 2 [ ( u m D 2 ) 2 + ( v n D 2 ) 2 ] } .
U f ( t , w ) = e j k z 1 e j k z 2 e j k z 3 j λ z 3 M exp [ j k 2 z 3 ( t 2 + w 2 ) ] m n exp { j k 2 f 2 [ ( m D 2 ) 2 + ( n D 2 ) 2 ] } d u d v P 2 ( u m D 2 , v n D 2 ) exp [ j k 2 ( 1 z 2 + 1 z 3 1 f 2 ) ( u 2 + v 2 ) ] exp { j k [ u ( t z 3 m D 2 f 2 ) + v ( w z 3 n D 2 f 2 ) ] } d ξ d η U o ( ξ M , η M ) exp { j k 2 z 1 [ ( ξ M ) 2 + ( η M ) 2 ] } h 1 ( u ξ , v η ) .
U o ( ξ , η ) U o * ( ξ ˜ , η ˜ ) = I o ( ξ , η ) δ ( ξ ξ ˜ , η η ˜ ) .
I f ( t , w ) = d ξ d η I o ( ξ M , η M ) | e j k z 1 e j k z 2 e j k z 3 j λ z 3 M exp [ j k 2 z 3 ( t 2 + w 2 ) ] m n exp { j k 2 f 2 [ ( m D 2 ) 2 + ( n D 2 ) 2 ] } d u d v P 2 ( u m D 2 , v n D 2 ) exp [ j k 2 ( 1 z 2 + 1 z 3 1 f 2 ) ( u 2 + v 2 ) ] exp { j k [ u ( t z 3 m D 2 f 2 ) + v ( w z 3 n D 2 f 2 ) ] } h 1 ( u ξ , v η ) | 2 .
PIF ξ , η t , w = PIF ( t , w , ξ , η ) = | e j k z 1 e j k z 2 e j k z 3 j λ z 3 M exp [ j k 2 z 3 ( t 2 + w 2 ) ] m n exp { j k 2 f 2 [ ( m D 2 ) 2 + ( n D 2 ) 2 ] } d u d v P 2 ( u m D 2 , v n D 2 ) exp [ j k 2 ( 1 z 2 + 1 z 3 1 f 2 ) ( u 2 + v 2 ) ] exp { j k [ u ( t z 3 m D 2 f 2 ) + v ( w z 3 n D 2 f 2 ) ] } h 1 ( u ξ , v η ) | 2 .
[ I v f ] = [ PIF v ] [ I v o ] ,
[ I f 1 , 1 I f 1 , 2 . . I f 2 , 1 I f 2 , 2 . I f T , W ] = [ PIF 1 , 1 1 , 1 PIF 1 , 2 1 , 1 . . PIF M , N 1 , 1 PIF 1 , 1 1 , 2 PIF 1 , 2 1 , 2 . . PIF M , N 1 , 2 . . . . . . . . . . PIF 1 , 1 1 , W PIF 1 , 2 1 , W . . PIF M , N 1 , W PIF 1 , 1 2 , 1 PIF 1 , 2 2 , 1 . . PIF M , N 2 , W . . . . . PIF 1 , 1 T , W PIF 1 , 2 T , W . . PIF M , N T , W ] [ I o , 1 , 1 I o , 1 , 2 . . I o , 2 , 1 I o , 2 , 1 . I o , M , N ] .
[ I ^ v o ] = argmin I v o [ PIF v ] [ I v o ] [ I v f ]
[ I v f ] = [ PIF v ] [ I v o ] + N .

Metrics