Abstract

Light field microscopy is a new technique for high-speed volumetric imaging of weakly scattering or fluorescent specimens. It employs an array of microlenses to trade off spatial resolution against angular resolution, thereby allowing a 4-D light field to be captured using a single photographic exposure without the need for scanning. The recorded light field can then be used to computationally reconstruct a full volume. In this paper, we present an optical model for light field microscopy based on wave optics, instead of previously reported ray optics models. We also present a 3-D deconvolution method for light field microscopy that is able to reconstruct volumes at higher spatial resolution, and with better optical sectioning, than previously reported. To accomplish this, we take advantage of the dense spatio-angular sampling provided by a microlens array at axial positions away from the native object plane. This dense sampling permits us to decode aliasing present in the light field to reconstruct high-frequency information. We formulate our method as an inverse problem for reconstructing the 3-D volume, which we solve using a GPU-accelerated iterative algorithm. Theoretical limits on the depth-dependent lateral resolution of the reconstructed volumes are derived. We show that these limits are in good agreement with experimental results on a standard USAF 1951 resolution target. Finally, we present 3-D reconstructions of pollen grains that demonstrate the improvements in fidelity made possible by our method.

© 2013 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
    [CrossRef]
  2. M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
    [CrossRef] [PubMed]
  3. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
    [CrossRef]
  4. R. Ng, “Fourier slice photography,” in Proceedings of ACM SIGGRAPH(2005). 735–744.
    [CrossRef]
  5. M. Bertero and C. de Mol, “III Super-resolution by data inversion,” in Progress in Optics (Elsevier, 1996) pp. 129–178.
    [CrossRef]
  6. T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
    [CrossRef]
  7. S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell.24. 1167–1183 (2002)
    [CrossRef]
  8. K. Grochenig and T. Strohmer, “Numerical and theoretical aspects of nonuniform sampling of band-limited images,” in “Nonuniform Sampling,” F. Marvasti, ed.. Information Technology: Transmission, Processing, and Storage, 283–324 (SpringerUS, 2010).
  9. T. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell.34. 972–986 (2012).
    [CrossRef]
  10. W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
    [CrossRef]
  11. M. Gu, Advanced Optical Imaging Theory (Springer, 1999).
  12. D. A. Agard, “Optical sectioning microscopy: cellular architecture in three dimensions,” Annual review of biophysics and bioengineering13, 191–219. (1984).
    [CrossRef] [PubMed]
  13. M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999).
  14. M. R. Arnison and C. J. R. Sheppard, “A 3D vectorial optical transfer function suitable for arbitrary pupil functions,” Optics communications211, 53–63 (2002).
    [CrossRef]
  15. A. Egner and S. W. Hell, “Equivalence of the Huygens–Fresnel and Debye approach for the calculation of high aperture point-spread functions in the presence of refractive index mismatch,” Journal of Microscopy193, 244–249 (1999).
    [CrossRef]
  16. J. Breckinridge, D. Voelz, and J. B. Breckinridge, Computational Fourier Optics: a MATLAB Tutorial (SPIE Press, 2011).
  17. J. M. Bardsley and J. G. Nagy, “Covariance-preconditioned iterative methods for nonnegatively constrained astronomical imaging,” SIAM journal on matrix analysis and applications27, 1184–1197 (2006).
    [CrossRef]
  18. M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
    [CrossRef]
  19. S. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Applied optics, 52, D22D31, (2013).
    [CrossRef]
  20. J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
    [CrossRef]
  21. I. J. Cox and C. J. R. Sheppard, “Information capacity and resolution in an optical system,” J. Opt. Soc. Am. A3, 1152 (1986).
    [CrossRef]
  22. R. Heintzmann, “Estimating missing information by maximum likelihood deconvolution,” Micron38, 136–144 (2007)
    [CrossRef]
  23. P. Favaro, “A split-sensor light field camera for extended depth of field and superresolution,” in “SPIE Conference Series,” 8436. (2012).
  24. C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in “Computational Optical Sensing and Imaging, Microscopy and Tomography I (CTh3B),” (2013).
  25. S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).
  26. M. Pluta, Advanced Light Microscopy, Vol. 1.(Elsevier, 1988).
  27. J. Goodman, Introduction to Fourier Optics, 2nd ed. (MaGraw-Hill, 1996).

2013

S. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Applied optics, 52, D22D31, (2013).
[CrossRef]

2012

P. Favaro, “A split-sensor light field camera for extended depth of field and superresolution,” in “SPIE Conference Series,” 8436. (2012).

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

T. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell.34. 972–986 (2012).
[CrossRef]

2011

J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
[CrossRef]

2009

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
[CrossRef] [PubMed]

2007

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

R. Heintzmann, “Estimating missing information by maximum likelihood deconvolution,” Micron38, 136–144 (2007)
[CrossRef]

2006

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

J. M. Bardsley and J. G. Nagy, “Covariance-preconditioned iterative methods for nonnegatively constrained astronomical imaging,” SIAM journal on matrix analysis and applications27, 1184–1197 (2006).
[CrossRef]

2005

R. Ng, “Fourier slice photography,” in Proceedings of ACM SIGGRAPH(2005). 735–744.
[CrossRef]

T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
[CrossRef]

2004

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

2002

S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell.24. 1167–1183 (2002)
[CrossRef]

M. R. Arnison and C. J. R. Sheppard, “A 3D vectorial optical transfer function suitable for arbitrary pupil functions,” Optics communications211, 53–63 (2002).
[CrossRef]

1999

A. Egner and S. W. Hell, “Equivalence of the Huygens–Fresnel and Debye approach for the calculation of high aperture point-spread functions in the presence of refractive index mismatch,” Journal of Microscopy193, 244–249 (1999).
[CrossRef]

1986

1984

D. A. Agard, “Optical sectioning microscopy: cellular architecture in three dimensions,” Annual review of biophysics and bioengineering13, 191–219. (1984).
[CrossRef] [PubMed]

Abrahamsson, S.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Adams, A.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

Agard, D. A.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

D. A. Agard, “Optical sectioning microscopy: cellular architecture in three dimensions,” Annual review of biophysics and bioengineering13, 191–219. (1984).
[CrossRef] [PubMed]

Arnison, M. R.

M. R. Arnison and C. J. R. Sheppard, “A 3D vectorial optical transfer function suitable for arbitrary pupil functions,” Optics communications211, 53–63 (2002).
[CrossRef]

Baker, S.

S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell.24. 1167–1183 (2002)
[CrossRef]

Bardsley, J. M.

J. M. Bardsley and J. G. Nagy, “Covariance-preconditioned iterative methods for nonnegatively constrained astronomical imaging,” SIAM journal on matrix analysis and applications27, 1184–1197 (2006).
[CrossRef]

Bargmann, C. I.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Berkner, K.

S. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Applied optics, 52, D22D31, (2013).
[CrossRef]

Bertero, M.

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

M. Bertero and C. de Mol, “III Super-resolution by data inversion,” in Progress in Optics (Elsevier, 1996) pp. 129–178.
[CrossRef]

Bishop, T.

T. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell.34. 972–986 (2012).
[CrossRef]

Boccacci, P.

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

Born, M.

M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999).

Breckinridge, J.

J. Breckinridge, D. Voelz, and J. B. Breckinridge, Computational Fourier Optics: a MATLAB Tutorial (SPIE Press, 2011).

Breckinridge, J. B.

J. Breckinridge, D. Voelz, and J. B. Breckinridge, Computational Fourier Optics: a MATLAB Tutorial (SPIE Press, 2011).

Brooker, G.

J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
[CrossRef]

Chan, W.

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

Chen, J.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Cox, I. J.

Dahan, M.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Darzacq, C. D.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Darzacq, X.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

de Mol, C.

M. Bertero and C. de Mol, “III Super-resolution by data inversion,” in Progress in Optics (Elsevier, 1996) pp. 129–178.
[CrossRef]

Desidera, G.

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

Egner, A.

A. Egner and S. W. Hell, “Equivalence of the Huygens–Fresnel and Debye approach for the calculation of high aperture point-spread functions in the presence of refractive index mismatch,” Journal of Microscopy193, 244–249 (1999).
[CrossRef]

Elad, M.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

Farsiu, S.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

Favaro, P.

T. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell.34. 972–986 (2012).
[CrossRef]

P. Favaro, “A split-sensor light field camera for extended depth of field and superresolution,” in “SPIE Conference Series,” 8436. (2012).

Fleischer, J.

C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in “Computational Optical Sensing and Imaging, Microscopy and Tomography I (CTh3B),” (2013).

Footer, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

Goodman, J.

J. Goodman, Introduction to Fourier Optics, 2nd ed. (MaGraw-Hill, 1996).

Grochenig, K.

K. Grochenig and T. Strohmer, “Numerical and theoretical aspects of nonuniform sampling of band-limited images,” in “Nonuniform Sampling,” F. Marvasti, ed.. Information Technology: Transmission, Processing, and Storage, 283–324 (SpringerUS, 2010).

Gu, M.

M. Gu, Advanced Optical Imaging Theory (Springer, 1999).

Gustafsson, M. G. L.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Hajj, B.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Heintzmann, R.

R. Heintzmann, “Estimating missing information by maximum likelihood deconvolution,” Micron38, 136–144 (2007)
[CrossRef]

Hell, S. W.

A. Egner and S. W. Hell, “Equivalence of the Huygens–Fresnel and Debye approach for the calculation of high aperture point-spread functions in the presence of refractive index mismatch,” Journal of Microscopy193, 244–249 (1999).
[CrossRef]

Horowitz, M.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

Kanade, T.

S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell.24. 1167–1183 (2002)
[CrossRef]

Katsov, A. Y.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Lam, E.

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

Levoy, M.

M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
[CrossRef] [PubMed]

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

Lu, C. H.

C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in “Computational Optical Sensing and Imaging, Microscopy and Tomography I (CTh3B),” (2013).

Mak, G.

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

McDowell, I.

M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
[CrossRef] [PubMed]

Milanfar, P.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

Mizuguchi, G.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Mueller, F.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Muenzel, S.

C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in “Computational Optical Sensing and Imaging, Microscopy and Tomography I (CTh3B),” (2013).

Nagy, J. G.

J. M. Bardsley and J. G. Nagy, “Covariance-preconditioned iterative methods for nonnegatively constrained astronomical imaging,” SIAM journal on matrix analysis and applications27, 1184–1197 (2006).
[CrossRef]

Ng, M.

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

Ng, R.

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

R. Ng, “Fourier slice photography,” in Proceedings of ACM SIGGRAPH(2005). 735–744.
[CrossRef]

Pham, T.

T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
[CrossRef]

Pluta, M.

M. Pluta, Advanced Light Microscopy, Vol. 1.(Elsevier, 1988).

Robinson, D.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

Rosen, J.

J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
[CrossRef]

Schutte, K.

T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
[CrossRef]

Sheppard, C. J. R.

M. R. Arnison and C. J. R. Sheppard, “A 3D vectorial optical transfer function suitable for arbitrary pupil functions,” Optics communications211, 53–63 (2002).
[CrossRef]

I. J. Cox and C. J. R. Sheppard, “Information capacity and resolution in an optical system,” J. Opt. Soc. Am. A3, 1152 (1986).
[CrossRef]

Shroff, S.

S. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Applied optics, 52, D22D31, (2013).
[CrossRef]

Siegel, N.

J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
[CrossRef]

Soule, P.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Stallinga, S.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Strohmer, T.

K. Grochenig and T. Strohmer, “Numerical and theoretical aspects of nonuniform sampling of band-limited images,” in “Nonuniform Sampling,” F. Marvasti, ed.. Information Technology: Transmission, Processing, and Storage, 283–324 (SpringerUS, 2010).

van Vliet, L.

T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
[CrossRef]

Vicidomini, G.

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

Voelz, D.

J. Breckinridge, D. Voelz, and J. B. Breckinridge, Computational Fourier Optics: a MATLAB Tutorial (SPIE Press, 2011).

Wisniewski, J.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Wolf, E.

M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999).

Wu, C.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Zhang, Z.

M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
[CrossRef] [PubMed]

Annual review of biophysics and bioengineering

D. A. Agard, “Optical sectioning microscopy: cellular architecture in three dimensions,” Annual review of biophysics and bioengineering13, 191–219. (1984).
[CrossRef] [PubMed]

Applied optics

S. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Applied optics, 52, D22D31, (2013).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell.

S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell.24. 1167–1183 (2002)
[CrossRef]

T. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell.34. 972–986 (2012).
[CrossRef]

International Journal of Imaging Systems and Technology

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology14, 47–57 (2004).
[CrossRef]

Inverse Problems

M. Bertero, P. Boccacci, G. Desidera, and G. Vicidomini, “Image deblurring with Poisson data: from cells to galaxies,” Inverse Problems25, 123006 (2009).
[CrossRef]

J. Opt. Soc. Am. A

Journal of Microscopy

A. Egner and S. W. Hell, “Equivalence of the Huygens–Fresnel and Debye approach for the calculation of high aperture point-spread functions in the presence of refractive index mismatch,” Journal of Microscopy193, 244–249 (1999).
[CrossRef]

M. Levoy, Z. Zhang, and I. McDowell, “Recording and controlling the 4D light field in a microscope using microlens arrays,” Journal of Microscopy235, 144–162 (2009).
[CrossRef] [PubMed]

Micron

R. Heintzmann, “Estimating missing information by maximum likelihood deconvolution,” Micron38, 136–144 (2007)
[CrossRef]

Multidimensional Systems and Signal Processing

W. Chan, E. Lam, M. Ng, and G. Mak, “Super-resolution reconstruction in a computational compound-eye imaging system,” Multidimensional Systems and Signal Processing18. 83–101. (2007).
[CrossRef]

Nat. Meth.

S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, X. Darzacq, C. Wu, C. I. Bargmann, D. A. Agard, M. G. L. Gustafsson, and M. Dahan, “Fast multicolor 3D imaging using aberration-corrected multifocus microscopy,” Nat. Meth.1–6. (2012).

Opt. Express

J. Rosen, N. Siegel, and G. Brooker, “Theoretical and experimental demonstration of resolution beyond the Rayleigh limit by FINCH fluorescence microscopic imaging,” Opt. Express19, 1506–1508 (2011).
[CrossRef]

Optics communications

M. R. Arnison and C. J. R. Sheppard, “A 3D vectorial optical transfer function suitable for arbitrary pupil functions,” Optics communications211, 53–63 (2002).
[CrossRef]

Proc. SPIE

T. Pham, L. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE5672, 169–180 (2005).
[CrossRef]

Proceedings of ACM SIGGRAPH

M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH. (2006) 924–934.
[CrossRef]

R. Ng, “Fourier slice photography,” in Proceedings of ACM SIGGRAPH(2005). 735–744.
[CrossRef]

SIAM journal on matrix analysis and applications

J. M. Bardsley and J. G. Nagy, “Covariance-preconditioned iterative methods for nonnegatively constrained astronomical imaging,” SIAM journal on matrix analysis and applications27, 1184–1197 (2006).
[CrossRef]

SPIE Conference Series

P. Favaro, “A split-sensor light field camera for extended depth of field and superresolution,” in “SPIE Conference Series,” 8436. (2012).

Other

C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in “Computational Optical Sensing and Imaging, Microscopy and Tomography I (CTh3B),” (2013).

M. Pluta, Advanced Light Microscopy, Vol. 1.(Elsevier, 1988).

J. Goodman, Introduction to Fourier Optics, 2nd ed. (MaGraw-Hill, 1996).

J. Breckinridge, D. Voelz, and J. B. Breckinridge, Computational Fourier Optics: a MATLAB Tutorial (SPIE Press, 2011).

M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999).

M. Bertero and C. de Mol, “III Super-resolution by data inversion,” in Progress in Optics (Elsevier, 1996) pp. 129–178.
[CrossRef]

M. Gu, Advanced Optical Imaging Theory (Springer, 1999).

K. Grochenig and T. Strohmer, “Numerical and theoretical aspects of nonuniform sampling of band-limited images,” in “Nonuniform Sampling,” F. Marvasti, ed.. Information Technology: Transmission, Processing, and Storage, 283–324 (SpringerUS, 2010).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1
Fig. 1

USAF 1951 resolution test target translated to depths below the native object plane (z = 0 μm) and imaged using a light field microscope with a 20× 0.5NA water-dipping objective. (a) Photographs taken with a conventional microscope as the target is translated to the z-heights denoted below each image. (b) Computational re-focusing using our 2009 method [2] while the microscope was defocused to the same heights as (a). While some computational refocusing is possible, there has been a significant loss of lateral resolution. (c) The reconstruction algorithm presented in this paper brings the target back into focus, achieving up to an 8-fold improvement in lateral resolution compared to (b) except at the native object plane (left image).

Fig. 2
Fig. 2

Optical model of the light field microscope. (a) A fluorescence microscope can be converted into a light field microscope by placing a microlens array at the native image plane. (b) A ray-optics schematic indicating the pattern of illumination generated by one point source. The gray grid delineates pixel locations, and the white circles depict the back-aperture of the objective imaged onto the sensor by each lenslet. Red level indicates the intensity of illumination arriving at the sensor. For a point source on the native object plane (red dot), all rays pass through a single lenslet. (c) A point source below the native object plane generates a more complicated intensity pattern involving many lenslets. (d) A real light field recorded on our microscope using a 60× 1.4NA oil objective and a 125 μm pitch f/20 microlens array of a 0.5 μm fluorescent bead placed 4 μm below the native object plane. Diffraction effects are present in the images formed behind each lenslet. (e) A schematic of our wave optics model based on the LFM optical path (not drawn to scale). In this model, the microlens array at the native image plane is modeled as a tiled phase mask operating on this wavefront, which is then propagated to the camera sensor. The xy cross section on the far right shows the simulated light field generated at the sensor plane. The simulation is in good agreement with the experimentally measured light field in (d).

Fig. 3
Fig. 3

Sampling of the volume recorded in a microlens-based light field. (a) A bundle of lenslet chief rays captured by the same pixel position relative to each lenslet (blue pixels) form a parallel projection through the volume, providing one of many angular views necessary to perform 3-D deconvolution. (b) When lenslet chief rays passing through every pixel in the light field are simultaneously projected back into the object volume, these rays cross at a diversity of x-positions (readers are encouraged to zoom into this figure in a PDF file to see how this pattern evolves with depth). This dense sampling pattern permits 3-D deconvolution with resolution finer than the lenslet spacing. The only place where this diversity does not occur is close to the native object plane; here resolution enhancement is not possible. (c) The distribution of the lenslet chief rays in xy cross-sections of the object volume changes at different distances from the native object plane. The outline of the lenslets are shown in light gray for scale. At z = 0 μm (rightmost image), the lack of spatial diversity in sample locations is evident.

Fig. 4
Fig. 4

The discrete light field imaging model, prior to accounting for sensor noise. The dimensionality of the light field f is fixed by the image sensor, but the dimensionality (or equivalently, the sampling rate) of the volume g is a user-selectable parameter in the reconstruction algorithm. (a) Column i of the measurement matrix H (purple) contains the discretized light field point spread function for voxel i, which corresponds to the forward projection of that point in the volume. (b) Row j of the measurement matrix (green) contains a pixel back projection: a visualization that shows how much each voxel in the volume contributes to a single pixel j in the light field. The cross sections in this figure were computed using our wave optics model for a 20× 0.5NA water dipping objective and a 125μm pitch f/20 microlens array.

Fig. 5
Fig. 5

Experimentally characterized resolution limits for two optical configurations of the light field microscope. (a) In a widefield microscope with no lenslet array, the target quickly goes out of focus when it is translated in z. (b) In a 3-D deconvolution from a light field, we lose resolution if the test target is placed at the native object plane (z = 0 μm), but we can reconstruct the target at much higher resolution than the spacing between lenslets when it is moved z = −15 μm (see also Fig. 1). Resolution falls off gradually beyond this depth (z = ±50 μm and ±100 μm). (c) Experimental MTF measured by analyzing the contrast of different line pair groupings in the USAF reconstruction. The colormap shows normalized contrast as measured using Eq. (10). The region of fluctuating resolution from z = −30 μm to 30 μm show that not all spatial frequencies are equally well reconstructed at all depths. (d) A slightly higher peak resolution (z = ±10 μm) can be achieved in the light field recorded with a 40× 0.8NA objective. However, the z = ±25 μm and ±50 μm planes in (d) have the same apparent resolution as the z = ±50 μm and ±100 μm planes in (b). (e) The experimental MTF for the 40× configuration shows that the region of fluctuating resolution (from −7.5 μm to 7.5 μm) is one quarter the size compared to (c). The solid green line in (c) and orange line in (e) are a 10% contrast cut-off representing the band limit of the reconstruction as a function of depth. Note that these plots are clipped to 645 cycles/mm, which is the highest resolution group on the USAF target.

Fig. 6
Fig. 6

Experimentally measured band-limits from Fig. 5 re-plotted along with the lenslet sampling rate (dotted black line) and the Nyquist sampling rate at the diffraction limit of a widefield fluorescence microscope (dashed blue line). For comparison, we have plotted the depth of field of a widefield microscope (thin blue lines). The theoretical band limit we propose in Eq. 11 (dotted purple line) is in good agreement with the experimental band limit. However, this criterion only predicts the resolution fall-off at moderate to large z-positions, and not near the native object plane where diffraction and sampling effects cause the band limit to fluctuate.

Fig. 7
Fig. 7

Lateral resolution fall-off as a function of depth for various optical design choices. (a) For a fixed 125μm pitch lenslet array, a larger objective magnification results in better peak resolution, but a more rapid fall-off and hence a diminished axial range over which good resolution can be achieved. (b) For a fixed magnification factor, decreasing the lenslet pitch achieves the same trade-off as in (a). In fact, Eq. (11) shows that multiplying the lenslet pitch by some constant has the same effect on the resolution criterion as dividing the objective magnification by that same amount.

Fig. 8
Fig. 8

Pollen grains imaged with a 60× 1.4NA oil dipping objective and a 125 μm f/20 microlens array. (a) Max-intensity projections of a volume reconstructed using the computational refocusing algorithm presented in our 2009 paper [2] shows the irregular shape of the pollen grains, but low contrast and little high frequency detail. (b) Maximum intensity projections of a volume reconstructed using the 3-D deconvolution algorithm introduced in this paper shows the structures of the pollen grain more clearly. A small region of reconstruction artifacts appears around the native object plane. (c) Individual xy slices from the computational refocusing algorithm. (d) Slices from the 3-D deconvolved volume. A gamma correction of 0.6 was applied to panels (a), (b), (c), and (d) to help visualize their full dynamic range. After gamma correction, each panel was separately normalized so that the 99% of the intensity range was represented by the colormap shown by the scale bar.

Fig. 9
Fig. 9

Geometric construction of the light field resolution criterion introduced in Eq. (11). In these figures conjugate images of the microlens array and image sensor are depicted on the object side of the microscope. Taking the magnification factor into account, the effective object-side pitch of the conjugated lenslet is d/M and its effective focal length is fμlens/M2. (a) The resolution criterion holds when point sources p1 and p2 are at sufficient depth |z| that they form diffraction limited spots on the sensor. Under this condition, the discriminability of the spots can be determined by simple geometric construction using similar triangles. Here, c is a constant that allows us to specify a particular resolution criterion (e.g., c = 1.22 selects the Rayleigh 2-point criterion, and c = 0.94 selects the Sparrow 2-point criterion [26, p. 340]). In this paper, we use the Sparrow 2-point criterion since it is best suited for measuring where contrast drops to zero between two point projections measured in a digital image [26, p. 340]. (b) To see where this resolution criterion holds, we measure the diameter b of the blur disk predicted using geometric optics. Our approximation is valid when this diameter is less than the diameter of a diffraction-limited spot (i.e. when b < /NA).

Equations (17)

Equations on this page are rendered with MathJax. Learn more.

f = H g ,
f ( x ) = | h ( x , p ) | 2 g ( p ) d p ,
U i ( x , p ) = M f obj 2 λ 2 exp ( i u 4 sin 2 ( α / 2 ) ) 0 α P ( θ ) exp ( i u sin 2 ( θ / 2 ) 2 sin 2 ( α / 2 ) ) J 0 ( sin ( θ ) sin ( α ) v ) sin ( θ ) d θ
v k ( x 1 p 1 ) 2 + ( x 2 p 2 ) 2 sin ( α ) u 4 k p 3 sin 2 ( α / 2 ) .
ϕ ( x ) = rect ( x / d ) exp ( i k 2 f μ lens x 2 2 ) .
Φ ( x ) = ϕ ( x ) * comb ( x / d ) .
h ( x , p ) = 1 { { Φ ( x ) U i ( x , p ) } exp [ i 4 π λ f μ lens ( ω x 2 + ω y 2 ) ] } ,
h i j = α j β i w i ( p ) | h ( x , p ) | 2 d p d x
f ^ ~ Pois ( H g + b ) ,
Pr ( f ^ | g , b ) = i ( ( H g + b ) i f ^ i exp ( ( H g + b ) i ) f ^ i ! ) ,
g ( k + 1 ) = diag ( H T 1 ) 1 diag ( H T diag ( H g ( k ) + b ) 1 f ) g ( k ) ,
C thr = ( I max I min ) / ( I max + I min ) ,
ν lf ( z ) = d 0.94 λ M | z | .
r | z | = c λ 2 NA M 2 f μ lens .
f μ lens = d M 2 NA .
ν lf ( z ) = d c λ M | z | .
b < c λ NA .

Metrics