Abstract

High-end lenses are usually composed of many optical elements to compensate various optical aberrations, e.g. geometric distortion, monochromatic and chromatic aberrations. The resulting complexity and machining accuracy requirements make high-end lenses too expensive, heavy, and fragile for day-to-day photography. To address this problem, we devised an optical computing approach to touch-up the low quality photos produced by simpler lenses. We propose a setup consisting of an easily accessible display and the original camera in order to perform optical aberration correction with a deconvolution framework. The equivalence of the degeneration model and the lens’s optical computing turns the traditional blind deconvolution algorithm into its non-blind counterpart and promises robust performance. A prototype system is implemented to verify the feasibility of the proposed method, and a series of experiments on both synthetic and captured images are applied to demonstrate effectiveness and performance.

© 2014 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Computational imaging using lightweight diffractive-refractive optics

Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich
Opt. Express 23(24) 31393-31407 (2015)

Satellite image restoration in the context of a spatially varying point spread function

Nasreddine Hajlaoui, Caroline Chaux, Guillaume Perrin, Frédéric Falzon, and Amel Benazza-Benyahia
J. Opt. Soc. Am. A 27(6) 1473-1481 (2010)

Deblurring adaptive optics retinal images using deep convolutional neural networks

Xiao Fei, Junlei Zhao, Haoxin Zhao, Dai Yun, and Yudong Zhang
Biomed. Opt. Express 8(12) 5675-5687 (2017)

References

  • View by:
  • |
  • |
  • |

  1. V. N. Mahajan and V. N. Mahajan, Aberration theory made simple (SPIE optical engineering press, Bellingham, Washington, USA, 1991).
    [Crossref]
  2. E. Logean, E. Dalimier, and C. Dainty, “Measured double-pass intensity point-spread function after adaptive optics correction of ocular aberrations,” Opt. Express 16, 17348–17357 (2008).
    [Crossref] [PubMed]
  3. H. Song, R. Fraanje, G. Schitter, H. Kroese, G. Vdovin, and M. Verhaegen, “Model-based aberration correction in a closed-loop wavefront-sensor-less adaptive optics system,” Opt. Express 18, 24070–24084 (2010).
    [Crossref] [PubMed]
  4. L. Mugnier, J.-F. Sauvage, T. Fusco, A. Cornia, and S. Dandy, “On-line long-exposure phase diversity: a powerful tool for sensing quasi-static aberrations of extreme adaptive optics imaging systems,” Opt. Express 16, 18406–18416 (2008).
    [Crossref] [PubMed]
  5. P. Bedggood, R. Ashman, G. Smith, and A. Metha, “Multiconjugate adaptive optics applied to an anatomically accurate human eye model,” Opt. Express 14, 8019–8030 (2006).
    [Crossref] [PubMed]
  6. K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
    [Crossref] [PubMed]
  7. F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
    [Crossref]
  8. M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.
  9. E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).
  10. T. Vettenburg and A. R. Harvey, “Holistic optical-digital hybrid-imaging design:wide-field reflective imaging,” Appl. Opt. 52, 3931–3936 (2013).
    [Crossref] [PubMed]
  11. Q. Luo, L. Huang, N. Gu, and C. Rao, “Experimental study of a modified phase diversity with a diffraction grating,” Opt. Express 20, 12059–12066 (2012).
    [Crossref] [PubMed]
  12. G. R. Brady, M. Guizar-Sicairos, and J. R. Fienup, “Optical wavefront measurement using phase retrieval with transverse translation diversity,” Opt. Express 17, 624–639 (2009).
    [Crossref] [PubMed]
  13. N. Joshi, R. Szeliski, and D. J. Kriegman, “PSF estimation using sharp edge prediction,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2008).
  14. Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations,” in European Conference on Computer Vision, (Springer, 2012), pp. 42–56.
  15. C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.
  16. S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
    [Crossref]
  17. R. D. Nowak and M. A. Figueiredo, “Fast wavelet-based image deconvolution using the EM algorithm,” in Asilomar Conference on Signals, Systems, and Computers, (2001), pp. 371–375.
  18. M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE T. Image Process. 12, 906–916 (2003).
    [Crossref]
  19. J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
    [Crossref]
  20. J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
    [Crossref]
  21. C. Vonesch and M. Unser, “A fast thresholded landweber algorithm for wavelet-regularized multidimensional deconvolution,” IEEE T. Image Process 17, 539–549 (2008).
    [Crossref]
  22. D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2011).
  23. A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
    [Crossref]
  24. R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
    [Crossref]
  25. Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” ACM T. Graphic 27, 73 (2008).
    [Crossref]
  26. N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).
  27. D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011).
  28. R. A. Athale and W. C. Collins, “Optical matrix-matrix multiplier based on outer product decomposition,” Appl. Opt. 21, 2089–2090 (1982).
    [Crossref] [PubMed]
  29. A. Dias, “Incoherent optical matrix-matrix multiplier,” NASA. Langley Research Center Opt. Inform. Process. for Aerospace Appl. 1, 71–83 (1981).
  30. R. A. Heinz, J. O. Artman, and S. H. Lee, “Matrix multiplication by optical methods,” Appl. Opt. 9, 2161–2168 (1970).
    [Crossref] [PubMed]
  31. P. Guilfoyle and R. Stone, “Digital optical computer II,” Proc. SPIE 1563, 214 (1991).
    [Crossref]
  32. H. Huang, L. Liu, and Z. Wang, “Parallel multiple matrix multiplication using an orthogonal shadow-casting and imaging system,” Opt. Lett. 15, 1085–1087 (1990).
    [Crossref] [PubMed]
  33. A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
    [Crossref]
  34. M. O’Toole and K. Kutulakos, “Optical computing for fast light transport analysis,” ACM T. Graphic 29, 164 (2010).
  35. D. Lefebvre, H. Arsenault, and S. Roy, “Nonlinear filter for pattern recognition invariant to illumination and to out-of-plane rotations,” Appl. Opt. 42, 4658–4662 (2003).
    [Crossref] [PubMed]
  36. F. Yu, S. Jutamulia, T. Lin, and D. Gregory, “Adaptive real-time pattern recognition using a liquid crystal TV based joint transform correlator,” Appl. Opt. 26, 1370–1372 (1987).
    [Crossref] [PubMed]
  37. P. Ambs, “A short history of optical computing: rise, decline, and evolution,” Proc. SPIE 7388, 73880H (2009).
    [Crossref]
  38. E. Leith, “The evolution of information optics,” IEEE J. Sel. Top Quant. 6, 1297–1304 (2000).
    [Crossref]
  39. C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.
  40. A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,” in Australasian Conference on Robotics and Automation, (ARAA, 2010).
  41. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
    [Crossref]

2014 (1)

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

2013 (2)

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

T. Vettenburg and A. R. Harvey, “Holistic optical-digital hybrid-imaging design:wide-field reflective imaging,” Appl. Opt. 52, 3931–3936 (2013).
[Crossref] [PubMed]

2012 (1)

2010 (2)

2009 (2)

2008 (4)

2007 (1)

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

2006 (2)

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

P. Bedggood, R. Ashman, G. Smith, and A. Metha, “Multiconjugate adaptive optics applied to an anatomically accurate human eye model,” Opt. Express 14, 8019–8030 (2006).
[Crossref] [PubMed]

2005 (1)

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

2004 (1)

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

2003 (4)

D. Lefebvre, H. Arsenault, and S. Roy, “Nonlinear filter for pattern recognition invariant to illumination and to out-of-plane rotations,” Appl. Opt. 42, 4658–4662 (2003).
[Crossref] [PubMed]

M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE T. Image Process. 12, 906–916 (2003).
[Crossref]

J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
[Crossref]

J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
[Crossref]

2000 (1)

E. Leith, “The evolution of information optics,” IEEE J. Sel. Top Quant. 6, 1297–1304 (2000).
[Crossref]

1997 (1)

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

1991 (1)

P. Guilfoyle and R. Stone, “Digital optical computer II,” Proc. SPIE 1563, 214 (1991).
[Crossref]

1990 (1)

1987 (1)

1982 (1)

1981 (1)

A. Dias, “Incoherent optical matrix-matrix multiplier,” NASA. Langley Research Center Opt. Inform. Process. for Aerospace Appl. 1, 71–83 (1981).

1970 (1)

Agarwala, A.

Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” ACM T. Graphic 27, 73 (2008).
[Crossref]

Albeck, Y.

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Ambs, P.

P. Ambs, “A short history of optical computing: rise, decline, and evolution,” Proc. SPIE 7388, 73880H (2009).
[Crossref]

Arsenault, H.

Artman, J. O.

Ashman, R.

Athale, R. A.

Bedggood, P.

Benchowski, J.

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Betzig, E.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Bovik, A. C.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

Brady, G. R.

Bronner, M. E.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Burger, M.

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

Candès, E.

J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
[Crossref]

Chen, S.

E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).

Collins, W. C.

Cornia, A.

Dainty, C.

Dalimier, E.

Dandy, S.

Dias, A.

A. Dias, “Incoherent optical matrix-matrix multiplier,” NASA. Langley Research Center Opt. Inform. Process. for Aerospace Appl. 1, 71–83 (1981).

Donoho, D.

J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
[Crossref]

Durand, F.

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

Engerer, P.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Fergus, R.

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2011).

Fienup, J. R.

Figueiredo, M.

M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE T. Image Process. 12, 906–916 (2003).
[Crossref]

Figueiredo, M. A.

R. D. Nowak and M. A. Figueiredo, “Fast wavelet-based image deconvolution using the EM algorithm,” in Asilomar Conference on Signals, Systems, and Computers, (2001), pp. 371–375.

Fraanje, R.

Freeman, W. T.

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

Fusco, T.

Goldfarb, D.

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

Gregory, D.

Gu, N.

Guenter, B.

Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations,” in European Conference on Computer Vision, (Springer, 2012), pp. 42–56.

Guilfoyle, P.

P. Guilfoyle and R. Stone, “Digital optical computer II,” Proc. SPIE 1563, 214 (1991).
[Crossref]

Guizar-Sicairos, M.

Harmeling, S.

M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.

Harvey, A. R.

Heide, F.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Heidrich, W.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Heinz, R. A.

Hertzmann, A.

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

Hirsch, M.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.

M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.

Huang, H.

Huang, L.

Hullin, M. B.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Jia, J.

Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” ACM T. Graphic 27, 73 (2008).
[Crossref]

Joshi, N.

N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).

Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations,” in European Conference on Computer Vision, (Springer, 2012), pp. 42–56.

N. Joshi, R. Szeliski, and D. J. Kriegman, “PSF estimation using sharp edge prediction,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2008).

Jutamulia, S.

Kassir, A.

A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,” in Australasian Conference on Robotics and Automation, (ARAA, 2010).

Kee, E.

E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).

Kolb, A.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Kriegman, D. J.

N. Joshi, R. Szeliski, and D. J. Kriegman, “PSF estimation using sharp edge prediction,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2008).

N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).

Krishnan, D.

D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2011).

Kroese, H.

Kutulakos, K.

M. O’Toole and K. Kutulakos, “Optical computing for fast light transport analysis,” ACM T. Graphic 29, 164 (2010).

Labitzke, B.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Lange, Z.

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Lee, S. H.

Lefebvre, D.

Leith, E.

E. Leith, “The evolution of information optics,” IEEE J. Sel. Top Quant. 6, 1297–1304 (2000).
[Crossref]

Levin, A.

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

Lewis, A.

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Lin, T.

Liu, L.

Logean, E.

Luo, Q.

Mahajan, V. N.

V. N. Mahajan and V. N. Mahajan, Aberration theory made simple (SPIE optical engineering press, Bellingham, Washington, USA, 1991).
[Crossref]

V. N. Mahajan and V. N. Mahajan, Aberration theory made simple (SPIE optical engineering press, Bellingham, Washington, USA, 1991).
[Crossref]

Metha, A.

Milkie, D. E.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Misgeld, T.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Mugnier, L.

Mumm, J.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Murtagh, F.

J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
[Crossref]

Nguyen, M.

J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
[Crossref]

Nowak, R.

M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE T. Image Process. 12, 906–916 (2003).
[Crossref]

Nowak, R. D.

R. D. Nowak and M. A. Figueiredo, “Fast wavelet-based image deconvolution using the EM algorithm,” in Asilomar Conference on Signals, Systems, and Computers, (2001), pp. 371–375.

O’Toole, M.

M. O’Toole and K. Kutulakos, “Optical computing for fast light transport analysis,” ACM T. Graphic 29, 164 (2010).

Osher, S.

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

Paris, S.

E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).

Peynot, T.

A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,” in Australasian Conference on Robotics and Automation, (ARAA, 2010).

Rao, C.

Rouf, M.

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

Roweis, S. T.

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

Roy, S.

Sauvage, J.-F.

Saxena, A.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Schitter, G.

Scholkopf, B.

M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.

Schölkopf, B.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.

Schuler, C. J.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.

Shan, Q.

Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” ACM T. Graphic 27, 73 (2008).
[Crossref]

Sheikh, H. R.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

Shih, Y.

Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations,” in European Conference on Computer Vision, (Springer, 2012), pp. 42–56.

Simoncelli, E. P.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

Singh, B.

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

Smith, G.

Song, H.

Sra, S.

M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.

Starck, J.

J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
[Crossref]

J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
[Crossref]

Stone, R.

P. Guilfoyle and R. Stone, “Digital optical computer II,” Proc. SPIE 1563, 214 (1991).
[Crossref]

Szeliski, R.

N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).

N. Joshi, R. Szeliski, and D. J. Kriegman, “PSF estimation using sharp edge prediction,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2008).

Tay, T.

D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2011).

Unser, M.

C. Vonesch and M. Unser, “A fast thresholded landweber algorithm for wavelet-regularized multidimensional deconvolution,” IEEE T. Image Process 17, 539–549 (2008).
[Crossref]

Vdovin, G.

Verhaegen, M.

Vettenburg, T.

Vonesch, C.

C. Vonesch and M. Unser, “A fast thresholded landweber algorithm for wavelet-regularized multidimensional deconvolution,” IEEE T. Image Process 17, 539–549 (2008).
[Crossref]

Wang, J.

E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).

Wang, K.

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Wang, Z.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

H. Huang, L. Liu, and Z. Wang, “Parallel multiple matrix multiplication using an orthogonal shadow-casting and imaging system,” Opt. Lett. 15, 1085–1087 (1990).
[Crossref] [PubMed]

Weiss, Y.

D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011).

Weizman, G.

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Xu, J.

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

Yin, W.

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

Yu, F.

Zitnick, C. L.

N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).

Zoran, D.

D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011).

ACM T. Graphic (5)

F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM T. Graphic 32, 149 (2013).
[Crossref]

A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” ACM T. Graphic 26, 70 (2007).
[Crossref]

R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph,” ACM T. Graphic 25, 787–794 (2006).
[Crossref]

Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” ACM T. Graphic 27, 73 (2008).
[Crossref]

M. O’Toole and K. Kutulakos, “Optical computing for fast light transport analysis,” ACM T. Graphic 29, 164 (2010).

Appl. Opt. (5)

Astronomy and Astrophysics (1)

J. Starck, D. Donoho, and E. Candès, “Astronomical image representation by the curvelet transform,” Astronomy and Astrophysics 398, 785–800 (2003).
[Crossref]

IEEE J. Sel. Top Quant. (1)

E. Leith, “The evolution of information optics,” IEEE J. Sel. Top Quant. 6, 1297–1304 (2000).
[Crossref]

IEEE T. Image Process (2)

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE T. Image Process 13, 600–612 (2004).
[Crossref]

C. Vonesch and M. Unser, “A fast thresholded landweber algorithm for wavelet-regularized multidimensional deconvolution,” IEEE T. Image Process 17, 539–549 (2008).
[Crossref]

IEEE T. Image Process. (1)

M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE T. Image Process. 12, 906–916 (2003).
[Crossref]

Multiscale Modeling & Simulation (1)

S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation 4, 460–489 (2005).
[Crossref]

NASA. Langley Research Center Opt. Inform. Process. for Aerospace Appl. (1)

A. Dias, “Incoherent optical matrix-matrix multiplier,” NASA. Langley Research Center Opt. Inform. Process. for Aerospace Appl. 1, 71–83 (1981).

Nature Methods (1)

K. Wang, D. E. Milkie, A. Saxena, P. Engerer, T. Misgeld, M. E. Bronner, J. Mumm, and E. Betzig, “Rapid adaptive optical recovery of optimal resolution over large volumes,” Nature Methods 11, 625–628 (2014).
[Crossref] [PubMed]

Opt. Express (6)

Opt. Lett. (1)

Proc. SPIE (2)

P. Guilfoyle and R. Stone, “Digital optical computer II,” Proc. SPIE 1563, 214 (1991).
[Crossref]

P. Ambs, “A short history of optical computing: rise, decline, and evolution,” Proc. SPIE 7388, 73880H (2009).
[Crossref]

Science (1)

A. Lewis, Y. Albeck, Z. Lange, J. Benchowski, and G. Weizman, “Optical computation with negative light intensity with a plastic bacteriorhodopsin film,” Science 275, 1462–1464 (1997).
[Crossref]

Signal Processing (1)

J. Starck, M. Nguyen, and F. Murtagh, “Wavelets and curvelets for image deconvolution: a combined approach,” Signal Processing 83, 2279–2283 (2003).
[Crossref]

Other (12)

R. D. Nowak and M. A. Figueiredo, “Fast wavelet-based image deconvolution using the EM algorithm,” in Asilomar Conference on Signals, Systems, and Computers, (2001), pp. 371–375.

N. Joshi, R. Szeliski, and D. J. Kriegman, “PSF estimation using sharp edge prediction,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2008).

Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations,” in European Conference on Computer Vision, (Springer, 2012), pp. 42–56.

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schölkopf, “Blind correction of optical aberrations,” in European Conference on Computer Vision, (Springer, 2012), pp. 187–200.

V. N. Mahajan and V. N. Mahajan, Aberration theory made simple (SPIE optical engineering press, Bellingham, Washington, USA, 1991).
[Crossref]

M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filter flow for space-variant multiframe blind deconvolution,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 607–614.

E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur,” in Proceedings of IEEE International Conference on Computational Photography, (IEEE, 2011).

D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2011).

N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman, “Image deblurring and denoising using color priors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009).

D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011).

C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Non-stationary correction of optical aberrations,” in Proceedings of IEEE International Conference on Computer Vision, (IEEE, 2011), pp. 659–666.

A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,” in Australasian Conference on Robotics and Automation, (ARAA, 2010).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1
Fig. 1 Prototype and results for image enhancement. (a) The setup. (b) Low-quality input image. (c) Aberration-compensated result.
Fig. 2
Fig. 2 Diagram for calculating the gradient gd in Eq. 7. We use red and blue outlines to highlight two types of convolutions, i.e. ∑m,n K(i, j, m, n)(·) and ∑i,j K(i, j, m, n)(·), respectively.
Fig. 3
Fig. 3 Optical computing for convolution modules. (a) Forward convolution— ∑m,n K(i, j, m, n)(·). (b) Backward convolution—∑i,j K(i, j, m, n)(·).
Fig. 4
Fig. 4 Steps for approximating backward convolution using forward operation.
Fig. 5
Fig. 5 Geometric (a–d) and photometric calibration (e,f) in the blue channel. (a) Original chessboard pattern. (b) Captured image. (c) Warping vectors of landmarks for interpolation. (d) Geometrically calibrated pattern. (e) Ratio image for dark-corner correction of the blue channel. (f) The chessboard pattern after correcting the geometry and dark corner distortions.
Fig. 6
Fig. 6 Cross-channel response calibration. (a) Response curves of Point Grey FL3-U3-13S2C-CS. (b) Blue channel gradation map for calibrating the linear component C and nonlinear component fr,g,b. (c) Response curves of r, g, b channels vs. input channel b. (d) and (e) An image before and after cross-channel response calibration.
Fig. 7
Fig. 7 The results of our approach with different patch sizes. (a) The convergence curves of our method with backward convolution conducted by accurate numerical calculation and patch-wise flip–FC–flip approximations with different patch sizes. (b)(c) Restoration results by accurate numerical calculation and the proposed approximation with a 60-pixel wide patch size.
Fig. 8
Fig. 8 The error map of our flip–FC–flip approximation method. (a) The ground-truth sharp image. (b) The local PSFs for the synthetic single lens camera. (c) Simulated image contaminated by optical aberrations (single lens). (d) Ground-truth image after backward convolution. (e) The approximate backward convolution result of our flip–FC–flip method. (f) The error map of our approximation.
Fig. 9
Fig. 9 The results of our approach with different display–camera distances. (a)(b) and (d)(e) show the captured degenerated images and corresponding PSFs with object distance at 40 cm and 4 m, respectively. (c)(f) display the results restoring the degenerated image in (a)(d) using the optical system with display–camera distance at 4 m and 40 cm, respectively.
Fig. 10
Fig. 10 Restoration performance with different regularization weights and noise levels. (a) SSIM curves with varying regularization weights and noise variances. (b) Aberration-corrupted images with noise variance at 18 (image intensity ranges from 0 to 255). (c) Our restoration result by setting the regularization weight to 18.
Fig. 11
Fig. 11 Intermediate results of the first several iterations.
Fig. 12
Fig. 12 More results for three different lenses.

Equations (13)

Equations on this page are rendered with MathJax. Learn more.

B ( i , j ) = m , n L ( m , n ) K ( i , j , m , n ) + N ( i , j ) ,
E d = i , j ( m , n L ( m , n ) K ( i , j , m , n ) B ( i , j ) ) 2 ,
E w ( L ) = 𝒲 ( L ) 1 ,
E = E d + λ E w ,
L g t ( i , j ) = L t ( i , j ) τ g d ( i , j ) ;
L t + 1 ( i , j ) = 𝒯 λ τ / 2 ( L g t ( i , j ) ) .
g d ( m , n ) = 2 i , k K ( i , j , m , n ) m , n K ( i , j , m , n ) L ( m , n ) 2 i , j K ( i , j , m , n ) B ( i , j ) .
FC ( L ) ( i , j ) = m , n L ( m , n ) K ( i , j , m , n ) .
BC ( L ) ( m , n ) = i , j L ( i , j ) K ( i , j , m , n ) = Δ i , Δ j K ( x 0 + Δ i , y 0 + Δ j , x 0 + Δ m , y 0 + Δ n ) B ( x 0 + Δ i , y 0 + Δ j ) ,
K ( i , j , m , n ) K ( i + Δ x , j + Δ y , m + Δ x , n + Δ y ) .
BC ( B ) ( x 0 + Δ m , y 0 + Δ n ) Δ i , Δ j K ( x 0 Δ m , y 0 Δ n , x 0 Δ i , y 0 Δ j ) B ( x 0 + Δ i , y 0 + Δ j ) .
BC ( B ) ( x 0 + Δ m , y 0 + Δ n ) F C ( B ) ( x 0 Δ m , y 0 Δ n ) ,
[ r g b ] = [ C r , r C g , r C b , r C r , g C g , g C b , g C r , b C g , b C b , b ] [ f r ( r ) f g ( g ) f b ( b ) ] .

Metrics