Abstract

Under strong turbulence conditions, object’s images can be severely distorted and become unrecognizable throughout the observing time. Conventional image restoring algorithms do not perform effectively in these circumstances due to the loss of good references on the object. We propose the use a plenoptic sensor as a light field camera to map a conventional camera image onto a cell image array in the image’s sub-angular spaces. Accordingly, each cell image on the plenoptic sensor is equivalent to the image acquired by a sub-aperture of the imaging lens. The wavefront distortion over the lens aperture can be analyzed by comparing cell images in the plenoptic sensor. By using a modified “Laplacian” metric, we can identify a good cell image in a plenoptic image sequence. The good cell image corresponds with the time and sub-aperture area on the imaging lens where wavefront distortion becomes relatively and momentarily “flat”. As a result, it will reveal the fundamental truths of the object that would be severely distorted on normal cameras. In this paper, we will introduce the underlying physics principles and mechanisms of our approach and experimentally demonstrate its effectiveness under strong turbulence conditions. In application, our approach can be used to provide a good reference for conventional image restoring approaches under strong turbulence conditions. This approach can also be used as an independent device to perform object recognition tasks through severe turbulence distortions.

© 2016 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Comparison between the plenoptic sensor and the light field camera in restoring images through turbulence

Chensheng Wu, Daniel A. Paulson, John R. Rzasa, and Christopher C. Davis
OSA Continuum 2(9) 2511-2525 (2019)

Research on high-resolution imaging technology based on light field manipulation for a lenslet-based plenoptic camera

Xincheng Liu, Haotong Ma, Ge Ren, Bo Qi, Zongliang Xie, Junqiu Chu, and Junjie Bai
Appl. Opt. 57(33) 9877-9886 (2018)

Plenoptic mapping for imaging and retrieval of the complex field amplitude of a laser beam

Chensheng Wu, Jonathan Ko, and Christopher C. Davis
Opt. Express 24(26) 29852-29871 (2016)

References

  • View by:
  • |
  • |
  • |

  1. K. T. Knox and B. J. Thompson, “Recovery of images from atmospherically degraded short-exposure photographs,” Astrophys. J. 193, L45–L48 (1974).
    [Crossref]
  2. X. Zhu and P. Milanfar, “Removing atmospheric turbulence via space-invariant deconvolution,” IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013).
    [Crossref] [PubMed]
  3. M. C. Roggemann, B. M. Welsh, and B. R. Hunt, Imaging through turbulence (CRC press, 1996).
  4. G. R. Ayers and J. C. Dainty, “Interative blind deconvolution method and its applications,” Opt. Lett. 13(7), 547–549 (1988).
    [Crossref] [PubMed]
  5. D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
    [Crossref]
  6. N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
    [Crossref]
  7. M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
    [Crossref]
  8. A. V. Kanaev, W. Hou, S. R. Restaino, S. Matt, and S. Gładysz, “Restoration of images degraded by underwater turbulence using structure tensor oriented image quality (STOIQ) metric,” Opt. Express 23(13), 17077–17090 (2015).
    [Crossref] [PubMed]
  9. E. Chen, O. Haik, and Y. Yitzhaky, “Detecting and tracking moving objects in long-distance imaging through turbulent medium,” Appl. Opt. 53(6), 1181–1190 (2014).
    [Crossref] [PubMed]
  10. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Computer Science Technical Report 2, no. 11 (2005): 1–11.
  11. T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
    [Crossref]
  12. C. Wu, J. Ko, and C. C. Davis, “Determining the phase and amplitude distortion of a wavefront using a plenoptic sensor,” J. Opt. Soc. Am. A 32(5), 964–978 (2015).
    [Crossref] [PubMed]
  13. D. L. Fried, “Optical resolution through a randomly inhomogeneous medium for very long and very short exposures,” J. Opt. Soc. Am. 56(10), 1372–1379 (1966).
    [Crossref]
  14. A. Zilberman, E. Golbraikh, and N. S. Kopeika, “Propagation of electromagnetic waves in Kolmogorov and non-Kolmogorov atmospheric turbulence: three-layer altitude model,” Appl. Opt. 47(34), 6385–6391 (2008).
    [Crossref] [PubMed]
  15. C. Wu, J. Ko, and C. Davis, “Object recognition through turbulence with a modified plenoptic camera,” Proc. SPIE 9354, 93540V (2015).
  16. J. W. Goodman, Introduction to Fourier Optics 3ed (Roberts and Company Publishers, 2005).
  17. A. K. Jain, Fundamentals of Digital Image Processing (Prentice-Hall Inc, 1989).
  18. A. S. Monin and A. M. Yaglom, Statistical Fluid Mechanics Volume II: Mechanics of Turbulence (Courier Corporation, 2013).
  19. M. A. Vorontsov and G. W. Carhart, “Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images,” J. Opt. Soc. Am. A 18(6), 1312–1324 (2001).
    [Crossref] [PubMed]
  20. C. Wu, J. Ko, and C. C. Davis, “Imaging through turbulence using a plenoptic sensor,” Proc. SPIE 9614, 961405 (2015).
  21. D. L. Fried, “Probability of getting a lucky short-exposure image through turbulence,” J. Opt. Soc. Am. 68(12), 1651–1657 (1978).
    [Crossref]
  22. K. Fliegel, “Modeling and measurement of image sensor characteristics,” Radio Eng. 13(4), 27–34 (2004).
  23. R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007).
    [Crossref]

2015 (4)

2014 (1)

2013 (2)

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

X. Zhu and P. Milanfar, “Removing atmospheric turbulence via space-invariant deconvolution,” IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013).
[Crossref] [PubMed]

2009 (1)

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

2008 (1)

2007 (2)

D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
[Crossref]

R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007).
[Crossref]

2006 (1)

N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
[Crossref]

2004 (1)

K. Fliegel, “Modeling and measurement of image sensor characteristics,” Radio Eng. 13(4), 27–34 (2004).

2001 (1)

1988 (1)

1978 (1)

1974 (1)

K. T. Knox and B. J. Thompson, “Recovery of images from atmospherically degraded short-exposure photographs,” Astrophys. J. 193, L45–L48 (1974).
[Crossref]

1966 (1)

Aubailly, M.

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

Ayers, G. R.

Baldwin, J. E.

N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
[Crossref]

Baraniuk, R. G.

R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007).
[Crossref]

Carhart, G. W.

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

M. A. Vorontsov and G. W. Carhart, “Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images,” J. Opt. Soc. Am. A 18(6), 1312–1324 (2001).
[Crossref] [PubMed]

Chen, E.

Dainty, J. C.

Davis, C.

C. Wu, J. Ko, and C. Davis, “Object recognition through turbulence with a modified plenoptic camera,” Proc. SPIE 9354, 93540V (2015).

Davis, C. C.

Fliegel, K.

K. Fliegel, “Modeling and measurement of image sensor characteristics,” Radio Eng. 13(4), 27–34 (2004).

Fried, D. L.

Georgiev, T.

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

Gladysz, S.

Golbraikh, E.

Goma, S.

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

Haik, O.

Hou, W.

Kanaev, A. V.

Knox, K. T.

K. T. Knox and B. J. Thompson, “Recovery of images from atmospherically degraded short-exposure photographs,” Astrophys. J. 193, L45–L48 (1974).
[Crossref]

Ko, J.

C. Wu, J. Ko, and C. C. Davis, “Imaging through turbulence using a plenoptic sensor,” Proc. SPIE 9614, 961405 (2015).

C. Wu, J. Ko, and C. Davis, “Object recognition through turbulence with a modified plenoptic camera,” Proc. SPIE 9354, 93540V (2015).

C. Wu, J. Ko, and C. C. Davis, “Determining the phase and amplitude distortion of a wavefront using a plenoptic sensor,” J. Opt. Soc. Am. A 32(5), 964–978 (2015).
[Crossref] [PubMed]

Kopeika, N. S.

Law, N. M.

N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
[Crossref]

Li, D.

D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
[Crossref]

Lumsdaine, A.

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

Mackay, C. D.

N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
[Crossref]

Matt, S.

Mersereau, R. M.

D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
[Crossref]

Milanfar, P.

X. Zhu and P. Milanfar, “Removing atmospheric turbulence via space-invariant deconvolution,” IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013).
[Crossref] [PubMed]

Restaino, S. R.

Simske, S.

D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
[Crossref]

Thompson, B. J.

K. T. Knox and B. J. Thompson, “Recovery of images from atmospherically degraded short-exposure photographs,” Astrophys. J. 193, L45–L48 (1974).
[Crossref]

Valley, M. T.

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

Vorontsov, M. A.

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

M. A. Vorontsov and G. W. Carhart, “Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images,” J. Opt. Soc. Am. A 18(6), 1312–1324 (2001).
[Crossref] [PubMed]

Wu, C.

C. Wu, J. Ko, and C. C. Davis, “Imaging through turbulence using a plenoptic sensor,” Proc. SPIE 9614, 961405 (2015).

C. Wu, J. Ko, and C. Davis, “Object recognition through turbulence with a modified plenoptic camera,” Proc. SPIE 9354, 93540V (2015).

C. Wu, J. Ko, and C. C. Davis, “Determining the phase and amplitude distortion of a wavefront using a plenoptic sensor,” J. Opt. Soc. Am. A 32(5), 964–978 (2015).
[Crossref] [PubMed]

Yitzhaky, Y.

Yu, Z.

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

Zhu, X.

X. Zhu and P. Milanfar, “Removing atmospheric turbulence via space-invariant deconvolution,” IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013).
[Crossref] [PubMed]

Zilberman, A.

Appl. Opt. (2)

Astron. Astrophys. (1)

N. M. Law, C. D. Mackay, and J. E. Baldwin, “Lucky imaging: high angular resolution imaging in the visible from the ground,” Astron. Astrophys. 446(2), 739–745 (2006).
[Crossref]

Astrophys. J. (1)

K. T. Knox and B. J. Thompson, “Recovery of images from atmospherically degraded short-exposure photographs,” Astrophys. J. 193, L45–L48 (1974).
[Crossref]

IEEE Geosci. Remote Sens. Lett. (1)

D. Li, R. M. Mersereau, and S. Simske, “Atmospheric turbulence-degraded image restoration using principal components analysis,” IEEE Geosci. Remote Sens. Lett. 4(3), 340–344 (2007).
[Crossref]

IEEE Signal Process. Mag. (1)

R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

X. Zhu and P. Milanfar, “Removing atmospheric turbulence via space-invariant deconvolution,” IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013).
[Crossref] [PubMed]

J. Opt. Soc. Am. (2)

J. Opt. Soc. Am. A (2)

Opt. Express (1)

Opt. Lett. (1)

Proc. SPIE (4)

M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, “Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach,” Proc. SPIE 7463, 74630C (2009).
[Crossref]

T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, “Lytro camera technology: theory, algorithms, performance analysis,” Proc. SPIE 8667, 86671J (2013).
[Crossref]

C. Wu, J. Ko, and C. Davis, “Object recognition through turbulence with a modified plenoptic camera,” Proc. SPIE 9354, 93540V (2015).

C. Wu, J. Ko, and C. C. Davis, “Imaging through turbulence using a plenoptic sensor,” Proc. SPIE 9614, 961405 (2015).

Radio Eng. (1)

K. Fliegel, “Modeling and measurement of image sensor characteristics,” Radio Eng. 13(4), 27–34 (2004).

Other (5)

J. W. Goodman, Introduction to Fourier Optics 3ed (Roberts and Company Publishers, 2005).

A. K. Jain, Fundamentals of Digital Image Processing (Prentice-Hall Inc, 1989).

A. S. Monin and A. M. Yaglom, Statistical Fluid Mechanics Volume II: Mechanics of Turbulence (Courier Corporation, 2013).

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Computer Science Technical Report 2, no. 11 (2005): 1–11.

M. C. Roggemann, B. M. Welsh, and B. R. Hunt, Imaging through turbulence (CRC press, 1996).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1 Structure diagram of using a plenoptic sensor to analyze image formation.
Fig. 2
Fig. 2 Illustration diagram of cell image difference generated by turbulence distortion in various regions.
Fig. 3
Fig. 3 Illustration of how a large curved wavefront distortion is affected by the restoring force.
Fig. 4
Fig. 4 Experimental arrangement for imaging through turbulence.
Fig. 5
Fig. 5 Experimental result of using the proposed image metric to auto select the best image cell.
Fig. 6
Fig. 6 Scatter plot between the cell image quality and cell image metric value.
Fig. 7
Fig. 7 Neighboring cell images of turbulence distorted target symbol “M” in the recorded plenoptic image sequence: (a) neighboring cell images sampled along vertical direction; (b) neighboring cell images sampled along horizontal direction; (c) neighboring cell images sampled along time direction.
Fig. 8
Fig. 8 Lucky imaging result on the normal camera branch for the same target under the same turbulence condition.

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

D( m 1 , n 1 ; m 2 , n 2 )= i,j [ I m 1 , n 1 (i,j) I m 2 , n 2 (i,j) ] 2
a vertical (m,n,t)= i,j [ I m+1,n,t (i,j)+ I m1,n,t (i,j)2 I m,n,t (i,j) ] 2
a horizontal (m,n,t)= i,j [ I m,n+1,t (i,j)+ I m,n1,t (i,j)2 I m,n,t (i,j) ] 2
a time ( m,n,t )= i,j [ I m,n,t+1 (i,j)+ I m,n,t1 (i,j)2 I m,n,t (i,j) ] 2 .
M(m,n,t)= a vertical ( m,n,t ) a horizontal ( m,n,t ) a time ( m,n,t )

Metrics