Abstract

The adaptive optics (AO) can be used to compensate for ocular aberrations to achieve near diffraction limited high-resolution retinal images. However, many factors such as the limited aberration measurement and correction accuracy with AO, intraocular scatter, imaging noise and so on will degrade the quality of retinal images. Image post processing is an indispensable and economical method to make up for the limitation of AO retinal imaging procedure. In this paper, we proposed a deep learning method to restore the degraded retinal images for the first time. The method directly learned an end-to-end mapping between the blurred and restored retinal images. The mapping was represented as a deep convolutional neural network that was trained to output high-quality images directly from blurry inputs without any preprocessing. This network was validated on synthetically generated retinal images as well as real AO retinal images. The assessment of the restored retinal images demonstrated that the image quality had been significantly improved.

© 2017 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Deconvolution of adaptive optics retinal images

Julian C. Christou, Austin Roorda, and David R. Williams
J. Opt. Soc. Am. A 21(8) 1393-1401 (2004)

Initialization of iterative parametric algorithms for blind deconvolution of motion-blurred images

Vadim Loyev and Yitzhak Yitzhaky
Appl. Opt. 45(11) 2444-2452 (2006)

Satellite image restoration in the context of a spatially varying point spread function

Nasreddine Hajlaoui, Caroline Chaux, Guillaume Perrin, Frédéric Falzon, and Amel Benazza-Benyahia
J. Opt. Soc. Am. A 27(6) 1473-1481 (2010)

References

  • View by:
  • |
  • |
  • |

  1. J. Liang, D. R. Williams, and D. T. Miller, “Supernormal vision and high-resolution retinal imaging through adaptive optics,” J. Opt. Soc. Am. A 14(11), 2884–2892 (1997).
    [PubMed]
  2. D. T. Miller and A. Roorda, Adaptive Optics in Retinal Microscopy and Vision, in Handbook of Optics (2009).
  3. H. Bao, C. Rao, Y. Zhang, Y. Dai, X. Rao, and Y. Fan, “Hybrid filtering and enhancement of high-resolution adaptive-optics retinal images,” Opt. Lett. 34(22), 3484–3486 (2009).
    [PubMed]
  4. J. Arines, “Partially compensated deconvolution from wavefront sensing images of the eye fundus,” Opt. Commun. 284, 1548–1552 (2011).
  5. C. Rao, Y. Tian, and H. Bao, AO-Based High Resolution Image Post-Processing (InTech, 2012).
  6. J. C. Christou, A. Roorda, and D. R. Williams, “Deconvolution of adaptive optics retinal images,” J. Opt. Soc. Am. A 21(8), 1393–1401 (2004).
    [PubMed]
  7. L. Blanco and L. M. Mugnier, “Marginal blind deconvolution of adaptive optics retinal images,” Opt. Express 19(23), 23227–23239 (2011).
    [PubMed]
  8. H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).
  9. G. Chenegros, L. M. Mugnier, F. Lacombe, and M. Glanc, “3D phase diversity: a myopic deconvolution method for short-exposure images: application to retinal imaging,” J. Opt. Soc. Am. A 24(5), 1349–1357 (2007).
    [PubMed]
  10. A. Lazareva, M. Asad, and G. Slabaugh, “Learning to Deblur Adaptive Optics Retinal Images,” International Conference Image Analysis and Recognition (2017).
  11. T. F. Chan and C. K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Process. 7(3), 370–375 (1998).
    [PubMed]
  12. W. Ouyang, C. C. Loy, X. Tang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, and S. Yang, “DeepID-Net: Deformable deep convolutional neural networks for object detection,” IEEE Transactions on Pattern Analysis & Machine Intelligence PP, 1–1 (2016).
  13. M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)
  14. C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
    [PubMed]
  15. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in International Conference on Neural Information Processing Systems, 2012), 1097–1105.
  16. L. Mariotti and N. Devaney, “Performance analysis of cone detection algorithms,” J. Opt. Soc. Am. A 32(4), 497–506 (2015).
    [PubMed]
  17. L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
    [PubMed]
  18. E. Tas, “Learning Parameter Optimization of Stochastic Gradient Descent with Momentum for a Stochastic Quadratic,” in EURO XXIV, 2010).
  19. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for Fast Feature Embedding,” Proceedings of the 22nd ACM international conference on Multimedia 675–678 (2014).
  20. K. Y. Li and A. Roorda, “Automated identification of cone photoreceptors in adaptive optics retinal images,” J. Opt. Soc. Am. A 24(5), 1358–1363 (2007).
    [PubMed]
  21. F. Šroubek and P. Milanfar, “Robust Multichannel Blind Deconvolution via Fast Alternating Minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
    [PubMed]
  22. A. K. Moorthy and A. C. Bovik, “A Two-Step Framework for Constructing Blind Image Quality Indices,” IEEE Signal Process. Lett. 17, 513–516 (2010).
  23. D. W. Arathorn, Q. Yang, C. R. Vogel, Y. Zhang, P. Tiruveedhula, and A. Roorda, “Retinally stabilized cone-targeted stimulus delivery,” Opt. Express 15(21), 13731–13744 (2007).
    [PubMed]
  24. Y. Zhang, S. Poonja, and A. Roorda, “MEMS-based adaptive optics scanning laser ophthalmoscopy,” Opt. Lett. 31(9), 1268–1270 (2006).
    [PubMed]
  25. F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).
  26. D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
    [PubMed]

2017 (1)

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

2016 (1)

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

2015 (2)

F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).

L. Mariotti and N. Devaney, “Performance analysis of cone detection algorithms,” J. Opt. Soc. Am. A 32(4), 497–506 (2015).
[PubMed]

2012 (1)

F. Šroubek and P. Milanfar, “Robust Multichannel Blind Deconvolution via Fast Alternating Minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[PubMed]

2011 (3)

J. Arines, “Partially compensated deconvolution from wavefront sensing images of the eye fundus,” Opt. Commun. 284, 1548–1552 (2011).

H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).

L. Blanco and L. M. Mugnier, “Marginal blind deconvolution of adaptive optics retinal images,” Opt. Express 19(23), 23227–23239 (2011).
[PubMed]

2010 (1)

A. K. Moorthy and A. C. Bovik, “A Two-Step Framework for Constructing Blind Image Quality Indices,” IEEE Signal Process. Lett. 17, 513–516 (2010).

2009 (1)

2007 (3)

2006 (1)

2004 (1)

2002 (1)

L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
[PubMed]

1998 (1)

T. F. Chan and C. K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Process. 7(3), 370–375 (1998).
[PubMed]

1997 (1)

Arathorn, D. W.

Arines, J.

J. Arines, “Partially compensated deconvolution from wavefront sensing images of the eye fundus,” Opt. Commun. 284, 1548–1552 (2011).

Bao, H.

Blanco, L.

Bovik, A. C.

A. K. Moorthy and A. C. Bovik, “A Two-Step Framework for Constructing Blind Image Quality Indices,” IEEE Signal Process. Lett. 17, 513–516 (2010).

Bradley, A.

L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
[PubMed]

Carroll, J.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Chan, T. F.

T. F. Chan and C. K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Process. 7(3), 370–375 (1998).
[PubMed]

Chenegros, G.

Christou, J. C.

Cooper, R. F.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Cunefare, D.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Dai, Y.

F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).

H. Bao, C. Rao, Y. Zhang, Y. Dai, X. Rao, and Y. Fan, “Hybrid filtering and enhancement of high-resolution adaptive-optics retinal images,” Opt. Lett. 34(22), 3484–3486 (2009).
[PubMed]

Devaney, N.

Dong, C.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

Dubra, A.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Fan, Y.

Fang, L.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Farsiu, S.

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Glanc, M.

He, K.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

Hinton, G. E.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in International Conference on Neural Information Processing Systems, 2012), 1097–1105.

Hong, X.

L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
[PubMed]

Hradis, M.

M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)

Kotera, J.

M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)

Krizhevsky, A.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in International Conference on Neural Information Processing Systems, 2012), 1097–1105.

Lacombe, F.

Li, H.

H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).

Li, K. Y.

Liang, J.

Loy, C. C.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

Lu, J.

H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).

Mariotti, L.

Milanfar, P.

F. Šroubek and P. Milanfar, “Robust Multichannel Blind Deconvolution via Fast Alternating Minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[PubMed]

Miller, D. T.

Moorthy, A. K.

A. K. Moorthy and A. C. Bovik, “A Two-Step Framework for Constructing Blind Image Quality Indices,” IEEE Signal Process. Lett. 17, 513–516 (2010).

Mugnier, L. M.

Poonja, S.

Rao, C.

Rao, X.

Roorda, A.

Shi, G.

H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).

Šroubek, F.

F. Šroubek and P. Milanfar, “Robust Multichannel Blind Deconvolution via Fast Alternating Minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[PubMed]

M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)

Sutskever, I.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in International Conference on Neural Information Processing Systems, 2012), 1097–1105.

Tang, X.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

Thibos, L. N.

L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
[PubMed]

Tiruveedhula, P.

Vogel, C. R.

Williams, D. R.

Wong, C. K.

T. F. Chan and C. K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Process. 7(3), 370–375 (1998).
[PubMed]

Xiao, F.

F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).

Yang, Q.

Yu-Dong, Z.

F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).

Zemcík, P.

M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)

Zhang, Y.

Acta Opt. Sin. (1)

F. Xiao, Y. Dai, and Z. Yu-Dong, “High-resolution retinal imaging with woofer-tweeter adaptive optics system,” Acta Opt. Sin. 35, 29–36 (2015).

IEEE Signal Process. Lett. (1)

A. K. Moorthy and A. C. Bovik, “A Two-Step Framework for Constructing Blind Image Quality Indices,” IEEE Signal Process. Lett. 17, 513–516 (2010).

IEEE Trans. Image Process. (2)

F. Šroubek and P. Milanfar, “Robust Multichannel Blind Deconvolution via Fast Alternating Minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[PubMed]

T. F. Chan and C. K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Process. 7(3), 370–375 (1998).
[PubMed]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016).
[PubMed]

J. Opt. Soc. Am. A (5)

Ophthalmic Physiol. Opt. (1)

L. N. Thibos, A. Bradley, and X. Hong, “A statistical model of the aberration structure of normal, well-corrected eyes,” Ophthalmic Physiol. Opt. 22(5), 427–433 (2002).
[PubMed]

Opt. Commun. (2)

J. Arines, “Partially compensated deconvolution from wavefront sensing images of the eye fundus,” Opt. Commun. 284, 1548–1552 (2011).

H. Li, J. Lu, G. Shi, and Y. Zhang, “Real-time blind deconvolution of retinal images in adaptive optics scanning laser ophthalmoscopy,” Opt. Commun. 284, 3258–3263 (2011).

Opt. Express (2)

Opt. Lett. (2)

Sci. Rep. (1)

D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017).
[PubMed]

Other (8)

E. Tas, “Learning Parameter Optimization of Stochastic Gradient Descent with Momentum for a Stochastic Quadratic,” in EURO XXIV, 2010).

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for Fast Feature Embedding,” Proceedings of the 22nd ACM international conference on Multimedia 675–678 (2014).

A. Lazareva, M. Asad, and G. Slabaugh, “Learning to Deblur Adaptive Optics Retinal Images,” International Conference Image Analysis and Recognition (2017).

C. Rao, Y. Tian, and H. Bao, AO-Based High Resolution Image Post-Processing (InTech, 2012).

D. T. Miller and A. Roorda, Adaptive Optics in Retinal Microscopy and Vision, in Handbook of Optics (2009).

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in International Conference on Neural Information Processing Systems, 2012), 1097–1105.

W. Ouyang, C. C. Loy, X. Tang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, and S. Yang, “DeepID-Net: Deformable deep convolutional neural networks for object detection,” IEEE Transactions on Pattern Analysis & Machine Intelligence PP, 1–1 (2016).

M. Hradis, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional Neural Networks for Direct Text Deblurring,” in British Machine Vision Conference, 2015)

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1
Fig. 1

Network Structure. A blurred image goes through layers and transforms into a restored one. We use 64 filters for each convolutional layer and some sample feature maps are drawn for visualization.

Fig. 2
Fig. 2

The learned first-layer filters.

Fig. 3
Fig. 3

Deconvolution of three representative synthetic images imitating at different eccentricities.

Fig. 4
Fig. 4

Result of cone detection on a representative retinal image obtained by the algorithm of Li and Roorda. (a). original image (1.2 mm eccentricity from the foveal center); (b). restored by the proposed method; (c). restored by the ALM method. The scale bar is 50 µm.

Fig. 5
Fig. 5

Deconvolution result of a retinal image captured by AOSLO system. (a) original image (0.9 mm eccentricity from the foveal center); (b) restored by the proposed method; (c) restored by the ALM method; (d) the corresponding normalized image power spectra. The scale bar is 50 µm.

Fig. 6
Fig. 6

Deconvolution result of a retinal image captured by AOSLO system. (a) original image (0.3 mm eccentricity from the foveal center); (b) restored by the proposed method; (c) restored by the ALM method; (d) the corresponding normalized image power spectra. The scale bar is 100 µm.

Tables (3)

Tables Icon

Table 1 The Performance and Complexity of the Network with Different Filter Numbers and Filter Sizes

Tables Icon

Table 2 Quality Assessment of Blurred and Processed Synthetic Retinal Images

Tables Icon

Table 3 Quality Assessment of Blurred and Restored Real AO Retinal Images

Equations (11)

Equations on this page are rendered with MathJax. Learn more.

y=xk+n,
x ^ = argmin x ||ykx| | 2 2 +r(x).
x ^ =F(y,θ)
L(θ)= 1 N i=1 N ||F( y i ;θ) x i | | 2 ,
F 1 (y)=max(0, W 1 y+ B 1 ),
F l (y)=max(0, W l F l1 (y)+ B l ),l=2,3,4,
F 5 (y)= W 5 F 4 (y)+ B 5 .
PSF= FFT{ P(x,y)exp(i 2π λ φ(x,y)) } 2 ,
φ(x,y)= m=3 44 a m Z m (x,y) ,
argmin W,B 1 2N i=1 N ||F( y i ) x i | | 2 2 +0.0005||W| | 2 2 .
ER= |coneNotruthNo| truthNo 100%