Abstract

A Wiener filter-based deconvolution algorithm is developed to restore vibration-degraded video imagery from an intensified CCD camera. The method is based on the use of azimuth and elevation angular optical line-of-sight data recorded from external sensors to estimate a two-dimensional vibration-blur impulse response on a per frame basis. Flight conditions are reproduced in the laboratory by use of prerecorded in-flight vibration data. The performance of the algorithm varies from frame to frame, following the time-varying characteristics of the vibration-blur impulse response. However, real-time display of the restored video minimizes these effects because of eye integration, and near-full restoration of the original uncorrupted imagery is observed for both high-light- and low-light-level conditions with minimal amplification of noise.

© 1999 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. J. M. Hilkert, M. Bowen, J. Wang, “Specifications for image stabilization systems,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 24–38 (1991).
    [CrossRef]
  2. C. J. Cooper, “Sensor line of sight stabilization,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 39–51 (1991).
    [CrossRef]
  3. M. K. Masten, “Electromechanical systems for optical target tracking sensors,” in Multitarget-Multisensor Tracking: Advanced Applications, Y. Bar-Shalom, ed. (YBS, Storrs, Conn., 1990), pp. 321–360.
  4. G. C. Holst, Electro-Optical Imaging System Performance (SPIE, Bellingham, Wash., 1995), pp. 110–118.
  5. D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations,” Opt. Eng. 26, 529–533 (1987).
    [CrossRef]
  6. O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
    [CrossRef]
  7. O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
    [CrossRef]
  8. O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
    [CrossRef]
  9. O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
    [CrossRef]
  10. A. Stern, N. S. Kopeika, “Analytical method to calculate optical transfer functions for image motion and vibrations using moments,” J. Opt. Soc. Am. A 14, 388–396 (1997).
    [CrossRef]
  11. N. S. Kopeika, A System Engineering Approach to Imaging (SPIE, Bellingham, Wash., 1998), Chaps. 14 and 18.
  12. Y. Yitzhaky, N. S. Kopeika, “Identification of blur parameters from motion blurred images,” CVGIP. Graph. Models Image Process.” 59, 310–320 (1997).
    [CrossRef]
  13. Y. Yitzhaky, I. Mor, A. Lantzman, N. S. Kopeika, “Direct method for restoration of motion-blurred images,” J. Opt. Soc. Am. A 15, 1512–1519 (1998).
    [CrossRef]
  14. A. Stern, N. S. Kopeika, “General restoration filter for vibrated-image restoration,” Appl. Opt. 37, 7596–7603 (1998).
    [CrossRef]
  15. C. W. Therrien, Discrete Random Signals and Statistical Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1992), pp. 587–588.
  16. L. Levi, “Motion blurring with decaying detector response,” Appl. Opt. 10, 38–41 (1971).
    [CrossRef] [PubMed]
  17. S. C. Som, “Analysis of the effect of linear smear on photographic images,” J. Opt. Soc. Am. 61, 859–864 (1971).
    [CrossRef]
  18. A. A. Sawchuk, “Space-variant image motion degradation and restoration,” Proc. IEEE 60, 854–861 (1972).
    [CrossRef]
  19. A. V. Oppenheim, R. W. Shafer, Discrete-Time Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1989), pp. 407–415.
  20. G. C. Holst, CCD Arrays, Cameras, and Displays (SPIE, Bellingham, Wash., 1996), pp. 269–270.
  21. For example, A. K. Jain, Fundamentals of Digital Image Processing, 2nd ed. (Addison-Wesley, Reading, Mass., 1987), pp. 275–289.
  22. A. M. Tekalp, Digital Video Processing (Prentice-Hall PTR, Upper Saddle River, N.J., 1995), pp. 95–116.
  23. For example, P. Gillett, Calculus and Analytic Geometry (Heath, Lexington, Mass., 1981), pp. 555–561.
  24. S. T. Hammett, P. J. Bex, “Motion sharpening: evidence for the addition of high spatial frequencies to the effective neural image,” Vision Res. 36, 2729–2733 (1996).
    [CrossRef] [PubMed]
  25. G. C. Holst, “Infrared imaging system testing,” in Electro-Optical Systems Design, Analysis, and Testing, Vol. 4 of The Infrared and Electro-Optical Systems Handbook, M. C. Dudzik, ed. (SPIE, Bellingham, Wash., 1993), pp. 235–241.

1998 (2)

1997 (3)

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

A. Stern, N. S. Kopeika, “Analytical method to calculate optical transfer functions for image motion and vibrations using moments,” J. Opt. Soc. Am. A 14, 388–396 (1997).
[CrossRef]

Y. Yitzhaky, N. S. Kopeika, “Identification of blur parameters from motion blurred images,” CVGIP. Graph. Models Image Process.” 59, 310–320 (1997).
[CrossRef]

1996 (2)

S. T. Hammett, P. J. Bex, “Motion sharpening: evidence for the addition of high spatial frequencies to the effective neural image,” Vision Res. 36, 2729–2733 (1996).
[CrossRef] [PubMed]

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

1994 (1)

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

1992 (1)

O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
[CrossRef]

1987 (1)

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

1972 (1)

A. A. Sawchuk, “Space-variant image motion degradation and restoration,” Proc. IEEE 60, 854–861 (1972).
[CrossRef]

1971 (2)

Adar, Z.

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

Bex, P. J.

S. T. Hammett, P. J. Bex, “Motion sharpening: evidence for the addition of high spatial frequencies to the effective neural image,” Vision Res. 36, 2729–2733 (1996).
[CrossRef] [PubMed]

Bowen, M.

J. M. Hilkert, M. Bowen, J. Wang, “Specifications for image stabilization systems,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 24–38 (1991).
[CrossRef]

Cooper, C. J.

C. J. Cooper, “Sensor line of sight stabilization,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 39–51 (1991).
[CrossRef]

Cotter, A.

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

Dror, I.

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

Fisher, M.

O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
[CrossRef]

Gillett, P.

For example, P. Gillett, Calculus and Analytic Geometry (Heath, Lexington, Mass., 1981), pp. 555–561.

Hadar, O.

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
[CrossRef]

Hammett, S. T.

S. T. Hammett, P. J. Bex, “Motion sharpening: evidence for the addition of high spatial frequencies to the effective neural image,” Vision Res. 36, 2729–2733 (1996).
[CrossRef] [PubMed]

Hilkert, J. M.

J. M. Hilkert, M. Bowen, J. Wang, “Specifications for image stabilization systems,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 24–38 (1991).
[CrossRef]

Holst, G. C.

G. C. Holst, Electro-Optical Imaging System Performance (SPIE, Bellingham, Wash., 1995), pp. 110–118.

G. C. Holst, CCD Arrays, Cameras, and Displays (SPIE, Bellingham, Wash., 1996), pp. 269–270.

G. C. Holst, “Infrared imaging system testing,” in Electro-Optical Systems Design, Analysis, and Testing, Vol. 4 of The Infrared and Electro-Optical Systems Handbook, M. C. Dudzik, ed. (SPIE, Bellingham, Wash., 1993), pp. 235–241.

Jain, A. K.

For example, A. K. Jain, Fundamentals of Digital Image Processing, 2nd ed. (Addison-Wesley, Reading, Mass., 1987), pp. 275–289.

Kaplan, D.

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

Kopeika, N. S.

Y. Yitzhaky, I. Mor, A. Lantzman, N. S. Kopeika, “Direct method for restoration of motion-blurred images,” J. Opt. Soc. Am. A 15, 1512–1519 (1998).
[CrossRef]

A. Stern, N. S. Kopeika, “General restoration filter for vibrated-image restoration,” Appl. Opt. 37, 7596–7603 (1998).
[CrossRef]

A. Stern, N. S. Kopeika, “Analytical method to calculate optical transfer functions for image motion and vibrations using moments,” J. Opt. Soc. Am. A 14, 388–396 (1997).
[CrossRef]

Y. Yitzhaky, N. S. Kopeika, “Identification of blur parameters from motion blurred images,” CVGIP. Graph. Models Image Process.” 59, 310–320 (1997).
[CrossRef]

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
[CrossRef]

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE, Bellingham, Wash., 1998), Chaps. 14 and 18.

Lantzman, A.

Levi, L.

Masten, M. K.

M. K. Masten, “Electromechanical systems for optical target tracking sensors,” in Multitarget-Multisensor Tracking: Advanced Applications, Y. Bar-Shalom, ed. (YBS, Storrs, Conn., 1990), pp. 321–360.

Mor, I.

Novogrozky, Y.

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

Oppenheim, A. V.

A. V. Oppenheim, R. W. Shafer, Discrete-Time Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1989), pp. 407–415.

Robbins, M.

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

Sawchuk, A. A.

A. A. Sawchuk, “Space-variant image motion degradation and restoration,” Proc. IEEE 60, 854–861 (1972).
[CrossRef]

Shafer, R. W.

A. V. Oppenheim, R. W. Shafer, Discrete-Time Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1989), pp. 407–415.

Som, S. C.

Stern, A.

Tekalp, A. M.

A. M. Tekalp, Digital Video Processing (Prentice-Hall PTR, Upper Saddle River, N.J., 1995), pp. 95–116.

Therrien, C. W.

C. W. Therrien, Discrete Random Signals and Statistical Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1992), pp. 587–588.

Wang, J.

J. M. Hilkert, M. Bowen, J. Wang, “Specifications for image stabilization systems,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 24–38 (1991).
[CrossRef]

Wulich, D.

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

Yitzhaky, Y.

Y. Yitzhaky, I. Mor, A. Lantzman, N. S. Kopeika, “Direct method for restoration of motion-blurred images,” J. Opt. Soc. Am. A 15, 1512–1519 (1998).
[CrossRef]

Y. Yitzhaky, N. S. Kopeika, “Identification of blur parameters from motion blurred images,” CVGIP. Graph. Models Image Process.” 59, 310–320 (1997).
[CrossRef]

Appl. Opt. (2)

CVGIP. Graph. Models Image Process. (1)

Y. Yitzhaky, N. S. Kopeika, “Identification of blur parameters from motion blurred images,” CVGIP. Graph. Models Image Process.” 59, 310–320 (1997).
[CrossRef]

J. Opt. Soc. Am. (1)

J. Opt. Soc. Am. A (2)

Opt. Eng. (4)

O. Hadar, M. Fisher, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part III: Numerical calculation of modulation transfer function,” Opt. Eng. 31, 581–589 (1992).
[CrossRef]

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations. Part IV: Real-time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibrations,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

O. Hadar, M. Robbins, Y. Novogrozky, D. Kaplan, “Image motion restoration from a sequence of images,” Opt. Eng. 35, 2898–2904 (1996).
[CrossRef]

Opt. Laser Technol. (1)

O. Hadar, Z. Adar, A. Cotter, N. S. Kopeika, “Restoration of images degraded by extreme mechanical vibrations,” Opt. Laser Technol. 29, 171–177 (1997).
[CrossRef]

Proc. IEEE (1)

A. A. Sawchuk, “Space-variant image motion degradation and restoration,” Proc. IEEE 60, 854–861 (1972).
[CrossRef]

Vision Res. (1)

S. T. Hammett, P. J. Bex, “Motion sharpening: evidence for the addition of high spatial frequencies to the effective neural image,” Vision Res. 36, 2729–2733 (1996).
[CrossRef] [PubMed]

Other (12)

G. C. Holst, “Infrared imaging system testing,” in Electro-Optical Systems Design, Analysis, and Testing, Vol. 4 of The Infrared and Electro-Optical Systems Handbook, M. C. Dudzik, ed. (SPIE, Bellingham, Wash., 1993), pp. 235–241.

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE, Bellingham, Wash., 1998), Chaps. 14 and 18.

J. M. Hilkert, M. Bowen, J. Wang, “Specifications for image stabilization systems,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 24–38 (1991).
[CrossRef]

C. J. Cooper, “Sensor line of sight stabilization,” in Tactical Infrared Systems, J. W. Tuttle, ed., Proc. SPIE1498, 39–51 (1991).
[CrossRef]

M. K. Masten, “Electromechanical systems for optical target tracking sensors,” in Multitarget-Multisensor Tracking: Advanced Applications, Y. Bar-Shalom, ed. (YBS, Storrs, Conn., 1990), pp. 321–360.

G. C. Holst, Electro-Optical Imaging System Performance (SPIE, Bellingham, Wash., 1995), pp. 110–118.

A. V. Oppenheim, R. W. Shafer, Discrete-Time Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1989), pp. 407–415.

G. C. Holst, CCD Arrays, Cameras, and Displays (SPIE, Bellingham, Wash., 1996), pp. 269–270.

For example, A. K. Jain, Fundamentals of Digital Image Processing, 2nd ed. (Addison-Wesley, Reading, Mass., 1987), pp. 275–289.

A. M. Tekalp, Digital Video Processing (Prentice-Hall PTR, Upper Saddle River, N.J., 1995), pp. 95–116.

For example, P. Gillett, Calculus and Analytic Geometry (Heath, Lexington, Mass., 1981), pp. 555–561.

C. W. Therrien, Discrete Random Signals and Statistical Signal Processing (Prentice-Hall, Englewood Cliffs, N.J., 1992), pp. 587–588.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1

Averaged periodogram power spectral density estimate of the LOS azimuth angle of the digitized flight data over 5 s.

Fig. 2
Fig. 2

Averaged periodogram power spectral density estimate of the LOS elevation angle of the digitized flight data over 5 s.

Fig. 3
Fig. 3

Plot of the LOS pointing angle over a 250-ms interval computed from the digitized azimuth and elevation flight data.

Fig. 4
Fig. 4

Histogram of the LOS azimuth angle of the digitized flight data.

Fig. 5
Fig. 5

Histogram of the LOS elevation angle of the digitized flight data.

Fig. 6
Fig. 6

Histogram of the computed pointing angle of the digitized flight data.

Fig. 7
Fig. 7

Block diagram of the system concept for image-processing-based electronic image stabilization through digital restoration. A/D, analog to digital; D/A digital to analog.

Fig. 8
Fig. 8

Detailed layout of laboratory experiment. LP, low pass; EFL, equivalent focal length.

Fig. 9
Fig. 9

Four-leaf rose scan pattern used to calibrate the piezoelectric scanning mirror.

Fig. 10
Fig. 10

Examples of the calculated vibration-blur impulse response and transfer function based on the azimuth and elevation scan data for individual image frames. The integration time, or gate interval, for all cases was 15.8 ms. The vibration-blur impulse response is shown before and after convolution with the intensifier impulse response. Larger magnitudes are indicated by a darker shade of gray. An x-axis profile of the MTF is also shown: (a) case 1, (b) case 2, (c) case 3.

Fig. 11
Fig. 11

Results of image restoration. The images, from top left clockwise, are (1) original imagery with no motion, (2) vibration-degraded imagery, (3) restored imagery, (4) registered-only imagery using subpixel block matching. The images were obtained by averaging three sequential frames from a 30-frame/s sequence. (a) Image of a standard EIA resolution chart under high-light-level conditions with a measured SNR of 18 dB, (b) image of a standard EIA resolution chart under low-light-level conditions with a measured SNR of 8 dB, (c) image of a truck on a test range at a distance of 2.2 km with a measured SNR of 16 dB. The dimensions of the truck are approximately 32 × 12 pixels.

Tables (1)

Tables Icon

Table 1 Limiting Resolution Measurements on Real-Time Videoa

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

gx=t0t0+T fx * hx * δx-αtdt+nx,
hvibx=t0t0+T δx-αtdt=δNormalddtαtt=α-1x=δNormalddt α1t2t=α1-1x1+ddt α2t2t=α2-1x11/2,
1s2ΔxΔt2-1/2=ΔtΔx2.
hintensifiern1, n2=ρ02 exp-πρ02Δx2n12+n22,
HWienerω=Hvib*ω|Hvibω|2+Pnω/Pimω,
x sinθAZcosθEL+y sinθEL+z-z0cosθAZcosθEL=0,
zi-xiθAZ+yiθEL+z0,
θAZn=θpeak cosnω0cosn2ω0, θELn=θpeak sinnω0cosn2ω0,
SNR=10 logσim2σnoise2,

Metrics