Abstract

An algorithm to increase the spatial resolution of digital video sequences captured with a camera that is subject to mechanical vibration is developed. The blur caused by vibration of the camera is often the primary cause for image degradation. We address the degradation caused by low-frequency vibrations (vibrations for which the exposure time is less than the vibration period). The blur caused by low-frequency vibrations differs from other types by having a random shape and displacement. The different displacement of each frame makes the approach used in superresolution (SR) algorithms suitable for resolution enhancement. However, SR algorithms that were developed for general types of blur should be adapted to the specific characteristics of low-frequency vibration blur. We use the method of projection onto convex sets together with a motion estimation method specially adapted to low-frequency vibration blur characteristics. We also show that the random blur characterizing low-frequency vibration requires selection of the frames prior to processing. The restoration performance as well as the frame selection criteria is dependent mainly on the motion estimation precision.

© 2001 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering, Bellingham, Wash., 1998), Chap. 14, pp. 422–430, and Chap. 18, pp. 517–524.
  2. D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration,” Opt. Eng. 26, 529–533 (1987).
    [CrossRef]
  3. O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
    [CrossRef]
  4. M. Elad, A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process. 6, 1646–1658 (1997).
    [CrossRef] [PubMed]
  5. A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
    [CrossRef]
  6. D. C. Youla, “Generalized image restoration by the method of alternating orthogonal projections,” IEEE Trans. Circuits Syst. CAS-25, 694–702 (1979).
  7. D. C. Youla, H. Webb, “Image restoration by the method of projection on convex sets. Part 1. Alternating orthogonal projection,” IEEE Trans. Med. Imaging TMI-1, 81–94 (1982).
    [CrossRef]
  8. R. W. Gerchberg, W. O. Saxton, “A practical algorithm for determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).
  9. A. Levi, H. Stark, “Image restoration by the method of generalized projection with application to restoration from magnitude,” J. Opt. Soc. Am. A 1, 932–943 (1984).
    [CrossRef]
  10. A. Papoulis, “A new algorithm in spectral analysis and band-limited extrapolation,” IEEE Trans. Circuits Syst. CAS-22, 735–742 (1975).
    [CrossRef]
  11. H. Stark, P. Oskoui, “High-resolution image recovery from image-plane arrays, using convex projections,” J. Opt. Soc. Am. A 6, 1715–1726 (1989).
    [CrossRef] [PubMed]
  12. A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
    [CrossRef] [PubMed]
  13. T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
    [CrossRef]
  14. C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
    [CrossRef]
  15. B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, Berlin, 1993).
    [CrossRef]
  16. R. Buschmann, “Efficiency of displacement estimation techniques,” Signal Process. Image Commun. 10, 43–61 (1997).
    [CrossRef]
  17. M. Bieling, “Displacement estimation by hierarchical block matching,” Proceedings of the Third SPIE Symposium on Visual Communications on Image Processing (SPIE, Bellingham, Wash., 1988), pp. 942–951.
    [CrossRef]
  18. A. Irani, S. Peleg, “Motion analysis for image enhancement: resolution, occlusion, and transparency,” J. Visual Commun. Image Represent. 4, 324–335 (1993).
    [CrossRef]
  19. Q. Tian, M. N. Hunhs, “Algorithms for subpixel registration,” Comput. Vision Graph. Image Process. 35, 220–233 (1986).
    [CrossRef]

2000 (1)

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

1998 (1)

C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
[CrossRef]

1997 (3)

R. Buschmann, “Efficiency of displacement estimation techniques,” Signal Process. Image Commun. 10, 43–61 (1997).
[CrossRef]

M. Elad, A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process. 6, 1646–1658 (1997).
[CrossRef] [PubMed]

A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
[CrossRef] [PubMed]

1995 (1)

T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
[CrossRef]

1994 (1)

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

1993 (1)

A. Irani, S. Peleg, “Motion analysis for image enhancement: resolution, occlusion, and transparency,” J. Visual Commun. Image Represent. 4, 324–335 (1993).
[CrossRef]

1989 (1)

1987 (1)

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

1986 (1)

Q. Tian, M. N. Hunhs, “Algorithms for subpixel registration,” Comput. Vision Graph. Image Process. 35, 220–233 (1986).
[CrossRef]

1984 (1)

1982 (1)

D. C. Youla, H. Webb, “Image restoration by the method of projection on convex sets. Part 1. Alternating orthogonal projection,” IEEE Trans. Med. Imaging TMI-1, 81–94 (1982).
[CrossRef]

1979 (1)

D. C. Youla, “Generalized image restoration by the method of alternating orthogonal projections,” IEEE Trans. Circuits Syst. CAS-25, 694–702 (1979).

1975 (1)

A. Papoulis, “A new algorithm in spectral analysis and band-limited extrapolation,” IEEE Trans. Circuits Syst. CAS-22, 735–742 (1975).
[CrossRef]

1972 (1)

R. W. Gerchberg, W. O. Saxton, “A practical algorithm for determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).

Bieling, M.

M. Bieling, “Displacement estimation by hierarchical block matching,” Proceedings of the Third SPIE Symposium on Visual Communications on Image Processing (SPIE, Bellingham, Wash., 1988), pp. 942–951.
[CrossRef]

Buschmann, R.

R. Buschmann, “Efficiency of displacement estimation techniques,” Signal Process. Image Commun. 10, 43–61 (1997).
[CrossRef]

Dror, I.

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

Elad, M.

M. Elad, A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process. 6, 1646–1658 (1997).
[CrossRef] [PubMed]

Feuer, A.

M. Elad, A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process. 6, 1646–1658 (1997).
[CrossRef] [PubMed]

Gerchberg, R. W.

R. W. Gerchberg, W. O. Saxton, “A practical algorithm for determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).

Hadar, O.

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

Hunhs, M. N.

Q. Tian, M. N. Hunhs, “Algorithms for subpixel registration,” Comput. Vision Graph. Image Process. 35, 220–233 (1986).
[CrossRef]

Irani, A.

A. Irani, S. Peleg, “Motion analysis for image enhancement: resolution, occlusion, and transparency,” J. Visual Commun. Image Represent. 4, 324–335 (1993).
[CrossRef]

Jähne, B.

B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, Berlin, 1993).
[CrossRef]

Kempner, E.

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

Kopeika, N. S.

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering, Bellingham, Wash., 1998), Chap. 14, pp. 422–430, and Chap. 18, pp. 517–524.

Kotzer, T.

T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
[CrossRef]

Levi, A.

Lin, J. Y.

C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
[CrossRef]

Liu, C. M.

C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
[CrossRef]

Oskoui, P.

Papoulis, A.

A. Papoulis, “A new algorithm in spectral analysis and band-limited extrapolation,” IEEE Trans. Circuits Syst. CAS-22, 735–742 (1975).
[CrossRef]

Patti, A. J.

A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
[CrossRef] [PubMed]

Peleg, S.

A. Irani, S. Peleg, “Motion analysis for image enhancement: resolution, occlusion, and transparency,” J. Visual Commun. Image Represent. 4, 324–335 (1993).
[CrossRef]

Rosen, J.

T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
[CrossRef]

Saxton, W. O.

R. W. Gerchberg, W. O. Saxton, “A practical algorithm for determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).

Sezan, M. I.

A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
[CrossRef] [PubMed]

Shamir, J.

T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
[CrossRef]

Shukrun, A.

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

Stark, H.

Stern, A.

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

Teklap, A. M.

A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
[CrossRef] [PubMed]

Tian, Q.

Q. Tian, M. N. Hunhs, “Algorithms for subpixel registration,” Comput. Vision Graph. Image Process. 35, 220–233 (1986).
[CrossRef]

Wang, C. N.

C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
[CrossRef]

Webb, H.

D. C. Youla, H. Webb, “Image restoration by the method of projection on convex sets. Part 1. Alternating orthogonal projection,” IEEE Trans. Med. Imaging TMI-1, 81–94 (1982).
[CrossRef]

Wulich, D.

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

Youla, D. C.

D. C. Youla, H. Webb, “Image restoration by the method of projection on convex sets. Part 1. Alternating orthogonal projection,” IEEE Trans. Med. Imaging TMI-1, 81–94 (1982).
[CrossRef]

D. C. Youla, “Generalized image restoration by the method of alternating orthogonal projections,” IEEE Trans. Circuits Syst. CAS-25, 694–702 (1979).

App. Opt. (1)

T. Kotzer, J. Rosen, J. Shamir, “Application of serial- and parallel-projection methods to correlation-filter design,” App. Opt. 34, 3883–3895 (1995).
[CrossRef]

Comput. Vision Graph. Image Process. (1)

Q. Tian, M. N. Hunhs, “Algorithms for subpixel registration,” Comput. Vision Graph. Image Process. 35, 220–233 (1986).
[CrossRef]

IEEE Trans. Circuits Syst. (2)

A. Papoulis, “A new algorithm in spectral analysis and band-limited extrapolation,” IEEE Trans. Circuits Syst. CAS-22, 735–742 (1975).
[CrossRef]

D. C. Youla, “Generalized image restoration by the method of alternating orthogonal projections,” IEEE Trans. Circuits Syst. CAS-25, 694–702 (1979).

IEEE Trans. Consumer Electron. (1)

C. M. Liu, C. N. Wang, J. Y. Lin, “A new postprocessing method for the block-based DCT coding based on the convex-projection theory,” IEEE Trans. Consumer Electron. 44, 1054–1061 (1998).
[CrossRef]

IEEE Trans. Image Process. (2)

A. J. Patti, M. I. Sezan, A. M. Teklap, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6, 1064–1076 (1997).
[CrossRef] [PubMed]

M. Elad, A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process. 6, 1646–1658 (1997).
[CrossRef] [PubMed]

IEEE Trans. Med. Imaging (1)

D. C. Youla, H. Webb, “Image restoration by the method of projection on convex sets. Part 1. Alternating orthogonal projection,” IEEE Trans. Med. Imaging TMI-1, 81–94 (1982).
[CrossRef]

J. Opt. Soc. Am. A (2)

J. Visual Commun. Image Represent. (1)

A. Irani, S. Peleg, “Motion analysis for image enhancement: resolution, occlusion, and transparency,” J. Visual Commun. Image Represent. 4, 324–335 (1993).
[CrossRef]

Opt. Eng. (3)

A. Stern, E. Kempner, A. Shukrun, N. S. Kopeika, “Restoration and resolution enhancement of a single image from a vibration-distorted image sequence,” Opt. Eng. 39, 2451–2457 (2000).
[CrossRef]

D. Wulich, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration,” Opt. Eng. 26, 529–533 (1987).
[CrossRef]

O. Hadar, I. Dror, N. S. Kopeika, “Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification,” Opt. Eng. 33, 566–578 (1994).
[CrossRef]

Optik (1)

R. W. Gerchberg, W. O. Saxton, “A practical algorithm for determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).

Signal Process. Image Commun. (1)

R. Buschmann, “Efficiency of displacement estimation techniques,” Signal Process. Image Commun. 10, 43–61 (1997).
[CrossRef]

Other (3)

M. Bieling, “Displacement estimation by hierarchical block matching,” Proceedings of the Third SPIE Symposium on Visual Communications on Image Processing (SPIE, Bellingham, Wash., 1988), pp. 942–951.
[CrossRef]

B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, Berlin, 1993).
[CrossRef]

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering, Bellingham, Wash., 1998), Chap. 14, pp. 422–430, and Chap. 18, pp. 517–524.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1

Motion functions during exposures of a low-frequency vibrating imaging platform. The spread function depends on the instant of the exposure (t x ), which is random.1-3

Fig. 2
Fig. 2

Distorted low-resolution frame {g k }k=1N simulation model.

Fig. 3
Fig. 3

Demonstration of the POCS concept. Starting with the initial point f 0 and applying the projections P 1, P 2, P 3 [Eq. (1) with λ i = 1)] successively on the sets C 1, C 2, C 3, we find that the iterations proceed in the directions of the arrows until they converge to the solution f (at the intersection of the sets).

Fig. 4
Fig. 4

(a) and (c) Examples of two vibrated low-resolution images blurred by (b) and (d) the respective PSFs.

Fig. 5
Fig. 5

Motion function r(t) is shown together with the estimated motion after the first (1)(t) and the tenth iteration (10)(t).

Fig. 6
Fig. 6

Comparison between the convergence of the reconstruction algorithms that used the selection method based on the SDE suggested here (bottom curve), the selection method based on blur extent suggested in Ref. 5 (top curve), and the whole image sequence (middle curve).

Fig. 7
Fig. 7

(a) Original high-resolution image. (b) and (c) Two images from a sequence of nine low-resolution images (160 × 160 pixels) distorted by vibration with amplitude of 10 pixels and frequency of 10 Hz. (d) The high-resolution (320 × 320 pixel) restoration of the image.

Fig. 8
Fig. 8

(a) Bar target composed from vertical bars at frequencies 1/2, 1/4, 1/6, and 1/8 lines/pixel. (b) An image from a sequence distorted by severe horizontal vibration. (vibration amplitude of 15 pixels, frequency of 5 Hz). (c) The restored image.

Equations (11)

Equations on this page are rendered with MathJax. Learn more.

fl+1=TmTm-1  T1f1,  l=0,1
Ckm1, m2=yn1, n2: |rkym1, m2|δ0m1, m2, k,
rkym1, m2=gkm1, m2-n1,n2 Gkyn1, n2×hkn1, n2; m1, m2.
Pkm1, m2fln1, n2=fln1, n2+λrkflm1, m2-δ0m1, m2hkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkflm1, m2>δ0m1, m2, k0,-δ0m1, m2, krflm1, m2δ0m1, m2, k,rkflm1, m2+δ0m1, m2, khkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkflm1, m2<-δ0m1, m2, k
CA=yn1, n2:0fn1, n2255
Δrˆkr=Δrˆk-Δrˆr.
rˆk=rˆk0+Δrˆkr.
Jkl=n1n2 λrkfm1, m2-δ0m1, m2hkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkfm1, m2>δ0m1, m2, k0,-δ0m1, m2, krfm1, m2δ0m1, m2, k.rkfm1, m2+δ0m1, m2, khkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkfm1, m2<-δ0m1, m2, k
Ck,lm1, m2=yn1, n2:|rkym1, m2|δ0m1, m2, k, Jkl-1<th
Pkm1, m2fln1, n2=fln1, n2+λrkflm1, m2-δ0m1, m2, khkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkflm1, m2>δ0m1, m2, k, Jkl-1<th0,-δ0m1, m2, krflm1, m2δ0m1, m2, k or Jkl-1th.rkflm1, m2+δ0m1, m2, khkn1, n2; m1, m2o1o2 hk2o1, o2; m1, m2,rkflm1, m2<-δ0m1, m2, k, Jkl-1<th
thl=minJklk=1N+αmedianJklk=1N-minJklk=1N,

Metrics