Abstract

We present an algorithm to realign images distorted by motion and vibrations captured in cameras that use a scanning vector sensor with an interlaced scheme. In particular, the method is developed for images captured by a staggered time delay and integration camera distorted by motion. The algorithm improves the motion-distorted image by adjusting its fields irrespective of the type of motion that occurs during the exposure. The algorithm performs two tasks: estimation of the field relative motion during the exposure by a normal least-squares estimation technique and improvement of the degraded image from such motion distortion. The algorithm uses matrix computations; therefore it has a computation advantage over algorithms based on the technique of searching for a match. The algorithm is successfully demonstrated on both simulated and real images.

© 2006 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. C. Holst, CCD Arrays Cameras and Displays (SPIE Optical Engineering Press, 1998).
  2. D. F. Barbe, "Time delay and integration image sensors" in Solid State Imaging, P. G. Jespers, F. van de Wiele, and M. H. White, eds. (Noordhoff, 1976), pp. 659-671.
  3. N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering Press, 1998).
  4. O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
    [CrossRef]
  5. A. Stern and N. S. Kopeika, "Motion-distorted composite-frame restoration," Appl. Opt. 38, 757-765 (1999).
    [CrossRef]
  6. S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
    [CrossRef]
  7. G. Hochman, Y. Yitzhaky, N. S. Kopeika, Y. Lauber, M. Citroen, and A. Stern, "Restoration of images captured by a staggered time delay and integration camera in the presence of mechanical vibrations," Appl. Opt. 43, 4345-4354 (2004).
    [CrossRef] [PubMed]
  8. C. Stiller and J. Konrad, "Estimating motion in image sequences--a tutorial on modeling and computation of 2D motion," IEEE Signal Process. Mag. 16(4), 70-91 (1999).
    [CrossRef]
  9. A. Tekalp, Digital Video Processing (Prentice Hall, 1995).
  10. Recommendation H.261: Video Codec for Audiovisual Services at p × 64 kbits/s." Rep. COM XV-R 37-E (CCITT, 1989).
  11. C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
    [CrossRef] [PubMed]
  12. C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
    [CrossRef] [PubMed]
  13. H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
    [CrossRef] [PubMed]
  14. B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, 1993).
  15. G. Zelniker and F. J. Taylor, Advanced Digital Signal Processing (Marcel Dekker, 1994).
  16. P. Papoulis, Probability, Random Variables, and Stochastic Processes, 2nd ed. (McGraw-Hill, 1985), Chap. 13, pp. 407-476.

2004 (1)

2003 (1)

S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
[CrossRef]

1999 (2)

A. Stern and N. S. Kopeika, "Motion-distorted composite-frame restoration," Appl. Opt. 38, 757-765 (1999).
[CrossRef]

C. Stiller and J. Konrad, "Estimating motion in image sequences--a tutorial on modeling and computation of 2D motion," IEEE Signal Process. Mag. 16(4), 70-91 (1999).
[CrossRef]

1995 (2)

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
[CrossRef] [PubMed]

1993 (1)

C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
[CrossRef] [PubMed]

1992 (1)

C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
[CrossRef] [PubMed]

Chan, C. L.

C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
[CrossRef] [PubMed]

Charalambous, C.

C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
[CrossRef] [PubMed]

Citroen, M.

Ghaddar, F. K.

C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
[CrossRef] [PubMed]

Hadar, O.

S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
[CrossRef]

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

Hochman, G.

Holst, G. C.

G. C. Holst, CCD Arrays Cameras and Displays (SPIE Optical Engineering Press, 1998).

Jähne, B.

B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, 1993).

Katsaggelos, A. K.

C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
[CrossRef] [PubMed]

Konrad, J.

C. Stiller and J. Konrad, "Estimating motion in image sequences--a tutorial on modeling and computation of 2D motion," IEEE Signal Process. Mag. 16(4), 70-91 (1999).
[CrossRef]

Kopeika, N. S.

G. Hochman, Y. Yitzhaky, N. S. Kopeika, Y. Lauber, M. Citroen, and A. Stern, "Restoration of images captured by a staggered time delay and integration camera in the presence of mechanical vibrations," Appl. Opt. 43, 4345-4354 (2004).
[CrossRef] [PubMed]

S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
[CrossRef]

A. Stern and N. S. Kopeika, "Motion-distorted composite-frame restoration," Appl. Opt. 38, 757-765 (1999).
[CrossRef]

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering Press, 1998).

Kouris, K.

C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
[CrossRef] [PubMed]

Kuntsevitsky, A.

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

Lauber, Y.

Papoulis, P.

P. Papoulis, Probability, Random Variables, and Stochastic Processes, 2nd ed. (McGraw-Hill, 1985), Chap. 13, pp. 407-476.

Raiter, S.

S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
[CrossRef]

Rotman, S. R.

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

Sahakian, A. V.

C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
[CrossRef] [PubMed]

Soltanian-Zadeh, H.

H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
[CrossRef] [PubMed]

Stern, A.

Stiller, C.

C. Stiller and J. Konrad, "Estimating motion in image sequences--a tutorial on modeling and computation of 2D motion," IEEE Signal Process. Mag. 16(4), 70-91 (1999).
[CrossRef]

Taylor, F. J.

G. Zelniker and F. J. Taylor, Advanced Digital Signal Processing (Marcel Dekker, 1994).

Tekalp, A.

A. Tekalp, Digital Video Processing (Prentice Hall, 1995).

Wasserblat, M.

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

Windham, J. P.

H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
[CrossRef] [PubMed]

Yagle, A. E.

H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
[CrossRef] [PubMed]

Yitzhaky, Y.

Zelniker, G.

G. Zelniker and F. J. Taylor, Advanced Digital Signal Processing (Marcel Dekker, 1994).

Appl. Opt. (2)

IEEE Signal Process. Mag. (1)

C. Stiller and J. Konrad, "Estimating motion in image sequences--a tutorial on modeling and computation of 2D motion," IEEE Signal Process. Mag. 16(4), 70-91 (1999).
[CrossRef]

IEEE Trans. Image Process. (1)

H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, "A multidimensional nonlinear edge-preserving filter for magnetic resonance image restoration," IEEE Trans. Image Process. 4, 147-161 (1995).
[CrossRef] [PubMed]

IEEE Trans. Med. Imaging (2)

C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, "Image sequence filtering in quantum-limited noise with applications to low-dose fluoroscopy," IEEE Trans. Med. Imaging 12, 610-621 (1993).
[CrossRef] [PubMed]

C. Charalambous, F. K. Ghaddar, and K. Kouris, "Two iterative image restoration algorithms with applications to nuclear medicine," IEEE Trans. Med. Imaging 11, 2-8 (1992).
[CrossRef] [PubMed]

Opt. Eng. (2)

O. Hadar, A. Kuntsevitsky, M. Wasserblat, N. S. Kopeika, and S. R. Rotman, "Automatic target recognition during sensor motion and vibration," Opt. Eng. 34, 3062-3068 (1995).
[CrossRef]

S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, "Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems," Opt. Eng. 42, 3253-3264 (2003).
[CrossRef]

Other (8)

G. C. Holst, CCD Arrays Cameras and Displays (SPIE Optical Engineering Press, 1998).

D. F. Barbe, "Time delay and integration image sensors" in Solid State Imaging, P. G. Jespers, F. van de Wiele, and M. H. White, eds. (Noordhoff, 1976), pp. 659-671.

N. S. Kopeika, A System Engineering Approach to Imaging (SPIE Optical Engineering Press, 1998).

A. Tekalp, Digital Video Processing (Prentice Hall, 1995).

Recommendation H.261: Video Codec for Audiovisual Services at p × 64 kbits/s." Rep. COM XV-R 37-E (CCITT, 1989).

B. Jähne, Spatio-Temporal Image Processing (Spring-Verlag, 1993).

G. Zelniker and F. J. Taylor, Advanced Digital Signal Processing (Marcel Dekker, 1994).

P. Papoulis, Probability, Random Variables, and Stochastic Processes, 2nd ed. (McGraw-Hill, 1985), Chap. 13, pp. 407-476.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1
Fig. 1

Sensor structure illustration: (a) TDI sensor, (b) staggered TDI. M is the number of sensor rows.

Fig. 2
Fig. 2

Scanning process to acquire one pixel with TDI.

Fig. 3
Fig. 3

Illustration of the TDI imaging process.

Fig. 4
Fig. 4

Space variance of motion image degradation. Two blocks of the image are enlarged to demonstrate the different types of distortion in different parts of the image.

Fig. 5
Fig. 5

Schematic representation of horizontal registration algorithm; horizontal shift between image fields and appropriate filter coefficients. From the K × (N∕2) block (matrix G) of the odd field (upper left) and the ith column of the even field (upper right) the coefficient parameters of the shifting filter h i are calculated. Then filtration of the G matrix performed to estimate the ith column in the field F 2 to the appropriate column in the field F 1 . Three examples of filters are shown for horizontal displacements s of 0, 1, and −2 pixels.

Fig. 6
Fig. 6

Illustration of the vertical relative motion estimation. Two fields of the image are shown schematically at the top. At the bottom the process of building a matrix G for vertical relative motion estimation is illustrated.

Fig. 7
Fig. 7

(a), (c) Estimated filter coefficients (vertical) as a function of time (horizontal) and (b), (d) the same parameters in the shift in pixels units; white pixels represent the shift of the column relative to the appropriate column of the second field.

Fig. 8
Fig. 8

Algorithm diagram.

Fig. 9
Fig. 9

(a) Original image, (b) image after simulation of staggered TDI imaging process, and (c) reconstructed image.

Fig. 10
Fig. 10

Relative vertical and horizontal motion function estimation. (a) Vertical and (b) horizontal relative motion between image fields. Vertical axis units are pixels; horizontal axis units are column numbers, which are relative to time because of constant-velocity horizontal scanning.

Fig. 11
Fig. 11

Dependence of horizontal and vertical motion estimation RMSE as a function of SNR. It can be seen that the horizontal motion estimation performance is more sensitive to the amount of noise added. Vertical estimation precision remains almost unchanged. Horizontal RMSE is similar to the horizontal RMSE at low noise levels. At high noise levels the horizontal RMSE increases but remains less than approximately 0.7 pixels even for a very high amount of noise added.

Fig. 12
Fig. 12

Restoration of a motion-degraded staggered TDI image. (a) Degraded and (b) restored images.

Fig. 13
Fig. 13

Registration of an electronic zoom image fragment: (a) original image, (b) improved image by four-field registration, and (c) interpolated image.

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

g ( k , m ) = 1 K t k , m t e t k , m [ ( x , y ) R h ( x 1 , y 1 ) ( x 1 , y 1 ) = Γ k , m ( t ) h 0 ( x x 1 , y y 1 ) × f ( x , y ) d x d y ] d t ,
K = t e h 0 ( x , y ) d x d y .
y ^ [ n ] = ω 1 x 1 [ n ] + ω 2 x 2 [ n ] + + ω m x m [ n ] , 1 n N .
E = n = 1 N | e [ i ] | 2 ,
X = [ x 1 ( 1 ) x 2 ( 1 ) x m ( 1 ) x 1 ( 2 ) x 2 ( 2 ) x m ( 2 ) x 1 ( N ) x 2 ( N ) x m ( N ) ] ,     y = [ y ( 1 ) y ( 2 ) y ( N ) ] ,     ω = [ ω 1 ω 2 ω m ] ,     e = [ e ( 1 ) e ( 2 ) e ( N ) ] ,
E = e H e = y H y y H X ω ω H X X H y + ω H X H X ω ,
E ω = X H y X H y = 2 X H X ω = 0 ,
X H X ω = X H y .
ω = ( X H X ) 1 X H y ,
E min = y H y y H X ω .
( y H y y H X X H y X H X ) ( 1 ω ) = ( E min O m ) .
A H X B = ( Σ O O O ) ,
X # = A ( Σ 1 O O O ) B H ,
X # = ( X H X ) 1 X H ,
X # = X H ( X X H ) 1 .

Metrics