Abstract

In this paper, we present a computationally efficient video restoration algorithm to address both blur and noise for a Nyquist sampled imaging system. The proposed method utilizes a temporal Kalman filter followed by a correlation-model based spatial adaptive Wiener filter (AWF). The Kalman filter employs an affine background motion model and novel process-noise variance estimate. We also propose and demonstrate a new multidelay temporal Kalman filter designed to more robustly treat local motion. The AWF is a spatial operation that performs deconvolution and adapts to the spatially varying residual noise left in the Kalman filter stage. In image areas where the temporal Kalman filter is able to provide significant noise reduction, the AWF can be aggressive in its deconvolution. In other areas, where less noise reduction is achieved with the Kalman filter, the AWF balances the deconvolution with spatial noise reduction. In this way, the Kalman filter and AWF work together effectively, but without the computational burden of full joint spatiotemporal processing. We also propose a novel hybrid system that combines a temporal Kalman filter and BM3D processing. To illustrate the efficacy of the proposed methods, we test the algorithms on both simulated imagery and video collected with a visible camera.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. Holst and T. Lomheim, CMOS/CCD Sensors and Camera Systems, 2nd ed. (SPIE, 2011).
  2. J. Goodman, Introduction to Fourier Optics, 3rd ed. (Roberts & Company, 1968).
  3. R. D. Fiete, “Image quality and λ FN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
    [CrossRef]
  4. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
    [CrossRef]
  5. A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
    [CrossRef]
  6. Y. Han and R. Chen, “Efficient video denoising based on dynamic nonlocal means,” Image Vis. Comput. 30, 78–85 (2012).
    [CrossRef]
  7. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
    [CrossRef]
  8. K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3d transform-domain collaborative filtering,” in Proceedings of the 15th European Signal Processing Conference (2007), Vol. 1, p. 145–149.
  9. M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
    [CrossRef]
  10. J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
    [CrossRef]
  11. J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
    [CrossRef]
  12. J. Kim and J. W. Woods, “Spatiotemporal adaptive 3-d Kalman filter for video,” IEEE Trans. Image Process. 6, 414–424 (1997).
    [CrossRef]
  13. R. Dugad and N. Ahuja, “Video denoising by combining Kalman and Wiener estimates,” in Proceedings of the International Conference on Image Processing (ICIP), Kobe (1999), pp. 152–156.
  14. R. D. Turney, A. M. Reza, and J. G. Delva, “FPGA implementation of adaptive temporal Kalman filter for real time video filtering,” in Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE, 1999), Vol. 4, pp. 2231–2234.
  15. F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
    [CrossRef]
  16. L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
    [CrossRef]
  17. R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16, 2953–2964 (2007).
    [CrossRef]
  18. R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging,” Opt. Express 19, 26208–26231 (2011).
    [CrossRef]
  19. R. C. Hardie and K. J. Barnard, “Fast super-resolution using an adaptive Wiener filter with robustness to local motion,” Opt. Express 20, 21053–21073 (2012).
    [CrossRef]
  20. G. Bishop and G. Welch, “An introduction to the Kalman filter,” Proc. SIGGRAPH, Course 8 (2001).
  21. M. Makitalo and A. Foi, “Poisson-Gaussian denoising using the exact unbiased inverse of the generalized anscombe transformation,” in Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2012), pp. 1081–1084.
  22. S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
    [CrossRef]
  23. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
    [CrossRef]
  24. M. A. Makar and H. K. Raghunandan, “Wiener and Kalman filters for denoising video signals,” EE378 Class Project, Spring (2008).
  25. S. E. Reichenbach and S. K. Park, “Small convolution kernels for high-fidelity image restoration,” IEEE Trans. Signal Process. 39, 2263–2274 (1991).
    [CrossRef]
  26. S. G. Johnson and M. Frigo, “A modified split-radix fft with fewer arithmetic operations,” IEEE Trans. Signal Process. 55, 111–119 (2007).
    [CrossRef]
  27. M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for Poisson-Gaussian noise,” IEEE Trans. Image Process. 22, 91–103 (2013).
    [CrossRef]

2013 (1)

M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for Poisson-Gaussian noise,” IEEE Trans. Image Process. 22, 91–103 (2013).
[CrossRef]

2012 (3)

Y. Han and R. Chen, “Efficient video denoising based on dynamic nonlocal means,” Image Vis. Comput. 30, 78–85 (2012).
[CrossRef]

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

R. C. Hardie and K. J. Barnard, “Fast super-resolution using an adaptive Wiener filter with robustness to local motion,” Opt. Express 20, 21053–21073 (2012).
[CrossRef]

2011 (1)

2009 (1)

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

2008 (1)

A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
[CrossRef]

2007 (3)

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16, 2953–2964 (2007).
[CrossRef]

S. G. Johnson and M. Frigo, “A modified split-radix fft with fewer arithmetic operations,” IEEE Trans. Signal Process. 55, 111–119 (2007).
[CrossRef]

2006 (2)

F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
[CrossRef]

S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
[CrossRef]

2003 (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

1999 (1)

R. D. Fiete, “Image quality and λ FN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

1998 (1)

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
[CrossRef]

1997 (1)

J. Kim and J. W. Woods, “Spatiotemporal adaptive 3-d Kalman filter for video,” IEEE Trans. Image Process. 6, 414–424 (1997).
[CrossRef]

1995 (1)

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

1991 (1)

S. E. Reichenbach and S. K. Park, “Small convolution kernels for high-fidelity image restoration,” IEEE Trans. Signal Process. 39, 2263–2274 (1991).
[CrossRef]

1983 (1)

J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
[CrossRef]

Ahuja, N.

R. Dugad and N. Ahuja, “Video denoising by combining Kalman and Wiener estimates,” in Proceedings of the International Conference on Image Processing (ICIP), Kobe (1999), pp. 152–156.

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
[CrossRef]

Barnard, K. J.

Biemond, J.

J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
[CrossRef]

Bishop, G.

G. Bishop and G. Welch, “An introduction to the Kalman filter,” Proc. SIGGRAPH, Course 8 (2001).

Bognar, J. G.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
[CrossRef]

Boracchi, G.

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

Brailean, J. C.

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Buades, A.

A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
[CrossRef]

Chen, R.

Y. Han and R. Chen, “Efficient video denoising based on dynamic nonlocal means,” Image Vis. Comput. 30, 78–85 (2012).
[CrossRef]

Coll, B.

A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
[CrossRef]

Dabov, K.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3d transform-domain collaborative filtering,” in Proceedings of the 15th European Signal Processing Conference (2007), Vol. 1, p. 145–149.

Delva, J. G.

R. D. Turney, A. M. Reza, and J. G. Delva, “FPGA implementation of adaptive temporal Kalman filter for real time video filtering,” in Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE, 1999), Vol. 4, pp. 2231–2234.

Dugad, R.

R. Dugad and N. Ahuja, “Video denoising by combining Kalman and Wiener estimates,” in Proceedings of the International Conference on Image Processing (ICIP), Kobe (1999), pp. 152–156.

Efstratiadis, S.

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Egiazarian, K.

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3d transform-domain collaborative filtering,” in Proceedings of the 15th European Signal Processing Conference (2007), Vol. 1, p. 145–149.

Elad, M.

S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
[CrossRef]

Farsiu, S.

S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
[CrossRef]

Fieguth, P.

F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
[CrossRef]

Fiete, R. D.

R. D. Fiete, “Image quality and λ FN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

Foi, A.

M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for Poisson-Gaussian noise,” IEEE Trans. Image Process. 22, 91–103 (2013).
[CrossRef]

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

M. Makitalo and A. Foi, “Poisson-Gaussian denoising using the exact unbiased inverse of the generalized anscombe transformation,” in Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2012), pp. 1081–1084.

K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3d transform-domain collaborative filtering,” in Proceedings of the 15th European Signal Processing Conference (2007), Vol. 1, p. 145–149.

Frigo, M.

S. G. Johnson and M. Frigo, “A modified split-radix fft with fewer arithmetic operations,” IEEE Trans. Signal Process. 55, 111–119 (2007).
[CrossRef]

Gerbrands, J.

J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
[CrossRef]

Goodman, J.

J. Goodman, Introduction to Fourier Optics, 3rd ed. (Roberts & Company, 1968).

Han, Y.

Y. Han and R. Chen, “Efficient video denoising based on dynamic nonlocal means,” Image Vis. Comput. 30, 78–85 (2012).
[CrossRef]

Hardie, R.

R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16, 2953–2964 (2007).
[CrossRef]

Hardie, R. C.

Holst, G.

G. Holst and T. Lomheim, CMOS/CCD Sensors and Camera Systems, 2nd ed. (SPIE, 2011).

Jin, F.

F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
[CrossRef]

Johnson, S. G.

S. G. Johnson and M. Frigo, “A modified split-radix fft with fewer arithmetic operations,” IEEE Trans. Signal Process. 55, 111–119 (2007).
[CrossRef]

Jovanov, L.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Kang, M. G.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Katkovnik, V.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

Katsaggelos, A. K.

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Kerre, E.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Kim, J.

J. Kim and J. W. Woods, “Spatiotemporal adaptive 3-d Kalman filter for video,” IEEE Trans. Image Process. 6, 414–424 (1997).
[CrossRef]

Kleihorst, R. P.

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Lagendijk, R. L.

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Lomheim, T.

G. Holst and T. Lomheim, CMOS/CCD Sensors and Camera Systems, 2nd ed. (SPIE, 2011).

Maggioni, M.

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

Makar, M. A.

M. A. Makar and H. K. Raghunandan, “Wiener and Kalman filters for denoising video signals,” EE378 Class Project, Spring (2008).

Makitalo, M.

M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for Poisson-Gaussian noise,” IEEE Trans. Image Process. 22, 91–103 (2013).
[CrossRef]

M. Makitalo and A. Foi, “Poisson-Gaussian denoising using the exact unbiased inverse of the generalized anscombe transformation,” in Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2012), pp. 1081–1084.

Milanfar, P.

S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
[CrossRef]

Morel, J.-M.

A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
[CrossRef]

Munteanu, A.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Ordonez, R.

Park, M. K.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Park, S. K.

S. E. Reichenbach and S. K. Park, “Small convolution kernels for high-fidelity image restoration,” IEEE Trans. Signal Process. 39, 2263–2274 (1991).
[CrossRef]

Philips, W.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Pizurica, A.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Raghunandan, H. K.

M. A. Makar and H. K. Raghunandan, “Wiener and Kalman filters for denoising video signals,” EE378 Class Project, Spring (2008).

Reichenbach, S. E.

S. E. Reichenbach and S. K. Park, “Small convolution kernels for high-fidelity image restoration,” IEEE Trans. Signal Process. 39, 2263–2274 (1991).
[CrossRef]

Reza, A. M.

R. D. Turney, A. M. Reza, and J. G. Delva, “FPGA implementation of adaptive temporal Kalman filter for real time video filtering,” in Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE, 1999), Vol. 4, pp. 2231–2234.

Rieske, J.

J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
[CrossRef]

Schelkens, P.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Schulte, S.

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

Turney, R. D.

R. D. Turney, A. M. Reza, and J. G. Delva, “FPGA implementation of adaptive temporal Kalman filter for real time video filtering,” in Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE, 1999), Vol. 4, pp. 2231–2234.

Watson, E. A.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
[CrossRef]

Welch, G.

G. Bishop and G. Welch, “An introduction to the Kalman filter,” Proc. SIGGRAPH, Course 8 (2001).

Winger, L.

F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
[CrossRef]

Woods, J. W.

J. Kim and J. W. Woods, “Spatiotemporal adaptive 3-d Kalman filter for video,” IEEE Trans. Image Process. 6, 414–424 (1997).
[CrossRef]

EURASIP J. Adv. Signal Process. (2)

F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” EURASIP J. Adv. Signal Process. 2006, 072705 (2006).
[CrossRef]

S. Farsiu, M. Elad, and P. Milanfar, “Video-to-video dynamic super-resolution for grayscale and color sequences,” EURASIP J. Adv. Signal Process. 2006, 061859 (2006).
[CrossRef]

IEEE Signal Process. Mag. (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

IEEE Trans. Acoust., Speech, Signal Process. (1)

J. Biemond, J. Rieske, and J. Gerbrands, “A fast Kalman filter for images degraded by both blur and noise,” IEEE Trans. Acoust., Speech, Signal Process. 31, 1248–1256 (1983).
[CrossRef]

IEEE Trans. Circuits Syst. Video Technol. (1)

L. Jovanov, A. Pizurica, S. Schulte, P. Schelkens, A. Munteanu, E. Kerre, and W. Philips, “Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods,” IEEE Trans. Circuits Syst. Video Technol. 19, 417–421 (2009).
[CrossRef]

IEEE Trans. Image Process. (5)

R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16, 2953–2964 (2007).
[CrossRef]

J. Kim and J. W. Woods, “Spatiotemporal adaptive 3-d Kalman filter for video,” IEEE Trans. Image Process. 6, 414–424 (1997).
[CrossRef]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
[CrossRef]

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
[CrossRef]

M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for Poisson-Gaussian noise,” IEEE Trans. Image Process. 22, 91–103 (2013).
[CrossRef]

IEEE Trans. Signal Process. (2)

S. E. Reichenbach and S. K. Park, “Small convolution kernels for high-fidelity image restoration,” IEEE Trans. Signal Process. 39, 2263–2274 (1991).
[CrossRef]

S. G. Johnson and M. Frigo, “A modified split-radix fft with fewer arithmetic operations,” IEEE Trans. Signal Process. 55, 111–119 (2007).
[CrossRef]

Image Vis. Comput. (1)

Y. Han and R. Chen, “Efficient video denoising based on dynamic nonlocal means,” Image Vis. Comput. 30, 78–85 (2012).
[CrossRef]

Int. J. Comput. Vis. (1)

A. Buades, B. Coll, and J.-M. Morel, “Nonlocal image and movie denoising,” Int. J. Comput. Vis. 76, 123–139 (2008).
[CrossRef]

Opt. Eng. (2)

R. D. Fiete, “Image quality and λ FN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37, 247–260 (1998).
[CrossRef]

Opt. Express (2)

Proc. IEEE (1)

J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: a review,” Proc. IEEE 83, 1272–1292 (1995).
[CrossRef]

Other (8)

M. A. Makar and H. K. Raghunandan, “Wiener and Kalman filters for denoising video signals,” EE378 Class Project, Spring (2008).

K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3d transform-domain collaborative filtering,” in Proceedings of the 15th European Signal Processing Conference (2007), Vol. 1, p. 145–149.

G. Holst and T. Lomheim, CMOS/CCD Sensors and Camera Systems, 2nd ed. (SPIE, 2011).

J. Goodman, Introduction to Fourier Optics, 3rd ed. (Roberts & Company, 1968).

R. Dugad and N. Ahuja, “Video denoising by combining Kalman and Wiener estimates,” in Proceedings of the International Conference on Image Processing (ICIP), Kobe (1999), pp. 152–156.

R. D. Turney, A. M. Reza, and J. G. Delva, “FPGA implementation of adaptive temporal Kalman filter for real time video filtering,” in Proceedings of the 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE, 1999), Vol. 4, pp. 2231–2234.

G. Bishop and G. Welch, “An introduction to the Kalman filter,” Proc. SIGGRAPH, Course 8 (2001).

M. Makitalo and A. Foi, “Poisson-Gaussian denoising using the exact unbiased inverse of the generalized anscombe transformation,” in Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2012), pp. 1081–1084.

Supplementary Material (7)

» Media 1: MOV (4241 KB)     
» Media 2: MOV (4385 KB)     
» Media 3: MOV (3808 KB)     
» Media 4: MOV (3770 KB)     
» Media 5: MOV (2651 KB)     
» Media 6: MOV (3906 KB)     
» Media 7: MOV (4527 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1.
Fig. 1.

Observation model used for the Kalman filter development.

Fig. 2.
Fig. 2.

Proposed video restoration using affine motion model based temporal Kalman filter followed by AWF. The AWF performs deconvolution and adapts to the spatially varying residual noise variance from the Kalman filter.

Fig. 3.
Fig. 3.

Detailed block diagram for the temporal Kalman filter.

Fig. 4.
Fig. 4.

PSFs for camera and AWF. (a) Impulse invariant PSF of the camera, (b) AWF impulse response with SNR=0.1, (c) SNR=36.45, and (d) SNR=400.

Fig. 5.
Fig. 5.

Imagery generated to test the various algorithms. (a) The original frame; (b) Poisson–Gaussian noise added to the blurred frame with α=5.90, σηi2=3222.99, and β=0.

Fig. 6.
Fig. 6.

PSNR results on the filtered simulated imagery.

Fig. 7.
Fig. 7.

Final frame of the filtered simulated imagery. (a) BM3D-Wiener, (b) VBM3D-Wiener (Media 1), (c) Kalman-BM3D-Wiener, (d) Kalman–Wiener, (e) Kalman-AWF (Media 2), and (f) MDKF-AWF.

Fig. 8.
Fig. 8.

Region of interest (ROI) of Kalman-AWF and BM3D-Wiener restored image. BM3D-Wiener results listed as (ϕ, PSNR). (a) Kalman-AWF result with PSNR=28.07, (b) (0.25, 24.34), (c) (0.50, 24.51), (d) (0.75, 24.88), (e) (1.05, 25.08), and (f) (1.25, 24.87).

Fig. 9.
Fig. 9.

ROI of (a) original noisy frame and (b) MDKF-AWF filtering result with fast local motion. A ROI of the local motion object for the (c) Kalman-AWF (Media 3), and (d) MDKF-AWF (Media 4, Media 5).

Fig. 10.
Fig. 10.

Final frame of the processed outdoor data imagery. (a) Original frame, (b) BM3D-Wiener, (c) VBM3D-Wiener (Media 6), (d) Kalman-BM3D-Wiener, (e) Kalman–Wiener, and (f) Kalman-AWF (Media 7).

Fig. 11.
Fig. 11.

Final frame ROI of the processed outdoor data imagery. (a) Original frame; (b) BM3D-Wiener, ϕ=1; (c) BM3D-Wiener, ϕ=0.6; (d) Kalman-AWF.

Tables (1)

Tables Icon

Table 1. Run Times for the Six Test Algorithms over 100 Frames

Equations (32)

Equations on this page are rendered with MathJax. Learn more.

yn=Fnyn1+qn,
zn=yn+vn=Hxn+vn,
z˜n=αpn+β1+η,
H(u,v)=Hdif(u,v)Hdet(u,v),
Hdif(u,v)={2π[cos1(ω)ω1ω2]ω<10else,
y¯n=Fny^n1,
E¯n=FnEn1FnT+Qn,
Kn=E¯n(E¯n+Vn)1.
y^n=(IKn)y¯n+Knzn.
En=(IKn)E¯n=E¯nKnE¯n.
ε¯n=Fnεn1+q˜n,
kn,i=ε¯n,iε¯n,i+σvn,i2,
y^n,i=(1kn,i)y¯n,i+kn,izn,i,
εn,i=(1kn,i)ε¯n,i,
qn=G(zny¯n),
σ^qn,i2=κ(1|Wi|jWiq^n,j2)γ.
y¯n(mj)=Fn,mjy^nmj,
ε¯n(mj)=Fn,mjεnmj+q˜n.
kn,i(mj)=ε¯n,i(mj)ε¯n,i(mj)+σvn,i2.
y^n,i(mj)=(1kn,i(mj))y¯n,i(mj)+kn,i(mj)zn,i.
εn,i(mj)=(1kn,i(mj))ε¯n,i(mj).
mn,i=argminmmJ[εn,i(m)].
y^n,i=y^n,i(mn,i),
εn,i=εn,i(mn,i).
y^n=Hxn+en.
x^n,i=wn,iTy^n[i],
wn,i=Rn,i1pn,i,
Rn,i=E{y^n[i]y^n[i]T},
pn,i=E{xn,iy^n[i]}.
rxx(n1,n2)=σx2ρn12+n22,
34(K+1)92Nlog2N
σA=ϕσT.

Metrics