Abstract

We present a new adaptive Wiener filter (AWF) super-resolution (SR) algorithm that employs a global background motion model but is also robust to limited local motion. The AWF relies on registration to populate a common high resolution (HR) grid with samples from several frames. A weighted sum of local samples is then used to perform nonuniform interpolation and image restoration simultaneously. To achieve accurate subpixel registration, we employ a global background motion model with relatively few parameters that can be estimated accurately. However, local motion may be present that includes moving objects, motion parallax, or other deviations from the background motion model. In our proposed robust approach, pixels from frames other than the reference that are inconsistent with the background motion model are detected and excluded from populating the HR grid. Here we propose and compare several local motion detection algorithms. We also propose a modified multiscale background registration method that incorporates pixel selection at each scale to minimize the impact of local motion. We demonstrate the efficacy of the new robust SR methods using several datasets, including airborne infrared data with moving vehicles and a ground resolution pattern for objective resolution analysis.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
    [CrossRef]
  2. R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
    [CrossRef]
  3. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proceedings of International Joint Conference on Artificial Intelligence (Vancouver, 1981), pp. 674–679.
  4. M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
    [CrossRef]
  5. M. Irani and S. Peleg, “Improving resolution by image registration,” CHIP: Graph. Models Image Process.53, 231–239 (1991).
    [CrossRef]
  6. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
    [CrossRef]
  7. R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive wiener filter and its application to airborne imaging,” Opt. Express19, 26208–26231 (2011).
    [CrossRef]
  8. M. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Processing13, 1185–1199 (2004).
    [CrossRef]
  9. M. D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Processing15, 1413–1428 (2006).
    [CrossRef]
  10. R. C. Hardie, “Super-resolution using adaptive wiener filters,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 35–61.
  11. R. C. Hardie, “A fast super-resolution algorithm using an adaptive wiener filter,” IEEE Trans. Image Processing16, 2953–2964 (2007).
    [CrossRef]
  12. B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
    [CrossRef]
  13. M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
    [CrossRef]
  14. T. R. Tuinstra and R. C. Hardie, “High resolution image reconstruction from digital video by exploitation on non-global motion,” Opt. Eng.38 (1999).
    [CrossRef]
  15. M. Tanaka and M. Okutomi, “Towards robust reconstruction-based superresolution,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 219–246.
  16. A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
    [CrossRef]
  17. M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
    [CrossRef]
  18. Z. Zhang and R. Wang, “Robust image superresolution method to handle localized motion outliers,” Opt. Eng.48, 077005 (2009).
    [CrossRef]
  19. J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
    [CrossRef]
  20. N. A. El-Yamany and P. E. Papamichalis, “Robust color image superresolution: an adaptive m-estimation framework,” J. Image Video Process. 2008, 16:1–16:12 (2008).
  21. M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
    [CrossRef]
  22. Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
    [CrossRef]
  23. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
    [CrossRef]
  24. S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
    [CrossRef]
  25. F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).
  26. R. D. Fiete, “Image quality and λ FN/ p for remote sensing systems,” Opt. Eng.38, 1229–1240 (1999).
    [CrossRef]
  27. R. Franzen, “Kodak lossless true color image suite,” http://r0k.us/graphics/kodak .
  28. P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Communications31, 532–540 (1983).
    [CrossRef]
  29. A. C. Bovik, The Essential Guide to Video Processing (Academic Press, 2009), 2nd ed.
  30. S. Coles, An introduction to statistical modeling of extreme values, Springer Series in Statistics (Springer-Verlag, London, 2001).

2011

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive wiener filter and its application to airborne imaging,” Opt. Express19, 26208–26231 (2011).
[CrossRef]

2010

A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
[CrossRef]

2009

Z. Zhang and R. Wang, “Robust image superresolution method to handle localized motion outliers,” Opt. Eng.48, 077005 (2009).
[CrossRef]

2008

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

N. A. El-Yamany and P. E. Papamichalis, “Robust color image superresolution: an adaptive m-estimation framework,” J. Image Video Process. 2008, 16:1–16:12 (2008).

2007

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
[CrossRef]

R. C. Hardie, “A fast super-resolution algorithm using an adaptive wiener filter,” IEEE Trans. Image Processing16, 2953–2964 (2007).
[CrossRef]

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
[CrossRef]

2006

M. D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Processing15, 1413–1428 (2006).
[CrossRef]

Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
[CrossRef]

2005

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
[CrossRef]

2004

M. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Processing13, 1185–1199 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

2003

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
[CrossRef]

2000

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

1999

T. R. Tuinstra and R. C. Hardie, “High resolution image reconstruction from digital video by exploitation on non-global motion,” Opt. Eng.38 (1999).
[CrossRef]

R. D. Fiete, “Image quality and λ FN/ p for remote sensing systems,” Opt. Eng.38, 1229–1240 (1999).
[CrossRef]

1998

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

1991

M. Irani and S. Peleg, “Improving resolution by image registration,” CHIP: Graph. Models Image Process.53, 231–239 (1991).
[CrossRef]

1983

P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Communications31, 532–540 (1983).
[CrossRef]

Adelson, E.

P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Communications31, 532–540 (1983).
[CrossRef]

Alam, M. S.

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

Barnard, K. J.

R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive wiener filter and its application to airborne imaging,” Opt. Express19, 26208–26231 (2011).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).

Barner, K. E.

R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
[CrossRef]

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
[CrossRef]

Baxley, F. O.

F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).

Bicknell, M. A.

F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).

Bognar, J. G.

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

Bovik, A. C.

A. C. Bovik, The Essential Guide to Video Processing (Academic Press, 2009), 2nd ed.

Burt, P.

P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Communications31, 532–540 (1983).
[CrossRef]

Chung, D.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Coles, S.

S. Coles, An introduction to statistical modeling of extreme values, Springer Series in Statistics (Springer-Verlag, London, 2001).

de Lange, D.-J. J.

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

Dijk, J.

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

Elad, M.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

El-Yamany, N. A.

N. A. El-Yamany and P. E. Papamichalis, “Robust color image superresolution: an adaptive m-estimation framework,” J. Image Video Process. 2008, 16:1–16:12 (2008).

Farsiu, S.

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

Fiete, R. D.

R. D. Fiete, “Image quality and λ FN/ p for remote sensing systems,” Opt. Eng.38, 1229–1240 (1999).
[CrossRef]

Han, D.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Hardie, R. C.

R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive wiener filter and its application to airborne imaging,” Opt. Express19, 26208–26231 (2011).
[CrossRef]

R. C. Hardie, “A fast super-resolution algorithm using an adaptive wiener filter,” IEEE Trans. Image Processing16, 2953–2964 (2007).
[CrossRef]

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
[CrossRef]

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

T. R. Tuinstra and R. C. Hardie, “High resolution image reconstruction from digital video by exploitation on non-global motion,” Opt. Eng.38 (1999).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

R. C. Hardie, “Super-resolution using adaptive wiener filters,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 35–61.

F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).

Irani, M.

M. Irani and S. Peleg, “Improving resolution by image registration,” CHIP: Graph. Models Image Process.53, 231–239 (1991).
[CrossRef]

Ivanovski, Z. A.

Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
[CrossRef]

Kanade, T.

B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proceedings of International Joint Conference on Artificial Intelligence (Vancouver, 1981), pp. 674–679.

Kang, M. G.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
[CrossRef]

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
[CrossRef]

Karam, L. J.

Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
[CrossRef]

Katsaggelos, A. K.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
[CrossRef]

Kim, M.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Ko, H.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Ku, B.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Lucas, B. D.

B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proceedings of International Joint Conference on Artificial Intelligence (Vancouver, 1981), pp. 674–679.

Milanfar, P.

M. D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Processing15, 1413–1428 (2006).
[CrossRef]

M. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Processing13, 1185–1199 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

Narayanan, B.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

Okutomi, M.

M. Tanaka and M. Okutomi, “Towards robust reconstruction-based superresolution,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 219–246.

Ordonez, R.

Panovski, L.

Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
[CrossRef]

Papamichalis, P. E.

N. A. El-Yamany and P. E. Papamichalis, “Robust color image superresolution: an adaptive m-estimation framework,” J. Image Video Process. 2008, 16:1–16:12 (2008).

Park, M. K.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
[CrossRef]

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
[CrossRef]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
[CrossRef]

Peleg, S.

M. Irani and S. Peleg, “Improving resolution by image registration,” CHIP: Graph. Models Image Process.53, 231–239 (1991).
[CrossRef]

Robinson, D.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

Robinson, M. D.

M. D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Processing15, 1413–1428 (2006).
[CrossRef]

M. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Processing13, 1185–1199 (2004).
[CrossRef]

Schultz, R. R.

R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
[CrossRef]

Schutte, K.

A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
[CrossRef]

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

Shao, M.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
[CrossRef]

Shin, H.

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

Tanaka, M.

M. Tanaka and M. Okutomi, “Towards robust reconstruction-based superresolution,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 219–246.

Tuinstra, T. R.

T. R. Tuinstra and R. C. Hardie, “High resolution image reconstruction from digital video by exploitation on non-global motion,” Opt. Eng.38 (1999).
[CrossRef]

van Eekeren, A. W. M.

A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
[CrossRef]

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

van Vliet, L. J.

A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
[CrossRef]

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

Wang, R.

Z. Zhang and R. Wang, “Robust image superresolution method to handle localized motion outliers,” Opt. Eng.48, 077005 (2009).
[CrossRef]

Watson, E. A.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

Yasuda, B. J.

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

Zhang, Z.

Z. Zhang and R. Wang, “Robust image superresolution method to handle localized motion outliers,” Opt. Eng.48, 077005 (2009).
[CrossRef]

CHIP: Graph. Models Image Process.

M. Irani and S. Peleg, “Improving resolution by image registration,” CHIP: Graph. Models Image Process.53, 231–239 (1991).
[CrossRef]

EURASIP J. Adv. Signal Process.

R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution enhancement of digital video,” EURASIP J. Adv. Signal Process.2007, 19–19 (2007).
[CrossRef]

M. Kim, B. Ku, D. Chung, H. Shin, D. Han, and H. Ko, “Robust video super resolution algorithm using measurement validation method and scene change detection,” EURASIP J. Adv. Signal Process.2011, 1–12 (2011). 10.1186/1687-6180-2011-103.
[CrossRef]

IEEE Signal Processing Mag

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Processing Mag. 20, 21–36 (2003).
[CrossRef]

IEEE Trans. Circuits Syst. Video Technol.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Trans. Circuits Syst. Video Technol.17, 621–634 (2007).
[CrossRef]

IEEE Trans. Communications

P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Communications31, 532–540 (1983).
[CrossRef]

IEEE Trans. Image Processing

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super-resolution,” IEEE Trans. Image Processing13, 1327–1344 (2004).
[CrossRef]

R. C. Hardie, “A fast super-resolution algorithm using an adaptive wiener filter,” IEEE Trans. Image Processing16, 2953–2964 (2007).
[CrossRef]

A. W. M. van Eekeren, K. Schutte, and L. J. van Vliet, “Multiframe super-resolution reconstruction of small moving objects,” IEEE Trans. Image Processing19, 2901–2912 (2010).
[CrossRef]

M. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Processing13, 1185–1199 (2004).
[CrossRef]

M. D. Robinson and P. Milanfar, “Statistical performance analysis of super-resolution,” IEEE Trans. Image Processing15, 1413–1428 (2006).
[CrossRef]

IEEE Trans. Instrum. Meas.

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49 (2000).
[CrossRef]

Int. J. Imag. Syst. Tech.

S. Farsiu, S. Farsiu, S. Farsiu, D. Robinson, D. Robinson, M. Elad, M. Elad, P. Milanfar, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Tech.14, 47–57 (2004).
[CrossRef]

J. Image Video Process

N. A. El-Yamany and P. E. Papamichalis, “Robust color image superresolution: an adaptive m-estimation framework,” J. Image Video Process. 2008, 16:1–16:12 (2008).

Opt. Eng.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng.46, 117004 (2007).
[CrossRef]

R. D. Fiete, “Image quality and λ FN/ p for remote sensing systems,” Opt. Eng.38, 1229–1240 (1999).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37, 247–260 (1998).
[CrossRef]

Z. Zhang and R. Wang, “Robust image superresolution method to handle localized motion outliers,” Opt. Eng.48, 077005 (2009).
[CrossRef]

J. Dijk, A. W. M. van Eekeren, K. Schutte, D.-J. J. de Lange, and L. J. van Vliet, “Superresolution reconstruction for moving point target detection,” Opt. Eng.47, 096401 (2008).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44, 107003–1–107003–14 (2005).
[CrossRef]

T. R. Tuinstra and R. C. Hardie, “High resolution image reconstruction from digital video by exploitation on non-global motion,” Opt. Eng.38 (1999).
[CrossRef]

Opt. Express

Proc. SPIE

Z. A. Ivanovski, L. Panovski, and L. J. Karam, “Robust super-resolution based on pixel-level selectivity,” Proc. SPIE6077, 607707 (2006).
[CrossRef]

Other

F. O. Baxley, K. J. Barnard, R. C. Hardie, and M. A. Bicknell, “Flight test results of a rapid step-stare and microscan midwave infrared sensor concept for persistent surveillance,” in Proceedings of MSS Passive Sensors (Orlando, FL, 2010).

R. Franzen, “Kodak lossless true color image suite,” http://r0k.us/graphics/kodak .

A. C. Bovik, The Essential Guide to Video Processing (Academic Press, 2009), 2nd ed.

S. Coles, An introduction to statistical modeling of extreme values, Springer Series in Statistics (Springer-Verlag, London, 2001).

R. C. Hardie, “Super-resolution using adaptive wiener filters,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 35–61.

B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proceedings of International Joint Conference on Artificial Intelligence (Vancouver, 1981), pp. 674–679.

M. Tanaka and M. Okutomi, “Towards robust reconstruction-based superresolution,” in Super-Resolution Imaging, P. Milanfar, ed. (Taylor & Francis/CRC Press, 2010), pp. 219–246.

Supplementary Material (4)

» Media 1: MOV (4141 KB)     
» Media 2: MOV (3792 KB)     
» Media 3: MOV (4066 KB)     
» Media 4: MOV (3975 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1
Fig. 1

Assumed observation model relating a desired 2-D continuous scene, d(x, y), to the observed LR image samples, g, from a set of K frames. This model is the basis for the nonuniform interpolation based SR algorithms, including the AWF SR method.

Fig. 2
Fig. 2

Robust AWF SR with local motion detection. Only pixels consistent with the motion model (i.e., valid pixels) are used to populate the HR grid. Samples on the HR grid are filtered to provide the final AWF SR output.

Fig. 3
Fig. 3

Shift registration error as a function of the image block sizes under test. The error is mean absolute positional error between the actual shifted block and the estimate. As larger blocks are used, the error goes down. The shift registration accuracy is also impacted by the level of aliasing indicated by the Q factor.

Fig. 4
Fig. 4

Robust multiscale registration. A Gaussian pyramid is formed and a Lucas-Kanade based affine registration [3] is applied iteratively at successive scales starting at the lowest resolution. Borders and pixels with the highest registration errors from the preceding level are excluded from the least squares parameter estimation (red contours) to provide robustness to moving objects and local motion.

Fig. 5
Fig. 5

Theoretical MSE between between two registered frames as a function of sub-pixel shift. (a) Highly aliasing imagery (Q=0.21) with bilinear interpolation, (b) no aliasing (Q=2.0) with bilinear interpolation, (c) highly aliasing (Q=0.21) with Wiener filter, (d) no aliasing (Q=2.0) with Wiener filter.

Fig. 6
Fig. 6

Simulated local motion detection study. (a) Simulated LR frame of static chirp pattern showing and a moving ruler. (b) R statistic with K = 12 registered frames, (c) RP statistic (standard deviation of 1.5 LR pixels), (d) absolute FME statistic. The red contours are the detection masks for pd = 0.85 and the green mask is the motion truth.

Fig. 7
Fig. 7

ROC curve for the local motion detection algorithms using the simulated moving ruler data. The benefit of the Gaussian prefilter is made clear here for both the frame error and range methods. Note that FME and MEE are more sensitive at very low FP rates, but the prefilter methods outperform at higher FP rates.

Fig. 8
Fig. 8

Robust AWF SR results for the simulated moving ruler image sequence (L = 4). (a) True HR image, (b) bicubic interpolation of the single LR reference frame, (c) AWF SR with no local motion detection, (d) robust AWF SR (R), (e) robust AWF SR (RP), (f) robust AWF SR (FME). All of the detection methods are set for pd = 0.85.

Fig. 9
Fig. 9

Robust AWF SR results (L = 4) using visible camera sequence using a translational background motion model ( Media 1, Media 2). (a) Bicubic interpolation of the reference frame, (b) AWF SR with no local motion detection, (c) robust AWF SR (R), (d) robust AWF SR (RP), (e) R statistic with detection mask, (f) RP statistic with detection mask. Thresholds are set for pfa = 10−4.

Fig. 10
Fig. 10

Robust AWF SR results for airborne data with 16 Hz frame rate using an affine background motion model (L = 3) ( Media 3, Media 4). (a) Bicubic interpolation of the reference frame, (b) AWF SR with no local motion detection, (c) robust AWF SR (RP), (d) robust AWF SR (FEP), (e) number of input frames contributing in (c), (f) number of input frames contributing in (d). Both detection methods are set for pfa = 10−5.

Fig. 11
Fig. 11

Probability density function estimation for the (a) RP statistic and (b) FEP statistic for the background in the 16Hz flight data sequence from Fig. 10.

Fig. 12
Fig. 12

Robust AWF SR results for airborne data with 50 Hz frame rate using an affine background motion model (L = 3). (a) Bicubic interpolation of the reference frame, (b) robust AWF SR (FEP) for pfa = 10−5, (c) moving vehicle ROI processed with AWF SR with no local motion detection, (d) moving vehicle ROI processed with robust AWF SR (FEP).

Tables (2)

Tables Icon

Table 1 Description of the local motion detection methods considered here.

Tables Icon

Table 2 Quantitative error analysis for the simulated moving ruler data.

Equations (4)

Equations on this page are rendered with MathJax. Learn more.

z ^ i = w ψ ( i ) T g i ,
w ψ ( i ) = R ψ ( i ) 1 p ψ ( i ) ,
r d d ( x , y ) = σ d 2 ρ x 2 + y 2 ,
J ( w ) = E { ( y i w T x i ) 2 } = E { y i 2 } 2 w T p + w T Rw

Metrics