Abstract

Multi-frame super-resolution algorithms offer resolution enhancement for sequences of images with sampling limited resolution. However, classical approaches have been constrained by the accuracy of motion estimation while nonlocal approaches that use implicit motion estimation have attained only modest resolution improvement. In this paper, we propose a new multi-frame optical flow based super-resolution algorithm, which provides significant resolution enhancement for image sequences containing complex motion. The algorithm uses the standard camera image formation model and a variational super-resolution formulation with an anisotropic smoothness term adapting to local image structures. The key elements enabling super-resolution of complex motion patterns are the computation of two-way optical flow between the images and use of two corresponding uncertainty measures that approximate the optical flow interpolation error. Using the developed algorithm, we are able to demonstrate super-resolution of images for which optical flow estimation experiences near breakdown, due to the complexity of the motion patterns and the large magnitudes of the displacements. In comparison, we show that for these images some conventional super-resolution approaches fail, while others including nonlocal super-resolution technique produce distortions and provide lower (1-1.8dB) image quality enhancement compared to the proposed algorithm.

©2013 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Enhancement of imagery of objects with highly dynamic brightness and large rotational motion

Andrey V. Kanaev, Christopher W. Miller, Collin J. Seanor, and Jeremy Murray-Krezan
Appl. Opt. 53(13) C32-C44 (2014)

Fast super-resolution using an adaptive Wiener filter with robustness to local motion

Russell C. Hardie and Kenneth J. Barnard
Opt. Express 20(19) 21053-21073 (2012)

Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging

Russell C. Hardie, Kenneth J. Barnard, and Raul Ordonez
Opt. Express 19(27) 26208-26231 (2011)

More Recommended Articles

References

  • View by:
  • Article Order
  • |
  • Year
  • |
  • Author
  • |
  • Publication
  1. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
    [Crossref]
  2. R. C. Hardie and K. J. Barnard, “Fast super-resolution using an adaptive Wiener filter with robustness to local motion,” Opt. Express 20(19), 21053–21073 (2012).
    [Crossref] [PubMed]
  3. S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
    [Crossref] [PubMed]
  4. M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
    [Crossref]
  5. O. A. Omer and T. Tanaka, “Multiframe image and video super-resolution algorithm with inaccurate motion registration errors rejection,” Proc. SPIE 6822, 682222, 682222-9 (2008).
    [Crossref]
  6. O. A. Omer and T. Tanaka, “Region-based weighted-norm with adaptive regularization for resolution enhancement,” Digit. Signal Process. 21(4), 508–516 (2011).
    [Crossref]
  7. D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
    [Crossref]
  8. M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
    [Crossref]
  9. C. Liu and D. Sun, “A Bayesian approach to adaptive video super resolution,” in Proceeding of Computer Vision and Pattern Recognition Conference (IEEE, 2011), pp. 209–216.
    [Crossref]
  10. A. Danielyan, A. Foi, V. Katkovnik, and K. Egiazarian, “Image and video super-resolution via spatially adaptive block-matching filtering,” in Proceedings of Int. Workshop on Local and Non-Local Approx. in Image Processing, Lausanne, Switzerland, 23–24 August, 2008.
  11. M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
    [Crossref] [PubMed]
  12. H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
    [Crossref] [PubMed]
  13. D. Sun, S. Roth, and M. J. Black, “Secrets of optical flow estimation and their principles,” in Proceeding of Computer Vision and Pattern Recognition Conference (IEEE, 2010), pp. 2432–2439.
    [Crossref]
  14. L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
    [Crossref] [PubMed]
  15. T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011).
    [Crossref] [PubMed]
  16. H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
    [Crossref]
  17. H. Zimmer, A. Bruhn, and J. Weickert, “Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement,” in Proceeding of Eurographics, Llandudno, UK, 11–15 April, 2011.
    [Crossref]
  18. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
    [Crossref]
  19. S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
    [Crossref]
  20. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
    [Crossref] [PubMed]
  21. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
    [Crossref]
  22. A. V. Kanaev, “Confidence measures of optical flow estimation suitable for multi-frame super-resolution,” Proc. SPIE 8399, 839903, 839903-12 (2012).
    [Crossref]
  23. A. Bruhn and J. Weickert, “A confidence measure for variational optic flow methods,” in Geometric Properties for Incomplete Data, (Springer-Verlag, 2006).
  24. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
    [Crossref]

2012 (3)

L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
[Crossref] [PubMed]

A. V. Kanaev, “Confidence measures of optical flow estimation suitable for multi-frame super-resolution,” Proc. SPIE 8399, 839903, 839903-12 (2012).
[Crossref]

R. C. Hardie and K. J. Barnard, “Fast super-resolution using an adaptive Wiener filter with robustness to local motion,” Opt. Express 20(19), 21053–21073 (2012).
[Crossref] [PubMed]

2011 (3)

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011).
[Crossref] [PubMed]

O. A. Omer and T. Tanaka, “Region-based weighted-norm with adaptive regularization for resolution enhancement,” Digit. Signal Process. 21(4), 508–516 (2011).
[Crossref]

2010 (2)

S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
[Crossref] [PubMed]

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

2009 (4)

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

2008 (2)

O. A. Omer and T. Tanaka, “Multiframe image and video super-resolution algorithm with inaccurate motion registration errors rejection,” Proc. SPIE 6822, 682222, 682222-9 (2008).
[Crossref]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

2007 (2)

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
[Crossref]

2004 (2)

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[Crossref]

2003 (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[Crossref]

Baker, S.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Barnard, K. J.

Belekos, S. P.

S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
[Crossref] [PubMed]

Bischof, H.

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

Black, M. J.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Brox, T.

T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011).
[Crossref] [PubMed]

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

Bruhn, A.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

Cremers, D.

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

Dabov, K.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

Egiazarian, K.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

Elad, M.

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

Foi, A.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

Galatsanos, N. P.

S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
[Crossref] [PubMed]

Hardie, R. C.

Jia, J.

L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
[Crossref] [PubMed]

Kanaev, A. V.

A. V. Kanaev, “Confidence measures of optical flow estimation suitable for multi-frame super-resolution,” Proc. SPIE 8399, 839903, 839903-12 (2012).
[Crossref]

Kang, M. G.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
[Crossref]

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[Crossref]

Katkovnik, V.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

Katsaggelos, A. K.

S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
[Crossref] [PubMed]

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
[Crossref]

Lewis, J. P.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Lowe, D. G.

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[Crossref]

Malik, J.

T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011).
[Crossref] [PubMed]

Matsushita, Y.

L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
[Crossref] [PubMed]

Milanfar, P.

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

Mitzel, D.

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

Omer, O. A.

O. A. Omer and T. Tanaka, “Region-based weighted-norm with adaptive regularization for resolution enhancement,” Digit. Signal Process. 21(4), 508–516 (2011).
[Crossref]

O. A. Omer and T. Tanaka, “Multiframe image and video super-resolution algorithm with inaccurate motion registration errors rejection,” Proc. SPIE 6822, 682222, 682222-9 (2008).
[Crossref]

Papenberg, N.

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

Park, M. K.

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
[Crossref]

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[Crossref]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[Crossref]

Pock, T.

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

Protter, M.

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

Rosenhahn, B.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

Roth, S.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Salgado, A.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

Scharstein, D.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Schoenemann, T.

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

Seidel, H.-P.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

Szeliski, R.

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Takeda, H.

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

Tanaka, T.

O. A. Omer and T. Tanaka, “Region-based weighted-norm with adaptive regularization for resolution enhancement,” Digit. Signal Process. 21(4), 508–516 (2011).
[Crossref]

O. A. Omer and T. Tanaka, “Multiframe image and video super-resolution algorithm with inaccurate motion registration errors rejection,” Proc. SPIE 6822, 682222, 682222-9 (2008).
[Crossref]

Unger, M.

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

Valgaerts, L.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

Weickert, J.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

Werlberger, M.

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

Xu, L.

L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
[Crossref] [PubMed]

Zimmer, H.

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

Digit. Signal Process. (1)

O. A. Omer and T. Tanaka, “Region-based weighted-norm with adaptive regularization for resolution enhancement,” Digit. Signal Process. 21(4), 508–516 (2011).
[Crossref]

IEEE Signal Process. Mag. (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[Crossref]

IEEE Trans. Image Process. (4)

S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” IEEE Trans. Image Process. 19(6), 1451–1464 (2010).
[Crossref] [PubMed]

M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009).
[Crossref] [PubMed]

H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-resolution without explicit subpixel motion estimation,” IEEE Trans. Image Process. 18(9), 1958–1975 (2009).
[Crossref] [PubMed]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007).
[Crossref] [PubMed]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012).
[Crossref] [PubMed]

T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011).
[Crossref] [PubMed]

Int. J. Comput. Vis. (2)

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[Crossref]

S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” Int. J. Comput. Vis. 92(1), 1–31 (2011).
[Crossref]

Lect. Notes Comput. Sci. (4)

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” Lect. Notes Comput. Sci. 3024, 25–36 (2004).
[Crossref]

H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, “Complementary optic flow,” Lect. Notes Comput. Sci. 5681, 207–220 (2009).
[Crossref]

D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video super resolution using duality based TV-L1 optical flow,” Lect. Notes Comput. Sci. 5748, 432–441 (2009).
[Crossref]

M. Unger, T. Pock, M. Werlberger, and H. Bischof, “A convex approach for variational super-resolution,” Lect. Notes Comput. Sci. 6376, 313–322 (2010).
[Crossref]

Opt. Eng. (1)

M. K. Park, M. G. Kang, and A. K. Katsaggelos, “Regularized high-resolution image reconstruction considering inaccurate motion information,” Opt. Eng. 46(11), 117004 (2007).
[Crossref]

Opt. Express (1)

Proc. SPIE (3)

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image restoration by sparse 3D transform-domain collaborative filtering,” Proc. SPIE 6812, 681207 (2008).
[Crossref]

O. A. Omer and T. Tanaka, “Multiframe image and video super-resolution algorithm with inaccurate motion registration errors rejection,” Proc. SPIE 6822, 682222, 682222-9 (2008).
[Crossref]

A. V. Kanaev, “Confidence measures of optical flow estimation suitable for multi-frame super-resolution,” Proc. SPIE 8399, 839903, 839903-12 (2012).
[Crossref]

Other (5)

A. Bruhn and J. Weickert, “A confidence measure for variational optic flow methods,” in Geometric Properties for Incomplete Data, (Springer-Verlag, 2006).

H. Zimmer, A. Bruhn, and J. Weickert, “Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement,” in Proceeding of Eurographics, Llandudno, UK, 11–15 April, 2011.
[Crossref]

D. Sun, S. Roth, and M. J. Black, “Secrets of optical flow estimation and their principles,” in Proceeding of Computer Vision and Pattern Recognition Conference (IEEE, 2010), pp. 2432–2439.
[Crossref]

C. Liu and D. Sun, “A Bayesian approach to adaptive video super resolution,” in Proceeding of Computer Vision and Pattern Recognition Conference (IEEE, 2011), pp. 209–216.
[Crossref]

A. Danielyan, A. Foi, V. Katkovnik, and K. Egiazarian, “Image and video super-resolution via spatially adaptive block-matching filtering,” in Proceedings of Int. Workshop on Local and Non-Local Approx. in Image Processing, Lausanne, Switzerland, 23–24 August, 2008.

Supplementary Material (5)

» Media 1: AVI (280 KB)     
» Media 2: AVI (115 KB)     
» Media 3: AVI (348 KB)     
» Media 4: AVI (388 KB)     
» Media 5: AVI (4378 KB)     

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1 The “Walking” image sequence (Media 1). Image courtesy of http://vision.middlebury.edu/flow/.

Download Full Size | PPT Slide | PDF

Fig. 2
Fig. 2 The “Basketball” image sequence (Media 2). Image courtesy of http://vision.middlebury.edu/flow/.

Download Full Size | PPT Slide | PDF

Fig. 3
Fig. 3 The “Foreman” image sequence (Media 3).

Download Full Size | PPT Slide | PDF

Fig. 4
Fig. 4 The “Walking” image sequence experiments with TVL1 OF algorithm. Warped image (in the center) and the warping error measure relative to 8-bit gray scale levels (lower right) which are obtained using OF estimation (upper right) between (a) frames 4 and 5; (b) frames 4 and 8; (c) frames 8 and 4; (d) using negative of OF estimation between frames 4 and 8. OF is color coded such that hue indicates flow direction according to presented scheme and saturation indicates flow magnitude.

Download Full Size | PPT Slide | PDF

Fig. 5
Fig. 5 Reference image of the “Walking” sequence with zoomed in region containing man’s left hand: (a) original; (b) degraded; (c) Lanczos interpolated; and super-resolved with (d) L2 C-SR; (e) L1 C-SR (f) NLS; (g) TVL1 OF DUDE; (h) CNL OF DUDE; (i) MDP OF DUDE; (g) WB OF DUDE.

Download Full Size | PPT Slide | PDF

Fig. 6
Fig. 6 Image PSNR dependence on number of super-resolution iterations. PSNR=10 log 10 ( 255 2 p/ ‖ I H − I ORIG ‖ L2 )[ dB ] , where p is a number of pixels.

Download Full Size | PPT Slide | PDF

Fig. 7
Fig. 7 Reference image of the “Basketball” sequence with zoomed in region containing man’s hands and the ball: (a) original; (b) degraded; (c) Lanczos interpolated; and super-resolved with (d) L2 C-SR; (e) L1 C-SR; (f) NLS; (g) DUDE.

Download Full Size | PPT Slide | PDF

Fig. 8
Fig. 8 Reference image of the “Foreman” sequence: (a) original; (b) degraded; (c) Lanczos interpolated; and super-resolved with (d) C-SR L2; (e) L1 C-SR; (f) NLS; (g) DUDE.

Download Full Size | PPT Slide | PDF

Fig. 9
Fig. 9 High frame rate sequence of images with human motion (Media 4).

Download Full Size | PPT Slide | PDF

Fig. 10
Fig. 10 High frame rate sequence of images (a) degraded and then Lanczos interpolated; and super-resolved using (b) the NLS; (c) the L1 C-SR; (d) the DUDE (Media 5).

Download Full Size | PPT Slide | PDF

Tables (2)

Tables Icon

Table 1 Warping error per pixel

View Table | View all tables in this article
Tables Icon

Table 2 PSNR results for the three test sequences.

View Table | View all tables in this article

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

I L n =DB W n I H + e n ,n=1…N,
E( I H )= ∑ n=1 N ‖ DB W n I H − I L n ‖ +λℛ( ∇ I H ),
ℛ=Ψ( ( ν 1 t ∇ I H ) 2 )+ ( ν 2 t ∇ I H ) 2 ,
J= G ρ 1 ∗[ ∇( G ρ 2 *I ) ( ∇( G ρ 2 *I ) ) t ],
I H ( k+1 ) = I H ( k ) +Δt( ∑ n=1 N W n T B T D T ( I L n −DB W n I H ( k ) ) + + λdiv( [ Ψ ′ ( ( ν 1 t ∇ I H ( k ) ) 2 ) ν 1 ν 1 t + ν 2 ν 2 t ]∇ I H ( k ) ) ),
E( I H )= ∑ n=1 N ( DB W n I H − I L n ) U n ( DB W n I H − I L n ) +λℛ( ∇ I H ),
I H ( k+1 ) = I H ( k ) +Δt( ∑ n=1 N U n→ref W n→ref B D T U ref→n ( I L n −DB W ref→n I H ( k ) ) + +λdiv( [ Ψ ′ ( ( ν 1 t ∇ I H ( k ) ) 2 ) ν 1 ν 1 t + ν 2 ν 2 t ]∇ I H ( k ) ) ),
E opt fl ( u )=( 1− γ 1 ) Ψ z ( I 1 ( r )− I 2 ( r+u ) )+ γ 2 Ψ z ( ∇ I 1 ( r )−∇ I 2 ( r+u ) )+ +α ℛ opt fl ( I 1 , I 2 ,∇u ),
ℛ opt fl =ω ‖ ∇u ‖ L1 ,
ℛ opt fl NL = | u− u ^ | 2 + ∑ r ∑ r ′ ∈ N r exp[ − | r− r ′ | 2 2 σ 1 2 − ( I( r )−I( r ′ ) ) 2 2 σ 2 2 ] o( r ′ ) o( r ) | u ^ (r)− u ^ ( r ′ ) |,
φ=| I 1 ( r )− I 2 ( r+u ) |.
U= G ρ *exp( −φ/a ),

Metrics