Abstract

Traditional superresolution techniques employ optimizers, priors, and regularizers to deliver stable, appealing restorations even though deviating from the real, ground-truth scene. We have developed a non-regularized superresolution algorithm that directly solves a fully-characterized multi-shift imaging reconstruction problem to achieve realistic restorations without being penalized by improper assumptions made in the inverse problem. An adaptive frequency-based filtering scheme is introduced to upper bound the reconstruction errors while still producing more fine details as compared with previous methods when inaccurate shift estimation, noise, and blurring scenarios are considered.

© 2015 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Spectral-overlap approach to multiframe superresolution image reconstruction

Edward Cohen, Richard H. Picard, and Peter N. Crabtree
Appl. Opt. 55(15) 3978-3992 (2016)

Digital superresolution and the generalized sampling theorem

Sudhakar Prasad
J. Opt. Soc. Am. A 24(2) 311-325 (2007)

Statistical performance modeling for superresolution: a discrete data-continuous reconstruction framework

Frédéric Champagnat, Guy Le Besnerais, and Caroline Kulcsár
J. Opt. Soc. Am. A 26(7) 1730-1746 (2009)

References

  • View by:
  • |
  • |
  • |

  1. Z. Zalevsky and D. Mendlovic, Optical Superresolution (Springer, 2003).
  2. T. Lukeš, Super-Resolution Methods for Digital Image and Video Processing (Czech Technical University, 2013).
  3. Z. Zalevsky, Super-Resolved Imaging: Geometrical and Diffraction Approaches (Springer, 2011).
  4. J. Hadamard, Lectures on Cauchy’s Problem in Linear Partial Differential Equation (Dover, 1923).
  5. A. Gilman and D. G. Bailey, “Near optimal non-uniform interpolation for image super-resolution from multiple images,” in Image and Vision Computing, (2006), pp. 31–35.
  6. A. N. Tikhonov and V. I. Arsenin, Solutions of Ill-Posed Problems (Winston, 1977).
  7. B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high resolution image from multiple degraded mis-registered low resolution images,” in Proceedings of IEEE Intl. Conf. on Image Processing (IEEE, 1994), pp. 553–557.
    [Crossref]
  8. R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
    [Crossref] [PubMed]
  9. S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
    [Crossref]
  10. P. Milanfar, Super-Resolution Imaging (CRC Press, 2010).
  11. H. Stark and P. Oskoui, “High-resolution image recovery from image-plane arrays, using convex projections,” J. Opt. Soc. Am. A 6(11), 1715–1726 (1989).
    [Crossref] [PubMed]
  12. F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
    [Crossref]
  13. A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 645–650.
  14. F. Šroubek and P. Milanfar, “Robust multichannel blind deconvolution via fast alternating minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
    [Crossref] [PubMed]
  15. P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
    [Crossref]
  16. D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacement,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 1988), pp. 742–746.
  17. J. N. Sarvaiya, S. Patnaik, and S. Bombaywala, “Image registration by template matching using normalized cross-correlation,” in Proceedings of IEEE Intl. Conf. on Advances in Computing, Control, and Telecommunication Technologies (IEEE, 2009), pp. 819–822.
    [Crossref]
  18. M. Johansson, Image Registration with Simulated Annealing and Genetic Algorithms (Royal Institute of Technology in Sweden, 2006).
  19. E. Y. Lam and J. W. Goodman, “Iterative statistical approach to blind image deconvolution,” J. Opt. Soc. Am. A 17(7), 1177–1184 (2000).
    [Crossref] [PubMed]
  20. J. S. Lim, Two-Dimensional Signal and Image Processing (Prentice Hall, 1990).
  21. A. K. Kaw, E. E. Kalu, and D. Nguyen, Numerical Methods with Applications (Univ. of South Florida, 2010), Chap. 4.
  22. R. Farebrother, Linear Least Squares Computations (CRC Press, 1988).

2013 (1)

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

2012 (1)

F. Šroubek and P. Milanfar, “Robust multichannel blind deconvolution via fast alternating minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[Crossref] [PubMed]

2006 (1)

P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
[Crossref]

2005 (1)

F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
[Crossref]

2000 (1)

1997 (1)

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[Crossref] [PubMed]

1989 (1)

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[Crossref] [PubMed]

Babacan, D.

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

Barnard, K. J.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[Crossref] [PubMed]

Barrett, E. B.

F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
[Crossref]

Bombaywala, S.

J. N. Sarvaiya, S. Patnaik, and S. Bombaywala, “Image registration by template matching using normalized cross-correlation,” in Proceedings of IEEE Intl. Conf. on Advances in Computing, Control, and Telecommunication Technologies (IEEE, 2009), pp. 819–822.
[Crossref]

Brada, R.

D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacement,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 1988), pp. 742–746.

Goodman, J. W.

Hardie, R. C.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[Crossref] [PubMed]

Hoctor, R. T.

F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
[Crossref]

Katsaggelos, A.

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

Katsaggelos, A. K.

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high resolution image from multiple degraded mis-registered low resolution images,” in Proceedings of IEEE Intl. Conf. on Image Processing (IEEE, 1994), pp. 553–557.
[Crossref]

Keren, D.

D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacement,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 1988), pp. 742–746.

Lam, E. Y.

Milanfar, P.

F. Šroubek and P. Milanfar, “Robust multichannel blind deconvolution via fast alternating minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[Crossref] [PubMed]

Molina, R.

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

Oskoui, P.

Patnaik, S.

J. N. Sarvaiya, S. Patnaik, and S. Bombaywala, “Image registration by template matching using normalized cross-correlation,” in Proceedings of IEEE Intl. Conf. on Advances in Computing, Control, and Telecommunication Technologies (IEEE, 2009), pp. 819–822.
[Crossref]

Peleg, S.

A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 645–650.

D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacement,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 1988), pp. 742–746.

Rav-Acha, A.

A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 645–650.

Sarvaiya, J. N.

J. N. Sarvaiya, S. Patnaik, and S. Bombaywala, “Image registration by template matching using normalized cross-correlation,” in Proceedings of IEEE Intl. Conf. on Advances in Computing, Control, and Telecommunication Technologies (IEEE, 2009), pp. 819–822.
[Crossref]

Šroubek, F.

F. Šroubek and P. Milanfar, “Robust multichannel blind deconvolution via fast alternating minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[Crossref] [PubMed]

Stark, H.

Süsstrunk, S.

P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
[Crossref]

Tom, B. C.

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high resolution image from multiple degraded mis-registered low resolution images,” in Proceedings of IEEE Intl. Conf. on Image Processing (IEEE, 1994), pp. 553–557.
[Crossref]

Vandewalle, P.

P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
[Crossref]

Vega, M.

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

Vetterli, M.

P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
[Crossref]

Villena, S.

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

Wheeler, F. W.

F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
[Crossref]

Zomet, A.

A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 645–650.

Digit. Signal Process. (1)

S. Villena, M. Vega, D. Babacan, R. Molina, and A. Katsaggelos, “Bayesian combination of sparse and non-sparse priors in image super-resolution,” Digit. Signal Process. 23(2), 530–541 (2013).
[Crossref]

EURASIP J. Appl. Signal Process. (1)

P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, 71459 (2006).
[Crossref]

IEEE Trans. Image Process. (2)

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[Crossref] [PubMed]

F. Šroubek and P. Milanfar, “Robust multichannel blind deconvolution via fast alternating minimization,” IEEE Trans. Image Process. 21(4), 1687–1700 (2012).
[Crossref] [PubMed]

J. Opt. Soc. Am. A (2)

Proc. SPIE (1)

F. W. Wheeler, R. T. Hoctor, and E. B. Barrett, “Super-resolution image synthesis using projections onto convex sets in the frequency domain,” Proc. SPIE 5674, 479–490 (2005).
[Crossref]

Other (15)

A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 645–650.

P. Milanfar, Super-Resolution Imaging (CRC Press, 2010).

Z. Zalevsky and D. Mendlovic, Optical Superresolution (Springer, 2003).

T. Lukeš, Super-Resolution Methods for Digital Image and Video Processing (Czech Technical University, 2013).

Z. Zalevsky, Super-Resolved Imaging: Geometrical and Diffraction Approaches (Springer, 2011).

J. Hadamard, Lectures on Cauchy’s Problem in Linear Partial Differential Equation (Dover, 1923).

A. Gilman and D. G. Bailey, “Near optimal non-uniform interpolation for image super-resolution from multiple images,” in Image and Vision Computing, (2006), pp. 31–35.

A. N. Tikhonov and V. I. Arsenin, Solutions of Ill-Posed Problems (Winston, 1977).

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high resolution image from multiple degraded mis-registered low resolution images,” in Proceedings of IEEE Intl. Conf. on Image Processing (IEEE, 1994), pp. 553–557.
[Crossref]

J. S. Lim, Two-Dimensional Signal and Image Processing (Prentice Hall, 1990).

A. K. Kaw, E. E. Kalu, and D. Nguyen, Numerical Methods with Applications (Univ. of South Florida, 2010), Chap. 4.

R. Farebrother, Linear Least Squares Computations (CRC Press, 1988).

D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacement,” in Proceedings of IEEE Intl. Conf. on Computer Vision and Pattern Recognition (IEEE, 1988), pp. 742–746.

J. N. Sarvaiya, S. Patnaik, and S. Bombaywala, “Image registration by template matching using normalized cross-correlation,” in Proceedings of IEEE Intl. Conf. on Advances in Computing, Control, and Telecommunication Technologies (IEEE, 2009), pp. 819–822.
[Crossref]

M. Johansson, Image Registration with Simulated Annealing and Genetic Algorithms (Royal Institute of Technology in Sweden, 2006).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (15)

Fig. 1
Fig. 1 Illustration of the shift impact on spatial sampling and angular bandwidth: sensor shift (left) and camera shift (right) as compared with the non-shifted case (center).
Fig. 2
Fig. 2 Simple imaging scheme illustrating the linear combinations of HR pixels including the boundaries forming the LR images (top) and the resulting vector-matrix representation that direct SR solves (bottom) in a noisy environment.
Fig. 3
Fig. 3 Structure of the H matrix (right) showing the sparse diagonal-like distribution (locations of nonzero elements) and the color-coded repetitive blocks for a 16×16 object (left) using 4×4 blocks and 4 LR images. The colored blocks below the horizontal axis of the H matrix are highlighting a group of columns that can be copied (with circular shifts) from any other block’s columns belonging to the same colored group.
Fig. 4
Fig. 4 The steps (top) and a block diagram (bottom) summarizing the direct SR technique.
Fig. 5
Fig. 5 Block diagram illustrating the adaptive frequency-based filtering scheme in the training stage (top) and the testing stage (bottom).
Fig. 6
Fig. 6 Training images used in adaptive frequency-based filtering scheme.
Fig. 7
Fig. 7 Visual comparison of various SR techniques at 2× and 4× subsampling factors using sensor shift model, known non-integer-pixel shifts, σ blur =0.5 pixel, and noiseless environment for U.S. Air Force (USAF) target and cameraman image. Shown at the first row: sample of LR images and the HR images. Rows 2-5 displays from left to right: bicubic interpolation, SR of mixed priors, MCBD, and direct SR. The associated RMSE% is shown at their left.
Fig. 8
Fig. 8 Reconstruction error vs. noise of various SR techniques and the AFFS results for the sharpness circle.
Fig. 9
Fig. 9 Visual comparison of the reconstructed sharpness circle (zoomed) at NS = 0.1% and 0.5% for various SR techniques and averaged over 20 noise realizations. A sample LR image and the ground-truth HR image are shown on the left of the first row. The three trained multi-gray masks ( w Direct ,  w Bicubic ,   and   w MCBD ) are color encoded as red for direct, green for bicubic, and blue for MCBD in the composite RGB masks displayed on the right of first row.
Fig. 10
Fig. 10 Reconstruction errors vs. σ blur when imaging the boat image through two observation models: sensor shift (SS) and camera shift (CS) using various SR techniques.
Fig. 11
Fig. 11 Visual comparison of the reconstructed boat at σ blur =1  pixel for various SR techniques using sensor shift (middle row) and camera shift (bottom row). A sample LR image and the original HR image are shown on the left of the first row. The three trained averaged masks ( w Direct ,  w Bicubic ,   and   w MCBD ) are color encoded as red for direct, green for bicubic, and blue for MCBD in the composite RGB masks displayed on the right of the first row.
Fig. 12
Fig. 12 Recon. errors vs. BTol when reconstructing peppers image with various SR techniques.
Fig. 13
Fig. 13 Visual comparison of the reconstructed pepper (zoomed) at BTol=10%  (middle row) and  20%  (bottom row) for various SR techniques. A sample LR image and the original HR image are shown on the left of the first row. The three trained multi-gray masks ( w Direct ,  w Bicubic ,   and   w MCBD ) are color encoded as red for direct, green for bicubic, and blue for MCBD in the composite RGB masks displayed on the right of the first row.
Fig. 14
Fig. 14 Reconstruction errors vs. STol when restoring endoscopy image with various SR techniques.
Fig. 15
Fig. 15 Visual comparison of the reconstructed endoscopy image (zoomed) at STol=0.1%  (middle row) and  1%  (bottom row) for various SR techniques. A sample LR image and the original HR image are shown on the left of the first row. The three trained multi-gray masks ( w Direct ,  w Bicubic ,   and   w MCBD ) are color encoded as red for direct, green for bicubic, and blue for MCBD in composite RGB masks displayed on the right of the first row.

Metrics