Abstract

Digital color cameras using a single detector array with a Bayer color filter array (CFA) require interpolation or demosaicing to estimate missing color information and provide full-color images. However, demosaicing does not specifically address fundamental undersampling and aliasing inherent in typical camera designs. Fast non-uniform interpolation based super-resolution (SR) is an attractive approach to reduce or eliminate aliasing and its relatively low computational load is amenable to real-time applications. The adaptive Wiener filter (AWF) SR algorithm was initially developed for grayscale imaging and has not previously been applied to color SR demosaicing. Here, we develop a novel fast SR method for CFA cameras that is based on the AWF SR algorithm and uses global channel-to-channel statistical models. We apply this new method as a stand-alone algorithm and also as an initialization image for a variational SR algorithm. This paper presents the theoretical development of the color AWF SR approach and applies it in performance comparisons to other SR techniques for both simulated and real data.

© 2013 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. B. E. Bayer, “Color imaging array,” Eastman Kodak Co., U.S. Patent 3,971,065 (Jul 20 1976).
  2. R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
    [CrossRef]
  3. B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
    [CrossRef]
  4. X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
    [CrossRef]
  5. R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng.38(7), 1229–1240 (1999).
    [CrossRef]
  6. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
    [CrossRef]
  7. P. Milanfar, Super-Resolution Imaging (CRC, 2010).
  8. J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
    [CrossRef]
  9. M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
    [CrossRef]
  10. S. Lertrattanapanich and N. K. Bose, “High resolution image formation from low resolution frames using Delaunay triangulation,” IEEE Trans. Image Process.11(12), 1427–1441 (2002).
    [CrossRef] [PubMed]
  11. K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).
  12. A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration,” Int. Conf. Acoust Spee.3, 169–172 (1992).
    [CrossRef]
  13. H. Shekarforoush and R. Chellappa, “Data-driven multichannel superresolution with application to video sequences,” J. Opt. Soc. Am. A16(3), 481–492 (1999).
    [CrossRef]
  14. T. Q. Pham, L. J. Van Vliet, and K. Schutte, “Robust fusion of irregularly sampled data using adaptive normalized convolution,” EURASIP J. Adv. Sig. Pr. 1–12 (2006).
  15. J. Shi, S. E. Reichenbach, and J. D. Howe, “Small-kernel superresolution methods for microscanning imaging systems,” Appl. Opt.45(6), 1203–1214 (2006).
    [CrossRef] [PubMed]
  16. R. C. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process.16(12), 2953–2964 (2007).
    [CrossRef] [PubMed]
  17. R. C. Hardie, D. A. LeMaster, and B. M. Ratliff, “Super-resolution for imagery from integrated microgrid polarimeters,” Opt. Express19(14), 12937–12960 (2011).
    [CrossRef] [PubMed]
  18. R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging,” Opt. Express19(27), 26208–26231 (2011).
    [CrossRef] [PubMed]
  19. B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
    [CrossRef]
  20. M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Process.10(8), 1187–1193 (2001).
    [CrossRef] [PubMed]
  21. M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).
  22. T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” Proc. CVPR IEEE2, 600–607 (2004).
    [CrossRef]
  23. R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).
  24. S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
    [CrossRef] [PubMed]
  25. M. Gevrekci, B. K. Gunturk, and Y. Altunbasak, “POCS-based restoration of Bayer-sampled image sequences,” IEEE Conf. Acoust. Spee. 1, 753–756 (2007).
    [CrossRef]
  26. Y. R. Li and D. Q. Dai, “Color superresolution reconstruction and demosaicing using elastic net and tight frame,” IEEE Circuits-I55(11), 3500–3512 (2008).
    [CrossRef]
  27. M. Trimeche, “Color demosaicing using multi-frame super-resolution,” Eur. Signal Process. Conf. (2008).
  28. D. A. Sorrentino and A. Antoniou, “Improved hybrid demosaicing and color super-resolution implementation using quasi-Newton algorithms,” Can. Con. El. Comp. En. 815–818 (2009).
  29. P. Vandewalle, K. Krichane, D. Alleysson, and S. Süsstrunk, “Joint demosaicing and super-resolution imaging from a set of unregistered aliased images” Proc. SPIE IS&T Elect. Im. 65020A (2007).
  30. T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).
  31. M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
    [CrossRef]
  32. M. T. Eismann, Hyperspectral Remote Sensing (SPIE, 2012).
  33. G. D. Boreman, Modulation Transfer Function in Optical and Electro-Optical Systems (SPIE, 2001).
  34. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
    [CrossRef]
  35. A. Oppenheim and R. Schafer, Discrete-Time Signal Processing (Prentice Hall, 1989).
  36. C. W. Therrien, Discrete Random Signals and Statistical Signal Processing (Prentice Hall PTR, 1992).
  37. A. K. Jain, Fundamentals of Digital Image Processing (Prentice Hall, 1989).
  38. P. Milanfar, “MDSP resolution enhancement software,” Copyright 2009 by University of California. http://users.soe.ucsc.edu/~milanfar/software/superresolution.html .

2012 (1)

T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).

2011 (2)

2008 (2)

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
[CrossRef]

Y. R. Li and D. Q. Dai, “Color superresolution reconstruction and demosaicing using elastic net and tight frame,” IEEE Circuits-I55(11), 3500–3512 (2008).
[CrossRef]

2007 (2)

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

R. C. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process.16(12), 2953–2964 (2007).
[CrossRef] [PubMed]

2006 (2)

J. Shi, S. E. Reichenbach, and J. D. Howe, “Small-kernel superresolution methods for microscanning imaging systems,” Appl. Opt.45(6), 1203–1214 (2006).
[CrossRef] [PubMed]

S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
[CrossRef] [PubMed]

2005 (3)

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
[CrossRef]

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

2004 (1)

M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).

2003 (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
[CrossRef]

2002 (2)

S. Lertrattanapanich and N. K. Bose, “High resolution image formation from low resolution frames using Delaunay triangulation,” IEEE Trans. Image Process.11(12), 1427–1441 (2002).
[CrossRef] [PubMed]

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

2001 (1)

M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Process.10(8), 1187–1193 (2001).
[CrossRef] [PubMed]

2000 (1)

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

1999 (2)

1998 (1)

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

1995 (1)

J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
[CrossRef]

1991 (1)

K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).

Aizawa, K.

K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).

Alam, M.

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

Altunbasak, Y.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

Barnard, K. J.

R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging,” Opt. Express19(27), 26208–26231 (2011).
[CrossRef] [PubMed]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

Barner, K. E.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
[CrossRef]

Bilbro, G.

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

Bognar, J. G.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

Bognar, M. S.

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

Bose, N. K.

S. Lertrattanapanich and N. K. Bose, “High resolution image formation from low resolution frames using Delaunay triangulation,” IEEE Trans. Image Process.11(12), 1427–1441 (2002).
[CrossRef] [PubMed]

Chellappa, R.

Dai, D. Q.

Y. R. Li and D. Q. Dai, “Color superresolution reconstruction and demosaicing using elastic net and tight frame,” IEEE Circuits-I55(11), 3500–3512 (2008).
[CrossRef]

Elad, M.

S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
[CrossRef] [PubMed]

M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Process.10(8), 1187–1193 (2001).
[CrossRef] [PubMed]

Farsiu, S.

S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
[CrossRef] [PubMed]

Fiete, R. D.

R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng.38(7), 1229–1240 (1999).
[CrossRef]

Gillette, J. C.

J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
[CrossRef]

Glotzbach, J.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

Gotoh, T.

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” Proc. CVPR IEEE2, 600–607 (2004).
[CrossRef]

Gunturk, B.

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
[CrossRef]

Gunturk, B. K.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

Hardie, R. C.

R. C. Hardie, K. J. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging,” Opt. Express19(27), 26208–26231 (2011).
[CrossRef] [PubMed]

R. C. Hardie, D. A. LeMaster, and B. M. Ratliff, “Super-resolution for imagery from integrated microgrid polarimeters,” Opt. Express19(14), 12937–12960 (2011).
[CrossRef] [PubMed]

R. C. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process.16(12), 2953–2964 (2007).
[CrossRef] [PubMed]

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
[CrossRef]

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
[CrossRef]

Hasegawa, H.

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

Heinze, T.

T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).

Hel-Or, Y.

M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Process.10(8), 1187–1193 (2001).
[CrossRef] [PubMed]

Howe, J. D.

Kang, M. G.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
[CrossRef]

Komatsu, T.

K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).

LeMaster, D. A.

Lertrattanapanich, S.

S. Lertrattanapanich and N. K. Bose, “High resolution image formation from low resolution frames using Delaunay triangulation,” IEEE Trans. Image Process.11(12), 1427–1441 (2002).
[CrossRef] [PubMed]

Li, X.

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
[CrossRef]

Li, Y. R.

Y. R. Li and D. Q. Dai, “Color superresolution reconstruction and demosaicing using elastic net and tight frame,” IEEE Circuits-I55(11), 3500–3512 (2008).
[CrossRef]

Mersereau, R. M.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

Milanfar, P.

S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
[CrossRef] [PubMed]

Narayanan, B.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

Okutomi, M.

M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” Proc. CVPR IEEE2, 600–607 (2004).
[CrossRef]

Ordonez, R.

Ozkan, M. K.

A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration,” Int. Conf. Acoust Spee.3, 169–172 (1992).
[CrossRef]

Park, M. K.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
[CrossRef]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
[CrossRef]

Polze, A.

T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).

Ramanath, R.

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

Ratliff, B. M.

Reichenbach, S. E.

Saito, T.

K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).

Sakaniwa, K.

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

Sander, W.

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

Sasahara, R.

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

Schafer, R. W.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

Sezan, M. I.

A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration,” Int. Conf. Acoust Spee.3, 169–172 (1992).
[CrossRef]

Shao, M.

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
[CrossRef]

Shekarforoush, H.

Shi, J.

Shimizu, M.

M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).

Snyder, W.

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

Stadtmiller, T. M.

J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
[CrossRef]

Tekalp, A. M.

A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration,” Int. Conf. Acoust Spee.3, 169–172 (1992).
[CrossRef]

Trimeche, M.

M. Trimeche, “Color demosaicing using multi-frame super-resolution,” Eur. Signal Process. Conf. (2008).

von Lowis, M.

T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).

Watson, E. A.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

Yamada, I.

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

Yano, T.

M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).

Yasuda, B. J.

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

Zhang, L.

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
[CrossRef]

Appl. Opt. (1)

IEEE Circ. Syst. Vid. (1)

B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A computationally efficient super-resolution algorithm for video processing using partition filters,” IEEE Circ. Syst. Vid.17(5), 621–634 (2007).
[CrossRef]

IEEE Circuits-I (1)

Y. R. Li and D. Q. Dai, “Color superresolution reconstruction and demosaicing using elastic net and tight frame,” IEEE Circuits-I55(11), 3500–3512 (2008).
[CrossRef]

IEEE Signal Proc. Mag. (1)

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Proc. Mag.22(1), 44–54 (2005).
[CrossRef]

IEEE Signal Process. Mag. (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag.20(3), 21–36 (2003).
[CrossRef]

IEEE Trans. Image Proc. (2)

R. Sasahara, H. Hasegawa, I. Yamada, and K. Sakaniwa, “A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method,” IEEE Trans. Image Proc.1, 857–860 (2005).

S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” IEEE Trans. Image Proc.15(1), 141–159 (2006).
[CrossRef] [PubMed]

IEEE Trans. Image Process. (3)

R. C. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process.16(12), 2953–2964 (2007).
[CrossRef] [PubMed]

S. Lertrattanapanich and N. K. Bose, “High resolution image formation from low resolution frames using Delaunay triangulation,” IEEE Trans. Image Process.11(12), 1427–1441 (2002).
[CrossRef] [PubMed]

M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Process.10(8), 1187–1193 (2001).
[CrossRef] [PubMed]

IEEE Trans. Instrum. Meas. (1)

M. Alam, M. S. Bognar, R. C. Hardie, and B. J. Yasuda, “Infrared image registration and high-resolution reconstruction using multiple translationally shifted aliased video frames,” IEEE Trans. Instrum. Meas.49(5), 915–923 (2000).
[CrossRef]

Int. C. Patt. Recog. (1)

M. Shimizu, T. Yano, and M. Okutomi, “Super-resolution under image deformation,” Int. C. Patt. Recog.3, 586–589 (2004).

IWSSIP (1)

T. Heinze, M. von Lowis, and A. Polze, “Joint multi-frame demosaicing and super-resolution with artificial neural networks,” IWSSIP2012, 540–543 (2012).

J. Electron. Imaging (1)

R. Ramanath, W. Snyder, G. Bilbro, and W. Sander, “Demosaicking methods for the Bayer color arrays,” J. Electron. Imaging11(3), 306–315 (2002).
[CrossRef]

J. Opt. Soc. Am. A (1)

Opt. Eng. (4)

R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng.38(7), 1229–1240 (1999).
[CrossRef]

J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Aliasing reduction in staring infrared imagers utilizing subpixel techniques,” Opt. Eng.34(11), 3130–3137 (1995).
[CrossRef]

M. Shao, K. E. Barner, and R. C. Hardie, “Partition-based interpolation for color filter array demosaicking and super-resolution reconstruction,” Opt. Eng.44(10), 107003 (2005).
[CrossRef]

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng.37(1), 247–260 (1998).
[CrossRef]

Opt. Express (2)

P. Soc. Photo-Opt. Ins. (1)

K. Aizawa, T. Komatsu, and T. Saito, “Acquisition of very high resolution images using stereo cameras,” P. Soc. Photo-Opt. Ins.1605, 318–328 (1991).

Proc. SPIE (1)

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE6822, 68221J, 68221J-15 (2008).
[CrossRef]

Other (15)

B. E. Bayer, “Color imaging array,” Eastman Kodak Co., U.S. Patent 3,971,065 (Jul 20 1976).

P. Milanfar, Super-Resolution Imaging (CRC, 2010).

A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration,” Int. Conf. Acoust Spee.3, 169–172 (1992).
[CrossRef]

T. Q. Pham, L. J. Van Vliet, and K. Schutte, “Robust fusion of irregularly sampled data using adaptive normalized convolution,” EURASIP J. Adv. Sig. Pr. 1–12 (2006).

A. Oppenheim and R. Schafer, Discrete-Time Signal Processing (Prentice Hall, 1989).

C. W. Therrien, Discrete Random Signals and Statistical Signal Processing (Prentice Hall PTR, 1992).

A. K. Jain, Fundamentals of Digital Image Processing (Prentice Hall, 1989).

P. Milanfar, “MDSP resolution enhancement software,” Copyright 2009 by University of California. http://users.soe.ucsc.edu/~milanfar/software/superresolution.html .

M. T. Eismann, Hyperspectral Remote Sensing (SPIE, 2012).

G. D. Boreman, Modulation Transfer Function in Optical and Electro-Optical Systems (SPIE, 2001).

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” Proc. CVPR IEEE2, 600–607 (2004).
[CrossRef]

M. Trimeche, “Color demosaicing using multi-frame super-resolution,” Eur. Signal Process. Conf. (2008).

D. A. Sorrentino and A. Antoniou, “Improved hybrid demosaicing and color super-resolution implementation using quasi-Newton algorithms,” Can. Con. El. Comp. En. 815–818 (2009).

P. Vandewalle, K. Krichane, D. Alleysson, and S. Süsstrunk, “Joint demosaicing and super-resolution imaging from a set of unregistered aliased images” Proc. SPIE IS&T Elect. Im. 65020A (2007).

M. Gevrekci, B. K. Gunturk, and Y. Altunbasak, “POCS-based restoration of Bayer-sampled image sequences,” IEEE Conf. Acoust. Spee. 1, 753–756 (2007).
[CrossRef]

Supplementary Material (1)

» Media 1: MOV (4486 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (16)

Fig. 1
Fig. 1

Bayer CFA showing color channel and fundamental detector sampling.

Fig. 2
Fig. 2

Forward observation model relating the desired continuous image to the observed data.

Fig. 3
Fig. 3

Alternate forward observation model combining frame motion and “mosaic” sampling.

Fig. 4
Fig. 4

MTF of the example detector, optical diffraction, and total system.

Fig. 5
Fig. 5

Example observation & estimation windows for 3 LR frames of 4-channel DoFP sensor.

Fig. 6
Fig. 6

Examples of the continuous (a) desired autocorrelation, (b) noise-free observed autocorrelation, and (c) cross-correlation models.

Fig. 7
Fig. 7

Image data used in simulation experiment #1. (a) Full-color (24-bit) high resolution image, (b) sample simulated LR Bayer CFA image, and (c) sample red color channel image.

Fig. 8
Fig. 8

Simulated data results for (a) original HR image, (b) demosaic/interpolation, (c) Shift & Add output, (d) MDSP output, (e) RLS with independent color channel regularization, (f) independent color channel AWF SR, (g) 12-parameter 3-PSF color AWF SR, (h) 6-parameter 1-PSF color AWF SR, (i) 3-parameter 1-PSF color AWF SR.

Fig. 9
Fig. 9

Region of interest from results in Fig. 9. (a) original HR image, (b) demosaic/interpolation, (c) Shift & Add output, (d) MDSP output, (e) RLS with independent color channel regularization, (f) independent color channel AWF SR, (g) 12-parameter 3-PSF color AWF SR, (h) 6-parameter 1-PSF color AWF SR, (i) 3-parameter 1-PSF color AWF SR.

Fig. 10
Fig. 10

Region of interest for simulated results using 4 LR input frames. (a) independent color channel AWF SR, (b) 12-parameter 3-PSF color AWF SR.

Fig. 11
Fig. 11

True shifts, AWF shift estimates, and quantized shift estimates in experiment #2.

Fig. 12
Fig. 12

HR image used in simulation experiment #3 variation SR performance comparison.

Fig. 13
Fig. 13

ROI from Fig. 13. (a) input HR image for simulation, (b) 6-parameter 1-PSF color AWF SR, (c) RLS initialized with interpolated LR input image, (d) RLS initialized with color AWF SR result.

Fig. 14
Fig. 14

Sample input from real data experiment. (a) sample LR Bayer CFA image, (b) red color channel from sample LR Bayer CFA image.

Fig. 15
Fig. 15

Chirp resolution chart ROI. (a) demosaic/interpolation (Media 1 left), (b) Shift & Add output, (c) MDSP output, (d) RLS with independent channel regularization, (e) 6-parameter 1-PSF color AWF SR (Media 1 right), (f) RLS initialized with color AWF SR result.

Fig. 16
Fig. 16

Textbooks ROI from real data experiment. (a) demosaic/interpolation (Media 1 left), (b) Shift & Add output, (c) MDSP output, (d) RLS with independent color channel regularization, (e) 6-parameter 1-PSF color AWF SR (Media 1 right), (f) RLS initialized with color AWF SR result.

Tables (6)

Tables Icon

Table 1 PSF parameters for simulation experiment #1

Tables Icon

Table 2 Model parameters for color AWF methods

Tables Icon

Table 3 Performance results for simulation experiment #1

Tables Icon

Table 4 Performance results for simulation experiment #2, randomly shifted input

Tables Icon

Table 5 Performance results for simulation experiment #2, quantized input shifts

Tables Icon

Table 6 Performance results for simulation experiment #3

Equations (22)

Equations on this page are rendered with MathJax. Learn more.

h d (x,y)= 1 ab rect( x a , y b ),
H d (u,v)=sinc(au,bv)= sin(πau)sin(πbv) π 2 (au)(bv) ,
H o (u,v)={ 2 π [ cos 1 ( ρ ρ c ) ρ ρ c 1 ( ρ ρ c ) 2 ],ρ ρ c 0, otherwise ,
ρ c = 1 λf/# (cyc/mm).
1 δ s 2 ρ c ,
K= P W x W y L 2 , i.
d ^ i = W i T g i .
W i = R i 1 P i .
R i =E{ f i f i T }+ σ n 2 I,
P i =E{ f i d i T }.
r f j f k (x,y)= r d j d k (x,y)* h j (x,y)* h k (x,y).
r d j f k (x,y)= r d j d k (x,y)* h k (x,y).
r d k f j (x,y)= r d j d k (x,y)* h j (x,y).
r d j d k (x,y)= σ j,k 2 ρ j,k x 2 + y 2 ,
ρ j,k =ρ.
σ j,k 2 ={ σ corr 2 , j=k α σ corr 2 , jk .
ρ j,k ={ ρ corr , j=k ρ r,g , j=R, k=G ρ r,b , j=R, k=B ρ g,b , j=G, k=B .
R i =E{ g i g i T }=E{( f i + n i ) ( f i + n i ) T } =E{( f i + n i )( f i T + n i T )} =E{ f i f i T }+E{ n i f i T }+E{ f i n i T }+E{ n i n i T } R i =E{ f i f i T }+ σ n 2 I
r f j f k (x,y)=E{ f j (α,β) f k (α+x,β+y) } =E{ [ d j (α,β)* h j (α,β) ][ d k (α+x,β+y)* h k (α+x,β+y) ] } =E{ [ d j (αγ,βη) h j (γ,η)dγdη ][ d k (α+xθ,β+yϕ) h k (θ,ϕ)dθdϕ ] } =E{ [ d j (αγ,βη) h j (γ,η)[ d k (α+xθ,β+yϕ) h k (θ,ϕ)dθdϕ ]dγdη ] } = [ E{ d j (αγ,βη) d k (α+xθ,β+yϕ) } h k (θ,ϕ)dθdϕ ] h j (γ,η)dγdη = [ r d j d k (x+γθ,y+ηϕ) h k (θ,ϕ)dθdϕ ] h j (γ,η)dγdη (let ξ=θγ, ψ=ϕ-η) r f j f k (x,y)= [ r d j d k (xξ,yψ) h k (γ+ξ,η+ψ)dξdψ ] h j (γ,η)dγdη = r d j d k (xξ,yψ)[ h j (γ,η) h k (γ+ξ,η+ψ)dγdη ]dξdψ = r d j d k (xξ,yψ)[ h j (ξ,ψ) h k (ξ,ψ) ]dξdψ r f j f k (x,y)= r d j d k (x,y)*[ h j (x,y)* h k (x,y) ],
P i =E{ g i d i T } =E{( f i + n i ) d i T } =E{ f i d i T }+E{ n i d i T } P i =E{ f i d i T }
r d j f k (x,y)=E{ d j (α,β) f k (α+x,β+y) } =E{ d j (α,β)[ d k (α+x,β+y)* h k (α+x,β+y) ] } =E{ d j (α,β)[ d k (α+xγ,β+yη) h k (γ,η)dγdη ] } = E{ d j (α,β) d k (α+xγ,β+yη) h k (γ,η) }dγdη = h k (γ,η)E{ d j (α,β) d k (α+xγ,β+yη) }dγdη = h k (γ,η) r d j d k (xγ,yη)dγdη r d j f k (x,y)= r d j d k (x,y)* h k (x,y).
r d k f j (x,y)=E{ d k (α,β) f j (α+x,β+y) } =E{ d k (α,β)[ d j (α+x,β+y)* h j (α+x,β+y) ] } =E{ d k (α,β)[ d j (α+xγ,β+yη) h j (γ,η)dγdη ] } = E{ d k (α,β) d j (α+xγ,β+yη) h j (γ,η) }dγdη = h j (γ,η)E{ d k (α,β) d j (α+xγ,β+yη) }dγdη = h j (γ,η) r d k d j (xγ,yη)dγdη = r d k d j (x,y)* h j (x,y) r d k f j (x,y)= r d j d k (x,y)* h j (x,y)

Metrics