Abstract

Trade studies used to design optical imaging systems frequently result in systems being undersampled. The resolution of such systems is limited by the finite size of the detector pixels rather than the cutoff spatial frequency of the optical system. Multiframe super-resolution techniques can be used to combine a number of spatially displaced images from such systems into a single, high-resolution image. Nonlinear optimization techniques have frequently been used to solve this problem. Such techniques define an objective function and use numerical optimization methods to obtain a solution. These numerical methods are often more efficient when derivatives of the objective function with respect to the variables can be calculated analytically rather than numerically. We demonstrate for the simple motion model of pure lateral translation that the derivatives of the objective function with respect to the image lateral shifts can be calculated analytically to speed optimization calculations.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
    [CrossRef]
  2. A. S. Fruchter and R. N. Hook, “A novel image reconstruction method applied to deep Hubble space telescope images,” Proc. SPIE 3164, 120–125 (1997).
    [CrossRef]
  3. M. S. Alam, J. G. Bognar, S. Cain, and B. J. Yasuda, “Fast registration and reconstruction of aliased low-resolution frames by use of a modified maximum-likelihood approach,” Appl. Opt. 37, 1319–1328 (1998).
    [CrossRef]
  4. S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
    [CrossRef]
  5. B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” in Proceedings of the International Conference on Image Processing (1995), Vol. I–III, pp. B539–B542.
  6. V. Bannore and L. Swierkowski, “An iterative approach to image super-resolution,” in Intelligent Information Processing III, Z. Shi, K. Shimohara, and D. Feng, eds. (Springer, 2006), pp. 473–482.
  7. H. Stark and P. Oskoui, “High-resolution image recovery from image-plane arrays, using convex projections,” J. Opt. Soc. Am. A 6, 1715–1726 (1989).
    [CrossRef]
  8. L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.
  9. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
    [CrossRef]
  10. J. Tian and K. K. Ma, “A survey on super-resolution imaging,” Signal Image Video Process. 5, 329–342 (2011).
  11. R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
    [CrossRef]
  12. Q. Wang and X. Song, “Joint image registration and super-resolution reconstruction based on regularized total least norm,” in 16th IEEE International Conference on Image Processing (IEEE, 2009), pp. 1537–1540.
  13. M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
    [CrossRef]
  14. J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. (Springer, 2006).
  15. R. Fletcher, Practical Methods of Optimization, 2nd ed. (Wiley, 1987).
  16. D. S. C. Biggs and M. Andrews, “Acceleration of iterative image restoration algorithms,” Appl. Opt. 36, 1766–1775 (1997).
    [CrossRef]
  17. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
    [CrossRef]
  18. M. Guizar-Sicairos, S. T. Thurman, and J. R. Fienup, “Efficient subpixel image registration algorithms,” Opt. Lett. 33, 156–158 (2008).
    [CrossRef]

2013

M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
[CrossRef]

2011

J. Tian and K. K. Ma, “A survey on super-resolution imaging,” Signal Image Video Process. 5, 329–342 (2011).

2008

2004

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

2003

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

1999

R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

1998

1997

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
[CrossRef]

A. S. Fruchter and R. N. Hook, “A novel image reconstruction method applied to deep Hubble space telescope images,” Proc. SPIE 3164, 120–125 (1997).
[CrossRef]

D. S. C. Biggs and M. Andrews, “Acceleration of iterative image restoration algorithms,” Appl. Opt. 36, 1766–1775 (1997).
[CrossRef]

1989

Alam, M. S.

Andrews, M.

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
[CrossRef]

Bannore, V.

V. Bannore and L. Swierkowski, “An iterative approach to image super-resolution,” in Intelligent Information Processing III, Z. Shi, K. Shimohara, and D. Feng, eds. (Springer, 2006), pp. 473–482.

Barnard, K. J.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
[CrossRef]

Biggs, D. S. C.

Bognar, J. G.

Bovik, A. C.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

Cain, S.

Capel, D.

L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.

Elad, M.

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

Farsiu, S.

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

Fienup, J. R.

Fiete, R. D.

R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

Fletcher, R.

R. Fletcher, Practical Methods of Optimization, 2nd ed. (Wiley, 1987).

Fruchter, A. S.

A. S. Fruchter and R. N. Hook, “A novel image reconstruction method applied to deep Hubble space telescope images,” Proc. SPIE 3164, 120–125 (1997).
[CrossRef]

Guizar-Sicairos, M.

Hardie, R. C.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
[CrossRef]

Hook, R. N.

A. S. Fruchter and R. N. Hook, “A novel image reconstruction method applied to deep Hubble space telescope images,” Proc. SPIE 3164, 120–125 (1997).
[CrossRef]

Kang, M. G.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Katsaggelos, A. K.

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” in Proceedings of the International Conference on Image Processing (1995), Vol. I–III, pp. B539–B542.

Kondi, L. P.

M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
[CrossRef]

Ma, K. K.

J. Tian and K. K. Ma, “A survey on super-resolution imaging,” Signal Image Video Process. 5, 329–342 (2011).

Milanfar, P.

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

Nikou, C.

M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
[CrossRef]

Nocedal, J.

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. (Springer, 2006).

Oskoui, P.

Park, M. K.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Pickup, L.

L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.

Roberts, S.

L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.

Robinson, M. D.

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

Sheikh, H. R.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

Simoncelli, E. P.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

Song, X.

Q. Wang and X. Song, “Joint image registration and super-resolution reconstruction based on regularized total least norm,” in 16th IEEE International Conference on Image Processing (IEEE, 2009), pp. 1537–1540.

Stark, H.

Swierkowski, L.

V. Bannore and L. Swierkowski, “An iterative approach to image super-resolution,” in Intelligent Information Processing III, Z. Shi, K. Shimohara, and D. Feng, eds. (Springer, 2006), pp. 473–482.

Thurman, S. T.

Tian, J.

J. Tian and K. K. Ma, “A survey on super-resolution imaging,” Signal Image Video Process. 5, 329–342 (2011).

Tom, B. C.

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” in Proceedings of the International Conference on Image Processing (1995), Vol. I–III, pp. B539–B542.

Vrigkas, M.

M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
[CrossRef]

Wang, Q.

Q. Wang and X. Song, “Joint image registration and super-resolution reconstruction based on regularized total least norm,” in 16th IEEE International Conference on Image Processing (IEEE, 2009), pp. 1537–1540.

Wang, Z.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

Wright, S. J.

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. (Springer, 2006).

Yasuda, B. J.

Zisserman, A.

L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.

Appl. Opt.

IEEE Signal Process. Mag.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

IEEE Trans. Image Process.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1633 (1997).
[CrossRef]

S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327–1344 (2004).
[CrossRef]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004).
[CrossRef]

J. Opt. Soc. Am. A

Opt. Eng.

R. D. Fiete, “Image quality and λFN/p for remote sensing systems,” Opt. Eng. 38, 1229–1240 (1999).
[CrossRef]

Opt. Lett.

Proc. SPIE

A. S. Fruchter and R. N. Hook, “A novel image reconstruction method applied to deep Hubble space telescope images,” Proc. SPIE 3164, 120–125 (1997).
[CrossRef]

Signal Image Video Process.

J. Tian and K. K. Ma, “A survey on super-resolution imaging,” Signal Image Video Process. 5, 329–342 (2011).

Signal Process. Image Commun.

M. Vrigkas, C. Nikou, and L. P. Kondi, “Accurate image registration for MAP image super-resolution,” Signal Process. Image Commun. 28, 494–508 (2013).
[CrossRef]

Other

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. (Springer, 2006).

R. Fletcher, Practical Methods of Optimization, 2nd ed. (Wiley, 1987).

B. C. Tom and A. K. Katsaggelos, “Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” in Proceedings of the International Conference on Image Processing (1995), Vol. I–III, pp. B539–B542.

V. Bannore and L. Swierkowski, “An iterative approach to image super-resolution,” in Intelligent Information Processing III, Z. Shi, K. Shimohara, and D. Feng, eds. (Springer, 2006), pp. 473–482.

Q. Wang and X. Song, “Joint image registration and super-resolution reconstruction based on regularized total least norm,” in 16th IEEE International Conference on Image Processing (IEEE, 2009), pp. 1537–1540.

L. Pickup, S. Roberts, A. Zisserman, and D. Capel, “Multiframe super-resolution from a Bayesian perspective,” in Super-Resolution Imaging, P. Milanfar, ed. (CRC Press, 2011), pp. 247–284.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (4)

Fig. 1.
Fig. 1.

(a) Sample low-resolution image, (b) reconstruction using Drizzle algorithm, and (c) derivative-based optimization reconstruction.

Fig. 2.
Fig. 2.

(a) Measured low-resolution image, (b) Drizzle reconstruction, (c) reconstruction after final optimization, and (d) vertical slices through patterns 1–6 on the right side of the target.

Fig. 3.
Fig. 3.

Estimated lateral shifts.

Fig. 4.
Fig. 4.

Central region of the resolution target for (a) 3× optimization reconstruction upsampled to 6× resolution via bilinear interpolation and (b) 6× optimization reconstruction.

Equations (41)

Equations on this page are rendered with MathJax. Learn more.

Q=λ·FNp,
L2(Ω)=def{v:ΩR:Ω|v(x)|2dx<}.
L(R2)=def{v:R2R:ess supxR2|v(x)|<},
(Hg)(ζ)=R2h(ζx)g(x)dx=(h*g)(ζ).
(Hgθ)(x)=R2h(xζ)g(ζθ)dζ=R2h((xθ)ξ)g(ξ)dξ.
(S(θ)g)(x)=(Hgθ)(x)=R2h((xθ)ξ)g(ξ)dξ.
Pγ=γ1ϕ1++γNϕN
Ω(Pγ)(x)g(x)dx==1mγ(P*g)
P*g=(g,ϕ1L2(Ω),,g,ϕNL2(Ω)),
f,gL2(Ω)=Ωf(x)g(x)dx.
C(θ)=defP*S(θ)P.
C(θ)ij=ϕi,S(θ)ϕjL2(Ω)=Ωϕi(x)Ωh((xθ)ξ)ϕj(ξ)dξdx.
J(z,Θ)=defk=1KN(DC(θk)zdk)+R(z),
minimize(z,Θ)RN+2KJ(z,Θ).
N(v)=1pvpp=1pj=1n|vej|p,
limh20N(v+h)N(v)N(v)h2h2=0
N(v)=j=1n(vej)p1sign(vej)ej,
2N(v)=(p1)j=1n|v,ej2|p2ejej.
{vRn:v,ej20,j=1,,n}.
N(v)=j=1nsign(v,ej2)ej
v+h,ej2>0ifv,ej2>0
v+h,ej2<0ifv,ej2<0.
|v+h,ej2||v,ej2|=sign(v,ej2)h,ej2
N(v+h)N(v)=N(v)h,
v1N(v)==1m(v,e22+ϵ)12
N(v)==1nv,e2e(v,e22+ϵ)12,
2N(v)==1m(ϵ(v,e22+ϵ)32)ee.
kC(θ)ij=C(θ)ijϑk=Ωϕi(x)Ω(kh((xθ)ξ))ϕj(ξ)dξdx
kC(θ)ij=2C(θ)ijϑkϑ=Ωϕi(x)Ω(kh((xθ)ξ))ϕj(ξ)dξdx
gk=defN(DC(θk)zdk)
Hk=def2N(DC(θk)zdk).
zJ(z,Θ)=k=1KC(θk)Dgk+R(z),
θkJ(z,Θ)=[gkD(1C(θk))zgkD(2C(θk))z.].
z2J(z,Θ)h=k=1KC(θk)DHkDC(θk)h+2R(z)h.
θk2J(z,Θ)θ=[d11ϑ1+d12ϑ2d21ϑ1+d22ϑ2],
dij=z(iC(θk))DHkD(jC(θk))z+gkD(ijC(θk))z
θkθJ(z,Θ)=0,
zθkJ(z,Θ)θ=d1ϑ1+d2ϑ2,
dj=C(θk)DHkD(jC(θk))z+(jC(θk))Dgk
θkzJ(z,Θ)h=[d1hd2h].
PSNR=10log10(P2MSE),

Metrics