Abstract

Computational complexity is a major impediment to the real-time implementation of image restoration and superresolution algorithms in many applications. Although powerful restoration algorithms have been developed within the past few years utilizing sophisticated mathematical machinery (based on statistical optimization and convex set theory), these algorithms are typically iterative in nature and require a sufficient number of iterations to be executed to achieve the desired resolution improvement that may be needed to meaningfully perform postprocessing image exploitation tasks in practice. Additionally, recent technological breakthroughs have facilitated novel sensor designs (focal plane arrays, for instance) that make it possible to capture megapixel imagery data at video frame rates. A major challenge in the processing of these large-format images is to complete the execution of the image processing steps within the frame capture times and to keep up with the output rate of the sensor so that all data captured by the sensor can be efficiently utilized. Consequently, development of novel methods that facilitate real-time implementation of image restoration and superresolution algorithms is of significant practical interest and is the primary focus of this study. The key to designing computationally efficient processing schemes lies in strategically introducing appropriate preprocessing steps together with the superresolution iterations to tailor optimized overall processing sequences for imagery data of specific formats. For substantiating this assertion, three distinct methods for tailoring a preprocessing filter and integrating it with the superresolution processing steps are outlined. These methods consist of a region-of-interest extraction scheme, a background-detail separation procedure, and a scene-derived information extraction step for implementing a set-theoretic restoration of the image that is less demanding in computation compared with the superresolution iterations. A quantitative evaluation of the performance of these algorithms for restoring and superresolving various imagery data captured by diffraction-limited sensing operations are also presented.

© 2002 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. J. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).
  2. W. H. Richardson, “Bayesian-based iterative method of image restoration,” J. Opt. Soc. Am. 62, 55–60 (1972).
    [CrossRef]
  3. L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J. 79, 745–759 (1974).
    [CrossRef]
  4. B. R. Hunt, “Super-resolution of images: Algorithms, principles and performance,” Int. J. Imaging Syst. Technol. 6, 297–304 (1995).
    [CrossRef]
  5. H. Y. Pang, M. K. Sundareshan, S. Amphay, “Optimized maximum-likelihood algorithms for superresolution of passive millimeter-wave imagery,” In Passive Millimeter-Wave Imaging Technology, R.M. Smith, ed., Proc. SPIE, 3378, 148–160 (1998)
  6. P. L. Combettes, “The foundations of set theoretic estimation,” Proc. of IEEE, 81, 182–208 (1993).
  7. M. I. Sezan, “An overview of convex projections theory and its application to image recovery problems,” Ultramicroscopy 40, 55–67 (1992)
    [CrossRef]
  8. S. Bhattacharjee, M. K. Sundareshan, “Hybrid Bayesian and convex set projections algorithms for restoration and resolution enhancement of digital images,” Proc. SPIEApplications of Digital Image Processing, E. Tischer, ed., 3922, 205–213 (2000).
  9. T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
    [CrossRef] [PubMed]
  10. M. K. Sundareshan, P. Zegers, “Role of oversampled data in super-resolution processing and a progressive upsampling scheme for optimized implementations of iterative restoration algorithms”, in Passive Millimeter-Wave Imaging Technology, Aerosense’99, R. M. Smith, ed., 3703, 155–166 (1999).
  11. S. Bhattacharjee, “Super-resolution of images using a convex set-theoretic approach,” M.S. Thesis, (University of Arizona, Tucson, Arizona, 2000).
  12. D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 13, 111–122 (1981).
  13. R. P. Lippman, “Pattern classification using neural networks,” IEEE Trans. Commun. 27, 47–64 (1989).
  14. R. Inampudi, “Region of interest extraction from images for optimized super-resolution performance,” M.S. thesis, (University of Arizona, Tucson, Arizona, 2000).
  15. B. R. Frieden, “Image enhancement and restoration,” in Picture Processing and Digital Filtering, T. S. Huang, ed. (Springer-Verlag, New York, 1975), pp. 177–247.
  16. B. R. Frieden, D. C. Wells, “Restoring with maximum entropy. III Poisson sources and backgrounds,” J. Opt. Soc. Am. 68, 93–103 (1978).
    [CrossRef]
  17. M. S. Nadar, P. J. Sementilli, B. R. Hunt, “Estimation techniques of the background and detailed portion of an object in image susperresolution,” in Inverse Optics III, M. A. Fiddy, ed., Proc. SPIE2241, 204–215 (1994).
  18. B. I. Hauss, H. Agravante, S. Chaiken, “Advanced radiometric millimeter-wave scene simulation: ARMSS,” in Passive Millimeter-Wave Imaging Technology, R. M. Smith, ed., Proc182–193 (1997).
  19. R. M. Haralick, L. G. Shapiro, Computer and Robot Vision (Addison-Wesley, New York, 1992).
  20. J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 372–381 (1986).
    [CrossRef]

1995 (1)

B. R. Hunt, “Super-resolution of images: Algorithms, principles and performance,” Int. J. Imaging Syst. Technol. 6, 297–304 (1995).
[CrossRef]

1994 (1)

T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
[CrossRef] [PubMed]

1993 (1)

P. L. Combettes, “The foundations of set theoretic estimation,” Proc. of IEEE, 81, 182–208 (1993).

1992 (1)

M. I. Sezan, “An overview of convex projections theory and its application to image recovery problems,” Ultramicroscopy 40, 55–67 (1992)
[CrossRef]

1989 (1)

R. P. Lippman, “Pattern classification using neural networks,” IEEE Trans. Commun. 27, 47–64 (1989).

1986 (1)

J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 372–381 (1986).
[CrossRef]

1981 (1)

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 13, 111–122 (1981).

1978 (1)

1974 (1)

L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J. 79, 745–759 (1974).
[CrossRef]

1972 (1)

Agravante, H.

B. I. Hauss, H. Agravante, S. Chaiken, “Advanced radiometric millimeter-wave scene simulation: ARMSS,” in Passive Millimeter-Wave Imaging Technology, R. M. Smith, ed., Proc182–193 (1997).

Amphay, S.

H. Y. Pang, M. K. Sundareshan, S. Amphay, “Optimized maximum-likelihood algorithms for superresolution of passive millimeter-wave imagery,” In Passive Millimeter-Wave Imaging Technology, R.M. Smith, ed., Proc. SPIE, 3378, 148–160 (1998)

Ballard, D. H.

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 13, 111–122 (1981).

Bhattacharjee, S.

S. Bhattacharjee, “Super-resolution of images using a convex set-theoretic approach,” M.S. Thesis, (University of Arizona, Tucson, Arizona, 2000).

S. Bhattacharjee, M. K. Sundareshan, “Hybrid Bayesian and convex set projections algorithms for restoration and resolution enhancement of digital images,” Proc. SPIEApplications of Digital Image Processing, E. Tischer, ed., 3922, 205–213 (2000).

Canny, J.

J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 372–381 (1986).
[CrossRef]

Chaiken, S.

B. I. Hauss, H. Agravante, S. Chaiken, “Advanced radiometric millimeter-wave scene simulation: ARMSS,” in Passive Millimeter-Wave Imaging Technology, R. M. Smith, ed., Proc182–193 (1997).

Combettes, P. L.

P. L. Combettes, “The foundations of set theoretic estimation,” Proc. of IEEE, 81, 182–208 (1993).

Frieden, B. R.

B. R. Frieden, D. C. Wells, “Restoring with maximum entropy. III Poisson sources and backgrounds,” J. Opt. Soc. Am. 68, 93–103 (1978).
[CrossRef]

B. R. Frieden, “Image enhancement and restoration,” in Picture Processing and Digital Filtering, T. S. Huang, ed. (Springer-Verlag, New York, 1975), pp. 177–247.

Goodman, J.

J. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).

Haralick, R. M.

R. M. Haralick, L. G. Shapiro, Computer and Robot Vision (Addison-Wesley, New York, 1992).

Hauss, B. I.

B. I. Hauss, H. Agravante, S. Chaiken, “Advanced radiometric millimeter-wave scene simulation: ARMSS,” in Passive Millimeter-Wave Imaging Technology, R. M. Smith, ed., Proc182–193 (1997).

Hunt, B. R.

B. R. Hunt, “Super-resolution of images: Algorithms, principles and performance,” Int. J. Imaging Syst. Technol. 6, 297–304 (1995).
[CrossRef]

M. S. Nadar, P. J. Sementilli, B. R. Hunt, “Estimation techniques of the background and detailed portion of an object in image susperresolution,” in Inverse Optics III, M. A. Fiddy, ed., Proc. SPIE2241, 204–215 (1994).

Inampudi, R.

R. Inampudi, “Region of interest extraction from images for optimized super-resolution performance,” M.S. thesis, (University of Arizona, Tucson, Arizona, 2000).

Ji, T. L.

T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
[CrossRef] [PubMed]

Lippman, R. P.

R. P. Lippman, “Pattern classification using neural networks,” IEEE Trans. Commun. 27, 47–64 (1989).

Lucy, L. B.

L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J. 79, 745–759 (1974).
[CrossRef]

Nadar, M. S.

M. S. Nadar, P. J. Sementilli, B. R. Hunt, “Estimation techniques of the background and detailed portion of an object in image susperresolution,” in Inverse Optics III, M. A. Fiddy, ed., Proc. SPIE2241, 204–215 (1994).

Pang, H. Y.

H. Y. Pang, M. K. Sundareshan, S. Amphay, “Optimized maximum-likelihood algorithms for superresolution of passive millimeter-wave imagery,” In Passive Millimeter-Wave Imaging Technology, R.M. Smith, ed., Proc. SPIE, 3378, 148–160 (1998)

Richardson, W. H.

Roehrig, H.

T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
[CrossRef] [PubMed]

Sementilli, P. J.

M. S. Nadar, P. J. Sementilli, B. R. Hunt, “Estimation techniques of the background and detailed portion of an object in image susperresolution,” in Inverse Optics III, M. A. Fiddy, ed., Proc. SPIE2241, 204–215 (1994).

Sezan, M. I.

M. I. Sezan, “An overview of convex projections theory and its application to image recovery problems,” Ultramicroscopy 40, 55–67 (1992)
[CrossRef]

Shapiro, L. G.

R. M. Haralick, L. G. Shapiro, Computer and Robot Vision (Addison-Wesley, New York, 1992).

Sundareshan, M. K.

T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
[CrossRef] [PubMed]

M. K. Sundareshan, P. Zegers, “Role of oversampled data in super-resolution processing and a progressive upsampling scheme for optimized implementations of iterative restoration algorithms”, in Passive Millimeter-Wave Imaging Technology, Aerosense’99, R. M. Smith, ed., 3703, 155–166 (1999).

S. Bhattacharjee, M. K. Sundareshan, “Hybrid Bayesian and convex set projections algorithms for restoration and resolution enhancement of digital images,” Proc. SPIEApplications of Digital Image Processing, E. Tischer, ed., 3922, 205–213 (2000).

H. Y. Pang, M. K. Sundareshan, S. Amphay, “Optimized maximum-likelihood algorithms for superresolution of passive millimeter-wave imagery,” In Passive Millimeter-Wave Imaging Technology, R.M. Smith, ed., Proc. SPIE, 3378, 148–160 (1998)

Wells, D. C.

Zegers, P.

M. K. Sundareshan, P. Zegers, “Role of oversampled data in super-resolution processing and a progressive upsampling scheme for optimized implementations of iterative restoration algorithms”, in Passive Millimeter-Wave Imaging Technology, Aerosense’99, R. M. Smith, ed., 3703, 155–166 (1999).

Astron. J. (1)

L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J. 79, 745–759 (1974).
[CrossRef]

IEEE Trans. Commun. (1)

R. P. Lippman, “Pattern classification using neural networks,” IEEE Trans. Commun. 27, 47–64 (1989).

IEEE Trans. Med. Imaging (1)

T. L. Ji, M. K. Sundareshan, H. Roehrig, “Adaptive image contrast enhancement based on human visual properties,” IEEE Trans. Med. Imaging 13, 573–586 (1994).
[CrossRef] [PubMed]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

D. H. Ballard, “Generalizing the Hough transform to detect arbitrary shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 13, 111–122 (1981).

J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 372–381 (1986).
[CrossRef]

Int. J. Imaging Syst. Technol. (1)

B. R. Hunt, “Super-resolution of images: Algorithms, principles and performance,” Int. J. Imaging Syst. Technol. 6, 297–304 (1995).
[CrossRef]

J. Opt. Soc. Am. (2)

Proc. of IEEE (1)

P. L. Combettes, “The foundations of set theoretic estimation,” Proc. of IEEE, 81, 182–208 (1993).

Ultramicroscopy (1)

M. I. Sezan, “An overview of convex projections theory and its application to image recovery problems,” Ultramicroscopy 40, 55–67 (1992)
[CrossRef]

Other (10)

S. Bhattacharjee, M. K. Sundareshan, “Hybrid Bayesian and convex set projections algorithms for restoration and resolution enhancement of digital images,” Proc. SPIEApplications of Digital Image Processing, E. Tischer, ed., 3922, 205–213 (2000).

M. K. Sundareshan, P. Zegers, “Role of oversampled data in super-resolution processing and a progressive upsampling scheme for optimized implementations of iterative restoration algorithms”, in Passive Millimeter-Wave Imaging Technology, Aerosense’99, R. M. Smith, ed., 3703, 155–166 (1999).

S. Bhattacharjee, “Super-resolution of images using a convex set-theoretic approach,” M.S. Thesis, (University of Arizona, Tucson, Arizona, 2000).

J. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).

H. Y. Pang, M. K. Sundareshan, S. Amphay, “Optimized maximum-likelihood algorithms for superresolution of passive millimeter-wave imagery,” In Passive Millimeter-Wave Imaging Technology, R.M. Smith, ed., Proc. SPIE, 3378, 148–160 (1998)

M. S. Nadar, P. J. Sementilli, B. R. Hunt, “Estimation techniques of the background and detailed portion of an object in image susperresolution,” in Inverse Optics III, M. A. Fiddy, ed., Proc. SPIE2241, 204–215 (1994).

B. I. Hauss, H. Agravante, S. Chaiken, “Advanced radiometric millimeter-wave scene simulation: ARMSS,” in Passive Millimeter-Wave Imaging Technology, R. M. Smith, ed., Proc182–193 (1997).

R. M. Haralick, L. G. Shapiro, Computer and Robot Vision (Addison-Wesley, New York, 1992).

R. Inampudi, “Region of interest extraction from images for optimized super-resolution performance,” M.S. thesis, (University of Arizona, Tucson, Arizona, 2000).

B. R. Frieden, “Image enhancement and restoration,” in Picture Processing and Digital Filtering, T. S. Huang, ed. (Springer-Verlag, New York, 1975), pp. 177–247.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (7)

Fig. 1
Fig. 1

Schematic of a processing scheme for optimized implementation of superresolution algorithms.

Fig. 2
Fig. 2

Schematic of the processing scheme with ROI extraction.

Fig. 3
Fig. 3

Schematic of the preprocessing filter for ROI extraction.

Fig. 4
Fig. 4

Results of ROI extraction and superresolution experiment: (a) acquired image, (b) extracted ROI, (c) processed image after 30 ML superresolution iterations of ROI.

Fig. 5
Fig. 5

Superresolution processing of PMMW image with ML and ML-BD algorithms: (a) acquired PMMW image, (b) ML processed image (20 iterations), (c) ML-BD processed image (5 iterations).

Fig. 6
Fig. 6

Restoration and superresolution of a blurred image using border constraint set: (a) test image, (b) blurred image, (c) spectrum of test image, (d) spectrum of blurred image, (e) output of border extraction filter, (f) restored image, (g) spectrum of restored image.

Fig. 7
Fig. 7

Superresolution of PMMW image with ML algorithm and border extraction filter: (a) acquired PMMW image, (b) image processed with 30 ML iterations, (c) preprocessing followed by 5 ML iterations.

Tables (1)

Tables Icon

Table 1 Typical processing capacities for iterative ML superresolution

Equations (23)

Equations on this page are rendered with MathJax. Learn more.

gy=xX hy, xfx+ny,
fˆk+1=fˆkgfˆkhh,
fˆ=arg minfi=1M wiJifˆ, Si,
Jifˆ, Si=minfpSifˆ-fp2,
fˆn+1=fˆn+λniIn wi,nPi,nfˆn-fˆn
fˆk+1i, j =fˆki, j× gi, jfˆk i, jh i, j hi, j, i, j=1, 2, 3,  N,
NC=2Nlog4 N10 N4+33+7+N4
NML12Nlog4 N10 N4+33+7+N4.
Cpˆ=αpˆ pˆ2β+pˆ2,
gy=xX hy, xfbx+fdx+ny,
fˆbj=hjgj, j=1, 2,  N,
fˆn+1dj=fˆnd+fˆbjgjfˆnd+fˆbjh jhj-fˆbj,j =1, 2, , N.
Cfˆn+1d=αfˆn+1dfˆn+1d2β+fˆn+1d2,
fˆn+1j=Cfˆn+1dj+fˆbj
fˆn+1j<0 set fˆn+1j=0,fˆn+1j0 set fˆn+1j=fˆn+1j, 
fˆn+1j=fˆn+1jj=1N fˆn+1jj=1N gj.
fˆn+1dj=fˆn+1j-fˆbj
Gn=Gn,
n=ΔGfΔGf,
2n2Gf=0.
Δfi, j=fi, j2x+fi, j2y and Θfi, j=arctanfi, jxfi, jy,
Sborder=fΞ: foreground of image f is bounded by ξ,
Pborderf=fp where fpi, j=fi, j:i, j lies inside ξ0 : i, j lies outside ξ.

Metrics