Abstract

Low-cost compact sensors for ultrasmall systems are a pressing need in many new applications. One potential solution is a shallow aspect ratio system using a lenslet array to form multiple undersampled subimages of a scene on a single focal plane array, where processing techniques then produce an upsampled restored image. We have investigated the optimization and theoretical limits of the performance of such arrays. We have built a hardware simulator and developed algorithms to process imagery similar to that of a full lenslet imaging sensor, which allowed us to quickly test optical components, algorithms, and complete system designs for future lenslet imaging systems.

© 2007 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, "Thin observation module by bound optics (TOMBO): concept and verification," Appl. Opt. 40, 1806-1813 (2001).
    [CrossRef]
  2. T. S. Huang, "Multiple frame image restoration and registration," in Advances in Computer Vision and Image Processing (JAI Press, 1984), Vol. 1.
  3. J. M. Schuler and D. A. Scribner, "Increasing spatial resolution through temporal supersampling of digital video," Opt. Eng. 38, 801-805 (1999).
    [CrossRef]
  4. M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
    [CrossRef]
  5. R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
    [CrossRef] [PubMed]
  6. M. S. Alam, J. G. Bognar, S. Cain, and B. J. Yasuda, "Fast registration and reconstruction of aliased, low-resolution frames by use of a modified maximum-likelihood approach," Appl. Opt. 37, 1319-1328 (1998).
    [CrossRef]
  7. A. Shaum and M. McHugh, Analytic Methods of Image Registration: Displacement Estimation and Resampling, NPL Rep. 9298 (Naval Research Laboratory, 1991).
  8. N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series (MIT Press, 1942).
  9. A. K. Jain, Fundamentals of Digital Image Processing (Prentice-Hall, 1989), pp. 278-299.
  10. D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).
  11. L. Brown, "A survey of image registration techniques," ACM Comput. Surv. 24, 326-376 (1992).
    [CrossRef]
  12. M. Elad and A. Feuer, "Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images," IEEE Trans. Image Process. 6, 1647-1658 (1997).
    [CrossRef]

2001 (1)

1999 (1)

J. M. Schuler and D. A. Scribner, "Increasing spatial resolution through temporal supersampling of digital video," Opt. Eng. 38, 801-805 (1999).
[CrossRef]

1998 (1)

1997 (3)

M. Elad and A. Feuer, "Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images," IEEE Trans. Image Process. 6, 1647-1658 (1997).
[CrossRef]

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
[CrossRef] [PubMed]

1992 (1)

L. Brown, "A survey of image registration techniques," ACM Comput. Surv. 24, 326-376 (1992).
[CrossRef]

1988 (1)

D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).

Alam, M. S.

M. S. Alam, J. G. Bognar, S. Cain, and B. J. Yasuda, "Fast registration and reconstruction of aliased, low-resolution frames by use of a modified maximum-likelihood approach," Appl. Opt. 37, 1319-1328 (1998).
[CrossRef]

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
[CrossRef] [PubMed]

Barnard, K. J.

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
[CrossRef] [PubMed]

Bognar, J. G.

M. S. Alam, J. G. Bognar, S. Cain, and B. J. Yasuda, "Fast registration and reconstruction of aliased, low-resolution frames by use of a modified maximum-likelihood approach," Appl. Opt. 37, 1319-1328 (1998).
[CrossRef]

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

Brown, L.

L. Brown, "A survey of image registration techniques," ACM Comput. Surv. 24, 326-376 (1992).
[CrossRef]

Cain, S.

Elad, M.

M. Elad and A. Feuer, "Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images," IEEE Trans. Image Process. 6, 1647-1658 (1997).
[CrossRef]

Feuer, A.

M. Elad and A. Feuer, "Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images," IEEE Trans. Image Process. 6, 1647-1658 (1997).
[CrossRef]

Hardie, R. C.

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
[CrossRef] [PubMed]

Huang, T. S.

T. S. Huang, "Multiple frame image restoration and registration," in Advances in Computer Vision and Image Processing (JAI Press, 1984), Vol. 1.

Ichioka, Y.

Ishida, K.

Jain, A. K.

A. K. Jain, Fundamentals of Digital Image Processing (Prentice-Hall, 1989), pp. 278-299.

Kondou, N.

Kruer, M. R.

D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).

Kumagai, T.

Longmire, M. S.

D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).

McHugh, M.

A. Shaum and M. McHugh, Analytic Methods of Image Registration: Displacement Estimation and Resampling, NPL Rep. 9298 (Naval Research Laboratory, 1991).

Miyatake, S.

Miyazaki, D.

Morimoto, T.

Schuler, J. M.

J. M. Schuler and D. A. Scribner, "Increasing spatial resolution through temporal supersampling of digital video," Opt. Eng. 38, 801-805 (1999).
[CrossRef]

Scribner, D. A.

J. M. Schuler and D. A. Scribner, "Increasing spatial resolution through temporal supersampling of digital video," Opt. Eng. 38, 801-805 (1999).
[CrossRef]

D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).

Shaum, A.

A. Shaum and M. McHugh, Analytic Methods of Image Registration: Displacement Estimation and Resampling, NPL Rep. 9298 (Naval Research Laboratory, 1991).

Tanida, J.

Wiener, N.

N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series (MIT Press, 1942).

Yamada, K.

Yasuda, B. J.

M. S. Alam, J. G. Bognar, S. Cain, and B. J. Yasuda, "Fast registration and reconstruction of aliased, low-resolution frames by use of a modified maximum-likelihood approach," Appl. Opt. 37, 1319-1328 (1998).
[CrossRef]

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

ACM Comput. Surv. (1)

L. Brown, "A survey of image registration techniques," ACM Comput. Surv. 24, 326-376 (1992).
[CrossRef]

Appl. Opt. (2)

IEEE Trans. Image Process. (2)

M. Elad and A. Feuer, "Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images," IEEE Trans. Image Process. 6, 1647-1658 (1997).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images," IEEE Trans. Image Process. 6, 1621-1628 (1997).
[CrossRef] [PubMed]

Opt. Eng. (1)

J. M. Schuler and D. A. Scribner, "Increasing spatial resolution through temporal supersampling of digital video," Opt. Eng. 38, 801-805 (1999).
[CrossRef]

Proc. SPIE (2)

M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda, "High resolution image reconstruction using multiple, randomly shifted, low resolution, aliased frames," Proc. SPIE 3063, 102-112 (1997).
[CrossRef]

D. A. Scribner, M. S. Longmire, and M. R. Kruer, "Analytic modeling of staring infrared systems with multidimensional matched filters," Proc. SPIE 890, 81-91 (1988).

Other (4)

T. S. Huang, "Multiple frame image restoration and registration," in Advances in Computer Vision and Image Processing (JAI Press, 1984), Vol. 1.

A. Shaum and M. McHugh, Analytic Methods of Image Registration: Displacement Estimation and Resampling, NPL Rep. 9298 (Naval Research Laboratory, 1991).

N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series (MIT Press, 1942).

A. K. Jain, Fundamentals of Digital Image Processing (Prentice-Hall, 1989), pp. 278-299.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1
Fig. 1

Conventionally designed imager with depth of optics h c equal to the focal length f is compared to TOMBO concept design containing n 2 lenslets and compacted depth of optics equal to fn.

Fig. 2
Fig. 2

Detector response function h d e t (dashed curves), optical PSF h o p (dashed–dotted curves), resulting system response function (solid curves), aliasing replicas due to sampling effects (dotted curves), and system MTF (thick solid curves) of the imager with 40   μm pixel FPA and (a) lens with 40   μm diameter PSF designed as standard system; (b) lens with 20   μm diameter PSF designed as standard system; (c) lens with 20   μm diameter PSF designed as superresolution system.

Fig. 3
Fig. 3

(a) Separations of 2 pixels, 3 pixels, and 4 pixels between δ functions in the comb with an optical PSF spot size of 1 pixel; (b) separation of 2 pixels between δ functions in the comb with an optical PSF spot size of 0.25 pixel, 0.5 pixel, and 1 pixel. SNR = 80 .

Fig. 4
Fig. 4

Image of Ansel Adams poster acquired using a traditional 25   mm camera lens. Image (a) fills the FPA at an object distance of 1 m. For comparison, image (b) is one of 64 lenslet subimages collected at an object distance of 4 m and upsampled four times using a polyphase filter. Image (c) is a reassembled image from the 64 subimages after applying the superresolution algorithm.

Fig. 5
Fig. 5

Image of Ansel Adams poster acquired using Lightpath Technologies model 350430 plano–convex aspheric lenslet ( 4 .32   mm focal length, 2.00   mm diameter, f∕2.2). Image (a) is taken at distance of 0 .35   m . For comparison, image (b) is one of 64 lenslet subimages collected at an object distance of 1 .4   m and upsampled four times using a polyphase filter. Image (c) is a reassembled image from the 64 subimages after applying the superresolution algorithm.

Equations (20)

Equations on this page are rendered with MathJax. Learn more.

s o p ( x , y ) = h o p s g .
s p ( x , y ) = η q h det s o p .
s ˜ p ( k x , k y ) = η q h ˜ det h ˜ o p s ˜ g ( k x , k y ) .
s ˜ s ( k x , k y ) = l = n = s ˜ p ( k x + 2 l k x N y q , k y + 2 n k y N y q ) .
s f ( x , y ) = r s s .
s f p ( x , y ) = s f q ( x + h i , y + v i ) ,
s f p ( x , y ) s f q ( x , y ) + h i x s f q ( x , y ) + v i y s f q ( x , y ) + o ( h i 2 , v i 2 ) .
[ x , y ( x s f q ( x , y ) ) 2 x , y x s f p ( x , y ) y s f q ( x , y ) x , y x s f p ( x , y ) y s f q ( x , y ) x , y ( y s f q ( x , y ) ) 2 ] [ h i v i ] = [ x , y ( s f p ( x , y ) s f q ( x , y ) ) x s f q ( x , y ) x , y ( s f p ( x , y ) s f q ( x , y ) ) y s f q ( x , y ) ] .
W ( k x , k y ) = h ˜ o p * ( k x , k y ) h ˜ det * ( k x , k y ) | h ˜ o p ( k x , k y ) h ˜ det ( k x , k y ) | 2 + w ˜ ( k x , k y ) / s ˜ ( k x , k y ) ,
w ˜ ( k x , k y ) / s ˜ ( k x , k y ) 1 SNR .
w ˜ ( k x , k y ) / s ˜ ( k x , k y ) γ | Δ ˜ ( k x , k y ) | 2 ,
h o p ( x , y ) = 1 / 2 π σ 2 exp [ ( x 2 + y 2 ) / 2 σ 2 ] ,
h det ( x , y ) = 1 a b   rect ( x a , y b ) ,
s ˜ f ( k x , k y ) = l = n = π τ o p m 2 η q 4 F n o 2 n ˜ ( k x , l , k y , n ) × exp [ 2 π 2 σ 2 ( k x , l 2 + k y , n 2 ) ] sin c ( k x , l a ) × sin c ( k y , n b ) ,
s ( x , y ) = 1 [ n = l = π τ o p m 2 η q n ˜ ( m k x , n , m k y , l ) h ˜ o p * ( k x , 0 , k y , 0 ) h ˜ det * ( k x , 0 , k y , 0 ) h ˜ o p ( k x , n , k y , l ) h ˜ det ( k x , n , k y , l ) 4 F n o 2 ( | h ˜ o p ( k x , 0 , k y , 0 ) h ˜ det ( k x , 0 , k y , 0 ) | 2 + 1 / SNR ) ] + w ( x , y ) .
s ( x , y ) = π τ o p m 2 η q 4 F n o 2 n ˜ ( m k x , m k y ) exp [ 4 π 2 σ 2 ( k x 2 + k y 2 ) + 2 π i ( k x x + k y y ) ] sin c 2 ( k x a ) sin c 2 ( k y b ) d k x d k y ( exp [ 4 π 2 σ 2 ( k x 2 + k y 2 ) ] sin c 2 ( k x a ) sin c 2 ( k y b ) + 1 / SNR ) + w ( x , y ) .
n ( x , y ) = l = n = δ ( x n Δ x ) δ ( y l Δ y ) .
n ˜ ( x , y ) = 1 Δ x Δ y l = n = δ ( k x n Δ x ) δ ( k y l Δ y ) .
s ( x , y ) = π τ o p m 2 η q 4 F n o 2 Δ x Δ y [ SNR 1 + SNR + 2 n , l = 1 A n , l cos ( 2 π x n Δ x ) × cos ( 2 π y l Δ y ) ] + w ( x , y ) ,
A n , l = exp [ 4 π 2 σ 2 ( n 2 Δ x 2 + l 2 Δ y 2 ) ] sin c 2 ( a n Δ x ) sin c 2 ( b l Δ y ) exp [ 4 π 2 σ 2 ( n 2 Δ x 2 + l 2 Δ y 2 ) ] sin c 2 ( a n Δ x ) sin c 2 ( b l Δ y ) + 1 / SNR .

Metrics