Abstract

Typical electro-optic imaging systems produce image aliasing artifacts. Superresolution algorithms process multiple aliased images to yield a single high-resolution image. We design imaging systems by jointly optimizing the optics and postprocessing to maximize such multiframe imaging performance. We describe efficient software methods that can be used to perform joint design by use of commercially available lens design software.

© 2008 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. D. G. Stork and M. D. Robinson, "Information-based methods for optics/image processing co-design," AIP Conf. Proc. 860, 125-135 (2006).
    [CrossRef]
  2. M. D. Robinson and D. G. Stork, "Joint design of lens system and digital image processing," in Proceedings of the International Optical Design Conference (Optical Society of America, 2006), paper WB4.
  3. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
    [CrossRef]
  4. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
    [CrossRef]
  5. M. Elad and A. Feuer, "Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images," IEEE Trans. Image Process. 6, 1646-1658 (1997).
    [CrossRef]
  6. N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
    [CrossRef]
  7. R. Tsai and T. Huang, "Multiframe image restoration and registration," Adv. Comput.Vision Image Process. 1, 317-339 (1984).
  8. A. Rajagopalan and P. Kiran, "Motion-free superresolution and the role of relative blur," J. Opt. Soc. Am. A 20, 2022-2032 (2003).
    [CrossRef]
  9. A. K. Jain, Fundamentals of Digital Image Processing, 1st ed. (Prentice-Hall, 1989).
  10. ZEMAX User's Guide (Zemax Development Corporation, San Diego, Calif., 2004).
  11. M. Stenner, A. Ashok, and M. Neifeld, "Multi-domain optimization for ultra-thin cameras," in OSA Frontiers in Optics Technical Digest (Opitcal Society of America, 2006).
  12. P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
    [CrossRef]

2006 (1)

D. G. Stork and M. D. Robinson, "Information-based methods for optics/image processing co-design," AIP Conf. Proc. 860, 125-135 (2006).
[CrossRef]

2005 (1)

P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
[CrossRef]

2004 (2)

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

2003 (1)

2001 (1)

N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
[CrossRef]

1997 (1)

M. Elad and A. Feuer, "Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images," IEEE Trans. Image Process. 6, 1646-1658 (1997).
[CrossRef]

1984 (1)

R. Tsai and T. Huang, "Multiframe image restoration and registration," Adv. Comput.Vision Image Process. 1, 317-339 (1984).

Ashok, A.

M. Stenner, A. Ashok, and M. Neifeld, "Multi-domain optimization for ultra-thin cameras," in OSA Frontiers in Optics Technical Digest (Opitcal Society of America, 2006).

Catrysse, P. B.

P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
[CrossRef]

Elad, M.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

M. Elad and A. Feuer, "Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images," IEEE Trans. Image Process. 6, 1646-1658 (1997).
[CrossRef]

Farsiu, S.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

Feuer, A.

M. Elad and A. Feuer, "Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images," IEEE Trans. Image Process. 6, 1646-1658 (1997).
[CrossRef]

Golub, G.

N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
[CrossRef]

Huang, T.

R. Tsai and T. Huang, "Multiframe image restoration and registration," Adv. Comput.Vision Image Process. 1, 317-339 (1984).

Jain, A. K.

A. K. Jain, Fundamentals of Digital Image Processing, 1st ed. (Prentice-Hall, 1989).

Kiran, P.

Maeda, P.

P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
[CrossRef]

Milanfar, P.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
[CrossRef]

Neifeld, M.

M. Stenner, A. Ashok, and M. Neifeld, "Multi-domain optimization for ultra-thin cameras," in OSA Frontiers in Optics Technical Digest (Opitcal Society of America, 2006).

Nguyen, N.

N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
[CrossRef]

Rajagopalan, A.

Robinson, D.

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

Robinson, M. D.

D. G. Stork and M. D. Robinson, "Information-based methods for optics/image processing co-design," AIP Conf. Proc. 860, 125-135 (2006).
[CrossRef]

M. D. Robinson and D. G. Stork, "Joint design of lens system and digital image processing," in Proceedings of the International Optical Design Conference (Optical Society of America, 2006), paper WB4.

Stenner, M.

M. Stenner, A. Ashok, and M. Neifeld, "Multi-domain optimization for ultra-thin cameras," in OSA Frontiers in Optics Technical Digest (Opitcal Society of America, 2006).

Stork, D. G.

D. G. Stork and M. D. Robinson, "Information-based methods for optics/image processing co-design," AIP Conf. Proc. 860, 125-135 (2006).
[CrossRef]

M. D. Robinson and D. G. Stork, "Joint design of lens system and digital image processing," in Proceedings of the International Optical Design Conference (Optical Society of America, 2006), paper WB4.

Tsai, R.

R. Tsai and T. Huang, "Multiframe image restoration and registration," Adv. Comput.Vision Image Process. 1, 317-339 (1984).

Wandell, B. A.

P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
[CrossRef]

Adv. Comput.Vision Image Process. (1)

R. Tsai and T. Huang, "Multiframe image restoration and registration," Adv. Comput.Vision Image Process. 1, 317-339 (1984).

AIP Conf. Proc. (1)

D. G. Stork and M. D. Robinson, "Information-based methods for optics/image processing co-design," AIP Conf. Proc. 860, 125-135 (2006).
[CrossRef]

IEEE Trans. Image Process. (1)

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super-resolution," IEEE Trans. Image Process. 13, 1327-1344 (2004).
[CrossRef]

IEEE Trans. Image Process. (2)

M. Elad and A. Feuer, "Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images," IEEE Trans. Image Process. 6, 1646-1658 (1997).
[CrossRef]

N. Nguyen, P. Milanfar, and G. Golub, "A computationally efficient image superresolution algorithm," IEEE Trans. Image Process. 10, 573-583 (2001).
[CrossRef]

Int. J. Imaging Syst. Technol. (1)

S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and challenges in super-resolution," Int. J. Imaging Syst. Technol. 14, 45-57 (2004).
[CrossRef]

J. Opt. Soc. Am. A (1)

Proc. SPIE (1)

P. Maeda, P. B. Catrysse, and B. A. Wandell, "Integrating lens design with digital camera simulation," Proc. SPIE 5678, 48-58 (2005).
[CrossRef]

Other (4)

A. K. Jain, Fundamentals of Digital Image Processing, 1st ed. (Prentice-Hall, 1989).

ZEMAX User's Guide (Zemax Development Corporation, San Diego, Calif., 2004).

M. Stenner, A. Ashok, and M. Neifeld, "Multi-domain optimization for ultra-thin cameras," in OSA Frontiers in Optics Technical Digest (Opitcal Society of America, 2006).

M. D. Robinson and D. G. Stork, "Joint design of lens system and digital image processing," in Proceedings of the International Optical Design Conference (Optical Society of America, 2006), paper WB4.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1
Fig. 1

(Color online) Representative MTFs of a simple diffraction-limited F # 4 optical system, sensor pixel, and combined total transfer function up to the optical diffraction limit. The vertical bar at 100 lp∕mm represents the Nyquist frequency for a sensor with 5 μm pixels. The system transfer function reveals significant signal preservation in the aliasing region to approximately 400 lp∕mm. It is the signal information from this region that is extracted with multiframe processing. Above 400 lp∕mm the transfer function shows limited contrast preservation.

Fig. 2
Fig. 2

(Color online) General block diagram of the proposed optimization framework that jointly optimizes both the optical design parameters (denoted Φ k ) and the multiframe digital filters (denoted R k ). The optical design parameters include both the static parameters that do not change over time as well as the variable parameters that define the mechanical actuation of the optical system.

Fig. 3
Fig. 3

(Color online) Effect of the downsampling operation in the Fourier domain. An original high-resolution image spectrum s ( ω , ν ) is shown at left. During downsampling, the high-resolution image spectrum is divided into U 2 tiles (in this case U = 2 ). These U 2 tiles are then averaged (shown in the middle) to produce the downsampled image spectrum shown at right. Thus, we observe that the spectral value for a particular spatial frequency in the downsampled image (denoted by the black square) is the average of U 2 spectral values of the high-resolution image.

Fig. 4
Fig. 4

(Color online) Image at left is a triplet lens system in which a back lens is capable of tilting independently in both the horizontal and the vertical directions. The tilt specifications are based on a commercially available actuator that is intended for an image stabilization application.

Fig. 5
Fig. 5

(Color online) Graph of the MTF for the optical system designed for VGA resolution single-frame imaging. The multiple curves show the tangential and sagittal MTF to the maximum horizontal field for the green channel. The MTF of the optical design shows reasonable performance to the native sampling frequency of the VGA image sensor.

Fig. 6
Fig. 6

(Color online) (a), (b) Sequentially designed optical MTF values for the green channels to the maximum horizontal field. We show only the positive and negative Y tilts that are due to the symmetry of the spot diagrams with respect to the X tilts along the positive Y field axis. (c) The system's spot diagrams at these field angles. Toward the edge of the field, the spot diagrams indicate significant astigmatism that explains the reduced contrast.

Fig. 7
Fig. 7

(Color online) Spot diagrams for the multiple capture modes as the number of total frames increase from (a) K = 4 , (b) K = 5, (c) K = 6, (d) K = 7 . The arrows within the boxes indicate the tilt actuator degrees of freedom, which were chosen to ensure a certain amount of symmetry during image capture and to reduce the search space. The spot diagrams show the tilt degrees of freedom and not actual spot separations.

Fig. 8
Fig. 8

(Color online) (a), (b) Jointly designed optical MTFs for the green channels to the maximum horizontal field. We show only the positive and negative Y tilts that are due to the symmetry of the spot diagrams with respect to the X tilts along the positive Y field axis. (c) The system's spot diagrams at these field angles. The MTF curves as well as the spot diagrams indicate a significant amount of spherical aberration and coma. These aberrations mask some of the astigmatism introduced by the lens tilt.

Fig. 9
Fig. 9

(Color online) Comparison of the RMSE performance of the multiframe imaging systems as a function of the number of images K. The results demonstrate the advantages of designing the multiframe imaging system in a joint fashion rather than retrofitting an existing optical design. As a point of reference, the standard deviation of the noise alone for an individual captured image is 0.03.

Fig. 10
Fig. 10

(Color online) Graph of the MTF for a jointly designed multiframe imaging system with the tilt actuators turned off. The multiple curves show the tangential and sagittal MTF to the maximum horizontal field for the green channel. As expected, the imaging system shows some loss in contrast because of the increased spherical and coma aberrations, but these aberrations can be corrected with digital sharpening filters.

Fig. 11
Fig. 11

(a) Input target object for the simulation example. The object comprises 6 × 6 = 36 tiles of one-dimensional chirped sinusoidal functions. (b) An example of a captured image after simulation of the effects of the optical and sensor subsystems.

Fig. 12
Fig. 12

(a) Input target object cropped at the upper leftmost image region. (b) A similarly cropped region of the single captured image for the jointly designed system. The aliasing artifacts associated with undersampling are visible. (c) Cropped region for the reconstructed image produced by the sequentially designed system. (d) Cropped region of the reconstructed image produced by the jointly designed system. Although both systems restore aliased signal content, the jointly designed system shows improved contrast for the diagonal signal content.

Tables (1)

Tables Icon

Table 1 Specifications of a 1.3 Mpixel Mobile Phone Camera

Equations (14)

Equations on this page are rendered with MathJax. Learn more.

y k = D A ( Φ k ) s + n k ,
s = k R k y k .
R k = ( k A T ( Φ k ) D T D A ( Φ k ) + σ 2 C s 1 ) 1 A T ( Φ k ) D T .
c ( m , n ) = σ s 2 ρ x | m | ρ y | n | ,
M S E ( Φ k ) = σ 2 T r ( k A T ( Φ k ) D T D A ( Φ k ) + σ 2 C s 1 ) 1 .
y k ( ω , ν ) = D { A ( ω , ν , Φ k ) s ( ω , ν ) } + n ( ω , ν ) ,
y k ( ω , ν ) = j = 0 U 1 i = 0 U 1 A ( ω + j 2 π U , ν + i 2 π U , Φ k ) × s ( ω + j 2 π U , ν + i 2 π U ) + n ( ω , ν ) .
p = M d + n ,
p = ( y 1 ( ω , ν ) y K ( ω , ν ) ) ,
d = ( s ( ω , ν ) s ( ω + 2 π ( U 1 ) U , ν + 2 π ( U 1 ) U ) ) ,
M = ( A ( ω , ν , Φ 1 ) A ( ω + 2 π ( U 1 ) U , ν + 2 π ( U 1 ) U , Φ 1 ) A ( ω , ν , Φ K ) A ( ω + 2 π ( U 1 ) U , ν + 2 π ( U 1 ) U , Φ K ) ) .
d ^ = ( M T M + σ 2 C d 1 ) 1 M T p .
M S E ( ω , ν ) = σ 2 T r ( M T M + σ 2 C d 1 ) 1 .
M S E = 1 4 π ω ν M S E b ( ω , ν ) .

Metrics