Abstract

A camera module employing spherical single-element lens imaging system (SSLIS) is introduced in this study. This type of imaging system can be used in compact digital cameras or mobile phone cameras, and it provides the advantages of simple design, reduced device bulkiness, and reduced manufacturing costs. When compared with conventional camera modules, our system produces radially variant blurred images, which can be satisfactorily restored by means of a polar domain deconvolution algorithm proposed in our previous study. In this study, we demonstrate an improved version of this algorithm that enables full-field-of-view (FOV) image restoration instead of the partial FOV restoration obtained via our previous algorithm. This improvement is realized by interpolating the upper and arc-shaped boundaries of the panoramic polar image such that the ringing artifacts around the center and four boundaries of the restored Cartesian image are greatly suppressed. The effectiveness of the improved algorithm is verified by image restoration of both computer simulated images and real-world scenes captured by the spherical single lens camera module. The quality of the restored image depends on the overall sparsity of all the point spread function (PSF) block Toeplitz with circulant blocks (BTCB) matrices used to restore a radially blurred image.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. Y. Koike, H. Hidaka, and Y. Ohtsuka, “Plastic axial gradient-index lens,” Appl. Opt.24(24), 4321–4325 (1985).
    [CrossRef] [PubMed]
  2. D. T. Moore, “Gradient-index optics: a review,” Appl. Opt.19(7), 1035–1038 (1980).
    [CrossRef] [PubMed]
  3. S. Kuiper and B. H. W. Hendriks, “Variable-focus liquid lens for miniature cameras,” Appl. Phys. Lett.85(7), 1128–1130 (2004).
    [CrossRef]
  4. W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
    [CrossRef]
  5. Y. Zhang and T. Ueda, “Field-dependent distortion coefficient and backward mapping for distortion correction of singlet lens cameras,” IEEE J. Trans. Elect. Electron. Eng.5(2), 203–210 (2010).
    [CrossRef]
  6. Y. Zhang and T. Ueda, “Design of a singlet lens and the corresponding aberration correction approaches for cell phone camera,” IEEE J. Trans. Elect. Electron. Eng.5(4), 474–485 (2010).
    [CrossRef]
  7. Y. Zhang and T. Ueda, “Deblur of radially variant blurred image for single lens system,” IEEE J. Trans. Elect. Electron. Eng.6(S1), S7–S16 (2011).
    [CrossRef]
  8. Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
    [CrossRef]
  9. H. C. Andrews and B. R. Hunt, Digital Image Restoration (Prentice-Hall, 1977), Chap.6,7,8 and 9.
  10. M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag.14(2), 24–41 (1997).
    [CrossRef]
  11. R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
    [CrossRef]
  12. S. J. Reeves, “Fast image restoration without boundary artifacts,” IEEE Trans. Image Process.14(10), 1448–1453 (2005).
    [CrossRef] [PubMed]
  13. M. Hanke and J. Nagy, “Toeplitz approximate inverse preconditioner for banded Toeplitz matrices,” Numer. Alg.7(2), 183–199 (1994).
    [CrossRef]
  14. J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
    [CrossRef] [PubMed]
  15. R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
    [CrossRef]

2011

Y. Zhang and T. Ueda, “Deblur of radially variant blurred image for single lens system,” IEEE J. Trans. Elect. Electron. Eng.6(S1), S7–S16 (2011).
[CrossRef]

Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
[CrossRef]

2010

Y. Zhang and T. Ueda, “Field-dependent distortion coefficient and backward mapping for distortion correction of singlet lens cameras,” IEEE J. Trans. Elect. Electron. Eng.5(2), 203–210 (2010).
[CrossRef]

Y. Zhang and T. Ueda, “Design of a singlet lens and the corresponding aberration correction approaches for cell phone camera,” IEEE J. Trans. Elect. Electron. Eng.5(4), 474–485 (2010).
[CrossRef]

2005

W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
[CrossRef]

S. J. Reeves, “Fast image restoration without boundary artifacts,” IEEE Trans. Image Process.14(10), 1448–1453 (2005).
[CrossRef] [PubMed]

2004

S. Kuiper and B. H. W. Hendriks, “Variable-focus liquid lens for miniature cameras,” Appl. Phys. Lett.85(7), 1128–1130 (2004).
[CrossRef]

1997

M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag.14(2), 24–41 (1997).
[CrossRef]

1996

J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
[CrossRef] [PubMed]

1994

M. Hanke and J. Nagy, “Toeplitz approximate inverse preconditioner for banded Toeplitz matrices,” Numer. Alg.7(2), 183–199 (1994).
[CrossRef]

1993

R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
[CrossRef]

1988

R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
[CrossRef]

1985

1980

Banham, M. R.

M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag.14(2), 24–41 (1997).
[CrossRef]

Biemond, J.

R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
[CrossRef]

Boekee, D. E.

R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
[CrossRef]

Chan, R. H.

R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
[CrossRef]

Fang, J.

W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
[CrossRef]

Hanke, M.

M. Hanke and J. Nagy, “Toeplitz approximate inverse preconditioner for banded Toeplitz matrices,” Numer. Alg.7(2), 183–199 (1994).
[CrossRef]

Hendriks, B. H. W.

S. Kuiper and B. H. W. Hendriks, “Variable-focus liquid lens for miniature cameras,” Appl. Phys. Lett.85(7), 1128–1130 (2004).
[CrossRef]

Hidaka, H.

Katsaggelos, A. K.

M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag.14(2), 24–41 (1997).
[CrossRef]

Koike, Y.

Kuiper, S.

S. Kuiper and B. H. W. Hendriks, “Variable-focus liquid lens for miniature cameras,” Appl. Phys. Lett.85(7), 1128–1130 (2004).
[CrossRef]

Lagendijk, R. L.

R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
[CrossRef]

Minema, I.

Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
[CrossRef]

Moore, D. T.

Nagy, J.

M. Hanke and J. Nagy, “Toeplitz approximate inverse preconditioner for banded Toeplitz matrices,” Numer. Alg.7(2), 183–199 (1994).
[CrossRef]

Nagy, J. G.

J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
[CrossRef] [PubMed]

R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
[CrossRef]

Ohtsuka, Y.

Plemmons, R. J.

J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
[CrossRef] [PubMed]

R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
[CrossRef]

Reeves, S. J.

S. J. Reeves, “Fast image restoration without boundary artifacts,” IEEE Trans. Image Process.14(10), 1448–1453 (2005).
[CrossRef] [PubMed]

Torgersen, T. C.

J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
[CrossRef] [PubMed]

Ueda, T.

Y. Zhang and T. Ueda, “Deblur of radially variant blurred image for single lens system,” IEEE J. Trans. Elect. Electron. Eng.6(S1), S7–S16 (2011).
[CrossRef]

Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
[CrossRef]

Y. Zhang and T. Ueda, “Field-dependent distortion coefficient and backward mapping for distortion correction of singlet lens cameras,” IEEE J. Trans. Elect. Electron. Eng.5(2), 203–210 (2010).
[CrossRef]

Y. Zhang and T. Ueda, “Design of a singlet lens and the corresponding aberration correction approaches for cell phone camera,” IEEE J. Trans. Elect. Electron. Eng.5(4), 474–485 (2010).
[CrossRef]

Varahramyan, K.

W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
[CrossRef]

Wang, W.

W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
[CrossRef]

Zhang, Y.

Y. Zhang and T. Ueda, “Deblur of radially variant blurred image for single lens system,” IEEE J. Trans. Elect. Electron. Eng.6(S1), S7–S16 (2011).
[CrossRef]

Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
[CrossRef]

Y. Zhang and T. Ueda, “Field-dependent distortion coefficient and backward mapping for distortion correction of singlet lens cameras,” IEEE J. Trans. Elect. Electron. Eng.5(2), 203–210 (2010).
[CrossRef]

Y. Zhang and T. Ueda, “Design of a singlet lens and the corresponding aberration correction approaches for cell phone camera,” IEEE J. Trans. Elect. Electron. Eng.5(4), 474–485 (2010).
[CrossRef]

Appl. Opt.

Appl. Phys. Lett.

S. Kuiper and B. H. W. Hendriks, “Variable-focus liquid lens for miniature cameras,” Appl. Phys. Lett.85(7), 1128–1130 (2004).
[CrossRef]

IEEE J. Trans. Elect. Electron. Eng.

Y. Zhang and T. Ueda, “Field-dependent distortion coefficient and backward mapping for distortion correction of singlet lens cameras,” IEEE J. Trans. Elect. Electron. Eng.5(2), 203–210 (2010).
[CrossRef]

Y. Zhang and T. Ueda, “Design of a singlet lens and the corresponding aberration correction approaches for cell phone camera,” IEEE J. Trans. Elect. Electron. Eng.5(4), 474–485 (2010).
[CrossRef]

Y. Zhang and T. Ueda, “Deblur of radially variant blurred image for single lens system,” IEEE J. Trans. Elect. Electron. Eng.6(S1), S7–S16 (2011).
[CrossRef]

IEEE Photon. Technol. Lett.

W. Wang, J. Fang, and K. Varahramyan, “Compact variable-focusing microlens with integrated thermal actuator and sensor,” IEEE Photon. Technol. Lett.17(12), 2643–2645 (2005).
[CrossRef]

IEEE Sens. J.

Y. Zhang, I. Minema, and T. Ueda, “Analysis of radially restored images for spherical single lens cellphone camera,” IEEE Sens. J.11(11), 2834–2844 (2011).
[CrossRef]

IEEE Signal Process. Mag.

M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag.14(2), 24–41 (1997).
[CrossRef]

IEEE Trans. Acoust. Speech

R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoust. Speech36(12), 1874–1888 (1988).
[CrossRef]

IEEE Trans. Image Process.

S. J. Reeves, “Fast image restoration without boundary artifacts,” IEEE Trans. Image Process.14(10), 1448–1453 (2005).
[CrossRef] [PubMed]

J. G. Nagy, R. J. Plemmons, and T. C. Torgersen, “Iterative image restoration using approximate inverse preconditioning,” IEEE Trans. Image Process.5(7), 1151–1162 (1996).
[CrossRef] [PubMed]

Numer. Alg.

M. Hanke and J. Nagy, “Toeplitz approximate inverse preconditioner for banded Toeplitz matrices,” Numer. Alg.7(2), 183–199 (1994).
[CrossRef]

SIAM J. Numer. Anal.

R. H. Chan, J. G. Nagy, and R. J. Plemmons, “FFT-based preconditioners for Toeplitz-Block least squares problems,” SIAM J. Numer. Anal.30(6), 1740–1768 (1993).
[CrossRef]

Other

H. C. Andrews and B. R. Hunt, Digital Image Restoration (Prentice-Hall, 1977), Chap.6,7,8 and 9.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (14)

Fig. 1
Fig. 1

Camera module with spherical single lens imaging system. (a) The spherical single-element lens fixed in a lens holder. (b)The CMOS image sensor IC. (c) Three-megapixel 2048[H] × 1536[V] image sensor with pixel size of 3.2 μm × 3.2 μm. (d) Side view of the camera module.

Fig. 2
Fig. 2

Schematic view of the SSLIS and optical aberrations obtained via real ray tracing for rays at different FOV values. (a) Schematic view. (b) Image sensor specifications. (c) Real ray tracing for rays in the semi-FOV range of 0°to 22.3°for infinite object distance, indicating severe off-axis aberrations. (d) Real ray tracing for rays at FOV of 0°for infinite object distance, indicating severe spherical aberration.

Fig. 3
Fig. 3

Distortion plot and distortion grid of the SSLIS. (a) Distortion in percentage vs. semi-FOV. (b) Distortion grid.

Fig. 4
Fig. 4

Lateral chromatic aberration of the SSLIS.

Fig. 5
Fig. 5

PSF distribution of our SSLIS. (a) 2D view, which indicates rotational symmetry (b) 3D view of PSFs along radial direction for different semi-FOV values. (c) Spot diagrams at different semi-FOV values, the full spot size is measured and marked in each sub images, the RMS spot size are also shown.

Fig. 6
Fig. 6

Image captured by our spherical single-lens camera module, which includes radially variant blur. (a) Captured image in the Cartesian domain (b) Panoramic polar image stretched out from (a). The corresponding boundaries in the Cartesian and panoramic polar images have been marked as A, B, C and D. (c) Nearest neighbor interpolation for the upper and arc-shaped boundaries of the polar image.

Fig. 7
Fig. 7

Coordinate system conversion and translation.

Fig. 8
Fig. 8

Visual comparison between an original image and the resulting images. (a) Original image. (b) Resulting image, the polar image resolution is 321 × 360. (c)Resulting image, the polar image resolution is 1280 × 7200.

Fig. 9
Fig. 9

Visual comparison between restored images with and without using the ringing reduction technique.. (a) Original image captured by a DSLR. (b) A software simulated image generated by the SSLIS. (c) Restored image without using the ringing reduction technique. (d) Restored image using the ringing reduction technique.(e) Magnified image of the central region of (c). (f) Magnified image of the central region of (d). (g) Magnified image of the upper region of (c). (h) Magnified image of the upper region of (d). (i) Magnified image of the right region of (c).(j) Magnified image of the right region of (d).

Fig. 10
Fig. 10

The full spot size versus semi-FOV values.

Fig. 11
Fig. 11

The building image. Visual comparison between the restored images by different number of segmented regions. (a)Original image captured by a DSLR.. (b) A software simulated image generated by the SSLIS. (c)Restored using PSF3.(d) Restored using PSF7.(e) Restored using PSF10.(f) Restored using PSF14(g) Restored using PSF3 and PSF7. (h) Restored using PSF7 and PSF14. (i) Restored using PSF7, PSF9, PSF12 and PSF14. (j)Restored using PSF10, PSF14, PSF16 and PSF18.(k)Relationship between the overall sparsity and RMSE

Fig. 12
Fig. 12

The English characters image. Visual comparison between the restored images by different number of segmented regions. (a)Original computer generated image. (b) A software simulated image generated by the SSLIS. (c)Restored using PSF3.(d) Restored using PSF7.(e) Restored using PSF10.(f) Restored using PSF14(g) Restored using PSF3 and PSF7. (h) Restored using PSF7 and PSF14. (i) Restored using PSF7, PSF9, PSF12 and PSF14. (j) Restored using PSF10, PSF14, PSF16 and PSF18.(k) Relationship between the overall sparsity and RMSE

Fig. 13
Fig. 13

Effectiveness of the improved restoration algorithm on real world scenes captured by the SSLIS camera. The upper row shows the captured images and the lower row shows the restored images.

Fig. 14
Fig. 14

Two dimensional structure of g e

Tables (4)

Tables Icon

Table 1 Lens specifications

Tables Icon

Table 2 Root mean square error (RMSE) between an original image and a resulting image obtained by coordinate conversions. Two different polar image resolutions are investigated.

Tables Icon

Table 3 Root mean square error (RMSE) between an original image and restored images with and without using the ringing reduction technique.

Tables Icon

Table 4 Restoration by different number of segmented regions

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

r ' =k ( i ' + i psfc i psfc ' i c ) 2 + ( j ' + j psfc j psfc ' j c ) 2 +1 r psfo + r psfo ' ,
θ ' = q 360 tan 1 i ' + i psfc i psfc ' i c j ' + j psfc j psfc ' j c +1 θ psfo + θ psfo ' ,
f ^ = ( H * H+α L * L) 1 H * g,
f ^ e = ( C H * C H +α C L * C L ) -1 C H * g e ,
( C H * C H +α C L * C L ) -1 C H * g e =F t * ( | Λ Η | 2 +α | Λ L | 2 ) -1 Λ Η * Ft g e ,
ifft{1./( [fft( c H )] 2 +α [fft( c L )] 2 ). [fft( c H )] * .fft( g e )},
( β/n ) wa = s=1 η w s · ( β/n ) s ,( s=1 η w s =1 )
( g g p )=( H 11 H 12 H 21 H 22 )( f 0 )+( n 0 )=( H 11 f+n H 21 f ),
g-H f ^ 2 +α L f ^ 2 .
min( g-H f ^ 2 +α L f ^ 2 )= ( g-H f ^ 2 +α L f ^ 2 ) f ^ =0.
( g-H f ^ 2 +α L f ^ 2 ) f ^ = [ (g-H f ^ ) * (g-H f ^ )] f ^ +α [ (L f ^ ) * (L f ^ )] f ^ = (gH f ^ ) * f ^ (gH f ^ )+ (gH f ^ ) * (gH f ^ ) f ^ +α (L f ^ ) * f ^ L f ^ +α (L f ^ ) * (L f ^ ) f ^ = (H) * (gH f ^ )+ (gH f ^ ) * (H)+α L * L f ^ +α (L f ^ ) * L =2 (H) * (gH f ^ )+2α L * L f ^ .
2 (H) * (gH f ^ )+2α L * L f ^ =0,
( H * H+α L * L) f ^ = H * g.
f ^ = ( H * H+α L * L) 1 H * g
( C H * C H +α C L * C L ) -1 C H * g e = [F t * ( | Λ H | 2 +α | Λ L | 2 )Ft] -1 (F t * Λ H * Ft) * g e =F t * ( | Λ H | 2 +α | Λ L | 2 ) -1 (F t * ) -1 F t * Λ H * Ft g e =F t * ( | Λ H | 2 +α | Λ L | 2 ) -1 Λ H * Ft g e ,

Metrics