Abstract

We propose a resolution enhancement method for mobile small f-number compact imaging systems based on wavefront coding and super-resolution image processing. Wavefront coding increases the focus depth of an optical system and produces point spread functions (PSFs) with similar characteristics at different field and defocus positions. The designed target wavefront is realized as a combination of wavefront errors of each rotationally symmetric lens, without including an additional phase plate. Finally, using one deconvolution filter containing all the characteristics of the PSFs, we achieve high resolution, breaking the diffraction limit of small f-number and the resolution limit of the image sensor by super-resolution image processing.

© 2008 Optical Society of America

1. Introduction

Wavefront coding can increase the focus depth of an optical system. Image quality is improved through an image restoration process in the spatial frequency domain called “super-resolution.” Wavefront-coded optical system uses a deconvolution filter that is designed based on point spread function (PSF) characteristics, which remain nearly constant at different field and defocus positions. In general, an asymmetric phase plate is placed in the optical system to realize the coded wavefront [1, 2]. In a compact imaging system, however, dimensions are fixed within the mobile devices, leaving no space for additional optical components. In addition, the image resolution of compact imaging systems is limited by their dimensions and by limitations on available lenses. Image resolution can be improved by decreasing the f-number that determines the resolving power of optics. However, the depth of focus becomes shorter with the f-number, limiting the possible decrease. Wavefront coding extends the depth of focus while maintaining a constant f-number. Finally, super-resolution enhances the resolution of the images over the diffraction limit by using a deconvolution filter containing the PSF characteristics of the optical system. The diffraction limit can be broken with the help of the calculation power of modern image processing and a high-resolution image sensor with a pixel size less than 1 µm.

In conventional optical design, small pixel sizes exceeding the Rayleigh resolution limit cannot be resolved even when the optical system exhibits no aberrations. As optics and digital image processing are separately developed, image resolution cannot break the limit of each part. Modern optical design utilizes several research fields simultaneously to break through the current limit of optical systems [3]. By optimizing optics and digital image processing simultaneously, such limits can be broken over the diffraction limit of optics and the spatial resolution limit of the image sensor.

However, commercial optical design software does not provide for simultaneous optics and digital image processing design. We developed an algorithm that optimizes the optics and digital image processing simultaneously, giving a final lens design that satisfies the given high-resolution specification. A wavefront-coded f-2.0 compact imaging system with four plastic lenses whose resolution is much higher than the conventional optical design is given as an example.

2. Overview of design methodology

Creating a wavefront-coded optical system involves optimizing the initial lens, wavefront, final lens, and deconvolution filter under various design constraints. Lack of phase plates is one such constraint. Designing the target wavefront to properly combine wavefront errors for each lens in the optical system is another. A schematic diagram of our design procedure is shown in Fig. 1.

Initial lenses are designed by conventional methods using commercial software, and the system is optimized to eliminate wavefront errors and ensure maximum spatial resolution at the best focus position.

The target wavefront, covering the whole field of view (over 60°) and defocus range (over 30 µm) of the optical system, must be constructed such that the PSFs of each position have similar shapes and characteristics. The wavefront has a rotationally symmetric shape, allowing it to be handled as a combination of wavefront errors for each lens without any additional optical components. Optimization is performed as a routine of PSF and modulation transfer function (MTF) calculations that finally satisfy the target of through focus MTF.

 

Fig. 1. Design procedure

Download Full Size | PPT Slide | PDF

Using a limited number of lenses from the initial lens design, the final optical system is designed for a target wavefront at the exit pupil; that is, the wavefront error of the final optical system matches the designed target wavefront.

Finally, a deconvolution filter is constructed as a weighted sum of PSFs over each field and defocus position. Using this deconvolution filter and a simulated image from the wavefront-coded optical system, the final image is restored in the spatial frequency domain and the spatial frequency response (SFR) is calculated. Optimization is performed as a calculation routine of filter and SFR for higher SFR values that satisfy the target specifications of resolution through the wide range of spatial frequency.

3. Initial lens design

We designed an f-2.0 compact imaging system for a mobile device using four plastic lenses. Design specifications are listed in Table 1. Figure 2 shows the diagram of lens design and spherical aberration. In conventional optical design and manufacturing processes, four plastic lenses barely manage to achieve resolutions corresponding to a 3 megapixel image sensor at the best focus position. SFR is calculated at 0.5 field position and the results of calculations are shown in Fig. 3. Results for the initial lens, designed by conventional methods, show degradation of resolution performance from the best focus position to the defocused position. This occurs because the PSFs of each defocus position have different characteristics, and the SFR is calculated by the Fourier transform of the image, which is simulated as a convolution of the PSFs and the original image.

As defined by the Rayleigh criterion, the small pixel size (under 1µm) of a high-resolution image sensor (over 10 megapixels) cannot be resolved by optics even when the system exhibits no aberrations. However, a high-resolution image sensor helps resolution enhancement in the deconvolution process with the calculation power of current hardware and memory.

Tables Icon

Table 1. Specification of the f-2.0 compact imaging system

 

Fig. 2. Initial lens design and spherical aberration

Download Full Size | PPT Slide | PDF

 

Fig. 3. Spatial frequency response of initial lens design.

Download Full Size | PPT Slide | PDF

4. Wavefront coding (target wavefront design)

To compensate for the insufficient performance of the initial lens design and obtain suitable optical characteristics for image restoration, the wavefront of the initial lens design is modified and optimized. The target wavefront extends the depth of focus and gives the PSFs of all fields of view similar characteristics.

4.1 Calculation of Target Specification

The depth of focus (δ) is defined by Eq. (1) using the Rayleigh criterion, which defines the resolution limit with the Airy Disc size. It gives only 8.6 µm for the f-2.0 camera. The short depth of focus of the small f-number camera is the main factor to be modified by wavefront coding. The conventional optics cannot generate enough depth of focus to match the large tilt of the auto-focus actuator, causing image blurring. The target specification of wavefront coding is calculated by the increase in the depth of focus desired to compensate for the maximum tilt of the auto-focus actuator.

δ=4λ(F/#)2
α=tan1(δd)
 

Fig. 4. Calculation of target specification

Download Full Size | PPT Slide | PDF

Figure 4 displays the schematic diagram of the target specification calculation. From the tilt (α) specification of the actuator, the target depth of focus is calculated, which will be the wavefront coding specification. Using Eq. (2), we find 30 µm of the target depth of focus.

4.2 Extending Depth of Focus Using Wavefront Coding

 

Fig. 5. Characteristics of conventional optical system (a) and Wavefront-coded optical system (b).

Download Full Size | PPT Slide | PDF

Infinite numbers of wavefronts are possible for the modified depth of focus. The pupil is divided into several zones and each zone is controlled for particular optical path differences (OPDs), as shown in Fig. 5. The wavefront is coded using several design parameters such as number of zones, sizes of each zone, and maximum OPDs of each zone.

4.3 Calculation of PSF and MTF

After the wavefront is coded, it is fitted with Zernike polynomials and implemented in the initial design, permitting calculations of the PSF and MTF. The PSF is calculated at each field of view and defocus position and the MTF is obtained as the Fourier transform of the PSF.

4.4 Optimization of the Wavefront

Optimization is performed as a calculation routine of MTFs at each field and defocus position using the design parameters (number of zones, sizes of each zone, and maximum OPDs of each zone) as variables. Finally, we generate a target wavefront with an extended depth of focus and nearly constant PSF characteristics through the broad range of defocus positions as shown in Fig. 6. Figure 6 represents the PSFs of conventional optical system (a) and wavefront-coded optical system (b) at 0.0 and 0.5 field position.

 

Fig. 6. PSFs of conventional optical system (a) and Wavefront-coded optical system (b).

Download Full Size | PPT Slide | PDF

5. Final lens design

The final lens is designed using the optimized wavefront. The optimized target wavefront is realized as a combination of wavefront errors of each lens, without any additional optical plate, as the compact camera is small (under 5mm) and is fixed within a mobile device, which uses the camera module as a capturing device.

Final lens design, particularly the construction of a target wavefront with few lenses, can be another research area in the development of wavefront-coded optical systems. In this paper, Zernike coefficients of target wavefront are used for the final lens design. Using the Zernike coefficients as design constraints, the lenses from the initial design are modified such that the target wavefront is realized at the exit pupil of the final optical system without the use of additional optical components such as wave plates. Individual lenses are easily optimized by commercial software; furthermore, specific lenses can be chosen to be modified in order to relax the tolerance sensitivities of the optical system. The final lens design and its spherical aberration are shown in Fig. 7, and the spatial frequency response is shown in Fig. 8.

 

Fig. 7. Final lens design and graph of spherical aberration

Download Full Size | PPT Slide | PDF

 

Fig. 8. Spatial frequency response of the final lens design

Download Full Size | PPT Slide | PDF

6. Deconvolution Filter Design and Image Restoration

6.1 Deconvolution Filter Design

Using super-resolution image processing, the captured image is restored by deconvolving the image with a spatial filter called the “deconvolution filter.” Breaking the diffraction limit is possible with the help of recent developments in digital sensors and memory technologies. The deconvolution filter is designed as a weighted sum of PSFs that best describes the response of point source as an optical transfer function.

6.2 Image Restoration and Calculation of Spatial Frequency Response

Images are restored in the spatial frequency domain for real-time implementation in mobile devices. Images are simulated at several defocus positions from the final lens design and the images have correspondence to the tilt angle of AF actuator. All the images are restored by only one deconvolution filter and SFRs are calculated to optimize the deconvolution filters. Figure 9 shows the process of image restoration and filter optimization.

 

Fig. 9. Process of image restoration

Download Full Size | PPT Slide | PDF

6.3 Optimization of the Deconvolution Filter and Final Result

Optimizing the deconvolution filter is one of the most important parts in design procedures with the optimization of target wavefront. Optimization is also performed as a calculation routine of filter and SFR as shown in Fig. 9.

The final result of SFR is calculated from the restored images (Fig. 11 and Fig. 12) at 0.5 field position. And it is overcoming the diffraction limit of f-2.0 camera at best focus position as shown in Fig. 10. The resolution data shows outstanding performance of the wavefront-coded optical system through the super-resolution image processing, which has SFR over 60% in the high-frequency region even when the system is defocused (within 30 µm) by the tilt of the AF actuator.

 

Fig. 10. Final result of SFR

Download Full Size | PPT Slide | PDF

Figures 11 and 12 are images of the conventional optical system and wavefront-coded optical system. Images are magnified for effective comparison from the central and outer regions. Final images represent the extended depth of focus of wavefront coding and the resolution enhancement of super-resolution image processing.

 

Fig. 11. Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the central region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.

Download Full Size | PPT Slide | PDF

 

Fig. 12. Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the outer region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.

Download Full Size | PPT Slide | PDF

7. Conclusions

Traditional optical design was focused on minimizing the wavefront error of optical system with a proper combination of spherical and aspherical lenses. In this paper, we generate certain amount of wavefront error which extends the focus depth of optical system and enables the final images to be restored compensating the tilt of auto-focus actuator. It is possible because the wavefront-coded optical systems display similar PSF shapes in a broad range of focus and wide field positions. The designed target wavefront is realized as a combination of the wavefront errors of each lens without any additional optical components. This means that compact imaging systems can be designed with special coded wavefront shapes while maintaining their dimensions and f-number. Using super-resolution image processing with only one deconvolution filter, the final image has a spatial resolution that breaks the diffraction limit. We successfully designed a small f-number optical system composed of only four plastic lenses whose resolution breaks the diffraction limit and meets the resolution specification of 10 megapixel image sensor. SFR is enhanced over 65% at the best focus position in the high-frequency region.

Acknowledgments

This work was supported by the Korea Science and Engineering Foundation (KOSEF) grant funded by the Korea government(MEST) (No. R17-2008-040-01001-0).

References and links

1. K. Kubala, E. R. Dowski, and W. T. Cathey, “Reducing complexity in computational imaging systems,” Opt. Express 11, 2102–2108 (2003). [CrossRef]   [PubMed]  

2. E. R. Dowski and W. T. Cathey, “Extended depth of field through wavefront coding,” Appl. Opt. 34, 1859–1866 (1995). [CrossRef]   [PubMed]  

3. W. T. Cathey and E. R. Dowski, “New paradigm for imaging systems,” Appl. Opt.41, 6080–6092 (2002).

References

  • View by:
  • |
  • |
  • |

  1. K. Kubala, E. R. Dowski, and W. T. Cathey, "Reducing complexity in computational imaging systems," Opt. Express 11, 2102-2108 (2003).
    [CrossRef] [PubMed]
  2. E. R. Dowski and W. T. Cathey, "Extended depth of field through wavefront coding," Appl. Opt. 34, 1859-1866 (1995).
    [CrossRef] [PubMed]
  3. W. T. Cathey and E. R. Dowski, "New paradigm for imaging systems," Appl. Opt. 41, 6080-6092 (2002).

2003 (1)

2002 (1)

1995 (1)

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1.
Fig. 1.

Design procedure

Fig. 2.
Fig. 2.

Initial lens design and spherical aberration

Fig. 3.
Fig. 3.

Spatial frequency response of initial lens design.

Fig. 4.
Fig. 4.

Calculation of target specification

Fig. 5.
Fig. 5.

Characteristics of conventional optical system (a) and Wavefront-coded optical system (b).

Fig. 6.
Fig. 6.

PSFs of conventional optical system (a) and Wavefront-coded optical system (b).

Fig. 7.
Fig. 7.

Final lens design and graph of spherical aberration

Fig. 8.
Fig. 8.

Spatial frequency response of the final lens design

Fig. 9.
Fig. 9.

Process of image restoration

Fig. 10.
Fig. 10.

Final result of SFR

Fig. 11.
Fig. 11.

Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the central region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.

Fig. 12.
Fig. 12.

Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the outer region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.

Tables (1)

Tables Icon

Table 1. Specification of the f-2.0 compact imaging system

Equations (2)

Equations on this page are rendered with MathJax. Learn more.

δ = 4 λ ( F / # ) 2
α = tan 1 ( δ d )

Metrics