1. Introduction

Light field data are widely used in many applications such as biomedical imaging, robotics and industrial inspection. The standard light field capturing device (light field camera) [1] proposed by R. Ng is made by placing a microlens array in front of a traditional digital camera’s sensor, which is possible to achieve different view angles via a single shot. However, the resolution of the captured light field is limited by the number of microlenses. The coded aperture based light field capturing systems [2,3] use a low-cost film mask instead of a microlens array, which can achieve higher spatial resolution. However, the reconstructed light field is darker because the mask limits the light throughput severely. Tajima et al. [4] proposed a lensless light-field camera composed by a multi-phased Fresnel zone aperture inserted before the image sensor. Although it presents the same problem as the mask based light field imaging system, this system has a very small overall size and a low-cost image reconstruction procedure. Singh et al. [5] discussed the possibility to use the diffuser as an imaging device that has a variable focal length and magnification. However, the proposed reconstruction method could only reconstruct the object at a determined depth. Antipa et al. [6] proposed an imaging system that use only a thin diffuser, which can also be used in communication [7,8] and cryptograph [9] applications, and a camera sensor to perform 3D volumetric object reconstruction. Its light throughput is higher than the mask based light field imaging systems because of the transparency of the diffuser. It has a very small form factor by system’s simple structure. However, the object reconstruction in this system is very complex as it needs to capture a great amount of PSFs under different depths in priori which limits its practical application. Xie et al. [10] proposed a geometric scaling way to obtain the PSFs at different depths by one single direct measurement and proved that the scaling method could extend the DOF of the diffuser camera. However, one single measurement could prone to large errors as the distance from the measured PSF increases.

It is of vital importance to obtain the PSFs in an easier and reliable way. The real capturing methods [11,12] require a lot of captures that is very time consuming, especially when the PSFs are spatially variant. It is not practical to capture all the point sources responses. Antipa et al. [13] calibrated the diffuser based light field imaging system via phase retrieval using transport of intensity. The retrieved phase is used to reconstruct the height map of diffuser, then the ray optics is used to model the system response. This method reduces the time to calibrate the system compared to direct capturing methods, but it still needs many through-focus images and the reconstruction algorithm is very time consuming. There are also some methods [14] of obtaining the PSFs for light field devices based on ray tracing and geometric optics. But when the optical element is scattering element like the diffusers, the ray tracing methods cannot solve the scatter effect unless there is a way to obtain diffuser’s transmission function which is very difficult to realize. Inspired by wave propagation [15] and computational Fourier optics [16], it is possible to construct a theoretical wave optics model for light field propagation. Such analyses have already been done on plenoptic camera 1.0 [17] and plenoptic camera 2.0 [18] systems. However, it lacks such analysis for the diffuser camera.

So, in this paper, we propose to derive diffuser camera’s PSFs under different depths by exploiting the forward propagation model. The model permits the estimation of PSFs in different depths which reduce the works of capturing PSFs. We will show that the only unknown variable for this model is diffuser’s phase which can be estimated by projection model. The correctness and effectiveness of the proposed model is demonstrated by evaluating the correlation between the theoretically derived PSF and the measured PSF, and comparing the reconstruction quality of the 2D objects at several depths. Experimental results showed that the reconstructions generated by the proposed model present comparable objective and subjective quality with those generated by the real captured PSFs with much lower reconstruction complexity, which will greatly benefit the application of diffuser camera.

The paper is organized as follows. The proposed model is described in detail in Section 2. Section 3 provides experimental results and discussions. Section 4 concludes the paper.

2. Proposed PSF model for the diffuser camera

2.1 Wave propagation modeling for the diffuser camera

The optical configuration of a diffuser camera is shown in Fig. 1. The light rays emitted from the object pass through an aperture, and then redirected by the diffuser toward the sensor.

 

Fig. 1 Optical configuration of a diffuser camera.

Download Full Size | PDF

According to scalar diffraction theory and Fresnel diffraction approximation [15], denoting the complex field at position (x₁,y₁) on the object plane and that at position (x₂,y₂) on the aperture plane by U₁(x₁,y₁) and U₂(x₂,y₂) respectively, the field propagation from the object plane to the plane before the aperture can be formulated by a convolutional integral:

As the PSF of the diffuser camera can be derived, the imaging target can be reconstructed once the object response can be captured by the sensor. The main challenge is to find the diffuser phase distribution θ_d in Eq. (7) since it is the only unknown variable to derivate the forward propagation PSF.

2.2 Diffuser phase estimation by projection model

Since the only unknown variable in the wave propagation model is the diffuser’s phase, we propose a method to estimate the diffuser’s phase inspired by laser beam shaping theory [19]. Laser beam shaping tries to adjust the laser beam irradiance profile by a far-field beam shaper, called light shaping diffuser. The light shaping diffuser is designed by adjusting the phase distribution under the given far-field irradiance constraints. It can be borrowed to our work to estimate the unknown phase distribution for the diffuser based on the measurement of the far-field and near-field intensity distributions through the diffuser.

The configuration to estimate the diffuser phase is shown in Fig. 2(a). The diffuser limited by a circular aperture placed on z = 0 is illuminated by a laser beam and the sensor is placed at z = 0 and z = z’ which correspond to near-field and far-field configurations, respectively. Suppose the diffuser plane is at position (x₂, y₂), and the diffuser and aperture are removed, the near-field of laser beam at z = 0 is given by

 

Fig. 2 Experimental setup to estimate diffuser’s phase: (a) general architecture to obtain the near-field and far-field intensity distributions; (b) experimental setup to obtain the far-field intensity distribution

Download Full Size | PDF

As the high spatial frequency component in diffuser is more evident than in laser beam, the shape of the energy envelope of the far-field intensity distribution is dominated by the phase function of the diffuser but not the divergence of the input beam. This indicates that the far-field distribution of laser beam combined with diffuser modulation is related with the near-field by a Fourier transform and diffuser’s phase has major influence to the final output pattern, which is independent with the shape of the input laser beam. Since the near-field and far-field intensity distributions can be measured, the phase of the diffuser which corresponds the near-field phase component can be easily retrieved by state-of-the-art phase retrieval methods [20–22].

As it is difficult to measure the far-field distribution in real experiment, we propose in using a Fourier lens to achieve the “Fourier transformation” as shown in Fig. 2(b). The diffuser is placed at front focal plane of the Fourier lens and sensor captures the intensity distribution at the back focal plane of the Fourier lens. For this arrangement, the field at the focal plane is given by

The intensity distributions $\left|U_{\text{2}}\right|$ and $\left|U_{\text{2}}^{\text{'}}\right|$ can be captured by a CMOS sensor. The diffuser’s phase θ_d associated with U₂ can thus be calculated via phase retrieval methods and the result will be inserted to Eq. (7) to calculate the PSF.

3. Experimental results

3.1 Simulated experiments

The proposed diffuser phase estimation is first verified by using the simulated near-field and far-field, comparing the estimated phases with the ground truth, and comparing the reconstructed results. The simulation grid is composed by a 4000 × 4000 pixels plane with each pixel corresponding to the physical size 5μm. The diffuser has plane size 512 × 512 pixels and one circular aperture with physical radius $\text{1}\text{.2}mm$ is added to the center of the diffuser.

Ground truth of the diffuser’s phase is created by a 512 × 512 gray “cameraman” image scaled proportionally from -π to π to represent the phase distribution as shown in Fig. 3(a). The near-field intensity distribution, as shown in Fig. 3(b), is given by uniform intensity 1. The far-field intensity distribution, as shown in Fig. 3(c), which in real experiment would be captured by the CMOS sensor, is obtained by applying the ground truth near-field to Eq. (13) to simulate the Fraunhofer propagation process, then calculate the square of the absolute value. The Fourier lens used here has focal length 5cm. Using near and far-field intensity distributions, the diffuser’s phase could be estimated by Gerchberg-Saxton algorithm. The iterative process is illustrated in Fig. 4. Figure 3(d) shows the estimated phase after 2000 iterations of Gerchberg-Saxton algorithm. The central 300 × 300 pixels for both the simulated and ground truth phase images are extracted for Pearson correlation coefficient calculation. Pearson correlation is given by:

 

Fig. 3 Phase estimation simulation: (a) Ground truth phase distribution; (b) Intensity distribution of the near field; (c) Far-field intensity distribution generated by Eq. (13); (d) Estimated phases.

Download Full Size | PDF

 

Fig. 4 Gerchberg-Saxton algorithm adapted for diffuser phase estimation illustration, where the known variables are near-field intensity and far-field intensity.

Download Full Size | PDF

Then, we apply the estimated phase information to the proposed PSF model and reconstruct the object by deconvolution. The testing object, as shown in Fig. 5(a), is placed with distances z₁ = 25mm, z₂ = 1mm (as shown in Fig. 1). The simulated image on the sensor is shown in Fig. 5(b), which is obtained by using Fresnel propagation, assuming that the diffuser has a pure phase distribution given in Fig. 3(a). The ground truth phase distribution and the ground truth PSF that is generated by the ground truth phase distribution are shown in the left-most column in Fig. 5(c). Also, the ground truth reconstruction result is shown in the same column, which is obtained by deconvolving the sensor response with the PSF generated by the ground truth phase. The deconvolution algorithm used here is the non-regularized Richardson-Lucy algorithm [23], which is an iterative algorithm that computes maximum likelihood estimation adapting to Poisson statistics. Its one iteration step is given by:

 

Fig. 5 (a) Simulated object for reconstruction with physical size 0.35mm × 0.35mm; (b) Simulated sensor response; (c) The first row corresponds to the estimated phase with different errors, second row corresponds to the model derived PSFs using the phase in the first row, and the last row corresponds to the reconstruction results using the PSFs in the second row.

Download Full Size | PDF

From Fig. 5, it can be observed that the reconstruction quality increases with the increase in the quality of the recovered phase, which means that the quality of the recovered phase affects the reconstruction quality proportionally. However, when the MSE of the recovered phase becomes 0.1373 and lower, corresponding to higher quality of the recovered phase, the reconstruction qualities of the object are visually the same and the improvement in the objective quality, measured by SSIM and P, is also marginal, which reflects that the phase quality does not benefit reconstructing the object “F”. To illustrate how the reconstruction performs for a more complex object in a more realistic phase distribution, we created a “virtual” diffuser using a random phase distribution, where the minimum feature size is 30μm, corresponding to 1-degree deflection angle, shown in the top-left corner in Fig. 6(c). The object is changed to a “flower”, which is more complex than “F” with tiny holes like that shown Fig. 6(a). The same experiment flow is performed as that in Fig. 5 and the results are shown in Fig. 6(c). For the complex object reconstruction, the positive correlation between the reconstruction quality and the phase quality still holds. However, it is observed that for the complex object, the reconstruction quality is not visually as good as that of “F”. Even the error in the recovered phase drops to 0.1603, the reconstruction quality is not improved obviously. It means that the influence of the error to the reconstruction quality is different between the complex texture object and simple object. Determining quantitatively the error influence to different objects and phase distribution is still an open problem to be solved.

 

Fig. 6 Simulation with a more complex object and phase distribution: (a) Simulated object for reconstruction with physical size 0.35mm × 0.35mm; (b) Simulated sensor response; (c) The first row corresponds to the estimated phase with different errors, second row corresponds to the model derived PSFs using the phase in the first row, and the last row corresponds to the reconstruction results using the PSFs in the second row.

Download Full Size | PDF

3.2 Real imaging experiments

The performance of the proposed model is further tested on a real diffuser camera shown in Fig. 7(a). The diffuser used here is a 1-degree Luminit light shaping diffuser [25]. The camera sensor is a point grey CMOS sensor GS3-U3-123S6C-C with pixel size 3.45μm and resolution 4096 × 3000. The illumination source for the phase estimation is a laser with wavelength 532nm. The laser passes through a beam expander to produce the parallel light source. The circular aperture we used has clear radius of $\text{2}\text{.3mm}$. The Fourier lens used to produce the far-field intensity distribution has focal length 5cm. The far-field and the near-field intensity captured are shown in Fig. 8(a) and (b), respectively. As the dynamic range between near-field and far-field intensities are large, the exposure time is adjusted to obtain images with similar brightness. Also, the background pixel value is subtracted for both images. They are further normalized by min-max normalization before entering the GS algorithm. The near field intensity distribution is captured by placing the diffuser on the camera sensor, where the separation distance is approximately the thickness of sensor’s cover glass (0.5mm). The far field intensity distribution is captured according to Fig. 2(b). The phase estimated is shown in Fig. 8(c).

 

Fig. 7 Diffuser camera system: (a)Vertical and side view of the diffuser camera system; (b)Test samples used for reconstruction.

Download Full Size | PDF

 

Fig. 8 (a) Captured far-field intensity distribution; (b) Captured near-field intensity distribution; (c) Estimated phase distribution.

Download Full Size | PDF

Using the estimated phase and Eq. (7), PSF can be generated at any object distance. The generated PSF is cropped within its central 900 × 900 pixels and the object can be reconstructed by deconvolving the image response with the corresponded cropped PSF. The PSFs calculated by the proposed model from z₁ = 22mm to z₁ = 30mm, shown in the third row of Fig. 9, are compared in terms of correlation with the PSFs captured by point sources (real PSFs) using a 3D moveable platform at z₁ from 22mm to 30mm, shown in the first row of Fig. 9, where the distance z₂ (distance from the cover glass) is adjusted by defocus level. The second row of Fig. 9 shows the PSFs obtained by the scaling method [10], where the captured PSF at z₁ = 22mm is made as the anchor and the corresponding scaling is applied to other depths. As shown in the figure, the scaled PSFs only present high correlation P(A, B) with the real captured ones as z₁ = 22mm and 24mm since their input PSF is the captured PSF at z₁ = 22mm. As they scale to other depths, P(A, B) drops quickly since the errors propagate intensively. In comparison, our model derived PSFs present high correlation with the real captured ones, even with the distance increase, both from visual distribution and the Pearson correlation P(A, C). The increment in P(A, C) is mainly caused by the decrease in z₂ (the distance between the diffuser and the sensor), which results in the captured PSF is close to the near field light field we captured for phase estimation.

 

Fig. 9 Comparison between the real captured PSFs, scaled PSFs and our model derived PSFs.

Download Full Size | PDF

Then, two real imaging objects in Fig. 7(b) are tested by recording sensor images under the corresponding object distances. The captured sensor images of each object are shown in the first row of Fig. 10(a) and 10(b), respectively. The reconstructed results in the second, the third and the fourth rows of Figs. 10(a) and 10(b) are generated by deconvolving the sensor images by the corresponding real captured PSFs (shown in the first row of Fig. 9), the corresponding PSFs generated by scaling method [10] (shown in the second row of Fig. 9) and our model derived PSFs (shown in the third row of Fig. 9), respectively.

 

Fig. 10 Reconstruction results comparison for: (a) Object 1; and (b) Object 2 using the captured PSFs, scaled PSFs and our model derived PSFs provided in Fig. 9 correspondingly.

Download Full Size | PDF

It is observed that the subjective quality of the reconstruction generated by the proposed PSFs is comparable to those generated by the captured PSFs, while the reconstruction generated by the scaling method [10] does not work well as the scaling distance increases. The reason for the scaling method does not work well is that the direct measurement of the PSF does contain the errors and the errors propagate more intense as the depth increases, which can be observed in Fig. 9 that the correlation index decreases very quick as the scaling distance increases. It is shown by experiment that our method can extend wider depth (ranging from z₁ = 22mm to z₁ = 30mm) than the scaling method (ranging from z₁ = 22mm to z₁ = 26mm) because it is more robust to noises. The objective quality of the reconstructions is further compared in Table 1 using structural similarity (SSIM) and Pearson correlation between the reconstructed object and the ground truth images in Fig. 7(b). We can see that our reconstructions also provide comparable objective quality with the reconstructions using the real captured PSFs, which are consistent with the subjective quality presented above. Furthermore, it is observed that our method outperforms the scaling method in all metrics evaluation for z₁ larger than 24mm, which is also consistent with the subjective quality presented above. The possible error sources for the proposed PSF are due to the condition of measurements, where there is no guarantee that the point source is placed exactly on-axis. Also, the non-ideality in illumination condition, the thickness of diffuser (energy absorption) and sensor noises contribute to the error. Actually, the real captured PSFs are also subject to the errors during the measurement, but it can serve as a good indicator whether our proposed model is valid. It demonstrates that the proposed PSF model is robust to different objects at different imaging distances and can provide the reconstruction good enough for the real applications to save a huge amount of acquisitions in retrieving real PSFs, which will greatly benefit the application of diffuser cameras.

Table 1. Objective quality comparison using the ground truths in Fig. 7(b) as the references.

View Table

4. Conclusion

In this paper we proposed a theoretical model to derive the PSFs for diffuser camera systems. The phase of diffuser is estimated by the projection model, which is inspired by laser beam shaping theory applied for diffuser design. The experimental results demonstrate that the proposed models can generate the PSFs in different depths with high accuracy and robustness. Only using two captures to estimate the phase distribution, it can reconstruct the object at any depth configurations. It provides comparable reconstruction performance with the real captured PSFs, which facilitates the application of diffuser cameras.

In the future work, we are going to analyze quantitatively how the error in the phase estimation is manifested in PSF’s error by establishing a statistical relationship between the object complexity and the reconstruction quality. Another point to work is to extend the reconstruction of planar object to 3D volumetric object. Also, the reconstruction quality for the off-axis locations of the proposed method is affected by the off-axis aberration and other high angle phenomena. Few works have been proposed to model the aberration for diffuser camera system. Therefore, one of our future works is to improve the reconstruction quality by modeling the off-axis aberration.