1. Introduction

Traditional optical systems achieve point-to-point imaging from the objects to the image plane through bulky and expensive optical lenses [1]. As an emerging imaging technology, the lensless imaging systems replace the lenses with a light-weight planar encoding device, or directly places the target on the surface of the detector, which can effectively reduce the system size and cost. Lensless imaging reduces the complexity of optical system design by transforming aberration optimization problems of lenses into computational imaging problems [2]. In the past few years, a variety of coherent lensless systems have been proposed, such as coherent diffraction imaging [3,4] and on-chip microscopy [5,6]. However, the imaging quality of these systems is closely dependent on the coherence of the light source, which greatly limits the application of imaging systems.

As a common light source in life, incoherent light can effectively avoid problems such as speckle noise caused by optical components in coherent light imaging, and can also cause interference and holographic imaging of objects through the design of the optical path, so that holographic technology has been applied in more fields. Incoherent lensless imaging technology originated in the early 1960s. Incoherent correlation holography technology, proposed by Mertz et al. [7], overcomes the limitations of using coherent light sources in the holographic recording process. It has been widely adopted in various fields and has become a hot research topic [8–12]. Rosen et al. proposed a Fresnel Incoherent Correlation Holography in 2007 [13]. This technology utilizes the correlation between the object light wave information and the Fresnel zone aperture (FZA) to record incoherent holograms. The use of incoherent light can expand the application of holographic technology to more fields.

Since the image sensor is only sensitive to the intensity of incident light, the phase information is easily lost during the image acquisition process, which will cause the twin image of the object to appear in the reconstructed image [14,15]. The traditional back propagation (BP) reconstruction algorithm is disturbed by twin image and has poor reconstruction quality. Over the past few decades, researchers have tried various ways to eliminate twin image [16]. Traditional methods for removing twin image include off-axis holography, phase shift, and compression sensing (CS). Wu et al. [2] used single-phase FZA mask plate for encoding under incoherent light illumination, and proposed an iterative reconstruction method based on CS, which is an embedded hologram-based view that leverages compressed sensing to remove the undesirable effects of twin images. However, the reconstruction quality of CS-based reconstruction methods is affected by calibration errors as well as manually selected parameters. Ma et al. [17] propose an explicit-restriction convolutional framework for lensless imaging, by tailoring its framework to specific factors, better perceived image quality can be achieved. This proposed framework can be extended to lensless imaging systems with different masks or structures. Moreover, it is difficult to obtain sufficient information from the encoding method of a single FZA phase, and the independent reconstruction strategy of each color channel is difficult to ensure the color fidelity.

In order to solve the problems of the above methods, Shimano et al. [18] acquired results using at least four coded images of band plates with different phases to achieve the reconstruction of non-twinning images [19,20]. Nakamura et al. arranged four FZAs with different phases on one mask to acquire multiple encoded images with different phases to eliminate the influence of twin images brought by coaxial holograms through color-channel synthesis [21]. However, their approach is to synthesize the modulation of multiple phases into one point-spread function, which does not effectively use the multiphase encoded information. Chen et al. [22] proposed the linear least-mean-square-error (LMSE), which based on wave optics to achieve super-resolution reconstruction by synthesizing high-resolution wavefront propagation functions. However, the multiphase FZA coding of this system is implemented by a phase-modulated spatial light modulator, which greatly increases the cost of lensless imaging systems. Although these two multiphase FZA lensless imaging have better image reconstruction quality, obtaining efficient high-dimensional image gradient distribution prior information and constraining the reconstruction process can further improve the quality of image reconstruction.

Deep learning technology provides new ideas for computational optical imaging. Wu et al. [23] present a prototype of a lensless camera that uses a deep neural network (DNN) to realize rapid reconstruction for FZA imaging. The network is designed with an error feedback mechanism, which realizes the self-correction of features to recover the image detail. Compared with traditional ADMMs, this method has comparable reconstruction quality, and the reconstruction time is two orders of magnitude faster. In recent years, deep generative learning models [24–27] based on score matching generative models [24,25] and Denoising Diffusion Probabilistic Model (DDPM) [26,27] have attracted widespread interest, which can generate high-quality samples without adversarial training. Song et al. [24,28] proposed a new model framework that incorporates generalized discrete fraction matching methods and diffusion models into the same framework. The model constructs the observed samples through the probability density function, and shows excellent performance in modeling the data distribution using known input invariance. Modeled with gradients of logarithmic probability densities, there is no need to normalize constants and learn directly using score matching [18]. The score model disturbs the data distribution by adding random noise in the forward process. To sample from the data distribution, a neural network can be trained to estimate the gradient of the logarithmic data distribution, i.e., the score function. The observed samples are then modeled by traversing the gradient of the target probability density function by Langevin dynamics [24]. The generative model maximizes the similarity of the data during the learning process of fitting parameters, so that prior information can be learned.

In order to solve the deficiencies of the existing technology, this paper proposes a multi-phase FZA lensless imaging algorithms based on diffusion model (MLDM). The training process uses a score-based generative model to continuously superimpose random noise on the original image to obtain a noisy image. The training part mainly obtains random samples of prior distribution by simulating the inverse of the stochastic process, and the result is to obtain the data gradient of the prior distribution. In this work, the MLDM algorithm is used to decode and reconstruct the encoded image, which uses multi-channel data distribution to encode image samples and sample data from the conditional distribution of the given observation data. The MLDM algorithm effectively characterizes the three-channel data distribution and its inline relationship of color images, thereby reconstructing high-quality images. During the reconstruction process, noisy images are predicted using the score function of the training process, where a fidelity step is added to enhance the consistency of the data [29]. The correction process uses annealing Langevin Dynamics, and the obtained noise reduction image then enters the prediction process iteration, and the data prior term and fidelity term are updated in each loop.

The rest of the manuscript is organized as follows. The second section introduces the principles of multi-phase lensless imaging and the background of score-based diffusion models. Detailed procedures and algorithms are described in Section 3. The experimental results and specifications regarding the implementation are given in Section 4. A discussion of the experimental results is arranged in Section 5. A conclusion is given in Section 6.

2. Preliminary

2.1 Lensless imaging principle

As shown in Fig. 1, lensless imaging directly applies an image sensor to record scene information encoded by an FZA mask, and then reconstructs the scene image through computer algorithms.

 

Fig. 1. The main processes of lensless imaging include optical encoding and computer decoding processes.

Download Full Size | PDF

The amplitude transfer function of an FZA is as follows [18]:

The set of all point sources from the object modulated by the FZA will be captured by the sensor under illumination of incoherent light. The imaging process can also be represented by the convolution of the target image and the FZA transmission function:

Under conditions of multi-phase imaging, a lensless encoding imaging system can be thought of as a linear transformation process which can be expressed as:

2.2 Score-based SDE

Generative models have achieved great success in the field of generating realistic and diverse data samples [24,32]. The forward stochastic differential equation (SDE) of the generative model achieves the purpose of perturbing the data distribution by adding Gaussian noise. The generative model can be trained to estimate the gradient of the logarithmic data distribution, and then used to numerically solve the reverse SDE to generate data from the noise to achieve the purpose of sampling from the data distribution.

Further elaborate on the above process, a continuous diffusion process $\{ x(t)\} _{t = 0}^T$ with $x(t) \in {{\mathbb R}^{{M_x}}}$ is indexed by a continuous-time variable $t \in [0,T]$. Where $x(0)\sim {p_0}$, ${p_0}$ represents the data distribution of the target image; and $x(T)\sim {p_T}$, ${p_T}$ is the prior distribution. The diffusion process can be modeled as the solution of the following SDE:

Different SDEs can be constructed by selecting different functions f and g. Since Variance Exploding (VE) SDE results in higher sample qualities, Song et al. developed the following VE-SDE:

Since all true ${\nabla _x}\log {p_t}(x)$ of t is uncertain, it can be estimated by training a score network ${S_\theta }(x,t) \simeq {\nabla _x}\log {p_t}(x)$ that changes over time. Then, samples can be generated by a variety of digital SDE solvers, such as the Euler-Maruyama method and the stochastic Runge-Kutta method [34].

Getting the score function of all t is necessary to solve Eq. (11), and estimating it using a time-conditioned neural network. ${S_\theta }(x,t) \simeq {\nabla _x}\log {p_t}(x)$ is a good way to do so, all weights and biases generated in interpolated and denoising networks are included in the training parameter $\theta $. Notably, the true score is unknown and it can be replaced by the unknown ${\nabla _x}\log {p_t}(x)$ with ${\nabla _x}\log {p_t}(x(t)|x(0))$, where ${\nabla _x}\log {p_t}(x(t)|x(0))$ is a Gaussian perturbation kernel centered on $x(0)$. Under certain conditions, training ${S_\theta }(x,t)$ with the ability to match denoising scores will definitely satisfy ${S_{\hat{\theta }}}(x,t) = {\nabla _x}\log {p_t}(x(t))$. Formally, the score network ground parameter $\theta$ is optimized using the following equation:

3. Proposed MLDM model

In the lensless imaging system, incoherent light illumination is used to transmit countless point light source lines in the scene through the FZA. These point sources are projected on the sensor and incoherently superimposed to finally form an encoded image. However, in the process of image acquisition, the image sensor can only acquire the intensity information of the image and lose the phase information, which leads to the appearance of twin image of objects in the reconstructed image [30,31]. On the problem of aforementioned, this work uses the correlation of different channels in the color space to realize the extraction of color multi-channel information, which solves the problem that the traditional single-channel reconstruction does not consider the correlation between channels. When using high-dimensional prior information for image processing, the prior information can be introduced into the processing model as a priori distribution, so as to realize the optimization of the processing process such as image repair and enhancement. Secondly, inspired by the phase-shifting techniques in the past few decades [18,19], different phase encoded image information is used in the iterative reconstruction process to further eliminate the twin effect.

As analyzed above, score-based generative model can obtain richer information and high-quality results by estimating the prior distribution of image data. Therefore, the learning of prior information is modeled using a VE-SDE, which learns more effective image distribution features. Two different schemes are proposed to solve the twin image problem in lensless imaging.

MLDM-I: Due to the high correlation of red (R), green (G), and blue (B) components in natural scene images, using components of other channels can effectively compensate for the lost information of a certain channel, thereby improving image quality reconstruction. Therefore, the distribution of RGB objects is modeled using a score-based generative model to learn prior information about the characteristics of the image distribution. Finally, the encoded information of different phases and the learned prior distribution are used in the iterative reconstruction process to improve the image reconstruction quality.

MLDM-II: In order to avoid restricting the data to low data density areas [35], this experiment effectively solves the above problems by improving the diffusion model, i.e., increasing the dimensionality of the processing objects. By broadening the input channel of the network and learning information representation in high-dimensional situations, it is easier to achieve the diversity of generation and further improve the quality of image reconstruction. Details will be presented in the following subsections.

3.1 MLDM-I: reconstruction in RGB object

The multi-phase lensless imaging problem boils down to solving Eq. (8), which is an underdetermined inverse problem. To solve this problem, various priors are incorporated into regularized objective function. The regularized objective function is expressed as:

Therefore, a score-based model trained on a target image dataset are used to introduce the learned priors into the lensless imaging inverse problem. Specifically, the images in the training dataset are fed one by one to the score-matching generative model for training. The model perturbs the data distribution by slowly adding Gaussian noise using an SDE. A continuous time-dependent score function ${S_\theta }(O,t)$ is trained to estimate the gradient ${\nabla _O}\log {p_t}(O(t))$ of the logarithmic data distribution by using the denoising score matching method of VE-SDE, Eq. (12) is further expressed as:

Through the above training, the reverse time VE-SDE can be solved once the trained score network ${S_\theta }(O,t)$ is brought into Eq. (11):

Comprehensively, the SDE bidirectional process is described in Fig. 2. Firstly, the forward VE-SDE diffuses the image data distribution $O(0) \sim {p_0}$ into a prior distribution $O(T) \sim {p_T}$. Secondly, to find the score of the distribution at each intermediate time step, one can estimate a score function with a time-conditional neural network ${S_\theta }(O(t),t) \simeq {\nabla _O}\log {p_t}(O(t))$. Finally, by reversing the forward process and starting from samples of $O(T) \sim {p_t}$, samples $O(0) \sim {p_0}$ can be obtained.

 

Fig. 2. Forward and reverse-time process of VE-SDE on RGB object.

Download Full Size | PDF

Then, Euler discrete method is used to solve VE-SDE numerically [24]. A predictor-corrector sampler is introduced to correct errors in the evolution of the discretized direction VE-SDE. The score-matching-based generative model first obtains a preliminary predicted reconstructed image from the obtained prior distribution of the target image by numerically solving the reverse time SDE, and then corrects this preliminary prediction result by using annealed Langevin as a corrector [36]. At the same time, when solving the reverse time SDE and annealing Langevin sampling, a data consistency is required after each unconditional sampling update step.

Specifically, the predictor refers to a numerical solver for the reverse-time SDE, which is used for giving an estimate of the sample at the next time step. In order to effectively enhance the consistency of the data, the variable ${V_i}$ is inserted into the DC formulation after the predictor updates it.

The variable ${O_i}$ obtained in Eq. (18) is input into the corrector, and the Langevin Markov chain correction algorithm is used to correct the gradient rising direction, and then the data consistency item is inserted into it.

Due to the inherent characteristics of lensless imaging, the reconstructed image has an obvious grid phenomenon, and there is a certain gap between the reconstructed image and the original image. To address this issue, total variation (TV) is applied to the reconstruction results.

Figure 3 shows the flowchart of the MLDM-I algorithm proposed in this work. The whole iterative reconstruction procedure consists of a two-level loop. In the iterative process of the outer loop, each loop updates the data prior items and data fidelity items of the prediction and correction process, and finally performs TV denoising. The number of outer loops is the number of discretization steps of the reverse time SDE. The correction process of the inner loop is performed by annealing the Langevin iteration. Comprehensively, the pseudo-code of the reconstruction algorithm of MLDM-I is shown in Algorithm 1.

 

Fig. 3. Reconstruction flow chart of MLDM-I algorithm. Top: Training stage to learn the gradient distribution of RGB object via denoising score matching. Bottom: Iterate between numerical SDE solver and data-consistency step to achieve reconstruction.

Download Full Size | PDF

3.2 MLDM-II: reconstruction in synthetic object

Inspired by the appealing performance of homotopic gradients of generative density priors over the naive score-based model NCSN, this experiment concludes that prior information learned from high-dimensional tensor is more effective for image reconstruction than the low-dimensional counterpart [37]. Therefore, replication technology is used for promoting the denoising score matching in generative model VE-SDE. In detail, a channel-copy transformation ${O^\ast } = B(O)$ is utilized to establish a $N$-channel higher-dimensional tensor. In our case dealing with lensless imaging, transforming $B(O)$ forms a high-dimensional tensor ${O^\ast }$. In this work ${O^\ast } = {(O,O)^T}$ with the setting of $N = 2$. The goal of achieving ${O^\ast }$ stacking is to create an object in high-dimensional manifold [38] that is favorable for subsequent network learning. This approach helps to avoid potential difficulties for both accuracy in score estimation and sampling with Langevin dynamics. Subsequently, the MLDM-II is trained with ${O^\ast }$ in high-dimensional space as network input and the parameterized ${S_\theta }({O^\ast },t)$ is obtained in the upper part of Fig. 5. The Eq. (14) is rewritten as:

Once the network is trained with Eq. (23), the approximation ${S_\theta }({O^\ast },t) \simeq {\nabla _{{O^\ast }}}\log {p_t}({O^\ast }(t))$ can be plugged to solve the reverse SDE in Eq. (11):

The Eq. (24) can be numerically solved with Euler discretization. This involves discretizing t in range [0, 1], which is uniformly separated into N intervals such that $0 = {t_0} < \cdots < {t_N} = 1$, $\Delta t = 1/N$. Additionally, the direction of gradient ascent can be corrected with Langevin Markov Chain Monte Carlo algorithm [36].

Here, the detail for implementing Eq. (24) is further given. An alternating optimization algorithm is used to minimize the decoupling of prior information items and data-consistency items. A predictor is used to estimate the predicted value of the sample at the next time step, and a DC step is added to enhance the consistency of the data.

Equation (26) can be further reduced as:

Next, perform a TV operation on the reconstruction result:

After the virtual variable $O_i^\ast $ is updated, it becomes an RGB three-channel data output via channel-average operator.

Annealed Langevin dynamics is used to further sample from the high-dimensional noisy data distribution with multi-view noise. To intuit the procedure of annealed Langevin dynamics, the intermediate samples are provided in Fig. 4, where each row shows how samples evolve from pure random noise to high-resolution images.

 

Fig. 4. Iterative generation of Langevin dynamics for annealing synthetic objects.

Download Full Size | PDF

 

Fig. 5. Reconstruction flow chart of MLDM-II algorithm. Top: Training stage to learn the gradient distribution of synthetic object via denoising score matching. Bottom: The lower part is there construction flow chart used for encoding image calculation and decoding.

Download Full Size | PDF

The flowchart of training phase and iterative reconstruction phase for MLDM-II is shown in Fig. 5. Specifically, this experiment first trains a score-based generative model to capture image prior distribution. Then, at the inference stage, the numerical SDE solver and data-consistency step is iteratively updated to achieve the image reconstruction. Since the Langevin dynamics updating can guarantee algorithms convergence, the overall MLDM-II algorithm will be convergent after finite iterations. The Algorithm 2 describes the reconstruction algorithm in detail.

4. Experiments

4.1 Data specification

Since there are currently no public datasets for FZA lensless imaging, all the training and testing in this experiment were carried out on the LSUN dataset [39]. LSUN is a large color image dataset with 10 scene categories and 20 object categories, each containing about one million labeled images. The outdoor scene church and indoor scene bedroom are chosen among them, then observe the training and testing effects of the network model on two different training sets, respectively. In addition, in order to further verify the performance of MLDM, 100 images of each dataset were selected as experimental test datasets.

4.2 Model training and parameter selection

The bedroom and church datasets given by the LSUN dataset were used for training, separately. Each image was reshaped into $256 \times 256$ pixels as preprocessing, and its pixel values were scaled to [0,1]. The data distribution was perturbed by Gaussian noise with noise values ranging from 0.01 to 380. Meanwhile, to avoid numerical problems, the parameter $\varepsilon$ was set to be 10⁻⁵. The signal to noise ratio (SNR) was set to be 0.16. The maximum value of gradient clipping was set to be 1.0. The parameters were exponential moving average. The flow rate was set to be 0.999 and each dataset was trained 500,000 times. For the PC sampling algorithm $N = 1000$, $M = 1$ iterations were used in this experiment. The largest peak signal to noise ratio (PSNR) and its corresponding structural similarity index measure (SSIM) were taken in this experiment, then the average of the 100 sets of data were calculated. A fixed learning rate of 0.00001 was used in the simulation experiment and Adam optimizer with ${\beta _1} = 0.9$ and ${\beta _2} = 0.999$ was utilized to optimize the network. The training and testing experiments were performed with NVIDIA 1080Ti GPU, 12 GB.

4.3 Quantitative indices

Due to the visual characteristics of the human eye, evaluation results often depend on the subjective perception of the person. For the effective quantitative evaluation of the experiments in this work, peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) are chosen as metrics.

The PSNR is based on the error between corresponding pixel points in dB. This indicator is based on error-sensitive image quality evaluation, the larger the value, the smaller the distortion. It is an objective evaluation index of images, defined by MSE as follows:

The SSIM is an index to measure the structural similarity of two images. It measures the similarity between images from three aspects: width, contrast and structure:

4.4 Simulation results

In the simulation experiments, a multiphase FZA mask was used as an encoder, the radius of the innermost area of the mask $r1$ was set to be 0.4 mm, and two phases of 0 and π/2 were used for encoding. The FZA mask was placed 3 mm away from the sensor. The sensor had a pixel count of 256 × 256 with a pixel pitch of 0.014 mm. A target with dimensions of 200 mm × 200 mm was placed 300 mm away from the FZA mask. Each pixel recorded a superposition of images from multiple apertures on the encoding mask. MLDM-I and MLDM-II were used to reconstruct the simulated data, respectively. In these experiments, the weight coefficients $v1$ and $v2$ used by MLDM-I were 0.05 and 1, and the weight coefficients $v1$ and $v2$ used by MLDM-II were 0.1 and 1.5, respectively.

Experiments on the LSUN bedroom dataset. For visual comparison, Fig. 6 selects two different sets of experimental results from the LSUN bedroom dataset for display, where each set of experimental results includes the ground truth, and the images reconstructed by the BP [2], CS [2], LSGM [40], MLDM-I and MLDM-II respectively. It can be seen intuitively that the reconstruction quality of the BP is the worst, and an obvious twin image effect can be observed. Besides, compared with the ground truth, the reconstruction results of the CS have obvious color fidelity, and image clarity is not well done. For example, in the second rows, the CS reconstruction results have obvious color gaps with the ground truth, with obvious color diffusion in the middle and discoloration of the areas at the edges of the images. Compared with BP and CS, LSGM is much better in the clarity and color accuracy of the reconstructed images, but there is still a relatively obvious gridding effect. Obviously, the image quality of the fifth and sixth column in Fig. 6 is significantly better than the other three sets of results, and the reconstruction results of MLDM-II are the best among the all algorithms.

 

Fig. 6. Visual comparison of reconstruction images on the LSUN bedroom dataset. (a) Ground Truth (b) BP (c) CS (d) LSGM (e) MLDM-I (f) MLDM-II.

Download Full Size | PDF

For further quantitative analysis, the average PSNR and SSIM of the all algorithms are shown in Table 1. It can be seen from the Table 1 that the average PSNR of the traditional BP is only 7.40 dB. In contrast, the other algorithms can achieve better results in an iterative manner. Comparing the data in the Table 1, the PSNR of CS and LSGM are nearly 10.28 dB and 17.06 dB higher than that of the BP. The MLDM-II proposed in this work has the best imaging quality. After iterative correction, its PSNR can reach 29.91 dB, and the SSIM can reach 0.941, which are 5.45 dB and 0.23 higher than that of the LSGM, respectively. In addition, from the reconstruction results of the fifth and sixth columns of Fig. 6, it can be concluded that the MLDM proposed in this work has a good reconstruction effect in both three and six channels. As shown in Table 1, PSNR can reach more than 29 dB and SSIM can reach more than 0.93. However, compared with MLDM-I, the PSNR and SSIM of MLDM-II reconstruction images will still have certain improvements, which proves that the increase in the number of channels can indeed improve the quality of the reconstructed images to some extent.

Table 1. Comparison of test results on LSUN bedroom dataset.

View Table | View all tables in this article

Experiments on the LSUN church dataset. To better verify the robustness of the MLDM, this experiment also selected the LSUN church dataset with richer grayscale, brightness and structural changes of the images for the experiment.

Figure 7 shows the comparison results of the five algorithms in this experiment on the LSUN church dataset. Same as the experiment in the previous section, this experiment shows two different groups of reconstructed images, each group contains the ground truth and the reconstruction results of the BP, CS, LSGM, MLDM-I and MLDM-II. Similar to the experimental results on the LSUN bedroom dataset, the reconstruction results of the BP are the worst, the images reconstructed by the CS always have colors that are not related to the images. For example, obvious purple and green twin image can be observed in the CS reconstruction results of the first row in Fig. 7, especially in the middle and the areas at the edges of the images. Although the reconstruction results of the CS do not affect the recognition of the image, the image quality is still blurred. Compared with CS and BP, LSGM has a great improvement in image quality and definition, but the MLDM-I and MLDM-II proposed in this experiment have better reconstruction effect than LSGM. And the gridding effect of MLDM-I is better solved than LSGM. Comparing the reconstruction results of the LSGM and MLDM-II in the second row in Fig. 7, it can be found that the MLDM-II has higher definition. According to the PSNR and SSIM given in Fig. 7, the quantitative evaluation indexes of the reconstruction of the MLDM-II are also the highest. Comparing the reconstruction results of each group in Fig. 7, the images in the last column are better than the images in the second, third, fourth and fifth columns in terms of image color, brightness and sharpness. Therefore, the MLDM-II is the best performer. It can clearly reconstruct the image details while ensuring the color accuracy, which is closer to the ground truth.

 

Fig. 7. Visual comparison of reconstruction images on the LSUN church dataset. (a) Ground Truth (b) BP (c) CS (d) LSGM (e) MLDM-I (f) MLDM-II.

Download Full Size | PDF

Quantitative comparison of PSNR and SSIM in Table 2 shows that among the five algorithms, MLDM-II has the highest PSNR and SSIM, and the average PSNR on the LSUN church dataset is close to the results on the LSUN bedroom dataset. After changing to a dataset with richer grayscale, brightness and structural changes, the evaluation indicators of the BP are almost unchanged. In conclusion, the MLDM-II proposed in this experiment perform relatively stable and the best in both indicators, with the highest PSNR reaching 28.87 dB and SSIM value reaching 0.933. Compared with the CS and LSGM, the average PSNR of the MLDM-II is 13.52 dB and 4.51 dB higher, and the average SSIM of the MLDM-II is 0.682 and 0.149 higher, respectively. Therefore, the method proposed in this experiment reconstructs the best image, not only can maintain the color, brightness and clarity of images, but also can significantly remove the artifact noise of twin image.

Table 2. Comparison of test results on LSUN church dataset.

View Table | View all tables in this article

Combining the two sets of experimental results, it can be concluded that on different datasets, compared with the BP, CS, LSGM, MLDM-I and MLDM-II, the MLDM-II can achieve higher reconstruction quality and the model performance is more stable.

4.5 Experimental validation

This section experimentally verified the proposed concept. As shown in Fig. 8, a self-designed multi-phase FZA mask is placed tightly onto the surface of the CMOS sensor (QHY163M, Light Speed Vision (Beijing) Co., Ltd, China) to capture the target displayed on the LCD screen. The resolution of the CMOS is 4650 × 3522. A chrome layer with a thickness of 0.1 mm is plated onto a substrate with dimensions of 127 × 127 × 2.3 mm. The entire mask plate is divided into 6 rows and 5 columns, resulting in a total of 30 regions, with each area measuring 20 × 20 mm. The r₁ parameters of FZA range from 0.2 mm to 0.8 mm, while the phases of FZA include 0, π/2, π, and 3π/2. According to the thickness of CMOS cover glass and FZA mask, the distance between the FZA and the sensor is 3.4 mm.

 

Fig. 8. The multi-phase lensless imaging system. The insets show the self-designed FZA mask and CMOS sensor, respectively

Download Full Size | PDF

Since this system uses a mono-chrome camera, the three-channel data is acquired by exposure of the three-channel image separately. The imaging target was loaded on the LCD screen at a distance of 300 mm from the FZA zone plate. The gathered coded image was saved onto the local computer. The 0 and π/2 phase-modulated sub-images, each with dimensions of 768 × 768, were cropped from the original images acquired by CMOS. They are then reduced to 256 × 256 pixels using pixel averaging and fed into the network for image reconstruction through the proposed MLDM methods.

To further verify the performance of the proposed MLDM algorithms, three images with significant color distributions and a mono-chrome image were used as imaging targets for the practical experiments. The $r1$ parameter of the FZA was chosen to be 0.4 mm and the phase was chosen to be 0 and π/2. The BP, CS, LSGM and ADMM algorithms are also used for comparison. Figure 9 shows the comparative experimental results of different methods on optical experiment. The BP algorithm produced the lowest quality results among all algorithms used, with significant information loss and severe color distortion. While the CS algorithm showed notable improvement compared to BP, significant color distortion was still present, along with uneven color distribution, excessive noise, overly smooth images, and poor sharpness. LSGM reconstruction improved clarity but still resulted in a noticeable grid effect, despite a reduction in uneven color distribution and noise. The grid effect of ADMM is improved compared to LSGM, but the clarity is not as good as LSGM, and artifacts are noticeable at the edges. The MLDM-I and MLDM-II proposed in this experiment exhibited excellent performance in restoring the target images, eliminating the twin image effect, and improving image clarity and color saturation. The reconstructed images of MLDM-II had more noticeable noise at the edges and lower clarity compared to MLDM-I, this can be attributed to the increase in noise introduced by the additional channels. Overall, the MLDM proposed in this experiment effectively eliminated the twin image effect in lensless imaging systems and produced reconstructed images of higher quality than other algorithms.

 

Fig. 9. Visual comparison of reconstructed images on optical experiment. (a) Ground Truth (b) BP (c) CS (d) LSGM (e) ADMM (f) MLDM-I (g) MLDM-II.

Download Full Size | PDF

To quantitatively analyze the performance of the system, the image reconstruction results of the practical experiment for PSNR and SSIM were calculated. As shown in Table 3, MLDM has the best reconstruction effect, with the highest PSNR values of 24.35 dB. In the comparison of different algorithms, BP has the worst reconstruction effect. The PSNR and SSIM of MLDM-I are 3.15 dB, 0.267 higher than CS in the third image, and the PSNR and SSIM of MLDM-I are 4.50 dB, 0.194 higher than LSGM in the second image. In the reconstruction results of all four images, the PSNRs of MLDM-I are higher than that of ADMM, especially in the second image, where the PSNR of MLDM-I is 5.30 dB higher than that of ADMM.

Table 3. PSNR and SSIM of different methods on the experiments

View Table | View all tables in this article

In the comparison of SSIM, the SSIMs of ADMM in the first and second images are higher than the other algorithms, while the SSIMs of MLDM-I are higher than the other algorithms in the third and fourth images. This is because the reconstruction effect of ADMM is dark, and the surrounding edges of the image to be reconstructed are black after pinhole imaging, which leads to the SSIM of ADMM reconstruction results being higher than other algorithms. However, from the perspective of the reconstructed images, the visual effect of MLDM is significantly better than that of ADMM. In summary, the MLDM proposed in this experiment can reconstruct higher-quality multi-channel color images compared with traditional algorithms.

5. Discussion

5.1 Effect of the number of FZA phases on the reconstruction results

In order to test the influence of the phase number on the quality of the reconstruction image, this section shows the reconstruction results of MLDM-I and MLDM-II under different FZA phase encodings. The LSUN church dataset were selected for simulation experiments. The objects were encoded with FZA phases of (0, π/2), (0, π), (π/2, 3π/2), and (0, π/2, π, 3π/2), individually.

As shown in Fig. 10, the first column is ground truth. The second to fourth columns are reconstruction images under bi-phase conditions and the fifth column are the reconstruction images under fourth-phase conditions. It can be clearly seen from the third column that the reconstructed results of the bi-phase of (0, π) have obvious color difference compared with the ground truth, and there is obvious gridding effect. In the fourth column, the color difference problem has been solved, and the color of the reconstructed image is basically the same as the ground truth. From the results in the second column to the fourth column of Fig. 10, it can be seen that under the bi-phase encoding, the phases of (0, π/2) has the best reconstruction performance, and the color fidelity and image clarity are better than other bi-phase reconstructed images. It can be seen from the results shown in the fifth column of Fig. 10 that the quality of the reconstructed images of the fourth-phase encoding method is better than that of the reconstructed images under the bi-phase condition.

 

Fig. 10. Visual comparison of reconstructed images with different phase numbers on the LSUN church dataset. (a) Ground Truth (b) 0, π/2 (c) 0, π (d) π/2, 3π/2 (e) 0, π/2, π, 3π/2.

Download Full Size | PDF

As shown in Table 4, the phases of (0, π/2) achieves the best reconstruction results in the bi-phase, and the fourth-phase reconstruction result has the highest PSNR and SSIM, which are 0.07 dB and 0.001 higher than the phases of (0, π/2) in the first image. In the second image, the fourth-phase reconstruction result still has the highest SSIM, while the phases of (0, π/2) has the highest PSNR. It can be seen that there is little difference between the reconstruction effects of the phases of (0, π/2) in the bi-phase and the fourth-phase.

Table 4. PSNR and SSIM of different number of phases on the LSUN church dataset

View Table | View all tables in this article

In conclusion, increasing the number of phases can improve the quality of recovered image, with the fourth phase producing the best result. Although there is a slight improvement when compared with the bi-phase of (0, π/2), it is important to consider the practicality of the approach, as increasing the number of phases also increases the computational cost. Therefore, the bi-phase of (0, π/2) is a better choice in practical scenarios.

5.2 Resolution analysis

In order to verify the minimum resolution that the proposed system can achieve, an USAF-1951 target was used for testing, the image was magnified to a size of 300 × 300 mm and displayed on the LCD monitor. This target was placed 500 mm away from FZA.

The image reconstruction results are shown in Fig. 11. The first column of images is the ground truth of the USAF-1951 test chart, and the second to seventh columns are the reconstruction results of BP, CS, LSGM, ADMM, MLDM-I, and MLDM-II.

 

Fig. 11. Visual comparison of resolution test reconstructions. (a) Ground Truth (b) BP (c) CS (d) LSGM (e) ADMM (f) MLDM-I (g) MLDM-II.

Download Full Size | PDF

Reconstruction results of BP were the worst. CS and LSGM have difficulty distinguishing group line -1. For the ADMM and MLDM, the 4th element of the group line -2 can be observed, corresponding to the actual line pair width of 10 mm. Comprehensive analysis, MLDM has a clear reconstruction effect, and even the edge details of the image are well reconstructed. And the reconstruction result is also affected by the sensor and FZA encoding parameters, by reducing the $r1$ of FZA, the image reconstruction quality can be further improved.

6. Conclusion

In this study, a multi-phase FZA lensless imaging method based on score model termed MLDM is proposed. The multi-phase FZA integrated on a single mask plate is used to expand the data channel of a single-shot image. Based on the data distribution prior information extraction strategy of multi-channel coding, the information association between the pixel channel and the coding phase channel of the color image is further obtained, and the iterative reconstruction method effectively improves the reconstruction quality. Compared with the results of traditional experiment, the PSNR and SSIM indices are significantly improved. It shows that the four-phase can effectively eliminate the influence of the twin image and is more conducive to generating high-quality images. In the simulations, the highest PSNR and SSIM of MLDM-I and MLDM-II were 29.42 dB, 0.933, and 29.91 dB, 0.941 respectively. The model exhibits good modeling performance and accurately captures the data distribution, without the need for adversarial training to generate high-quality samples. By utilizing the multi-phase method to collect more encoded information, the extraction of prior information is made more effective, leading to an improvement in the reconstruction quality. The method proposed in this paper is expected to reduce the cost and size of cameras while ensuring imaging quality, and is expected to be applied to biomedical imaging, digital holography and other fields.