1. INTRODUCTION

Correlated-photon imaging, relying on the inherent quantum correlations between entangled photon pairs, has emerged as a novel technique that brings quantum enhancement to many research fields [1–5]. The direct imaging of non-classical correlations can reveal entanglement between position and momentum [6,7] or entanglement among optical angular momentum modes [8]. The new imaging technique enabled by single-photon-sensitive cameras can test fundamental quantum physics [9–16] and improve the conventional imaging systems in spatial resolution and signal-to-noise ratio (SNR) [17–19].

Unfortunately, the imaging system’s performance at low-light conditions is affected by shot noise due to the quantum nature of light. Intensified scientific complementary metal-oxide-semiconductor (I-sCMOS) cameras are able to capture a single photon by virtue of image intensifier technology [20–24]. To extract the single-photon signal from the noise, a reasonable threshold is set to binarize the data in each pixel, a signal over which is defined as a successful registration of one photon [25,26].

Reconstructing such photon-limited images can be converted to solve inverse problems [27,28]. Numerical algorithms [29,30] can solve the problems by treating Poisson statistics as the prior knowledge and performing complex iterative operations, such as least squares, maximum likelihood, and convex optimization. Apparently, thousands of raw images have to be collected to form a proper statistical result, which prevents the reconstruction process from being implemented in real time. Machine learning, also known as an “end-to-end” approach, can merge multiple stages into one neural network [31–36]. It implies finding a direct relationship between the original objects and measured ultra-low-light images by learning from large datasets.

Here, we experimentally report a deep learning of correlated-photon imaging with a convolutional neural network (CNN), whose architecture is inspired by autoencoders and trained for extracting effective signals from various noises. As a result, deep learning algorithms show superior performance over the numerical reconstruction algorithms in image restoration and super-resolution at the single-photon level, especially the ability to achieve high-contrast imaging in real time. Our results suggest emerging deep-learning-enhanced applications in quantum imaging and quantum information processing.

2. THEORY AND EXPERIMENT

In general, the imaging measurement, $y$, is noisy compared with the original image, $x$, due to the imperfect imaging system, such as the laser’s Poisson properties, the optical elements’ limited size, and the camera’s low quantum detection efficiency. However, a statistical model describing the forward imaging system becomes ill-posed when the number of photons decreases. Thus regularization is often introduced into a designed numerical algorithm ${R_{{\rm{reg}}}}$ to search solutions that match the prior knowledge about objects [37],

As shown in Fig. 1(a), numerical algorithms usually consist of the following process: the pre-processing step rescales the measurements to fit the inputs in the modeling step, where the reconstruction process is converted to a convex optimization program from Eq. (1). After sequential quadratic approximations, we get an optimal image corresponding to the original object.

 

Fig. 1. Reconstruction algorithms in an inverse imaging problem. (a) Schematic of the numerical algorithm. This process only optimizes a single-frame image to achieve a globally optimal solution by multiple steps such as least squares, maximum likelihood, and convex optimization. (b) Schematic of the learning algorithm. This process builds a network structure to directly connect the input and output images. Then a large set of training images is fed into the network to learn features of the imaging system by optimizing joint parameters. Once the training step is finished, we can rebuild images in real time.

Download Full Size | PDF

As an effective method for feature extracting and image denoising, deep learning has revolutionized our ability to use computers to perform challenging tasks. It provides another alternative, called the learning algorithm, as shown in Fig. 1(b). Original objects and their corresponding measurements, $\{{({{x_n},{y_n}})} \}_{n = 1}^N$, are fed into a neural network as inputs, the reconstruction algorithm, ${R_{{\rm{learn}}}}$, is trained for optimizing the following process:

Our experimental arrangement is shown in Fig. 2(a). We use a 780 nm mode-locked Ti:sapphire oscillator with a pulse duration of 140 fs and a repetition rate of 80 MHz. The laser is frequency-doubled to 390 nm in a ${\rm{Li}}{{\rm{B}}_3}{{\rm{O}}_5}$ crystal, and then the ultraviolet laser pumps a 2 mm thick $\beta - {\rm{Ba}}{{\rm{B}}_2}{{\rm{O}}_4}$ crystal to create the correlated-photon pairs via a type-II spontaneous parametric down-conversion (SPDC) process. The dichroic mirror allows the light of 390 nm to pass through, while the light of 780 nm is reflected. The single-channel count rate and two-channel coincidence reach about 500,000 and 100,000, respectively. The idler photons are detected by a single-photon avalanche diode (SPAD) to trigger the image intensifier coupled to the sCMOS camera. It is a single-photon-sensitive camera with a 5.5 megapixel sensor and quantum efficiency of about 20% at 780 nm. The signal photons, encoded with image information by a spatial light modulator (SLM), pass through a linear polarization analyzer and then are captured by the I-sCMOS camera. A wave plate and polarization beam splitter are employed to initiate and optimize signal photons by monitoring and adjusting the signal path’s photon level. To ensure the synchronization of correlated-photon pairs, we compensate for the electronic delay by adding a 10 m fiber delay line to the signal path.

 

Fig. 2. Experimental setup and CNN model. (a) Sketch of the experimental setup. Correlated-photon pairs with a wavelength of 780 nm are simultaneously generated from $\beta$-barium borate crystal cut for Type-II phase matching. The signal photons probe an object displayed on a SLM and reflected photons are detected by an I-sCMOS camera, whose intensifier is triggered by a SPAD detector responding to the corrected idler photons. A 10 m fiber delay line is necessary to compensate for the electronic delays between two arms. LBO, ${\rm{Li}}{{\rm{B}}_3}{{\rm{O}}_5}$ crystal; DM, dichroic mirror; BBO, $\beta - {\rm{Ba}}{{\rm{B}}_2}{{\rm{O}}_4}$ crystal; WP, wave plate; PBS, polarization beam splitter; APD, avalanched photon diode; POL, polarizer; FL, filter lens (${{780}}\pm{{10}}\;{\rm{nm}}$). (b) The CNN model. The structure includes two parts: encoder and decoder. The encoder compresses the inputs to a lower-dimensional representation, while the decoder decompresses the representation to reconstruct the inputs as best as possible.

Download Full Size | PDF

 

Fig. 3. Reconstruction results with both numerical and learning algorithms. (a) Original object intensity of six letters representing the word “photon” from the EMNIST dataset. (b) Direct measurements captured by the I-sCMOS camera with 5 s exposure time. (c) TV regularization reconstruction. (d) The CNN algorithm reconstruction. (e) Intensity plots from one line of the “n” letter corresponding to the original intensity.

Download Full Size | PDF

The amplitude-only SLM pixel size is ${{8}}\;{{\unicode{x00B5}{\rm m}}} \times {{8}}\;{{\unicode{x00B5}{\rm m}}}$, and the number of pixels is ${{1920}} \times {{1080}}$. On the one hand, liquid crystal molecules can modulate photons, simulating reflection and other optical phenomena in real-world scenes. On the other hand, the SLM can keep the optical system stable and update the samples in real time, which provides excellent convenience for obtaining a large number of measured images. The entire optical system makes real-time correlated-photon imaging possible.

In low-light situations, such as in the long-range imaging where the number of photons in each laser pulse decay exponentially with propagation distance, or in the fluorescence microscopic imaging where the samples are sensitive to bright light, imaging would face a dilemma, in which only a few numbers of photons can be obtained. The limited photons are vulnerable to noise, which can lead to a significant difference in imaging results. These uncertain noises are primary from the shot noise of the light source itself, shot noise of the dark signal, and readout noise. Therefore, optimizing these inherent noises becomes crucial in correlated-photon imaging. See Supplement 1 for details.

We choose two different sample categories: handwritten images from the Extended MNIST (EMNIST) [38] and MNIST [39] database. For each noise level, a corresponding CNN model is trained. The layout of our specific CNN structure is schematically shown in Fig. 2(b), which is inspired by the “encoder-decoder network” architecture [40,41]. The encoder consists of a stack of convolutional layers and maxpooling layers. Specifically, each convolutional layer has a filter size of ${{3}} \times {{3}}$, stride size of 1, and zero-padding with a size of 1, followed by a max pooling layer with a kernel size of ${{2}} \times {{2}}$ and a stride size of 2. The decoder comprises deconvolutional layers and unpooling layers to perform the opposite operations to the encoder. The activate function of all layers is rectified linear units (ReLU), which allows for fast and effective training on the large and complex databases. See Supplement 1 for details.

The data process consists of two phases: training and test. The EMNIST database images are randomly split into a training set and a test set containing 10,521 and 1169 images, respectively. All images are uploaded to the SLM frame by frame, and the I-sCMOS camera records the results. The measured images constitute the inputs to the network, and the true EMNIST images are the targets. The training procedure updates the weights of the network, and once it is completed, the model performances are evaluated throughout the test set. The CNN optimization process is based on Python version 3.7 performed on a GTX1650 graphics card (NVIDIA).

3. RESULTS

We choose the “photon” letter samples to demonstrate the performance from different reconstruction algorithms, as shown in Figs. 3(a)–3(d). Direct measurements in the camera plane are very noisy compared with the original objects in the low-light condition. For TV regularization, we display the optimal results after minimizing Eq. (1). (See Supplement 1 for details.) When there is only one frame of data, this scheme has little effect on image reconstruction. In contrast, the CNN algorithm is very efficient in suppressing noise. Besides, we plot an intensity map of the “n” image, as shown in Fig. 3(e). The reconstruction results from the CNN model can fit the original images well, which is superior to TV regularization.

The negative films of original images are collected, but sharp edges about the objects can still be well reconstructed by the end-to-end method. After the training process, the CNN model can recreate the original images but is insensitive to the training data because the inputs and outputs are no longer the same. Our CNN model does not simply develop a mapping that memorizes the training data but learns to map the input images toward a lower dimension. If this dimension accurately characterizes the original data, we can effectively filter out the unwanted noise. Therefore, the CNN model learns a general encoding and decoding process independent of samples. Moreover, the negative films are common in medical imaging, especially x ray imaging, where minimizing the radiation is beneficial to the patient. This is a completely different mechanism from numerical algorithms and is also the kernel to accelerate correlated-photon imaging.

To measure the difference between the original and reconstructed images, we compare the root mean square error (RMSE) defined as follows:

Table 1. Comparison of the Root Mean Square Error for Different Algorithms

View Table

 

Fig. 4. Reconstruction results with learning algorithms at 0.8 photons/pixel on average. (a) Original object intensity of four digits “1905” from the MNIST dataset. (b) Direct measurements in the plane of the camera. (c) The CNN algorithm reconstruction verifies strong robustness in noisy environment. (d) The mean square error between the network outputs and original handwritten digits drops down as the epoch increases.

Download Full Size | PDF

Further, to verify the robustness of the learning algorithm in ultra-low-light conditions, fewer correlated photons are illuminated on the samples with ${\sim}0.8$ photons per pixel on average. We prepare another dataset containing 6690 handwritten digits downloaded from the MNIST database, including 6021 images as a training set and 669 images as a test set. After optimizing the CNN model weights, we display the “1905” digits in Figs. 4(a)–4(c). Fewer photons result in indistinguishable raw signal measurements. Interestingly, the reconstructed images still give high contrast. Thus, the CNN algorithm can protect the signals from noise damage and demonstrate strong robustness.

Besides, to optimize the CNN structure, we construct the networks with five layers, seven layers, and nine layers, as shown in Fig. 4(d). After 1000 epochs, the mean square error (MSE) between the network outputs and the original handwritten digits drops down to 0.25 for five layers and becomes steady, which indicates that our network has not overfitted to the training dataset. However, for seven layers and nine layers, the cost difference becomes smaller after 2000 epochs. Using the least layers is necessary to realize the optimal denoising results since it can reduce the computational cost significantly and save huge computing resources. See Supplement 1 for details.

 

Fig. 5. Summary of the state-of-the-art imaging experiments at a single-photon level. To reconstruct a high-contrast image, numerical algorithms require sparse single photons per frame; thus, intensified CCD and CMOS cameras have to accumulate thousands of frames. The emergence of new imaging devices like SPAD cameras makes less necessary photons and high contrast possible, but numerical algorithms are still a barrier to realize fast imaging. Deep learning algorithms effectively solve this problem, achieving a win-win for both imaging speed and quality.

Download Full Size | PDF

We summarize the state-of-the-art single-photon imaging experiments, as shown in Fig. 5. To quantify this comparison, we define the image contrast as

Imaging systems differ in various applications, leading to a trade-off in visibility and the time spent on collecting data. Compared with the passive imaging schemes, active imaging enables higher contrast by high-precision nanosecond time gate filtering out noise from the signals. However, intensified CCD and CMOS architectures suffer from the low frame rate. As a result, the traditional reconstruction algorithms can enhance the image only by collecting thousands of sparse-photon frames, which wastes a lot of time [42–44]. Another active imaging scheme is based on SPAD cameras capable of counting and time stamping single photons with picosecond time resolution. Single-pixel scanning imaging [45] with excellent visibility acquires a megapixel scan in approximately 20 min. SPAD array imaging [46] achieves great improvement, but it still requires hundreds of seconds. It can be seen that the balance between imaging quality and imaging speed is the result of the simultaneous improvement of hardware devices and algorithms. Our deep-learning-based reconstruction algorithm based on the I-sCMOS camera can realize fast imaging at a second-level speed while simultaneously maintaining high visibility.

4. DISCUSSION

In summary, we experimentally demonstrate a fast correlated-photon imaging scheme enhanced by deep learning algorithms to recover objects illuminated with a SPDC source. We show the CNN model is superior to the classical algorithm in denoising and fast reconstruction with ultra-weak illumination. First, the reconstruction process is independent of prior knowledge, such as Poisson distribution or point spread function. Besides, large network parameters trained with many images can overcome the uncertainty in the optimization process, where we can find the best solutions for the non-convex problem. Furthermore, once trained, the CNN model remains nonlinear and highly complex, which keeps a good response to the imaging system. Finally, it can complete high-contrast image restoration by collecting a limited number of faint pictures, which provides the possibility of real-time imaging. In this work, the optimization results are built on a single frame, which breaks the rule that it will take more time to collect more photons to obtain a high contrast image. Although the CNN structure is a standard model for denoising in this experiment, in the long term, we believe the pioneering work boosts a closer connection between deep learning and quantum imaging, which will pave the way for broader and practical exploitation of quantum-enhanced imaging.