1. Introduction

Single-pixel imaging (SPI) [1–37], also referred to as ghost imaging (GI), or correlated imaging, is based on sampling an object using a sequence of structured light patterns generated by a programmable spatial light modulator (SLM), and the light intensity is measured by a single-pixel detector. The detected signals in combination with knowledge of illumination patterns enable a computational reconstruction of an image using different algorithms. The single-pixel detector used in SPI is inexpensive compared with plane array detectors, such as charge coupled device (CCD), especially for spectrum ranges for which array-based sensor are costly or even unavailable. SPI has the ability to image at nonvisible wavelengths and under low light conditions and thus holds great potential for low-cost and compact imaging. To date, SPI has achieved success in special imaging and various applications, such as microscopy imaging [6, 7], fluorescent imaging [8–10] and gas imaging [11]. However, SPI encounters limitations associated with low imaging quality and efficiency. To push SPI towards practical applications, researchers have made considerable efforts to resolve these issues through various methods, such as differential ghost imaging (DGI) [12], normalized ghost imaging (NGI) [13], high-order ghost imaging (HGI) [14], compressed sampling ghost imaging (CGI) [15], etc. Besides, the illumination pattern model has an important influence on the imaging quality and efficiency. Random patterns have been widely employed for SPI from the outset. However, such patterns require a large number of measurements and a long data-acquisition time for recording signals [16]. Recently, deterministic orthogonal basis patterns have been proposed for greatly improving imaging quality and efficiency. Hadamard single-pixel imaging (HSI) [11, 17–20] and Fourier single-pixel imaging (FSI) [21–25] are two such representative SPI techniques. HSI uses Hadamard basis patterns for illumination. Subsequently, HSI acquires the Hadamard spectrum of the object image and reconstructs the object image by applying an inverse Hadamard transform. Whereas, FSI uses grayscale Fourier basis patterns for illumination, and acquires the Fourier spectrum for the object image, which is then reconstructed by applying an inverse Fourier transform. ZIBANG ZHANG et al. [26] compared the performance of HSI and FSI using a theoretical analysis and experiments. The results show that FSI is more efficient than HSI, while HSI is more noise-robust than FSI.

In SPI, the programmable SLM is a crucial device for generating structured light patterns. Generally, three types of SLMs are available: transmissive Liquid Crystal (LC), Liquid Crystalon Silicon (LCoS) and digital micro-mirror device (DMD). Compared with LC and LCoS, DMD shows the advantages of higher reflectivity, higher light throughput and higher frame rates [38]. The latest digital DMD is able to generate ~22,000 binary patterns per-second, which is much faster than LC and LCoS. Therefore, DMD is a popularly used SLM in SPI systems. Because of the binarized nature of the Hadamard basis patterns, HSI is well compatible with DMD; thus, HSI is perfectly suitable for DMD-based SPI systems. However, since DMD generates grayscale patterns by temporal dithering and works much slower in the grayscale mode, DMD-based FSI system is inherently slow and needs a longer data acquisition time compared with HSI. Thus, to improve the imaging efficiency of DMD-based FSI system, if a transformation can be performed in which the grayscale patterns are equivalently transformed to the corresponding binary patterns. As such, a DMD-based FSI system via binary illumination will obtain high imaging efficiency by taking advantage of the higher binary modulation rate of a DMD. Following this type of approach, ZIBANG ZHANG et al. [27] presented a spatial dithering strategy to increase the speed of FSI by two orders of magnitude. In their strategy, the Fourier basis patterns are binarized based on upsampling and error diffusion dithering and they captured 256 × 256 pixels dynamic scenes at a speed of 10 frames per-second under a 20 KHz projection rate of a DMD. The reported technique accelerates the image acquisition speed of FSI at the expense of the imaging spatial resolution.

Here, we present a novel technique to improve the imaging efficiency of FSI based on a detected signal computational-weighted via binary pattern illumination by a DMD. Such a scheme can be termed signal dithering. The proposed technique makes a tradeoff between spatial resolution and temporal resolution without sacrificing spatial resolution compared with the spatial dithering method in [27], and it increases the speed for DMD-based FSI by one order of magnitude compared with temporal dithering method. Technically, the original grayscale Fourier basis illumination patterns will be split into a pair of grayscale patterns that are then decomposed into a cluster of binary illumination patterns in terms of the principle of decimalization to binary [39]. A single-pixel detector records the reflected light intensities, which arise from the interactions between the imaged object and the binary illumination patterns. The recorded intensities will be multiplied by the factors of the weighted decomposed coefficients, and then summed. The Fourier spectrum for the imaged object can be constructed from the sum of the intensities. Finally, an inverse Fourier transform is applied to the Fourier spectrum to recover the object image. The lay-out of the present paper is organized as follows. In Section 2, we introduce the imaging principles and deduction methods. In Section 3, the quantity simulations and experiments are carried out to evaluate our proposed methods. In Section 4, the conclusions of this work are summarized.

2. Principles and methods

The proposed computational-weighted FSI technique is based on the theorem of the Fourier transform. The discrete 2-D Fourier transform indicates that an M × N digital image $f(x,y)$ in the 2-D spatial domain can be expressed as a weighted sum of M × N 2-D sinusoid structured light patterns by the corresponding Fourier coefficients, which can be denoted as follows:

where (x,y) represents the 2-D Cartesian coordinate, (u, v) is the spatial frequency, and $F(u,v)$ is the Fourier coefficient. According to the above equation, the object image can be reconstructed if the Fourier coefficient of each corresponding frequency is known. The Fourier coefficient $F(u,v)$ can be obtained by the corresponding transform as follows:

Thus, when the sinusoid-structured light illumination patterns (called as Fourier basis patterns) are utilized in an SPI system, a single-pixel detector acquires the corresponding light intensities, which will be constructed the Fourier coefficients by a combined operation. The mathematical formula for the detected light intensity $I_{\varphi }(u,v)$ can be denoted as follows:

where the parameter $P_{\varphi }(x,y;u,v)$ is the illumination pattern and can be expressed as follows:

Here, the parameter R represents the quantification level. ϕ is the initial phase for the illumination patterns. When the phase is ϕ = 0, the intensities for the grayscale Fourier basis illumination patterns $P_{\varphi \text{=}0}\text{(}x,y;u,v\text{)}$ interacting with the imaged object correspond to the real components of Fourier spectrum of the imaged object; however, when the phase is ϕ = π/2, the intensities for the grayscale Fourier basis illumination patterns $P_{\varphi \text{=}\pi \text{/2}}\text{(}x,y;u,v\text{)}$ interacting with the imaged object correspond to the imaginary components of the Fourier spectrum of the imaged object.

State-of-the-art DMD can generate ~22000 binary patterns per-second. However, when the DMD is used to modulate 8 grayscale patterns (such as Fourier basis patterns), the modulation rate is decreased to ~250Hz.The reason for this decrease is that the grayscale patterns are decomposed to binary patterns, which are displayed according to the weighted time. For example, a grayscale pattern with 256 grayscale levels (8-bit image) will be decomposed into 8 binary patterns, as shown in Fig. 1, and the i-th binary patterns will illuminate according to the weighted time 2^(i-1)*T. (T is the basis time for one binary illumination pattern). The illumination process, which is termed temporal dithering [27], can be expressed as follows:

 

Fig. 1 Traditional method used for generation grayscale Fourier basis illumination patterns by temporal dithering.B0, B1 ... B7 are successive binary illumination patterns that are decomposed from the grayscale Fourier pattern. T is the basis illumination time.

Download Full Size | PDF

Where $B_{\varphi ,i}(x,y;u_m,v_n)$are binary patterns that are decomposed from the corresponding grayscale Fourier basis pattern. The use of the temporal dithering method to create grayscale patterns requires considerable time to illuminate the imaged target, which results in low imaging efficiency for the DMD-based FSI. In [27], ZIBANG ZHANG et al. proposed a spatial dithering strategy to transfer grayscale illumination to binary illumination and improve the imaging efficiency for DMD-based FSI; however, this strategy reduces the imaging spatial resolution.

When grayscale Fourier $P_{\varphi }\text{(}x,y;u_m,v_n\text{)}$ pattern with a given spatial frequency (u_m, v_n) illuminates the imaged object $f(x,y)$, the resulting reflected light $I_{\varphi }(u_m,v_n)$ can be collected by a single-pixel detector. This process can be expressed as follows:

When Eq. (5) is incorporated into Eq. (6), we can obtain the following equation:

where the parameters $I_{\varphi ,1}(u_m,v_n)$…$I_{\varphi ,R}(u_m,v_n)$ are the detected intensities corresponding to the binary illumination patterns $B_{\varphi ,1}(x,y;u_m,v_n)$…$B_{\varphi ,R}(x,y;u_m,v_n)$ interacting with the imaged object under the same exposure/illumination time T, respectively. Equation (7) shows that the weighted sum of the detected intensities for the binary illumination patterns interacting with the imaged object can be equivalent to the intensities for the grayscale Fourier basis illumination patterns interacting with the imaged object. In other words, FSI can be indirectly realized by using binary illumination patterns with DMD. Due to the high binary modulation rate of DMD, this process will greatly improve the imaging efficiency for FSI, which represents the core idea of this paper.

Due to the negative grayscale value for the original Fourier pattern illumination $P_{\varphi }\text{(}x,y;u,v\text{)}$ cannot be projected directly by DMD; thus we first split the original Fourier pattern illumination $P_{\varphi }\text{(}x,y;u,v\text{)}$ into a pair of grayscale patterns $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ and $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$, which fulfills the following relation:

For example, Fig. 2 shows a schematic demonstration of a 3 × 3 pixels grayscale pattern splitting to a pair of grayscale patterns with R = 6.

 

Fig. 2 Splitting of original grayscale Fourier patterns into a pair of grayscale patterns.

Download Full Size | PDF

The particular splitting strategies are described as follows:

For the pattern $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$, if the value of the spatial coordinate (x, y) for the grayscale Fourier pattern $P_{\varphi }\text{(}x,y;u,v\text{)}$ is positive, then the value of the spatial coordinate (x, y) for the grayscale pattern $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ is equivalent to the value of the spatial coordinate (x, y) for$P_{\varphi }\text{(}x,y;u,v\text{)}$; otherwise, the value of the spatial coordinate (x, y) for the grayscale pattern $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ is zero.

For the pattern $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$, if the value of the spatial coordinate (x, y) for the grayscale Fourier pattern $P_{\varphi }\text{(}x,y;u,v\text{)}$ is negative, then the value of the spatial coordinate (x, y) for the grayscale pattern $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$ is the absolute value of the corresponding spatial coordinate of $P_{\varphi }\text{(}x,y;u,v\text{)}$; otherwise, the value of the spatial coordinate (x, y) for the grayscale pattern $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$ is zero.

The split patterns $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ and $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$ are still grayscale. Thus, the grayscale patterns should be further decomposed to binary patterns. These decomposing processes follow the principle of decimalization to binary. Figure 3 shows a schematic representation of a certain grayscale pattern $P_{\varphi }^{\text{+}}$(shown in Fig. 2) decomposing to corresponding binary illumination patterns. In Fig. 3, the number of decomposed binary patterns is 6. Similarly, the processes of decomposing $P_{\varphi }^{\text{-}}$ into binary patterns are equivalent to that for $P_{\varphi }^{\text{+}}$in Fig. 3.

 

Fig. 3 Decomposition of the grayscale pattern into binary illumination patterns.

Download Full Size | PDF

Figure 4 shows a demonstration of the entire decomposing processes. As shown in Fig. 4, $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ and $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$ are a pair of grayscale patterns that are split from the original Fourier pattern $P_{\varphi }\text{(}x,y;u,v\text{)}$; and $B_{\varphi }^{\text{+}}{}_{,i}(x,y;u,v)$ and $B_{\varphi }^{\text{-}}{}_{,i}(x,y;u,v)$ are R binary patterns, which are decomposed from $P_{\varphi }^{\text{+}}\text{(}x,y;u,v\text{)}$ and $P_{\varphi }^{\text{-}}\text{(}x,y;u,v\text{)}$, respectively. When the binary patterns $B_{\varphi }^{\text{+}}{}_{,i}(x,y;u,v)$are used to successively illuminate the imaged objects, the corresponding light intensities are detected by a single-pixel detector, which are then summed by the weighted coefficients of 2 orders. The sum can be termed $I_{\varphi }^{\text{+}}(u_m,v_n)$ and expressed as follows:

 

Fig. 4 Entire process for decomposing the grayscale Fourier patterns to binary patterns. OGP: Original grayscale pattern; DGP: Decomposed grayscale patterns; and BIP: Binary illumination patterns.

Download Full Size | PDF

The parameters $I_{\varphi }^{\text{+}}{}_{,1}(u_m,v_n)$,$I_{\varphi }^{\text{+}}{}_{,2}(u_m,v_n)$…$I_{\varphi }^{\text{+}}{}_{,R}(u_m,v_n)$ are reflected light intensities corresponding to the binary illumination patterns $B_{\varphi }^{\text{+}}{}_{,1}(x,y;u_m,v_n)$, $B_{\varphi }^{\text{+}}{}_{,2}(x,y;u_m,v_n)$… $B_{\varphi }^{\text{+}}{}_{,R}(x,y;u_m,v_n)$ interacting with the imaged object, respectively. Using the same operations, we can obtain the intensities for $I_{\varphi }^{\text{-}}(u_m,v_n)$:

Similarly, the parameters $I_{\varphi }^{\text{-}}{}_{,1}(u_m,v_n)$, $I_{\varphi }^{\text{-}}{}_{,2}(u_m,v_n)$… $I_{\varphi }^{\text{-}}{}_{,R}(u_m,v_n)$ are the corresponding reflected light intensities for the binary illumination patterns $B_{\varphi }^{\text{-}}{}_{,1}(x,y;u_m,v_n)$, $B_{\varphi }^{\text{-}}{}_{,2}(x,y;u_m,v_n)$… $B_{\varphi }^{\text{-}}{}_{,R}(x,y;u_m,v_n)$ interacting with the imaged object, respectively. For the phase is ϕ = 0, the real component of the Fourier coefficients for the imaged object under a given spatial frequency (u_m, v_n) can be expressed as:

for the phase is ϕ = π/2, using a similar operation, the imaginary component of the Fourier coefficients under a certain spatial frequency (u_m, v_n) can be obtained and expressed as:

Due to the differential operation, our method can also restrain background light. When all the detected light intensities under different spatial frequencies are recorded, the entire set of Fourier spectrum can be received. Finally, by applying of an inverse Fourier transform to the complex-valued Fourier spectrum, the image of the object can be reconstructed as follows:

where F⁻¹ denotes the inverse Fourier transform operator. With the prior knowledge that the Fourier spectrum of any real-valued image is conjugated symmetrically, fully sampling an M × N-pixel image using our method takes 2 × M × N × R measurements and 2 × M × N × R binary illumination patterns.

The recorded signal will be multiplied by the weighted decomposed coefficients in our method and then used to construct the Fourier spectrum of the imaged object; therefore, compared with the temporal dithering and spatial dithering methods, our proposed method can be termed signal dithering. If the binary modulation rate of the DMD is 20 KHz, the modulation rate in our proposed method is increased to 2500 Hz to realize the equivalent function of an 8-bit grayscale pattern, which is 10 times higher than that for temporal dithering. A comparison among our proposed signal dithering method and the two other methods is shown in Table. 1. The results show that our strategy significantly reduces the illumination time compared with temporal dithering method and does not sacrifice imaging spatial resolution compared with spatial dithering method.

Table 1. Comparison among three methods

View Table

3. Experiments

3.1 Computational simulations

First, we use four different images (Mandril, Baboon, Cameraman and Peppers) to evaluate our proposed strategy with varying quantization R values in the range from 2 to 8 for the computational simulation. Thus, the pixel values for the original grayscale Fourier basis patterns range from [-3 + 3] to [-255 + 255]. By using different quantization R values, the original grayscale Fourier basis patterns are decomposed to different quantities of binary illumination patterns, which will result in a diversified reconstruction error for the imaging system. For a certain spatial frequency (u_m, v_n) to the original grayscale Fourier basis patterns, the patterns will be decomposed into 4 binary illumination patterns at R = 2 or 16 binary illumination patterns at R = 8. The simulation results are shown in Fig. 5. The quality of the reconstruction images improves with an increase in the R value. The percentage of root mean squared error (RMSE) is used to further quantitatively evaluate the quality of the reconstruction for the different R values, which is calculated by Eq. (14):

 

Fig. 5 Reconstructed Fourier spectrum and images for four different objects with a quantization parameter R ranging from 2 to 8. The resolution of the reconstructed images is 128 × 128 pixels. FS: Fourier spectrum (the absolute value of the Fourier spectrum in a logarithm scale), RI: Reconstructed image.

Download Full Size | PDF

 

Fig. 6 RMSEs and illumination time under different quantization levels.

Download Full Size | PDF

3.2 Laboratory experiments

The experimental setup is shown in Fig. 7. The setup consists of an illumination system, detection system and imaged object. The illumination system consists of a Texas Instruments DLP Discovery 4100 development kit, lens system, and white LED. The DLP development kit is equipped with a 0.7-inch DMD, which contains 1024 × 768 micro mirrors. The maximum binary modulation rate of the DMD is up to 22.7 KHz. And the size of each mirror is 13.6 × 13.6 μm². The detection system consists of a photo-diode (Thorlabs PMT-PMM02, active area of 25 mm²) that is employed as a single-pixel detector, collecting lens, data acquisition board (NI DAQ USB-6211), and computer.

 

Fig. 7 Experimental setup.

Download Full Size | PDF

We present two experiments to verify our proposed strategy. The first is a static imaging experiment. The imaged objects are composed of a fabrictoy and an enlarged 1951 USAF resolution test pattern printed on a piece of white paper. In the beginning the original Fourier basis patterns with a spatial resolution 128 × 128 are generated by MATLAB program. Then, the Fourier basis patterns are finally decomposed into a cluster of binary illumination patterns (128 × 128-pixel), which are loaded into the DMD to modulate the white LED light. According to the analysis of the above simulations, illumination time and measurement time linearly increases with an increment of quantization level. Thus, if the modulation rate of the used DMD is constant, the increment of quantization level will lead to reduce the imaging speed/efficiency. Here, a quantization level of R = 6 is used as a tradeoff between imaging efficiency and reconstruction quality. For a certain spatial frequency (u_m, v_n), the total binary illumination patterns are equal to 12. Since the Fourier spectrum of a real-valued signal is conjugated symmetrically, half of the coefficients are redundant. Thus, in this case, the number of measurements for the Fourier spectrum with coverage of 100% is 2 × 128 × 128 × 6 = 196608. With the strategy that the Fourier spectrum is acquired in the order from lower to higher frequencies, the quality of the reconstructed images improves with rapid convergence. Figure 8 shows the sampled Fourier spectrum and the corresponding reconstructed images with the spectrum coverage from 5% to 100%. As shown by the figures, the proposed technique can be used to realize clear images. Note that we show the absolute value of the Fourier spectrum on a logarithm scale to increase their visibility. The images are reconstructed by using the 2-D discrete inverse digital Fourier transform (IDFT) algorithm. No postprocessing was applied to the reconstructed images.

 

Fig. 8 Static imaging with spectrum coverage from 5% to 100% and the corresponding reconstructed images. (A–D): Density distributions for the Fourier spectrum acquired at 5%, 15%, 25%, and 100%, the corresponding reconstructed images, respectively. Scale bar: 1cm.

Download Full Size | PDF

The second experiment is a dynamic imaging experiment in which imaged object is a moving hand. For the experimental imaging of dynamic scenes, we set the quantization parameter R = 2 to improve the imaging speed. For a given spatial frequency (u_m, v_n), the total binary illumination patterns are equal to 4. The resolution of the image is 128 × 128. In general, most information in a natural scene is concentrated at low-frequency components. Thus, we acquire only 313 coefficients in the low-frequency range of the Fourier domain to recover the images. Each image takes 2504 measurements, which enables the capture of ~9 images per-second at the 22 KHz modulation rate of the DMD. The sampling ratio is ~2%( = 313/128*128). The results for the dynamic imaging are shown in Fig. 9. We capture 35 images and reconstruct a ~4 seconds’ video (see Visualization 1). The video is created from the sequence of reconstructed images, with the playback frame rate set to 9. No postprocessing was applied to the reconstructed images and the video. When the low-frequency components and relatively low quantification level are used, our method can also be used to image dynamic scenes.

 

Fig. 9 Dynamic imaging results. (a-h): eight of 35 reconstructed images. (See Visualization 1 for the complete video).

Download Full Size | PDF

4. Discussion and conclusions

We present a novel technique to improve the imaging efficiency of the DMD-based FSI by using computational-weighted method via binary pattern illumination. Technically, the grayscale Fourier basis illumination patterns are replaced by a cluster of binary illumination patterns to project the imaged objects, which takes advantage of the high binary modulation rate of the DMD. The light intensities for the binary illumination patterns interacting with the imaged object are multiplied by weighted decomposed coefficients and then summed, which can be equal to the light intensities of the grayscale Fourier basis illumination patterns interacting with the imaged object. This strategy overcomes the low grayscale patterns modulation rate for the DMD-based FSI system. The simulation and experimental results validated the present approach. Objectively, our method presents a tradeoff between the imaging spatial resolution and temporal resolution. For DMD-based FSI, the proposed method does not limit the imaging spatial resolution compared with spatial dithering method and overcomes the limitation of low illumination efficiency compared with the temporal dithering method. Of course, the use of the three-step phase-shifting algorithm [26, 27] can result in an advantage in which of 25% fewer measurements are required, thereby further reducing the imaging time. The proposed scheme using computational-weighted DMD-based FSI provides an optional imaging solution for 3-D imaging, living microscopy and many other applications. In short, the reported technique accelerates the imaging speed of DMD-based FSI and provides an alternative approach to improve FSI efficiency.