Single-pixel imaging using discrete Zernike moments

Wenchang Lai; Wenchang Lai; Guozhong Lei; Guozhong Lei; Qi Meng; Qi Meng; Dongfeng Shi; Wenda Cui; Wenda Cui; Pengfei Ma; Pengfei Ma; Yan Wang; Yan Wang; Yan Wang; Kai Han; Kai Han; Kai Han; Kai Han

doi:10.1364/OE.473912

1. Introduction

Single-pixel imaging (SPI) is a novel imaging technique that is substantially different from traditional array imaging technique. The array imaging scheme usually employs a pixelated detector such as CCD or CMOS based on silicon materials, which works at the visible waveband. By contrast, single-pixel imaging technique utilizes a single-pixel detector such as photodiode or photon multiplier tube that is cheaper and has broad working spectral range. Therefore, single-pixel imaging technique can be applied especially in non-visible imaging system such as infrared [1], X-ray [2,3], or terahertz [4,5]. Additionally, compared with array pixelated detectors, single-pixel detectors can be fabricated with higher quantum efficiency and higher light sensitivity by using low-cost materials. Given such advantages, single-pixel imaging can also have substantial potential in weak-light imaging [6] and remote sensing imaging [7]. Due to the wide applications, the single-pixel imaging scheme has attracted increasing attention in recent years [8–11]. Since the spatial light modulation technology develops rapidly, the contemporary single-pixel imaging often uses modulated light illumination methods [10,11]. Here a sequence of time-varying structured patterns is applied to modulate the illuminating light via a spatial light modulator (SLM). Then the time-varying reflected light intensities from the object can be collected by the single-pixel detector. By combining the known illumination patterns and the measured light intensities, the object image can be reconstructed by computational algorithms.

In the single-pixel imaging based on the spatial light modulation, the modulated illumination patterns are important for the reconstructed image quality and the imaging efficiency. Initially, the single-pixel imaging was inspired by ghost imaging (GI), where random patterns were employed for illumination in most circumstances [12]. However, random patterns illumination requires a large number measurements and it takes long data-acquisition time. In addition, the quality of reconstructed images cannot be comparable with that obtained by the traditional array imaging methods [8,9]. Then the compressive sensing (CS) theory was introduced to SPI for reducing the total measurements [13,14]. It can break the limit of the Nyquist-Shannon sampling theorem and reconstruct the object image in under-sampling circumstance. However, the CS-SPI scheme requires more computational time due to the convex optimization algorithm in reconstruction process [14]. In addition to the random patterns, some deterministic orthogonal basis patterns such as Hadamard or Fourier patterns are also applied widely in SPI [15–21]. Hadamard SPI (H-SPI) uses Hadamard basis patterns for illumination to get the Hadamard spectrum of object image. And the object image can be reconstructed by applying inverse Hadamard transform [15–17]. Similarly, the Fourier SPI (F-SPI) method uses Fourier basis patterns and reconstructs images through inverse Fourier transform [17–19]. The application of these orthogonal basis patterns can reduce the reconstructing time and obtain perfect reconstructed image in principle [17]. Besides, the above patterns can be combined with some mathematical transforms such as Radon transform or Morlet wavelet to generate special illuminating patterns [22–24]. The designed illuminating patterns can usually achieve high efficiency or high-resolution imaging.

As we know, orthogonal image moments have been used as characteristic descriptors in a variety of applications in image analysis. There are a lot of orthogonal image moments with different characteristics including Zernike [25], Tchebichef [26], Krawtchouk [27] moments and so on. Among of them, Zernike moments have been usually used for image reconstruction and object recognition [25,28]. The Zernike polynomials are orthogonal to each other on the unit disk and the Zernike moments can keep their amplitude under arbitrary rotation. Therefore, the Zernike moments have special rotation invariants in image analysis, which could be used for specific application such as object classification. In fact, the Zernike moments can be measured by using a single-pixel detector through specific Zernike patterns illumination in SPI system. The Zernike patterns can be applied in SPI system for object reconstruction and classification.

In SPI system, the object classification is a critical issue and has wide applications in microcosmic or remote sensing. Conventionally, the object classification can be performed after obtaining the object image with high quality. However, it needs a large number of illuminations to reconstruct object image, which costs long imaging time. The deep learning techniques are introduced to reduce the illumination patterns and achieve fast classification [29–31], which needs to train the neural network in advance. Jiao proposes a fast object classification method with only 13 illuminations and 80% accuracy based on Fourier SPI [32]. There are only simulation results without experimental demonstration in this work. In our work, the Zernike illuminating patterns are demonstrated for classifying object directly without image reconstructing process. Only 4 Zernike patterns are needed for obtaining one specific Zernike moment of the object. This method can classify the rotated object images fast with high accuracy due to the rotation invariant of Zernike moments.

In this paper, the Zernike moments is firstly introduced to single-pixel imaging method. Here, the orthogonal illuminating patterns are calculated from Zernike polynomials. The Zernike moments can be acquired by measuring the reflected light intensities through a single-pixel detector. The object images can be reconstructed by combining the known Zernike patterns and the measured light intensities. The simulated and experimental results show that the proposed imaging scheme can obtain high image quality under compressive sampling with a few low-order patterns. The computational time is short due to the simple reconstructing integral functions. Besides, by using a few specific Zernike patterns for illumination, we measure the amplitudes of the Zernike moments through SPI system. Due to the rotation invariant of Zernike moments, the amplitude values are maintained well when the image is rotated. As a result, the images with different rotating angles and the same content are classified into one category accurately.

2. Principles and methods

2.1 Zernike moments

Since the Zernike polynomials are orthogonal on a unit disk, the Zernike moment are usually defined on polar coordinates. The Zernike moment of the n-th order with repetition m for a continuous image function f (r, θ) is defined as [25]

(1)$${\textrm{A}_{nm}}\textrm{ = }\frac{{n + 1}}{\pi }\int_0^{2\pi } {\int_0^1 {Z_{nm}^ \ast (r,\theta )} } f(r,\theta )r dr d\theta $$

where n = 0, 1, 2, …, N, N > 0, m is an integer between -n and n. The difference n-|m| is always even [25]. Z_nm (r, θ) is the corresponding Zernike polynomial. The asterisk means the complex conjugate. For a digital image f (x, y) on Cartesian coordinates, the integrals can be replaced by summation to get

(2)$${\textrm{A}_{nm}}\textrm{ = }\frac{{n + 1}}{\pi }\sum\limits_{x = 0}^{M - 1} {\sum\limits_{y = 0}^{N - 1} {Z_{nm}^\ast } } (x,y)f(x,y) , {x^2} + {y^2} \le 1$$

In Eq. (1) and Eq. (2), the Zernike polynomials are defined as products

(3)$${Z_{nm}}(x,y) = {Z_{nm}}(r,\theta ) = {R_{nm}}(r)\exp (jm\theta )$$

where R_nm is called the radial part which is defined as

(4)$${R_{nm}}(r) = \sum\limits_{s = 0}^{n - |m|/2} {\frac{{{{( - 1)}^s}(n - s)! {r^{n - 2s}}}}{{s! (\frac{{n + |m|}}{2} - s)! (\frac{{n - |m|}}{2} - s)!}}} $$

Note that R_n,-m = R_nm. The Zernike polynomials satisfy the following orthogonality

(5)$$\int\!\!\!\int_{r \le 1} {{{[{Z_{nm}}(r,\theta )]}^\ast }{Z_{pq}}(r,\theta ) } r dr d\theta = \frac{\pi }{{n + 1}}{\delta _{np}}{\delta _{mq}}$$

with

{\delta _{ab}} = \left\{ {\begin{array}{cc} 1 &a = b\\ 0 &a \ne b \end{array}} \right.

According to Eq. (2) and Eq. (5), the reconstructed discrete image function f_re (x, y) can be calculated by

(6)$${f_{re}} (x,y) = \sum\limits_{n = 0}^{{n_{\max }}} {\sum\limits_m {{A_{nm}}{Z_{nm}}(x,y)} }$$

Where n_max is the given max order of Zernike moments. When n_max approaches infinity, the f_re (x, y) will approach f (x, y). Combining Eq. (3) and Eq. (6) and noting that Z*_nm (r, θ) = Z_{n, -m} (r, θ), the reconstructed discrete image function could be expanded as

(7)$${f_{re}} (x,y) = \sum\limits_n {\sum\limits_{m > 0} {({C_{nm}}\cos (m\theta ) + {S_{nm}} \sin (m\theta )){R_{nm}}(r) + \frac{{{C_{n0}}}}{2}{R_{n0}}(r)} }$$

with

{C_{nm}} = 2{\textrm{Re}} ({A_{nm}}) = \frac{{2(n + 1)}}{\pi }\int\!\!\!\int_{{x^2} + {y^2} \le 1} {f(x,y) \cdot {R_{nm}}(r) \cos (m\theta ) dx dy}

{S_{nm}} ={-} 2{\mathop{\rm Im}\nolimits} ({A_{nm}}) = \frac{{ - 2(n + 1)}}{\pi }\int\!\!\!\int_{{x^2} + {y^2} \le 1} {f(x,y) \cdot {R_{nm}}(r) \sin (m\theta ) dx dy}

From Eq. (6) and Eq. (7), we can see that the object image can be reconstructed from the Zernike moments and Zernike polynomials. In the reconstruction process, the reconstruction error would be reduced as the number (or the order) of Zernike moments increases. However, higher-order discrete Zernike moments calculated from Eq. (4) lose their precision due to the factorial functions, which will impair the reconstructing accuracy. Therefore, some optimized recursive algorithms have been proposed to compute Zernike polynomials. Here the q-recursive algorithm is selected for computing Zernike polynomials due to the short time and high precision [28].

In addition to the orthogonality, the rotation invariant of Zernike moments also deserve attention. If we rotate the image by an angle α, the rotated image in the same polar coordinates is

(8)$${f^{\prime}}(r,\theta ) = f(r,\theta - \alpha )$$

And the Zernike moments of the rotated image in the same coordinates is

(9)$$A_{nm}^{\prime} = \frac{{n + 1}}{\pi }\int_0^{2\pi } {\int_0^1 {f(r,\theta - \alpha ){R_{nm}}(r)\exp ( - jm\theta )r dr d\theta } }$$

By changing the variable to θ’=θ – α, Eq. (9) can be calculated as

(10)$$\begin{aligned} A_{nm}^{\prime} &= \frac{{n + 1}}{\pi }\int_0^{2\pi } {\int_0^1 {f(r,{\theta ^{\prime}}){R_{nm}}(r)\exp ( - jm({\theta ^{\prime}} + \alpha ))r dr d{\theta ^{\prime}}} } \\ &= [\frac{{n + 1}}{\pi }\int_0^{2\pi } {\int_0^1 {f(r,{\theta ^{\prime}}){R_{nm}}(r)\exp ( - jm{\theta ^{\prime}})r dr d{\theta ^{\prime}}} } ]\exp ( - jm\alpha )\\ &= {A_{nm}} \exp ( - jm\alpha ) \end{aligned}$$

Equation (10) indicates that each Zernike moment merely has a phase shift on rotation and the magnitude remains unchanged. Therefore, the magnitude of Zernike moments |A_nm| can be acted as the rotation invariant feature of the object image, which can be applied in object recognition.

2.2 Proposed single-pixel imaging method from Zernike moments

According to the theorem of Zernike moments above, the single-pixel imaging process can be divided into four steps as follows.

1. Generating Zernike basis patterns. It can be found in Eq. (3) that the Zernike polynomials are complex and cannot be achieved directly through a spatial light modulator. Therefore, the corresponding real part and imaginary part should be extracted to generate the Zernike basis patterns.
2. Achieving differential modulation. Even though each Zernike basis pattern satisfies the real part or imaginary part of Zernike polynomials, there are also positive and negative values in the pixels of each pattern. The positive and negative values of the pixels in each pattern can be separated to achieve differential modulation for background noise suppression.
3. Loading the Zernike basis patterns onto the spatial light modulator and illuminating the object, then measuring the reflected light intensity by a single-pixel detector.
4. Reconstructing the object image according to Eq. (7).

Now the single-pixel imaging will be explained in detail. In order to generate Zernike basis patterns in reality, according to the reconstructing process in Eq. (7), the mathematical form of Zernike basis patterns can be extracted from Zernike polynomials as below

(11)$${P_{nm}}(x,y) = {P_{nm}}(r,\theta ) = \left\{ {\begin{array}{cc} {R_{nm}}(r) \cos (m\theta ) &m \ge 0\\ - {R_{nm}}(r) \sin (m\theta ) &m < 0 \end{array}} \right.$$

Then Eq. (7) can be expressed as

(12)$${f_{re}} (x,y) = \sum\limits_n {[\frac{{2(n + 1)}}{\pi }(\sum\limits_{m > 0} {{I_{nm}}{P_{nm}}(r,\theta )} - } \sum\limits_{m < 0} {{I_{nm}}{P_{nm}}(r,\theta )} + \frac{{{I_{n0}}}}{2}{P_{n0}}(r,\theta ))]$$

with

{I_{nm}} = \int\!\!\!\int_{{x^2} + {y^2} \le 1} {f(x,y) \cdot {P_{nm}}(x,y) dx dy}

Since f (x, y) is the image function and P_nm (x, y) is the illuminating pattern, I_nm is the reflected intensity from image detected by the single-pixel detector. Equation (11) indicates that each Zernike basis pattern can have both positive and negative values in pixels located in a unit circle, which may not be achieved on experiment. To solve this problem, each pattern P_nm (x, y) could be split into two complementary patterns P ⁺_nm (x, y) and P⁻_nm (x, y), which satisfy the following expression [21,22]

(13)$${P_{nm}}(x,y) = P_{nm}^ + (x,y) - P_{nm}^ - (x,y)$$

In Eq. (13), under the specific pattern’s order (n_i, m_j), the pattern P ⁺_nm (x, y) retains the positive values of pattern P_nm (x, y) in the corresponding spatial coordinates and other pixels values are set to be zero. Similarly, the pattern P⁻_nm (x, y) retains the absolute values of the negative values of pattern P_nm (x, y) in the corresponding spatial coordinates and other pixels values are set to be zero. Therefore, the patterns P ⁺_nm (x, y) and P⁻_nm (x, y) can satisfy Eq. (13) and show a differential modulation method that the background noise could be suppressed well [21,22]. Figure 1 shows the gray images of Zernike basis patterns up to the third degree on the unit disk located in a rectangle with 128${\times} $128 pixels. The black and white pixels mean values of zero and one, respectively. It can be concluded that the number of m under the specific nth order is (n + 1). Then the number of Zernike basis patterns up to the nth order is calculated to be (n + 1) ${\times} $(n + 2).

Fig. 1. The 128${\times} $128 Zernike patterns up to the third degree generated from Eq. (11) and Eq. (13). Black = 0 and white = 1. 1^st row: n = 0, m = 0, 2^nd row: n = 1, m = −1, 1, 3^rd row: n = 2, m = −2, 0, 2, 4^th row: n = 3, m = −3, −1, 1, 3.

Download Full Size | PDF

After the differential operation, the reconstructed image can be expressed as

(14)$$\begin{aligned} {f_{re}} (x,y) = &\sum\limits_n {[\frac{{2(n + 1)}}{\pi }(\sum\limits_{m > 0} {(I_{nm}^ +{-} I_{nm}^ - ){P_{nm}}(r,\theta )} - } \sum\limits_{m < 0} {(I_{nm}^ +{-} I_{nm}^ - ){P_{nm}}(r,\theta )} \\ &+ \frac{{(I_{n0}^ +{-} I_{n0}^ - )}}{2}{P_{n0}}(r,\theta ))] \end{aligned}$$

with

I_{nm}^ +{=} \int\!\!\!\int_{{x^2} + {y^2} \le 1} {f(x,y) \cdot P_{nm}^ + (x,y) dx dy}

I_{nm}^ -{=} \int\!\!\!\int_{{x^2} + {y^2} \le 1} {f(x,y) \cdot P_{nm}^ - (x,y) dx dy}

where I ⁺_nm and I⁻_nm are the reflected intensity values which can be detected by a single-pixel detector. And P_nm (r, θ) is the Zernike basis pattern which can be calculated from Eq. (11).

3. Computational simulations

3.1 Single-pixel imaging by Zernike moments

When the Zernike patterns are used as the illuminating patterns, the illuminating sequence is important. As we know, the low-order Zernike polynomials include more information of image function [25]. Hence, the illuminating sequence is in order of increasing Zernike order (n). In the same order n, the sequence of m is going from -n to n. Based on the SPI method mentioned above, we reconstruct the typical Cameraman image at different sampling ratios (as show in Fig. 2). The sampling ratio increases from 3% to 30%. As we can see, the reconstructed images become clearer when the sampling ratio increases from 3% to 10%. When the sampling ratio exceeds 10%, the edges of the reconstructed image become blurred. The blurred area increases with the further increase of sampling ratio. It is because that the higher order Zernike polynomials will lose the precision on discrete coordinates especially on edge area of the unit circle. Then the numerical errors of reconstructed images will become larger as the sampling ratio increases.

Fig. 2. The reconstructed results of Z-SPI technique at different sampling ratios.

Download Full Size | PDF

In order to analyze the reconstructed images quantitatively, the root mean squared error (RMSE) is used as the criterion for evaluating the quality of reconstructed image. The expression of RMSE can be described as

(15)$$RMSE = \sqrt {\frac{{\sum\limits_{x = 1}^M {\sum\limits_{y = 1}^N {{{[{I_R}(x,y) - I(x,y)]}^2}} } }}{{M \times N}}} $$

where I_R (x, y) and I (x, y) are the values of specific spatial coordinate (x, y) in reconstructed and original images, respectively. M and N are the pixel numbers of image along x and y dimensions, respectively. As we can see, the smaller RMSE value means the better reconstructed quality. Figure 3(a) shows the RMSEs of the reconstructed images at different sampling ratios. When the sampling ratio increases from 3% to 10%, the RMSEs decrease. When the sampling ratio further increases after 10%, the RMSEs increase rapidly. Therefore, it exists an optimal sampling ratio value on Z-SPI where the reconstructed image has best quality. Figure 3(b) shows the calculation time of the reconstructing process at different sampling ratios. The reconstructing process based on Eq. (14) is calculated on a computer with a i5 Intel Core and Windows 10 system. It is obvious that the calculation time increases with the sampling ratio linearly. It illustrates that the reconstructing process which mainly include the inverse transform of Zernike moments is a linear process. The calculation time is lower than 0.16 seconds even at sampling ratio of 0.3. In Ref. [20], the running time of F-SPI is about 0.12 seconds while reconstructing a 64 × 64 image at a sampling ratio of 0.3. Therefore, the reconstructing time of the Z-SPI and the F-SPI is almost in the same level that can be regarded as a fast process.

Fig. 3. (a) The RMSEs of reconstructed images at different sampling ratios, (b) the calculation time of Z-SPI at different sampling ratios.

Download Full Size | PDF

As we know, both the Zernike-SPI (Z-SPI) method and Fourier-SPI (F-SPI) adopt gray-scale orthogonal pattens. And the reconstructing process is based on the matrix inversion that takes short computing time. In addition, the F-SPI has high imaging efficiency due to that the information of natural images mainly locates in the low-frequency region of Fourier spectrum [17,18]. Here a comparative simulation is done to compare the F-SPI method and the Z-SPI method in this section. Two typical images Peppers and Cameraman with a resolution of 128 ${\times} $128 pixels are selected as the object images. Both the Zernike and Fourier illuminating patterns need to be expressed as numerical matrix in simulation or experiment. Since the Zernike polynomials are defined in a circular area, the values of the corners outside the circle in Zernike patterns are set to be zeros. In order to compare the reconstructed images of F-SPI and Z-SPI fairly, only the values in the incircle of reconstructed image are remained and the values for other pixels are set to be zero while calculating the RMSEs. In F-SPI, the imaging process adopts Four-step phase-shifting method with a circular sampling path in the Fourier spectrum, which has been proved to have high reconstructing efficiency [17]. Figure 4 shows the simulation results of F-SPI and Z-SPI at different sampling ratios. And the RMSEs of reconstructed images are listed in Table 1. Here the sampling ratios are set to be 3%, 5%, 8% and 10%, respectively. When the sampling ratio increases from 3% to 10%, the imaging quality is improved and the details of the images become clearer. In Table 1, the RMSEs of Z-SPI are smaller than those of F-SPI when the sampling ratios are less than 10%. It indicates that the Z-SPI can provide better imaging quality than F-SPI especially at extremely low sampling ratios.

Fig. 4. The reconstructed results of (a) Peppers and (b) Cameraman images by F-SPI and Z-SPI techniques.

Download Full Size | PDF

Table 1. Comparisons of RMSEs (${\times} $10⁻²) of the reconstructed images under different sampling ratios

View Table | View all tables in this article

3.2 Rotation invariant of Zernike moments

As we analyzed above, the Zernike moments of rotated objects are rotation invariant that each Zernike moment merely acquires a phase shift and keeps the unchanged amplitude. This property can also be verified with a single-pixel detector and be used for object recognition and classification. Note that A_{n, -m} = A*_nm, then |A_nm|=|A_{n, -m}|. Therefore, we can concentrate on the |A_nm| with m $\ge \; $0. According to the theoretical analysis above, the amplitude of each Zernike moment can be expressed as

(16)$$\textrm{|}{A_{nm}}|= \sqrt {I_{nm}^2 + I_{n, - m}^2} , m \ge 0$$

where I_nm and I_n.-m can be derived from Eq. (11) and Eq. (12). Note that a differential modulation method was used in reality. Hence

(17)$$\textrm{|}{A_{nm}}|= \sqrt {{{(I_{nm}^ +{-} I_{nm}^ - )}^2} + {{(I_{n, - m}^ +{-} I_{n, - m}^ - )}^2}} , m \ge 0$$

where I ⁺_nm, I⁻_nm, I ⁺_{n, -m} and I⁻_n.-m are the intensity of single-pixel detector when the object is illustrated by P ⁺_nm(x, y), P⁻_nm(x, y), P ⁺_{n, -m}(x, y) and P⁻_n.-m(x, y), respectively. According to Eq. (17), calculating the amplitude of each Zernike moment needs 4 Zernike light patterns for illumination.

Here a simulation is done to verify the rotation invariant of Zernike moments. Figure 5 shows some binary images of letter A and gray images of Cameraman with six rotated versions. The rotation angles are set to be 0°, 30°, 60°, 120°, 180° and 300°, respectively. Each image has a resolution of 128 ${\times} $128 pixels. Table 2 is the list of the corresponding amplitudes of several Zernike moments calculated by above theory. In fact, each Zernike moment can have the rotation invariant during the rotating process on theory. However, the Zernike moments with high order can bring the complexity and numerical error of computation. Here we select several low order Zernike moments such as A₃₁, A₅₁ and A₇₁ for analysis without loss of generality. It is obvious that the amplitudes of all Zernike moments can be maintained well for either binary or gray images with different rotation angles. In Table 2, the tiny difference of a specific Zernike moment with different rotation angles may come from the discrete forms of image function and Zernike polynomials rather than a continuous one.

Fig. 5. The images of binary letter A and gray Cameraman with six rotated versions. The rotation angles from left to right are: 0°, 30°, 60°, 120°, 180°, 300°.

Download Full Size | PDF

Table 2. The amplitudes of several Zernike moments at different rotation angles

View Table | View all tables in this article

4. Laboratory experiments

4.1 Single-pixel imaging experiment

The proposed SPI technique is also verified by an experiment system. The experiment setup is shown in Fig. 6. The light source was a LASEVER LSR532NL-100 532-nm continuous wave laser. The output laser entered a beam expander (BE) and was projected to a DMD screen. The DMD system (Texas Instruments Discovery V7001) had 1024${\times} $768 individually addressable micro-mirrors and was used to modulate the illuminating laser. By loading the varying calculated images (as shown in Fig. 1) on the DMD screen, the illumination patterns can be generated. Then the modulated illuminating laser was projected to the object by a projection lens (PL) with a focal length of 300 mm. The PL was located 600 mm from the DMD screen and the object was also located 600 mm from the PL. The object and the DMD screen were in conjugate position of PL. The reflected light from object was collected by a collecting lens (CL) and then was detected by a single-pixel detector (SD, Thorlabs PMT-PMM02). The detected light intensities were transferred to the computer through a data-acquisition system (DAS, NI USB-6361). The computer was also used to load pattern images on the DMD system. A self-developed data acquisition software was used to record the illuminating patterns and the corresponding reflected light intensities synchronously.

Fig. 6. Experimental setup. BE (beam expander), PL (projection lens), CL (collecting lens), SD (single-pixel detector), DAS (data-acquisition system).

Download Full Size | PDF

In the experimental system above, the object consisted of another DMD system where a gray image with an English letter R was loaded on the central area. This DMD system had the same parameters with the one generating illumination patterns. When the illuminating laser was modulated, the intermediate 512${\times} $512 mirrors of the first DMD were utilized to generate illumination patterns. Each 4 mirrors were combined to show an image pixel. Then the resolution of the whole image was 128${\times} $128 pixels. To demonstrate the imaging effect of Z-SPI at low sampling ratios, a series of 128${\times} $128-pixels gray-scale Zernike basis patterns and Fourier basis patterns (Four-step phase-shifting) were generated to illuminate the object. And the object image was reconstructed under the sampling ratio at 3%, 5%, 8% and 10%, respectively. The reconstructed results were shown in Fig. 7. The images in the top line were F-SPI reconstructed results and the images in the bottom line were the Z-SPI reconstructed results. It can be seen that the Z-SPI technique can effectively reconstruct the object image under sampling ratio lower than 10%. The experimental RMSEs results of F-SPI and Z-SPI at different sampling ratios were shown in Table 3. We can see that the RMSEs of Z-SPI reconstructed images were lower than that of F-SPI reconstructed images when the sampling ratio was below 10%. Therefore, the Z-SPI method had outstanding information extraction capability especially at low sampling ratios.

Fig. 7. The experimental reconstructed images of letter R by F-SPI and Z-SPI techniques.

Download Full Size | PDF

Table 3. The RMSEs of reconstructed letter R under different sampling ratio

View Table | View all tables in this article

4.2 Object classification experiment

The rotation invariant of Zernike moments can be used for object classification. Here an object classification experiment based on the above experimental setup was done to demonstrate the classification power of Zernike moments. Figure 8 showed the database, five groups of rotated images of letters from A to E with rotation angles of 0°, 30°, 60°, 120°, 180° and 300°. Each letter was located in a circular dark background to avoid problems about the definition domain of Zernike polynomials. In order to demonstrate the object classification ability of the proposed Z-SPI technique, we loaded the five groups of rotated images on the objecting DMD system as the target. Then the specific Zernike patterns, such as P ⁺_nm(x, y), P⁻_nm(x, y), P ⁺_{n, -m}(x, y) and P⁻_n.-m(x, y), were loaded on the modulating DMD system for illumination. The corresponding reflected light intensities I ⁺_nm, I⁻_nm, I ⁺_{n, -m} and I⁻_n.-m were detected by the single-pixel detector. Then the amplitudes of corresponding Zernike moments can be calculated according to Eq. (17).

Fig. 8. The five groups of images of letters from A to E with rotation angles of 0°, 30°, 60°, 120°, 180° and 300° from left to right.

Download Full Size | PDF

In order to classify the above images, a classifier namely minimum-mean-distance (MMD) is used in this experiment [25]. Here an unknown image is represented by an m-dimensional feature vector X = [x₁, x₂, …, x_m]. Each class has several training samples that let $t_k^{(i )} = [{t_{k1}^{(i )},\; t_{k2}^{(i )}, \ldots ,\; t_{km}^{(i )}} ]$ denote the kth m-dimensional training feature vector of the ith class. The MMD classifier measures the sum of the squared distance between the feature vector of the test image X and the mean of the feature vectors of each class. Let d (X, i) be distance between test image X and the representation of class i. Then

(18)$$d(X, i) = \sum\limits_{j = 1}^m {({x_j} - } \frac{{sum(t_j^{(i)})}}{N}{)^2}$$

where N is the number of the training samples and $\frac{{sum(t_j^{(i)})}}{N}$ represents the sample mean of the jth element of the m-dimensional training feature vector of class i. The unknown test image X is classified to class i* when the distance is minimum among {d (X, i) I = 1, 2, …, c} where c is the number of classes.

Since the amplitudes of Zernike moments are invariant in the rotating process, the amplitudes of Zernike moments can act as the elements of feature vector X in this experiment. And the larger number of elements m means the better classifying effect. However, high order Zernike moments will bring large numerical and experimental errors. Therefore, without loss of generality, we measured the amplitude of Zernike moments |A₃₁|, |A₅₁| and |A₇₁| of image database shown in Fig. 8. Each Zernike moment needs four Zernike patterns for illumination. Therefore, a total of 12 Zernike patterns are required in this experiment. Additionally, the images without rotation (the leftmost images with 0° rotation angle in Fig. 8) act as the training samples of each letter class. The other images act as the test images. According to Eq. (18), the distance between each test image and each class can be calculated by

(19)$$d(X, i) = {(|{{A_{31}}} |- |{A_{31}^{(i)}} |)^2} + {(|{{A_{51}}} |- |{A_{51}^{(i)}} |)^2} + {(|{{A_{71}}} |- |{A_{71}^{(i)}} |)^2}, i = {A_0}, {B_0}, {C_0}, {D_0}, {E_0}$$

The experimental results of distances between the test images (letter A and B) and the training image of each class are shown in Table 4. Obviously, for both test rotated images of letter A and B, the distances from the corresponding class training image are minimum compared with those from other classes. Additionally, the cases of letter C, D and E coincides with the cases of letter A and B. When any test image is selected, we can measure the magnitudes of several specific Zernike moments and calculate the distances from different classes. It can be classified into the class with the minimum distance. In most cases, the distance between images in the same class is at least an order of magnitude smaller than the distance between images in different classes. It is not difficult to classify the rotated images by using the Z-SPI technique. Besides, the proposed Z-SPI technique can be used for object classification directly without any advanced training neural network shown in Ref. [29–30].

Table 4. The distance between the test images (letter A and B) and the training image of each class (${\times} {10^{ - 2}})$

View Table | View all tables in this article

In order to show the classification effect more directly, we establish a 2D subspace to show the distributions of different rotating letters (as shown in Fig. 9). The Zernike moment amplitudes |A₃₁| and |A₇₁| are set as the horizontal axis and the vertical axis, respectively. The values of |A₅₁| can also be used here which has the similar exhibition effect. As we can see, the images with the same letter (or class) and different rotating angles can be classified into one group obviously. And the images having different letters can be distinguished significantly. Therefore, by applying the Zernike rotation invariants, the Z-SPI technique can perform excellent object classification ability especially for rotating images. Moreover, only a few illuminating patterns are needed to measure the amplitudes of Zernike moments. Therefore, this method can achieve fast and accurate classification due to the high modulation speed of DMD system and the high sensitivity of single-pixel detector.

Fig. 9. The two Zernike moments amplitudes |A₃₁| and |A₇₁| of different letters.

Download Full Size | PDF

5. Conclusion

In summary, this paper demonstrates a novel single-pixel imaging technique based on Zernike patterns. Through theoretical and experimental analysis, the Zernike SPI can retrieve an image efficiently under low sampling ratio such as lower than 10% for images with 128${\times} $128 pixels. The reconstruction algorithm is based on the inverse Zernike moment transform so that it can realize rapid reconstruction with negligible calculation time. In addition, by using the Zernike rotation invariant, the proposed method can achieve object classification through a single-pixel detector. In the experiment, the images that have the same content and different rotating angles are classified into one class successfully. Different from the imaging process, the classification process only needs 4 illumination patterns for calculating the specific Zernike moment amplitude acting as the rotation invariant. Due to the high modulation speed of DMD system and the high sensitivity of single-pixel detector, it can achieve fast and accurate classification. Therefore, the proposed Zernike SPI can achieve both image reconstruction and classification effectively through a single-pixel detector, which can have great application prospects.

Funding

Youth Innovation Promotion Association of the Chinese Academy of Science (2020438).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Sci. Rep. 5(1), 10669 (2015). [CrossRef]

2. J. Greenberg, K. Krishnamurthy, and D. Brady, “Compressive single-pixel snapshot x-ray diffraction imaging,” Opt. Lett. 39(1), 111–114 (2014). [CrossRef]

3. A. Zhang, Y. He, L. Wu, L. Chen, and B. Wang, “Tabletop x-ray ghost imaging with ultra-low radiation,” Optica 5(4), 374–377 (2018). [CrossRef]

4. W. L. Chan, K. Charan, D. Takhar, K. F. Kelly, R. G. Baraniuk, and D. M. Mittleman, “A single-pixel terahertz imaging system based on compressed sensing,” Appl. Phys. Lett. 93(12), 121105 (2008). [CrossRef]

5. C. M. Watts, D. Shrekenhamer, J. Montoya, G. Lipworth, J. Hunt, T. Sleasman, S. Krishna, D. R. Smith, and W. J. Padilla, “Ter- ahertz compressive imaging with metamaterial spatial light modulators,” Nat. Photonics 8(8), 605–609 (2014). [CrossRef]

6. P. A. Morris, R. S. Aspden, J. E. C. Bell, R. W. Boyd, and M. J. Padgett, “Imaging with a small number of photons,” Nat. Commun. 6(1), 5913 (2015). [CrossRef]

7. C. Zhao, W. Gong, M. Chen, E. Li, H. Wang, W. Xu, and S. Han, “Ghost imaging lidar via sparsity constraints,” Appl. Phys. Lett. 101(14), 141123 (2012). [CrossRef]

8. G. M. Gibson, S. D. Johnson, and M. J. Padgett, “Single-pixel imaging 12 years on: a review,” Opt. Express 28(19), 28190–28208 (2020). [CrossRef]

9. T. Lu, Z. Qiu, Z. Zhang, and J. Zhong, “Comprehensive comparison of single-pixel imaging methods,” Opt. Lasers Eng. 134, 106301 (2020). [CrossRef]

10. B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3D computational imaging with single-pixel detectors,” Science 340(6134), 844–847 (2013). [CrossRef]

11. S. Sun, H. Lin, Y. Xu, J. Gu, and W. Liu, “Tracking and imaging of moving objects with temporal intensity difference correlation,” Opt. Express 27(20), 27851–27861 (2019). [CrossRef]

12. M. D’Angelo, A. Valencia, and H. M. Rubin, “Resolution of quantum and classical ghost imaging,” Phys. Rev. A 72(1), 013810 (2005). [CrossRef]

13. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). [CrossRef]

14. J. H. Shapiro and B. I. Erkmen, “Ghost imaging: from quantum to classical to computational,” Adv. Opt. Photonics 2(4), 405–450 (2010). [CrossRef]

15. M. J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nat. Commun. 7(1), 12010 (2016). [CrossRef]

16. P. G. Vaz, D. Amaral, L. F. R. Ferreira, M. Morgado, and J. Cardoso, “Image quality of compressive single-pixel imaging using different Hadamard orderings,” Opt. Express 28(8), 11666–11681 (2020). [CrossRef]

17. Z. Zhang, X. Wang, G. Zheng, and J. Zhong, “Hadamard single-pixel imaging versus Fourier single-pixel imaging,” Opt. Express 25(16), 19619–19639 (2017). [CrossRef]

18. Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of Fourier spectrum acquisition,” Nat. Commun. 6(1), 6225 (2015). [CrossRef]

19. J. Huang, D. Shi, K. Yuan, S. Hu, and Y. Wang, “Computational-weighted Fourier single-pixel imaging via binary illumination,” Opt. Express 26(13), 16547–16559 (2018). [CrossRef]

20. B. Liu, Z. Yang, X. Liu, and L. Wu, “Coloured computational imaging with single-pixel detectors based on a 2D discrete cosine transform,” J. Mod. Opt. 64(3), 259–264 (2017). [CrossRef]

21. Y. Chen, X. Yao, Q. Zhao, S. Liu, X. Liu, C. Wang, and G. Zhai, “Single-pixel compressive imaging based on the transformation of discrete orthogonal Krawtchouk moments,” Opt. Express 27(21), 29838–29853 (2019). [CrossRef]

22. D. Shi, J. Huang, W. Meng, K. Yin, B. Sun, Y. Wang, K. Yuan, C. Xie, D. Liu, and W. Zhu, “Radon single-pixel imaging with projective sampling,” Opt. Express 27(10), 14594–14609 (2019). [CrossRef]

23. K. M. Czajkowski, A. Pastuszczak, and R. Kotynski, “Single-pixel imaging with Morlet wavelet correlated random patterns,” Sci. Rep. 8(1), 466 (2018). [CrossRef]

24. P. Stojek, A. Pastuszczak, P. Wrobel, and P. Kotynski, “Single pixel imaging at high pixel resolutions,” Opt. Express 30(13), 22730–22745 (2022). [CrossRef]

25. A. Khotanzad and Y. H. Hong, “Invariant image recognition by Zernike moments,” IEEE Trans. Pattern Anal. Machine Intell. 12(5), 489–497 (1990). [CrossRef]

26. R. Mukundan, S. H. Ong, and P. A. Lee, “Image analysis by Tchebichef moments,” IEEE Trans. on Image Process. 10(9), 1357–1364 (2001). [CrossRef]

27. P. T. Yap, R. Paramesran, and S. H. Ong, “Image analysis by Krawtchouk moments,” IEEE Trans. on Image Process. 12(11), 1367–1377 (2003). [CrossRef]

28. C. W. Chong, P. Raveendran, and R. Mukundan, “A comparative analysis of algorithms for fast computation of Zernike moments,” Pattern Recongnit. 36(3), 731–742 (2003). [CrossRef]

29. S. Jiao, J. Feng, Y. Gao, T. Lei, Z. Xie, and X. Yuan, “Optical machine learning with incoherent light and a single-pixel detector,” Opt. Lett. 44(21), 5186–5189 (2019). [CrossRef]

30. J. Li, D. Mengu, N. T. Yardimci, Y. Luo, X. Li, M. Veli, Y. Rivenson, M. Jarrahi, and A. Ozcan, “Spectrally encoded single-pixel machine vision using diffractive networks,” Sci. Adv. 7(13), eabd7690 (2021). [CrossRef]

31. M. Yao, S. Zheng, Y. Hu, Z. Zhang, J. Peng, and J. Zhong, “Single-pixel moving object classification with differential measuring in transform domain and deep learning,” Photonics 9(3), 202 (2022). [CrossRef]

32. S. Jiao, “Fast Object Classification in Single-pixel Imaging,” arXiv, arXiv.1805.07582 (2018). [CrossRef]

	Peppers		Cameraman
Sampling ratio	F-SPI	Z-SPI	F-SPI	Z-SPI
3%	9.78	8.59	10.95	10.34
5%	8.00	7.16	9.60	9.07
8%	6.77	6.19	8.41	8.20
10%	6.24	5.88	7.86	7.90

Rotation angles	Binary letter A			Gray Cameraman
Rotation angles	\|A₃₁\|	\|A₅₁\|	\|A₇₁\|	\|A₃₁\|	\|A₅₁\|	\|A₇₁\|
0°	0.0267	0.0136	0.0151	0.1160	0.0483	0.0051
30°	0.0269	0.0138	0.0149	0.1155	0.0492	0.0045
60°	0.0269	0.0138	0.0149	0.1155	0.0492	0.0045
120°	0.0269	0.0138	0.0149	0.1155	0.0492	0.0045
180°	0.0267	0,0136	0.0151	0.1160	0.0483	0.0051
300°	0.0269	0.0138	0.0149	0.1155	0.0492	0.0045

Sampling ratio	3%	5%	8%	10%
F-SPI	0.1657	0.1619	0.1605	0.1598
Z-SPI	0.1584	0.1532	0.1498	0.1494

Test images	Rotation angles	Training image of each class
Test images	Rotation angles	A₀	B₀	C₀	D₀	E₀
A	30°	0.02	3.02	20.84	6.68	13.54
	60°	0.36	3.50	17.35	7.05	11.92
	120°	0.30	3.11	17.24	6.71	11.30
	180°	0.39	3.01	16.53	6.51	10.79
	300°	0.15	4.06	21.36	8.25	14.42
B	30°	4.36	0.18	10.39	0.68	3.53
	60°	3.29	0.07	10.37	0.80	4.10
	120°	5.06	0.35	10.37	0.86	3.06
	180°	4.35	0.22	9.78	0.79	3.13
	300°	2.23	0.07	11.92	1.24	5.43

	Peppers		Cameraman
Sampling ratio	F-SPI	Z-SPI	F-SPI	Z-SPI
3%	9.78	8.59	10.95	10.34
5%	8.00	7.16	9.60	9.07
8%	6.77	6.19	8.41	8.20
10%	6.24	5.88	7.86	7.90

Single-pixel imaging using discrete Zernike moments

Abstract

1. Introduction

2. Principles and methods

2.1 Zernike moments

2.2 Proposed single-pixel imaging method from Zernike moments

3. Computational simulations

3.1 Single-pixel imaging by Zernike moments

3.2 Rotation invariant of Zernike moments

4. Laboratory experiments

4.1 Single-pixel imaging experiment

4.2 Object classification experiment

5. Conclusion

Funding

Disclosures

Data availability

References

Data availability

Cited By

Figures (9)

Tables (4)

Equations (25)

Optics Express