1. Introduction

Automatic object recognition in scenes has been an interesting subject in military and industrial applications [1–16]. The aim of the automatic target recognition (ATR) system is to detect and identify objects in input scenes and label them as one of hypothesized classes. The discrimination capability of the recognition system is often challenged by arbitrary distortion of objects or the external noise of the sensor environment. Various imaging and signal processing techniques have been developed to improve the recognition performance. There are numerous techniques using two-dimensional (2D) object recognition in context, while recently there has been growing interest in three-dimensional (3D) imaging and recognition system. For example, we can utilize 3D complex images reconstructed at arbitrary depths and perspectives by means of digital holography [4,6].

Integral imaging (II) is a 3D sensing and visualization technique [17–20]. We record the directional information of rays using a micro-lenslet array. Each elemental image has its own perspective of objects according to the corresponding micro-lenslet. 3D scenes can be reconstructed optically or numerically in the opposite way of the recording [17,18]. The application of the II system has been extended to the object recognition and depth estimation [12–16].

There are various applications of photon-counting imaging such as night vision, laser radar imaging, radiological imaging, stellar imaging, and medical imaging [21–35]. Photon-counting imaging systems in general require less power than the conventional imaging systems that generate irradiance images. Image recognition with photon counting has been proposed in [23,24]. Recognition techniques with photon counting have been applied to infrared imaging and thermal imaging [25,26]. The classification of photon-limited images has been researched in [27,28], and 3D active sensing by LADAR [29–32]. In [33], maximum likelihood estimator was constructed to match photon-limited scenes for astronomical image reconstruction. Estimation of irradiance images has been researched in [34]. In [35], the Bhattacharyya distance is employed as an efficient merit for target detection in a low flux coherent active imagery. Recently, nonlinear matched filtering using photon-counting II has been developed in [16].

In this paper, we address a classification scheme for distorted objects by means of photon-counting II. Photon-counting linear discriminant analysis (LDA) is proposed to classify distorted objects. In photon-counting LDA, the irradiance images of the objects are used to train the classifier and pattern recognition is performed using the photon counts detected from unknown objects. The Fisher’s LDA has appeared in numerous researches for the classification of irradiance images [10–11,15]. The Fisher’s LDA maximizes the ratio of the between-class scatter to the within-class scatter [7–9]. The photon-counting LDA maximizes the Fisher’s criterion in the application of photon-limited images. It is shown that the Fisher’s criterion is solved in a slightly different way in the case of photon-limited images. By contrast to the Fisher’s LDA, the photon-counting LDA does not suffer from the singularity problem which is inevitable when the number of available data for the training is small. The photon-counting LDA determines linear hyper planes during the training on the feature space. The feature space is composed of the number of photo-counts per pixel. Thus, the dimension of the feature space is the same as the number of pixels in the image. It will be shown that an advantage of the photon-counting LDA is that it can handle such a high dimensional feature space without any dimensional reduction process. Although there have been investigations of nonlinear dimensional reduction or nonlinear kernel methods in the literature [36,37], the photon-counting LDA in this paper is shown to be robust using only linear boundaries in the high dimensional feature space. One may notice that there is difficulty in the high dimensional space for statistical learning which is called the curse of dimensionality (the required number of training data sets increases exponentially as the dimension increases). However, this notion becomes problematic not in the photo-counting LDA which is the parametric learning system, but in the non-parametric learning system [7,8].

Euclidean distance between unknown input vector and the trained class-conditional mean vectors is adopted for decision making. In the experiments, the 3D distortion is simulated by out-of-plane rotation as illustrated in Fig. 1 and photon-limited images are simulated from experimentally-captured irradiance images. The discrimination capability is evaluated by the correct and false classification rates as a function of varying number of photons, varying average irradiances. In addition we also consider an approximated within-class covariance estimation that is computationally more efficient. The performance is analyzed in terms of mean-squared distance (MSD) between the irradiance images of the distorted objects. We may anticipate that the smaller the distance between the images of the same classes and the larger the distance between the images of the different classes, the better the performance of the classifier. The experimental and simulation results show the photon-counting LDA can classify the distorted objects with a low level of photons.

The organization of the paper is as follows. In Section 2, we briefly review the advantages of II. The photon-counting model is described in Section 3. The photon-counting LDA with the overview of the Fisher’s LDA is presented in Section 4. The decision rule and performance analysis are discussed in Section 5. Experimental and simulation results are presented in Section 6. Conclusions follow in Section 7.

 

Fig. 1. A schematic diagram of the photon-counting II system for the distortion-tolerant object recognition.

Download Full Size | PDF

2. Overview of integral imaging

Integral imaging (II) or integral photography (IP) is a traditional 3D sensing and display technique [17–20]. A micro-lenslet or pinhole array is placed between 3D objects and an imaging sensor to sense directional irradiance information of rays from the objects as illustrated in Fig. 2(a). 3D display or reconstruction is the reversal process of the recording that the sensed image is illuminated from behind to form a 3D real image [See Fig. 2(b)]. The reconstruction can be performed optically or numerically. Figure 3 shows the multiple perspective imaging in ray optics. Each micro-lens in the lenslet array generates an elemental image on an imaging sensor which is placed on the image plane of the micro-lenses. An elemental image is a 2D image which has its own perspective and the field of view. Therefore, consecutive perspective scenes (a set of elemental images) are obtained by a single shot.

 

Fig. 2. (a) II sensing process, (b) II display process

Download Full Size | PDF

 

Fig. 3. Multiple perspective imaging in ray optics: each elemental image records the point A at the pixels which are located differently to its own optical axes (O₁~O₃) by rays 2,3,4, however, the same directional (parallel) information (rays 1,3,5) is recorded at the pixels of the same relative location.

Download Full Size | PDF

The scope of II has been extended to the object recognition and the depth estimation [12–16]. One advantage of II for 3D image recognition lies in its compactness of multiple-perspective imaging. Multi-view scenes of 3D objects are recorded by a single shot without using multiple-sensors or changing the position of the sensor. Therefore, we can build a compact system in capturing 3D information. In this paper, we assume that photon-limited scenes are generated by a photon-counting detector. Photon-limited scenes with multiple perspectives are recorded according to the corresponding lenslet as shown in Fig. 1.

3. Photon-counting detection model

The probability of counting y photons in a time interval τ can be shown to be Poisson distributed [21,22]. However, such a distribution assumes that the irradiance at the detector is perfectly uniform in time and space. Therefore, the statistics of the irradiance fluctuations should be considered if the irradiance is not uniform. However, for many cases of interest (i.e. blackbody radiation in the visible, including blackbody radiation from sources as hot as the sun) the fluctuations in irradiance are small compared to the fluctuations produced by the quantized nature of the radiation [22]. Therefore, we can assume that the probability distribution of photon-counting is Poisson distribution:

where y is the number of photon-counts produced by a detector during a time interval τ; and a is a rate parameter. The rate parameter can be given by

where η is the quantum efficiency of the detection process; P_o is the optical power incident on the detector; h is Plank’s constant; and ν̄ is the mean frequency of the quasi-monochromatic light source. The mean of photo-counts n_p is given by [38]:

where E(·) denotes the expectation operator.

Without the loss of generality we can assume that Poisson parameter for the photon-counts at each pixel is proportional to the irradiance of the pixel in a detector [16]. Therefore, the probability of photon event at pixel i can be given by

where y_i is the number of photons detected at pixel i. The parameter n_p(i) is given by

where N_P is an expected number of photon-counts in the scene; x_i is normalized irradiance at NT pixel i such that $\sum _{i=1}^{N_T}{x}_i=1;N_T$ is the total number of pixels in the scene, that is, N_T = N_x×N_y; and N_x and N_y are the size of the image in the x and y directions, respectively.

We can define a random vector y with the number of photon-counts of all pixels as

where each component of y follows independent Poisson distribution in Eq. (4); and the superscript t denotes transpose. The conditional mean vector and conditional covariance matrix of y given x, respectively become

and

where diag(·) denotes the diagonal matrix operator.

4. Classification of photon-limited images

In this section, first, the Fisher’s LDA is overviewed, and then, a new approach of the photon-counting LDA is discussed. Lastly, the parameter estimation is presented.

4.1 Overview of Fisher’s LDA

In conventional image classification, we often deal with irradiance images. Let a column vector composed of the irradiance values of the pixels be one realization of a random vector x∈R ^d×1, where R ^d×1 is d-dimensional Euclidean space; and d is the same with the number of pixels (N_T). The Fisher’s LDA maximizes the ratio of determinant of between-class covariance matrix to determinant of within-class covariance matrix. The within-class covariance is a measure of the concentration of each class, which is defined as

where w_j represents the event that the random vector x is a member of class j; and μ _x|wjis the class-conditional mean vector of x:

The between-class covariance is a measure of the separations of the classes. The between-class covariance matrix is defined as

where μ _x is the mean vector of x:

It is noted that ∑_xx=∑^W_xx+∑^B_xx, where ∑_xx is the covariance matrix of x:

The Fisher’s LDA can be implemented as

where the linear projector W_F∈R ^d×r satisfies the following Fisher’s criterion:

It is well known that Eq. (15) is equivalent with the generalized eigenvalues problem. The column vectors of W_F are eigenvectors of (∑^W_xx)^-1∑^B_xx corresponding to non-zero eigenvalues of (∑^W_xx)^-1∑^B_xx. It is noted that rank(∑_B)≤min[n_c-1,d]. Therefore W_F is composed of at most n_c-1 orthogonal vectors when d is larger than n_c-1. The maximum value of Eq. (15), |W^t_F∑^B_xxW_F|/|W^t_F∑^W_xxW_F| is equal to the summation of non-zero eigenvalues of (∑^W_xx)^-1∑^B_xx.

4.2 Photon-counting LDA

For photon-limited images, a random vector y is composed of numbers of photons detected. Let one realization of the random vector y be photon events at all pixels as in Eq. (6). Each component of y follows independent Poisson distribution. We find the following relationships of the first and the second moments between the irradiance random vector x and the photon event random vector y [See Appendix A]:

and

The within-class covariance matrix and the between-class covariance matrixes of y are, respectively derived as [See Appendix A]:

and

As in the Fisher’s LDA, the photon-counting linear discriminant function can be defined as

where w_P maximizes the following criterion:

It is noted that Eq. (21) is solved the same as in Eq. (15). The column vectors of W_F are eigenvectors of (∑^W_yy)^-1∑^B_yy corresponding to non-zero eigenvalues of (∑^W_yy)^-1∑^B_yy. The rank of ∑^B_yy is the same with that of ∑^B_xx. The maximum value of Eq. (21), |W^t_p∑^B_yyW_p|/|W^t_p∑^W_yyW_p| is equal to the summation of non-zero eigenvalues of {W^t_p[diag(μ _x)/N_p+∑^W_xx]W_p}^-1∑^B_xx. Also note that as N_p becomes large, Eq. (21) becomes the same as Eq. (15).

4.3 Parameter estimation

In the case that no probability information of x is available (true for many of the practical classification examples we encounter), needed parameters should be estimated in a proper way. In the following, the conventional estimators of the within-class covariance matrix and the between-class covariance matrix of the Fisher’s LDA are described, and then they are applied to the photon-counting LDA.

The within-class covariance matrix can be estimated by $\widehat{\sum }$ ^W_xx, which is derived in Appendix B:

where x _j(n) is the n-th realization of the object that belongs to class j; n_c is the number of classes; n_j is the number of realizations in class j; and n_t is the total number of realizations used to train the classifier, i.e., $n_t=\sum _{j=1}^{n_c}{n}_j$ ; and μ̂_x|wj is the class-conditional sample mean vector for class j:

The between-class covariance matrix can be estimated by ∑^B_xx [See Appendix B]:

where μ̂_x. is the sample mean vector of x:

Similarly, the within-class covariance matrix and the between-class covariance matrix for the photon-counting LDA can be estimated by [See Appendix B]:

and

For the Fisher’s LDA of irradiance images, the number of images used in the training process n_t is usually smaller than the dimension d of the image. Therefore, $\widehat{\sum }$ ^W_xx is a singular matrix since rank($\widehat{\sum }$ ^W_xx)≤min[n_t-n_c,d]. To overcome the singularity problem of $\widehat{\sum }$ ^W_xx, dimension reduction techniques such as principal component analysis (PCA) can be applied prior to the Fisher’s LDA [10,11,15] while direct implementation methods have been developed in [39]. However, as shown in Eq. (27), $\widehat{\sum }$ ^W_yy is nonsingular with the non-zero components of μ̂_x, therefore no additional process is required to avoid the singularity problem.

It is also noted that the sample covariance $\widehat{\sum }$ _xx|wj in Appendix B may not be a proper estimator when n_j≤d since $\widehat{\sum }$ _xx|wj is not a positive definite matrix which is the nature of the covariance matrix of any nonsingular random vector x. In the literature, there have been efforts to estimate the covariance matrix with a small sample (available training data) size and a large scale (dimension) [40]. For the photon-counting LDA, the within-class covariance matrix estimator $\widehat{\sum }$ ^W_yy in Eq. (27) is more governed by the diagonal components since it is anticipated that the covariance between the irradiance values of two different components (pixels) is much smaller than the mean values and closer to zero when they are further apart. Therefore, one may postulate that the following approximation holds:

and

where x _j,i(n) and μĮ_x|wj,i. are the i-th component of the n-th training vector x _j(n) and the class-conditional sample mean vector μ̂_x|wj, respectively, that is, ∑^W_yy in Eq. (28) is approximated by merely adopting the diagonal components of $\widehat{\sum }$ ^W_yy in Eq. (27). It is noted that the computational complexity of the inversion of Eq. (28) is significantly lower than that of Eq. (27) when d is large. In the experiments described in Section 6 we compare the performance of the photon-counting LDA between using Eq. (28) and Eq. (27).

5. Decision rule and performance evaluation

In this section, the decision rule and measures for performance evaluation and analysis are discussed. Our decision rule adopts Euclidean distance between unknown input vector and the trained class-conditional mean vectors. In the experiments, the photon-limited integral image (a set of elemental images from one lenslet array) is obtained from different object orientations. The random vectors x and y, respectively are the irradiance and the photon event vectors corresponding to one elemental image. Therefore, multiple-perspective training data of a specific orientation can be constituted by a single exposure. During the test, the multiple photon event vectors are used to take advantage of the multiple perspective imaging, thus, the test vector for an unknown input scene is

where y _test(n) is the photon event vector corresponding to the n-th photon-limited elemental image and n_test is the number of elemental images tested. The images used for testing are gathered from unknown rotations of object.

We classify a vector z _test as the member of class ĵ if

where ∥·∥ stands for Euclidean norm; and μ̂_z|wj is the estimate of the class-conditional mean vector. Assuming the distribution of y _test is of the same as the distribution of the images y used for training, we can show that

To evaluate the performance, two performance measures are calculated: correct classification rate and false classification rate which are, respectively defined as

and

The performance is analyzed in terms of MSD between elemental images used for training and elemental images used testing. The MSD between two random vectors is defined as

and estimated as

where x _j is the irradiance random vector for training representing class j; x _j(n) is the n-th realization of x _j; x _test(s,i_s) is the irradiance random vector for testing labeled as class s and rotation angle i_s; and x _test(m;s,i_s) is the m-th realization of x _test(s,i_s). In the experiments, the rotation angle for training is fixed. It is noted that the irradiance vectors in Eq. (36) are the normalized elemental images used for training and testing due to the normalized condition of the irradiance image in Eq. (5).

6. Experimental and simulation results

In this section, we present the experimental and simulation results. The experiments and simulation consist of (1) capturing irradiance of the integral images of objects with varying orientations, (2) aligning elemental images and training the estimator with the photon-counting LDA, and (3) simulating photon-counts and testing them with the NN decision rule. The validity of Eq. (28) and the effects of the average irradiance variation are experimented. The performances are analyzed in terms of MSD in Eq. (36).

6.1 Integral imaging acquisition and preprocessing

The optical set-up is composed of a micro-lenslet array, an imaging lens, and a CCD camera. The focal length of each micro lenslet is about 3 mm, the focal length of the imaging lens is 50 mm, and the f-number of the imaging lens is 2.5. The imaging lens is placed between the lenslet array and the CCD camera due to the short focal length of the lenslets. Three classes of toy cars are used in the experiments as shown in Fig. 4. The size of three toy cars is about 2.5 cm×2.5 cm×4.5 cm. The distance between the CCD camera and the imaging lens is about 7.2 cm, and the distance between the micro-lenslet array and the imaging lens is about 2.9 cm. Integral images of the toy cars are gathered at rotation angles of 30°, 33°, 36°, 39°, 42°, and 45°. Rotation is with respect to the perpendicular to the optical axis of the micro-lenslet array. Thus, six integral images for each toy car are obtained; one at each of the six different out-of-plane rotation angles. Captured irradiance images are the same ones in [15] except that 30 (5 by 6) elemental images located in the center are used in this paper.

Before simulating photon-limited images, each elemental image is aligned and cropped. The alignment is the pre-process to enhance the system performance. Through the alignment, we aim to minimize the MSD in the same class. All elemental images are shifted by maximizing cross-correlation coefficients as in [15]. The reference elemental image for the alignment is the central elemental image in the integral image of the object rotated at 36° for each class. After the alignment, each elemental image is cropped by 60×125 pixels considering the computational load and accuracy of computing the linear discriminant function. Therefore, the dimension (d) of the vectors x and y is 7500 (=60×125). The sizes of the integral image in the row and the column directions are 300 (=60×5) and 750 (=125×6), respectively. Figure 5 shows the movies of 6 integral images (frames) for each toy car with the out-of-plane rotation. In a practical situation, photon-limited elemental images can be aligned and cropped by a pre-process like non-linear correlation filtering of photon events [16].

 

Fig. 4. Three toy cars used in the experiments; they represent class 1, 2 and 3 from right to left.

Download Full Size | PDF

 

Fig. 5. Movies of integral images of three objects with the out-of-plane rotation, (a) (the movie file size: 3.64 MB) class 1, [Media 1] (b) (the movie file size: 3.64 MB) class 2, [Media 2] (c) (the movie file size: 3.64 MB) class 3. [Media 3]

Download Full Size | PDF

6.2 Classification results and performance analysis

Our goal is to categorize each integral image into its correct class of the toy cars. For training, only one integral image is used for each class (object). That image is the one associated with a rotation angle of 36°. It corresponds to the third frame of each movie in Fig. 5. The other five integral images for each class are used only for testing. The integral image used for training in each class is composed of 30 elemental images, thus, the number of vectors (n_j) associated with training from each integral image is 30. The number of classes (n_c) is 3 so the total number of vectors (n_t) used in training the estimator is 90 (=30×3).

For the test, all of 18 integral images are used, including the three integral images used in training. Each integral image is considered an unknown input scene. 1000 photon-counting scenes are generated for each integral image and the correct and false classification rates in Eqs. (33) and (34) are obtained from these 1000 realizations. The photon number is simulated by the Poisson random number generator in MATLAB with N_p = 3 for each elemental image normalized. The number of test elemental images, n_test in Eqs. (30) and (36) is 30 since each integral image is composed of 30 elemental images. Therefore, the mean number of photon-counts (N_p×n_test) in the entire scene (integral image) is 90 (=3×30). The averaged classification results are illustrated in Fig. 6.

The training and test are repeated with N_p = 5 and 10 when the corresponding mean numbers of photon-counts are 150 and 300, respectively. The averaged classification results are presented in Figs. 7 and 8, respectively. As illustrated in Figs. 6 to 8, a low level of photons can classify the distorted objects. The averaged correct classification rates increase when a larger number of photons are used while the averaged false classification rates decrease. Figure 9 shows an example of the test input scene (photon-limited integral image) with N_p = 10. Figure 9 is generated from the integral image of the class 1 with the rotation angle 30° which corresponds to the first frame of the movie in Fig. 5(a). The actual number of photons in Fig. 9 is 289.

 

Fig. 6. Classification results when the mean photon number in the test scene is 90. (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs. ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

 

Fig. 7. Classification results when the mean photon number in the test scene is 150. (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs. ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

 

Fig. 8. Classification results when the mean photon number in the test scene is 300. (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs. ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

 

Fig. 9. An example of the test input scene corresponding to class 1 with the rotation angle 30°, the mean photon number in the test scene is 300, and the actual number of photon-counts in this scene is 289.

Download Full Size | PDF

The performance is analyzed in terms of the MSD in Eq. (36). The different classification results between the classes and the rotation angles may be correlated with the MSD between the irradiance images. Figure 10 shows the MSD between the elemental images used for training and elemental images used for testing. The MSD is computed for images within the same and different classes. Overall performance of the class 3 is shown to be the best since the MSD between class 3 and other classes are larger than other MSD’s as illustrated in Figs. 10(a) and 10(b). For the correct classification rates, the performance of the class 1 is worse than others since the MSD between the class 1 and other classes are smaller as illustrated in Figs. 10(b) and 10(c).

 

Fig. 10. MSD in Eq. (36) when (a) j = 1, (b) j = 2, (c) j = 3.

Download Full Size | PDF

Figure 11 shows the averaged classification results when the approximated within-class covariance estimator in Eq. (28) is used. The mean number of photon-counts in the test scene is 150 (N_p = 5), which is the same as in Fig. 7. Figure 11 shows very similar performance with Fig. 7.

 

Fig. 11. Classification results when the mean photon number in the test scene is 150, Eq. (28) is used for the within-class covariance estimation, (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs, ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

6.3 Classification results with the average irradiance variation

In a practical situation, the average irradiance might vary. Even though the irradiances are normalized, the resulting histogram of the normalized images may narrow or widen as a result of the change in average irradiance. Therefore, we performed experiments in which the average irradiance is increased. In the first experiment, it is assumed that the increased average irradiance is known, and in the second experiment, the average irradiance is unknown.

The irradiance values of the integral images in Fig. 5 are gray scaled between 0 and 255. The average irradiance values of three trained integral images are 129.86, 127.26, and 129.84, respectively. In the first experiment, the irradiance values of all the pixels are increased by 50 before normalizing elemental images, and the classification performance is tested with N_p = 10 over 1000 runs, that is, the mean photon number in the test input scene is 300. Figure 12 shows the averaged classification results and Fig. 13 shows the MSD between the elemental images used for training and the elemental images used for testing. As shown in Fig. 12, the performance is degraded compared with Fig. 8. The smaller MSD presented in Fig. 13 than in Fig. 10 may cause the performance degradation. Figure 14 shows the classification results with the approximated within-class covariance estimator in Eq. (28), showing very similar results with Fig. 12.

 

Fig. 12. Classification results when the mean photon number in the test scene is 300, (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs, ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

 

Fig. 13. MSD in Eq. (36) when (a) j = 1, (b) j = 2, (c) j = 3.

Download Full Size | PDF

 

Fig. 14. Classification results when the mean photon number in the test scene is 300, Eq. (28) is used for the within-class covariance estimation, (a) averaged correct classification rate for each class over 1000 runs, (b) averaged false classification rate for each class over 1000 runs, ‘Avg’ denotes the average value of 6 rotation angles.

Download Full Size | PDF

There might be the case that the average irradiance in the scenes is unexpectedly changed. In the following experiments, the performance of the photon-counting LDA is empirically tested in the case of the unknown average irradiance variation. The average irradiance is increased by 50 only for the test input integral images before the normalization of the elemental images and the photon-counts simulation while the trained images are the same ones in subsection 6b. Therefore, there is significant average irradiance variation between the trained scenes and test scenes. Figure 15 shows the averaged classification results over the all rotation angles and 1000 runs. The mean photon number in the test scene varies from 90, 450, 900, 4500, and 9000, that correspond to N_p = 3, 15, 30, 150, and 300, respectively. As illustrated in Fig. 15, when the mean photon number is 900, reliable classification results (more than about 0.95 for the correct classification rate and less than about 0.05 for the false classification rate) are obtained for all the classes. It is noted that the mean photon number 900 in the entire scene can be considered a very low level of photons (900 photons at an average wavelength of 4000nm represents an energy of less than 10^-16 J incident on the focal plane). The experimental results with the approximated within-class covariance matrix estimator in Eq. (28) are shown in Fig. 16 showing very similar results with Fig. 15. From Figs. 15 and 16, the photon-counting LDA is shown to overcome the unknown average irradiance problem with a moderate number of photon-counts. It can be thought that with a considerably large expected photon number N_p, the photon-counting LDA criterion in Eq. (21) becomes closer to the Fisher’s criterion in Eq. (15), and the effect of the average irradiance variation can be approximately compensated by the ratio of the between-class covariance matrix to the within-class covariance matrix in Eq. (15). However, because the class-condition mean vectors in the decision rule in Eq. (31) are estimated without the average irradiance variation, the performance can be degraded as shown in Figs. 15 and 16.

 

Fig. 15. Classification results when the average irradiance is increased by 50 only for the test images, (a) averaged correct classification rates versus the mean photon numbers, (b) averaged false classification rates versus the mean photon numbers.

Download Full Size | PDF

 

Fig. 16. Classification results when the average irradiance is increased by 50 only for the test images, Eq. (28) is used for the within-class covariance estimation, (a) averaged correct classification rates versus the mean photon numbers, (b) averaged false classification rates versus the mean photon numbers.

Download Full Size | PDF

7. Conclusions

In this paper, we propose a distortion-tolerant automatic recognition system using the photon-counting integral imaging. The photon-counting detector combined with the micro-lenslet array generates photon-limited multi-view scenes. Photon events are modeled by Poisson distribution. A new approach of the photon-counting LDA is proposed for the classification of photon-limited images. In the photon-counting LDA, the irradiance values are used for training while photon-limited images are tested to classify unknown input objects. A compact 3D information processing is possible and the performance can be enhanced by means of the multiple perspective photon-limited scenes of II.

It may be possible to extend the proposed photon-counting 3D image recognition system to broader applications including ordinary irradiance imaging since the photon-limited images can be extracted from the irradiance values by either reducing the input light and/or using the Poisson model for photon counting. In that case, we may take advantage of the low number of photons generated by the photon-counting detector to improve the computational efficiency of the recognition process. Further experimentations are necessary to investigate the potential benefit of the proposed method to a broad class of image recognition systems.