In this paper, we propose photon counting three-dimensional (3D) passive sensing and object recognition using integral imaging. The application of this approach to 3D automatic target recognition (ATR) is investigated using both linear and nonlinear matched filters. We find there is significant potential of the proposed system for 3D sensing and recognition with a low number of photons. The discrimination capability of the proposed system is quantified in terms of discrimination ratio, Fisher ratio, and receiver operating characteristic (ROC) curves. To the best of our knowledge, this is the first report on photon counting 3D passive sensing and ATR with integral imaging.
©2005 Optical Society of America
Automatic target recognition (ATR), which is the ability of a machine to detect and identify objects in a scene and categorizing them into a class, has been the subject of intense research [1–9]. ATR systems need to deal with uncooperative objects in complex scenes. Statistical fluctuations of the scene and the objects caused by noise, clutter, distortion, rotation, and scaling changes create many challenges toward achieving a reliable ATR system. Numerous techniques using two-dimensional (2D) image processing have been developed while in recent years there has been growing research interest in 3D object recognition to enhance discrimination capability of unlabeled objects [6,7]. Additional benefits of 3D imaging include the ability to segment the object of interest from the background and to change the point of view of the observer with respect to the image. Many 3D imaging techniques involve some form of active illumination; the waveform that is transmitted is used to derive the range dimension of the image. However, for imaging applications in which cost and covertness are important, the use of an illumination source may not be feasible.
Photon counting imaging has been applied in many fields such as night vision, laser radar imaging, radiological imaging, and stellar imaging [10–18]. Advances in the field, have produced sensitive receivers that can detect single photons. Devices with high gain produce large charge packet upon absorption of a single incident photon. The number of carriers in charge packet is ignored and only location of the charge packet is recorded. Therefore, imagery is built up a photo-count at a time, and 2D imagery (irradiance) is developed over time. These highly sensitive receivers allow decrease in required transmitted power over conventional imaging sensors, and trade power for integration time. 2D image recognition using photon counting techniques have been demonstrated [11,12]. Photon counting techniques have been applied to infrared imaging and thermal imaging [13,14]. Photon counting detectors have been considered for 3D active sensing by LADAR [16–18].
Integral imaging is a sensing and display technique for 3D visualization [19–22]. An integral imaging system consists of two stages. In the pickup (sensing) stage, an array of micro-lenslets generates a set of 2D elemental images which are captured by a detector array (image sensor). Each elemental image has a different perspective of the 3D object. Therefore, the image sensor records a set of projections of the object from different perspectives. In the reconstruction stage, the recorded 2D elemental images are projected through a similar micro-lens array to produce the original 3D scene. Integral imaging is a passive sensor and unlike holography or LADAR, it does not require active illumination of the scene under investigation. The application of integral imaging has been extended to object recognition and longitudinal distance estimation [23–26]. The resolution of the 3D image reconstruction and the accuracy of depth estimation have been improved by moving array lenslet technique in . Statistical pattern recognition technique has been applied for distortion-tolerant 3D object recognition .
In this paper, we propose a photon counting passive 3D object sensing and recognition by integral imaging. Target recognition is implemented using a nonlinear matched filter. We analyze the statistical properties of the nonlinear correlation normalized according to the power law of the sum of photon numbers. We investigate the potential of the proposed system for 3D sensing and recognition with a low number of photons. The photon counting can be binary at low light level [11–14]. The advantage of the photon-counting detector may be enhanced by the processing of binary photon numbers which can be simpler and faster . We recognize that in special cases in which we are imaging very bright objects in dark backgrounds that the binary assumption will not be valid (more than 1 count per pixel will be obtained). However for low numbers of detected photons and for scenes in which the total irradiance is distributed more evenly over the image, the assumption is valid, allowing us to use the benefits of binary imaging. The discrimination capability of the proposed system will be demonstrated in terms of discrimination ratio, Fisher ratio, and ROC curve.
In Section 2, we provide an overview of the 3D sensing and visualization technique using integral imaging. In Section 3, the photon counting model is presented for the simulation of photon-limited images. Pattern recognition application is illustrated with performance evaluation metrics in Section 4. In Section 5, simulated photon-limited images are generated from experimentally-derived integral images. These simulated images are used to verify the results presented in Section 4. Conclusions are presented in Section 6.
2. Integral imaging
Techniques based on integral imaging have been considered for 3D sensing and display. As illustrated in Fig. 1, we can use a pinhole array or a micro-lenslet array to sense irradiance and directional information of rays from 3D objects during the pickup process. One advantage of integral imaging for 3D image recognition lies in its capability of multiple perspective imaging. The depth and perspective information in the multiple perspective imaging can be utilized to build a compact 3D recognition system. In this paper, we assume that each lenslet in the lenslet array generates a photon-limited elemental image on an image sensor. Multiple perspectives of photon-limited scenes are recorded according to the relative position and the field of view (FOV) of the lenslet. In the following sections, we will show that different perspective information of 3D object can be acquired by the photon counting integral imaging. The multi-view effect of integral imaging can be beneficial for the pattern recognition although it can degrade 3D visualization quality .
Reconstruction is the reverse of the pick-up process. It can be carried out optically and computationally from elemental images. In display, 3D images are formed by the intersection of discrete rays coming from elemental images. The 3D scene of the pseudoscopic real image is formed by propagating the intensity of elemental images through the lenslet array which is placed in front of the display device. The pseudoscopic real image is displayed by the rays from opposite directions but having the same intensities as in the sensing process. The 3D display of integral imaging provides autostereoscopic images with full parallax and continuously varying viewpoints [21,22]. In the computational method, the reconstruction is processed numerically using a computer. Computational reconstruction has been researched with various techniques [28,29].
3. Photon counting detection model
Given fairly standard assumptions, the probability of counting k photons in a time interval τ can be shown to be Poisson distributed . However, such a distribution assumes that the irradiance at the detector is perfectly uniform in time and space. In the case that the irradiance is not uniform, the statistics of the irradiance fluctuations must also be considered. However, it can be shown  for many cases of interest (i.e., blackbody radiation in the visible, including blackbody radiation from sources as hot as the sun) that the fluctuations in irradiance are small compared to the fluctuations produced by the quantized nature of the radiation. Therefore, in this section we model the probability distribution as Poisson:
where k is the number of counts produced by a detector centered on a position vector x during a time interval τ; and a(x) is a rate parameter. The rate parameter can be given by:
where χ is the quantum efficiency of the detection process; E is the power incident on the detector; h is Plank’s constant; and ν¯ is the mean frequency of the quasi-monochromatic light source. It is noted that the mean of photo-counts np (x) is given by:
where 〈·〉 stands for the expectation operator.
To simulate photon-limited images from an intensity image, we assume the probability of detecting more than one photon in a pixel is zero. While this assumption does place restrictions on the allowed irradiance distribution in the image, we make it in anticipation that the image will contain very few photons, i.e. np (x) << 1.
It can be shown  that the probability of detecting a photo-event at the ith pixel is given by the normalized irradiance image. Since the event location for each count is an independent random variable, the mean of photo-counts at the ith pixel is given by:
where NP is a predetermined number of counts in the entire scene; S(xi ) is the irradiance of the input image at the pixel i; and NT is the total number of pixels. From Eq. (1), we can obtain the probability that no photon arrives at pixel i:
According to the assumption above, the probability that only one photon is detected:
To simulate an image consisting of individual photo-counts, we generate a random number [rand(xi )] which is uniformly distributed between 0 and 1 at each pixel. If the random number is less than the probability that no photon arrives [Eq. (5)], we assume that no photon is detected, otherwise, one photon is assumed to be detected:
where rand(xi ) stands for the random number generated for pixel i.
If the image contains pixels whose irradiance is much larger than the mean image irradiance, then the above assumption may not be valid. In this case, np (x) can be large enough so that the assumption np (x)<<1 is not valid even at the anticipated low-level of photons in the scene. In this case, our hypothesis could be revised to allow multiple photon detection per pixel.
4. Pattern recognition using a nonlinear matched filter
As shown in Eq. (7), the photon counting is assumed to be a binary image:
where bi is a random number which follows Bernoulli distribution. The probability function of bi is:
Equivalently, one realization of a photon-limited image can be described as:
where N is the total number of photon detection events occurred in the scene; δ is a kronecker delta function; and xk represents the position of the pixel k where a photon detection event occurs. It is noted that N and xk are random numbers.
Matched filtering of photon-limited images estimates the correlation between the intensity images of a reference and an unknown input image obtained during the photon counting event. We define our matched filtering as the nonlinear correlation normalized with the power v of the photon-limited image intensity as shown below:
where N is the total number of photons detected; R is the radiance of the reference image from the object class r; and s represents an unknown input object from which the photon-limited image Ŝ is generated. The second term in Eq. (14) is derived by Eq. (13). Without loss of generality, we may assume that R(xi ) and S(xi ) are normalized:
It is noted that Crs (xj ;v) has the maximum value at xj = 0 in our experiments:
One advantage of photon counting detection is that the computational time of the matched filtering is much faster than conventional image correlation. As shown in the second term in Eq. (14) the correlation becomes merely the sum of the reference radiance at particular pixels (photon arrivals) .
where “var” stands for the variance.
We will prove the statistical properties of nonlinear correlation peak Crs (0;1) [see Eq. (14)] in Appendix B:
The nonlinear matched filtering shows different behaviors according to v. When v = 0, both the mean [Eq. (18)] and variance [Eq. (19)] of the correlation peak Crs (0;0) are approximately proportional to NP since the second term including NP in Eq. (19) affects very minimally. However, the mean of Crs (0;1) [Eq. (20)] does not depend on the number of photons, i.e., we can theoretically achieve the same correlation value with any small number of photons. Although the variance of Crs (0;1) [Eq. (21)] increases when using lower number of photons, this property of photon-limited images might be beneficial for pattern recognition applications. We would like to point out that a number of filtering algorithms may be used for ATR of photon-limited images including a variety of nonlinear filters [1–6,31].
We define discrimination ratio (DR) and Fisher ratio (FR) as our performance metrics :
where mrs and σrs are the sample mean and the sample standard deviation of Crs (0;v), respectively, which are suitable estimates for the mean and the variance. Receiver operating characteristic (ROC) curves are also illustrated in the experimental results to investigate the discrimination capability of the proposed system.
5. Experimental and simulation results
5.1. 3D sensing with integral imaging
In this section, we present the results of simulations derived from experimentally produced integral images for a number of 3D objects. The experimental system is composed of a micro-lens array and a pick-up camera as shown in Fig. 2(a). The pitch of each microlens is 1.09 mm and the focal length of each microlens is about 3 mm. The focal length and the f-number of the mount lens in the pick-up camera are 50 mm and 2.5, respectively. Three types of toy cars are used in the experiments as shown in Fig. 2(b). Each car is about 4.5 cm × 2.5 cm × 2.5 cm. The distance between the pick-up camera lens and the microlens array is 11.5 cm, and the distance between the microlens array and the objects is 7.5 cm.
A set of 20×24 elemental images is captured at one exposure. One set of elemental images for one object is composed of 1334×1600 pixels and the size of one elemental image is approximately 67×67 pixels. Three sets of elemental images are shown in Fig 3. The intensity image of the reference (r) or the unlabeled input (s) corresponds to one set of elemental images captured during the pick-up process.
5.2 Photon counting simulation
We simulate photon-limited images using the irradiance of the elemental images to calculate np (xi ) from Eq. (4). Equations (5) and (7) are then used to generate the photon-limited images. Several values of NP , mean number of photo-counts in the entire image, are used to test the proposed image recognition. Figure 4 shows the probability map that no photon arrives on the detector [Eq. (5)] for car 1 when NP = 1,000. Figure 5 shows photon-limited images generated for cars 1–3 [Fig. 3(a)-(c)] when NP = 1,000. Figure 6(a) shows magnified elemental images for car 1 [Fig. 3(a)] in the center and Figs. 6(b) and (c) show magnified photon-limited elemental images corresponding to Fig. 6(a) when NP = 10,000 and 1,000, respectively. The numbers of photons shown in Fig. 6(b) and (c) are 272 and 25, respectively.
5.3 Image recognition results
We generate photon-limited images, each with a random number of photons. To compute the statistical means and variances we generate 1000 images for each car. We also vary the mean value photon numbers from 10 to 1,000. The intensity image of car 1 [Fig. 3(a)] is used as our reference image. Figure 7(a) shows the experimental results (sample mean) of correlation coefficients and their fluctuations (error bars) when v = 0 with theoretical prediction in Eq. (18). Error bars stand for mrs ±σrs. The red solid line graph represents the sample mean of autocorrelation between the intensity image and photon-limited images of car 1, and the blue dotted line graph is the sample mean of cross-correlation between the intensity image of car 1 and photon-limited images of car 2, and the black dashed line graph is the sample mean of cross-correlation between the intensity image of car 1 and photon-limited images of car 3. Figure 7(b) shows the sample variance of Crs (0;0) with theoretical prediction in Eq. (19). Figure 8(a) and (b) show the sample mean and the sample variance of Crs (0;0.5), respectively. Figure 9(a) and (b) show, respectively, the sample mean and the sample variance of Crs (0;1) with the theoretical values in Eq. (20) and (21). As shown in Fig. 7 and Fig. 9(a), theoretical values are presented to be very close to the experimental results. Figure 9(b) shows the approximated theoretical value of the variance. The deviation from the theoretical prediction becomes larger as the number of photons decreases as shown in Fig. 9(b).
Table 1 shows the discrimination ratios defined in Eq. (22) for Crs (0;0), Crs (0;0.5), and Crs (0;1). There appears to be only small variation of the discrimination ratio with varying number of photo-counts. Table 2 shows the Fisher ratio defined in Eq. (23) for Crs (0;0), Crs (0;0.5), and Crs (0;1). Fisher ratio decreases when using a lower number of photo-counts, but for photo-counts greater than one hundred, the Fisher ratios show good separability [13,14] between the reference and false objects when v = 1. Fisher ratios are larger when v = 1 than other values since the approximated theoretical value of the variance is proportional to 1/NP .
In the paper, we propose a photon counting passive 3D sensing for ATR using integral imaging. Micro-lenslet array generates photon-limited multi-view (elemental) images to discriminate unknown 3D objects. Photon events are modeled with Poisson distribution. The proposed nonlinear correlation of photon-limited images can improve the system performance. The first and second order statistical properties of the nonlinear correlation output are determined. We have observed in the experiments that the output of the nonlinear correlation provides better performance than the linear matched filter.
The proposed photon counting passive 3D sensing with integral imaging seems to be robust for pattern recognition. Unknown objects captured by integral imaging may be recognized with a small number of photons. Additional investigation of the proposed system which includes detector noise and photon counting events with more than one photon arrival per pixel remains for future studies.
since Ŝ(x 1),…,Ŝ(x NT) are assumed to be independent Bernoulli random variables; and var(Ŝ(x i))=np (xi )-(xi ).
With the assumption of pj <<1, the moment generating function of Ni becomes:
Since e αi (es -1) is the moment generating function of the Poisson distribution, we assume the distribution of Ni approximately follows the Poisson distribution with the parameter αi :
Next, we derive as followings:
The moment generating function of yi is:
Thus, the mean of yi becomes:
In the above equation, we use:
since S(xi ) <<1, and e -αi <<1.
Also, we derive var(yi ) and cov(yi , yi ) as:
We derive 〈〉 as followings:
To derive (yi yj ), we define Nij as:
where Nij approximately follows the Poisson distribution as Ni in Eq. (B7):
Therefore, the joint moment generating function of yi and yj is:
Since 〈yi 〉 ≈ S(xi ) in Eq. (B10),
We are thankful to Mr. Yong-wook Song and Mr. Rodrigo Ponce-Diaz’s help for integral imaging experiments. We are also thankful to Mr. Seongho Song in statistics department for valuable discussion.
References and links
1 . F. Sadjadi , “ Improved target classification using optimum polarimetric SAR signatures ,” IEEE Trans. on Aerosp. Electron. Syst. 38 , 38 – 49 ( 2002 ). [CrossRef]
2 . A. Mahalanobis , R. R. Muise , S. R. Stanfill , and A. V. Nevel , “ Design and application of quadraticcorrelation filters for target detection ,” IEEE Trans. on Aerosp. Electron. Syst. 40 , 837 – 850 ( 2004 ). [CrossRef]
3 . H. Sjoberg , F. Goudail , and P. Refregier , “ Optimal algorithms for target location in nonhomogeneous binary images ,” J. Opt. Soc. Am. A. 15 , 2976 – 2985 ( 1998 ). [CrossRef]
4 . H. Kwon and N. M. Nasrabadi , “ Kernel RX-algorithm: a nonlinear anomaly detector for hyperspectral imagery ,” IEEE Trans. on Geosci. Remote Sens. 43 , 388 – 397 ( 2005 ). [CrossRef]
5 . F. Sadjadi , ed., Selected Papers on Automatic Target Recognition , (SPIE-CDROM, 1999 ).
6 . B. Javidi , ed., Image Recognition and Classification: Algorithms , Systems, and Applications, ( Marcel Dekker, New York , 2002 ). [CrossRef]
7 . B. Javidi and F. Okano , eds., Three-dimensional television, video, and display technologies , ( Springer, New York , 2002 ).
8 . R. O. Duda , P. E. Hart , and D. G. Stork , Pattern classification 2nd , ( Wiley Interscience, New York , 2001 ).
9 . C. M. Bishop , Neural networks for pattern recognition , ( Oxford University Press, New York , 1995 ).
10 . E. Hecht , Optics 4 th Edition , ( Addison Wesley , 2001 ).
11 . G. M. Morris , “ Scene matching using photon-limited images ,” J. Opt. Soc. Am. A. 1 , 482 – 488 ( 1984 ). [CrossRef]
13 . E. A. Watson and G. M. Morris , “ Comparison of infrared upconversion methods for photon-limited imaging ,” J. Appl. Phys. 67 , 6075 – 6084 ( 1990 ). [CrossRef]
15 . D. Stucki , G. Ribordy , A. Stefanov , H. Zbinden , J. G. Rarity , and T. Wall , “ Photon counting for quantumkey distribution with Peltier cooled InGaAs/InP APDs ,” J. Mod. Opt. 48 , 1967 – 1981 ( 2001 ). [CrossRef]
16 . P. A. Hiskett , G. S. Buller , A. Y. Loudon , J. M. Smith , I Gontijo , A. C. Walker , P. D. Townsend , and M. J. Robertson , “ Performance and design of InGaAs/InP photodiodes for single-photon counting at 1.55 um ,” Appl. Opt. 39 , 6818 – 6829 ( 2000 ). [CrossRef]
17 . L. Duraffourg , J.-M. Merolla , J.-P. Goedgebuer , N. Butterlin , and W. Rhods , “ Photon Counting in the 1540-nm Wavelength Region with a Germanium Avalanche Photodiode ,” IEEE J. Quantum Electron. 37 , 75 – 79 ( 2001 ). [CrossRef]
18 . J. G. Rarity , T. E. Wall , K. D. Ridley , P. C. M. Owens , and P. R. Tapster , “ Single-photon counting for the1300-1600-nm range by use of Peltier-cooled and passively quenched InGaAs avalanche photodiodes ,” Appl. Opt. 39 , 6746 – 6753 ( 2000 ). [CrossRef]
19 . M. G. Lippmann , “ Epreuves reversibles donnant la sensation du relief ,” J. Phys. (Paris) 7 , 821 – 825 ( 1908 ).
20 . H. E. Ives , “ Optical properties of a Lippmann lenticulated sheet ,” J. Opt. Soc. Am. 21 , 171 – 176 ( 1931 ). [CrossRef]
21 . T. Okoshi , “ Three-dimensional displays ,” Proceedings of the IEEE 68 , 548 – 564 ( 1980 ). [CrossRef]
22 . J.-S. Jang and B. Javidi , “ Time-multiplexed integral imaging for 3D sensing and display ,” Optics and Photonics News 15 , 36 – 43 ( 2004 ), http://www.osa-opn.org/abstract.cfm? URI=OPN-15-4-36.
23 . O. Matoba , E. Tajahuerce , and B. Javidi , “ Real-time three-dimensional object recognition with multiple perspectives imaging ,” Appl. Opt. 40 , 3318 – 3325 ( 2001 ). [CrossRef]
25 . S. Kishk and B. Javidi , “ Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging ,” Opt. Express 11 , 3528 – 3541 ( 2003 ), http://www.opticsexpress.org/abstract.cfm?URI=OPEX-11-26-3528. [CrossRef] [PubMed]
26 . S. Yeom and B. Javidi , “ Three-dimensional distortion tolerant object recognition using integral imaging ,” Opt. Express 12 , 5795 – 5809 ( 2004 ), http://www.opticsexpress.org/abstract.cfm?URI=OPEX-12-23-5795. [CrossRef] [PubMed]
27 . M. Martínez-Corral , B. Javidi , R. Martínez-Cuenca , and G. Saavedra , “ Multifacet structure of observedreconstructed integral images ,” J. Opt. Soc. Am. A 22 , 597 – 603 ( 2005 ). [CrossRef]
28 . H. Arimoto and B. Javidi , “ Integrate three-dimensional imaging with computed reconstruction ,” Opt. Lett. 26 , 157 – 159 ( 2001 ). [CrossRef]
29 . A. Stern and B. Javidi , “ Shannon number and information capacity of integral imaging ,” J. Opt. Soc. Am. A. 21 , 1602 – 1612 ( 2004 ). [CrossRef]
30 . J. W. Goodman , Statistical optics , ( John Wiley & Sons, Inc. , 1985 ), Chap 9.
32 . F. Sadjadi , ed., Milestones in performance evaluations of signal and image processing systems , (SPIE Press, 1993 ).
33 . A. Papoulis , Probability, random variables, and stochastic processes 3 rd , ( McGraw-Hill, Inc. 1991 ).