Real-time concealed-object detection and recognition with passive millimeter wave imaging

Seokwon Yeom; Dong-Su Lee; YuShin Jang; Mun-Kyo Lee; Sang-Won Jung

doi:10.1364/OE.20.009371

1. Introduction

Millimeter wave (MMW) imaging has been widely adopted in security and military applications since it can penetrate fabrics and clothing [1–5]. Moreover, passive MMW imaging can generate interpretable images even in low-visibility conditions such as fog, rain, dust, and smoke.

Unfortunately, the image quality in such applications is often degraded by the diffraction limit and low-signal-to-noise ratio (SNR) [6,7]. Furthermore, the system conventionally affords trade-offs between aperture size and spatial resolution, and temperature resolution is directly affected by integration time and bandwidth of the system.

There have been researches for the automatic detection of concealed objects [8–14]. Multi-level expectation-maximization (EM) has been proposed to cluster pixels according to the Gaussian mixture model (GMM) [11,12]. Object identification has been performed by means of the radiation characteristics of the target in close range [13]. Studies for concealed weapon identification and navigation were presented in [14]. Shape-based object matching and recognition can be found in [15].

In the paper, we propose a real-time concealed object detection and recognition with passive MMW imaging. Concealed object regions are extracted by the multi-level segmentation method [11,12]. A novel approach is proposed to measure similarity between two binary images. Principal component analysis (PCA) provides a convenient way to normalize the object in terms of translation and rotation [16]. A feature vector is extracted after PCA, which is invariant to scale and orientation, and tolerant to distortion. The feature vector is composed of several geometric descriptors. A decision rule classifies an unknown object into one of trained classes. The decision rule is based on Euclidean distance between feature vectors normalized. The passive MMW imaging system generates images at a frame rate of 1 Hz, thus the computational time is very limited in order to achieve real-time process.

In experiments, two different objects (gun and metal plate) are concealed under clothing of a moving human subject. They are detected and recognized by the proposed method. Our algorithms are implemented in both MATLAB and C++ environment on a standard PC. The performance is evaluated by the detection probability and the computational time.

The paper is organized as follows: In Sections 2 and 3, the passive MMW imaging system and the multilevel segmentation are described briefly. The concealed object recognition is presented in Section 4. In Section 5, the experimental and simulation results are illustrated, and the conclusion follows in Section 6.

2. Passive millimeter wave imaging system

The passive MMW imaging system operating in W band (around 94 GHz) consists of a high-density polyethylene lens with a diameter of 50 cm, reflective mirror, one-dimensional (1D) receiver array composed of 30 receiver channels, and a mechanical scanner, which moves the receiver array. The incoming W-band electromagnetic wave is focused on the feed antennas of a 1D receiver array located at the focal plane of the lens, thus images are visualized by mechanical scanning of the 1D receiver array [9]. The focal length and the field of view (FOV) of the lens are 500 mm and 17 × 17°, respectively, thus the 30-channel 1D receiver array corresponds to 30 pixels in the range of 17°. The lens of 500 mm-diameter satisfies the 0.57° spatial resolution. The 1D receiver is the direct detection total power type [9, 10]. Each receiver channel is composed of a dielectric rod antenna, four MMIC (monolithic microwave integrated circuit) low noise amplifiers, and a detector. The measured receiver gain is more than 45 dB in the desired frequency band (90-98 GHz). The detector is implemented by an unbiased Schottky barrier diode (SBD). The simulated return loss of the detector is less than −15 dB in the same frequency band [10]. There are three factors in determining the signal sensitivity: the receiver bandwidth, the noise figure of the amplifier, and the integration time. This passive MMW imaging system has 10 mrad spatial resolution and 1.5 K temperature resolution. The integration time of one channel is 10 msec. Figures 1(a) and 1(b) show the passive MMW imaging system and a block diagram of the system, respectively.

Fig. 1 (a) passive MMW imaging system, (b) a block diagram of the system.

Download Full Size | PDF

3. Concealed object segmentation

The multilevel segmentation comprises the global and local segmentation. The global segmentation separates the subject’s body area from the background area. During the local segmentation, only the inside of the body area is processed and the concealed object is segmented from the body area. Figure 2 shows the block diagram of concealed object segmentation.

Fig. 2 Block diagram of concealed object segmentation.

Download Full Size | PDF

This multilevel segmentation adopts vector quantization (VQ), EM algorithm for the GMM parameter estimation, and Bayesian decision rule. The VQ algorithm initializes the parameters of the GMM at each segmentation level. The EM process estimates the GMM parameters iteratively and the Bayesian decision rule decides which cluster each pixel belongs to. An alternative method is also developed for faster processing, which utilizes only the VQ during the global segmentation. Threshold values for EM are $P_{0} (G {}_{1}) = 0.999$ and $P_{0} (G {}_{2}) = 0.001$ , respectively. They are chosen heuristically when the best results are produced. The segmented areas less than 4 pixels are considered clutters and discarded in the experiments. More detailed procedures of the segmentation process are presented in [10,11].

4. Concealed object recognition

The recognition of the concealed object is composed of preprocessing for magnification, principal component analysis (PCA), size normalization, feature vector extraction, and a decision rule as illustrated in Fig. 3 . Magnification process reduces errors occurring when geometric shape features are extracted in the low resolution binary image. In this paper, we enlarge the binary image five times before applying PCA.

Fig. 3 Block diagram of the concealed object recognition.

Download Full Size | PDF

4.1 Principal component analysis (PCA) and size normalization

PCA and size normalization provide a convenient way to normalize the object region in terms of size, translation, and rotation. The pixels in the object area can be treated as two dimensional vectors, $x_{j} = {[x_{j}, y_{j}]}^{t}$ , where $x_{j}$ and $y_{j}$ are the coordinate values of any pixel along the $x$ - and $y$ -axis, respectively. All the pixels in the region constitute a 2-D vector population which can be used to compute the covariance matrix $Σ_{xx}$ and mean vector $m_{x}$ .

m_{x} = \frac{1}{n_{o}} \sum_{j = 1}^{n_{o}} x_{j}

Σ_{xx} = \frac{1}{n_{o}} \sum_{j = 1}^{n_{o}} (x_{j} - m_{x}) {(x_{j} - m_{x})}^{t}

where

n_{o}

is the number of pixels of the object, and superscript

t

denotes the transpose matrix. The principal components transform is calculated by the eigenvalue matrix (

Λ = [\begin{matrix} λ_{1} & 0 \\ 0 & λ_{2} \end{matrix}] \begin{matrix} , & λ_{1} > λ_{2} > 0 \end{matrix}

) and eigenvectors (

Ε = [μ_{1}, μ_{2}]

) of the covariance matrix.

Σ_{xx} Ε = Ε Λ

y_{j} = Ε^{t} (x_{j} - m_{x}) \begin{matrix} , & j = 1, ..., n_{o} \end{matrix}

The coordinates of $y_{j}$ are translated and rotated versions of the $x_{j}$ ’s, so that the eigenvectors align with axes centered at the origin. This transform gives the translation and rotation invariant property. One of eigenvalues ( $λ_{1}$ ) is normalized to $10^{4}$ by the resizing process for the size-invariant property. Figure 4 shows the PCA and the resizing processes of the binary object.

Fig. 4 (a) target image, (b) eigenvectors of the target, (c) transformed target by PCA, (d) size normalization.

Download Full Size | PDF

4.2 Feature vector extraction

A geometric feature is composed of several shape descriptors which are the object size, the major or minor-axis lengths, and the major and minor principal components, and size of each quadrant. Figure 5 shows the shape descriptors. The perimeter ( $T$ ) of the concealed object is marked with ‘□’ in Fig. 5(a). The area ( $A$ ) is calculated as the total number of pixels marked by ‘ × ’ in Fig. 5(a). The major ( $w$ ) and minor ( $h$ ) axis are the longer and the shorter side of a basic rectangle, respectively [16]. The proposed feature vector is composed of several descriptors as illustrated in Table 1 . The feature vector has six components ( $f_{1}$ ~ $f_{6}$ ). The feature components are based on shape descriptors and PCA in [16, 17]. Some of the components are contrived in order to classify several objects captured by a passive MMW imaging system [18].

Fig. 5 Geometric feature descriptors, (a) size number of pixels, lengths of major and minor axis, and the perimeter (number of boundary pixels), (b) size of each quadrant.

Download Full Size | PDF

Table 1. Feature Vector Components

View Table | View all tables in this article

4.3 Decision rule

Our decision rule is based on the Euclidean distance between feature vectors normalized. An unknown object in the image is classified into class $\hat{j}$ , which minimizes the Euclidean distance as

\hat{j} = \underset{j = 1, .., n_{c}}{\arg \min} ‖ \sum_{k = 1}^{c} {(\frac{o_{t e s t} (k)}{x_{m | j} (k)} - 1)}^{2} ‖,

where

o_{t e s t} (k)

is k-th component of the feature vector of the target image,

x_{m | j} (k)

is k-th component of the conditional mean vector of the

j

-th class, obtained from the training images,

c

is the number of components of the feature vector,

n_{c}

is the number of classes, and

‖ \cdot ‖

denotes the Euclidean norm. We define a probability of correct decision (

P_{D}

) for the evaluation of recognition results.

P_{D} = \frac{Number of decision for class j}{Number of frames capturing class j}

5. Experimental and simulation results

The passive MMW images of a human subject hiding a gun (Colt 45 – M1911A1) and metal plate are captured for 10 seconds with a 1 Hz frame rate. A human subject is captured at 5 m outdoors. Figures 6(a) and 6(b) are the pictures of the gun and metal plate, respectively. Figures 7(a) and 7(b) are the binary images for training, which describes Fig. 6(a) and 6(b), respectively. Figure 7(c) shows enlarged training images three, five, and seven times from the original image.

Fig. 6 Training target of real images, (a) gun (class 1, C1), (b) metal plate (class 2, C2).

Download Full Size | PDF

Fig. 7 Binary image of training target, (a) gun, (b) metal plate, (c) enlarged image by 3, 5, and 7 times of Fig. 7(a) and 7(b), respectively.

Download Full Size | PDF

Figure 8(a) and 8(d) are the passive MMW images capturing human subject hiding a gun and metal plate, respectively. Figure 8(b) and 8(e) are the results of concealed object detection by the standard procedure of the multi-level segmentation. Figure 8(c) and 8(f) are the feature extraction of the binary image representing concealed objects. The image quality depends on SNR and temperature sensitivity whereas the spatial resolution is an important factor to decide the size of the object to be recognized.

Fig. 8 Feature extraction of the target objects, gun of, (a) passive MMW images, (b) multi-level segmentation, (c) feature extraction, metal plate of, (d) passive MMW images, (e) multi-level segmentation, (f) feature extraction.

Download Full Size | PDF

Table 2 shows the feature vectors of the mean of the training targets and unknown target objects. Table 3 is the recognition results of the concealed object. All frames are correctly classified excepted for the 8-th and 9-th frames of the gun. This is caused by the shape of the object, which is more similar to the metal shape after the segmentation. Figure 9 shows the total computational time of the concealed objects recognition process which is implemented in both MATALB and C + + environments on a standard computer.

Table 2. Feature Vectors of the Target Object

View Table | View all tables in this article

Table 3. Probabilities of Correct Decision

View Table | View all tables in this article

Fig. 9 Computational time of the concealed object recognition.

Download Full Size | PDF

6. Conclusion

In this paper, we have addressed the detection and recognition of concealed objects. The imaging system generates frames every second by scanning a human subject in outdoor open space. Segmented binary images are analyzed by the geometrical feature vector, which are extracted from the target shape after the PCA transform. Further investigation on multiple objects captured from a flow of people remains for future studies.

Acknowledgments

This work was supported by the Samsung Thales and Mid-career Researcher Program through NRF grant funded by the MEST (No. 2010-0027695).

References and links

1. L. Yujiri, M. Shoucri, and P. Moffa, “Passive millimeter-wave imaging,” IEEE Microw. Mag. 4(3), 39–50 (2003). [CrossRef]

2. R. Appleby and R. N. Anderton, “Millimeter-wave and submillimeter-wave imaging for security and surveillance,” Proc. IEEE 95(8), 1683–1690 (2007). [CrossRef]

3. H.-M. Chen, S. Lee, R. M. Rao, M.-A. Slamani, and P. K. Varshney, “Imaging for concealed weapon detection: a tutorial overview of development in imaging sensors and processing,” IEEE Signal Process. Mag. 22(2), 52–61 (2005). [CrossRef]

4. National Research Council, Assessment of millimeter-wave and terahertz technology for detection and identification of concealed explosives and weapons (National Academies Press, Washington, D.C., 2007).

5. A. Denisov, A. Gorishnyak, S. Kuzmin, V. Miklashevich, V. Obolonskv, V. Radzikhovsky, B. Shevchuk, B. Yshenko, V. Uliz’ko, and J. Son, “Some experiments concerning resolution of 32 sensors passive 8 mm wave imaging system,” in Proceeding of the 20th International Symposium on Space Terahertz Technology, ISSTT 2009 (National Radio Astronomy Observatory, 2009), 227–229.

6. M. R. Fetterman, J. Grata, G. Jubic, W. L. Kiser Jr, and A. Visnansky, “Simulation, acquisition and analysis of passive millimeter-wave images in remote sensing applications,” Opt. Express 16(25), 20503–20515 (2008), http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-16-25-20503. [CrossRef] [PubMed]

7. K. B. Cooper, R. J. Dengler, N. Llombart, T. Bryllert, G. Chattopadhyay, I. Mehdi, and P. H. Siegel, “An approach for sub-second imaging of concealed objects using terahertz (THz) radar,” J. Infrared, Millimeter Terahertz Waves 30, 1297–1307 (2009).

8. H. Sato, K. Sawaya, K. Mizuno, J. Uemura, M. Takeda, J. Takahashi, K. Yamada, K. Morichika, T. Hasegawa, H. Hirai, H. Niikura, T. Matsuzaki, S. Kato, and J. Nakada, “Passive millimeter-wave imaging for security and safety applications,” Proc. SPIE 7672, 76720V (2010).

9. M.-K. Jung, Y.-S. Chang, S.-H. Kim, W.-G. Kim, and Y.-H. Kim, “Development of passive millimeter wave imaging system at W-band,” in Proceedings of the 34th International Conference on Infrared, Millimeter, and Terahertz Waves, (Busan, Korea, , 2009), pp. 1–2.

10. H. Lee, W. Kim, J. Seong, D. Kim, K. Na, M. Jung, Y. Chang, S. Kim, and Y. Kim, “Test model of millimeter-wave imaging radiometer equipment (MIRAE), presented at the International Symposium of Remote Sensing, Busan, South Korea Oct. 23-Nov. 2 2007.

11. S. Yeom, D.-S. Lee, J. Son, M. K. Jun, Y. Jang, S.-W. Jung, and S.-J. Lee, “Real-time outdoor concealed-object detection with passive millimeter wave imaging,” Opt. Express 19(3), 2530–2536 (2011). [CrossRef] [PubMed]

12. D.-S. Lee, S. Yeom, M.-K. Lee, S.-W. Jung, and Y. Chang, “Real-time computational processing and implementation for concealed object detection,” Opt. Eng. Computational Imaging Special Section, (to be published).

13. L. C. Li, J. Y. Yang, G. L. Cui, Z. M. Jiang, and X. Zheng, “Method of passive MMW image detection and identification for close target,” J. Infrared, Millimeter Terahertz Waves 32(1), 102–115 (2011). [CrossRef]

14. E. L. Jacobs and O. Furxhi, “Target identification and navigation performance modeling of a passive millimeter wave imager,” Appl. Opt. 49(19), E94–E105 (2010). [CrossRef] [PubMed]

15. S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object recognition using shape contexts,” IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–522 (2002). [CrossRef]

16. R. C. Gonzalez, Digital Image Processing 2/E (Prentice-Hall Inc., 2003).

17. I. Pitas, Digital image processing algorithms and applications (John Wiley & Sons, Inc., 2000).

18. S. Yeom and D. Lee, “Radiometer automatic detection and image fusion algorithm development,” Samsung Thales Final Report, 2011.

$f_{1}$	$\frac{\sqrt{λ_{1}} + \sqrt{λ_{2}}}{w + h}$
$f_{2}$	$\frac{\sqrt{λ_{1}} \sqrt{λ_{2}}}{w \times h}$
$f_{3}$	$\begin{array}{l} \begin{matrix} \frac{Q_{3} + Q_{4}}{Q_{1} + Q_{2}} & \begin{matrix} i f & Q_{1} + Q_{2} > Q_{3} + Q_{4} \end{matrix} \end{matrix} \\ \begin{matrix} \frac{Q_{1} + Q_{2}}{Q_{3} + Q_{4}} & o t h e r w i s e \end{matrix} \end{array}$
$f_{4}$	$\begin{array}{l} \begin{matrix} \frac{Q_{2} + Q_{3}}{Q_{1} + Q_{4}} & \begin{matrix} i f & Q_{1} + Q_{4} > Q_{2} + Q_{3} \end{matrix} \end{matrix} \\ \begin{matrix} \frac{Q_{1} + Q_{4}}{Q_{2} + Q_{3}} & o t h e r w i s e \end{matrix} \end{array}$
$f_{5}$	$\frac{λ_{1} + λ_{2}}{A}$
$f_{6}$	$\frac{\sqrt{λ_{1} λ_{2}}}{A}$

	$x_{m \| 1}$ *	$o_{1}$	$o_{2}$	$o_{3}$	$o_{4}$	$o_{5}$	$o_{6}$	$o_{7}$	$o_{8}$	$o_{9}$	$o_{10}$
$f_{1}$	0.248	0.248	0.257	0.251	0.252	0.257	0.256	0.242	0.235	0.232	0.250
$f_{2}$	0.052	0.055	0.051	0.044	0.052	0.053	0.052	0.052	0.053	0.052	0.053
$f_{3}$	0.964	0.986	0.983	0.964	0.947	0.942	0.925	0.992	0.948	0.952	0.999
$f_{4}$	0.789	0.749	0.834	0.827	0.777	0.781	0.831	0.730	0.854	0.812	0.696
$f_{5}$	0.460	0.416	0.586	0.598	0.520	0.538	0.554	0.397	0.212	0.258	0.519
$f_{6}$	0.010	0.011	0.010	0.010	0.010	0.011	0.010	0.010	0.007	0.008	0.012

	$x_{m \| 2}$ **	$o_{1}$	$o_{2}$	$o_{3}$	$o_{4}$	$o_{5}$	$o_{6}$	$o_{7}$	$o_{8}$	$o_{9}$	$o_{10}$
$f_{1}$	0.251	0.255	0.248	0.248	0.248	0.254	0.253	0.253	0.247	0.250	0.252
$f_{2}$	0.057	0.058	0.052	0.057	0.053	0.055	0.058	0.065	0.056	0.059	0.060
$f_{3}$	0.940	0.934	0.917	0.946	0.918	0.898	0.970	0.988	0.953	0.928	0.945
$f_{4}$	0.975	0.960	0.972	0.949	0.975	0.992	0.952	0.988	0.999	0.992	0.973
$f_{5}$	0.370	0.408	0.364	0.383	0.443	0.477	0.418	0.331	0.268	0.296	0.307
$f_{6}$	0.007	0.007	0.007	0.007	0.007	0.007	0.007	0.007	0.007	0.007	0.007

$P_{D}$ (gun)	$P_{D}$ (metal plate)
0.8	1.0

$f_{1}$	$\frac{\sqrt{λ_{1}} + \sqrt{λ_{2}}}{w + h}$
$f_{2}$	$\frac{\sqrt{λ_{1}} \sqrt{λ_{2}}}{w \times h}$
$f_{3}$	$\begin{array}{l} \begin{matrix} \frac{Q_{3} + Q_{4}}{Q_{1} + Q_{2}} & \begin{matrix} i f & Q_{1} + Q_{2} > Q_{3} + Q_{4} \end{matrix} \end{matrix} \\ \begin{matrix} \frac{Q_{1} + Q_{2}}{Q_{3} + Q_{4}} & o t h e r w i s e \end{matrix} \end{array}$
$f_{4}$	$\begin{array}{l} \begin{matrix} \frac{Q_{2} + Q_{3}}{Q_{1} + Q_{4}} & \begin{matrix} i f & Q_{1} + Q_{4} > Q_{2} + Q_{3} \end{matrix} \end{matrix} \\ \begin{matrix} \frac{Q_{1} + Q_{4}}{Q_{2} + Q_{3}} & o t h e r w i s e \end{matrix} \end{array}$
$f_{5}$	$\frac{λ_{1} + λ_{2}}{A}$
$f_{6}$	$\frac{\sqrt{λ_{1} λ_{2}}}{A}$

Real-time concealed-object detection and recognition with passive millimeter wave imaging

Abstract

1. Introduction

2. Passive millimeter wave imaging system

3. Concealed object segmentation

4. Concealed object recognition

4.1 Principal component analysis (PCA) and size normalization

4.2 Feature vector extraction

4.3 Decision rule

5. Experimental and simulation results

6. Conclusion

Acknowledgments

References and links

Cited By

Figures (9)

Tables (3)

Equations (6)

Optics Express