1. Introduction

The human face plays a very important role in our social interaction and in conveying a person’s identity. Since Kanade developed the first automated face recognition system (see box AFR) system in 1973 [1], face recognition technologies have received more and more attention due to their wide variety of civilian and military applications.

Over the years, numerous computer-based vision methods have been applied to develop facial recognition or tracking algorithms with high discrimination and robustness [2], e.g., geometric feature-based methods, eigenface (sub-space)-based methods [3,4], neural networks, the support vector machine (SVM), and correlation-based methods [5]. Yet there are still many technical challenges to overcome. For example, face images of the same subject may appear very different due to changes of the expression, illumination conditions, head orientation, and digital post-processing modules [6]. Unexpected noise interference and imposters make it more difficult for a recognition system to make a correct decision. To construct a reliable discrimination algorithm that can cope with all of these changes, almost all existing numerical schemes involve complicated and time-consuming computations [7].

Ever since the first use of an optical correlator for implementing a matched spatial filter [8,9], correlation methods have attracted extensive interest among researchers in the face recognition field because of the following advantages: (1) inherent parallelism, (2) shift invariance, (3) desired noise robustness, and (4) high ability for discrimination [6]. Many correlation filters (CFs) can be expressed in closed forms (see Appendix A) in the frequency domain. Based on these closed form expressions, many optoelectronic hybrid implementation schemes of CFs have been suggested. To obtain improvement in discrimination, the design procedures of CF often involve optimization of some performance metrics such as the average correlation energy (ACE) and correlation output noise variance (ONV). During the recognition stage, CF methods generally involve multiplying the test image spectrum $T$ (2D Fourier transform spectrum of the image to be recognized) with a CF $H$ [generated from a single or a set of training (reference) images whose identity or class are known] and then performing an inverse Fourier transformation (FT). The correlation output is given as

In the ideal case, one can obtain an apparent correlation peak with high PSR (or PCE, exceeding a threshold value) when the test image of $k$th class is correlated with a CF synthesized from a training image that also belongs to the same class. In the case of real-time face recognition, the receiver operating characteristic curve (ROC) is also often used to measure the performance of correlation algorithms (see box ROC).

Robust, efficient, and discriminating face recognition schemes based on correlation algorithms have been reported in the past decade [5,13]. Some of them have been implemented successfully and used in practice. Much work has been done to combine conception of optical correlation with other spatial-domain recognition methods, aiming to achieve new face recognition algorithms with higher discrimination and robustness characteristics. Rapid development of powerful optoelectronic interfaces, such as, charge-coupled devices (CCDs) [14] and spatial light modulators (SLMs) [15], further accelerates the integration of optical correlation and computer vision methods in this field.

 

Figure 1 ROC curve.

Download Full Size | PDF

In this tutorial review, we are interested in the design of face recognition algorithms using correlation theory, and their implementation and potential application. We hope that this study will prove useful in helping the reader get a broad overview of CF-based face recognition methods, especially the most recent progress in this area. Before discussing typical correlation-based face recognition algorithms, we will first present a brief historical background to help the reader appreciate the importance of this research. Our main goal is to pay more attention to the correlation techniques that have been exploited for face recognition.

As mentioned earlier, the correlation technique can be considered a filtering that extracts relevant information for pattern recognition in a complex 2D scene. The earliest CF is the well-known matched filter (MF; see box MF), which is fabricated from a single reference image [8]. MF can be implemented optically with a Vander Lugt 4-$f$ setup (VLC) [9]. The VLC 4-$f$ setup consists of two cascaded Fourier transform structures carried out by a lens, as shown in Fig. 2. In the input plane, the test image $O$ is first Fourier transformed by illuminating it with a collimated coherent beam. The Fourier spectrum $S_O$ is generated at the rear focal plane of the lens where a MF, $H_{\mathrm{MF}}$, is inserted to filter the resulted spectrum. By performing a second FT with another lens, the correlation result is recorded on the correlation plane (the rear focal plane of the second lens). An alternative implementation structure of a MF is a joint transform correlator (JTC) [16–18], which can avoid the precise alignment of optical elements and holographic recording of a complex filter, but spatial size of the input is seriously confined and much of the energy is consumed by the extra autocorrelation signal.

 

Figure 2 Principle of the VLC setup.

Download Full Size | PDF

To improve the light efficiency of a MF, Horner and Gianino proposed a phase-only matched filter (POF; see box POF). Compared with a MF, the POF can produce sharp correlation peaks with improved discrimination capability [19].

Binary processing of a POF leads to another filter, a binary POF (BPOF), which can effectively reduce the accommodation burden of the SLM [20]. Since then, numerous nonlinear numerical methods have been applied to modify the robustness and discrimination of the original MF, resulting in wavelet MF [21], joint wavelet transform correlator [22], morphological correlation [23], adaptive JTC [24], fringe-adjusted JTC [17], and so on. Furthermore, in order to realize in-plane rotation invariance and scale invariance of correlation peaks, researchers have devised a circular-harmonic filter (CHF) [25] and radial-harmonic filters (RHF) [26] by extracting circular harmonic components (CHCs) and radial harmonic components (RHCs) from the training images.

A single MF is vulnerable when a wide variety of face samples of the same subject are considered for recognition [6,14], because it contains feature information of only a single face sample. In the ideal case, a face recognition system needs to exhibit distortion invariance to intra-class changes while maintaining sufficient discrimination capability to reject imposter classes. There are several approaches proposed in the literature aiming at handling this problem by the design of CFs. They generally fall into two categories. The first approach is to prepare a large number of MFs corresponding to all kinds of changes of each subject beforehand. The advantage of this solution is that it does not involve complicated training of filters. However, in order to use this solution to carry out high-speed recognition tasks with low error rates, one needs a large-volume digital memory device, such as a hard disk drive (HDD) or holographic optical memory system, to store these filters while ensuring high transfer and processing speed. If the shape or appearance of a target object is relatively simple, or the captured conditions of test images are given with many rigorous confinements, such recognition systems sometimes can also work very well, such as the high-speed holographic correlation system for face recognition in [27], and the road sign recognition system proposed in [28].

In the more complicated cases, this solution is clearly impractical because we cannot prepare a MF (or other filters generated from single facial image of one subject) for each combination of variance. To address the issue, the second approach based on composite filters (also called advanced CFs) should be used. In many applications, the first solution and the composite filters are often integrated together by using different performance optimization strategies [29]. A composite filter is a single filter that carries feature information extracted from two or more training images [6]. Development of composite filters can be classified into two different categories: (i) linear composite filters and (ii) nonlinear composite filters. Linear composite filters can be expressed as linear summation of multiple training spectra (i.e., Fourier spectra of training images) in the frequency domain. This class of filters can be further categorized into two classes: linear constrained filters and linear unconstrained filters. The constrained filters are designed to satisfy the hard constraints that specify the peak values for the training images. The earliest composite filter, the equal-correlation-peak synthetic discrimination function (ECP-SDF) filter, is a typical linear constrained filter [30] where the correlation peak intensities produced by authentic training images are set to be “1” while the peaks for imposter training images are set to be “0”. To help resolve the wide sidelobe problem of ECP-SDF, Mahalanobis et al. proposed minimum average correlation energy (MACE) filters that can produce sharp correlation peaks at the origin [31]. Different from the isolated point (desired peak values) constraints of ECP-SDF, MACE is an attempt to control the entire correlation plane where output values are suppressed at all points except at the origin center. To achieve this, MACE is designed to minimize average correlation energy (ACE) resulting from the training images (see box) while constraining the correlation peaks to predefined values. Because ECP-SDF and MACE were not designed to tolerate input noise, Kumar proposed the minimum variance SDF (MVSDF), which is capable of minimizing correlation output noise variance (ONV) (see box) while satisfying hard constraints of peak values [32]. To achieve a trade-off between noise tolerance and peak sharpness, Refregier presented the optimal trade-off SDF (OTSDF), which elegantly integrates the design ideas of MACE and MVSDF by introducing an adjustable trade-off factor [33]. Ravichandran and Casasent introduced another trade-off filter, i.e., the minimum noise and correlation energy (MINACE) filter, where an envelope equal to or larger than the noise is employed at each frequency of the power spectra of the training images [34].

A drawback of linear constrained filters is that their correlation peaks cannot take on the specified values for the non-training images. Moreover, studies have shown that hard constraints on the correlation peak outputs can be detrimental [5,6]. As a result, a wide variety of linear unconstrained filters were achieved by relaxing or removing the abovementioned hard constraints. One example is the maximum average correlation height (MACH) filter where the average correlation height (ACH) (see box) produced by true-class training images is maximized without any hard constraints required on the correlation peaks [35]. In addition to retaining large correlation peaks, MACH achieves distortion tolerance by minimizing the average similarity measure (ASM) (see box) yielded by the true class training image. By making an appropriate approximate fabrication of MACH, one can get the unconstrained MACE (UMACE) filter [35]. Although hard constraints are replaced with softer requirements in UMACE filters, in Kumar’s face recognition tests, they exhibited discrimination ability and distortion invariance comparable to those of MACE filters [11]. Optimal trade-off approaches are introduced by relating different correlation metrics (ACE, ONV, ASM, and so on), which results in unconstrained OTSDF (UOTSDF) [36] and optimal trade-off MACH (OTMACH) [37,38]. In order to reduce the dependence on the average of training images, two trade-off variations of MACH filters, termed extended MACH (EMACH) [39] and eigen-extended MACH (EEMACH) [40], were also introduced.

An important advantage of the linear composite filters is that they can be computed efficiently in the frequency domain. In addition to the linear composite filter, some nonlinear composite filters that retain the computation efficiency of linear methods in the frequency domain were also suggested. Many filter designs are based on the judicious combination of linear composite filters and nonlinear optimization procedures, which include correntropy MACE (CMACE) [41], action MACH (AMACH) [42], wavelet modified MACH (WMMACH) [43], phase-only UMACE (POUMACE) [44], quad phase UMACE (QPUMACE) [45], local binary pattern UMACE (LBP-UMACE) [46], and generalized MACH (GMACH) [47]. In two contrasting approaches, researchers designed the quadratic CFs (QCFs) [48,49] and the polynomial CFs (PCFs) [50,51]. The design of QCFs is based on the fabrication of a quadratic discriminating function that can be decomposed into a set of linear correlators. However, in PCFs, a set of linear correlators are first applied to process input images, and then the correlation outputs are summed together. Other important nonlinear optimization approaches of composite filters include the segmented composite filter (SCF) [52], adaptive robust CF (ARCF) [53], minimum output sum of squared error (MOSSE) filter [54], and average exact synthetic function (AESF) filter [55].

In addition to the design of CFs mentioned earlier, several researchers found other ways to achieve high efficiency, distortion invariance, and discrimination ability by combining CFs with other pattern recognition methods. The combination of CFs and distance classifiers results in the distance-classifier CF (DCCF) [56], where the distance between the obtained correlation array and a prototype correlation array is calculated to make the class decision. By increasing the separation between true-class correlation outputs and false-class correlation outputs, a modified correlation-based classifier, termed the minimum distance transform CF (MDTCF) [57], achieves better discrimination than the previous DCCF. Alkanhal and Kumar devised the polynomial DCCF (PDCCF) by combining the DCCF and polynomial CFs (PCFs) for human face recognition [58]. Another class of composite filters is obtained by applying subspace-based methods. For example, Alfalou and Brosseau proposed independent component analysis (ICA)-based composite POF (ICA-CPOF) [59], where ICA (see box ICA) is employed to optimize the representation of training images. In order to achieve illumination invariance, Datta et al. proposed class-specific subspace-dependent nonlinear CF where class-specific subspace analysis is carried out for formulation of CFs [5]. Typical cases based on the combination of a SVM with CFs include linear shift-invariant maximum margin SVM CF (LMM-SVM-CF) [60] and maximum margin CF (MMCF) [61].

In what follows, we will present some important specially designed CFs for face recognition application. The MACE, UMACE, and their extensions were among the earliest CFs used for face recognition. In [11], the authors used the MACE and UMACE to study the applicability of composite filters for face verification. Composite CFs can offer very good matching performance in the presence of variance in the facial images, e.g., face expressions and illumination changes. In another study, Wijaya et al. found that the optimal trade-off SDF (OTSDF) and MACE can perform illumination-tolerant face verification of compressed target images at low bit rates by using the JPEG2000 wavelet compression model, but the OTSDF was found to be more suitable for this task due to its built-in noise robustness [63]. In spite of this, one can greatly improve the discrimination performance of the MACE by making use of the logarithmic transformation on the training images [63,64]. In [65], a comparative study of the MACE and individual eigenface subspace method for face recognition in terms of margin of separations was proposed. Additionally, [66] described a method to reduce the complexity of the MACE design while the localization and discrimination performances are retained. In [67], a principle component analysis (PCA; see box PCA) is added to preprocess the phase spectrum of the training image because the obtained subspace can represent a larger set of changes, which is helpful to further enhance the PSR of the authentic facial class. Face verification based on the LBP-UMACE can be found in [46], where the local binary pattern (LBP) is operated on the training images of UMACE, aiming at enhancing the recognition rate and reducing the error rate. Simulation tests demonstrate that the LBP-UMACE produces better performance than the traditional UMACE. Correntropy MACE (CMACE) is another nonlinear variance of MACE used for face recognition applications [41]. The CMACE equations are formulated in a reproducing kernel Hilbert space (RKHS) induced by correntropy. A correntropy function can generalize the concept of correlation by introducing second- and higher-order moments of signal statistics, which is helpful to enhance the discrimination of CMACE. In experiments using pose illumination and expression (PIE) dataset of human faces, the CMACE outperforms the conventional MACE in both distortion invariance and rejection capabilities. Illumination invariant face recognition with the MINACE filter is presented in [68], where two forms of MINACE filters and two decisional criteria, i.e., peak intensity and PCE ratio, are combined for simulation tests on the PIE database. Satisfying performance criteria were achieved for both face identification and verification. In [69], several typical composite filters, including the MACE, UMACE, POUMACE, DCCF, and MDTCF, were compared with other traditional learning methods for several face and head pose databases. The experimental results show that the correlation-based classifiers outperform the traditional appearance-based methods (for example, PCA) in robustness and accuracy. The POUMACE filter provided the best performance with 100% accuracy for the CMU facial expression database and the Yale frontal face illumination database. In other comparison studies, polynomial DCCF (PDCCF) also provided a better discrimination capability than PCA and the numerical LBP method [58].

Thornton et al. found that application of SVM during the design of the correlation algorithm can enhance robustness against white additive noise and further optimize the relationship between peak and sidelobe in the correlation plane of training faces. Their work resulted in the proposal of LMM-SVM-CF [61]. Another successful combination of SVM and CF is the Rodriguez et al.’s MMCF [62], where the localization capability of CF and the generalization capability of SVM [71,72] are retained simultaneously. The face class code (FCC) method based on SVM and CF was also proposed in [73]. This method can perform face recognition when the class number is large. Good recognition results can be achieved under different illumination conditions.

A facial feature extraction method using kernel CF and cosine distance criterion was reported in [74,75], where satisfying experimental results were obtained when the FGRC2.0 dataset was used. In [76], kernel correlation feature analysis (KCFA) exhibits good representation capability for unseen datasets and better verification and identification properties using the PIE and AR facial datasets. In recent papers, some face trackers based on the kernel CF have been presented with better accuracy and distortion-tolerance capability obtained for dynamic tracking [77–79]. In another study, a 2D correlation feature analysis is extended to the high-order tensor cases. This tensor-based extension obtains better recognition rates using standard face datasets than previous 2D analysis [80].

In conventional composite filter design, increasing the number of training images leads to local saturation phenomenon [52]. To help resolve this problem, Alfalou and co-workers proposed a SCF, where the CF is segmented into several independent sections, and each section is assigned to a single training spectrum [52,81]. In [82], a comparison study of the SCF and ICA for human face recognition is proposed. Simulation results show that a better classification capability is obtained with SCF using the Pointing Head Pose Image Database (PHPID). Actually, a judicious combination of the ICA and CF can also be employed to optimize the decision performance. In [59], an ICA-based composite POF can perform face classification tasks with an impressive low false recognition rate.

The QCF for face recognition was proposed in [83], where the quaternion data is extracted from the wave decomposition of a face image for filter training. The proposed filter achieves a significant improvement in recognition compared with conventional advanced correlators for handling illumination-invariant recognition using the CMU PIE database. The representation of face image features using the structure of quaternion numbers alleviates the negative illumination disturbance on face images in an effective way. In a subsequent work, the QCF was extended into two variants: the phase-only QCF and separable trade-off QCF [84]. These filters need only one training facial image of a subject while retaining illumination-invariant performance.

By introducing zero-aliasing constraints into the training algorithms of many existing composite filters, Fernandez and co-workers proposed zero-aliasing CFs (ZACFs) in order to reduce the wide sidelobe caused by circular correlations [85,86]. Bolme et al.’s average of synthetic exact filter (ASEF) is another innovative approach to optimize correlation algorithms [55]. ASEFs can be contrasted with composite filters, which pose a limited number of constraints on the correlation output because ASEFs specify the entire correlation plane of training images. ASEFs do not involve complicated computations and can be trained using larger training sets because the over-fitting effect [55] is alleviated by average computations. In addition to the optimization of correlation algorithms, the decision optimization can also improve the discrimination capability of a recognition system. In [87], the linear functional model (LFM) [88] and singular value decomposition (SVD) [89] were employed to denoise the correlation plane, aiming at enhancing the recognition accuracy of correlators. Numerical tests demonstrated that this strategy is compatible with SCF.

Next, considering the close relationship between pattern recognition and visual tracking, we provide a brief discussion of several CFs used for face tracking. Similar to pattern recognition in a static scene, a tracking algorithm needs to overcome various critical issues associated with robustness, such as deformation, occlusion, illumination variation, and rotation. Even more critical is that visual tracking requires filters to be trained from a single frame and dynamically adapted to the changes of the target appearance. Although many conventional composite filters (even non-composite filters such as the fringe-adjusted JTC) can effectively perform localization tasks required by visual tracking, their training needs make them unsuitable for online tracking. Bolme et al.’s MOSSE filter thoroughly changed this situation [54]. The MOSSE filter is an ASEF-like filter that needs very few training images for initialization. The filter can quickly adapt to appearance changes by running a simple average computation. MOSSE filters were often used for face recognition and comparison purposes [62,86]. Motivated by the design of MOSSE filters, several researchers have suggested numerous CF-based tracking algorithms [77–79,90]. Compared with conventional trackers based on an appearance model, CF-based methods demonstrate better performance in efficiency and robustness. The interested reader may wish to consult Chen et al.’s study [91] for more details.

Despite the fact that CF approaches have achieved significant results for face recognition, detection, and tracking in the past decade, we must point out that these techniques encounter some recurrent problems that need to be addressed for optimizing their potentiality. Although many of these techniques can provide satisfying discrimination performance in the laboratory or with using benchmark datasets, they often have difficulties to perform well when some unexpected factors induce variations of facial appearance, i.e., intrapersonal, interpersonal, and extrinsic factors [5]. Intrapersonal factors include age, facial expression, and facial paraphernalia, such as facial hair, glasses, and cosmetics, that vary the facial appearance of a given subject. Interpersonal factors are responsible for the difference in the facial appearance of different persons, such as gender and ethnicity. Extrinsic factors include pose, illumination, scale, and some technical parameters such as focus and resolution, which vary the facial appearance through the interaction of light with the observer and the face. The facial appearance caused by these factors can potentially deteriorate the performance of an automated face recognition real-time system. Several key factors that may jeopardize practical application of CFs are the following. (1) Many CF methods can handle recognition tasks only under moderate illumination variations. Performance of these methods decreases noticeably when these variations become very large or when the illumination is not uniform. Although some correlation algorithms exhibit better illumination invariance [11,46,73,83], the experiments are still confined in very few standard datasets. (2) Projection deformations and self-occlusion caused by pose changes (e.g., 3D head rotation) can degrade the recognition performance of CFs. (3) Occlusion (often introduced by sunglasses, caps, veils, and other persons in the same scenario) on the upper side of the face can dramatically influence the identification process of CFs due to the loss of facial features such as eyes and nose. (4) Low image resolution and poor focus quality are also important factors that result in false recognition because of the loss of specific features. (5) Sometimes, extreme changes in facial expression may also result in recognition failure.

Obviously, it is impossible to resolve all the above-mentioned problems by using only one composite CF. This would require an excessive number of training face images to generalize all of these uncontrolled variances for classifying an array representing a test face in high dimensions. Moreover, increasing the number of training images may also seriously deteriorate the recognition performance of CFs because of the saturation effect. Schemes based on multiple composite filters can alleviate the problems to a certain extent. However, this may pose rigorous technique requirements on the computation capacity and speed of the hardware. Moreover, some practical recognition tasks are perhaps very difficult to be resolved by using only existing CF methods. For example, the changes in the human face over a period time due to aging are not trivial [5].

Based on the above considerations, we believe that devising new methods by judicially combining CFs and other face detection and recognition methods together is more effective and practical for resolving the deficiencies of the existing CFs. Due to highly complex exterior environmental interference and the varied nature of facial appearance it is unlikely that all face detection and recognition tasks can be resolved by using a single strategy. In past reviews, we demonstrated many examples for which CF and other techniques can be combined for performance optimization [41–51,56–59,61,62,67,68,71,72,82,87]. Here, we further supplement several possible approaches:

The remainder of this paper is arranged as follows. Section 2 presents some typical correlation algorithms applied to face recognition. In Section 3, we discuss some recently implemented schemes of correlation techniques. Although some of these applications are not strictly devoted to face recognition, they still demonstrate the potential of CF-based techniques to be compatible with standard optical or electronic hardware devices.

2. Correlation-Based Face Recognition Methods

In this section, we first present some classical composite correlation algorithms applied for face recognition, namely, the MACE, UMACE (Subsection 2.1 [11,31]), and OTSDF (Subsection 2.2 [37,63]). To the best of our knowledge, these are the earliest CFs employed to tackle basic issues involved in face recognition, such as robustness under different conditions (including facial expression changes, off-plane rotation, illumination variation, noisy interference, and image compression), optimization of discrimination ability, and filtering adaptability. These pioneering works gave a solid theoretical basis for application of CF methods in this area. We also pay special attention to recently proposed algorithms, such as ASEF (Subsection 2.3, [55]), MMCF (Subsection 2.6, [62]), LBP-based UMACE (Subsection 2.7, [46]), ICA-based POF (Subsection 2.8, [59]), and ZACF (Subsection 2.9, [85,86]). The simple ASEF design is described in Subsection 2.3. MOSSE [17], a milestone work for correlation-based tracking, is introduced in Subsection 2.4. Another visual tracking method using CNN-learned features in the CF framework is discussed in Subsection 2.5 [100]. Face recognition schemes described in Subsections 2.6 to 2.8 demonstrate the flexibility of correlation technique design for a wide variety of applications. Subsection 2.9 reports another approach of algorithm optimization by correcting a commonly neglected fault in the design of conventional composite filters. In Subsection 2.10, we will present a decision optimization scheme whose design idea is very different from the above filter optimization algorithms [20]. Finally, in Subsection 2.11, a recognition based on class-specific subspace analysis is introduced to realize illumination-invariant recognition (i.e., Chap. 11 of [5]). Because of space limitations, several correlation algorithms, such as the QCF [83,84], kernel CFs [74–79], DCCFs [56–58], and correntropy MACE [41], cannot be discussed in this section. Interested readers may reference the related literature.

2.1. Face Verification with MACE Filters

The MACE filter ([31], see Appendix A.2, item 2) is aimed at minimizing the ACE generated from the training images, and constraining the correlation peak intensities with specified values. In contrast to ECP-SDF [30], which controls only one isolated point of the correlation plane, to obtain a MACE filter, one needs to resolve the ACE minimization problem subject to the constraints of correlation peak values. In several studies, well-designed MACE filters exhibit impressive discrimination with sharp peaks in response to the training images in the authenticated class, as well as small output values in response to the imposter classes. In spite of this, two drawbacks of conventional MACE filters have been found. One is the lack of built-in noise tolerance. Another is that MACE exhibits more sensitivity to intraclass variation than the other types of composite filters [6].

In recent years, verification based on biometric characteristics (face, finger, and eye iris) has been regarded as an effective alternative to the conventional authentication systems using password and/or personal identification because (i) these characteristics are physically attached to the authentic subjects and cannot be lost or stolen, and (ii) they differ from one person to another one. Motivated by the excellent discrimination ability of MACE, Kumar et al. tried to use it for face verification under different facial expressions and illumination conditions [11]. In many methods, the recognition decision is based on the observation of a correlation peak that is strongly related to the brightness level of the input image. To avoid the dependence on brightness and to base the recognition decision on a larger region of correlation plane, the PSR can be used as a decision criterion. The datasets used for simulation tests were provided by the Advanced Multimedia Processing Laboratory in the Department of Electrical and Computer Engineering at Carnegie Mellon University. Facial images of 13 subjects are collected in this dataset. For each subject, 75 images ($64\times 64$ in size) with different facial expressions are recorded. Figure 3 shows example images. In [11], each subject’s MACE filter was synthesized from three training images (images 1, 21, and 41 out of the 75 images). To test the discrimination performance of these MACE filters, the cross correlations of each subject’s filter with every image of the dataset (75 true-class images and 900 false-class images) were calculated.

In Figs. 4 and 5, we show the best PSR values of the MACE filter (for subject 1) and the worst PSR values (for subject 2), respectively. The horizontal axis represents the image index. The solid curves show the PSRs produced from the authentic faces, and the dotted curves correspond to the imposter faces. For subject 1, one can find a distinct separation between the solid and the dotted curves, which means a perfect discrimination between the authenticated person and the imposters. Although the results for subject 2 show the smallest margin between the true and false classes, it was concluded that the MACE filter for subject 2 still yields a 99.1% verification performance [11].

 

Figure 3 Typical images for one subject. Reprinted with permission from [11]. Copyright 2004 Optical Society of America.

Download Full Size | PDF

 

Figure 4 PSR values generated from the MACE filter fabricated with subject 1. Reprinted with permission [11]. Copyright 2004 Optical Society of America.

Download Full Size | PDF

For the 13 MACE filters, each generated from three training images (images 1, 21, and 41) of one subject, an average EER of 0.15% was obtained [11]. As a comparison, if an individual eigenface subspace method [107] is devised with the same three training images for comparison tests using the same database, the obtained EER takes a larger value of 0.85%. When the number of training images for each subject’s MACE filter is increased to 25, a 100% verification accuracy in the facial expression database is obtained [11].

The test and training images used for illumination tests in [11] were taken from the illumination subset of the Carnegie Mellon University’s PIE dataset, which contains 65 subjects with 21 different illumination conditions. Figure 6 displays 21 images of subject 2. A single MACE filter and a single UMACE filter (see Appendix A) were fabricated for each person by using three training images under extreme lighting variations (image 3, image 7, and image 16). In Fig. 7, we show the PSR results when the MACE and UMACE filters of subject 2 were cross correlated with the whole database. From Fig. 7, one can find that the PSR for the authenticated class is larger than those obtained for the imposters. These results demonstrate that by using three linearly independent training images that generalize the extreme lighting variations, one can obtain high PSR values from any face image that lies in the convex envelope of these training images. Moreover, there is no pronounced difference between the PSR values generated from the MACE and UMACE filters. The same simulation tests were repeated with another combination of training images (7, 10, and 19 with frontal facial lighting). A verification accuracy of 93.5% at a FAR of 0% was reported in [11]. The built-in tolerance of MACE filters to illumination variations is mainly due to the emphasis of these filters on high spatial frequencies. As a result, low spatial frequencies, which are affected mostly by lighting changes, are attenuated by MACEs.

 

Figure 5 PSR values generated from the MACE filter fabricated with subject 2. Reprinted with permission from [11]. Copyright 2004 Optical Society of America.

Download Full Size | PDF

 

Figure 6 Images for one subject obtained for the illumination variation tests. Reprinted with permission from [11]. Copyright 2004 Optical Society of America.

Download Full Size | PDF

Overall, Kumar et al.’s report can be considered a milestone in the design of CFs. Since this seminal work, much research has been done in designing and applying advanced CFs for biometric verification.

2.2. OTSDF for Face Verification in the Presence of Illumination Variance and Image Compression

The OTSDF [37] is actually designed to optimally trade-off ONV and ACE. As mentioned earlier, in order that a MACE filter can achieve sharp correlation peaks with high discrimination, high frequencies are emphasized by minimizing the ACE. In contrast, low frequencies are emphasized by minimizing the ONV so that MVSDF [32] (see Appendix A.2, item 3) can obtain optimal noise robustness. To maintain the balance between noise robustness and high discrimination, one must resolve an optimal trade-off problem of the two criteria, ONV and ACE, subject to the peak constraints. An feature adjust factor is introduced in the fabrication of the OTSDF (see Appendix A.2, item 4) to meet specific application use [37]. The MACE filter is actually a special case of the OTSDF with $\alpha =1$. If $\alpha =0$, the OTSDF simplifies to the maximally noise tolerant filter, which is MVSDF.

In many recognition applications, the situation is often encountered in which the training images are uncompressed, but authentication is performed with mobile devices that transmit compressed images at low bit rates. To identify how well the MACE filters and the OTSDFs perform for compressed images at different bit rates, Wijaya et al. carried out a series of simulation tests [63]. The CMU PIE dataset used in [63] consists of 65 subjects captured with 21 different illumination conditions. Figure 8 shows an example of gray images ($100\times 100$ pixels) of subject 2 in the dataset. These high-quality uncompressed images do not have background lighting. In [63], two sets of training images were used to design a MACE filter and an OTSDF for each of the 65 persons. Training set 1 includes three facial images captured with extreme lighting variances, and training set 2 uses three images that are captured under near-frontal lighting, as shown in Fig. 9.

 

Figure 7 PSR values for subject 2: authentic subjects (top plots) and imposters (bottom plots). Reprinted with permission from [11]. Copyright 2004 Optical Society of America.

Download Full Size | PDF

 

Figure 8 Images for subject 2 in the dataset. Reprinted with permission from [63]. Copyright 2005 Optical Society of America.

Download Full Size | PDF

 

Figure 9 Training sets 1 (top) and 2 (bottom). Reprinted with permission from [63]. Copyright 2005 Optical Society of America.

Download Full Size | PDF

These authors tested the performance of these filters with the complete database. At the recognition stage, the tested images were first compressed by making use of the JPEG2000 scheme at various bit rates and then were reconstructed at the authentication end. In Table 1, we display the average verification rates of the 65 filters (one from each person) at 0% false acceptance rate (FAR) [63]. According to these results, the MACE filters synthesized from training set 1 provide better performance. One can achieve 92.8% verification even at 0.5 bpp (bit/pixel) using these filters. The ROC curves of MACE filters trained with training sets 1 and 2 of subject 2 (see Figs. 10 and 11) were also compared. The ROC of the MACE filter trained with training set 1 has better performance without error for bit rates as low as 0.8 bpp. The recognition performance strongly depends on a suitable enrollment choice of training set, which can generalize for the authenticated subject. The corresponding average verification rates of the OTSDFs are also given in Ref. 63] (see Table 2). The highest recognition rates in each column are shown in bold, from which one can see that the choice of parameter $\alpha$ is closely related with the bit rate of the reconstructed images.

Table 1. Average Verification Rate (at 0% FAR) Achieved by Use of Test Images Compressed to Various Bit Rates with MACE Filters Synthesized with the Two Training Schemes [63]

View Table | View all tables in this article

Table 2. Average Verification Rate (at 0% FAR) Achieved by Use of Test Images Compressed at Various Bit Rates with OTSDFS Synthesized with Different Amounts of Noise Tolerance [63]

View Table | View all tables in this article

 

Figure 10 ROC of the MACE filter obtained with training set 1 of subject 2. Reprinted with permission from [63]. Copyright 2005 Optical Society of America.

Download Full Size | PDF

It was also found that calculating the logarithm of the intensity of the tested images was an effective way to improve the performance of the MACE filter fabricated with training set 2, because it can reduce the contrast between the dark and illuminated regions. From the results summarized in Table 3, one can conclude that a filter with more built-in noise tolerance works better at lower bit rates, because a lower bit rate means larger compression artifacts [63]. However, more noise tolerance results in degradation of the discrimination ability. One can easily realize an optimal trade-off between the discrimination and noise tolerance by adjusting the parameter factor $\alpha$.

Table 3. Average Verification Performance of 65 People (at 0% FAR) with Compressed Logarithm-Transformed Test Images at Various Bit Rates [63]

View Table | View all tables in this article

Environmental noise interference is very common in images captured under low-light conditions. To evaluate the robustness of these filters to noisy tested images, Wijaya et al. introduced a synthesized Gaussian additive noise in the tested images with different signal-to-noise ratios (SNRs) [63]. These noisy images are then compressed and reconstructed for authentication tests. They calculated the verification rates for tested images with SNRs of 7, 10, and 15 dB. In Table 4, we present the calculated results for SNRs of only 7 dB (i.e., the worst case). For more complete data, see [63]. Table 4 verifies that the noise interference of test images can be resolved to some degree by the use of OTSDFs. By comparing with simulation results obtained for other SNRs, they found that lower SNRs require a larger amount of noise tolerance, that is, smaller values of $\alpha$, in order to obtain larger recognition rates.

Table 4. Verification Rates (at 0% FAR) of Test Images with 7 dB Additive White Gaussian Noise Compressed at Various Bit Rates and Tested with the OTSDF at Different Amounts of Noise Tolerance [63]

View Table | View all tables in this article

From these simulation results, Wijaya et al. checked that the MACE filter and the OTSDF can resolve face verification of compressed test images at low bit rates (JPEG2000 scheme) in the presence of illumination variations and additive white Gaussian noise. By selecting suitable training sets and the value of $\alpha$ of OTSDFs, one can achieve high verification rates (above 92% for $\mathrm{FAR}=0\%$) with 0.5 bpp compression. To the best of our knowledge, [63] is the only report in which the influences of image compression, illumination variance, and noise interference on the performance of CFs were simultaneously considered in the face verification task.

2.3. ASEFs

In this section, we summarize the main features of the ASEF (see Appendix A.2, item 13), i.e., a class of CFs proposed by Bolme et al. first designed for exploring eye localization tasks in face images [55]. The ASEF has two marked features that differ from many previous composite filters that specify a only single peak value of correlation output per training image. First, ASEF is an over-constrained filter that exploits the convolution theorem to specify the entire surface of the designed correlation for each training image. This constraint is helpful to deal with the structured background because the resulting filter can learn to ignore other features on the human face. Second, to get the final ASEF one needs only to average the resulting filters that are generated from different training images, therefore reducing the over-fitting effect of training images that often appears in many SDFs with hard constraints. Thus, the training of such filters can be made over large and more inclusive image sets.

In recent literature, ASEFs along with the modified algorithm MOSSE [54] were also selected for performance comparison with other CFs [85,86]. Although some newly proposed CFs exhibited some advantages over ASEF in recognition rates for some face recognition or tracking tasks, we think that ASEFs deserve special mention because of their rapidity and simplicity. Moreover, performing localization of eyes is often regarded as an important first step in face detection and recognition.

According to the explanation provided by the authors of [55], the over-fitting effect found in many previous composite filters such as the MVSDF, MACE filters, and optimal trade-off filters arises mainly because there are few constraints relative to the freedom of the filter surface. To address this problem, they proposed to formulate a complete “exact filter” for each training image. This exact filter ensures that the corresponding training image has an entirely predefined training output (e.g., a bright peak centered on the target of interest). They also obtained the additional benefit that the target of interest does not need to be strictly centered in the training image because the location of the desired correlation peak can be selected freely. Using the convolution theorem, they solved for the exact filter in the following form:

 

Figure 11 ROC of the MACE filter obtained with training set 2 of subject 2. Reprinted with permission from [63]. Copyright 2005 Optical Society of America.

Download Full Size | PDF

The performance tests of ASEF in face recognition or tracking can be found in Subsections 2.6 and 2.9. Here, we present simulation results only for eye localization [55]. In these tests, the FERET dataset was employed to compare the localization ability of the ASEF and several other filters. The FERET dataset contains 3368 images of 1204 subjects. The scheme proceeds as follows. First, this dataset is randomly separated into two sets of 602 people and 1699 images. One of these sets is further separated into the training set with 1024 training images and the verifying set with 675 images. The training and verifying sets are used for the training and adjustment of filters, respectively. Another set is used for testing. To make the task more challenging, each of the training images is perturbed eight times by random similarity transforms, resulting in 8192 training images ($128\times 128$). In [55], the quantitative parameter used to evaluate localization is the deviation distance normalized by the intraocular distance. The distance $D<0.1$ is chosen as the criterion of successful localization. Several experiments are performed under different conditions using several CFs. In Fig. 13, we present the localization results when the approximate position of the eye is not known in advance. Figure 13(a) demonstrates that the ASEF and the UMACE can avoid an over-fitting effect when the number of training images is a large number, while the localization accuracy of the other filters decreases rapidly due to the over-fitting effect. Here, the alphas are adjustment factors of the composite filters, and sigma denotes the standard deviation of the specified Gaussian output of the ASEF. Figure 13(b) displays the correct rates of three filters as functions of the fraction of the intraocular distance for which the ASEFs provide the best localization accuracy. Bolme et al. attributed the good performance of ASEF to two causes. First, the ASEF is trained on the entire face image, including the “wrong” eyes, nose, and mouth [55]. Note that the other composite filters, such OTSDF and UMACE, are centered on the correct eyes and therefore have no exposure to these or other perturbations. Second, because the ASEF specifies the correlation output surface for the entire training image, it can lead to a high response of the correct eye and to a low response for the rest of the face [55].

 

Figure 12 Training images $f_{\mathrm{1,2},3}$, specified correlation outputs $g_{\mathrm{1,2},3}$, and exact filters $h_{\mathrm{1,2},3}$ of ASEF [55].

Download Full Size | PDF

Despite the fact that several researchers developed new CFs dedicated to face recognition tasks that outperform the ASEF, this algorithm is still used by researchers for feature localization and face recognition.

2.4. MOSSE Filter

In recent years, the MOSSE filter (see Appendix A.2, item 14) proposed by Bolme et al. [54] has been recognized as a typical example of a correlation-based tracker [77–79,90,91]. Different from its previous counterparts that are poorly suited to tracking due to time-consuming training procedures, the MOSSE filter is an adaptive CF that can be used by trackers with high efficiency to model the actual appearance of the target. By a comparative study, Bolme et al. found that trackers based on the MOSSE filter can achieve performance consistent with that of other robust tracking schemes at a higher update rate. Moreover, the implementation of the MOSSE-filter-based tracker is much simpler than many other trackers that often require complex appearance models and optimization algorithms [108–110].

Although the original MOSSE filter was specially aimed at object tracking, it has often been employed as comparison benchmark of CFs used to perform object recognition or identification. When Bolme et al. devised the MOSSE filters, their aim was to develop a modified filter that could keep the robustness of ASEFs [55] while requiring fewer training images to meet the high-speed tracking need. The optimization goal of the MOSSE filter is to find a filter that can minimize the sum of squared error (SSE) between the actual correlation of training images and the desired correlation output. The idea of minimizing SSE is not a new concept since it has been considered in earlier CF designs [35,111]. Different from the previous designs where the target was always carefully placed in the center of the training image and correlation output was fixed for the entire training set, the MOSSE filter customizes the correlation output for each training image without any target-centering requirement. Bolme et al. [54] deduced the following closed form of the MOSSE filter:

 

Figure 13 Localization accuracy of several CFs when a priori approximate eye localization is unknown. © 2009 IEEE. Reprinted, with permission, from Bolme et al., IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 2105–2112 [55].

Download Full Size | PDF

 

Figure 14 Comparison of the second frame PSR values of the MOSSE, ASEF, and UMACE filters. © 2010 IEEE. Reprinted, with permission, from Bolme et al., IEEE Conference on Computer Vision and Pattern Recognition (2010), pp. 2544–2550 [54].

Download Full Size | PDF

During the tracking period, the filter can adapt rapidly to keep up with the changes of the target appearance in pose, scale, rotation, or under different illumination conditions. In [54], the MOSSE filter learned from the $i$th frame is computed as

 

Figure 15 Comparison of the second frame PSR values of the regulated CFs when different regulation parameters are added. © 2010 IEEE. Reprinted, with permission, from Bolme et al., IEEE Conference on Computer Vision and Pattern Recognition (2010), pp. 2544–2550 [54].

Download Full Size | PDF

Bolme et al. also discussed the ability of some current trackers, such as, incremental visual tracking (IVT) [109], robust fragments-based tracking (FragTrack) [108], multiple instance learning (MILtrack) [112], robust online appearance models (ROAMs) [113], and Online Ada-Boost (OAB) [114]. They compared tracking scenes on the same test sequences with results provided by CF-based trackers. Bolme et al. found that, although these state-of-the-art schemes involve computation of heavy weight classifiers, complex appearance models, and stochastic search techniques, they did not exhibit significant advantages over the correlation-based trackers in tracking stability and efficiency [54]. Interested readers can check [109,112], where tracking results of IVT, ROAM MILTrack, OAB, and FragTrack are presented on the same video sequences for comparison.

MOSSE-based trackers are adapted to visual object tracking. Based on the framework of MOSSE filters, several variations of MOSSE filters and other CF-based tracking algorithms have been recently developed with great strengths in efficiency and robustness [77–79,90,91]. Attenuating training needs by optimizing existing algorithms is critical for the application of advanced CFs in online and real-time object tracking.

2.5. Correlation-Based Visual Tracking Using Convolutional Features

Shortly after the introduction of the MOSSE filter, histograms of oriented gradient (HOG) [79], color names (CN) [115], and channel representations [116] were also successfully employed in CF-based tracking schemes because such representations demonstrated good results for tracking. Recently, CNN techniques have drawn considerable attention because they achieved promising results for face recognition and detection [92–101]. In a recent paper, Danelljan et al. [100] employed activations from the convolutional layer of CNNs in a CF-based tracking framework. Making use of the online tracking benchmark (OTB), the Amsterdam library of ordinary videos for tracking (ALOV300++), and the visual object tracking (VOT) challenge 2015, the experiments demonstrated superior performance compared to other representations [100]. CNN is a special case of ANNs. These networks perform a sequence of convolution, local normalization, and pooling operations (called layers) on a fixed sized RGB image (input image). The final layers are fully connected (FC) layers. All the CNNs are trained using raw pixels with a fixed input size. Training these networks requires a large amount of labeled training data [100]. The standard deep features contained in the FC layers of a trained CNN are genetic and typically employed in a variety of vision areas. Interestingly, recent articles have shown that activations from the convolutional layers can lead to better results for classification tasks than those from the FC layer of the same network [117]. These convolutional layers are discriminating, semantically meaningful, and contain structural information crucial for the localization task. Additionally, the need to apply task-specific fine-tuning is also mitigated by using convolutional layers [117].

In Danelljan et al.’s scheme [100], spatial structures of these convolutional layers are employed for learning two different CFs: discriminating CF (DCF) and spatially regularized discriminating CF (SRDCF) [118]. The learned DCF and SRDCF can be viewed as the final classification layer in the network. The DCF framework uses the properties of circular correlation to train and applies the resulting correlation classifier to the input feature channels in a sliding window fashion. In a DCF-based tracking framework, the CF $f_t$ is learned from a set of patches $x_k$ that are sampled at each frame ($k=\mathrm{1,2},\dots ,t$, and $t$ denotes the current frame number). The objective is to find a CF that minimizes the following loss:

In their experiments, the authors employed the imagenet-vgg-2048 network [119], which is trained on the ImageNet dataset for image classification tasks. This network contains five convolutional layers and a $224\times 224$ RGB image is used as input. Figure 17 compares the tracking performance when different convolutional layers are employed in the DCF framework in the OTB dataset [120]. The results of mean overlap precision (OP) are given to measure the tracking performance. OP is calculated as the percentage of frames in a sequence where the intersection-over-union overlap with the ground-truth bounding box is larger than a specific threshold $T\in (\mathrm{0,1})$ (here $T=0.5$ is chosen). Layer 0 represents the intensity of the RGB input, which is directly used as feature descriptor. The involvement of convolutional layers improves significantly the tracking performance. The best results are achieved for the first layer. Figure 18 shows a comparison of the first convolutional layer with other hand-crafted features [HOG, CN, and image intensity (I)] commonly used in the DCF-based trackers. Here OP is plotted over the range of thresholds, and area-under-the-curve (AUC) is employed to rank the different methods. The convolutional features achieved the best result with an AUC of 52.1% [100]. On the full OTB datasets (50 sequences), the authors compare their tracker with 15 state of-the-art trackers: SRDCF [118], discriminative scale space tracker (DSST) [90], kernel correlation filter (KCF) [79], scale adaptive with multiple features (SAMF) tracker [78], advanced combat tracker (ACT) [115], tracker using Gaussian processes regression (TGPR) [121], multiple experts using entropy minimization (MEEM) [122], Struck [123], correlation filters with limited boundaries (CFLB) [124], multiple instance learning (MIL) [125], compressive tracking (CT) [126], tracking learning detection (TLD) [127], distribution fields tracker (DFT) [128], enhanced distribution field tracker (EDFT) [129], and the adaptive structural local sparse appearance model (ASLA) [130]. The success plot on the full OTB dataset is displayed in Fig. 19, where the AUC scores for the top 10 trackers are presented in the caption. Here, DCF and SRDCF trackers using convolutional features are named DeepDCF and DeepSRDCF in the legend [100]. DeepSRDCF outperforms all the other trackers in terms of AUC scores.

 

Figure 16 Performance of three filter-based trackers on three video sequences for face tracking. © 2010 IEEE. Reprinted, with permission, from Bolme et al., IEEE Conference on Computer Vision and Pattern Recognition (2010), pp. 2544–2550 [54].

Download Full Size | PDF

 

Figure 17 Performance comparison when using different convolutional layers in the network. © 2015 IEEE. Reprinted, with permission, from Danelljan et al., Proceedings of the IEEE International Conference on Computer Vision Workshops (2015), pp. 58–66 [100].

Download Full Size | PDF

 

Figure 18 Performance comparison of several feature representations in the DCF framework. © 2015 IEEE. Reprinted, with permission, from Danelljan et al., Proceedings of the IEEE International Conference on Computer Vision Workshops (2015), pp. 58–66 [100].

Download Full Size | PDF

They also evaluate these trackers by using attribute-based analysis. Figure 20 shows the success plots of four different attributes: scale variation, in-plane rotation, fast motion, and occlusion. For clarity, only the top ten trackers are given in each plot. The number in each plot title indicates the number of sequences related to a particular attribute. The proposed trackers, especially DeepSRDCF, achieve better performance compared to other methods. The VOT challenge is a competition among short-term, model-free visual tracking methods [131]. The trackers in VOT are evaluated in terms of accuracy and robustness scores. In Table 5, the authors of [100] compare the results by the VOT2015 toolkit. In each column, the first, the second, and the third ranks are marked with superscripts $a$, $b$, and $c$, respectively. The proposed DeepSRDCF tracker achieves the best final rank on this dataset. Due to space limitations, we do not show the experiment results on the ALOV300++ dataset [132]. The interested reader may wish to consult [100] for a detailed discussion. The improved results achieved by DeepSRDCF in visual tracking proved the potential of convolutional features contained in the CNNs. It is worth noting the excellent robustness of DeepSRDCF shown in Fig. 20.

Table 5. Results Generated by the VOT2015 Benchmark Toolkit [100]

View Table | View all tables in this article

 

Figure 19 Success plot showing a comparison of our trackers with state-of-the-art methods on the OTB dataset containing all 50 videos. © 2015 IEEE. Reprinted, with permission, from Danelljan et al., Proceedings of the IEEE International Conference on Computer Vision Workshops (2015), pp. 58–66 [100].

Download Full Size | PDF

2.6. MMCF

In many computer-based pattern recognition applications, the SVM classifier is very popular [71,72]. SVMs are often designed by extracting features from training images and then using a feature vector to represent the image. For a two-class problem, the task of the SVM classifier is to find a hyperplane that maximizes the smallest L2 norm distance between the hyperplane and any data sample with specific constraints. The resulted hyperplane is represented by a solution vector (the normal to the hyperplane) and the bias (or offset) from the origin (all called the margin). The solution vector can be expressed as a linear combination of training vectors (feature extracted from the training image). The training vectors corresponding to non-zero coefficients are called support vectors. One can use the resulting solution vector for simultaneous object localization and classification by cross correlating the template converted by the solution vector with features extracted from the test image. SVMs are designed to maximize margin and thus generally exhibit generalization for the desired class, but they exhibit poor localization because the peaks resulting from the cross correlation of the SVM template and the test image are not sharp [62]. In contrast, CFs can produce sharp peaks and thus offer good localization performance, but they are not explicitly designed to offer good generalization. Rodriguez et al. devised a MMCF (see Appendix A.2, item 12) that contains the localization capability of CFs and the generalization capability of SVM [62].

Design principles of SVMs and CFs are simultaneously considered in the fabrication of MMCF. In [62], the multi-objective function of an MMCF is written as

The MMCF design can be implemented with the standard SVM solver by making appropriate data transformation. Detailed procedures and analysis of the design can be found in [62]. It is important to observe that Rodriguez et al. introduced an adjustment factor $\lambda$ that can flexibly balance the localization and generalization of the MMCF (see Appendix A.2, item 12). To evaluate the performance of the MMCF, Rodriguez et al. conducted a set of computer vision tasks [62]. Here, we present simulation results for face recognition. Five other classifiers or filters, namely, SVM, ASEF, MOSSE filter, OTSDF, and UOTSDF were also used for comparison.

In their tests, the tested faces or training faces ($64\times 64$) were provided from the Multi-PIE database, which contains 337 subjects (persons) with different face poses, expressions, and illumination variations. The images in this database were captured over multiple sessions (five sessions at most). Figure 21 shows different images for one subject. Rodriguez et al. presented the results using frontal images of neutral expressions with different illumination (more than 23,000 images) [62]. In their scheme, to build the filters or SVM templates, a set of true-class training images was selected per subject, while the true-class training images from all the imposters were used as false-class training images. They performed eight experiments, which are described below.

 

Figure 20 Attribute-based comparison of our trackers with some state of-the-art methods on the OTB-2013 dataset. © 2015 IEEE. Reprinted, with permission, from Danelljan et al., Proceedings of the IEEE International Conference on Computer Vision Workshops (2015), pp. 58–66 [100].

Download Full Size | PDF

In each experiment, each test image is cross correlated with all 337 templates (each template is designed to positively classify one subject). Table 6 shows the classification accuracy for the first six tests using different filters or classifiers [62]. Tests 1–3 are more challenging than Tests 4–6, since the testing session is different from the training session. From these results, the MMCF exhibits higher recognition ability than any other CFs in Tests 1–3. Rodriguez et al. attributed this good performance to the introduction of the SVM concept because it can improve the generalization of filters. The classification accuracy of the MMCF outperforms that of the SVM classifier. According to Rodriguez et al.’s interpretation, this originates mainly from the delta-function-like requirement of the MMCF for correlation output, in which the inner products of filters with the centered training images are constrained to be 1 and the inner products with the shifted images are constrained to be 0. Thus, the centered training images as well as all shifted versions are effectively used in designing a MMCF. The implicit increase in the number of training images also contributes to the classification accuracy of the MMCF. In contrast, SVMs constrain only the inner product of the template and the centered training images, and does not constrain the other values of the correlation plane.

Table 6. Classification Accuracy (%) [62]

View Table | View all tables in this article

Tests 7 and 8 were carried out to estimate the occlusion performance of these CFs and SVM. In Fig. 22, three tested images with variant degrees of occlusion are displayed as reference. Figures 23 and 24 present the variances of the rank-1 identification accuracy with the increasing percent of missing pixels in the image (calculated from Tests 7 and 8, respectively) [62]. In Test 7, the MMCFs exhibit more robustness than the other CFs and the SVM within the range of a low percent of occlusion. However, they were more sensitive to information loss than the OTSDF and the UOTSDF in Test 8. Finally, Table 7 [62] compares the computation complexity of these algorithms using the MATLAB platform, where $N$ ($N=100$) denotes the number of training images, $d$ ($d=40\times 70$) represents the total pixels of one training image, and $d_s$ ($d_s=512\times 640$) is the total pixels of one test image. Although training a MMCF is more time consuming, its recognition speed is the same as those of the other CFs and SVM.

 

Figure 21 Test images with 25%, 50%, and 75% occlusion. © 2013 IEEE. Reprinted, with permission, from Rodriguez et al., IEEE Trans. Image Process. 22, 631–643 (2013) [62].

Download Full Size | PDF

Table 7. Computational Complexity ($\mathsf{O}$, see box) and Measured Time [62]

View Table | View all tables in this article

 

Figure 22 Multi-PIE database test images with illumination variations. © 2013 IEEE. Reprinted, with permission, from Rodriguez et al., IEEE Trans. Image Process. 22, 631–643 (2013) [62].

Download Full Size | PDF

 

Figure 23 Accuracy loss in Test 7 for face recognition. © 2013 IEEE. Reprinted, with permission, from Rodriguez et al., IEEE Trans. Image Process. 22, 631–643 (2013) [62].

Download Full Size | PDF

It is instructive to enhance the generalization capability of CFs by introducing the design concept of the SVM classifier. The presented simulation results have demonstrated the potential value of such a hybrid scheme in many face recognition applications. We believe that successful combination of the two classical recognition techniques, CF and SVM, will breathe new life into the design field of pattern recognition.

2.7. UMACE Filters Using a LBP Operator

As mentioned earlier, during the training period, the MACE filter and its generalization, the OTSDF filter [31,37], are designed to satisfy the hard constraints of correlation peaks in response to the training images. Although the OTSDF filters realize the trade-off optimization between noise tolerance and discrimination, such filters cannot yet provide high correlation peaks in response to the unconstrained images from the desired class. Moreover, these hard constraints can actually be replaced with softer requirements [35–40]. Recently, Maddah and Mozaffari proposed the UMACE filter based on a LBP for face verification [46]. Simulation results showed that satisfying recognition performance is achieved in comparison with the UMACE filters.

LBP is a powerful descriptor of texture that encodes local primitives into a feature histogram [133]. Within the LBP operation of an image, the $3\times 3$ neighborhood of each pixel in the image is binarized by using the center value as a threshold (see Fig. 25 for illustration, where 211 is taken as the gray level value of the center pixel). The resulted 256-bin histograms of the LBP labels are employed as a texture descriptor. Figure 26 shows the flowchart of the LBP-UMACE, from which we can find that the LBP is used as a pre-processing operation of training images before the fabrication of the UMACE.

 

Figure 24 Accuracy loss in Test 8 for face recognition. © 2013 IEEE. Reprinted, with permission, from Rodriguez et al., IEEE Trans. Image Process. 22, 631–643 (2013) [62].

Download Full Size | PDF

 

Figure 25 Basic LBP operator. Reprinted with permission from [46]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

In simulation tests, the extended Yale Face Database B was used to evaluate the performance of the LBP-UMACE in the presence of illumination variations. This database contains 38 persons (subjects) under 64 different illumination conditions. In [46], the first twelve images of each person were selected to design the LBP-UMACE filters and the conventional UMACE filters. Then, these CFs were tested with the complete database (64 authenticated person and $37\times 64$ imposters). Figure 27 shows images of one person from the Extended Yale B frontal database [46]. Tables 8 and 9 display the corresponding recognition results of both CFs when the PSR threshold is equal to 10 (bold numbers represent successful recognition). The upward and downward arrows represent the increase and decrease relative to the UMACE filter, respectively. The recognition rates yielded by LBP-UMACE are found to be larger than those of the conventional UMACE filter. The zero error rates obtained for imposter images constitute an attractive feature of this scheme.

Table 8. Recognition Results of UMACE Filters for the First Five Subjects [46]

View Table | View all tables in this article

Table 9. Recognition Data of LBP-UMACE Filters for the First Five Subjects [46]

View Table | View all tables in this article

 

Figure 26 Flowchart of the LBP-UMACE filter. Reprinted with permission from [46]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

In Fig. 28, the recognition rates for each person are given for comparison purposes ($\mathrm{PSR}=10$). The horizontal axis denotes the subject index. For all subjects, the average recognition rates of the LBP-UMACE filter and the conventional UMACE were found to be 86.06 and 73.43, respectively [46]. In addition, Maddah and Mozaffari computed the error rates for all subjects to further verify the effectiveness of their scheme (see Fig. 29). The obtained average error rates were 0.46 for LBP-UMACE and 7.42 for the conventional UMACE.

 

Figure 27 Typical images of one subject. Reprinted with permission from [46]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

 

Figure 28 Comparison of recognition rates. Reprinted with permission from [46]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

According to Maddah and Mozaffari’s explanation, because the information of adjacent pixels is considered in each pixel of the LBP output, it contains more frequency content than a common image [46]. As a result, UMACE using LBP outputs as training images leads to significant improvement in recognition ability compared to conventional UMACE. Indeed, the numerical pre-processing of training images, i.e., the LBP operator, is helpful to improve the performance of existing MACE filters. However, several important questions remain, i.e., when other neighborhood regions, e.g., $4\times 4$ pixels, are considered for the LBP operation, how well can the LBP-UMACE filters perform?

2.8. Face Recognition Based on Correlation and the ICA Model

One of the main drawbacks of existing face recognition methods comes from their sensitivity to the rotation of the target image with respect to the training images. Although many composite filters have been presented to optimize the correlation output in robustness and discrimination, their performance is deteriorated by saturation effects when the number of training images increases [52]. Motivated by the enhanced face recognition performance of ICA models in a compressed and whitened space, and their high sensitivity to high-order relationships among pixels, Alfalou and Brosseau proposed a hybrid scheme based on VLC and ICA [59], in which ICA [134,135] is carried out for the optimal pre-processing of the target image, and the recognition stage is performed with a POF.

The ICA algorithm can provide an optimal signal representation technique in the MSE sense [136]. In [59], a given face is considered a linear combination of several independent components (ICs) arising from reference facial images contained in a learning base. The learning base is a set of faces of the same person (e.g., subject X) differing in head orientation. In Fig. 30(a), we show the construction procedure of the IC training base. First, training faces in the learning base are converted to the vector forms $V_i$ ($i=\mathrm{1,2},\dots ,n$). Then, these vectors are processed by a specific nonlinear function operation that ensures good convergence for the subsequent fast ICA algorithm [59]. Finally, fast ICA converts the original learning base into a new learning base consisting of ICs. In this study, the authors chose the ICs that can provide the best result.

 

Figure 29 Comparison of error rates. Reprinted with permission from [46]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

Next, each face $V_i$ of the learning base is correlated with different POFs, $F_j={\mathrm{POF}}_j$, and ${\mathrm{PCE}}_{ij}$ values of the correlation outputs are calculated. Here, each CF is generated from one IC, $C_j$. It is worth observing that $V_i$ and $C_j$ need to be transformed to 2D forms to perform the correlation operations. By carrying out a series of such correlation filtering, one can get a reference matrix of PCEs [see Fig. 30(b)]. Once the components $C_j$ and the ${\mathrm{PCE}}_{ij}$ values are determined, the recognition algorithm shown in Fig. 30(c) is carried out. In this algorithm, the vector $V_{?}$ representing the target face can be considered a linear combination in terms of the IC base. By correlating $V_{?}$ with the different POFs $F_j$, one can achieve a set of PCE values $\mathrm{Val}\_\mathrm{Corr}=({\mathrm{PCE}}_1^{\prime },{\mathrm{PCE}}_2^{\prime },\dots ,{\mathrm{PCE}}_n^{\prime })$ used to identify the target face among the reference images. Finding the ${\mathrm{PCE}}_j^{\prime }$ can be realized by introducing the target image into the input plane of a standard VLC using a POF generated from a single IC. By comparing with the PCE matrix, a set of error vectors are defined and calculated as

Figure 31 shows simulation results obtained from images of PHPID. Three training images X(9) (right rotation), X(27) (without rotation), and X(45) (left rotation) are chosen in these simulation tests [represented by the red frames in the Fig. 31(b)]. The horizontal axis of Fig. 31(a) denotes the face index, and the vertical axis represents the error values obtained with different faces of person $X(i)$ ($1<i<53$) and the 32nd face of another person $Y(32)$, which is framed in green in Fig. 31(d). From Fig. 31(a), we find that a good performance is provided by this algorithm. However, the appropriate choice of threshold is critical to make a reliable decision. Figure 31(c) shows the ROCs as another test of this algorithm. Every point of this ROC is calculated with a specific threshold value within the range [2000,⋯,2200]. The horizontal and vertical axes of Fig. 31(c) correspond to the false recognition and the true recognition rates, respectively. The solid red line represents the reference values when the true recognition is equal to the false recognition. By fixing the threshold arbitrarily to 2008, a recognition rate of 83% is obtained with a false recognition rate of 3.7% [59].

 

Figure 30 Schematics of the algorithm: (a) definition of the independent component learning base, (b) definition of the PCEs matrix, and (c) recognition procedure. Reprinted with permission from [59]. Copyright 2011 Optical Society of America.

Download Full Size | PDF

Interestingly, Alfalou and Brosseau’s scheme can be optically implemented. This renders this technique suitable for designing improved image processing systems. The introduction of ICA plays an important role in providing better recognition outputs with a considerably smaller false alarm rate in comparison with those obtained from a simple composite filter. Remarkably, Alfalou and Brosseau’s work is a fruitful test of the combination of the optical correlation method with another conventional numerical pattern recognition method. Moreover, it is also possible to further improve the performance of this scheme by replacing the POF with other optimal filters. Another novelty of this work is that the PCE error is selected as the decision criterion that is to be contrasted with many conventional schemes. How the choice and number of ICs can impact the performance of this scheme remain to be studied in more detail.

2.9. Zero-Aliasing CFs for Face Recognition

To help resolve the aliasing effect in many existing CFs, Fernandez and co-workers proposed zero-aliasing CFs (ZACFs) [85,86] (see Appendix A.2, items 15–18) in which a set of ZA constraints are introduced in the design stage of composite filters to eliminate the template’s non-zero tail that results in aliasing the correlation output. This aliasing, i.e., parts of correlations being overlapped with other parts of themselves, can seriously reduce the sharpness of correlation peaks. ZA constraints can be flexibly incorporated into the designs of many existing CFs to optimize both localization and recognition rate. Using two databases of face images, the ZA versions of CFs are found to outperform the original ones in recognition ability [86].

CFs can perform simultaneous classification and localization of authenticated objects in a given scene without assuming prior segmentation of objects in test scenes. However, it has been recognized that there exists a potential design default in the training stage of these filters that can deteriorate the recognition ability of many CFs [85]. To increase the computational efficiency when performing optimizations of various metrics (such as minimization of MSE) during filter design, the cross correlation in the spatial domain is often converted to the element-wise multiplication of the CF [the discrete FT of 2D CF template array in spatial domain] and the conjugate of the discrete FT of the training images in the spatial frequency domain. However, the element-wise multiplication between two discrete FTs does not correspond to the intended linear correlation in the spatial domain but results in a circular correlation, i.e., an aliased version of linear correlation shown in Fig. 32 [86]. The wide sidelobe of circular correlation can affect the localization performance of the CFs. The optimization and training stages of almost all CFs used involve the discrete FT-based operations mentioned earlier with the wrong assumption that circular correlation is identical to linear correlation.

 

Figure 31 Simulation results illustrating the ICA algorithm. Reprinted with permission from [59]. Copyright 2011 Optical Society of America.

Download Full Size | PDF

 

Figure 32 Linear correlation (left) and circular correlation (right). © 2015 IEEE. Reprinted, with permission, from Fernandez et al., IEEE Trans. Pattern Anal. Mach. Intell. 37, 1702–1715 (2015) [86].

Download Full Size | PDF

The simple zero padding of test images and the CF template in the test stage cannot resolve the aliasing problem arising from the design stage of CFs. To thoroughly remove the aliasing effects due to circular correlation, Fernandez et al. [86] proposed the ZACF, in which ZA constraints that can force the template’s tail to zero in the design stage are applied into the optimization procedure of CFs. To overcome the intensive computation in the training stage of ZACF, Fernandez et al. developed two methods based on reduced aliasing CFs (RACF) and the iterative proximal gradient descent. The former can provide a computationally more tractable closed = form solution while allowing parts of aliasing to remain. In addition, a proper zero-padding operation needs to be made in the training images. The latter is a fast and more efficient iterative approach for the numerical solution from the computational memory standpoint.

In [86], the authors display the 3D output of the conventional MACE and zero-aliasing MACE (ZAMACE) filters (Fig. 33) to illustrate the benefit of the ZACF. The ZAMACE filter yields a sharper but somewhat lower correlation peak, which is more consistent with the original design of the MACE filter. The performance of the ZACF was quantitatively evaluated for several recognition tasks. Here, we report simulation results dedicated to face recognition. Two different databases, the AT&T Database of Faces (ORL dataset) and the face recognition grand challenge (FRGC) dataset were used to perform the tests.

 

Figure 33 Conventional MACE (left) and ZAMACE (right). © 2015 IEEE. Reprinted, with permission, from Fernandez et al., IEEE Trans. Pattern Anal. Mach. Intell. 37, 1702–1715 (2015) [86].

Download Full Size | PDF

The AT&T Database of Faces (ORL dataset) contains the facial images ($112\times 92$ pixels in size) of 40 subjects (persons). For each person, 10 different images are recorded. A “leave one out” cross-validation approach is employed in the tests, i.e., nine training images are selected to build one CF for each person, and then the 40 filters are tested on the remaining 40 images (one for each person) for all subjects, i.e., a total of 1600 correlations. The above tests are repeated 10 times. The PCE is taken as a decision criterion. In addition, the EER and the rank-1 ID rates are evaluated using different CFs, namely, the OTSDF, MOSSE filter, and MMCF, and their zero-aliasing versions. The corresponding data are summarized in Table 10. The first column corresponds to the original CFs. The calculated results shown in column 2 are obtained by calculating the closed-form solution of the ZACF. Column 3 shows the results provided by the accelerated proximal gradient descent method. The other columns correspond to various versions of RACFs with different zero paddings in the training images. The recognition rate of the ZACF based on the closed-form expression (column 2) is larger than that of the original CFs (column 2) in terms of EER and rank-ID. However, arriving at the closed-form solution is complicated and time consuming. It is worth noting that the results obtained by the proximal gradient decent method (column 3) are very close to those generated from the closed-form solution.

Table 10. Comparison of Performance for the Baseline CFs and ZACFs Using the ORL Dataset^a [86]

View Table | View all tables in this article

The FRGC dataset contains face images ($128\times 128$) of 410 subjects. The number of images for each subject varies from eight (minimum) to 88 (maximum). In [86], three training and test sets are considered by randomly choosing 25% images of each subject for training, and the remaining 75% for testing. One CF was fabricated for each subject and every CF was correlated with all test images, respectively. Table 11 shows the results for the OTSDF, MACE, MOSSE, and MMCF. An accelerated proximal gradient approach is used to construct the optimal approach of zero aliasing. By viewing the data listed in Table 11, one arrives to the same conclusion as that found in Table 10. In the above test, it is worth noting that the delta function is used as the desired correlation output for the design of these filters.

Table 11. Comparison of Performance for the Baseline CFs and ZACFs Using the FRGC Dataset [86]

View Table | View all tables in this article

The ZACF proposed by Fernandez et al. [86] presents a solution to solve the aliasing problem caused by circular correlation. This algorithm optimization makes the output results of composite CFs more consistent with the intended linear correlation, and improves the recognition ability with sharper correlation. However, to fully realize the potential of this approach, one needs to find a trade-off between its computational complexity and its discrimination ability. Additionally, more accurate tests against noise, occlusion, and illumination variation need to be completed in future studies.

2.10. Decision Optimization Algorithm Based on the Denoised Decomposition of the Correlation Plane

In previous subsections, typical CFs proposed in recent years for face recognition or tracking tasks were discussed. All these schemes endeavor to design modified CF algorithms and/or choose specific decision criteria, such as PCE and PSR, to improve the decisional performance. Using another perspective, Alfalou et al. [87] proposed a new decisional optimization algorithm based on the denoised decomposition of the correlation plane. Their aim was not to develop a new CF but to apply a post-processing step in the correlation plane before using a decision metric. In their scheme, the linear functional model (LFM) [88] and SVD method [89] are employed to separate the background noise from the correlation signal. Simulation results verify that the proposal can increase the TPR by a factor of 5 for a FPR set to 0% for the SCF [52].

The principle of this algorithm is illustrated in Fig. 34, where the correlation plane $P_c$ is set to be a finite linear combination of weighted regressors and residual signal $R$ as

 

Figure 34 Illustration of the denoised separation algorithm. Reprinted with permission from [87]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

Here we summarize the main points and focus on the ROC curves (Fig. 35) related to the simulation tests worked out in [87]. First, the algorithm is tested with a three-reference segmented filter in a VLC configuration. The training images are chosen from P(1), as shown in the inset of Fig. 35(a). The PCE was selected as the final decision criterion in the simulations. Figure 35(a) shows the results obtained by correlating the face images of P(1) and P(3) with the three-reference segmented filter. Figure 35(b) shows the ROC curves obtained by applying the proposed algorithm (12 models from each person) to post-process the correlation outputs. Without the denoising optimization, the TPR is close to 15% with zero false positives. When using the optimization, a TPR close to 70% is obtained for a FPR set to 0%. For comparison, Figs. 35(c) and 35(d) show the ROC curves obtained with a five-reference segmented filter and a five-reference trade-off maximum average correlation height (OTMACH) filter, respectively. These filters achieve a TPR close to 50% and 30%, respectively, for a 0% FPR. Although the composite filters shown in Figs. 35(c) and 35(d) are fabricated with five reference persons, their TPR values are obviously lower than that of an optimized scheme based on three reference persons.

The LFM-SVD-based recognition algorithm puts emphasis on the post-processing optimization of the correlation plane rather than on the filter’s algorithm itself, in strong contrast with many conventional optimization methods of CFs. A major contribution to decisional optimization of CFs has been proposed by this work. However, much remains to be learned to check whether or not the effectiveness of this method is confined to the SCFs.

2.11. Class-Specific Nonlinear Correlation Filter for Illumination-Invariant Face Recognition

In this section, we present briefly a novel subspace-based face recognition scheme (see Chap. 11 of [5]). This design is based on the subspace-based reconstruction of faces. The proposed CFs can dynamically change according to the input face image in order to achieve robust recognition for all possible enlightening variations that lie in a 3D linear subspace for a Lambertian source. Figure 36 shows the face reconstruction procedure while the class specific subspace analysis is performed. From Fig. 36, one can find that the test face image is almost perfectly reconstructed when the test face is selected from the class of training faces (person-1 in PIE). When the image of person-3 (imposter) is projected onto the subspace developed by person-1, the reconstructed image looks like person-1. By evaluating the difference between the test face and its corresponding reconstruction, it is possible to discriminate the authentic face from the imposter one.

 

Figure 35 ROC curves for performance comparison. Reprinted with permission from [87]. Copyright 2012 Optical Society of America.

Download Full Size | PDF

Within this context, the authors of [5] proposed a class-specific subspace CF. The detailed process of this scheme is presented in Fig. 37. First, by computing the class-specific subspace over the total $M$ classes of face images, $M$ class-specific subspaces $E^k$ ($\mathrm{1,2},\cdots ,M$) are obtained. Then, a test face image $T^{C_j}$ (from any $j$th class, $j\in M$) is projected onto the subspaces, resulting in $M$ reconstructed images $R^k$. The reconstructed correlation filter ${\mathbf{H}}_r^k$ for each class is calculated from the reconstructed image. The authors consider only the phase component (amplitude is set to unity) and denote it as ${\mathbf{H}}_{r\phi }^k$. A phase-only optimal projecting image CF ${\mathbf{H}}_{p\phi }^j$ is simultaneously calculated from the test face $T^{C_j}$. This CF is designed by minimizing the energy at the correlation plane and maximizing the correlation peak height. The final correlation output is computed as

 

Figure 36 Face reconstruction while the subspace analysis is performed over person-1 (PIE dataset). Reprinted from Face Detection and Recognition: Theory and Practice (Chapman & Hall/CRC Press, 2015) [5].

Download Full Size | PDF

Next, we demonstrate some experimental results obtained with nonlinear class-specific CFs. In [5], the PIE and YaleB (extended) datasets are employed to evaluate the illumination invariant performance of the filters. To demonstrate the improvement of PSR (see Table 12), 10 sets of three randomly selected training images are taken and the top-left corner image of Subset-5 (YaleB dataset) is chosen for test. The test images are not included in the training set but belong to the same subject. The authors of [5] also use ROC to further check the performance improvement. For this specific purpose, two images (index 10 and index 19) are chosen for training out of 21 face images from any one subject of the PIE dataset. Two training images are chosen from YaleB subset-1. The ROCs are shown in Figs. 38(a) and 38(b). Figures 38(c)–38(f) show the ROCs that are generated from different set of training images from YaleB as Fig. 38(c) two random, Fig. 38(d) three random, and from PIE as Fig. 38(e) three random, and Fig. 38(f) four random. It is worth observing that these ROC curves achieve considerable improvement against illumination variances.

Table 12. PSR Value Comparison of Different Filters [5]

View Table | View all tables in this article

 

Figure 37 Detailed process of the face recognition method. Reprinted from Face Detection and Recognition: Theory and Practice (Chapman & Hall/CRC Press, 2015) [5].

Download Full Size | PDF

2.12. Summary

In summary, we reviewed typical CF design for face recognition. To fully realize the potential of these approaches, many correlation algorithms were described in some detail. Overall, it is a general observation that these correlation-based schemes can achieve generally better performance than complicated numerical methods in several respects, such as noise tolerance, discrimination ability, and computational resources. Next, we continue our review by showing how to implement these schemes.

3. Implementation and Application of Correlation Methods

3.1. Implementation

As mentioned earlier, spatial matched filters can be implemented optically with either a VLC or a JTC. By adding suitable nonlinear optimization procedures, one can also implement more complicated correlation schemes, such as POF, CHF, and composite filters, at an optoelectronic hybrid platform. At this point, it is worth recalling that the optical correlation concept originates mainly from the inherent parallelism of optical information processing and the large light velocity. All-optical computing is a major topic within optical information science. However, compared with the rapid advancement of microelectronics-based devices in the past two decades, the development of all-optical computing devices encountered serious scientific and technological issues. Hence, the optical computing units of current optical correlators are still confined to performing several optical integral transforms, such as the Fresnel transform, FT, fractional FT, wavelet transform, and gyrator transform. To optically realize these transforms, one has to arrange optical elements intelligently under rigorous conditions and address various limitations that do not arise in numerical implementation, e.g., alignment of optical components or unwanted interference. Moreover, numerous optimization strategies of optical correlation must be completed in a numerical manner: this implies more conversion interfaces in the setup involving some specific technical problems, such as electronic noise and diffraction effects. For numerous civilian applications, optical correlators are still unable to compete with numerical correlators in terms of flexibility, stability, and cost versus performance balance. In the following sections, we present and discuss one example of a numerical correlator based on a graphics processing unit (GPU), and two examples of optical correlators based on either a compact VLC configuration, or using a holographic memory disk. In this section, we focus our attention on the hardware implementation structure of CFs, not on the algorithms and optimization design.

3.1a. GPU Implementation of Segmented Composite POF for Face Recognition

Recently, an all-numerical implementation has been studied for face recognition using a segmented composite POF [29,52]. An iterative pre-processing method of the target image is developed to ensure high discrimination while maintaining robustness against distortion of the subject, e.g., head rotation. In addition, a two-level decision tree learning approach is employed to optimize the recognition procedure with a GPU processor (NVIDIA GPU GeForce 8400GS) (see Fig. 39). For more detail about the iterative processing algorithm and the two-level decision tree learning approach, the interested reader may wish to consult [29]. Experimental results show that the proposed scheme can perform the assigned face recognition task efficiently with high discrimination, i.e., the true recognition rate is larger than 85% with $t<120\mathrm{ms}$ for stationary images from the PHPID, and larger than 77% using a real video sequence with 2 frames per second for the database containing 100 persons. It is also worth noting that a higher processing speed with 4 frames per second can be obtained using the more recent NVIDIA-GPU Quadro FX 770M processor.

 

Figure 38 Comparison of ROC plots for different training sets. Reprinted from Face Detection and Recognition: Theory and Practice (Chapman & Hall/CRC Press, 2015) [5].

Download Full Size | PDF

The GPU processer shown in Fig. 40 is the core hardware of this all-numerical implementation in which one or several stream multiprocessors are integrated together to perform intensive computing in parallel. In this scheme, compute unified device architecture (CUDA; Fig. 41) is exploited to program the GPU. In CUDA, each kernel is processed by parallel threads that are grouped together in a block. Ouerhani et al. found that the face recognition algorithm running on the GPU can achieve impressive speed compared to the conventional CPU. In [29], comparative results of the computing time were reported for four architectures. Figure 42 demonstrates that the influence of the image size on computing time of the GPU family processors is smaller than for CPU architectures.

 

Figure 39 Two-level decision tree learning approach. (a) First level, classification; (b) second level, identification [29].

Download Full Size | PDF

 

Figure 40 GPU architecture [29].

Download Full Size | PDF

 

Figure 41 GPU architecture [29].

Download Full Size | PDF

Ouerhani et al.’s GPU-based all-numerical implementation platform complemented with two-level decision tree learning provides a high-speed tool for investigating face recognition using computationally intensive composite filters. Making a high-speed all-numerical implementation of CFs can broaden the technological applications of CF techniques.

3.1b. All-Optical Implementation System of Correlation Based on the Vander Lugt Configuration

Recently, it was reported that the optical FT simulating Fraunhofer diffraction leads to better performance than the numerical FFT. When the target image can be only optically determined, optical implementation becomes preferable. Figure 43 displays a recent all-optical implementation setup of CF [137]. All required optical components, such as the laser source, SLM, lens, and reflector are efficiently integrated in this setup.

 

Figure 42 Influence of the image size on the run time for four architectures [29].

Download Full Size | PDF

This setup was employed to implement segmented composite correlation filters [52]. Figure 44 shows an optical input image and the class domains used for segmented filtering. The PCE data listed in Fig. 45 for composite, multichannel, and segmented filters are plotted as functions of training image number. Although the generated PCE values decrease rapidly, owing to filter saturation, the relatively higher PCE values obtained for the segmented filter suggest that it is more appropriate compared to the other tested filters. To illustrate the intra-class distortion invariance of segmented filtering, the optical outputs in response to the training and non-training faces (belonging to the authentic class) are shown in Fig. 46.

 

Figure 43 All-optical correlation setup [137].

Download Full Size | PDF

 

Figure 44 (a) Optical input image and (b) class domains of segmented filtering. Reprinted with permission from [52]. Copyright 1999 Optical Society of America.

Download Full Size | PDF

 

Figure 45 PCE versus number of training images for fabrication of the composite filters. Reprinted with permission from [52]. Copyright 1999 Optical Society of America.

Download Full Size | PDF

3.1c. Optical Implementation of Face Recognition Correlator Using Holographic Memory

This optical correlation implementation system, also known as S-FARCO, was proposed and assembled by Watanabe and Kodate [27,138]. In this design, a filtering-correlation technique and a coaxial holographic optical storage system are combined to realize high-speed face recognition. In S-FARCO, a filtering-correlation optimization algorithm is employed to obtain a binary real-only matched filter that can meet the requirements of short calculation time, low false acceptance, and rejection rates. During the recording period of the database, the matched filters generated from different reference facial subjects are written on an optical disk by a co-axis FT holographic configuration [27] and then optimized in a PC server. Results presented in [27] show the schematic setup of the high-speed optical correlator with a holographic optical disk. The novel disk-shaped holographic memory system allows for a huge amount of data stored in the form of matched filters due to its 3D storage ability (typically, about 200 GB) [27]. During the recognition period, an input image on the same position is illuminated by the laser beam. This leads to a correlation signal through the matched filter on the output plane. In addition, the parallel transformation of holographic optic memory leads to higher processing speed than that of conventional digital signal processing architecture. One can speed up the correlation process by simply rotating the optical disk at higher latency. Table 13 summarizes the experimental conditions used in [27]. Preliminary correlation experiments conducted showed that the system can produce a high correlation peak and low recognition error rates at a multiplexing pitch of 10 μm and rotational speed of 300 rpm (equal error rate of 0% when the threshold is optimally chosen) [27]. The dependence of the recognition error rates (FAR, FRR, and EER) on the threshold have been studied in [27], where the shadow area represents the range of threshold that produces an EER of 0%. Table 14 summarizes correlation speed values under different conditions when Watanabe and Kodate conducted the correlation experiments. It is notable that a rotation speed of 2400 rpm is equivalent to data transfer of more than 100 Gbps.

Table 13. Experimental Conditions for Holographic Optical Disk Correlator [27]

View Table | View all tables in this article

Table 14. Correlation Speed of the Outermost Track of an Optical Holographic Optical Disk [27]

View Table | View all tables in this article

 

Figure 46 (a) Training image and (b) corresponding output; (c) non-training image and (d) corresponding output. Reprinted with permission from [52]. Copyright 1999 Optical Society of America.

Download Full Size | PDF

Watanabe and Kodate developed an online numerical correlation face recognition system (FARCO) with high operation speed (less than 10 ms/correlation for 3 GHz CPU and 2 GB memory) and high accuracy for low-resolution facial images ($64\times 64$ pixels). They tested their system using 30 subjects and obtained 0% FAR and 0% FRR [27]. Finally, it is worth noting that a cell phone face recognition system [27] and a video copyright monitoring system [139] based on numerical correlation have also been assembled for technological applications.

3.1d. Summary

In summary, the implementation schemes show that correlation algorithms can run very fast using either a numerical or an optical hardware platform. Powerful GPU processors provide high-efficiency numerical computation sources for fabrication and real-time application of composite filters. Creating a compact VLC architecture such as that developed in [137] is an extremely important task for miniaturization of an optical correlator. The value of a state-of-the-art technique, i.e., the holographic optical memory disk, is an efficient tool for CF databases and data transfer [27].

3.2. Applications

In this section, we describe recent applications of correlation methods including face recognition, tracking in video surveillance, video copyright monitoring, cell phone face recognition, tracker of swimmers, image comparators, underwater mine recognition, and road sign recognition.

3.2a. Correlation-Based Face Recognition for Patient Monitoring Application in Home Video Surveillance

In [140], Elbouz et al. devised a patient monitoring system by combining correlation techniques with a fuzzy logic method. Different from previous monitoring systems, two basic correlation configurations, namely, a VLC and a JTC, are employed numerically to make (high level) decisions on the identity of patients and their 3D behavior change, e.g., a sudden fall. Figure 47 shows the overall block diagram of the monitoring system [140]. In this system, first, hue, saturation, and value (HSV) color space and moment methods are employed to detect the subject’s head information at the entrance of a room. Figure 48 shows the experimental procedures for detection of the head area in a captured image. The target image containing head information is sent to a numerical VLC fabricated with an optimized, segmented composite POF for identification. At the same time, a numerical fringe adjusted JTC (FAJTC) is initialized to guide the tracking procedure, with the reference images transformed by a stereovision system. To ensure that the tracking is reliable and robust against noise, a database containing possible head motion is stored beforehand as a reference base of JTC. Figure 49 shows several example images from a reference database. Moreover, a fuzzy logic method is also introduced to analyze the trajectory of the subject’s head motion in order to make a decision whether or not to send an alarm signal.

 

Figure 47 Block diagram of the monitoring system [140].

Download Full Size | PDF

 

Figure 48 Experimental results for head detection: (a) captured image, (b) H component thresholding, (c) S component thresholding, and (d) head-detection results. (e) This figure shows the result by using the proposed method [140].

Download Full Size | PDF

An experimental platform (see Fig. 50 [140]) is used to check the reliability of the monitoring system. A series of tests are performed using video recording of the subject’s body movement as input to the proposed algorithm. Figure 51 shows the user interface for real-time monitoring, in which the head motion is captured by four webcams and recorded from either the front, back, or side. Four video sequences of the scene can be visualized in this interface. The pink frames represent the head position of the identified subject. The software is run on a PC (Intel Dual Core E5300, 2.6 GHz, 2 GoRAM CPU). Logitech webcams are used for the tracking video with $352\times 288$ pixels in size. The main focus of this study is on fall detection corresponding to a rapid decrease of the subject’s head position. Videos of experimental results can be viewed online (https://www.youtube.com/watch?v=3nNFTHOsm0E). The videos demonstrate that the segmented composite POF in the VLC configuration can correctly identify the subject without alarm, and the numerical FAJTC can track efficiently the trajectory of the subject’s head position in all experiments, i.e., entering, rapid falling, slow bending, and sitting of the subject. The alarm rate provided by this system is close to 89% and can be interesting for potential surveillance applications.

 

Figure 49 Example of reference database used in our study [140].

Download Full Size | PDF

 

Figure 50 Experimental platform developed [140].

Download Full Size | PDF

3.2b. Monitoring of Video Copyright Based on Optical Correlation

In the past two decades, unauthorized video content on the Internet has grown exponentially. Finding fast and reliable video monitoring techniques for copyright protection has become a most active research area. Using numerical correlation algorithms, Kodate’s group developed a powerful online video-matching application called the fast recognition correlation system (FReC) [139] to monitor the copyright information of video on the Internet. The FReC software system can correlate the content on multiple videos with the registered content provided by a copyright holder at very high speeds. By patrolling a video-sharing website, FReCs can filter out a number of unauthorized video content on the Internet and send automatic warning messages to the website’s manager. In 2009, major publishers and television networks in Japan began to use FReCs to protect their online video products [139]. To further speed up the detection of the video-filtering system, Kodate’s group is developing a new optical-correlation-based system called FARCO 3.0 [139]. Different from the numerical processing of the correlator in FReC, the optical correlator hardware in FARCO 3.0 uses a holographic memory disk. This permits processing correlations more than 100 times faster than the previous FReC system.

 

Figure 51 User interface of the fuzzy logic and optical correlation based face-recognition method developed using MATLAB software and a CPU: Intel Dual Core E5300, 2.6 GHz, 2Go RAM computer [140].

Download Full Size | PDF

3.2c. Cellular Phone Face Recognition System for Attendance Management of Students

This is another interesting online application of the numerical correlation method developed by Kodate et al. [27]. In this application, the filtering-correlation optimization algorithm mentioned in Subsection 3.1c. is applied to increase the recognition accuracy and efficiency of matched filters. The block diagram depicting the cellular phone face recognition system is shown in [27], which includes two working units (online registration and recognition). The system consists of the facial recognition software, a control server for pre- and post-processing, and a cellular camera phone. During the registration period, the students access the URL provided by the administrator and download the Java application. Then, they input their facial image using the Java application on their cell phone. Finally, they send the image with their ID to the administrator. After checking whether the IDs and images in the server match, the facial images are uploaded into the reference dataset. The recognition unit consists of the following steps: (1) a student takes their facial image with the Java application on their cell phone and transmits it back to the recognition server, along with their ID; (2) the face recognition server use the optimization and correlation algorithm to analyze and recognize the input image; (3) if the input is recognized as the registered person, a one-time password is created and provided to the student; and (4) using the password, the student logs on to the remote lecture server.

This cell phone face recognition system was employed as a lecture attendance management system. This system was implemented 12 times on 30 students over a 20-week duration. The D505is and D506i (Mitsubishi Co.) were chosen as cell phone types for this experiment. The facial images were in JPEG format ($120\times 120$ pixels). Experimental error rates of this system are given in Table 15. The results show very low error rates.

Table 15. Experimental Error Rates of this Cell Phone Face Recognition System [27]

View Table | View all tables in this article

3.2d. Optimized Swimmer Tracking System by a Dynamic Fusion of Correlation and Color Histogram Techniques

In this application, the nonlinear JTC (NL-NZ-JTC) and the color histogram method are combined to perform robust swimmer tracking [141]. This combination is completed by the dynamic fusion of the correlation plane and the color score map. Figure 52 shows the environment of this application. For the experiments, two video sequences of junior swimmers from the Brest nautical club were captured. The videos were captured with a 4K resolution ($3840\times$ 2160 pixels) in 25 frames per second for a period of 15 s each. Three measures [tracking percentage, PCE, and local standard deviation (Local-STD)] were used to evaluate the tracking performance of this system [141]. The tracking percentage is calculated by measuring the Euclidean distance between the coordinates given by the tracking system and the actual measurement obtained beforehand by manually adjusting the swimmer position in the videos. The PCE measures the detection accuracy. The Local-STD estimates the accuracy and robustness in terms of localization by measuring the noise level locally in the correlation plane.

Figures 53 and 54 show the PCE curves and the Local-STD curves of three tracking algorithms (the dynamic fusion method, NL-NZ-JTC, and color histogram), respectively. In Fig. 53, the higher PCEs of the fusion method demonstrate the improvement of the detection accuracy. Figure 54 shows that the fusion method has comparable robustness to the tracking method based on color histograms. Table 16 summarizes the results of the tests performed on the two videos. From these data, one obtains conclusions that are similar to those made from inspection of Figs. 53 and 54. The superiority of the dynamic fusion method in tracking percentage is also clear.

Table 16. Comparison among NL-NZ-JTC, Color Histogram, and Dynamic Fusion Technique in Terms of Tracking Percentage, PCE, and Local-STD [141]

View Table | View all tables in this article

 

Figure 52 Shooting environment: (a) camera Blackmagic 4K used for shooting the videos of the test, (b) example of a frame extracted from a video used in the tests [141].

Download Full Size | PDF

 

Figure 53 Comparison among the NL-NZ-JTC correlator, color histograms, and dynamic fusion in terms of PCE (tested on a video sequence (360 frames) of a backstroke competition). A high value of the PCE criterion implies greater confidence in the localization (sharp peak) [141].

Download Full Size | PDF

3.2e. Image Comparator Based on the Optical JTC

In this application [142], an integrated optical JTC configuration serves as comparator of dynamic video frames (Fig. 55). Different from the conventional JTC configuration, the input plane is formed by combining the current frame with the previous one in a video sequence. When the frames from the reference and the input sequences do match, a correlation peak is generated on the output plane; otherwise, the correlation peak between them decreases rapidly and eventually vanishes. Any possible movement of the chosen target can be detected by the movement of this correlation peak based on the shift-invariance property of linear correlation. The JTC application consists of a digital CMOS camera, a single liquid-crystal-over-silicon SLM, and a FT lens. Interestingly, numerical nonlinear procedures can be introduced in the joint power spectrum plane, allowing tuning and/or optimizing its recognition and robustness. The value of this image comparator has been demonstrated in many computer vision devices [142]. Figure 56 presents the basic configuration of this comparator.

 

Figure 54 Comparison among the NL-NZ-JTC correlator, color histogram, and dynamic fusion in terms of Local-STD (tested on a video sequence (360 frames) of a backstroke competition). A low value of the Local-STD criterion implies a greater confidence toward the detection (less noisy plane) [141].

Download Full Size | PDF

 

Figure 55 Image comparator based on the JTC [142]. Available from http://www-g.eng.cam.ac.uk/CMMPE/pattern.html.

Download Full Size | PDF

Figure 57 shows the correlation outputs of the JTC comparator when the reference and input sequences match and do not match. One can observe a change of brightness of the correlation signals, which is caused by the mismatch of sequences. Based on correlation shift invariance, the image comparator can be used to evaluate the direction of a moving object, e.g., jet pilot head tracker. For this purpose, an image comparator is used to detect the spatial translations of the camera view. The working principle of this application is illustrated in Fig. 58 where the distance and displacement measurement of the pilot’s head are characterized by the translation of the cross-correlation signal (blue dots). Figure 59 shows experimental results of this head tracker.

 

Figure 56 Picture showing the configuration of the image comparator studied in [142]. Available from http://www-g.eng.cam.ac.uk/CMMPE/pattern.html.

Download Full Size | PDF

 

Figure 57 Correlation output of the image comparator based on JTC [142]. Available from http://www-g.eng.cam.ac.uk/CMMPE/pattern.html.

Download Full Size | PDF

 

Figure 58 Head tracker based on JTC comparator [142]: the blue spots represent the cross-correlation peaks, and the red spot is the auto-correlation peak. Available from http://www-g.eng.cam.ac.uk/CMMPE/pattern.html.

Download Full Size | PDF

3.2f. Underwater Mine Detection and Recognition

In this application, an adaptive nonlinear fringe-adjusted JTC is devised to perform detection and recognition of underwater mines [137,143]. To optimize the recognition performance of correlation filters, a numerical nonlinear procedure is introduced in the Fourier plane of the fringe-adjusted JTC [143]. Moreover, a pre-processing treatment is also carried out to enhance the image quality within the underwater environment, e.g., polarized light is used to denoise the input target image. Figure 60 shows several types of underwater mines used for experimental detection.

 

Figure 59 Experimental results of head tracker based on JTC comparator [142]. Available from http://www-g.eng.cam.ac.uk/CMMPE/pattern.html.

Download Full Size | PDF

In Figure 61, detection results of the optimized nonlinear fringe-adjusted JTC are shown to verify its robustness and recognition ability. The horizontal axis of this figure represents the frame index of video sequence. The vertical axis measures a specific decisional criterion [143] that is related to the sharpness of the correlation peaks and takes its value between 0 and 1. A large value means that an underwater mine is identified in the video frame. To optimize the automatic decision making, a threshold needs to be set properly. At the bottom of Fig. 61, several images show the real detection environment. From Fig. 61, one finds that the optimized nonlinear fringe-adjusted JTC can detect most of the image frames containing the underwater mine with very few false alarms. Detailed results, in terms of recognition and false alarm probabilities, are summarized in Table 17, where three types of fringe-adjusted JTC are compared. Here $k$ is a nonlinear factor [143]. The optimized nonlinear fringe-adjusted JTC produces the smallest false alarm rate among the three JTCs. Compared to the nonlinear fringe-adjusted JTC, an increase of the recognition rate is also obtained by this optimized scheme.

Table 17. Robustness Results of the Different JTCs [143]

View Table | View all tables in this article

 

Figure 60 Three target images showing different types of mine [137].

Download Full Size | PDF

3.2g. Road Sign Recognition Using Phase-Only Correlation and the VIAPIX Module

Road sign recognition is not only interesting for the computer vision community, but also fundamentally important to address questions of road asset management, navigation assistance, and driver assistance systems. Here we discuss the road sign recognition system based on correlation introduced in [28]. Basically, it combines the VIAPIX device module with a numerical computation of a POF [28]. The VIAPIX module consists of a VIAPIX acquisition device (Fig. 62) and a processing software platform, i.e., VIAPIX Exploitation. First, the street images captured by three cameras in the acquisition unit are combined by the VIAPIX exploitation module to produce a panoramic image covering an angle of 180° view. Figure 63 illustrates several images captured by the three cameras. The corresponding panoramic image generated by VIAPIX is shown in Fig. 64.

 

Figure 61 Results of the optimized nonlinear fringe-adjusted JTC [143].

Download Full Size | PDF

 

Figure 62 VIAPIX acquisition [28].

Download Full Size | PDF

 

Figure 63 Examples of images used to obtain panoramic images [28].

Download Full Size | PDF

Then, the road signs appearing in the panoramic image are extracted by a color segmentation method based on the HSV space. Figure 65 shows an example of red segmentation using the HSV representation. The final recognition task is completed by a POF. In Fig. 66, we show the schematic diagram of this numerical POF, for which the filters are fabricated offline from different sign images and stored. The calculated PCE is chosen as the decisional metric of the recognition method.

 

Figure 64 Panoramic image obtained using images from [28].

Download Full Size | PDF

 

Figure 65 Red color segmentation using HSV [28].

Download Full Size | PDF

Figure 67 shows an experimental panoramic image and the corresponding recognition results [28]. In this experiment, the VIAPIX module is installed on the roof of a vehicle at a speed between 30 and 50 km/h. The acquisition frequency of the image is one image per meter. From Fig. 67, one can see that the POF can identify correctly the road signs appearing in the panoramic images. This recognition system identified 61 road signs from a database of 1500 images containing 72 road signs with a true recognition rate of 84.7% [28].

 

Figure 66 Schematic diagram of the POF used in [28].

Download Full Size | PDF

 

Figure 67 Panoramic image and identification results of POF [28].

Download Full Size | PDF

4. Concluding Remarks and Outlook

From a general perspective, the examples described in this review demonstrate that correlation methods are promising platforms for face and pattern recognition applications. In practice, these correlation-based systems exhibit reliable discrimination and/or real-time tracking performance with very low error rate and high processing speed. However, few applications based on correlation methods have been reported to date. We believe that this is partly because researchers have spent much effort in designing correlation schemes that can be optically implemented. However, as discussed above, the optical implementation of correlation schemes is generally complicated, time consuming, and has low performance versus cost. To achieve a high-efficiency optical recording of a complex-valued filter, a holographic configuration is often introduced, but it renders the implementation system more complicated to process and vulnerable to attackers. Additionally, expensive optoelectronic devices (SLMs and CCDs) need to be employed to perform numerically a flexible algorithm optimization with a PC because optimal optical signals can be processed only in the Fourier plane. Consequently, the low resolution, limited modulation ability, and low processing speed of these optoelectronic devices constitute the main limitations of optical correlators. Furthermore, progress in this field has been hampered by the diffraction effects and electronic noise arising from multiple optoelectronic conversions, which deteriorate the final correlation performance. Although researchers have proposed a number of optical correlators in the past three decades, the technical problems mentioned earlier reduce significantly the value of these devices. Similar advancements as the illustrating examples considered here are expected, but all-optical correlators have not yet been realized. In contrast, numerical implementation can benefit the development and application of correlation algorithms for many reasons. First, the numerical implementation needs only a PC with a high-efficiency processor, thus avoiding the construction of a complicated optical platform and using optoelectronic conversion. Second, many numerical methods can be easily applied to optimize the correlation performance in the input, Fourier, and output planes, e.g., numerical pre-processing of target images for edge detection and correlation post-processing for decision optimization. Third, a more reliable decision making on object’s identity, position, and behavior can be obtained by combining correlation methods and numerical pattern recognition approaches, such as the fuzzy logic techniques.

Thanks to the rapid development of digital processing techniques, we believe that more and more exciting practical applications will be developed in the field of correlation techniques for face and pattern recognition tasks. The high efficiency and flexibility of digital processing methods can greatly enrich the design of correlation techniques. At least four research directions for performance optimization of correlation methods can be of fundamental importance: (1) extracting the feature components representing the training images or test images by using some image representation (e.g., ICA, PCA), artificial neutral network techniques (such as “deep learning” represented by CNNs), and iterative processing methods before the correlation computation step; (2) separating or suppressing some components that may influence correct decisions in the correlation plane after the correlation computation step; (3) designing new decisional criteria that can ensure high recognition rates and robustness against distortion or noisy interference; and (4) combining correlation algorithms with numerical recognition methods. We hope to return to some of these issues in the near future.