Abstract

Although visible face recognition has been an active area of research for several decades, cross-modal face recognition has only been explored by the biometrics community relatively recently. Thermal-to-visible face recognition is one of the most difficult cross-modal face recognition challenges, because of the difference in phenomenology between the thermal and visible imaging modalities. We address the cross-modal recognition problem using a partial least squares (PLS) regression-based approach consisting of preprocessing, feature extraction, and PLS model building. The preprocessing and feature extraction stages are designed to reduce the modality gap between the thermal and visible facial signatures, and facilitate the subsequent one-vs-all PLS-based model building. We incorporate multi-modal information into the PLS model building stage to enhance cross-modal recognition. The performance of the proposed recognition algorithm is evaluated on three challenging datasets containing visible and thermal imagery acquired under different experimental scenarios: time-lapse, physical tasks, mental tasks, and subject-to-camera range. These scenarios represent difficult challenges relevant to real-world applications. We demonstrate that the proposed method performs robustly for the examined scenarios.

© 2015 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Improving cross-modal face recognition using polarimetric imaging

Nathaniel Short, Shuowen Hu, Prudhvi Gurram, Kristan Gurton, and Alex Chan
Opt. Lett. 40(6) 882-885 (2015)

Effects of surface materials on polarimetric-thermal measurements: applications to face recognition

Nathaniel J. Short, Alex J. Yuffa, Gorden Videen, and Shuowen Hu
Appl. Opt. 55(19) 5226-5233 (2016)

Midwave infrared and visible sensor performance modeling: small craft identification discrimination criteria for maritime security

Keith Krapels, Ronald G. Driggers, Dawne Deaver, Steven K. Moyer, and John Palmer
Appl. Opt. 46(30) 7345-7353 (2007)

References

  • View by:
  • |
  • |
  • |

  1. S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
    [Crossref]
  2. L. B. Wolff, D. A. Socolinsky, and C. K. Eveland, “Face recognition in the thermal infrared,” in Computer Vision Beyond the Visible Spectrum (Springer, 2005), pp. 167–191.
  3. D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.
  4. B. Klare and A. K. Jain, “Heterogeneous face recognition: matching NIR to visible light images,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1513–1516.
  5. T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.
  6. F. Nicolo and N. A. Schmid, “Long range cross-spectral face recognition: matching SWIR against visible light images,” IEEE Trans. Inf. Forensics Security 7, 1717–1726, 2012.
    [Crossref]
  7. D. A. Socolinsky and A. Selinger, “A comparative analysis of face recognition performance with visible and thermal infrared imagery,” in Proc. International Conference on Pattern Recognition (IEEE, 2002).
  8. P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
    [Crossref]
  9. S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
    [Crossref]
  10. J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.
  11. T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.
  12. B. F. Klare and A. K. Jain, “Heterogeneous face recognition using kernel prototype similarity,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 1410–1422, 2013.
    [Crossref]
  13. X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” IEEE Trans. Image Process. 19, 374–383, 2010.
    [Crossref]
  14. N. Dalal and B. Triggs, “Histogram of oriented gradients for human detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2005), pp. 886–893.
  15. D. G. Lowe, “Object recognition from local scale-invariant features,” in Proc. International Conference on Computer Vision (IEEE, 1999), pp. 1150–1157.
  16. O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
    [Crossref]
  17. H. Wold, “Estimation of principal components and related models by iterative least squares,” in Multivariate Analysis (Academic, 1966).
  18. W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.
  19. A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
    [Crossref]
  20. A. Sharma and D. Jacobs, “Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 593–600.
  21. W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
    [Crossref]
  22. I. Helland, “Partial least squares regression,” in Encyclopedia of Statistical Sciences (Wiley, 2006), pp. 5957–5962.
  23. H. Abdi, “Partial least squares regression and projection on latent structure regression (PLS Regression),” Wiley Interdisciplinary Reviews: Computational Statistics 2, 433–459 (2010).
    [Crossref]
  24. R. Manne, “Analysis of two partial-least-squares algorithms for multivariate calibration,” Chemometr. Intell. Lab. 2, 187–197, 1987.
    [Crossref]
  25. M. Barker and W. Rayens, “Partial least squares for discrimination,” J. Chemometrics 17, 166–173, 2003.
    [Crossref]
  26. P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
    [Crossref]
  27. X. Chen, P. J. Flynn, and K. W. Bowyer, “Visible-light and Infrared Face Recognition,” in Proc. ACM Workshop on Multimodal User Authentication (ACM, 2003), pp. 48–55.
  28. X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
    [Crossref]
  29. K. A. Byrd, “Preview of the newly acquired NVESD-ARL multimodal face database,” Proc. SPIE 8734, 8734-34, 2013.
  30. L. J. Kozlowski and W. F. Kosonocky, “Infrared detector arrays,” in Handbook of Optics Volume II: Design, Fabrication and Testing, Sources and Detectors, Radiometry and Photometry, 3rd Ed. (Optical Society of America, 2009).
  31. S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” in NIST IR 6281, National Institute of Standards and Technology (NIST, 1998).
  32. J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
    [Crossref]
  33. C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.
  34. D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
    [Crossref]
  35. H. J. Mitchell and C. Salvaggio, “The MWIR and LWIR Spectral Signatures of Water and Associated Materials,” Proc. SPIE 5093, 195–205, 2003.
    [Crossref]
  36. B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

2013 (2)

B. F. Klare and A. K. Jain, “Heterogeneous face recognition using kernel prototype similarity,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 1410–1422, 2013.
[Crossref]

K. A. Byrd, “Preview of the newly acquired NVESD-ARL multimodal face database,” Proc. SPIE 8734, 8734-34, 2013.

2012 (4)

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

F. Nicolo and N. A. Schmid, “Long range cross-spectral face recognition: matching SWIR against visible light images,” IEEE Trans. Inf. Forensics Security 7, 1717–1726, 2012.
[Crossref]

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

2011 (2)

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
[Crossref]

2010 (2)

X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” IEEE Trans. Image Process. 19, 374–383, 2010.
[Crossref]

H. Abdi, “Partial least squares regression and projection on latent structure regression (PLS Regression),” Wiley Interdisciplinary Reviews: Computational Statistics 2, 433–459 (2010).
[Crossref]

2009 (1)

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

2007 (2)

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

2005 (2)

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
[Crossref]

2003 (3)

M. Barker and W. Rayens, “Partial least squares for discrimination,” J. Chemometrics 17, 166–173, 2003.
[Crossref]

P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
[Crossref]

H. J. Mitchell and C. Salvaggio, “The MWIR and LWIR Spectral Signatures of Water and Associated Materials,” Proc. SPIE 5093, 195–205, 2003.
[Crossref]

2001 (1)

J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
[Crossref]

1987 (1)

R. Manne, “Analysis of two partial-least-squares algorithms for multivariate calibration,” Chemometr. Intell. Lab. 2, 187–197, 1987.
[Crossref]

Abdi, H.

H. Abdi, “Partial least squares regression and projection on latent structure regression (PLS Regression),” Wiley Interdisciplinary Reviews: Computational Statistics 2, 433–459 (2010).
[Crossref]

Abidi, B. R.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

Abidi, M. A.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

Barker, M.

M. Barker and W. Rayens, “Partial least squares for discrimination,” J. Chemometrics 17, 166–173, 2003.
[Crossref]

Bazakos, M.

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

Beumer, G. M.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

Boom, B. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

Boughorbel, F.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

Bourlai, T.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

Bowyer, K. W.

X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
[Crossref]

P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
[Crossref]

X. Chen, P. J. Flynn, and K. W. Bowyer, “Visible-light and Infrared Face Recognition,” in Proc. ACM Workshop on Multimodal User Authentication (ACM, 2003), pp. 48–55.

Buddharaju, P.

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

Bueno, G.

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

Byrd, K. A.

K. A. Byrd, “Preview of the newly acquired NVESD-ARL multimodal face database,” Proc. SPIE 8734, 8734-34, 2013.

Chen, C.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

Chen, X.

X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
[Crossref]

X. Chen, P. J. Flynn, and K. W. Bowyer, “Visible-light and Infrared Face Recognition,” in Proc. ACM Workshop on Multimodal User Authentication (ACM, 2003), pp. 48–55.

Choi, J.

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

Chu, R. F.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

Cooper, M.

J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
[Crossref]

Cukic, B.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

Dalal, N.

N. Dalal and B. Triggs, “Histogram of oriented gradients for human detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2005), pp. 886–893.

Davis, L. S.

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
[Crossref]

W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.

De La Torre, F.

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

Deniz, O.

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

Eveland, C. K.

L. B. Wolff, D. A. Socolinsky, and C. K. Eveland, “Face recognition in the thermal infrared,” in Computer Vision Beyond the Visible Spectrum (Springer, 2005), pp. 167–191.

Flynn, P. J.

X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
[Crossref]

P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
[Crossref]

X. Chen, P. J. Flynn, and K. W. Bowyer, “Visible-light and Infrared Face Recognition,” in Proc. ACM Workshop on Multimodal User Authentication (ACM, 2003), pp. 48–55.

Guo, H.

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

Harwood, D.

A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
[Crossref]

W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.

Helland, I.

I. Helland, “Partial least squares regression,” in Encyclopedia of Statistical Sciences (Wiley, 2006), pp. 5957–5962.

Heo, J.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

Hornak, L.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

Hu, S.

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

Jacobs, D.

A. Sharma and D. Jacobs, “Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 593–600.

Jain, A. K.

B. F. Klare and A. K. Jain, “Heterogeneous face recognition using kernel prototype similarity,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 1410–1422, 2013.
[Crossref]

B. Klare and A. K. Jain, “Heterogeneous face recognition: matching NIR to visible light images,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1513–1516.

Kalka, N.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

Kembhavi, A.

A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
[Crossref]

W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.

Klare, B.

B. Klare and A. K. Jain, “Heterogeneous face recognition: matching NIR to visible light images,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1513–1516.

Klare, B. F.

B. F. Klare and A. K. Jain, “Heterogeneous face recognition using kernel prototype similarity,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 1410–1422, 2013.
[Crossref]

Kong, S. G.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

Koschan, A.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

Kosonocky, W. F.

L. J. Kozlowski and W. F. Kosonocky, “Infrared detector arrays,” in Handbook of Optics Volume II: Design, Fabrication and Testing, Sources and Detectors, Radiometry and Photometry, 3rd Ed. (Optical Society of America, 2009).

Kozlowski, L. J.

L. J. Kozlowski and W. F. Kosonocky, “Infrared detector arrays,” in Handbook of Optics Volume II: Design, Fabrication and Testing, Sources and Detectors, Radiometry and Photometry, 3rd Ed. (Optical Society of America, 2009).

Lei, Z.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

Levine, J.

J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
[Crossref]

Li, S. Z.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

Liu, R.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

Lowe, D. G.

D. G. Lowe, “Object recognition from local scale-invariant features,” in Proc. International Conference on Computer Vision (IEEE, 1999), pp. 1150–1157.

Manne, R.

R. Manne, “Analysis of two partial-least-squares algorithms for multivariate calibration,” Chemometr. Intell. Lab. 2, 187–197, 1987.
[Crossref]

Merla, A.

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

Mitchell, H. J.

H. J. Mitchell and C. Salvaggio, “The MWIR and LWIR Spectral Signatures of Water and Associated Materials,” Proc. SPIE 5093, 195–205, 2003.
[Crossref]

Moon, H.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” in NIST IR 6281, National Institute of Standards and Technology (NIST, 1998).

Nicolo, F.

F. Nicolo and N. A. Schmid, “Long range cross-spectral face recognition: matching SWIR against visible light images,” IEEE Trans. Inf. Forensics Security 7, 1717–1726, 2012.
[Crossref]

Olson, L.

C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.

Paik, J.

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

Pavlidis, I.

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
[Crossref]

C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.

Pavlidis, I. T.

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

Phillips, J. P.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” in NIST IR 6281, National Institute of Standards and Technology (NIST, 1998).

Phillips, P. J.

P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
[Crossref]

Puri, C.

C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.

Rayens, W.

M. Barker and W. Rayens, “Partial least squares for discrimination,” J. Chemometrics 17, 166–173, 2003.
[Crossref]

Rizvi, S. A.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” in NIST IR 6281, National Institute of Standards and Technology (NIST, 1998).

Ross, A.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

Salido, J.

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

Salvaggio, C.

H. J. Mitchell and C. Salvaggio, “The MWIR and LWIR Spectral Signatures of Water and Associated Materials,” Proc. SPIE 5093, 195–205, 2003.
[Crossref]

Schmid, N. A.

F. Nicolo and N. A. Schmid, “Long range cross-spectral face recognition: matching SWIR against visible light images,” IEEE Trans. Inf. Forensics Security 7, 1717–1726, 2012.
[Crossref]

Schwartz, W. R.

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.

Selinger, A.

D. A. Socolinsky and A. Selinger, “A comparative analysis of face recognition performance with visible and thermal infrared imagery,” in Proc. International Conference on Pattern Recognition (IEEE, 2002).

Sharma, A.

A. Sharma and D. Jacobs, “Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 593–600.

Shastri, D.

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

Socolinsky, D. A.

D. A. Socolinsky and A. Selinger, “A comparative analysis of face recognition performance with visible and thermal infrared imagery,” in Proc. International Conference on Pattern Recognition (IEEE, 2002).

L. B. Wolff, D. A. Socolinsky, and C. K. Eveland, “Face recognition in the thermal infrared,” in Computer Vision Beyond the Visible Spectrum (Springer, 2005), pp. 167–191.

Spreeuwers, L. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

Starren, J.

C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.

Tan, X.

X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” IEEE Trans. Image Process. 19, 374–383, 2010.
[Crossref]

Triggs, B.

X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” IEEE Trans. Image Process. 19, 374–383, 2010.
[Crossref]

N. Dalal and B. Triggs, “Histogram of oriented gradients for human detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2005), pp. 886–893.

Tsiamyrtzis, P.

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

Veldhuis, N. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

Wold, H.

H. Wold, “Estimation of principal components and related models by iterative least squares,” in Multivariate Analysis (Academic, 1966).

Wolff, L. B.

L. B. Wolff, D. A. Socolinsky, and C. K. Eveland, “Face recognition in the thermal infrared,” in Computer Vision Beyond the Visible Spectrum (Springer, 2005), pp. 167–191.

Yi, D.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

Yi, M.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

Young, S. S.

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

Zheng, Y.

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

Audio and Video-Based Biometric Person Authentication (1)

P. J. Flynn, K. W. Bowyer, and P. J. Phillips, “Assessment of time dependency in face recognition: An initial study,” Audio and Video-Based Biometric Person Authentication 3, 44–51 (2003).
[Crossref]

Chemometr. Intell. Lab. (1)

R. Manne, “Analysis of two partial-least-squares algorithms for multivariate calibration,” Chemometr. Intell. Lab. 2, 187–197, 1987.
[Crossref]

Comput. Vis. Image Und. (2)

X. Chen, P. J. Flynn, and K. W. Bowyer, “IR and visible light face recognition,” Comput. Vis. Image Und. 99, 332–358, 2005.
[Crossref]

S. G. Kong, J. Heo, B. R. Abidi, J. Paik, and M. A. Abidi, “Recent advances in visual and infrared face recognition–a review,” Comput. Vis. Image Und. 97, 103–135 (2005).
[Crossref]

IEEE Trans. Biomed. Eng. (1)

D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis, “Imaging facial signs of neurophysiological responses,” IEEE Trans. Biomed. Eng. 56, 477–484, 2009.
[Crossref]

IEEE Trans. Image Process. (2)

W. R. Schwartz, H. Guo, J. Choi, and L. S. Davis, “Face identification using large feature sets,” IEEE Trans. Image Process. 21, 2245–2255, 2012.
[Crossref]

X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” IEEE Trans. Image Process. 19, 374–383, 2010.
[Crossref]

IEEE Trans. Inf. Forensics Security (1)

F. Nicolo and N. A. Schmid, “Long range cross-spectral face recognition: matching SWIR against visible light images,” IEEE Trans. Inf. Forensics Security 7, 1717–1726, 2012.
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (3)

A. Kembhavi, D. Harwood, and L. S. Davis, “Vehicle detection using partial least squares,” IEEE Trans. Pattern Anal. Mach. Intell. 33, 1250–1265, 2011.
[Crossref]

P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 613–626, 2007.
[Crossref]

B. F. Klare and A. K. Jain, “Heterogeneous face recognition using kernel prototype similarity,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 1410–1422, 2013.
[Crossref]

Int. J. Comput. Vision (1)

S. G. Kong, J. Heo, F. Boughorbel, Y. Zheng, B. R. Abidi, A. Koschan, M. Yi, and M. A. Abidi, “Adaptive fusion of visual and thermal IR images for illumination-invariant face recognition,” Int. J. Comput. Vision 71, 215–233, 2007.
[Crossref]

J. Chemometrics (1)

M. Barker and W. Rayens, “Partial least squares for discrimination,” J. Chemometrics 17, 166–173, 2003.
[Crossref]

Lancet (1)

J. Levine, I. Pavlidis, and M. Cooper, “The face of fear,” Lancet 357, 1757 (2001).
[Crossref]

Pattern Recogn. Lett. (1)

O. Deniz, G. Bueno, J. Salido, and F. De La Torre, “Face recognition using histogram of oriented gradients,” Pattern Recogn. Lett. 32, 1598–1603, 2011.
[Crossref]

Proc. SPIE (4)

J. Choi, S. Hu, S. S. Young, and L. S. Davis, “Thermal to visible face recognition,” Proc. SPIE 8371, 83711L, 2012.

T. Bourlai, A. Ross, C. Chen, and L. Hornak, “A study on using mid-wave infrared images for face recognition,” Proc. SPIE 8371, 83711K, 2012.

H. J. Mitchell and C. Salvaggio, “The MWIR and LWIR Spectral Signatures of Water and Associated Materials,” Proc. SPIE 5093, 195–205, 2003.
[Crossref]

K. A. Byrd, “Preview of the newly acquired NVESD-ARL multimodal face database,” Proc. SPIE 8734, 8734-34, 2013.

Wiley Interdisciplinary Reviews: Computational Statistics (1)

H. Abdi, “Partial least squares regression and projection on latent structure regression (PLS Regression),” Wiley Interdisciplinary Reviews: Computational Statistics 2, 433–459 (2010).
[Crossref]

Other (16)

I. Helland, “Partial least squares regression,” in Encyclopedia of Statistical Sciences (Wiley, 2006), pp. 5957–5962.

X. Chen, P. J. Flynn, and K. W. Bowyer, “Visible-light and Infrared Face Recognition,” in Proc. ACM Workshop on Multimodal User Authentication (ACM, 2003), pp. 48–55.

L. J. Kozlowski and W. F. Kosonocky, “Infrared detector arrays,” in Handbook of Optics Volume II: Design, Fabrication and Testing, Sources and Detectors, Radiometry and Photometry, 3rd Ed. (Optical Society of America, 2009).

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” in NIST IR 6281, National Institute of Standards and Technology (NIST, 1998).

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proc. 7th Int. Conference on Control, Automation, Robotics, and Vision (IEEE, 2006).

C. Puri, L. Olson, I. Pavlidis, and J. Starren, “Stresscam: non-contact measurement of users’ emotional states through thermal imaging,” in Proc. 2005 ACM Conference on Human Factors in Computing Systems (CHI) (ACM, 2005), pp. 1725–1728.

L. B. Wolff, D. A. Socolinsky, and C. K. Eveland, “Face recognition in the thermal infrared,” in Computer Vision Beyond the Visible Spectrum (Springer, 2005), pp. 167–191.

D. Yi, R. Liu, R. F. Chu, Z. Lei, and S. Z. Li, “Face matching between near infrared and visible light images,” in Advances in Biometrics: Lecture Notes in Computer Science (Springer, 2007), Vol. 4642, pp. 523–530.

B. Klare and A. K. Jain, “Heterogeneous face recognition: matching NIR to visible light images,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1513–1516.

T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak, “Cross-spectral face verification in the short wave infrared (SWIR) band,” in Proc. International Conference on Pattern Recognition (IEEE, 2010), pp. 1343–1347.

H. Wold, “Estimation of principal components and related models by iterative least squares,” in Multivariate Analysis (Academic, 1966).

W. R. Schwartz, A. Kembhavi, D. Harwood, and L. S. Davis, “Human detection using partial least squares analysis,” in Proc. IEEE International Conference on Computer Vision (IEEE, 2009), pp. 24–31.

A. Sharma and D. Jacobs, “Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 593–600.

D. A. Socolinsky and A. Selinger, “A comparative analysis of face recognition performance with visible and thermal infrared imagery,” in Proc. International Conference on Pattern Recognition (IEEE, 2002).

N. Dalal and B. Triggs, “Histogram of oriented gradients for human detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2005), pp. 886–893.

D. G. Lowe, “Object recognition from local scale-invariant features,” in Proc. International Conference on Computer Vision (IEEE, 1999), pp. 1150–1157.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1.
Fig. 1. Face signatures of a subject from the NVESD dataset simultaneously acquired in visible, SWIR, MWIR, and LWIR.
Fig. 2.
Fig. 2. Raw thermal and visible images with manually labeled fiducial points from the WSRI dataset are geometrically normalized to canonical coordinates and cropped.
Fig. 3.
Fig. 3. Comparison of aligned thermal and visible images, showing original intensity images, as well as after DOG filtering and contrast normalization.
Fig. 4.
Fig. 4. Illustration of one-vs-all PLS model building. Note that face images shown here actually represent the feature vectors used for model building.
Fig. 5.
Fig. 5. Sample visible and corresponding LWIR images of a subject from the UND X1 database with neutral expression and smiling expression.
Fig. 6.
Fig. 6. Rank-1 identification rate as a function of the number of thermal cross-examples for the UND X1 dataset.
Fig. 7.
Fig. 7. Cumulative match characteristics curves showing Rank-1 to Rank-15 performance for UND X1 dataset.
Fig. 8.
Fig. 8. Cumulative match characteristics curves showing Rank-1 to Rank-15 performance for the overall evaluation on the WSRI dataset.
Fig. 9.
Fig. 9. Preprocessed MWIR and LWIR images at 1 m, 2 m, and 4 m ranges for a subject from the NVESD dataset. All images were resized to 272 × 322 for display.
Fig. 10.
Fig. 10. Cumulative match characteristics curve showing Rank-1 to Rank-15 performance at three ranges for (a) MWIR-to-visible face recognition, and (b) LWIR-to-visible face recognition for the NVESD dataset.
Fig. 11.
Fig. 11. ROC curves showing thermal-to-visible verification performance for UND X1 dataset, WSRI dataset, and NVESD dataset (with separate curves for MWIR and LWIR).
Fig. 12.
Fig. 12. For the one-vs-all PLS model belonging to the subject shown at the left, the heat maps for the weight vectors w 1 and w 2 are computed without and with thermal cross-examples. The mean square difference between the without and with weight vectors is displayed at the right column.

Tables (5)

Tables Icon

Table 1. Summary of Sensor Resolutions for UND Collection X1, WSRI Dataset, and NVESD Dataset

Tables Icon

Table 2. Mean and Standard Deviation (in Parentheses) of Eye-to-Eye Pixel Distance across Subjects for NVESD Dataset

Tables Icon

Table 3. Number of Subjects Who Participated in Each Condition, Total Number Of Thermal Images for Each Condition, and the Corresponding Rank-1 Identification Rate for WSRI Dataset

Tables Icon

Table 4. MWIR-to-Visible Rank-1 Identification Rate Using Before Exercise MWIR Probe Imagery and after Exercise MWIR Probe Imagery with Thermal Cross-Examples for NVESD Dataset

Tables Icon

Table 5. LWIR-to-Visible Rank-1 Identification Rate Using before Exercise LWIR Probe Imagery and after Exercise LWIR Probe Imagery with Thermal Cross-Examples for NVESD Dataset

Equations (6)

Equations on this page are rendered with MathJax. Learn more.

I DOG ( x , y | σ 0 , σ 1 ) = [ 1 2 π σ 0 2 e ( x 2 + y 2 ) / 2 σ 0 2 1 2 π σ 1 2 e ( x 2 + y 2 ) / 2 σ 1 2 ] * I ( x , y )
I C ( x , y ) = τ [ tanh ( I DOG ( x , y ) τ ) ] .
X = TP T + X res ,
y = Uq T + y res .
max [ cov ( t i , u i ) ] 2 = max | w i = 1 | [ cov ( Xw i , y ) ] 2 i = 1 , , p .
y f = y ¯ + β T f = y ¯ + [ W ( P T W ) 1 T T y ] f .

Metrics