Abstract

With the prevalence of surveillance systems, face recognition is crucial to aiding the law enforcement community and homeland security in identifying suspects and suspicious individuals on watch lists. However, face recognition performance is severely affected by the low face resolution of individuals in typical surveillance footage, oftentimes due to the distance of individuals from the cameras as well as the small pixel count of low-cost surveillance systems. Superresolution image reconstruction has the potential to improve face recognition performance by using a sequence of low-resolution images of an individual’s face in the same pose to reconstruct a more detailed high-resolution facial image. This work conducts an extensive performance evaluation of superresolution for a face recognition algorithm using a methodology and experimental setup consistent with real world settings at multiple subject-to-camera distances. Results show that superresolution image reconstruction improves face recognition performance considerably at the examined midrange and close range.

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.
  2. D. M. Blackburn, M. Bone, and P. J. Phillips, “Facial Recognition Vendor Test 2000,” http://www.frvt.org/FRVT2000/ .
  3. Pennsylvania Justice Network, “JNET facial recognition investigative search tool and watchlist,” http://www.pajnet.state.pa.us/ .
  4. T. E. Boult, M.-C. Chiang, and R. J. Micheals, “Super-resolution via image warping,” in Super-Resolution Imaging, S. Chaudhuri, ed. (Springer, 2001), pp. 131–169.
  5. S. Baker and T. Kanade, “Hallucinating faces,” in Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (IEEE, 2000), pp. 83–88.
  6. F. W. Wheeler, X. Liu, and P. H. Tu, “Multi-frame super-resolution for face recognition,” in Proceedings of IEEE 1st International Conference on Biometrics: Theory, Applications and Systems (IEEE, 2007), pp. 1–6.
  7. B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
    [CrossRef]
  8. P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar, “Simultaneous super-resolution and feature extraction for recognition of low-resolution faces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.
  9. H. Huang and H. He, “Super-resolution method for face recognition using nonlinear mappings on coherent features,” IEEE Trans. Neural Netw. 22, 121–130 (2011).
    [CrossRef]
  10. C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
    [CrossRef]
  11. D. S. Bolme and J. R. Beveridge, “CSU LRPCA baseline algorithm,” www.cs.colostate.edu/facerec/algorithms/lrpca2010.php .
  12. D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
    [CrossRef]
  13. S. S. Young and R. G. Driggers, “Super-resolution image reconstruction from a sequence of aliased imagery,” Appl. Opt. 45, 5073–5085 (2006).
    [CrossRef]
  14. A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
    [CrossRef]
  15. S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” NIST IR 6281 (National Institute of Standards and Technology, 1998).
  16. R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
    [CrossRef]

2012 (1)

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

2011 (1)

H. Huang and H. He, “Super-resolution method for face recognition using nonlinear mappings on coherent features,” IEEE Trans. Neural Netw. 22, 121–130 (2011).
[CrossRef]

2006 (1)

2005 (1)

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

2004 (1)

R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
[CrossRef]

2003 (2)

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Abdi, H.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Altunbasak, Y.

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

Ayyad, J. H.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Baker, S.

S. Baker and T. Kanade, “Hallucinating faces,” in Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (IEEE, 2000), pp. 83–88.

P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar, “Simultaneous super-resolution and feature extraction for recognition of low-resolution faces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Batur, A. U.

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

Beumer, G. M.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.

Beveridge, J. R.

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Bolle, R. M.

R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
[CrossRef]

Bolme, D. S.

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Boom, B. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.

Boult, T. E.

T. E. Boult, M.-C. Chiang, and R. J. Micheals, “Super-resolution via image warping,” in Super-Resolution Imaging, S. Chaudhuri, ed. (Springer, 2001), pp. 131–169.

Chandran, V.

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

Chiang, M.-C.

T. E. Boult, M.-C. Chiang, and R. J. Micheals, “Super-resolution via image warping,” in Super-Resolution Imaging, S. Chaudhuri, ed. (Springer, 2001), pp. 131–169.

Draper, B. A.

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Driggers, R. G.

Fookes, C.

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

Gunturk, B. K.

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

Harms, J.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Hayes, M. H.

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

He, H.

H. Huang and H. He, “Super-resolution method for face recognition using nonlinear mappings on coherent features,” IEEE Trans. Neural Netw. 22, 121–130 (2011).
[CrossRef]

Hennings-Yeomans, P. H.

P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar, “Simultaneous super-resolution and feature extraction for recognition of low-resolution faces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Huang, H.

H. Huang and H. He, “Super-resolution method for face recognition using nonlinear mappings on coherent features,” IEEE Trans. Neural Netw. 22, 121–130 (2011).
[CrossRef]

Hurst, D. R.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Kanade, T.

S. Baker and T. Kanade, “Hallucinating faces,” in Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (IEEE, 2000), pp. 83–88.

Kumar, B. V. K. V.

P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar, “Simultaneous super-resolution and feature extraction for recognition of low-resolution faces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Lin, F.

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

Liu, X.

F. W. Wheeler, X. Liu, and P. H. Tu, “Multi-frame super-resolution for face recognition,” in Proceedings of IEEE 1st International Conference on Biometrics: Theory, Applications and Systems (IEEE, 2007), pp. 1–6.

Mersereau, R. M.

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

Micheals, R. J.

T. E. Boult, M.-C. Chiang, and R. J. Micheals, “Super-resolution via image warping,” in Super-Resolution Imaging, S. Chaudhuri, ed. (Springer, 2001), pp. 131–169.

Moon, H.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” NIST IR 6281 (National Institute of Standards and Technology, 1998).

O’Toole, A. J.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Pankanti, S.

R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
[CrossRef]

Pappas, M. R.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Phillips, J. P.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” NIST IR 6281 (National Institute of Standards and Technology, 1998).

Ratha, N. K.

R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
[CrossRef]

Rizvi, S. A.

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” NIST IR 6281 (National Institute of Standards and Technology, 1998).

Snow, S. L.

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

Spreeuwers, L. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.

Sridharan, S.

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

Teixeria, M.

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Tu, P. H.

F. W. Wheeler, X. Liu, and P. H. Tu, “Multi-frame super-resolution for face recognition,” in Proceedings of IEEE 1st International Conference on Biometrics: Theory, Applications and Systems (IEEE, 2007), pp. 1–6.

Veldhuis, N. J.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.

Wheeler, F. W.

F. W. Wheeler, X. Liu, and P. H. Tu, “Multi-frame super-resolution for face recognition,” in Proceedings of IEEE 1st International Conference on Biometrics: Theory, Applications and Systems (IEEE, 2007), pp. 1–6.

Young, S. S.

Appl. Opt. (1)

Comput. Vis. Image Underst. (1)

R. M. Bolle, N. K. Ratha, and S. Pankanti, “Error analysis of pattern recognition systems—the subsets bootstrap,” Comput. Vis. Image Underst. 93, 1–33 (2004).
[CrossRef]

IEEE Trans. Image Process. (1)

B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau, “Eigenface-domain super-resolution for face recognition,” IEEE Trans. Image Process. 12, 597–606 (2003).
[CrossRef]

IEEE Trans. Neural Netw. (1)

H. Huang and H. He, “Super-resolution method for face recognition using nonlinear mappings on coherent features,” IEEE Trans. Neural Netw. 22, 121–130 (2011).
[CrossRef]

IEEE Trans. Pattern Anal. Machine Intell. (1)

A. J. O’Toole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H. Ayyad, and H. Abdi, “A video database of moving faces and people,” IEEE Trans. Pattern Anal. Machine Intell. 27, 812–816 (2005).
[CrossRef]

J. Vis. Commun. Image Represent. (1)

C. Fookes, F. Lin, V. Chandran, and S. Sridharan, “Evaluation of image resolution and super-resolution on face recognition performance,” J. Vis. Commun. Image Represent. 23, 75–93 (2012).
[CrossRef]

Lect. Notes Comput. Sci. (1)

D. S. Bolme, J. R. Beveridge, M. Teixeria, and B. A. Draper, “The CSU face identification evaluation system: its purpose, features, and structure,” Lect. Notes Comput. Sci. 2626, 304–313 (2003).
[CrossRef]

Other (9)

S. A. Rizvi, J. P. Phillips, and H. Moon, “The FERET verification testing protocol for face recognition algorithms,” NIST IR 6281 (National Institute of Standards and Technology, 1998).

D. S. Bolme and J. R. Beveridge, “CSU LRPCA baseline algorithm,” www.cs.colostate.edu/facerec/algorithms/lrpca2010.php .

P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar, “Simultaneous super-resolution and feature extraction for recognition of low-resolution faces,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

B. J. Boom, G. M. Beumer, L. J. Spreeuwers, and N. J. Veldhuis, “The effect of image resolution on the performance of face recognition system,” in Proceedings of the 7th International Conference on Control, Automation, Robotics, and Vision (IEEE, 2006), pp. 1–6.

D. M. Blackburn, M. Bone, and P. J. Phillips, “Facial Recognition Vendor Test 2000,” http://www.frvt.org/FRVT2000/ .

Pennsylvania Justice Network, “JNET facial recognition investigative search tool and watchlist,” http://www.pajnet.state.pa.us/ .

T. E. Boult, M.-C. Chiang, and R. J. Micheals, “Super-resolution via image warping,” in Super-Resolution Imaging, S. Chaudhuri, ed. (Springer, 2001), pp. 131–169.

S. Baker and T. Kanade, “Hallucinating faces,” in Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (IEEE, 2000), pp. 83–88.

F. W. Wheeler, X. Liu, and P. H. Tu, “Multi-frame super-resolution for face recognition,” in Proceedings of IEEE 1st International Conference on Biometrics: Theory, Applications and Systems (IEEE, 2007), pp. 1–6.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1.
Fig. 1.

Sample frame extracted from a subject’s parallel gait video in the database of moving faces and people [14]. Subject is at the far range (resulting eye-to-eye distance of 5–10 pixels).

Fig. 2.
Fig. 2.

(a) Close range face image of a subject, (b) close range face image downsampled by a factor of 3 to simulate far range using procedure of Fookes et al. [10], and (c) far range face image of the subject taken from the same video.

Fig. 3.
Fig. 3.

Low-resolution (LR) imagery and superresolved imagery (4 frames—SR4, 8 frames—SR8) at eye-to-eye distances of 5–10, 15–20, and 25–30 pixels. All images at all ranges have been resized to a fixed size for comparison.

Fig. 4.
Fig. 4.

Computed kx-domain cumulative power spectrums of LR eye region and SR8 eye region at the midrange. The circled part of the plot represents high-frequency band recovered from using a sequence of eight aliased low-resolution frames.

Fig. 5.
Fig. 5.

Pixel intensity value plots of LR and SR8 along a profile across the eye region at the midrange, showing the improved edge contrast with SR8 in the spatial domain.

Fig. 6.
Fig. 6.

ROC curves at the far range for original low resolution (LR510) query set and the corresponding superresolved query sets using four (SR4510) and eight (SR8510) frames.

Fig. 7.
Fig. 7.

ROC curves at the midrange for original low resolution (LR1520) query set and the corresponding superresolved query sets using four (SR41520) and eight (SR81520) frames.

Fig. 8.
Fig. 8.

ROC curves at the close-range for original low resolution (LR2530) query set and the corresponding superresolved query sets using four (SR42530) and eight (SR82530) frames.

Fig. 9.
Fig. 9.

Performance as a function of range at FARs of (a) 0.01 and (b) 0.05. Error bars show the 95% confidence interval for each correct verification rate.

Fig. 10.
Fig. 10.

ROC curves at the far-range for each low resolution frame (superscript 1–8). The ROC curve for LRave is generated by averaging similarity matrices of the eight individual frames and generating the ROC curve.

Fig. 11.
Fig. 11.

ROC curves at the midrange for each low resolution frame (superscript 1–8).

Fig. 12.
Fig. 12.

ROC curves at the close-range for each low resolution frame (superscript 1–8).

Fig. 13.
Fig. 13.

Performance as a function of range at FARs of (a) 0.01 and (b) 0.05. Error bars show the 95% confidence interval for each correct verification rate (not shown for LRave because LRave represents averaged similarity scores, and not actual similarity measurements). LR* denotes the best of the eight LR frame (in terms of AUC) at each range.

Tables (3)

Tables Icon

Table 1. Query Set Nomenclaturea

Tables Icon

Table 2. Query Set Nomenclature for Evaluation of Face Recognition with Respect to Low-Resolution Framea

Tables Icon

Table 3. Area under the Curves for LRi, Where i[1,8] Denotes the Frame Number, AUC for LRave (Computed from the ROC of the Average Similarity Scores across the Eight Frames), and AUC for SR8 at Far, Mid-, and Close Rangesa

Equations (3)

Equations on this page are rendered with MathJax. Learn more.

F^(t0)=1Mi=1M1(Xit0),
S1(kx)=ky|F(kx,ky)|2,
S2(ky)=kx|F(kx,ky)|2.

Metrics