Abstract

In this paper we present a novel health monitoring method by estimating the heart rate and respiratory rate using an RGB camera. The heart rate and the respiratory rate are estimated from the photoplethysmography (PPG) and the respiratory motion. The method mainly operates by using the green spectrum of the RGB camera to generate a multivariate PPG signal to perform multivariate de-noising on the video signal to extract the resultant PPG signal. A periodicity based voting scheme (PVS) was used to measure the heart rate and respiratory rate from the estimated PPG signal. We evaluated our proposed method with a state of the art heart rate measuring method for two scenarios using the MAHNOB-HCI database and a self collected naturalistic environment database. The methods were furthermore evaluated for various scenarios at naturalistic environments such as a motion variance session and a skin tone variance session. Our proposed method operated robustly during the experiments and outperformed the state of the art heart rate measuring methods by compensating the effects of the naturalistic environment.

© 2017 Optical Society of America

Full Article  |  PDF Article

Corrections

9 November 2017: A typographical correction was made to the article title.


OSA Recommended Articles
DistancePPG: Robust non-contact vital signs monitoring using a camera

Mayank Kumar, Ashok Veeraraghavan, and Ashutosh Sabharwal
Biomed. Opt. Express 6(5) 1565-1588 (2015)

Detail-preserving pulse wave extraction from facial videos using consume-level camera

Dingliang Wang, Xuezhi Yang, Xuenan Liu, Jin Jing, and Shuai Fang
Biomed. Opt. Express 11(4) 1876-1891 (2020)

References

  • View by:
  • |
  • |
  • |

  1. M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation,” Opt. Express 18 (10), 10762–10774 (2010).
    [Crossref] [PubMed]
  2. M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
    [Crossref]
  3. M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.
  4. S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (IEEE, 2012), pp. 2174–2177.
  5. T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in: 2012 IEEE International Conference On Consumer Electronics, (IEEE, 2012), pp. 544–545.
  6. H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
    [Crossref]
  7. Y.-P. Yu, P. Raveendran, and C.-L. Lim, “Heart rate estimation from facial images using filter bank,” in: Communications, Control and Signal Processing, 2014 6th International Symposium on, (IEEE, 2014), pp. 69–72.
  8. X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.
  9. A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in: Proceedings of the IEEE International Conference on Computer Vision, (IEEE, 2015), pp. 3640–3648.
  10. M. Kumar, A. Veeraraghavan, and A. Sabharwal, Distanceppg: “Robust non-contact vital signs monitoring using a camera,” Biomed. Opt. Express 6 (5), 1565–1588 (2015).
    [Crossref] [PubMed]
  11. L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
    [Crossref]
  12. S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.
  13. M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
    [Crossref]
  14. G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomedical Engineering 60 (10), 2878–2886 (2013).
    [Crossref]
  15. A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6 (9), 3320–3338 (2015).
    [Crossref] [PubMed]
  16. C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
    [Crossref]
  17. W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
    [Crossref]
  18. W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).
  19. Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Vol. 5, (Prentice HallUpper Saddle River, 2002).
  20. A. Krishnaswamy and G. V. Baranoski, A study on skin optics, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo, Canada, Technical Report. 1, 1–17 (2004).
  21. T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
    [Crossref]
  22. G. R. Cooper and C. D. McGillem, Probabilistic Methods of Signal and System Analysis (Oxford University Press, 1986).
  23. S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way (Academic Press, 2008).
  24. A. Antoniadis, “Wavelets in statistics: a review,” Journal of the Italian Statistical Society 6 (2), 97–130 (1997).
    [Crossref]
  25. W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
    [Crossref]
  26. M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
    [Crossref]
  27. S. Mallat, A Wavelet Tour of Signal Processing (Academic Press, 1999).
  28. S. Mallat and W. L. Hwang, “Singularity detection and processing with wavelets,” IEEE transactions on information theory 38 (2), 617–643 (1992).
    [Crossref]
  29. P. J. Rousseeuw and K. V. Driessen, “A fast algorithm for the minimum covariance determinant estimator,” Technometrics 41 (3), 212–223 (1999).
    [Crossref]
  30. J. Shi and C. Tomasi, “Good features to track,” in: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, (IEEE, 1994), pp. 593–600.
  31. L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.
  32. M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
    [Crossref]
  33. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
    [Crossref]

2017 (2)

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

2016 (2)

W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
[Crossref]

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

2015 (4)

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
[Crossref]

A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6 (9), 3320–3338 (2015).
[Crossref] [PubMed]

M. Kumar, A. Veeraraghavan, and A. Sabharwal, Distanceppg: “Robust non-contact vital signs monitoring using a camera,” Biomed. Opt. Express 6 (5), 1565–1588 (2015).
[Crossref] [PubMed]

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

2014 (1)

H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
[Crossref]

2013 (1)

G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomedical Engineering 60 (10), 2878–2886 (2013).
[Crossref]

2012 (3)

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
[Crossref]

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

2011 (1)

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
[Crossref]

2010 (1)

2006 (1)

M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
[Crossref]

1999 (1)

P. J. Rousseeuw and K. V. Driessen, “A fast algorithm for the minimum covariance determinant estimator,” Technometrics 41 (3), 212–223 (1999).
[Crossref]

1997 (1)

A. Antoniadis, “Wavelets in statistics: a review,” Journal of the Italian Statistical Society 6 (2), 97–130 (1997).
[Crossref]

1992 (1)

S. Mallat and W. L. Hwang, “Singularity detection and processing with wavelets,” IEEE transactions on information theory 38 (2), 617–643 (1992).
[Crossref]

Alameda-Pineda, X.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Ali, Y. S.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Aminghafari, M.

M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
[Crossref]

Antoniadis, A.

A. Antoniadis, “Wavelets in statistics: a review,” Journal of the Italian Statistical Society 6 (2), 97–130 (1997).
[Crossref]

Baranoski, G. V.

A. Krishnaswamy and G. V. Baranoski, A study on skin optics, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo, Canada, Technical Report. 1, 1–17 (2004).

Bernacchia, N.

L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.

Bowles, H. R.

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

Calvo, R.

H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
[Crossref]

Chappell, P. H.

T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
[Crossref]

Chen, J.

X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.

Cheze, N.

M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
[Crossref]

Cohn, J. F.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Cooper, G. R.

G. R. Cooper and C. D. McGillem, Probabilistic Methods of Signal and System Analysis (Oxford University Press, 1986).

Daly, J.

de Haan, G.

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
[Crossref]

G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomedical Engineering 60 (10), 2878–2886 (2013).
[Crossref]

den Brinker, B.

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

Driessen, K. V.

P. J. Rousseeuw and K. V. Driessen, “A fast algorithm for the minimum covariance determinant estimator,” Technometrics 41 (3), 212–223 (1999).
[Crossref]

Ercoli, I.

L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.

Feng, L.

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

Fofi, D.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Frise, M. C.

Guazzi, A. R.

Hagströmer, M.

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

Haque, M. A.

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

Hassan, M. A.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Hwang, W. L.

S. Mallat and W. L. Hwang, “Singularity detection and processing with wavelets,” IEEE transactions on information theory 38 (2), 617–643 (1992).
[Crossref]

Irani, R.

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

Jeanne, V.

G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomedical Engineering 60 (10), 2878–2886 (2013).
[Crossref]

Jorge, J.

Kim, H.

S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (IEEE, 2012), pp. 2174–2177.

Kocejko, T.

M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.

Krajewski, J.

T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in: 2012 IEEE International Conference On Consumer Electronics, (IEEE, 2012), pp. 544–545.

Krishnaswamy, A.

A. Krishnaswamy and G. V. Baranoski, A study on skin optics, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo, Canada, Technical Report. 1, 1–17 (2004).

Kumar, M.

Kuno, Y.

A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in: Proceedings of the IEEE International Conference on Computer Vision, (IEEE, 2015), pp. 3640–3648.

Kwon, S.

S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (IEEE, 2012), pp. 2174–2177.

Lam, A.

A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in: Proceedings of the IEEE International Conference on Computer Vision, (IEEE, 2015), pp. 3640–3648.

Lewandowska, M.

M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.

Li, X.

X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.

Li, Y.

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

Lichtenauer, J.

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

Lim, C.-L.

Y.-P. Yu, P. Raveendran, and C.-L. Lim, “Heart rate estimation from facial images using filter bank,” in: Communications, Control and Signal Processing, 2014 6th International Symposium on, (IEEE, 2014), pp. 69–72.

Lister, T.

T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
[Crossref]

Ma, R.

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

Malik, A. S.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Mallat, S.

S. Mallat and W. L. Hwang, “Singularity detection and processing with wavelets,” IEEE transactions on information theory 38 (2), 617–643 (1992).
[Crossref]

S. Mallat, A Wavelet Tour of Signal Processing (Academic Press, 1999).

S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way (Academic Press, 2008).

Marchionni, P.

L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.

Matthews, C. E.

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

McDuff, D. J.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
[Crossref]

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation,” Opt. Express 18 (10), 10762–10774 (2010).
[Crossref] [PubMed]

McGillem, C. D.

G. R. Cooper and C. D. McGillem, Probabilistic Methods of Signal and System Analysis (Oxford University Press, 1986).

Meriaudeau, F.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Moeller, R.

T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in: 2012 IEEE International Conference On Consumer Electronics, (IEEE, 2012), pp. 544–545.

Moeslund, T. B.

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

Monkaresi, H.

H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
[Crossref]

Nasrollahi, K.

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

Nowak, J.

M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.

Ostermann, J.

Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Vol. 5, (Prentice HallUpper Saddle River, 2002).

Pantic, M.

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

Park, K. S.

S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (IEEE, 2012), pp. 2174–2177.

Picard, R. W.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
[Crossref]

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation,” Opt. Express 18 (10), 10762–10774 (2010).
[Crossref] [PubMed]

Pietikainen, M.

X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.

Po, L.-M.

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

Pober, D. M.

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

Poggi, J.-M.

M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
[Crossref]

Poh, M.-Z.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
[Crossref]

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation,” Opt. Express 18 (10), 10762–10774 (2010).
[Crossref] [PubMed]

Pun, T.

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

Pursche, T.

T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in: 2012 IEEE International Conference On Consumer Electronics, (IEEE, 2012), pp. 544–545.

Raveendran, P.

Y.-P. Yu, P. Raveendran, and C.-L. Lim, “Heart rate estimation from facial images using filter bank,” in: Communications, Control and Signal Processing, 2014 6th International Symposium on, (IEEE, 2014), pp. 69–72.

Ricci, E.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Robbins, P. A.

Rousseeuw, P. J.

P. J. Rousseeuw and K. V. Driessen, “A fast algorithm for the minimum covariance determinant estimator,” Technometrics 41 (3), 212–223 (1999).
[Crossref]

Rumiski, J.

M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.

Saad, N. M.

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

Sabharwal, A.

Scalise, L.

L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.

Sebe, N.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Shi, J.

J. Shi and C. Tomasi, “Good features to track,” in: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, (IEEE, 1994), pp. 593–600.

Soleymani, M.

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

Stuijk, S.

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
[Crossref]

Tarassenko, L.

Tomasi, C.

J. Shi and C. Tomasi, “Good features to track,” in: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, (IEEE, 1994), pp. 593–600.

Tulyakov, S.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Veeraraghavan, A.

Villarroel, M.

Wang, W.

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
[Crossref]

Wang, Y.

Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Vol. 5, (Prentice HallUpper Saddle River, 2002).

Wright, P. A.

T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
[Crossref]

Xu, X.

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

Yan, H.

H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
[Crossref]

Yin, L.

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

Yu, Y.-P.

Y.-P. Yu, P. Raveendran, and C.-L. Lim, “Heart rate estimation from facial images using filter bank,” in: Communications, Control and Signal Processing, 2014 6th International Symposium on, (IEEE, 2014), pp. 69–72.

Zhang, Y.-Q.

Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Vol. 5, (Prentice HallUpper Saddle River, 2002).

Zhao, G.

X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.

Biomed. Opt. Express (2)

Computational Statistics Data Analysis (1)

M. Aminghafari, N. Cheze, and J.-M. Poggi, “Multivariate denoising using wavelets and principal component analysis,” Computational Statistics Data Analysis 50, 2381–2398 (2006).
[Crossref]

IEEE Intelligent Systems (1)

M. A. Haque, R. Irani, K. Nasrollahi, and T. B. Moeslund, “Heartbeat rate measurement from facial video,” IEEE Intelligent Systems 31 (3), 40–48 (2016).
[Crossref]

IEEE J. Biomed. Health Informatics (1)

H. Monkaresi, R. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE J. Biomed. Health Informatics 18 (4) 1153–1160 (2014).
[Crossref]

IEEE Sensors Journal (1)

M. A. Hassan, A. S. Malik, D. Fofi, N. M. Saad, Y. S. Ali, and F. Meriaudeau, “Video-based heartbeat rate measuring method using ballistocardiography,” IEEE Sensors Journal 17 (14), 4544–4557 (2017).
[Crossref]

IEEE Trans. Biomed. Engineering (1)

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Engineering 58 (1), 7–11 (2011).
[Crossref]

IEEE Trans. Biomedical Engineering (3)

G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomedical Engineering 60 (10), 2878–2886 (2013).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomedical Engineering 62 (2), 415–425 (2015).
[Crossref]

W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomedical Engineering 63 (9), 1974–1984 (2016).
[Crossref]

IEEE Trans. Biomedical Engineering. (1)

W. Wang, B. den Brinker, S. Stuijk, and G. de Haan, “lgorithmic principles of remote-ppg,” IEEE Trans. Biomedical Engineering. 64 (7), 1479–1491 (2017).

IEEE Trans. Circuits and Systems for Video Technology (1)

L. Feng, L.-M. Po, X. Xu, Y. Li, and R. Ma, “Motion-resistant remote imaging photoplethysmography based on the optical properties of skin,” IEEE Trans. Circuits and Systems for Video Technology 25 (5), 879–891 (2015).
[Crossref]

IEEE Transactions on Affective Computing (1)

M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing 3 (1), 42–55 (2012).
[Crossref]

IEEE transactions on information theory (1)

S. Mallat and W. L. Hwang, “Singularity detection and processing with wavelets,” IEEE transactions on information theory 38 (2), 617–643 (1992).
[Crossref]

J. Biomed. Opt. (1)

T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human skin,” J. Biomed. Opt. 17 (9), 0909011 (2012).
[Crossref]

Journal of the Italian Statistical Society (1)

A. Antoniadis, “Wavelets in statistics: a review,” Journal of the Italian Statistical Society 6 (2), 97–130 (1997).
[Crossref]

Medicine and science in sports and exercise (1)

C. E. Matthews, M. Hagströmer, D. M. Pober, and H. R. Bowles, “Best practices for using physical activity monitors in population-based research,” Medicine and science in sports and exercise 44 (1 Suppl 1), S68 (2012).
[Crossref]

Opt. Express (1)

Technometrics (1)

P. J. Rousseeuw and K. V. Driessen, “A fast algorithm for the minimum covariance determinant estimator,” Technometrics 41 (3), 212–223 (1999).
[Crossref]

Other (14)

J. Shi and C. Tomasi, “Good features to track,” in: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, (IEEE, 1994), pp. 593–600.

L. Scalise, N. Bernacchia, I. Ercoli, and P. Marchionni, “Heart rate measurement in neonatal patients using a webcamera,” in: Medical Measurements and Applications Proceedings, 2012 IEEE International Symposium on, (IEEE, 2012), pp. 1–4.

S. Mallat, A Wavelet Tour of Signal Processing (Academic Press, 1999).

G. R. Cooper and C. D. McGillem, Probabilistic Methods of Signal and System Analysis (Oxford University Press, 1986).

S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way (Academic Press, 2008).

Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Vol. 5, (Prentice HallUpper Saddle River, 2002).

A. Krishnaswamy and G. V. Baranoski, A study on skin optics, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo, Canada, Technical Report. 1, 1–17 (2004).

S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 2396–2404.

M. Lewandowska, J. Rumiski, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam–a non-contact method for evaluating cardiac activity,” in: Computer Science and Information Systems, 2011 Federated Conference on, (IEEE, 2011), pp. 405–410.

S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (IEEE, 2012), pp. 2174–2177.

T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in: 2012 IEEE International Conference On Consumer Electronics, (IEEE, 2012), pp. 544–545.

Y.-P. Yu, P. Raveendran, and C.-L. Lim, “Heart rate estimation from facial images using filter bank,” in: Communications, Control and Signal Processing, 2014 6th International Symposium on, (IEEE, 2014), pp. 69–72.

X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 4264–4271.

A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in: Proceedings of the IEEE International Conference on Computer Vision, (IEEE, 2015), pp. 3640–3648.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1 Illustration of the 3-D to 1-D video signal processing of the Photoplethysmography Model. The black arrows highlight the incident light, the blue arrows highlight the surface reflectance and the red arrows highlight the subsurface reflectance. The green signal is the 1-D video signal.
Fig. 2
Fig. 2 Description of PPG measurement from facial skin by considering fixed L and V.
Fig. 3
Fig. 3 Overall description of the framework to measure heartbeat rate using the Photoplethysmography model.
Fig. 4
Fig. 4 Illustration of the noise profile of ψ(t) by using normal autocorrelation.
Fig. 5
Fig. 5 Description of the first five harmonics selection from the (a) de-noised video signal from the (b) PSD.
Fig. 6
Fig. 6 Illustration of the heart rate estimation by voting scheme.
Fig. 7
Fig. 7 Description of the first three harmonics selection from the (a) de-noised video signal from the (b) PSD.
Fig. 8
Fig. 8 Illustration of the ROI selection. (a) the ROI selection in prospect of outlier skin, (b) ROI selection in terms of tissue and muscle coverage, (c) ROI selection in terms of arteries, arterioles coverage and (d) ROI selection in terms of upper respiratory system coverage (i.e. sinus).
Fig. 9
Fig. 9 Illustration of the ROI detection after performing foreground object detection. (a) Input Image/frame, (b) CIELab color space conversion, (c) ‘a’ color opponent extraction, (d) binary map extraction, (e) RGB image cropping using binary map, (f) Face detection using Viola Jones, (g) Refining binary map, (h) Foreground extraction and (i) multiple ROI detection.
Fig. 10
Fig. 10 Performance validation of the respiratory rate estimation using Bland-Altman plot (a) and Correlation plot (b)
Fig. 11
Fig. 11 Illustration for the effect of spatial illumination variance at (a) controlled environment and (b) uncontrolled environment.

Tables (3)

Tables Icon

Algorithm 1 Wavelet based Multivariate De-noising

Tables Icon

Table 1 Performance validation of the methods for benchmarking experiment of 30s videos and 10s videos

Tables Icon

Table 2 Performance validation of the proposed method for naturalistic environment experiment

Equations (19)

Equations on this page are rendered with MathJax. Learn more.

C ( L , V , N , X , t , λ ) = r ( L , V , N , X , t , λ ) . E ( L , N , X , t , λ )
C ( N , X , t , λ ) = r ( N , X , t , λ ) . E ( N , X , t , λ )
r ( t , λ ) = r s ( t , λ ) + r d ( t , λ )
r d ( t , λ ) = α . p ( t )
C ( N , X , t , λ ) = ( r s ( t , λ ) + α . p ( t ) ) . E ( N , X , t , λ )
ψ ( x , t ) = λ = 400 700 C ( N , X , t , λ ) . ( a c ( λ ) + q ( t ) ) d λ
ψ ( t ) = 1 K × M i = k K j = m M x i , j ( t )
ψ ( t ) = P ( t ) + ε ( t )
ψ h ( t ) = 1 K × M h = 1 H i = k K j = m M x h , i , j ( t )
ψ h ( t ) = P h ( t ) + ε h ( t ) , 1 h H
W φ ( j 0 , k ) = 1 N n = 1 N ψ ( n ) φ j 0 , k ( n )
W ξ ( j , k ) = 1 N n = 1 N ψ ( n ) ξ j , k ( n )
ξ ( n ) = k h ξ ( k ) 2 φ ( 2 n k )
φ ( n ) = k h φ ( k ) 2 φ ( 2 n k )
Σ ε = T
γ h = 2 μ h log ( n )
ϕ ( ω ) = lim N E { 1 N | n = 0 N 1 p h ( n ) e i ω t | 2 }
H f ( ϕ 1 : 5 , h ( ω ) ) = h = 1 H [ ϕ 1 , h ( ω ) ϕ 2 , h ( ω ) ϕ 3 , h ( ω ) ϕ 4 , h ( ω ) ϕ 5 , h ( ω ) ]
H e = | M hr G hr |

Metrics