Abstract

Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. We also find that the PPG-related prior knowledge may still be helpful for the CNN-based PPG extraction, and recommend further investigation of hybrid CNN-based methods that include prior knowledge in their design.

© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Amplitude-selective filtering for remote-PPG

Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan
Biomed. Opt. Express 8(3) 1965-1980 (2017)

Skin inhomogeneity as a source of error in remote PPG-imaging

Andreia Vieira Moço, Sander Stuijk, and Gerard de Haan
Biomed. Opt. Express 7(11) 4718-4733 (2016)

DistancePPG: Robust non-contact vital signs monitoring using a camera

Mayank Kumar, Ashok Veeraraghavan, and Ashutosh Sabharwal
Biomed. Opt. Express 6(5) 1565-1588 (2015)

References

  • View by:
  • |
  • |
  • |

  1. W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light,” Opt. Express 16(26), 21434–21445 (2008).
    [Crossref]
  2. M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.
  3. M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
    [Crossref]
  4. G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).
    [Crossref]
  5. W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
    [Crossref]
  6. W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
    [Crossref]
  7. G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiol. Meas. 35(9), 1913–1926 (2014).
    [Crossref]
  8. W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), (2018), pp. 349–365.
  9. X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.
  10. R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the British Machine Vision Conference, Newcastle, UK, (2018), pp. 3–6.
  11. Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” in Proc. BMVC, (2019), pp. 1–12.
  12. Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.
  13. A. Tran and L.-F. Cheong, “Two-stream flow-guided convolutional attention networks for action recognition,” in Proceedings of the IEEE International Conference on Computer Vision, (2017), pp. 3110–3119.
  14. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
    [Crossref]
  15. G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2013), pp. 3430–3437.
  16. L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
    [Crossref]
  17. A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015).
    [Crossref]
  18. W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
    [Crossref]
  19. W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
    [Crossref]
  20. B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.
  21. A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.
  22. M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv preprint arXiv:1212.5701 (2012).
  23. R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, (IEEE, 2014), pp. 1056–1062.

2017 (2)

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

2016 (1)

W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
[Crossref]

2015 (3)

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
[Crossref]

2014 (2)

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiol. Meas. 35(9), 1913–1926 (2014).
[Crossref]

2013 (1)

G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).
[Crossref]

2011 (1)

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref]

2008 (1)

Balakrishnan, G.

G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2013), pp. 3430–3437.

Berg, A. C.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Bernstein, M.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Bittner, M.

A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.

Chen, W.

W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), (2018), pp. 349–365.

Chen, X.

X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.

Cheong, L.-F.

A. Tran and L.-F. Cheong, “Two-stream flow-guided convolutional attention networks for action recognition,” in Proceedings of the IEEE International Conference on Computer Vision, (2017), pp. 3110–3119.

Chung, M.-L.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

Clifton, D.

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Daly, J.

de Haan, G.

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
[Crossref]

G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiol. Meas. 35(9), 1913–1926 (2014).
[Crossref]

G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).
[Crossref]

den Brinker, A. C.

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

Deng, J.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Durand, F.

G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2013), pp. 3430–3437.

Fei-Fei, L.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Franc, V.

R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the British Machine Vision Conference, Newcastle, UK, (2018), pp. 3–6.

Frise, M. C.

Gross, H.-M.

R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, (IEEE, 2014), pp. 1056–1062.

Guazzi, A.

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Guazzi, A. R.

Gudi, A.

A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.

Guttag, J.

G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2013), pp. 3430–3437.

Han, H.

X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.

Hong, X.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

Huang, P.-W.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

Huang, Z.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Jeanne, V.

G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).
[Crossref]

Jorge, J.

A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015).
[Crossref]

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Karpathy, A.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Khosla, A.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Kocejko, T.

M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.

Krause, J.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Lewandowska, M.

M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.

Li, X.

Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” in Proc. BMVC, (2019), pp. 1–12.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

Lin, T.-M.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

Lochmans, R.

A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.

Ma, S.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Matas, J.

R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the British Machine Vision Conference, Newcastle, UK, (2018), pp. 3–6.

McDuff, D.

W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), (2018), pp. 349–365.

McDuff, D. J.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref]

Müller, S.

R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, (IEEE, 2014), pp. 1056–1062.

Nelson, J. S.

Niu, X.

X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.

Nowak, J.

M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.

Peng, W.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

Picard, R. W.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref]

Poh, M.-Z.

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref]

Pugh, C.

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Robbins, P. A.

Ruminski, J.

M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.

Russakovsky, O.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Satheesh, S.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Shan, S.

X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.

Špetlík, R.

R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the British Machine Vision Conference, Newcastle, UK, (2018), pp. 3–6.

Stricker, R.

R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, (IEEE, 2014), pp. 1056–1062.

Stuijk, S.

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
[Crossref]

Su, H.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Svaasand, L. O.

Tarassenko, L.

A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015).
[Crossref]

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Tran, A.

A. Tran and L.-F. Cheong, “Two-stream flow-guided convolutional attention networks for action recognition,” in Proceedings of the IEEE International Conference on Computer Vision, (2017), pp. 3110–3119.

Tsou, T.-Y.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

van Gemert, J.

A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.

Van Leest, A.

G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiol. Meas. 35(9), 1913–1926 (2014).
[Crossref]

Verkruysse, W.

Villarroel, M.

A. R. Guazzi, M. Villarroel, J. Jorge, J. Daly, M. C. Frise, P. A. Robbins, and L. Tarassenko, “Non-contact measurement of oxygen saturation with an rgb camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015).
[Crossref]

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

Wang, W.

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
[Crossref]

Wu, B.-F.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

Yu, Z.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” in Proc. BMVC, (2019), pp. 1–12.

Zeiler, M. D.

M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv preprint arXiv:1212.5701 (2012).

Zhao, G.

Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” in Proc. BMVC, (2019), pp. 1–12.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

Biomed. Opt. Express (1)

IEEE Trans. Biomed. Eng. (5)

M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref]

G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Algorithmic principles of remote ppg,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).
[Crossref]

W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rppg,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015).
[Crossref]

Int. J. Comput. Vis. (1)

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015).
[Crossref]

Opt. Express (1)

Physiol. Meas. (3)

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014).
[Crossref]

W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Robust heart rate from fitness videos,” Physiol. Meas. 38(6), 1023–1044 (2017).
[Crossref]

G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiol. Meas. 35(9), 1913–1926 (2014).
[Crossref]

Other (12)

W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), (2018), pp. 349–365.

X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), (IEEE, 2018), pp. 3580–3585.

R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the British Machine Vision Conference, Newcastle, UK, (2018), pp. 3–6.

Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” in Proc. BMVC, (2019), pp. 1–12.

Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 151–160.

A. Tran and L.-F. Cheong, “Two-stream flow-guided convolutional attention networks for action recognition,” in Proceedings of the IEEE International Conference on Computer Vision, (2017), pp. 3110–3119.

G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2013), pp. 3430–3437.

B.-F. Wu, P.-W. Huang, T.-Y. Tsou, T.-M. Lin, and M.-L. Chung, “Camera-based heart rate measurement using continuous wavelet transform,” in 2017 International Conference on System Science and Engineering (ICSSE), (IEEE, 2017), pp. 7–11.

A. Gudi, M. Bittner, R. Lochmans, and J. van Gemert, “Efficient real-time camera based estimation of heart rate and its variability,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2019), p. 0.

M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv preprint arXiv:1212.5701 (2012).

R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, (IEEE, 2014), pp. 1056–1062.

M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity,” in 2011 federated conference on computer science and information systems (FedCSIS), (IEEE, 2011), pp. 405–410.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1.
Fig. 1. The CNN-PPG flowchart includes four major parts: (1) Pre-processing: normalized image difference (AC/DC image); (2) Training reference: the derivative of the reference PPG signal; (3) The architecture of the CNN used in our experiments (ten convolution layers, five pooling layers, two fully-connected layers); (4) Post-processing: a bandpass filter with low cut-off frequency of 0.7 Hz and high cut-off frequency of 3 Hz.
Fig. 2.
Fig. 2. The snapshots of different image resolutions used in the experiment on the spatial context learning of CNN. (a)–(g) are the raw images (face area) with different image resolutions.
Fig. 3.
Fig. 3. The RMSE and accuracy of the CNN trained on the R-G-B channel order and tested on the R-G-B channel, B-G-R channel and R-B-G channel on the HNU dataset. And the RMSE and accuracy of the CNN trained and tested on the G-G-G channel, B-B-B channel and R-R-R channel order on the HNU dataset. In each panel, the median values are indicated by red bars inside the boxes, the quartile range by boxes, the full range by whiskers, the outliers by red cross.
Fig. 4.
Fig. 4. (a) The RMSE and accuracy of the CNN trained and tested on the HNU dataset by the labels with different phase delays; (b) The RMSE and accuracy of the CNN trained and tested on the PURE dataset with four kinds of labels. F-PPG: Finger-PPG; C-PPG: Camera-PPG; PF-PPG: Phase corrected finger-PPG; PFF-PPG: Phase corrected and filtered finger-PPG.
Fig. 5.
Fig. 5. (a) The RMSE and accuracy of the CNN trained and tested on different image resolutions on the HNU dataset; (b) The RMSE and accuracy of the CNN trained on the HNU dataset and tested on the HNU dataset with different spatial context information (e.g. the images are rotated by 90 degrees and 180 degrees, respectively).
Fig. 6.
Fig. 6. (a) The scatter plot of the sum of the weights in the R-G-B channels from the first convolution layer trained on the image with 1$\times$1 pixel on the HNU dataset. Each scatter point represents the sum of the weights of each convolution kernel in the R-G-B channels. The black vector $[0.46,-0.85,0.25]^{\mathrm {T}}$ indicates the exact projection direction to combined the R, G and B channel. (b) The scatter plots of the sum of the weights in the 2 projected channels from the first convolution layer (CNN+POS, CNN+CHROM) trained on the HNU dataset. Each scatter point represents the sum of the weights of each convolution kernel in the two projected channels. The red vector $[0.55,0.83]^{\mathrm {T}}$ and the blue vector $[0.63,-0.78]^{\mathrm {T}}$ indicate the exact projection direction for the two projected channels in the CNN+POS and the CNN+CHROM, respectively.
Fig. 7.
Fig. 7. Visualization of the different activation maps extracted by the first convolution layer of the CNN trained on the HNU dataset: (a) The activation map extracted by the CNN from the raw DC image; (b) The activation map extracted by the CNN from the normalized frame difference image; (c) The activation map extracted by the CNN from the normalized frame difference image with the content of a palm. The bright areas indicate the larger weights in the activation map.
Fig. 8.
Fig. 8. The RMSE and accuracy of the CNN, CNN+Noise and CNN+POS trained on the HNU dataset and tested on the noisy-perturbed (periodic noise) video data.
Fig. 9.
Fig. 9. (a) The RMSE and accuracy of CNN, CNN+POS and CNN+CHROM trained on the HNU dataset and tested on the PURE dataset; (b) The RMSE and accuracy of CNN, CNN+POS and CNN+CHROM trained and tested on the PURE dataset.
Fig. 10.
Fig. 10. (a) The RMSE and accuracy of the CNN trained on the HNU dataset with the R-G-B channel order and tested on the PURE dataset with the R-G-B-channel, B-G-R channel and R-B-G channel order. And the RMSE and accuracy of the CNN trained on the HNU dataset and tested on PURE dataset with the G-G-G channel, B-B-B channel and R-R-R channel order. (b) The RMSE and accuracy of the CNN trained on the HNU dataset by the labels with different phase delays and tested on the PURE dataset.
Fig. 11.
Fig. 11. (a) The RMSE and accuracy of the CNN trained on the HNU dataset and tested on the PURE dataset with different image resolutions; (b) The RMSE and accuracy of the CNN trained on the HNU dataset and tested on the PURE dataset with different spatial context information (e.g. the images are rotated by 90 degrees and 180 degrees, respectively).

Tables (1)

Tables Icon

Table 1. The exact split of groups in the HNU dataset.