Abstract

The ability to monitor the respiratory rate, one of the vital signs, is extremely important for the medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake everyday activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Alternatively, contactless digital image sensor based remote-photoplethysmography (PPG) can be used. However, remote PPG requires an ambient source of light, and does not work properly in dark places or under varying lighting conditions. Recent advances in thermographic systems have shrunk their size, weight and cost, to the point where it is possible to create smart-phone based respiration rate monitoring devices that are not affected by lighting conditions. However, mobile thermal imaging is challenged in scenes with high thermal dynamic ranges (e.g. due to the different environmental temperature distributions indoors and outdoors). This challenge is further amplified by general problems such as motion artifacts and low spatial resolution, leading to unreliable breathing signals. In this paper, we propose a novel and robust approach for respiration tracking which compensates for the negative effects of variations in the ambient temperature and motion artifacts and can accurately extract breathing rates in highly dynamic thermal scenes. The approach is based on tracking the nostril of the user and using local temperature variations to infer inhalation and exhalation cycles. It has three main contributions. The first is a novel Optimal Quantization technique which adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking. The second is the Thermal Gradient Flow method that computes thermal gradient magnitude maps to enhance the accuracy of the nostril region tracking. Finally, we introduce the Thermal Voxel method to increase the reliability of the captured respiration signals compared to the traditional averaging method. We demonstrate the extreme robustness of our system to track the nostril-region and measure the respiratory rate by evaluating it during controlled respiration exercises in high thermal dynamic scenes (e.g. strong correlation (r = 0.9987) with the ground truth from the respiration-belt sensor). We also demonstrate how our algorithm outperformed standard algorithms in settings with different amounts of environmental thermal changes and human motion. We open the tracked ROI sequences of the datasets collected for these studies (i.e. under both controlled and unconstrained real-world settings) to the community to foster work in this area.

© 2017 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Remote monitoring of breathing dynamics using infrared thermography

Carina Barbosa Pereira, Xinchi Yu, Michael Czaplik, Rolf Rossaint, Vladimir Blazek, and Steffen Leonhardt
Biomed. Opt. Express 6(11) 4378-4394 (2015)

Angle-selective optical filter for highly sensitive reflection photoplethysmogram

Chan-Sol Hwang, Sung-Pyo Yang, Kyung-Won Jang, Jung-Woo Park, and Ki-Hun Jeong
Biomed. Opt. Express 8(10) 4361-4368 (2017)

Fast varifocal two-photon microendoscope for imaging neuronal activity in the deep brain

Masaaki Sato, Yuki Motegi, Shogo Yagi, Keiko Gengyo-Ando, Masamichi Ohkura, and Junichi Nakai
Biomed. Opt. Express 8(9) 4049-4060 (2017)

References

  • View by:
  • |
  • |
  • |

  1. J. B. West, Respiratory Physiology: The Essentials (Lippincott Williams & Wilkins, 2012).
  2. P. Grossman, “Respiration, Stress, and Cardiovascular Function,” Psychophysiology 20(3), 284–300 (1983).
    [Crossref] [PubMed]
  3. M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
    [PubMed]
  4. M. Kumar, A. Veeraraghavan, and A. Sabharwal, “DistancePPG: Robust non-contact vital signs monitoring using a camera,” Biomed. Opt. Express 6(5), 1565–1588 (2015).
    [Crossref] [PubMed]
  5. M. van Gastel, S. Stuijk, and G. de Haan, “Robust respiration detection from remote photoplethysmography,” Biomed. Opt. Express 7(12), 4941–4957 (2016).
    [Crossref] [PubMed]
  6. C. B. Pereira, X. Yu, M. Czaplik, R. Rossaint, V. Blazek, and S. Leonhardt, “Remote monitoring of breathing dynamics using infrared thermography,” Biomed. Opt. Express 6(11), 4378–4394 (2015).
    [Crossref] [PubMed]
  7. F. Birklein, “Complex regional pain syndrome,” J. Neurol. 252(2), 131–138 (2005).
    [Crossref] [PubMed]
  8. M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
    [Crossref] [PubMed]
  9. A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
    [Crossref]
  10. R. Murthy and I. Pavlidis, “Noncontact measurement of breathing function,” IEEE Eng. Med. Biol. Mag. 25(3), 57–67 (2006).
    [Crossref] [PubMed]
  11. G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
    [Crossref] [PubMed]
  12. J. Fei and I. Pavlidis, “Thermistor at a Distance: Unobtrusive Measurement of Breathing,” IEEE Trans. Biomed. Eng. 57(4), 988–998 (2010).
    [Crossref] [PubMed]
  13. A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
    [Crossref] [PubMed]
  14. W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light,” Opt. Express 16(26), 21434–21445 (2008).
    [Crossref] [PubMed]
  15. S. Xu, L. Sun, and G. K. Rohde, “Robust efficient estimation of heart rate pulse from video,” Biomed. Opt. Express 5(4), 1124–1135 (2014).
    [Crossref] [PubMed]
  16. X. Mei and H. Ling, “Robust Visual Tracking and Vehicle Classification via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2259–2272 (2011).
    [Crossref] [PubMed]
  17. P. Ledda, A. Chalmers, T. Troscianko, and H. Seetzen, “Evaluation of Tone Mapping Operators Using a High Dynamic Range Display,” in ACM SIGGRAPH 2005 Papers, SIGGRAPH ’05 (ACM, 2005), pp. 640–648.
  18. J. Shi and C. Tomasi, “Good features to track,” in 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1994), pp. 593–600.
  19. B. D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” in Proceedings of the 7th International Joint Conference on Artificial Intelligence-Volume 2, IJCAI’81 (Morgan Kaufmann Publishers Inc., 1981), pp. 674–679.
  20. T. Ridler and S. Calvard, “Picture Thresholding Using an Iterative Selection Method,” IEEE Trans. Syst. Man Cybern. 8(8), 630–632 (1978).
    [Crossref]
  21. Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-Backward Error: Automatic Detection of Tracking Failures,” in 2010 20th International Conference on Pattern Recognition (ICPR) (2010), pp. 2756–2759.
    [Crossref]
  22. M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
    [Crossref]
  23. J. P. Lewis, “Fast normalized cross-correlation,” in Vision Interface: Canadian Image Processing and Pattern Recognition Society (1995), pp. 120–123.
  24. B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
    [Crossref] [PubMed]
  25. B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
    [Crossref] [PubMed]
  26. J. Ashburner and K. J. Friston, “Voxel-Based Morphometry−The Methods,” Neuroimage 11(6), 805–821 (2000).
    [Crossref] [PubMed]
  27. C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
    [Crossref] [PubMed]
  28. R. M. Gray and L. D. Davisson, An Introduction to Statistical Signal Processing (Cambridge University Press, 2004).
  29. J. Steketee, “Spectral emissivity of skin and pericardium,” Phys. Med. Biol. 18(5), 686–694 (1973).
    [Crossref] [PubMed]
  30. Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 790–799 (1995).
    [Crossref]
  31. E. R. Girden, ANOVA: Repeated Measures (Sage, 1992).
  32. J. M. Lloyd, Thermal Imaging Systems (Springer Science & Business Media, 2013).
  33. V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
    [Crossref] [PubMed]
  34. Y. Cho, N. Bianchi-Berthouze, and S. J. Julier, “DeepBreath: Deep Learning of Breathing Patterns for Automatic Stress Recognition using Low-Cost Thermal Imaging in Unconstrained Settings,” in 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII) (2017).

2017 (1)

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

2016 (1)

2015 (2)

2014 (2)

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

S. Xu, L. Sun, and G. K. Rohde, “Robust efficient estimation of heart rate pulse from video,” Biomed. Opt. Express 5(4), 1124–1135 (2014).
[Crossref] [PubMed]

2013 (1)

C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
[Crossref] [PubMed]

2011 (5)

X. Mei and H. Ling, “Robust Visual Tracking and Vehicle Classification via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2259–2272 (2011).
[Crossref] [PubMed]

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref] [PubMed]

G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
[Crossref] [PubMed]

2010 (1)

J. Fei and I. Pavlidis, “Thermistor at a Distance: Unobtrusive Measurement of Breathing,” IEEE Trans. Biomed. Eng. 57(4), 988–998 (2010).
[Crossref] [PubMed]

2009 (1)

A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
[Crossref]

2008 (3)

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light,” Opt. Express 16(26), 21434–21445 (2008).
[Crossref] [PubMed]

2006 (1)

R. Murthy and I. Pavlidis, “Noncontact measurement of breathing function,” IEEE Eng. Med. Biol. Mag. 25(3), 57–67 (2006).
[Crossref] [PubMed]

2005 (1)

F. Birklein, “Complex regional pain syndrome,” J. Neurol. 252(2), 131–138 (2005).
[Crossref] [PubMed]

2000 (1)

J. Ashburner and K. J. Friston, “Voxel-Based Morphometry−The Methods,” Neuroimage 11(6), 805–821 (2000).
[Crossref] [PubMed]

1995 (1)

Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 790–799 (1995).
[Crossref]

1983 (1)

P. Grossman, “Respiration, Stress, and Cardiovascular Function,” Psychophysiology 20(3), 284–300 (1983).
[Crossref] [PubMed]

1978 (1)

T. Ridler and S. Calvard, “Picture Thresholding Using an Iterative Selection Method,” IEEE Trans. Syst. Man Cybern. 8(8), 630–632 (1978).
[Crossref]

1973 (1)

J. Steketee, “Spectral emissivity of skin and pericardium,” Phys. Med. Biol. 18(5), 686–694 (1973).
[Crossref] [PubMed]

Abbas, A. K.

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Abouelenien, M.

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

Ashburner, J.

J. Ashburner and K. J. Friston, “Voxel-Based Morphometry−The Methods,” Neuroimage 11(6), 805–821 (2000).
[Crossref] [PubMed]

Avants, B. B.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

Bellomo, R.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Birklein, F.

F. Birklein, “Complex regional pain syndrome,” J. Neurol. 252(2), 131–138 (2005).
[Crossref] [PubMed]

Blazek, V.

Boric-Lubecke, O.

A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
[Crossref]

Brüser, C.

C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
[Crossref] [PubMed]

Burzo, M.

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

Calvard, S.

T. Ridler and S. Calvard, “Picture Thresholding Using an Iterative Selection Method,” IEEE Trans. Syst. Man Cybern. 8(8), 630–632 (1978).
[Crossref]

Cardone, D.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Chen, J.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Cheng, Y.

Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 790–799 (1995).
[Crossref]

Cook, P. A.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

Cretikos, M. A.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Czaplik, M.

de Haan, G.

Droitcour, A. D.

A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
[Crossref]

Engert, V.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Epstein, C. L.

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

Fei, J.

J. Fei and I. Pavlidis, “Thermistor at a Distance: Unobtrusive Measurement of Breathing,” IEEE Trans. Biomed. Eng. 57(4), 988–998 (2010).
[Crossref] [PubMed]

Finfer, S.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Flabouris, A.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Friston, K. J.

J. Ashburner and K. J. Friston, “Voxel-Based Morphometry−The Methods,” Neuroimage 11(6), 805–821 (2000).
[Crossref] [PubMed]

Gatto, R. G.

G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
[Crossref] [PubMed]

Gee, J. C.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

Grant, J. A.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Grossman, M.

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

Grossman, P.

P. Grossman, “Respiration, Stress, and Cardiovascular Function,” Psychophysiology 20(3), 284–300 (1983).
[Crossref] [PubMed]

Heimann, K.

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Hillman, K.

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Jergus, K.

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Klein, A.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

Kovacs, G. T. A.

A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
[Crossref]

Kumar, M.

Leonhardt, S.

C. B. Pereira, X. Yu, M. Czaplik, R. Rossaint, V. Blazek, and S. Leonhardt, “Remote monitoring of breathing dynamics using infrared thermography,” Biomed. Opt. Express 6(11), 4378–4394 (2015).
[Crossref] [PubMed]

C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
[Crossref] [PubMed]

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Lewis, G. F.

G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
[Crossref] [PubMed]

Ling, H.

X. Mei and H. Ling, “Robust Visual Tracking and Vehicle Classification via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2259–2272 (2011).
[Crossref] [PubMed]

McDuff, D. J.

M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref] [PubMed]

Mei, X.

X. Mei and H. Ling, “Robust Visual Tracking and Vehicle Classification via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2259–2272 (2011).
[Crossref] [PubMed]

Merla, A.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Mihalcea, R.

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

Murthy, R.

R. Murthy and I. Pavlidis, “Noncontact measurement of breathing function,” IEEE Eng. Med. Biol. Mag. 25(3), 57–67 (2006).
[Crossref] [PubMed]

Nelson, J. S.

Orlikowsky, T.

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Pavlidis, I.

J. Fei and I. Pavlidis, “Thermistor at a Distance: Unobtrusive Measurement of Breathing,” IEEE Trans. Biomed. Eng. 57(4), 988–998 (2010).
[Crossref] [PubMed]

R. Murthy and I. Pavlidis, “Noncontact measurement of breathing function,” IEEE Eng. Med. Biol. Mag. 25(3), 57–67 (2006).
[Crossref] [PubMed]

Pereira, C. B.

Pérez-Rosas, V.

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

Picard, R. W.

M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref] [PubMed]

Poh, M. Z.

M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref] [PubMed]

Porges, S. W.

G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
[Crossref] [PubMed]

Ridler, T.

T. Ridler and S. Calvard, “Picture Thresholding Using an Iterative Selection Method,” IEEE Trans. Syst. Man Cybern. 8(8), 630–632 (1978).
[Crossref]

Rohde, G. K.

Rossaint, R.

Sabharwal, A.

Singer, T.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Song, G.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

Steketee, J.

J. Steketee, “Spectral emissivity of skin and pericardium,” Phys. Med. Biol. 18(5), 686–694 (1973).
[Crossref] [PubMed]

Stuijk, S.

Sun, L.

Svaasand, L. O.

Tusche, A.

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Tustison, N. J.

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

van Gastel, M.

Veeraraghavan, A.

Verkruysse, W.

Winter, S.

C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
[Crossref] [PubMed]

Xu, S.

Yu, X.

Biomed. Eng. Online (1)

A. K. Abbas, K. Heimann, K. Jergus, T. Orlikowsky, and S. Leonhardt, “Neonatal non-contact respiratory monitoring based on real-time infrared thermography,” Biomed. Eng. Online 10(1), 93–109 (2011).
[Crossref] [PubMed]

Biomed. Opt. Express (4)

IEEE Eng. Med. Biol. Mag. (1)

R. Murthy and I. Pavlidis, “Noncontact measurement of breathing function,” IEEE Eng. Med. Biol. Mag. 25(3), 57–67 (2006).
[Crossref] [PubMed]

IEEE Trans. Biomed. Eng. (2)

J. Fei and I. Pavlidis, “Thermistor at a Distance: Unobtrusive Measurement of Breathing,” IEEE Trans. Biomed. Eng. 57(4), 988–998 (2010).
[Crossref] [PubMed]

M. Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam,” IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).
[Crossref] [PubMed]

IEEE Trans. Inf. Forensics Security (1)

M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo, “Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities,” IEEE Trans. Inf. Forensics Security 12(5), 1042–1055 (2017).
[Crossref]

IEEE Trans. Microw. Theory Tech. (1)

A. D. Droitcour, O. Boric-Lubecke, and G. T. A. Kovacs, “Signal-to-Noise Ratio in Doppler Radar System for Heart and Respiratory Rate Measurements,” IEEE Trans. Microw. Theory Tech. 57(10), 2498–2507 (2009).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

X. Mei and H. Ling, “Robust Visual Tracking and Vehicle Classification via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2259–2272 (2011).
[Crossref] [PubMed]

Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 790–799 (1995).
[Crossref]

IEEE Trans. Syst. Man Cybern. (1)

T. Ridler and S. Calvard, “Picture Thresholding Using an Iterative Selection Method,” IEEE Trans. Syst. Man Cybern. 8(8), 630–632 (1978).
[Crossref]

J. Neurol. (1)

F. Birklein, “Complex regional pain syndrome,” J. Neurol. 252(2), 131–138 (2005).
[Crossref] [PubMed]

Med. Image Anal. (1)

B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain,” Med. Image Anal. 12(1), 26–41 (2008).
[Crossref] [PubMed]

Med. J. Aust. (1)

M. A. Cretikos, R. Bellomo, K. Hillman, J. Chen, S. Finfer, and A. Flabouris, “Respiratory rate: the neglected vital sign,” Med. J. Aust. 188(11), 657–659 (2008).
[PubMed]

Neuroimage (2)

J. Ashburner and K. J. Friston, “Voxel-Based Morphometry−The Methods,” Neuroimage 11(6), 805–821 (2000).
[Crossref] [PubMed]

B. B. Avants, N. J. Tustison, G. Song, P. A. Cook, A. Klein, and J. C. Gee, “A reproducible evaluation of ANTs similarity metric performance in brain image registration,” Neuroimage 54(3), 2033–2044 (2011).
[Crossref] [PubMed]

Opt. Express (1)

Phys. Med. Biol. (1)

J. Steketee, “Spectral emissivity of skin and pericardium,” Phys. Med. Biol. 18(5), 686–694 (1973).
[Crossref] [PubMed]

Physiol. Meas. (1)

C. Brüser, S. Winter, and S. Leonhardt, “Robust inter-beat interval estimation in cardiac vibration signals,” Physiol. Meas. 34(2), 123–138 (2013).
[Crossref] [PubMed]

PLoS One (1)

V. Engert, A. Merla, J. A. Grant, D. Cardone, A. Tusche, and T. Singer, “Exploring the Use of Thermal Infrared Imaging in Human Stress Research,” PLoS One 9(3), e90782 (2014).
[Crossref] [PubMed]

Psychophysiology (2)

G. F. Lewis, R. G. Gatto, and S. W. Porges, “A novel method for extracting respiration rate and relative tidal volume from infrared thermography,” Psychophysiology 48(7), 877–887 (2011).
[Crossref] [PubMed]

P. Grossman, “Respiration, Stress, and Cardiovascular Function,” Psychophysiology 20(3), 284–300 (1983).
[Crossref] [PubMed]

Other (10)

J. B. West, Respiratory Physiology: The Essentials (Lippincott Williams & Wilkins, 2012).

P. Ledda, A. Chalmers, T. Troscianko, and H. Seetzen, “Evaluation of Tone Mapping Operators Using a High Dynamic Range Display,” in ACM SIGGRAPH 2005 Papers, SIGGRAPH ’05 (ACM, 2005), pp. 640–648.

J. Shi and C. Tomasi, “Good features to track,” in 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1994), pp. 593–600.

B. D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” in Proceedings of the 7th International Joint Conference on Artificial Intelligence-Volume 2, IJCAI’81 (Morgan Kaufmann Publishers Inc., 1981), pp. 674–679.

Y. Cho, N. Bianchi-Berthouze, and S. J. Julier, “DeepBreath: Deep Learning of Breathing Patterns for Automatic Stress Recognition using Low-Cost Thermal Imaging in Unconstrained Settings,” in 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII) (2017).

E. R. Girden, ANOVA: Repeated Measures (Sage, 1992).

J. M. Lloyd, Thermal Imaging Systems (Springer Science & Business Media, 2013).

J. P. Lewis, “Fast normalized cross-correlation,” in Vision Interface: Canadian Image Processing and Pattern Recognition Society (1995), pp. 120–123.

R. M. Gray and L. D. Davisson, An Introduction to Statistical Signal Processing (Cambridge University Press, 2004).

Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-Backward Error: Automatic Detection of Tracking Failures,” in 2010 20th International Conference on Pattern Recognition (ICPR) (2010), pp. 2756–2759.
[Crossref]

Supplementary Material (3)

NameDescription
» Visualization 1       An example clip to show the performance of the proposed method (from Dataset1 - controlled respiration in environments with non-constant temperature)
» Visualization 2       An example clip to show the performance of the proposed method (from Dataset2 - unconstrained respiration with natural motion artefact)
» Visualization 3       An example clip to show the performance of the proposed method (from Dataset3 - unconstrained respiration in fully mobile context and varied thermal dynamic range scenes)

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (14)

Fig. 1
Fig. 1

Key challenges in thermal imaging-based respiration tracking. (a) high thermal dynamic range scenes: fixed thermal range of interest is not suitable in preserving the morphological facial shape within varying ambient temperature: [top] examples of thermogram shots collected from a person walking outdoor (for 6 minutes), [bottom] temperature histograms, (b) motion and breathing artifacts: the shape of the nostril is affected by mobility and respiration dynamics, (c) respiration signal quality: the four example shots show the tracked nostril region while breathing. The traditional average temperature is not ideal in extracting the respiratory feature when the respiration-induced thermal variance is weak, e.g. during shallow breathing in Case 2 compared with deep breathing in Case 1. The low spatial resolution of mobile thermal imaging also leads to the weak signal.

Fig. 2
Fig. 2

(a) Overall procedure of the imaging-based respiration tracking system; (b) key components: 1) Optimal Quantization – convert from the absolute temperature distributions to the color-mapped images by analyzing the temperature histogram of every frame, 2) Thermal Gradient Flow – nostril-region tracking method using the thermal gradient magnitude and points tracking methods, and 3) Thermal Voxel-based Respiratory Rate Estimation – extracting the respiratory signals by integrating the unit thermal voxels inside the nostril.

Fig. 3
Fig. 3

Example shots of conversion from thermal images to thermal-gradient magnitude maps: the proposed method can help to preserve the morphology of the nostril region during motion (Zoomed-in-areas are manually rotated for the visual representation).

Fig. 4
Fig. 4

Extraction of respiratory patterns through Thermal Voxel integration: (a) a person’s nostril and its thermogram sequences along the time in 3D (top) and 2D (bottom), (b) the concave volume corresponding to heat variances in the nostril, (c) the extracted respiratory signals compared with ground truth signals, and (d) a comparison of the filtered voxel-based signals with the traditional method as the participant changes their head. The voxel method closely tracks ground truth, but the traditional method fails.

Fig. 5
Fig. 5

Experiment 1: (a) to obtain different thermal dynamic range scenes (i.e. environments with non-constant dynamic temperature), four different places were chosen (Place A: room, B: entrance of the building, C: corner on the street, D: park), the last image-shot is a thermal image collected in Place D, the experiment was run in winter, (b) the guiding breathing patterns are composed of three different rates (10(slow), 15(normal), 30(fast) breaths/min).

Fig. 6
Fig. 6

(a) experiment 2: unconstrained respiration in desk activities, (b) experiment 3: unconstrained respiration in both indoor and outdoor light physical activities.

Fig. 7
Fig. 7

Statistical methods for evaluations: (a) automated-synchronization between estimated signals and reference signals using the maximum-amplitude of cross-correlation (MACC), (b) respiration-related goodness probability as a respiratory signal quality index (rSQI).

Fig. 8
Fig. 8

Overall results of the nostril-tracking performance of Thermal Gradient Flow compared with existing methods: (a) Dataset 1 (controlled but in non-constant temperature scenes), (b) Dataset 2 (unconstrained respiration during sedentary activity), (c) Dataset 3 (unconstrained respiration during physical activity).

Fig. 9
Fig. 9

Quantified motion artifacts using the relative Euclidean distance from the origin of the nostril-ROI at the first frame: (a) example from Dataset 1 (fully controlled), (b) from Dataset 2 (sedentary behavior), and (c) from Dataset 3 (physical activity).

Fig. 10
Fig. 10

Results of respiratory signature extraction: (a,d,g) time-domain signal, (b,e,h) frequency-domain signal (respiration rate), and (c,f,i) the respiration-related goodness metric. The Thermal Voxel-based method is more robust than the traditional averaging-based method for Dataset 1 (a-c) and Dataset 2 (d-f). For Dataset 3 (g-i) (i.e. fully mobile contexts), the ground truth shows less reliability in the respiration tracking, except for the segment A in (g) (i.e. standing with small movement).

Fig. 11
Fig. 11

Dataset 1 (overall): Bland-Altman plots of (a) Thermal Voxel integration method, (b) the traditional averaging method, and (c) overall RMSE comparisons.

Fig. 12
Fig. 12

Dataset 1 (separate results along with the different environment): (a) Bland-Altman plots of Thermal Voxel integration method and the traditional averaging method, and (b) RMSE comparisons for Place A – D.

Fig. 13
Fig. 13

Dataset 2: Bland-Altman plots of (a) Thermal Voxel integration method, (b) the traditional averaging based estimation method, and (c) overall RMSE comparisons. The mean Pr value (rSQI) from the ground truth in Dataset 1 was set as exclusion criterion (Pr ≥0.9825).

Fig. 14
Fig. 14

Dataset 3: Bland-Altman plots of (a) Thermal Voxel integration method, (b) the traditional averaging method, and (c) overall RMSE comparisons. The mean Pr value (rSQI) from the ground truth in Dataset 1 was set as exclusion criterion (Pr ≥0.9825).

Tables (5)

Tables Icon

Table 1 Effects of different environments (i.e. different thermal dynamic range scenes) and algorithms on the success rate of the nostril-region tracking from Dataset 1 using two-way repeated measures ANOVA

Tables Icon

Table 2 Significance test to assess effects of different algorithms on the success rate in the outdoor condition (i.e. Place B-D in Dataset 1, high thermal dynamics) using one-way repeated measures ANOVA

Tables Icon

Table 3 Significance test to assess effects of different algorithms on the success rate in the indoor condition (i.e. Place A in Dataset 1, low thermal dynamics) using one-way repeated measures ANOVA

Tables Icon

Table 4 Effect of different algorithms on ROI tracking performance in sedentary activities (i.e. Dataset 2) using one-way repeated measures ANOVA

Tables Icon

Table 5 Effect of different algorithms on ROI tracking during physical activities (i.e. Dataset 3) using one-way repeated measures ANOVA

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

T min = c ¯ 1.96 σ c nm , T max = c ¯ +1.96 σ c nm
T opt (0)= T min
T opt (t+1)= μ 1 (t)+ μ 2 (t) 2
T 0 = T opt (p), T k1 = T max .
Φ(x,y)= ( u(x,y) x ) 2 + ( u(x,y) y ) 2
e( T f k |S)= x t x ^ t
γ(x)= i ( Φ ROI ( x i ) μ Φ ROI ( x i ) )(Φ( x i x) μ Φ( x i x) ) i ( Φ ROI ( x i ) μ Φ ROI ( x i ) ) 2 i (Φ( x i x) μ Φ( x i x) ) 2
v ^ (t)= T min (t) T δ (t) Λ t (T)dT i j T δ (t) u ^ ij (t), T δ (t) u ^ ij (t)>0
Δ γ 1 t =E [ u ^ ij (t) μ t σ t ] 3 E [ u ^ ij (t1) μ t1 σ t1 ] 3 .
w i (k)= v ^ ( n i +k) e 1 2 ( k σ ) 2 , k{ t ^ max f s ,..., t ^ max f s }.
S V (f)= F f ( R ww )= k R ww (k) e j2πfk
P r ( f ^ min f f ^ max ) f ^ min f ^ max S V (f)df total S V (f)df