In this paper, we propose a non-invasive method for measuring pulse waves corresponding to heart rate (HR) by capturing the color change on the soles of rats’ feet using a high-speed RGB camera. Remote photoplethysmography (rPPG) with a camera has been used as a non-invasive biometric method. However, the rPPG method has been challenging to apply to rats with body hair. We applied the rPPG method using a high frame rate to the sole where the skin was directly visible and successfully and accurately detected pulse waves under non-invasive, non-restraint, and non-anesthetized conditions.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Animal studies are essential for investigating biological mechanisms of an entire living system that would be too difficult to perform in human subjects [1–3]. However from the perspective of animal welfare, the 3Rs (replacement, reduction, and refinement) are essential issues in animal experiments [4,5]. In animal experiments, heart rate (HR) measurement is widely used to observe vitals, and electrocardiography (ECG) is the gold standard [6–8].
Non-contact measuring of pulse waves without attaching a device, such as using microwaves and cameras, have been proposed [9–11]. A vital signal measurement method using cameras including RGB, infrared, and Time-of-Flight (TOF), has been proposed. Among them, RGB cameras are easy to use because they are general purpose. An RGB camera-based method has been proposed to measure rats and wild animals’ HR and respiration rate (RR) by capturing their skin movements using a camera [12–15]. These methods are affected by ambient light and require observation in a sedation state. In human HR measurement, a photoplethysmography (PPG) method that directly utilizes the color channels of an RGB camera has been proposed [16–18]. Use of the green (G) channel signal of an RGB camera has the advantage in that it can be quickly realized by spatially averaging the G signals, and it can be processed in real-time [19,20].
This paper proposes to use the G channel of the camera to measure the HR of rodents. This is a non-invasive method for measuring pulse waves corresponding to HR by capturing the color change on the soles of rats’ feet using a high-speed RGB camera. These results and discussion of the paper give a new analysis compared to the conference proceedings SPIE-BIOS . We used the same video datasets as in the conference proceeding. There are two challenges in adapting the remote PPG (rPPG) to rats. The first challenge is that the human pulse rate is 60–150 beats per minute (bpm), whereas the rat pulse rate is 350–450 bpm . Therefore, the image and signal processing should be improved. The second challenge is that rPPG captures skin color changes; thus, rats covered with body hair need to be depilated. To solve these problems, we decided to capture images at a faster frame rate than that for humans and focus on the rat soles, where the skin is directly visible, to capture changes in skin color. As a result, pulse waves could be detected from videos of the sole captured at a frame rate of 250 fps. Furthermore, we were able to detect arrhythmia that occurred when fear was inflicted.
2. Material and methods
2.1 Creating the dataset for analysis
The dataset used in this study is common to the dataset described in the SPIE-BIOS manuscript. In this section, the dataset is briefly described.
Sixteen eight-week-old male Wistar rats (Japan SLC, Hamamatsu, Japan) were used. The rats were housed under 12 h light/12 h dark cycles at 21 °C with unrestricted access to food and water. All images were captured in the afternoon. The Committee of Animal Research at the International University of Health and Welfare approved our experiments (no. 19021).
2.1.2 Experimental setup for creating the dataset
Figure 1 and Fig. 2 show an overview of the experimental setup for video and ECG acquisition. The experiment was conducted in the laboratory of the International University of Health and Welfare, Ohtawara.
Figure 1 shows the experimental setup used to acquire data from rats under sedation. To observe color change, we removed the hair from the head, back, and abdomen areas using a hair removal cream (Hair Removing Body Cream “epilat”, Kracie Holdings, Tokyo, Japan) 3 h before the experiment. The intravenous anesthetic propofol (500 µL/kg body weight, Wako Pure Chemical Industries, Tokyo, Japan), which was used for sedation, was injected intraperitoneally 30 min before the experiment. The camera was set up on a tripod 60 cm above the rat skin, and two LED lights were placed on either side of the camera. We used a Memrecam-Q1m high-speed RGB camera (NAC Image Technology Inc., Tokyo, Japan) with a frame size of 1024 × 768 pixels. The lens (FL-CC1614A-2M, RICHO, Tokyo, Japan) had a focal length of 16 mm (f = 1.4), and only LED (ViltroxL116T, 810LX/0.5m) lighting was used with 100% output at a color temperature of 5600 K. The camera-specific software HXLink64 was installed on a laptop computer for camera control and data acquisition. The frame rate was set at 250 fps, which is approximately eight times higher than the human experiment considering to measure heart rate variability (HRV)  in our future work, to account for the temporal resolution because the pulse rate of rats is 350-450 bpm , while that of humans is 60-150 bpm. The true value was determined using a gold standard 3-point ECG system (SP-2000, EP95U, Softron Co., Ltd., Tokyo, Japan). The ECG was measured simultaneously with a video recording. The sampling rate for converting the ECG analog signals to digital signals was 1000 Hz.
Figure 2 shows the experimental setup used to acquire data from the sole of normal rats. The rat was placed in a rectangular box with a partition and was illuminated from below using LED lights. The distance between the floor and the tip of the camera lens is 10 cm. The floor is made of transparent plastic, and the partition allows indirect placement of odorous substances. The lighting and camera apparatus are the same ones used in under sedation state. The lens (FL-CC0814A-2M, RICHO, Tokyo, Japan) with focal length of 8 mm (f = 1.4) was used. The HR was determined using an implant ECG system (ATR-1001, ATE-02s, EP95U, Softron Co., Ltd, Tokyo, Japan).
Three different states were set up to determine the detection sensitivity: normal, fox smell-induced, and sedentary. The fox smell was included in the experiment because the smell of foxes, which are natural predators of rats, causes stress. We used 2-methylthiazoline (2-MT, Tokyo Chemical Industries, Tokyo, Japan), which creates a unique stress-like pattern of dynamic and endocrine activation in rats . For the sedation experiment, intravenous anesthetic propofol (500 µL/kg body weight, Wako Pure Chemical Industries, Tokyo, Japan) was injected into the abdominal cavity 10 min before the experiment.
2.2 Pulse rate detection procedure
The data processing procedure used in this experiment is illustrated in Fig. 3. To extract the video of the regions of interest (ROIs) for the experiment, we used the camera-specific software HXLink64. First, the video dataset was visually checked to identify the locations of the ROIs to be extracted. Next, the ROIs were selected, and the videos were converted to uncompressed AVI files. MATLAB2019b was used for the subsequent data processing. The G signal was obtained by extracting the pixel values of the G channel from a file in the AVI format and averaging the pixel values. The obtained G signal was detrended, and bandpass filtered, and its peaks were detected. The signal was passed between the 3 and 9 Hz bands, considering the HR range for the rat. The findpeaks function in MATLAB was used for peak detection. For the ECG signals, we used the threshold method because the peak values were almost constant. For rPPG, because the peak values fluctuated, we used the nearest-neighbor method. The definitions of the peak interval (PI) in the rPPG and ECG are shown in Fig. 3 (b).
The PI was obtained from the extracted peak detection time T = [ T2,…, Tk,…, TN-1 ]. To avoid false peak detection, T was set from 2 to N-1.
HR was calculated from PI
3. Results and discussion
3.1 Pulse wave analysis from the head and back ROIs
3.1.1 Signal analysis in the prone position
Table 1 shows the HR, relative error (RE) based on the ECG, and standard deviation (SD) for 20 s in the prone position. The HR measured from the gold standard 3-point ECG was used as the correct value and was compared with the HR calculated from the G signal. The ROIs set up for the rPPG measurements were the depilated head and back. The ROIs were set to the same size (60 × 52 pixels) to obtain the same denoising effect as the averaging process. The results showed that the values obtained from the head were in excellent agreement with the ECG signals. In comparison, the signals from the back were sometimes inconsistent with the correct ECG values. This might have been due to the noise caused by respiration, which prevented accurate peak detection.
Figure 4 shows the ECG signal compared with the unprocessed G signals from the ROIs on the rat skin removal area in experiment 2. The ROIs were positioned in four different areas: the depilated head, depilated back, back of the hair, and paper as a dummy. Every ROI was set to the same size. No well-defined waveforms corresponding to RR and HR were observed in the paper area (ROI-1). The signal from the hair-removed head showed peaks at intervals similar to those of the HR, and the signals from the hair-removed areas (ROI-4, 5) showed the same periodic signal. In comparing the signals of ROI-2 and ROI-5, we also observed a periodic peak of about 0.8 Hz, which was different from that of the HR. To investigate the origin of this periodic signal, we observed the original video. We found that the periodic signal corresponds to the body movement caused by the respiration of the rat.
These results showed that the pulse wave signal could be obtained from the unprocessed G signal at a frame rate of 250 fps. The respiratory signal could also be obtained by analyzing the G signal in the ROI of the area with large movements due to respiration. This result is similar to the results of several studies [13,14] that obtained the RR from the luminance change in rPPG.
Figure 5 shows the results of comparing the ECG and pulse wave signal from the hair-removed area of the head in experiment 2. A graph comparing the temporal changes in the pulse wave PI is also presented. The pulse wave peaks in ECG and rPPG are almost identical, and the pulse wave interval graph supports this. Since the ECG and rPPG devices are manually synchronized, it is not easy to compare the ECG and rPPG data accurately against the time axis. The rPPG shows more significant variation in the pulse wave interval than the ECG in PI variation. The rPPG shows an amplitude variation that is within 0.04 s. From the above results, it can be concluded that the signal processing worked well in the head area where respiratory noise was low. The rPPG had a more comprehensive range of variability than the ECG. The accuracy of peak detection needs to be improved to support applications that measure HRV. The G signal peak for calculating the HRV, the same as PI, was not sharp, as shown in Fig. 4. In order to reliably capture the peak, higher frame rate is necessary in the imaging system.
3.1.2 Signal analysis in the supine position
Table 2 shows the result in the supine position. The measurement conditions were the same as those described in the previous section. The ROIs set up for rPPG measurements were the hair-removed abdomens. The results showed that the values obtained from the abdomen were in good agreement with the ECG signals. The accuracy was not as good as that of the signal from the head, as shown in Table 1.
Figure 6 shows the locations of the ROIs taken from the abdomen and their respective unprocessed G signals. From the ROIs set on the rat, low-frequency signals different from the ECG signals were observed from the graphs. The ROI-4 and ROI-5 had reversed phases. The periodic signal set on the rat’s body fluctuated in the same cycle as the respiratory rate, as seen in the video. As for the HR signal, the unprocessed G signal obtained from the video taken from the ventral side did not show a clear HR signal, similar to the signal obtained from the head in the prone position. However, in the skin area, pulse waves could be observed from the signal processing, as in the prone position.
The correlation between heart rate and the mean value of rPPG for experiment number 1-4 was 0.70 to show the correlation over time. From the above results, the pulse wave signal could be extracted from the abdomen ROI by processing the G signal at a frame rate of 250 fps, but the noise due to the influence of respiration was larger than that of the head ROI.
3.2 Pulse wave detection from different body locations
This section describes the results of pulse wave detection from different body locations of a rat under three conditions: normal, stress-added, and under sedation. The time events of the experiments are sequenced by experiment number. ROIs were set at two locations on the soles and two locations on the abdomen. The size of the ROI varies due to the rat's movement and the camera's viewing angle. In experiments 5 and 10, only one sole was visible in the camera's field of view, so ROIs were set to two different locations on the same sole. Table 3 lists the HR, ECG-based RE, and SD obtained from data measured for 20 s under three different conditions and at different body parts. The standard deviation of the pulse wave was calculated using the pulse wave interval time converted to pulse rate per minute. The correlation between heart rate and the mean value of rPPG for experiment number 5-10 was 0.86 to show the correlation over changing heart rates.
Except for experiments 6 and 8, rPPG obtained the exact HR in RE. The comparison of variances showed that rPPG tended to be wider than ECG, which was the same as the experimental results in the previous section.
We investigated the cause of the large errors in experiments 6 and 8. Figure 7 shows the ROIs of experiment 6 and the pulse wave data obtained from each ROI. Figure 7 (b) shows the G-channel source signal obtained from each ROIs. The pixel values changed significantly between 11 and 12 s. The original video showed fluctuating illumination. While the signal intensity of the pulse wave component in rPPG, which captures pulse wave fluctuations, is less than 0.5, the noise estimated to be due to lighting fluctuation is 5. Since the lighting fluctuation range was extensive, the pulse wave fluctuations were presumably undetected. The PI in Fig. 7 (c) is also longer in this area, and the pulse wave signal is not detected. From this result, although the pulse wave signal was normally captured correctly, our detection software was unable to cope with the large fluctuations in the base signal. Therefore to improve the accuracy of the obtained average pulse wave, approaches such as cutting off the large variation portion in the calculation can be considered in future.
In experiment 8 there was an over-detection of pulses. Figure 8 shows the data for experiment 8, i.e., for the rat that sniffed 2-MT. In this experiment, the rat experienced extreme fear and was immobile as a result. According to the ECG data in Fig. 8 (b) and (d), the pulse wave was inconsistent and showed an arrhythmia-like waveform. In Fig. 8 (c), the PI at approximately 8 s showed that the pulse was lost for one beat. The peak differences between the three locations shown in the ECG and rPPG PIs were all 0.33 s. The waveforms of the ECG and rPPG peaks are shifted by two factors: the origin of the signal and the coincidence of the trigger. The origin of the signal, which is the gap between the ECG and rPPG, is estimated to be around 40 msec in mice . Since automatic synchronization was not available, the trigger operation in this experiment was performed manually. The deviation of the manual operation of the triggers is considered to be within 0.5 seconds. Based on this result, we concluded that the time deviation between ECG and rPPG was 0.33 s. By correcting for this time difference, the peak positions of ECG and rPPG can be correctly compared. The red asterisks on the rPPG graph in Fig. 8 (d) show the instances when peaks were recognized using peak detection. Comparison of rPPG and ECG peak detection showed that rPPG incorrectly detected peaks in the area indicated by the red arrows at around 12 seconds. We used the findpeaks function for peak extraction. In this experiment, we adjusted the parameters for optimality. It is noted that various methods have been proposed for peak extraction. In the future, we will continue to conduct and test the various state-of-the-art methods.
3.3 Comparison of pulse waves obtained from different areas of the body
The differences in the pulse waves obtained from the four different parts of the body were compared. Figure 9 shows the ROIs of experiment 9 and the G signals obtained from each ROI in 4 s after detrending. Similar signals were observed at all four ROIs. Then, to investigate the possibility of detecting the pulse wave propagation velocity, we analyzed the signal shift for the four ROIs of rPPG. The calculation was performed for experiment numbers 6–9, where both feet could be measured. The xcorr function in MATLAB was used to detect the signal shift. The maximum value of the shift width was set to 0.1 s. The 20-s detrended signal was divided into time regions of every 4 s, i.e., 0-4, 4-8, 8-12, 12-16, and 16-20 s, and the signal in each region was calculated. Table 4. Calculation result of signal time delay of three conditions. Table 4 summarizes the results of the signal shift from xcorr function for 4 s. According to this result, the signals obtained from the four ROIs set at different body locations showed no time deviation. The delay due to pulse wave propagation between the ROIs is estimated to be less than 1/250 s. In experiment 6-8, the SD values were larger than in experiment 9, whereby rats were under sedation. We plan to investigate this issue of large SD values in future studies.
4. Conclusion and outlook
We developed a non-invasive pulse wave measurement method for rats using a high-speed RGB camera to obtain rPPG signals with high accuracy from changes in skin color using only general signal processing. The feature of this method is that heart rate can be detected simply by using a high-speed camera system (250 fps) that generates high-quality pulse signals, selecting an appropriate ROI, and detecting all signal peaks in the band-pass filtered average signal of that ROI. This method yielded the same pulse rate measurement as the gold standard ECG. Furthermore, using this method, we found that pulse waves could be detected by video recording from the sole, which did not require hair removal. Our proposed method is non-contact and non-invasive, and is valuable for detecting the pulse wave of rats in a non-restrained manner. In addition, because various parts of the body can be video-recorded using a single camera, we have demonstrated the possibility of simultaneously detecting respiration from body movement.
In an experiment to measure the normal state of the rat, we detected the signals during the periods when the feet were not moving. When the feet moves a lot, blood distribution on the sole changes and noise can be expected to increase. In the future, we plan to take these into account when performing continuous measurements.
For continuous measurements, the system may need to be equipped with a program that tracks the location of the sole and also automatically excludes areas of the foot that frequently move. Rats perform grooming behaviors such as scratching their heads with their hands and standing up, even when their feet are not moving. Collecting measurements from a hair-removed area is challenging when rats are standing because the position of their hair changes. Therefore, imaging of the sole is considered to be superior to imaging of depilated skin.
In this study, we were able to obtain the same HR from the G signal in the video as from the ECG, which represents the true value of our research. Because the rats used in this study did not have melanin, the G signal was used, but for mice with melanin or under fluctuations in illumination, the signal processing would need to be modified. In a previous study, Fukunishi et al. used a method to separate shade, hemoglobin, and melanin components from RGB signals using independent component analysis (ICA) . This component separation method may be useful for future studies involving animals with melanin. Moreover, wang et al. used RGB channels to improve the robustness of the method . In the future, we plan to apply these findings to improve the accuracy in rats.
Because we were able to measure PI in rats under normal conditions, we were able to find a way to measure stress using PI fluctuations. The measurement of stress in animals is currently conducted by observation, which is time-consuming and costly. Our proposed method has the potential to reduce the labor of researchers. In human stress research, blood samples are taken to measure biomarkers such as inflammatory cytokines. However, animal blood collection is sometimes difficult. Our method has the potential to solve these problems.
In the future, we plan to improve the signal processing to achieve more accurate measurements.
MT and NT: Imaging Tech Laboratory LLC, 727-2, Konakadai, Inage-ku, Chiba-shi, Chiba 263-0044, Japan (I, P, S).
NI: Imaging Tech Laboratory LLC, 727-2, Konakadai, Inage-ku, Chiba-shi, Chiba 263-0044, Japan (P).
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
1. S. Festing and R. Wilkinson, “The ethics of animal research: talking point on the use of animals in scientific research,” EMBO Rep. 8(6), 526–530 (2007). [CrossRef]
2. P. Camacho, H. Fan, Z. Liu, and J.-Q. He, “Small mammalian animal models of heart disease,” Am. J. Cardiovasc. Dis. 6(3), 70–80 (2016). [CrossRef]
3. M. R. Fernandes and A. R. Pedroso, “Animal experimentation: A look into ethics, welfare and alternative methods,” Rev. Assoc. Med. Bras. 63(11), 923–928 (2017). [CrossRef]
4. E. O. Kehinde, “They see a rat, we seek a cure for diseases: the current status of animal experimentation in medical practice,” Med. Princ. Pract. 22(s1), 52–61 (2013). [CrossRef]
5. N. Levy, “The use of animal as models: ethical considerations,” Int. J. Stroke 7(5), 440–442 (2012). [CrossRef]
6. D. Ho, X. Zhao, S. Gao, C. Hong, D. E. Vatner, and S. F. Vatner, “Heart rate and electrocardiography monitoring in mice,” Curr. Protoc. Mouse Biol. 1, 123–139 (2011). [CrossRef]
7. D. Park, M. Lee, S. E. Park, J.-K. Seong, and I. Youn, “Determination of optimal heart rate variability features based on SVM-recursive feature elimination for cumulative stress monitoring using ECG sensor,” Sensors 18(7), 2387 (2018). [CrossRef]
8. Y. Shikano, T. Sasaki, and Y. Ikegaya, “Simultaneous recordings of cortical local field potentials, electrocardiogram, electromyogram, and respiration rhythm from a freely moving rat,” J. Visualized Exp. 134, 56980 (2018). [CrossRef]
9. C. González-Sánchez, J.-C. Fraile, J. Pérez-Turiel, E. Damm, J. G. Schneider, H. Zimmermann, D. Schmitt, and F. R. Ihmig, “Capacitive sensing for non-invasive respiration and heart monitoring in non-restrained, non-sedated laboratory mice,” Sensors 16(7), 1052 (2016). [CrossRef]
10. C. Li, Z. Peng, T.-Y. Huang, T. Fan, F.-K. Wang, T.-S. Horng, J. M. Muñoz-Ferreras, R. Gómez-García, L. X. Ran, and J. S. Lin, “A review on recent progress of portable short-range noncontact microwave radar systems,” IEEE Trans. Microwave Theory Tech. 65(5), 1692–1706 (2017). [CrossRef]
11. S. H. Oh, S. B. Lee, S. M. Kim, and J. H. Jeong, “Development of a heart rate detection algorithm using a non-contact doppler radar via signal elimination,” Biomed. Signal Process. Control. 64, 102314 (2021). [CrossRef]
12. A. Al-Naji, Y. Tao, I. Smith, and J. Chahl, “A pilot study for estimating the cardiopulmonary signals of diverse exotic animals using a digital camera,” Sensors 19(24), 5445 (2019). [CrossRef]
13. C. Barbosa Pereira, J. Kunczik, L. Zieglowski, R. Tolba, A. Abdelrahman, D. Zechner, B. Vollmar, H. Janssen, T. Thum, and M. Czaplik, “Remote welfare monitoring of rodents using thermal imaging,” Sensors 18(11), 3653 (2018). [CrossRef]
14. J. Kunczik, C. B. Pereira, L. Zieglowski, R. Tolba, L. Wassermann, C. Häger, A. Bleich, H. Janssen, T. Thum, and M. Czaplik, “Remote vitals monitoring in rodents using video recordings,” Biomed. Opt. Express 10(9), 4422–4436 (2019). [CrossRef]
15. M. Froesel, Q. Goudard, M. Hauser, M. Gacoin, and S. B. Hamed, “Automated video-based heart rate tracking for the anesthetized and behaving monkey,” Sci. Rep. 10(1), 17940 (2020). [CrossRef]
16. W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light,” Opt. Express 16(26), 21434–21445 (2008). [CrossRef]
17. K. Alghoul, S. Alharthi, H. A. Osman, and A. E. Saddik, “Heart rate variability extraction from videos signals: ICA vs. EVM comparison,” IEEE Access 5, 4711–4719 (2017). [CrossRef]
18. R. Mitsuhashi, K. Iuchi, T. Goto, A. Matsubara, T. Hirayama, H. Hshizume, and N. Tsumura, “Video-based stress level measurement using imaging photoplethysmography,” in 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (2019), pp. 90–95.
19. Y. Maeda, M. Sekine, and T. Tamura, “Relationship between measurement site and motion artifacts in wearable reflected photoplethysmography,” J. Med. Syst. 35(5), 969–976 (2011). [CrossRef]
20. B. A. Fallow, T. Tarumi, and H. Tanaka, “Influence of skin type and wavelength on light wave reflectance,” J. Clin. Monit. Comput. 27(3), 313–317 (2013). [CrossRef]
21. M. Takahashi, T. Yamaguchi, R. Takahashi, N. Tsumura, and N. Iijima, “Non-contact measurement of pulse wave in rats using an RGB camera,” in SPIE Optical Diagnostics and Sensing XXI: Toward Point-of-Care Diagnostics (2021), p. 116510E.
22. M. Kuwahara, M. D. Hue, H. Hirose, and S. Sugano, “Alteration of the intrinsic heart rate and autonomic nervous tone during the growing process of rats and pigs,” Jpn. J. Vet. Sci. 48(4), 703–709 (1986). [CrossRef]
23. M. Fukunishi, K. Kurita, S. Yamamoto, and N. Tsumura, “Non-contact video-based estimation of heart rate variability spectrogram from hemoglobin composition,” Artif. Life Robot. 22(4), 457–463 (2017). [CrossRef]
24. T. Isosaka, T. Matsuo, T. Yamaguchi, K. Funabiki, S. Nakanishi, R. Kobayakawa, and K. Kobayakawa, “Htr2a-expressing cells in the central amygdala control the hierarchy between innate and learned fear,” Cell 163(5), 1153–1164 (2015). [CrossRef]
25. H. Zakaria and P. Hasimun, “Non-invasive pulse wave velocity measurement in mice,” in 2017 International Seminar on Sensors, Instrumentation, Measurement and Metrology (ISSIMM) (2017), pp. 95–98.
26. W. Wang, S. Stuijk, and G. De Haan, “Exploiting spatial redundancy of image sensor for motion robust rPPG,” IEEE Trans. Biomed. Eng. 62(2), 415–425 (2015). [CrossRef]