Abstract

Three-dimensional geometrical models with incorporated surface temperature data provide important information for various applications such as medical imaging, energy auditing, and intelligent robots. In this paper we present a robust method for mobile and real-time 3D thermographic reconstruction through depth and thermal sensor fusion. A multimodal imaging device consisting of a thermal camera and a RGB-D sensor is calibrated geometrically and used for data capturing. Based on the underlying principle that temperature information remains robust against illumination and viewpoint changes, we present a Thermal-guided Iterative Closest Point (T-ICP) methodology to facilitate reliable 3D thermal scanning applications. The pose of sensing device is initially estimated using correspondences found through maximizing the thermal consistency between consecutive infrared images. The coarse pose estimate is further refined by finding the motion parameters that minimize a combined geometric and thermographic loss function. Experimental results demonstrate that complimentary information captured by multimodal sensors can be utilized to improve performance of 3D thermographic reconstruction. Through effective fusion of thermal and depth data, the proposed approach generates more accurate 3D thermal models using significantly less scanning data.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article

Corrections

30 March 2018: A typographical correction was made to the author affiliations.


OSA Recommended Articles
Bound constrained bundle adjustment for reliable 3D reconstruction

Yuanzheng Gong, De Meng, and Eric J. Seibel
Opt. Express 23(8) 10771-10785 (2015)

3D reconstruction of cystoscopy videos for comprehensive bladder records

Kristen L. Lurie, Roland Angst, Dimitar V. Zlatev, Joseph C. Liao, and Audrey K. Ellerbee Bowden
Biomed. Opt. Express 8(4) 2106-2123 (2017)

References

  • View by:
  • |
  • |
  • |

  1. S. Vidas and P. Moghadam, “Heatwave: A handheld 3D thermography system for energy auditing,” Energy Build. 66(5), 445–460 (2013).
    [Crossref]
  2. S. Vidas, P. Moghadam, and M. Bosse, “3D thermal mapping of building interiors using an RGB-D and thermal camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 2311–2318.
  3. S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
    [Crossref]
  4. A. O. Müller and A. Kroll, “Generating high fidelity 3-D thermograms with a handheld real-time thermal imaging system,” IEEE Sensors J. 17(3), 774–783 (2017).
    [Crossref]
  5. S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.
  6. T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.
  7. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
    [Crossref]
  8. A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
    [Crossref]
  9. Y. Cao and J. McDonald, “Improved feature extraction and matching in urban environments based on 3D viewpoint normalization,” Comput. Vis. Image Underst. 116(1), 86–101 (2012).
    [Crossref]
  10. M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
    [Crossref]
  11. Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
    [Crossref]
  12. T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
    [Crossref]
  13. J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.
  14. S. Zhang, D. Royer, and S.-T. Yau, “GPU-assisted high-resolution, real-time 3-D shape measurement,” Opt. Express 14(20), 9120–9129 (2006).
    [Crossref] [PubMed]
  15. M. Nießner, A. Dai, and M. Fisher, “Combining inertial navigation and ICP for real-time 3D surface reconstruction,” in Proceedings of Eurographics (Short Papers), (ACM, 2014), pp. 13–16.
  16. J. Shi and C. Tomasi, “Good features to track,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 1994), pp. 593–600.
  17. S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).
  18. Y. An and S. Zhang, “High-resolution, real-time simultaneous 3D surface geometry and temperature measurement,” Opt. Express 24(13), 14552–14563 (2016).
    [Crossref] [PubMed]
  19. H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
    [Crossref]
  20. G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
    [Crossref]
  21. D. Borrmann, J. Elseberg, and A. Nüchter, “Thermal 3D mapping of building façades,” in Proceedings of International Conference on Intelligent Autonomous Systems (Springer, 2012) pp. 173–182.
  22. S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
    [Crossref]
  23. W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.
  24. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis Mach. Intell. 22(11), 1330–1334 (2000).
    [Crossref]
  25. J.T Lussier and S. Thrun, “Automatic calibration of RGBD and thermal cameras,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2014), pp. 451–458.
  26. S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
    [Crossref]
  27. G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
    [Crossref]
  28. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, (Cambridge University, 2003).
  29. F. Steinbrücker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-D images,” in Proceedings of IEEE International Conference on Computer Vision Workshops (IEEE, 2011), pp. 719–722.
  30. N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.
  31. K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
    [Crossref]
  32. R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.
  33. B. Zeise and B. Wagner, “Temperature Correction and Reflection Removal in Thermal Images using 3D Temperature Mapping,” in Proceedings of International Conference on Informatics in Control, Automation and Robotics (Springer, 2016), pp. 158–165.
  34. Y. Li and M.S. Brown, “Exploiting reflection change for automatic reflection removal,” in Proceedings of IEEE International Conference on Computer Vision (IEEE, 2013), pp. 2432–2439.

2018 (1)

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

2017 (1)

A. O. Müller and A. Kroll, “Generating high fidelity 3-D thermograms with a handheld real-time thermal imaging system,” IEEE Sensors J. 17(3), 774–783 (2017).
[Crossref]

2016 (2)

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Y. An and S. Zhang, “High-resolution, real-time simultaneous 3D surface geometry and temperature measurement,” Opt. Express 24(13), 14552–14563 (2016).
[Crossref] [PubMed]

2015 (4)

G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
[Crossref]

S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
[Crossref]

Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
[Crossref]

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

2013 (1)

S. Vidas and P. Moghadam, “Heatwave: A handheld 3D thermography system for energy auditing,” Energy Build. 66(5), 445–460 (2013).
[Crossref]

2012 (4)

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Y. Cao and J. McDonald, “Improved feature extraction and matching in urban environments based on 3D viewpoint normalization,” Comput. Vis. Image Underst. 116(1), 86–101 (2012).
[Crossref]

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

2011 (1)

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

2008 (2)

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
[Crossref]

2006 (1)

2000 (1)

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

1987 (1)

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
[Crossref]

Akbarzadeh, A.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

An, Y.

Arias, P.

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

Armesto, J.

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

Arun, K. S.

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
[Crossref]

Avidan, S.

N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.

Blostein, S. D.

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
[Crossref]

Borrmann, D.

G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
[Crossref]

D. Borrmann, J. Elseberg, and A. Nüchter, “Thermal 3D mapping of building façades,” in Proceedings of International Conference on Intelligent Autonomous Systems (Springer, 2012) pp. 173–182.

Bosse, M.

S. Vidas, P. Moghadam, and M. Bosse, “3D thermal mapping of building interiors using an RGB-D and thermal camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 2311–2318.

Brown, M.S.

Y. Li and M.S. Brown, “Exploiting reflection change for automatic reflection removal,” in Proceedings of IEEE International Conference on Computer Vision (IEEE, 2013), pp. 2432–2439.

Burgard, W.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

Cao, Y.

Y. Cao and J. McDonald, “Improved feature extraction and matching in urban environments based on 3D viewpoint normalization,” Comput. Vis. Image Underst. 116(1), 86–101 (2012).
[Crossref]

Cardone, G.

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

Cho, Y. K.

Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
[Crossref]

Chung, S. H.

A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
[Crossref]

Clipp, B.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Cremers, D.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

F. Steinbrücker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-D images,” in Proceedings of IEEE International Conference on Computer Vision Workshops (IEEE, 2011), pp. 719–722.

Dai, A.

M. Nießner, A. Dai, and M. Fisher, “Combining inertial navigation and ICP for real-time 3D surface reconstruction,” in Proceedings of Eurographics (Short Papers), (ACM, 2014), pp. 13–16.

Davison, A.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

de Sorbier, F.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Dello Ioio, G.

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

Demisse, G. G.

G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
[Crossref]

Denman, S.

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

Elseberg, J.

D. Borrmann, J. Elseberg, and A. Nüchter, “Thermal 3D mapping of building façades,” in Proceedings of International Conference on Intelligent Autonomous Systems (Springer, 2012) pp. 173–182.

Endres, F.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

Engelhard, N.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

Engels, C.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Fallon, M.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

Fisher, M.

M. Nießner, A. Dai, and M. Fisher, “Combining inertial navigation and ICP for real-time 3D surface reconstruction,” in Proceedings of Eurographics (Short Papers), (ACM, 2014), pp. 13–16.

Fitzgibbon, A.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Fookes, C.

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

Fox, D.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Frahm, J.-M.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Freeman, D.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Frisch, J.

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

Gallup, D.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Golpavar-Fard, M.

Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
[Crossref]

Grisetti, G.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

Ham, Y.

Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
[Crossref]

Hartley, R.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, (Cambridge University, 2003).

Henry, P.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Herbst, E.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Hilliges, O.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Hodges, S.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Hong, J.

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Huang, T. S.

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
[Crossref]

Ianiro, A.

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

Iketani, A.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Izadi, S.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Johannsson, H.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

Kaess, M.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

Kim, D.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Kim, S.-J.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Kohli, P.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Konolige, K.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

Krainin, M.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Kroll, A.

A. O. Müller and A. Kroll, “Generating high fidelity 3-D thermograms with a handheld real-time thermal imaging system,” IEEE Sensors J. 17(3), 774–783 (2017).
[Crossref]

Kümmerle, R.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

Lagüela, S.

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

Lakemond, R.

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

Leonard, J. J.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

Li, B.

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Li, X.

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Li, Y.

Y. Li and M.S. Brown, “Exploiting reflection change for automatic reflection removal,” in Proceedings of IEEE International Conference on Computer Vision (IEEE, 2013), pp. 2432–2439.

Lussier, J.T

J.T Lussier and S. Thrun, “Automatic calibration of RGBD and thermal cameras,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2014), pp. 451–458.

Martínez, J.

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

Matsumoto, K.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Matusik, W.

N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.

McDonald, J.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

Y. Cao and J. McDonald, “Improved feature extraction and matching in urban environments based on 3D viewpoint normalization,” Comput. Vis. Image Underst. 116(1), 86–101 (2012).
[Crossref]

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

Merrell, P.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Metzmacher, H.

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

Moghadam, P.

S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
[Crossref]

S. Vidas and P. Moghadam, “Heatwave: A handheld 3D thermography system for energy auditing,” Energy Build. 66(5), 445–460 (2013).
[Crossref]

S. Vidas, P. Moghadam, and M. Bosse, “3D thermal mapping of building interiors using an RGB-D and thermal camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 2311–2318.

Molyneaux, D.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Mordohai, P.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Morris, N. J.

N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.

Müller, A. O.

A. O. Müller and A. Kroll, “Generating high fidelity 3-D thermograms with a handheld real-time thermal imaging system,” IEEE Sensors J. 17(3), 774–783 (2017).
[Crossref]

Nakagawa, W.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Newcombe, R.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Ng, A. Y.

A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
[Crossref]

Nießner, M.

M. Nießner, A. Dai, and M. Fisher, “Combining inertial navigation and ICP for real-time 3D surface reconstruction,” in Proceedings of Eurographics (Short Papers), (ACM, 2014), pp. 13–16.

Nistér, D.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Nüchter, A.

G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
[Crossref]

D. Borrmann, J. Elseberg, and A. Nüchter, “Thermal 3D mapping of building façades,” in Proceedings of International Conference on Intelligent Autonomous Systems (Springer, 2012) pp. 173–182.

Passaro, A.

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

Pfister, H.

N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.

Pollefeys, M.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Ren, X.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Royer, D.

Saito, H.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Salmi, C.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Saxena, A.

A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
[Crossref]

Schmidt, C.

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

Senda, S.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Shi, J.

J. Shi and C. Tomasi, “Good features to track,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 1994), pp. 593–600.

Shibata, T.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Shotton, J.

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

Sinha, S.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Sridharan, S.

S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
[Crossref]

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

Steinbrücker, F.

F. Steinbrücker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-D images,” in Proceedings of IEEE International Conference on Computer Vision Workshops (IEEE, 2011), pp. 719–722.

Stewenius, H.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Strasdat, H.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

Sturm, J.

F. Steinbrücker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-D images,” in Proceedings of IEEE International Conference on Computer Vision Workshops (IEEE, 2011), pp. 719–722.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

Sugimoto, M.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

Talton, B.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Thrun, S.

J.T Lussier and S. Thrun, “Automatic calibration of RGBD and thermal cameras,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2014), pp. 451–458.

Tomasi, C.

J. Shi and C. Tomasi, “Good features to track,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 1994), pp. 593–600.

Towles, H.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

van Treeck, C.

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

Vidas, S.

S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
[Crossref]

S. Vidas and P. Moghadam, “Heatwave: A handheld 3D thermography system for energy auditing,” Energy Build. 66(5), 445–460 (2013).
[Crossref]

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

S. Vidas, P. Moghadam, and M. Bosse, “3D thermal mapping of building interiors using an RGB-D and thermal camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 2311–2318.

Wagner, B.

B. Zeise and B. Wagner, “Temperature Correction and Reflection Removal in Thermal Images using 3D Temperature Mapping,” in Proceedings of International Conference on Informatics in Control, Automation and Robotics (Springer, 2016), pp. 158–165.

Wang, L.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Wark, T.

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

Welch, G.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Whelan, T.

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

Wölki, D.

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

Yang, Q.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Yang, R.

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

Yau, S.-T.

Zeise, B.

B. Zeise and B. Wagner, “Temperature Correction and Reflection Removal in Thermal Images using 3D Temperature Mapping,” in Proceedings of International Conference on Informatics in Control, Automation and Robotics (Springer, 2016), pp. 158–165.

Zhang, K.

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Zhang, S.

Zhang, Z.

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

Zheng, S.

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Zisserman, A.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, (Cambridge University, 2003).

Adv. Eng. Informatics (1)

Y. K. Cho, Y. Ham, and M. Golpavar-Fard, “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art,” Adv. Eng. Informatics 29(2), 184–195 (2015).
[Crossref]

Comput. Des. (1)

S. Zheng, J. Hong, K. Zhang, B. Li, and X. Li, “A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM,” Comput. Des. 78(C), 107–117 (2016).

Comput. Vis. Image Underst. (1)

Y. Cao and J. McDonald, “Improved feature extraction and matching in urban environments based on 3D viewpoint normalization,” Comput. Vis. Image Underst. 116(1), 86–101 (2012).
[Crossref]

Energy Build. (3)

H. Metzmacher, D. Wölki, C. Schmidt, J. Frisch, and C. van Treeck, “Real-time human skin temperature analysis using thermal image recognition for thermal comfort assessment,” Energy Build. 158, 1063–1078 (2018).
[Crossref]

S. Lagüela, J. Martínez, J. Armesto, and P. Arias, “Energy efficiency studies through 3D laser scanning and thermographic technologies,” Energy Build. 43(6), 1216–1221 (2011).
[Crossref]

S. Vidas and P. Moghadam, “Heatwave: A handheld 3D thermography system for energy auditing,” Energy Build. 66(5), 445–460 (2013).
[Crossref]

Exp. fluids (1)

G. Cardone, A. Ianiro, G. Dello Ioio, and A. Passaro, “Temperature maps measurements on 3D surfaces with infrared thermography,” Exp. fluids 52(2), 375–385 (2012).
[Crossref]

IEEE Sensors J. (2)

S. Vidas, P. Moghadam, and S. Sridharan, “Real-time mobile 3D temperature mapping,” IEEE Sensors J. 15(2), 1145–1152 (2015).
[Crossref]

A. O. Müller and A. Kroll, “Generating high fidelity 3-D thermograms with a handheld real-time thermal imaging system,” IEEE Sensors J. 17(3), 774–783 (2017).
[Crossref]

IEEE Transactions on Instrumentation Meas. (1)

S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” IEEE Transactions on Instrumentation Meas. 61(6), 1625–1635 (2012).
[Crossref]

IEEE Transactions on Pattern Analysis Mach. Intell. (2)

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis Mach. Intell. 9(5), 698–700 (1987).
[Crossref]

Int. J. Comput. Vis. (2)

M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, and H. Towles, “Detailed real-time urban 3D reconstruction from video,” Int. J. Comput. Vis. 78(2), 143–167 (2008).
[Crossref]

A. Saxena, S. H. Chung, and A. Y. Ng, “3-D depth reconstruction from a single still image,” Int. J. Comput. Vis. 76(1), 53–69 (2008).
[Crossref]

J. Intell. & Robotic Syst. (1)

G. G. Demisse, D. Borrmann, and A. Nüchter, “Interpreting thermal 3D models of indoor environments for energy efficiency,” J. Intell. & Robotic Syst. 77(1), 55–72 (2015).
[Crossref]

Opt. Express (2)

The Int. J. Robotics Res. (2)

T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense RGB-D SLAM with volumetric fusion,” The Int. J. Robotics Res. 34(4-5), 598–626 (2015).
[Crossref]

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments,” The Int. J. Robotics Res. 31(5), 647–663 (2012).
[Crossref]

Other (15)

S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, “KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera,” in, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, (ACM, 2011), pp. 559–568.

T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-D mapping,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 5724–5731.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 573–580.

M. Nießner, A. Dai, and M. Fisher, “Combining inertial navigation and ICP for real-time 3D surface reconstruction,” in Proceedings of Eurographics (Short Papers), (ACM, 2014), pp. 13–16.

J. Shi and C. Tomasi, “Good features to track,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 1994), pp. 593–600.

D. Borrmann, J. Elseberg, and A. Nüchter, “Thermal 3D mapping of building façades,” in Proceedings of International Conference on Intelligent Autonomous Systems (Springer, 2012) pp. 173–182.

W. Nakagawa, K. Matsumoto, F. de Sorbier, M. Sugimoto, H. Saito, S. Senda, T. Shibata, and A. Iketani, “Visualization of temperature change using RGB-D camera and thermal camera,” in Proceedings of European Conference on Computer Vision Workshops (Springer, 2014), pp. 386–400.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, (Cambridge University, 2003).

F. Steinbrücker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-D images,” in Proceedings of IEEE International Conference on Computer Vision Workshops (IEEE, 2011), pp. 719–722.

N. J. Morris, S. Avidan, W. Matusik, and H. Pfister, “Statistics of infrared images,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–7.

R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “g2o: A general framework for graph optimization,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 3607–3613.

B. Zeise and B. Wagner, “Temperature Correction and Reflection Removal in Thermal Images using 3D Temperature Mapping,” in Proceedings of International Conference on Informatics in Control, Automation and Robotics (Springer, 2016), pp. 158–165.

Y. Li and M.S. Brown, “Exploiting reflection change for automatic reflection removal,” in Proceedings of IEEE International Conference on Computer Vision (IEEE, 2013), pp. 2432–2439.

J.T Lussier and S. Thrun, “Automatic calibration of RGBD and thermal cameras,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2014), pp. 451–458.

S. Vidas, P. Moghadam, and M. Bosse, “3D thermal mapping of building interiors using an RGB-D and thermal camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2013), pp. 2311–2318.

Supplementary Material (2)

NameDescription
» Visualization 1       Robustness against large displacement
» Visualization 2       Accuracy of 3D reconstruction

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1 Thermal camera and depth sensor are rigidly attached using an acrylic frame.
Fig. 2
Fig. 2 Distortion correction of an infrared camera. (a) A raw infrared image with apparent lens distortions and (b) Result of lens distortion compensation. The edge of a square calibration board appears a straight line after distortion correction.
Fig. 3
Fig. 3 3D alignment of thermal and depth camera coordinate systems. (a) Initial alignment without extrinsic calibration, (b) Alignment result using the estimated extrinsic matrix. Note the heat colored point clouds are obtained by mapping each pixel of 2D thermal image to its counterpart in the 3D point cloud.
Fig. 4
Fig. 4 Flowchart of the proposed Thermal-guided ICP method.
Fig. 5
Fig. 5 The computed optimal 2D translation u ^ can be used to shift the projection p and establish correct correspondences between two wide baseline scenes. The correspondences are shown in thermal images for visualization.
Fig. 6
Fig. 6 The multimodal benchmark used for evaluation of 3D thermographic reconstruction solutions including (a) A person with clothes, (b) A person without clothes, (c) A water boiler, (d) An office chair.
Fig. 7
Fig. 7 Samples of 3D thermal reconstruction. A successful 3D reconstruction (left) and a failed example (right).
Fig. 8
Fig. 8 3D thermal reconstruction results when the frame rate is decreased to 1/72. (a) Results of ICP, (b) Results of RGBD-ICP, and (c) Results of T-ICP. More comparative results are provided in Visualization 1.
Fig. 9
Fig. 9 Wide baseline alignment using ICP [5], Visible-guided ICP (V-ICP) and Thermal-guided ICP (T-ICP).
Fig. 10
Fig. 10 3D thermographic reconstruction by considering different pose constrains. (a) Results of ICP, (b) Results of RGBD-ICP, and (c) Results of T-ICP. Please see Visualization 2 to check the highlighted details.

Tables (4)

Tables Icon

Algorithm 1 Correspondence Accumulation For Wide Baseline 3D Alignment

Tables Icon

Algorithm 2 Accumulation of 3D Correspondences and Thermal Interest Points at k-th Iteration for Pose Refinement

Tables Icon

Table 1 Averaged minimum number of frames required to complete 3D reconstruction.

Tables Icon

Table 2 Absolute trajectory error (RMSE between the estimated and the ground truth trajectories) of camera pose estimation using different pose constrains.

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

E t c ( u ) = x Ω ( I i ( x + u ) I i 1 ( x ) ) 2 ,
u k = u k 1 + Δ u .
E t c ( Δ u ) x Ω ( I i ( x + u k 1 ) Δ u + I i ( x + u k 1 ) I i 1 ( x ) ) 2 = J r t c Δ u + r t c 2 ,
J r t c J r t c Δ u = J r t c r t c .
E ( T i ) = E i c p ( T i ) + ω E t d ( T i ) ,
E i c p ( T i ) = 1 M ( x , p ) T ( V i 1 g ( x ) T i V i ( p ) ) N i 1 g ( x ) 2 = 1 M ( x , p ) T ( v i 1 g T i v i ) n i 1 g 2 ,
E t d ( T i ) = 1 N x P ( I i ( κ ( Ψ ( V i 1 g ( x ) , T i ) ) ) I i 1 ( x ) ) 2 = 1 N x P ( I i ( κ ( Ψ ( v i 1 g , T i ) ) ) I i 1 ( x ) ) 2 ,
Ψ ( v i 1 g , T i ) = T i 1 v i 1 g ,
κ ( v ) = [ v x f x v z + c x , v y f y v z + c y ] ,
T i k = Δ T T i k 1 ( I + ξ ^ ) T i k 1 ,
ξ ^ = ( 0 γ β t x γ 0 α t y β α 0 t z 0 0 0 0 ) .
E i c p ( ξ ) 1 M ( x , p ) T ( v i 1 g ( I + ξ ^ ) T i k 1 v i ) n i 1 g 2 = 1 M ( x , p ) T [ v i × n i 1 g n i 1 g ] ξ + ( v i 1 g T i k 1 v i ) n i 1 g 2 = 1 M J r i c p ξ + r i c p 2 ,
E t d ( ξ ) 1 N x P ( I i ( κ ( Ψ ( v i 1 g , ( I + ξ ^ ) T i k 1 ) ) ) I i 1 ( x ) ) 2 1 N x P ( I i ( κ ) J κ ( Ψ ) J Ψ ( ξ ) | ξ = 0 ξ + I i ( κ ( Ψ ( v i 1 g , T i k 1 ) ) ) I i 1 ( x ) ) 2 = 1 N J r t d ξ + r t d 2 ,
[ 1 M J r i c p μ N J r t d ] [ 1 M J r i c p μ N J r t d ] ξ = [ 1 M J r i c p μ N J r t d ] [ 1 M r i c p μ N r t d ] ,
( 1 M J r i c p J r i c p + ω N J r t d J r t d ) ξ = 1 M J r i c p r i c p μ N J r t d J r t d ,

Metrics