Abstract

Hybrid concepts are often used to improve existing methods in many fields. We developed a hybrid optical system that consists of multiple color cameras and one depth camera to make up the concavity problem of the visual hull construction. The heterogeneous data from the color cameras and the depth camera is fused in an effective way. The experimental results show that the proposed hybrid system can reconstruct concave objects successfully by combining the visual hull and the depth data.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. Y. Furukawa and J. Ponce, “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010).
    [CrossRef]
  2. G. H. Liu, X. Y. Liu, and Q. Y. Feng, “High-accuracy three-dimensional shape acquisition of a large-scale object from multiple uncalibrated camera views,” Appl. Opt. 50, 3691–3701 (2011).
    [CrossRef]
  3. A. Laurentini, “The visual hull concept for silhouette based image understanding,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 150–162 (1994).
    [CrossRef]
  4. E. Simioni, F. Ratti, I. Calliari, and L. Poletto, “Three-dimensional modeling using x-ray shape-from-silhouette,” Appl. Opt. 50, 3282–3288 (2011).
    [CrossRef]
  5. W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.
  6. J. Feng, B. Song, and B. Zhou, “Bottom and concave surface rendering in image-based visual hull,” in Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (2008), paper 3.
  7. http://www.cyberware.com/products/scanners/px.html .
  8. http://www.pmdtec.com/ .
  9. http://en.wikipedia.org/wiki/Canesta .
  10. http://www.mesa-imaging.ch/ .
  11. http://www.microsoft.com/en-us/kinectforwindows/ .
  12. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
    [CrossRef]
  13. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.
  14. http://www.3d3solutions.com/products/3d-scanner/hdi-advance/ .
  15. F. Keith, V. Anthon, and B. Ndimi, “Using silhouette consistency constraints to build 3D models,” in Proceedings of Fourteenth Annual South African Workshop on Pattern Recognition (PRASA) (2003).
  16. J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).
  17. U. Hahne and M. Alexa, “Depth imaging by combining time-of-flight and on-demand stereo,” in Proceedings of the Dynamic 3D Imaging Workshop (Dyn3D) (2009), pp. 70–83.
  18. Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.
  19. E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).
  20. C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.
  21. A. Bogomjakov, C. Gotsman, and M. Magnor, “Free-viewpoint video from depth cameras,” in Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV) (2006), pp. 89–96.
  22. L. Guan, J. S. Franco, and M. Pollefeys, “3D object reconstruction with heterogeneous sensor data,” in Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008), paper 108.
  23. C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for 3D object modeling,” Comput. Vis. Image Underst. 96, 367–392 (2004).
    [CrossRef]
  24. J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
    [CrossRef]
  25. A. Weiss, D. Hirshberg, and M. J. Black, “Home 3D body scans from noisy image and range data,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1951–1958.
  26. J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).
  27. K. Nakano and H. Chikatsu, “Camera calibration techniques using multiple cameras of different resolutions and bundle of distances,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII, 484–489 (2010).
  28. http://www.vision.caltech.edu/bouguetj/calib_doc/ .
  29. W. Matusik, C. Buehler, and L. McMillan, “Polyhedral visual hulls for real-time rendering,” in Proceedings of Twelfth Eurographics Workshop on Rendering (EGWR) (2001), pp. 115–125.
  30. B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in SIGGRAPH, Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 303–312.
  31. T. Akenine-Möller and E. Hanines, “Acceleration algorithms: backface culling,” in Real-Time Rendering, 2nd ed. (2002), Chap. 9.3, pp. 359–363.
  32. N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. Syst. Man Cybern. SMC-8, 62–66 (1979).
  33. Y. K. Liu and B. Zalik, “An efficient chain code with Huffman coding,” Pattern Recogn. 38, 553–557 (2005).
    [CrossRef]
  34. E. Haines, “Point in polygon strategies,” in Graphics Gems IV, P. S. Heckbert, ed. (Morgan Kauffman, 1994), pp. 24–46.
  35. O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.
  36. http://pointclouds.org/ .

2012 (2)

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

2011 (2)

2010 (2)

Y. Furukawa and J. Ponce, “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010).
[CrossRef]

K. Nakano and H. Chikatsu, “Camera calibration techniques using multiple cameras of different resolutions and bundle of distances,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII, 484–489 (2010).

2005 (1)

Y. K. Liu and B. Zalik, “An efficient chain code with Huffman coding,” Pattern Recogn. 38, 553–557 (2005).
[CrossRef]

2004 (1)

C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for 3D object modeling,” Comput. Vis. Image Underst. 96, 367–392 (2004).
[CrossRef]

2003 (1)

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

1994 (1)

A. Laurentini, “The visual hull concept for silhouette based image understanding,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 150–162 (1994).
[CrossRef]

1979 (1)

N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. Syst. Man Cybern. SMC-8, 62–66 (1979).

Akenine-Möller, T.

T. Akenine-Möller and E. Hanines, “Acceleration algorithms: backface culling,” in Real-Time Rendering, 2nd ed. (2002), Chap. 9.3, pp. 359–363.

Alexa, M.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

U. Hahne and M. Alexa, “Depth imaging by combining time-of-flight and on-demand stereo,” in Proceedings of the Dynamic 3D Imaging Workshop (Dyn3D) (2009), pp. 70–83.

Anthon, V.

F. Keith, V. Anthon, and B. Ndimi, “Using silhouette consistency constraints to build 3D models,” in Proceedings of Fourteenth Annual South African Workshop on Pattern Recognition (PRASA) (2003).

Black, M. J.

A. Weiss, D. Hirshberg, and M. J. Black, “Home 3D body scans from noisy image and range data,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1951–1958.

Bogomjakov, A.

A. Bogomjakov, C. Gotsman, and M. Magnor, “Free-viewpoint video from depth cameras,” in Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV) (2006), pp. 89–96.

Buehler, C.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

W. Matusik, C. Buehler, and L. McMillan, “Polyhedral visual hulls for real-time rendering,” in Proceedings of Twelfth Eurographics Workshop on Rendering (EGWR) (2001), pp. 115–125.

Cai, Q.

E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).

Calliari, I.

Carranza, J.

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

Chikatsu, H.

K. Nakano and H. Chikatsu, “Camera calibration techniques using multiple cameras of different resolutions and bundle of distances,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII, 484–489 (2010).

Cohen-Or, D.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

Curless, B.

B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in SIGGRAPH, Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 303–312.

Davis, J.

J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).

Davison, A. J.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Diebel, J.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Esteban, C. H.

C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for 3D object modeling,” Comput. Vis. Image Underst. 96, 367–392 (2004).
[CrossRef]

Feng, J.

J. Feng, B. Song, and B. Zhou, “Bottom and concave surface rendering in image-based visual hull,” in Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (2008), paper 3.

Feng, Q. Y.

Fitzgibbon, A.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Fox, D.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Franco, J. S.

L. Guan, J. S. Franco, and M. Pollefeys, “3D object reconstruction with heterogeneous sensor data,” in Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008), paper 108.

Furukawa, Y.

Y. Furukawa and J. Ponce, “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010).
[CrossRef]

Gortler, S. J.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

Gotsman, C.

A. Bogomjakov, C. Gotsman, and M. Magnor, “Free-viewpoint video from depth cameras,” in Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV) (2006), pp. 89–96.

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

Gross, M.

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

Guan, L.

L. Guan, J. S. Franco, and M. Pollefeys, “3D object reconstruction with heterogeneous sensor data,” in Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008), paper 108.

Hahne, U.

U. Hahne and M. Alexa, “Depth imaging by combining time-of-flight and on-demand stereo,” in Proceedings of the Dynamic 3D Imaging Workshop (Dyn3D) (2009), pp. 70–83.

Haines, E.

E. Haines, “Point in polygon strategies,” in Graphics Gems IV, P. S. Heckbert, ed. (Morgan Kauffman, 1994), pp. 24–46.

Hanines, E.

T. Akenine-Möller and E. Hanines, “Acceleration algorithms: backface culling,” in Real-Time Rendering, 2nd ed. (2002), Chap. 9.3, pp. 359–363.

Henry, P.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Herbst, E.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Hilliges, O.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Hirshberg, D.

A. Weiss, D. Hirshberg, and M. J. Black, “Home 3D body scans from noisy image and range data,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1951–1958.

Hodges, S.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Izadi, S.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Keith, F.

F. Keith, V. Anthon, and B. Ndimi, “Using silhouette consistency constraints to build 3D models,” in Proceedings of Fourteenth Annual South African Workshop on Pattern Recognition (PRASA) (2003).

Kim, D.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Kim, Y. M.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Kohli, P.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Kosecka, J.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Krainin, M.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Kuster, C.

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

Laurentini, A.

A. Laurentini, “The visual hull concept for silhouette based image understanding,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 150–162 (1994).
[CrossRef]

Levoy, M.

B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in SIGGRAPH, Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 303–312.

Lipman, Y.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

Liu, G. H.

Liu, L.

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Liu, X. Y.

Liu, Y. K.

Y. K. Liu and B. Zalik, “An efficient chain code with Huffman coding,” Pattern Recogn. 38, 553–557 (2005).
[CrossRef]

Magnor, M.

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

A. Bogomjakov, C. Gotsman, and M. Magnor, “Free-viewpoint video from depth cameras,” in Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV) (2006), pp. 89–96.

Matusik, W.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

W. Matusik, C. Buehler, and L. McMillan, “Polyhedral visual hulls for real-time rendering,” in Proceedings of Twelfth Eurographics Workshop on Rendering (EGWR) (2001), pp. 115–125.

McMillan, L.

W. Matusik, C. Buehler, and L. McMillan, “Polyhedral visual hulls for real-time rendering,” in Proceedings of Twelfth Eurographics Workshop on Rendering (EGWR) (2001), pp. 115–125.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

Misusik, B.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Molyneaux, D.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Nakano, K.

K. Nakano and H. Chikatsu, “Camera calibration techniques using multiple cameras of different resolutions and bundle of distances,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII, 484–489 (2010).

Ndimi, B.

F. Keith, V. Anthon, and B. Ndimi, “Using silhouette consistency constraints to build 3D models,” in Proceedings of Fourteenth Annual South African Workshop on Pattern Recognition (PRASA) (2003).

Newcombe, R. A.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Otsu, N.

N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. Syst. Man Cybern. SMC-8, 62–66 (1979).

Pan, Z.

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Poletto, L.

Pollefeys, M.

L. Guan, J. S. Franco, and M. Pollefeys, “3D object reconstruction with heterogeneous sensor data,” in Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008), paper 108.

Ponce, J.

Y. Furukawa and J. Ponce, “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010).
[CrossRef]

Popa, T.

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

Raskar, R.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

Ratti, F.

Ren, X.

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Rossl, C.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

Schmitt, F.

C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for 3D object modeling,” Comput. Vis. Image Underst. 96, 367–392 (2004).
[CrossRef]

Seidel, H. P.

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

Shotton, J.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

Simioni, E.

Song, B.

J. Feng, B. Song, and B. Zhou, “Bottom and concave surface rendering in image-based visual hull,” in Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (2008), paper 3.

Sorkine, O.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

Theobalt, C.

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Thrun, S.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

Tola, E.

E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).

Tong, J.

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Wang, L.

J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).

Weiss, A.

A. Weiss, D. Hirshberg, and M. J. Black, “Home 3D body scans from noisy image and range data,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1951–1958.

Yan, H.

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Yang, R.

J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).

Zach, C.

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

Zalik, B.

Y. K. Liu and B. Zalik, “An efficient chain code with Huffman coding,” Pattern Recogn. 38, 553–557 (2005).
[CrossRef]

Zhang, C.

E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).

Zhang, Z.

E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).

Zhou, B.

J. Feng, B. Song, and B. Zhou, “Bottom and concave surface rendering in image-based visual hull,” in Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (2008), paper 3.

Zhou, J.

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Zhu, J.

J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).

ACM Trans. Graph. (1)

J. Carranza, C. Theobalt, M. Magnor, and H. P. Seidel, “Free-viewpoint video of human actors,” ACM Trans. Graph. 22, 569–577 (2003).
[CrossRef]

Appl. Opt. (2)

Comput. Vis. Image Underst. (1)

C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for 3D object modeling,” Comput. Vis. Image Underst. 96, 367–392 (2004).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

Y. Furukawa and J. Ponce, “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1362–1376 (2010).
[CrossRef]

A. Laurentini, “The visual hull concept for silhouette based image understanding,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 150–162 (1994).
[CrossRef]

IEEE Trans. Syst. Man Cybern. (1)

N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. Syst. Man Cybern. SMC-8, 62–66 (1979).

IEEE Trans. Vis. Comput. Graph. (1)

J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3D full human bodies using Kinects,” IEEE Trans. Vis. Comput. Graph. 18, 643–650 (2012).

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. (1)

K. Nakano and H. Chikatsu, “Camera calibration techniques using multiple cameras of different resolutions and bundle of distances,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXVIII, 484–489 (2010).

Int. J. Rob. Res. (1)

P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, “RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments,” Int. J. Rob. Res. 31, 647–663 (2012).
[CrossRef]

Pattern Recogn. (1)

Y. K. Liu and B. Zalik, “An efficient chain code with Huffman coding,” Pattern Recogn. 38, 553–557 (2005).
[CrossRef]

Other (25)

E. Haines, “Point in polygon strategies,” in Graphics Gems IV, P. S. Heckbert, ed. (Morgan Kauffman, 1994), pp. 24–46.

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H. P. Seidel, “Laplacian surface editing,” in Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP) (2004), pp. 175–184.

http://pointclouds.org/ .

A. Weiss, D. Hirshberg, and M. J. Black, “Home 3D body scans from noisy image and range data,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1951–1958.

http://www.vision.caltech.edu/bouguetj/calib_doc/ .

W. Matusik, C. Buehler, and L. McMillan, “Polyhedral visual hulls for real-time rendering,” in Proceedings of Twelfth Eurographics Workshop on Rendering (EGWR) (2001), pp. 115–125.

B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in SIGGRAPH, Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 303–312.

T. Akenine-Möller and E. Hanines, “Acceleration algorithms: backface culling,” in Real-Time Rendering, 2nd ed. (2002), Chap. 9.3, pp. 359–363.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: real-time dense surface mapping and tracking,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2011), pp. 127–136.

http://www.3d3solutions.com/products/3d-scanner/hdi-advance/ .

F. Keith, V. Anthon, and B. Ndimi, “Using silhouette consistency constraints to build 3D models,” in Proceedings of Fourteenth Annual South African Workshop on Pattern Recognition (PRASA) (2003).

J. Zhu, L. Wang, R. Yang, and J. Davis, “Fusion of time-of-flight depth and stereo for high accuracy depth maps,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008).

U. Hahne and M. Alexa, “Depth imaging by combining time-of-flight and on-demand stereo,” in Proceedings of the Dynamic 3D Imaging Workshop (Dyn3D) (2009), pp. 70–83.

Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Misusik, and S. Thrun, “Multi-view image and ToF sensor fusion for dense 3D reconstruction,” in Proceedings of IEEE Conference on Computer Vision Workshops (IEEE, 2009), pp. 1542–1549.

E. Tola, C. Zhang, Q. Cai, and Z. Zhang, “Virtual view generation with a hybrid camera array,” CVLAB-Report-2009-001 (École Polytechnique Fédérale de Lausanne, 2009).

C. Kuster, T. Popa, C. Zach, C. Gotsman, and M. Gross, “A hybrid camera system for interactive free-viewpoint video,” in Proceedings of Vison, Modeling, and Visualization (VMV) (2011), pp. 17–24.

A. Bogomjakov, C. Gotsman, and M. Magnor, “Free-viewpoint video from depth cameras,” in Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV) (2006), pp. 89–96.

L. Guan, J. S. Franco, and M. Pollefeys, “3D object reconstruction with heterogeneous sensor data,” in Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008), paper 108.

W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, and L. McMillan, “Image-based visual hulls,” in SIGGRAPH, Proceedings of the Conference on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 369–374.

J. Feng, B. Song, and B. Zhou, “Bottom and concave surface rendering in image-based visual hull,” in Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (2008), paper 3.

http://www.cyberware.com/products/scanners/px.html .

http://www.pmdtec.com/ .

http://en.wikipedia.org/wiki/Canesta .

http://www.mesa-imaging.ch/ .

http://www.microsoft.com/en-us/kinectforwindows/ .

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1.
Fig. 1.

(a) Original cup, (b) the image produced by a 3D scanner [14], (c) as reconstructed by the classical visual hull [15], and (d) as estimated by multiview stereo [1].

Fig. 2.
Fig. 2.

Overall framework of the proposed method.

Fig. 3.
Fig. 3.

Measurement of depth and scale of a planar object for the depth camera calibration.

Fig. 4.
Fig. 4.

Depth linearization.

Fig. 5.
Fig. 5.

Linearization of the scale. (a) The measured width of subject, and (b) the scale errors and the scale compensation function P.

Fig. 6.
Fig. 6.

(a) Visual hull and (b) the depth hull.

Fig. 7.
Fig. 7.

Signed distance function in 3D. In practice, no depth values align along the z axis as in the figure due to measurement error and noise. The depth value at the first measurement may be greater than the one at the others and vice versa. If we generate depth surfaces using the raw depth images, the surfaces may be shuffled and interpenetrate each other. The SDFs are computed over the depth images and their combination, subject to D=0, results in an optimal isosurface [30].

Fig. 8.
Fig. 8.

Overall procedure of depth enhancement.

Fig. 9.
Fig. 9.

Procedure for extracting the region of interest.

Fig. 10.
Fig. 10.

Extraction of the front surface. The color of the model represents the angle between each normal vector and the view direction.

Fig. 11.
Fig. 11.

Normal vectors: (a) the local artifacts, and (b) estimation of the normal vectors.

Fig. 12.
Fig. 12.

This figure shows that the concave region can be extracted even at the diagonal view. (a) The projected depth image of the front surface, (b) the captured depth image, (c) adaptive thresholding applied to the difference of (a) and (b), (d) the outlier removed using the iterative erosion and dilation operations, and (e) the inner region after the extraction of the contour.

Fig. 13.
Fig. 13.

Results. (a)–(d) show reconstruction of a concave region containing convex objects. It indicates why the concave region must be reconstructed based on the real measured data. (a) The visual hull, (b) the raw depth data of a concave region, (c) the geometry reconstructed by the proposed method, and (d) the textured model. (e)–(g) show the reconstructions of concave objects having color homogeneous regions. (h)–(j) show the concave region of earthenware reconstructed by the wireframe, depth map, and the textured model, respectively.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

D(x)=ikwi(x)di(x)ikwi,
Bd(x)=max(d(x))min(d(x)),did,{di|i=1k},
Gi=med(f(n⃗j,d⃗)|n⃗jnormal{V⃗i,N(V⃗i)}),
S={ViR3|Gi>T,i=1n},
F=DsDk,σF=i=1nj=1m(F(i,j)μ)2mn,g={adaptivethreshold(F)ifσF>αotherwise,
M=(Vi,Ei),ViS.
L(Vi)=Vi1|Ni|jNiVj.
ξi=himi,{imk}.
S=L1(M+ξ).

Metrics