Abstract

In this paper, we propose a progressive reliable points growing matching scheme to estimate the depth from the speckle projection image. First a self-adapting binarization is introduced to reduce the influence of inconsistent intensity. Then we apply local window-based correlation matching to get the initial disparity map. After the initialization, we formulate a progressive updating scheme to update the disparity estimation. There are two main steps in each round of updation. At first new reliable points are progressively selected based on three aspects of criterion including matching degree, confidence, and left–right consistency; then prediction-based growing matching is adopted to recalculate the disparity map from the reliable points. Finally, the more accurate depth map can be obtained by subpixel interpolation and transformation. The experimental results well demonstrate the effectiveness and low computational cost of our scheme.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. Y. Chen and B. Chen, “Measuring of a three-dimensional surface by use of a spatial distance computation,” Appl. Opt. 42, 1958–1972 (2003).
    [CrossRef]
  2. M. Schaffer, M. Grosse, and R. Kowarschik, “High-speed pattern projection for three-dimensional shape measurement using laser speckles,” Appl. Opt. 49, 3622–3629 (2010).
    [CrossRef]
  3. H. Dai and X. Su, “Shape measurement by digital speckle temporal sequence correlation with digital light projector,” Opt. Eng. 40, 793–800 (2001) (in Chinese).
    [CrossRef]
  4. B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).
  5. A. Shpunt and Z. Zalevsky, “Three-dimensional sensing using speckle patterns,” U.S. patent 0,096,783 (16April2009).
  6. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vis. 47, 7–42 (2002).
    [CrossRef]
  7. Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
    [CrossRef]
  8. C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
    [CrossRef]
  9. C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
    [CrossRef]
  10. K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
    [CrossRef]
  11. I. Haller and S. Nedevschi, “Design of interpolation functions for sub-pixel accuracy stereo-vision systems,” IEEE Trans. Image Process. 21, 889–898 (2011).
    [CrossRef]
  12. R. Zahib and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Proc. ECCV 801, 151–158 (1994).
    [CrossRef]
  13. K. Khoshelham and S. Elberink, “Accuracy and resolution of kinect depth data for indoor mapping applications,” Sensors 12, 1437–1454 (2012).
    [CrossRef]

2012 (3)

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

K. Khoshelham and S. Elberink, “Accuracy and resolution of kinect depth data for indoor mapping applications,” Sensors 12, 1437–1454 (2012).
[CrossRef]

2011 (1)

I. Haller and S. Nedevschi, “Design of interpolation functions for sub-pixel accuracy stereo-vision systems,” IEEE Trans. Image Process. 21, 889–898 (2011).
[CrossRef]

2010 (1)

2009 (1)

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

2003 (1)

2002 (2)

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vis. 47, 7–42 (2002).
[CrossRef]

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

2001 (1)

H. Dai and X. Su, “Shape measurement by digital speckle temporal sequence correlation with digital light projector,” Opt. Eng. 40, 793–800 (2001) (in Chinese).
[CrossRef]

1994 (1)

R. Zahib and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Proc. ECCV 801, 151–158 (1994).
[CrossRef]

Arieli, Y.

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).

Chen, B.

Chen, Y.

Dai, H.

H. Dai and X. Su, “Shape measurement by digital speckle temporal sequence correlation with digital light projector,” Opt. Eng. 40, 793–800 (2001) (in Chinese).
[CrossRef]

Elberink, S.

K. Khoshelham and S. Elberink, “Accuracy and resolution of kinect depth data for indoor mapping applications,” Sensors 12, 1437–1454 (2012).
[CrossRef]

Freedman, B.

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).

Grosse, M.

Haller, I.

I. Haller and S. Nedevschi, “Design of interpolation functions for sub-pixel accuracy stereo-vision systems,” IEEE Trans. Image Process. 21, 889–898 (2011).
[CrossRef]

He, B.

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

Hesser, J.

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

Khoshelham, K.

K. Khoshelham and S. Elberink, “Accuracy and resolution of kinect depth data for indoor mapping applications,” Sensors 12, 1437–1454 (2012).
[CrossRef]

Kowarschik, R.

Lin, X.

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

Machline, M.

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).

Maier, D.

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

Manner, R.

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

Muhlmann, K.

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

Nedevschi, S.

I. Haller and S. Nedevschi, “Design of interpolation functions for sub-pixel accuracy stereo-vision systems,” IEEE Trans. Image Process. 21, 889–898 (2011).
[CrossRef]

Nister, D.

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

Pei, X.

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

Schaffer, M.

Scharstein, D.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vis. 47, 7–42 (2002).
[CrossRef]

Shi, C.

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

Shpunt, A.

A. Shpunt and Z. Zalevsky, “Three-dimensional sensing using speckle patterns,” U.S. patent 0,096,783 (16April2009).

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).

Stewenius, H.

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

Su, X.

H. Dai and X. Su, “Shape measurement by digital speckle temporal sequence correlation with digital light projector,” Opt. Eng. 40, 793–800 (2001) (in Chinese).
[CrossRef]

Szeliski, R.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vis. 47, 7–42 (2002).
[CrossRef]

Wang, G.

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

Wang, L.

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

Woodfill, J.

R. Zahib and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Proc. ECCV 801, 151–158 (1994).
[CrossRef]

Yang, Q.

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

Yang, R.

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

Zahib, R.

R. Zahib and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Proc. ECCV 801, 151–158 (1994).
[CrossRef]

Zalevsky, Z.

A. Shpunt and Z. Zalevsky, “Three-dimensional sensing using speckle patterns,” U.S. patent 0,096,783 (16April2009).

Appl. Opt. (2)

IEEE Trans. Image Process. (1)

I. Haller and S. Nedevschi, “Design of interpolation functions for sub-pixel accuracy stereo-vision systems,” IEEE Trans. Image Process. 21, 889–898 (2011).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 492–504 (2009).
[CrossRef]

IEICE Trans. Inf. Syst. (2)

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “An interleaving updating framework of disparity and confidence map for stereo matching,” IEICE Trans. Inf. Syst. E95-D, 1552–1555 (2012).
[CrossRef]

C. Shi, G. Wang, X. Pei, B. He, and X. Lin, “Stereo matching using local plane fitting in confidence-based support window,” IEICE Trans. Inf. Syst. E95-D, 699–702 (2012).
[CrossRef]

Int. J. Comput. Vis. (2)

K. Muhlmann, D. Maier, J. Hesser, and R. Manner, “Calculating dense disparity maps from color stereo images, an efficient implementation,” Int. J. Comput. Vis. 47, 79–88 (2002).
[CrossRef]

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vis. 47, 7–42 (2002).
[CrossRef]

Opt. Eng. (1)

H. Dai and X. Su, “Shape measurement by digital speckle temporal sequence correlation with digital light projector,” Opt. Eng. 40, 793–800 (2001) (in Chinese).
[CrossRef]

Proc. ECCV (1)

R. Zahib and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Proc. ECCV 801, 151–158 (1994).
[CrossRef]

Sensors (1)

K. Khoshelham and S. Elberink, “Accuracy and resolution of kinect depth data for indoor mapping applications,” Sensors 12, 1437–1454 (2012).
[CrossRef]

Other (2)

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, “Depth mapping using projected patterns,” U.S. patent 0,118,123 (13May2010).

A. Shpunt and Z. Zalevsky, “Three-dimensional sensing using speckle patterns,” U.S. patent 0,096,783 (16April2009).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1.
Fig. 1.

Structure of a typical speckle projection system.

Fig. 2.
Fig. 2.

Illustration of a typical input image pair of laser speckle projection system. (a) Target speckle image and (b) reference speckle image.

Fig. 3.
Fig. 3.

Flow chart of proposed PRPGM with samples.

Fig. 4.
Fig. 4.

Sample of SAB result. (a) Patch of input image, (b) results of SAB, and (c) calculated shadow mask.

Fig. 5.
Fig. 5.

Illustration of the effect of SAB. (a) Original target image; (b) original reference image captured at a fixed distance; (c), (d) depth result from 13×13 local matching in original image using SAD and its error map; (e), (f) depth result using 13×13 census [12] and its error map; (g), (h) depth result from 13×13 window matching after SAB and its error map.

Fig. 6.
Fig. 6.

Results during the progressive growing rounds for a typical example. Upper: the intermediate depth results along with the progress, with the truth depth and NCC matching result as comparison. Lower: error rate and the execution time for each round.

Fig. 7.
Fig. 7.

Comparison of proposed method’s RMSE with theoretical error.

Fig. 8.
Fig. 8.

Depth estimation for human face surface. (a) Depth map produced by the proposed algorithm, and (b) synthesized view of the surface depth.

Fig. 9.
Fig. 9.

Some frames in our dataset videos. The upper row shows the captured IR speckle images, the middle row shows the result of NCC-based local window matching, and the bottom row shows the estimated depth with our method.

Fig. 10.
Fig. 10.

Comparison of the proposed algorithm with conventional method under different scenes.

Tables (2)

Tables Icon

Table 1. Symbols and Explanations in this Section

Tables Icon

Table 2. Results of the Plane Test

Equations (18)

Equations on this page are rendered with MathJax. Learn more.

I(x,y)={0,ifI(x,y)T(x,y)1,ifI(x,y)>T(x,y).
T(x,y)=1(2l+1)2s=llt=llI(x+s,y+t).
T(x,y)=μ(x,y)+Δc,
S(I1,I2)={1,ifI1=I20,ifI1I2.
f(x,y,d)=1NqΩ(p)S(L(xq,yq),R(xqd,yq)),
D(x,y)=argmaxdminddmaxf(x,y,d).
Conf(x,y)=f(x,y,D(x,y))maxdD(x,y)f(x,y,d)1maxdD(x,y)f(x,y,d).
LRC(x,y)=|DLR(x,y)DRL(xDLR(x,y),y)|.
RSet={(x,y)|f(x,y,D(x,y))TfHConf(x,y)TConfLRC(x,y)<TLRC},
TfHt=TfHt1ΔTfH,
TConft=TConft1ΔTConf.
TfMt=TfMt1ΔTfM.
TfH0nΔTfH>=TfM0.
x={ΔLΔR,ifΔLΔRΔRΔL,ifΔL>ΔR.
dfinal={d0.5+interpF(x),ifΔLΔRd+0.5interpF(x),ifΔL>ΔR,
interpF(x)=x2.
Z=sZ0s+dZ0,
B=1N(x,y)(|dC(x,y)dT(x,y)|>δd).

Metrics