Abstract

In this paper we present a new denoising method for the depth images of a 3D imaging sensor, based on the time-of-flight principle. We propose novel ways to use luminance-like information produced by a time-of flight camera along with depth images. Firstly, we propose a wavelet-based method for estimating the noise level in depth images, using luminance information. The underlying idea is that luminance carries information about the power of the optical signal reflected from the scene and is hence related to the signal-to-noise ratio for every pixel within the depth image. In this way, we can efficiently solve the difficult problem of estimating the non-stationary noise within the depth images. Secondly, we use luminance information to better restore object boundaries masked with noise in the depth images. Information from luminance images is introduced into the estimation formula through the use of fuzzy membership functions. In particular, we take the correlation between the measured depth and luminance into account, and the fact that edges (object boundaries) present in the depth image are likely to occur in the luminance image as well. The results on real 3D images show a significant improvement over the state-of-the-art in the field.

© 2010 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. C. L. Zitnick, and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
    [CrossRef]
  2. W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
    [CrossRef] [PubMed]
  3. S. K. Nayar, and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
    [CrossRef]
  4. A. Torralba, and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
    [CrossRef]
  5. S. Soatto, and P. Perona, ““Reducing ”structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 933–942 (1998).
    [CrossRef]
  6. R. Lange, and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
    [CrossRef]
  7. R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
    [CrossRef]
  8. J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
    [CrossRef] [PubMed]
  9. S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.
  10. S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).
  11. M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
    [CrossRef]
  12. T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).
  13. L. Jovanov, A. Pi?zurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).
  14. Lj. Jovanov, N. Petrovi’c, A. Pi?zurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).
  15. S. Schulte, B. Huysmans, Pi?zurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).
  16. S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
    [CrossRef]
  17. A. Benazza-Benyahia, and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
    [CrossRef] [PubMed]
  18. . “3D TV Production [online],” (2009).
  19. P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst. I Regul. Pap. 55(8), 2368–2377 (2008).
    [CrossRef]
  20. I. Daubechies, Ten Lectures on Wavelets (SIAM, Philadelphia, 1992).
  21. J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
    [CrossRef]
  22. S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
    [CrossRef]
  23. A. Pi?zurica, and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
    [CrossRef]
  24. S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process. 55, 319–323 (1993).
    [CrossRef]
  25. M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
    [CrossRef]
  26. A. Amer, and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 15, 113–118 (2005).
    [CrossRef]
  27. V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
    [CrossRef]
  28. R. Bracho, and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).
  29. D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
    [CrossRef]
  30. A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
    [CrossRef]
  31. “Swissranger sr4000 overview [online],” http://www.mesa-imaging.ch/prodview4k.php (2009).
  32. J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
    [CrossRef] [PubMed]
  33. G. J. Iddan, and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE 4298, 48–55 (2001).
    [CrossRef]
  34. D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
    [CrossRef]
  35. L. Zhang, and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast 51, 191–199 (2005).
    [CrossRef]

2009

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[CrossRef] [PubMed]

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
[CrossRef]

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
[CrossRef]

2008

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[CrossRef] [PubMed]

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst. I Regul. Pap. 55(8), 2368–2377 (2008).
[CrossRef]

2007

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
[CrossRef]

C. L. Zitnick, and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[CrossRef]

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
[CrossRef]

2006

A. Pi?zurica, and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[CrossRef]

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
[CrossRef]

2005

L. Zhang, and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast 51, 191–199 (2005).
[CrossRef]

A. Amer, and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 15, 113–118 (2005).
[CrossRef]

A. Benazza-Benyahia, and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[CrossRef] [PubMed]

2003

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

2002

A. Torralba, and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[CrossRef]

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

2001

G. J. Iddan, and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE 4298, 48–55 (2001).
[CrossRef]

R. Lange, and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[CrossRef]

2000

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[CrossRef]

1998

S. Soatto, and P. Perona, ““Reducing ”structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 933–942 (1998).
[CrossRef]

1996

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
[CrossRef] [PubMed]

1994

S. K. Nayar, and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[CrossRef]

1993

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process. 55, 319–323 (1993).
[CrossRef]

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[CrossRef]

Acheroy, M.

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

Amer, A.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
[CrossRef]

A. Amer, and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 15, 113–118 (2005).
[CrossRef]

Benazza-Benyahia, A.

A. Benazza-Benyahia, and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[CrossRef] [PubMed]

Chang, S.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[CrossRef]

De Backer, S.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

De Silva, D.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
[CrossRef]

Donoho, D.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[CrossRef]

Dubois, E.

A. Amer, and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 15, 113–118 (2005).
[CrossRef]

Fernando, W.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
[CrossRef]

Frank, M.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
[CrossRef]

Gauch, J.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
[CrossRef] [PubMed]

Ghazal, M.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
[CrossRef]

Ghrayeb, A.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
[CrossRef]

Guerrero-Colon, J. A.

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[CrossRef] [PubMed]

Hamprecht, F. A.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
[CrossRef]

Huysmans, B.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

Iddan, G. J.

G. J. Iddan, and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE 4298, 48–55 (2001).
[CrossRef]

Johnstone, I.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[CrossRef]

Johnstone, I. M.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[CrossRef]

Kang, S. B.

C. L. Zitnick, and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[CrossRef]

Kuijk, M.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
[CrossRef]

Lange, R.

R. Lange, and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[CrossRef]

Lemahieu, I.

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

Mancera, L.

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[CrossRef] [PubMed]

Miled, W.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[CrossRef] [PubMed]

Nakagawa, Y.

S. K. Nayar, and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[CrossRef]

Nayar, S. K.

S. K. Nayar, and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[CrossRef]

Nieuwenhove, R. G. J. S. D. V.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
[CrossRef]

Oliva, A.

A. Torralba, and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[CrossRef]

Olsen, S. I.

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process. 55, 319–323 (1993).
[CrossRef]

Parent, M.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[CrossRef] [PubMed]

Perona, P.

S. Soatto, and P. Perona, ““Reducing ”structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 933–942 (1998).
[CrossRef]

Pesquet, J.

A. Benazza-Benyahia, and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[CrossRef] [PubMed]

Pesquet, J.-C.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[CrossRef] [PubMed]

Philips, W.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

A. Pi?zurica, and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[CrossRef]

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
[CrossRef]

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

Pi?zurica, A.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

A. Pi?zurica, and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[CrossRef]

Pien, H.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
[CrossRef] [PubMed]

Pizurica, A.

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
[CrossRef]

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

Plaue, M.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
[CrossRef]

Portilla, J.

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[CrossRef] [PubMed]

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

Scheunders, P.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

Seitz, P.

P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst. I Regul. Pap. 55(8), 2368–2377 (2008).
[CrossRef]

R. Lange, and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[CrossRef]

Shah, J.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
[CrossRef] [PubMed]

Simoncelli, E. P.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

Soatto, S.

S. Soatto, and P. Perona, ““Reducing ”structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 933–942 (1998).
[CrossRef]

Strela, V.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

Tam, W.

L. Zhang, and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast 51, 191–199 (2005).
[CrossRef]

Torralba, A.

A. Torralba, and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[CrossRef]

van der Tempel, W.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
[CrossRef]

Vetterli, M.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[CrossRef]

Wainwright, M. J.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

Yahav, G.

G. J. Iddan, and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE 4298, 48–55 (2001).
[CrossRef]

Yasakethu, S.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
[CrossRef]

Yu, B.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[CrossRef]

Zhang, L.

L. Zhang, and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast 51, 191–199 (2005).
[CrossRef]

Zitnick, C. L.

C. L. Zitnick, and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[CrossRef]

Zlokolica, V.

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
[CrossRef]

Biometrika

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[CrossRef]

CVGIP: Graph. Models Image Process.

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process. 55, 319–323 (1993).
[CrossRef]

IEEE J. Quantum Electron.

R. Lange, and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[CrossRef]

IEEE Sens. J.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J. 7, 317–318 (2007).
[CrossRef]

IEEE Signal Process. Lett.

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett. 13, 337–340 (2006).
[CrossRef]

IEEE Trans. Broadcast

L. Zhang, and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast 51, 191–199 (2005).
[CrossRef]

IEEE Trans. Circ. Syst. Video Tech.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 17, 1690–1699 (2007).
[CrossRef]

A. Amer, and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circ. Syst. Video Tech. 15, 113–118 (2005).
[CrossRef]

IEEE Trans. Circuits Syst. I Regul. Pap.

P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst. I Regul. Pap. 55(8), 2368–2377 (2008).
[CrossRef]

IEEE Trans. Consum. Electron.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron. 55, 1699–1706 (2009).
[CrossRef]

IEEE Trans. Image Process.

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[CrossRef]

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[CrossRef] [PubMed]

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[CrossRef]

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[CrossRef]

A. Pi?zurica, and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[CrossRef]

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243–1251 (1996).
[CrossRef] [PubMed]

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[CrossRef] [PubMed]

A. Benazza-Benyahia, and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[CrossRef] [PubMed]

IEEE Trans. Pattern Anal. Mach. Intell.

S. K. Nayar, and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[CrossRef]

A. Torralba, and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[CrossRef]

S. Soatto, and P. Perona, ““Reducing ”structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 933–942 (1998).
[CrossRef]

Image Vis. Comput.

S. De Backer, A. Pi?zurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vis. Comput. 26, 1038–1051 (2008).
[CrossRef]

Int. J. Comput. Vis.

C. L. Zitnick, and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[CrossRef]

Opt. Eng.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng.,  48, 077003 (2009).
[CrossRef]

Proc. SPIE

G. J. Iddan, and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE 4298, 48–55 (2001).
[CrossRef]

Other

“Swissranger sr4000 overview [online],” http://www.mesa-imaging.ch/prodview4k.php (2009).

I. Daubechies, Ten Lectures on Wavelets (SIAM, Philadelphia, 1992).

R. Bracho, and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

L. Jovanov, A. Pi?zurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).

Lj. Jovanov, N. Petrovi’c, A. Pi?zurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

S. Schulte, B. Huysmans, Pi?zurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

. “3D TV Production [online],” (2009).

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (20)

Fig. 1
Fig. 1

3D camera using the Time-Of-Flight principle.

Fig. 2
Fig. 2

(a) Luminance image of the scene. (b) Depth map of the scene.

Fig. 3
Fig. 3

(a) The membership function LARGE COEFFICIENT denoted as μw for the fuzzy set large coefficient and (b) the membership function LARGE NEIGBOURHOOD denoted as μz for the fuzzy set large variable.

Fig. 6
Fig. 6

(a) Amplitude image D1. (b) The corresponding scatter plot of the measured noise standard deviation σ versus the amplitude A and the fitted noise model (C = 4.11). (c) Another amplitude image D2. (d) The corresponding experimental σA scatter plot with fitted noise models C/A using C estimated from the corresponding data (C = 4.19) and using C estimated from image D1 (C = 4.11).

Fig. 7
Fig. 7

(a) An amplitude image. (b) Noise in the amplitude image from (a);(c). (c) Raster scan of the noise image in (b). (d) Raster scan of the noise in the corresponding depth image.

Fig. 5
Fig. 5

(a) Noise standard deviation estimate using Donoho’s noise estimator. (b) Noise estimation using the proposed approach. (c) Noise in the depth map.

Fig. 16
Fig. 16

(a)Depth image denoised using Donoho’s noise estimation. (b) Depth image de-noised using the proposed noise estimation method.

Fig. 4
Fig. 4

(a) Segments of depth image. (b) Segments of luminance image (both ordered by standard deviation of luminance segments).

Fig. 8
Fig. 8

(a) Horizontal wavelet band of depth image. (b) Detected signal of interest.

Fig. 9
Fig. 9

a) “LARGE COEFFICIENTS” membership function and b) “LARGE ACTIVITY INDICATOR” membership function as generalizations of the corresponding membership functions from Fig. 3.

Fig. 10
Fig. 10

An illustration of the proposed estimator functional dependence on luminance indicator and noisy coefficient value.

Fig. 11
Fig. 11

(a) Shrinkage functions for different values of noise variance. (b) Shrinkage functions for different values of spatial indicator.

Fig. 12
Fig. 12

An illustration of the directional windows. Spatial indicators are formed by summing along the directions di.

Fig. 13
Fig. 13

(a) Horizontal (LH) band of the luminance image from Fig. 2. (b) Spatial indicator obtained by directional filtering and combining depth and luminance images.

Fig. 14
Fig. 14

Denoising result for the “Closet” depth image. (a) Noisy depth image produced by the TOF camera. (b) Image denoised using method from [17]. (c) Image denoised using method from [16]. (d) Image denoised using method from [13]. (e) Image denoised using method from [11]. (f) Image denoised using our extension of GSM vector method from [16]. (g) Image denoised using proposed method. (h) Noise-free reference image.

Fig. 15
Fig. 15

Results for the “Bookshelf” depth image. (a) Noisy depth image produced by the TOF camera (b) Noise-free reference image. (c) Image denoised using method from [17]. (d) Image denoised using proposed method.

Fig. 17
Fig. 17

Denoising results for the “Table” depth image. (a) Noisy depth image produced by the TOF camera. (b) Noise-free reference image. (c) Image denoised using method from [11]. (g) Image denoised using proposed method.

Fig. 19
Fig. 19

(a) Rendering of a scene with noise-free reference depth map. (b) Rendering of the scene using noisy depth map. (c) Rendering of the scene using depth map denoised using the method from [11]. (d) Rendering of a scene with the depth map denoised using proposed method.

Fig. 18
Fig. 18

(a)Depth image denoised non-local method from [12]. (b) Depth image denoised using the proposed method.

Fig. 20
Fig. 20

Virtual views generated using (a) noisy depth image, (b) depth image denoised using method from [11], (c) depth map denoised using depth image using the proposed method, (d) noise-free depth image.

Tables (1)

Tables Icon

Table 1 PSNR values of the denoised depth images

Equations (29)

Equations on this page are rendered with MathJax. Learn more.

d h ; i , j = Σ k i = n 1 2 n 1 2 Σ k j = n 1 2 n 1 2 d i + k i , j + k j f k i , k j h A i + k i , j + k j 2 Σ k i = n 1 2 n 1 2 Σ k j = n 1 2 n 1 2 f k i , k j h . A i + k i , j + k j 2 ,
σ h ; i , j 2 = k i = n 1 2 n 1 2 k j = n 1 2 n 1 2 ( f k i , k j h A i + k i , j + k j 2 ) 2 ( k i = n 1 2 n 1 2 k j = n 1 2 n 1 2 f k i , k j h A i + k i , j + k j 2 ) 2 .
v ( i ) = j W i w ( i , j ) v ( j ) ,
w ( i , j ) = 1 Z i e 1 h k N ξ ik G a ( k 2 ) ( v ( i + k ) v ( j + k ) ) 2 ,
C ( τ ) = s ( t ) g ( t ) = lim T > 1 T T 2 + T 2 s ( t ) g ( t + τ ) dt .
s ( t ) = B + A cos ( 2 π f m t ϕ ) , g ( t ) = cos ( 2 π f m t ) ,
ϕ = arctan A 3 A 1 A 0 A 2 ,
B = A 0 + A 1 + A 2 + A 3 4 Δ t ,
A = δ Δ t sin δ ( A 3 A 1 ) 2 + ( A 0 A 2 ) 2 2 ,
Δ ϕ = i = 1 3 ( ϕ A i ) 2 A i .
Δ L = L 8 B 2 A ,
w k = y k + n k
IF z k is a large activity indicator AND w k is a large coefficient OR z k is a large activity indicator THEN w k is a signal of interest .
y ^ k = γ ( w k , z k ) w k ,
γ ( w k , z k ) = α + μ z ( | z k | ) α μ z ( | z k | ) with α = μ z ( | z k | ) μ w ( | w k | ) .
σ ^ l = C 1 A l ,
σ ^ b l 2 = 1 N k = 1 N ( X l , k m l ) 2 ,
C = 1 P B l = 1 PB A b l σ ^ b l ,
y ^ k D = γ ˜ ( w k , z k ) w k D ,
μ w ˜ ( w k ) = μ w ( || C n 1 2 w k || ) .
μ w ˜ ( w k ) = μ w ( C n 1 2 w k ) and μ z ˜ ( z k ) = μ w ( C n 1 2 z k ) .
C n 1 / 2 w k 2 = w k T C n 1 w k = T 2
μ z ˜ ( z k ) = μ z ( || C w 1 2 z k || ) ,
γ ˜ ( w k , z k ) = α + μ z ˜ ( | z k | ) α μ z ˜ ( | z k | ) with α = μ z ( | z k | ) μ w ˜ ( | w k | ) ,
[ σ ^ D σ ^ L ] = [ σ ^ l med ( | HH 1 L | ) 0.6745 ] ,
σ ^ DL = σ ^ LD = ( med ( | HH s D 1 + HH s L 1 | ) med ( | HH s D 1 HH s L 1 | ) ) 2 .
ρ LD = σ ^ D σ ^ L σ ^ LD .
C n = [ σ D ρ LD ρ LD σ L ] .
C y = [ h h s 1 D h h s 1 L ] T × [ h h s 1 D h h s 1 L ] ,

Metrics