Abstract

In this paper we present a new denoising method for the depth images of a 3D imaging sensor, based on the time-of-flight principle. We propose novel ways to use luminance-like information produced by a time-of flight camera along with depth images. Firstly, we propose a wavelet-based method for estimating the noise level in depth images, using luminance information. The underlying idea is that luminance carries information about the power of the optical signal reflected from the scene and is hence related to the signal-to-noise ratio for every pixel within the depth image. In this way, we can efficiently solve the difficult problem of estimating the non-stationary noise within the depth images. Secondly, we use luminance information to better restore object boundaries masked with noise in the depth images. Information from luminance images is introduced into the estimation formula through the use of fuzzy membership functions. In particular, we take the correlation between the measured depth and luminance into account, and the fact that edges (object boundaries) present in the depth image are likely to occur in the luminance image as well. The results on real 3D images show a significant improvement over the state-of-the-art in the field.

© 2010 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. C. L. Zitnick and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
    [Crossref]
  2. W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
    [Crossref] [PubMed]
  3. S. K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
    [Crossref]
  4. A. Torralba and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
    [Crossref]
  5. S. Soatto and P. Perona, “Reducing “structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell.20, 933–942 (1998).
    [Crossref]
  6. R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron.37, 390 –397 (2001).
    [Crossref]
  7. R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J.7, 317–318 (2007).
    [Crossref]
  8. J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243 –1251 (1996).
    [Crossref] [PubMed]
  9. S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.
  10. S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).
  11. M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
    [Crossref]
  12. T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).
  13. L. Jovanov, A. Pižurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).
  14. Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).
  15. S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).
  16. S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
    [Crossref]
  17. A. Benazza-Benyahia and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
    [Crossref] [PubMed]
  18. , “3D TV Production [online],” (2009).
  19. P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst., I: Regul. Pap.55(8), 2368–2377 (2008).
    [Crossref]
  20. I. Daubechies, Ten Lectures on Wavelets (SIAM, Philadelphia, 1992).
  21. J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
    [Crossref]
  22. S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
    [Crossref]
  23. A. Pižurica and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
    [Crossref]
  24. S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process.  55, 319–323 (1993).
    [Crossref]
  25. M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 17, 1690–1699 (2007).
    [Crossref]
  26. A. Amer and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 15, 113–118 (2005).
    [Crossref]
  27. V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett.13, 337 – 340 (2006).
    [Crossref]
  28. R. Bracho and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).
  29. D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
    [Crossref]
  30. A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
    [Crossref]
  31. “Swissranger sr4000 overview [online],” http://www.mesa-imaging.ch/prodview4k.php (2009).
  32. J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
    [Crossref] [PubMed]
  33. G. J. Iddan and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE4298, 48–55 (2001).
    [Crossref]
  34. D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron.55, 1699–1706 (2009).
    [Crossref]
  35. L. Zhang and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast. 51, 191–199 (2005).
    [Crossref]

2009 (1)

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
[Crossref]

2008 (1)

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

2007 (1)

C. L. Zitnick and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[Crossref]

1993 (2)

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process.  55, 319–323 (1993).
[Crossref]

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[Crossref]

Acheroy, M.

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[Crossref]

Amer, A.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 17, 1690–1699 (2007).
[Crossref]

A. Amer and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 15, 113–118 (2005).
[Crossref]

Bamji, C.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.

Benazza-Benyahia, A.

A. Benazza-Benyahia and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[Crossref] [PubMed]

Bracho, R.

R. Bracho and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).

Chang, S.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[Crossref]

Daubechies, I.

I. Daubechies, Ten Lectures on Wavelets (SIAM, Philadelphia, 1992).

Davis, J.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

De Backer, S.

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

De Silva, D.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron.55, 1699–1706 (2009).
[Crossref]

Donoho, D.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[Crossref]

Dubois, E.

A. Amer and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 15, 113–118 (2005).
[Crossref]

Fernando, W.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron.55, 1699–1706 (2009).
[Crossref]

Frank, M.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
[Crossref]

Gauch, J.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243 –1251 (1996).
[Crossref] [PubMed]

Ghazal, M.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 17, 1690–1699 (2007).
[Crossref]

Ghrayeb, A.

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 17, 1690–1699 (2007).
[Crossref]

Gokturk, S. B.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.

Guerrero-Colon, J. A.

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[Crossref] [PubMed]

Hamprecht, F. A.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
[Crossref]

Huhle, B.

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

Huysmans, B.

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

Iddan, G. J.

G. J. Iddan and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE4298, 48–55 (2001).
[Crossref]

Jenke, P.

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

Johnstone, I.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[Crossref]

Johnstone, I. M.

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[Crossref]

Jovanov, L.

L. Jovanov, A. Pižurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).

Jovanov, Lj.

Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

Kang, S. B.

C. L. Zitnick and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[Crossref]

Kerre, E.

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

Kuijk, M.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J.7, 317–318 (2007).
[Crossref]

Lange, R.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron.37, 390 –397 (2001).
[Crossref]

Lemahieu, I.

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[Crossref]

Mancera, L.

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[Crossref] [PubMed]

Miled, W.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[Crossref] [PubMed]

Nakagawa, Y.

S. K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[Crossref]

Nayar, S. K.

S. K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[Crossref]

Nieuwenhove, R. G. J. S. D. V.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J.7, 317–318 (2007).
[Crossref]

Oliva, A.

A. Torralba and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[Crossref]

Olsen, S. I.

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process.  55, 319–323 (1993).
[Crossref]

Parent, M.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[Crossref] [PubMed]

Perona, P.

S. Soatto and P. Perona, “Reducing “structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell.20, 933–942 (1998).
[Crossref]

Pesquet, J.

A. Benazza-Benyahia and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[Crossref] [PubMed]

Pesquet, J.-C.

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[Crossref] [PubMed]

Petrovic, N.

Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

Philips, W.

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

A. Pižurica and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[Crossref]

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett.13, 337 – 340 (2006).
[Crossref]

L. Jovanov, A. Pižurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[Crossref]

Pien, H.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243 –1251 (1996).
[Crossref] [PubMed]

Pizurica, A.

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett.13, 337 – 340 (2006).
[Crossref]

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[Crossref]

Pižurica,

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

Pižurica, A.

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

A. Pižurica and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[Crossref]

L. Jovanov, A. Pižurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).

Plaue, M.

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
[Crossref]

Portilla, J.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[Crossref]

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[Crossref] [PubMed]

Sanderson, A.

R. Bracho and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).

Schairer, T.

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

Scheunders, P.

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

Schulte, S.

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

Schuon, S.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

Seitz, P.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron.37, 390 –397 (2001).
[Crossref]

P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst., I: Regul. Pap.55(8), 2368–2377 (2008).
[Crossref]

Shah, J.

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243 –1251 (1996).
[Crossref] [PubMed]

Simoncelli, E. P.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[Crossref]

Soatto, S.

S. Soatto and P. Perona, “Reducing “structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell.20, 933–942 (1998).
[Crossref]

Straßer, W.

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

Strela, V.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[Crossref]

Tam, W.

L. Zhang and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast. 51, 191–199 (2005).
[Crossref]

Theobalt, C.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

Thrun, S.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

Torralba, A.

A. Torralba and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[Crossref]

van der Tempel, W.

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J.7, 317–318 (2007).
[Crossref]

Vetterli, M.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[Crossref]

Wainwright, M. J.

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[Crossref]

Yahav, G.

G. J. Iddan and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE4298, 48–55 (2001).
[Crossref]

Yalcin, H.

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.

Yasakethu, S.

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron.55, 1699–1706 (2009).
[Crossref]

Yu, B.

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[Crossref]

Zhang, L.

L. Zhang and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast. 51, 191–199 (2005).
[Crossref]

Zitnick, C. L.

C. L. Zitnick and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[Crossref]

Zlokolica, V.

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett.13, 337 – 340 (2006).
[Crossref]

Biometrika (1)

D. Donoho, I. Johnstone, and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika 81, 425–455 (1993).
[Crossref]

CVGIP: Graph. Models Image Process (1)

S. I. Olsen, “Estimation of noise in images: an evaluation,” CVGIP: Graph. Models Image Process.  55, 319–323 (1993).
[Crossref]

Image Vision Comput. (1)

S. De Backer, A. Pižurica, B. Huysmans, W. Philips, and P. Scheunders, “Denoising of multicomponent images using wavelet least-squares estimators,” Image Vision Comput. 26, 1038–1051 (2008).
[Crossref]

Int. J. Comput. Vis. (1)

C. L. Zitnick and S. B. Kang, “Stereo for image-based rendering using image over-segmentation,” Int. J. Comput. Vis. 75, 49–65 (2007).
[Crossref]

Opt. Eng. (1)

M. Frank, M. Plaue, and F. A. Hamprecht, “Denoising of continuous-wave time-of-flight depth images using confidence measures,” Opt. Eng. 48 (2009).
[Crossref]

Other (30)

T. Schairer, B. Huhle, P. Jenke, and W. Straßer, “Parallel non-local denoising of depth maps,” in International Workshop on Local and Non-Local Approximation in Image Processing (EUSIPCO Satellite Event) (2008).

L. Jovanov, A. Pižurica, and W. Philips, “Wavelet based joint denoising of depth and luminance images,” in 3D TV Conference, Kos Island, Greece (2007).

Lj. Jovanov, N. Petrović, A. Pižurica, and W. Philips, “Content adaptive wavelet based method for joint denoising of depth and luminance images,” in SPIE Wavelet Applications in Industrial Processing V, (Boston, Massuchusetts, USA, 2007).

S. Schulte, B. Huysmans, Pižurica, E. Kerre, and W. Philips, “A new fuzzy-based wavelet shrinkage image denoising technique,” in Advanced Concepts for Intelligent Vision Systems (Acivs 2006), (Antwerp, Belgium, 2006).

M. Ghazal, A. Amer, and A. Ghrayeb, “A real-time technique for spatiotemporal video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 17, 1690–1699 (2007).
[Crossref]

A. Amer and E. Dubois, “Fast and reliable structure-oriented video noise estimation,” IEEE Trans. Circuits Syst. Video Technol. 15, 113–118 (2005).
[Crossref]

V. Zlokolica, A. Pizurica, and W. Philips, “Noise estimation for video processing based on spatio-temporal gradients,” IEEE Signal Process. Lett.13, 337 – 340 (2006).
[Crossref]

R. Bracho and A. Sanderson, “Segmentation of images based on intensity gradient information,” in Proc. IEEE Computer Soc. Conf. on Computer Vision, 341–347(1985).

A. Pizurica, W. Philips, I. Lemahieu, and M. Acheroy, “A joint inter- and intrascale statistical model for bayesian wavelet based image denoising,” IEEE Trans. Image Process. 11, 545–557 (2002).
[Crossref]

“Swissranger sr4000 overview [online],” http://www.mesa-imaging.ch/prodview4k.php (2009).

J. A. Guerrero-Colon, L. Mancera, and J. Portilla, “Image restoration using space-variant gaussian scale mixtures in overcomplete pyramids,” IEEE Trans. Image Process. 17, 27–41 (2008).
[Crossref] [PubMed]

G. J. Iddan and G. Yahav, “G.: 3d imaging in the studio (and elsewhere,” Proc. SPIE4298, 48–55 (2001).
[Crossref]

D. De Silva, W. Fernando, and S. Yasakethu, “Object based coding of the depth maps for 3d video coding,” IEEE Trans. Consum. Electron.55, 1699–1706 (2009).
[Crossref]

L. Zhang and W. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcast. 51, 191–199 (2005).
[Crossref]

W. Miled, J.-C. Pesquet, and M. Parent, “A convex optimization approach for depth estimation under illumination variation,” IEEE Trans. Image Process. 18, 813–830 (2009).
[Crossref] [PubMed]

S. K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994).
[Crossref]

A. Torralba and A. Oliva, “Depth estimation from image structure,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 1226–1238 (2002).
[Crossref]

S. Soatto and P. Perona, “Reducing “structure from motion”: A general framework for dynamic vision part 1: Modeling,” IEEE Trans. Pattern Anal. Mach. Intell.20, 933–942 (1998).
[Crossref]

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron.37, 390 –397 (2001).
[Crossref]

R. G. J. S. D. V. Nieuwenhove, W. van der Tempel, and M. Kuijk, “Photonic demodulator with sensitivity control,” IEEE Sens. J.7, 317–318 (2007).
[Crossref]

J. Shah, H. Pien, and J. Gauch, “Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework,” IEEE Trans. Image Process. 5, 1243 –1251 (1996).
[Crossref] [PubMed]

S. B. Gokturk, H. Yalcin, and C. Bamji, “A time-of-flight depth sensor - system description, issues and solutions,” in CVPRW ’04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3, (IEEE Computer Society, 2004), p. 35.

S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “High-quality scanning using time-of-flight depth superresolution,” CVPR Workshop on Time-of-Flight Computer Vision (2008).

A. Benazza-Benyahia and J. Pesquet, “Building robust wavelet estimators for multicomponent images using Stein’s principle,” IEEE Trans. Image Process. 14, 1814–1830 (2005).
[Crossref] [PubMed]

, “3D TV Production [online],” (2009).

P. Seitz, “Quantum-noise limited distance resolution of optical range imaging techniques,” IEEE Trans. Circuits Syst., I: Regul. Pap.55(8), 2368–2377 (2008).
[Crossref]

I. Daubechies, Ten Lectures on Wavelets (SIAM, Philadelphia, 1992).

J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using Gaussian scale mixtures in the wavelet domain,” IEEE Trans. Image Process. 12, 1338–1351 (2003).
[Crossref]

S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
[Crossref]

A. Pižurica and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Trans. Image Process. 15, 654–665 (2006).
[Crossref]

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (20)

Fig. 1
Fig. 1

3D camera using the Time-Of-Flight principle.

Fig. 2
Fig. 2

(a) Luminance image of the scene. (b) Depth map of the scene.

Fig. 3
Fig. 3

(a) The membership function LARGE COEFFICIENT denoted as μw for the fuzzy set large coefficient and (b) the membership function LARGE NEIGBOURHOOD denoted as μz for the fuzzy set large variable.

Fig. 6
Fig. 6

(a) Amplitude image D1. (b) The corresponding scatter plot of the measured noise standard deviation σ versus the amplitude A and the fitted noise model (C = 4.11). (c) Another amplitude image D2. (d) The corresponding experimental σA scatter plot with fitted noise models C/A using C estimated from the corresponding data (C = 4.19) and using C estimated from image D1 (C = 4.11).

Fig. 7
Fig. 7

(a) An amplitude image. (b) Noise in the amplitude image from (a);(c). (c) Raster scan of the noise image in (b). (d) Raster scan of the noise in the corresponding depth image.

Fig. 5
Fig. 5

(a) Noise standard deviation estimate using Donoho’s noise estimator. (b) Noise estimation using the proposed approach. (c) Noise in the depth map.

Fig. 16
Fig. 16

(a)Depth image denoised using Donoho’s noise estimation. (b) Depth image de-noised using the proposed noise estimation method.

Fig. 4
Fig. 4

(a) Segments of depth image. (b) Segments of luminance image (both ordered by standard deviation of luminance segments).

Fig. 8
Fig. 8

(a) Horizontal wavelet band of depth image. (b) Detected signal of interest.

Fig. 9
Fig. 9

a) “LARGE COEFFICIENTS” membership function and b) “LARGE ACTIVITY INDICATOR” membership function as generalizations of the corresponding membership functions from Fig. 3.

Fig. 10
Fig. 10

An illustration of the proposed estimator functional dependence on luminance indicator and noisy coefficient value.

Fig. 11
Fig. 11

(a) Shrinkage functions for different values of noise variance. (b) Shrinkage functions for different values of spatial indicator.

Fig. 12
Fig. 12

An illustration of the directional windows. Spatial indicators are formed by summing along the directions di.

Fig. 13
Fig. 13

(a) Horizontal (LH) band of the luminance image from Fig. 2. (b) Spatial indicator obtained by directional filtering and combining depth and luminance images.

Fig. 14
Fig. 14

Denoising result for the “Closet” depth image. (a) Noisy depth image produced by the TOF camera. (b) Image denoised using method from [17]. (c) Image denoised using method from [16]. (d) Image denoised using method from [13]. (e) Image denoised using method from [11]. (f) Image denoised using our extension of GSM vector method from [16]. (g) Image denoised using proposed method. (h) Noise-free reference image.

Fig. 15
Fig. 15

Results for the “Bookshelf” depth image. (a) Noisy depth image produced by the TOF camera (b) Noise-free reference image. (c) Image denoised using method from [17]. (d) Image denoised using proposed method.

Fig. 17
Fig. 17

Denoising results for the “Table” depth image. (a) Noisy depth image produced by the TOF camera. (b) Noise-free reference image. (c) Image denoised using method from [11]. (g) Image denoised using proposed method.

Fig. 19
Fig. 19

(a) Rendering of a scene with noise-free reference depth map. (b) Rendering of the scene using noisy depth map. (c) Rendering of the scene using depth map denoised using the method from [11]. (d) Rendering of a scene with the depth map denoised using proposed method.

Fig. 18
Fig. 18

(a)Depth image denoised non-local method from [12]. (b) Depth image denoised using the proposed method.

Fig. 20
Fig. 20

Virtual views generated using (a) noisy depth image, (b) depth image denoised using method from [11], (c) depth map denoised using depth image using the proposed method, (d) noise-free depth image.

Tables (1)

Tables Icon

Table 1 PSNR values of the denoised depth images

Equations (29)

Equations on this page are rendered with MathJax. Learn more.

d h ; i , j = Σ k i = n 1 2 n 1 2 Σ k j = n 1 2 n 1 2 d i + k i , j + k j f k i , k j h A i + k i , j + k j 2 Σ k i = n 1 2 n 1 2 Σ k j = n 1 2 n 1 2 f k i , k j h . A i + k i , j + k j 2 ,
σ h ; i , j 2 = k i = n 1 2 n 1 2 k j = n 1 2 n 1 2 ( f k i , k j h A i + k i , j + k j 2 ) 2 ( k i = n 1 2 n 1 2 k j = n 1 2 n 1 2 f k i , k j h A i + k i , j + k j 2 ) 2 .
v ( i ) = j W i w ( i , j ) v ( j ) ,
w ( i , j ) = 1 Z i e 1 h k N ξ ik G a ( k 2 ) ( v ( i + k ) v ( j + k ) ) 2 ,
C ( τ ) = s ( t ) g ( t ) = lim T > 1 T T 2 + T 2 s ( t ) g ( t + τ ) dt .
s ( t ) = B + A cos ( 2 π f m t ϕ ) , g ( t ) = cos ( 2 π f m t ) ,
ϕ = arctan A 3 A 1 A 0 A 2 ,
B = A 0 + A 1 + A 2 + A 3 4 Δ t ,
A = δ Δ t sin δ ( A 3 A 1 ) 2 + ( A 0 A 2 ) 2 2 ,
Δ ϕ = i = 1 3 ( ϕ A i ) 2 A i .
Δ L = L 8 B 2 A ,
w k = y k + n k
IF z k is a large activity indicator AND w k is a large coefficient OR z k is a large activity indicator THEN w k is a signal of interest .
y ^ k = γ ( w k , z k ) w k ,
γ ( w k , z k ) = α + μ z ( | z k | ) α μ z ( | z k | ) with α = μ z ( | z k | ) μ w ( | w k | ) .
σ ^ l = C 1 A l ,
σ ^ b l 2 = 1 N k = 1 N ( X l , k m l ) 2 ,
C = 1 P B l = 1 PB A b l σ ^ b l ,
y ^ k D = γ ˜ ( w k , z k ) w k D ,
μ w ˜ ( w k ) = μ w ( || C n 1 2 w k || ) .
μ w ˜ ( w k ) = μ w ( C n 1 2 w k ) and μ z ˜ ( z k ) = μ w ( C n 1 2 z k ) .
C n 1 / 2 w k 2 = w k T C n 1 w k = T 2
μ z ˜ ( z k ) = μ z ( || C w 1 2 z k || ) ,
γ ˜ ( w k , z k ) = α + μ z ˜ ( | z k | ) α μ z ˜ ( | z k | ) with α = μ z ( | z k | ) μ w ˜ ( | w k | ) ,
[ σ ^ D σ ^ L ] = [ σ ^ l med ( | HH 1 L | ) 0.6745 ] ,
σ ^ DL = σ ^ LD = ( med ( | HH s D 1 + HH s L 1 | ) med ( | HH s D 1 HH s L 1 | ) ) 2 .
ρ LD = σ ^ D σ ^ L σ ^ LD .
C n = [ σ D ρ LD ρ LD σ L ] .
C y = [ h h s 1 D h h s 1 L ] T × [ h h s 1 D h h s 1 L ] ,

Metrics