Abstract

We present a method to enhance depth quality of a time-of-flight (ToF) camera without additional devices or hardware modifications. By controlling the turn-off patterns of the LEDs of the camera, we obtain depth and normal maps simultaneously. Sixteen subphase images are acquired with variations in gate-pulse timing and light emission pattern of the camera. The subphase images allow us to obtain a normal map, which are combined with depth maps for improved depth details. These details typically cannot be captured by conventional ToF cameras. By the proposed method, the average of absolute differences between the measured and laser-scanned depth maps has decreased from 4.57 to 3.77 mm.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
    [CrossRef]
  2. S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
    [CrossRef]
  3. M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.
  4. D. Falie and V. Buzuloiu, in Proceedings of International Symposium on Signals, Circuits and Systems (IEEE, 2007), pp. 229–232.
  5. D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
    [CrossRef]
  6. M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
    [CrossRef]
  7. Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.
  8. S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.
  9. R. J. Woodham, Opt. Eng. 19, 191139 (1980).
    [CrossRef]
  10. R. T. Frankot and R. Chellappa, IEEE Trans. Pattern Anal. Mach. Intell. 10, 439 (1988).
    [CrossRef]
  11. R. C. Gonzalez and R. E. Woods, Digital Image Processing (McGraw-Hill, 2002).

2011

S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
[CrossRef]

S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
[CrossRef]

2009

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

2005

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

1988

R. T. Frankot and R. Chellappa, IEEE Trans. Pattern Anal. Mach. Intell. 10, 439 (1988).
[CrossRef]

1980

R. J. Woodham, Opt. Eng. 19, 191139 (1980).
[CrossRef]

Alenya, G.

S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
[CrossRef]

Buzuloiu, V.

D. Falie and V. Buzuloiu, in Proceedings of International Symposium on Signals, Circuits and Systems (IEEE, 2007), pp. 229–232.

Chellappa, R.

R. T. Frankot and R. Chellappa, IEEE Trans. Pattern Anal. Mach. Intell. 10, 439 (1988).
[CrossRef]

Davis, J.

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

Driewer, F.

M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.

Edeler, T.

S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
[CrossRef]

Falie, D.

D. Falie and V. Buzuloiu, in Proceedings of International Symposium on Signals, Circuits and Systems (IEEE, 2007), pp. 229–232.

Foix, S.

S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
[CrossRef]

Frank, M.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Frankot, R. T.

R. T. Frankot and R. Chellappa, IEEE Trans. Pattern Anal. Mach. Intell. 10, 439 (1988).
[CrossRef]

Gonzalez, R. C.

R. C. Gonzalez and R. E. Woods, Digital Image Processing (McGraw-Hill, 2002).

Hamprecht, F. A.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Hermanski, A.

S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
[CrossRef]

Hussmann, S.

S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
[CrossRef]

Jähne, B.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Kang, B.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Kim, C.-Y.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Kim, J.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Kim, K.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Kim, S.-J.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Köthe, U.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Lee, K.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

Matsushita, Y.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

Nehab, D.

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

Plaue, M.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Ramamoorthi, R.

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

Rapp, H.

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Rusinkiewicz, S.

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

Sauer, M.

M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.

Schilling, K.

M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.

Torras, C.

S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
[CrossRef]

Wiedemann, M.

M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.

Wilburn, B.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

Woodham, R. J.

R. J. Woodham, Opt. Eng. 19, 191139 (1980).
[CrossRef]

Woods, R. E.

R. C. Gonzalez and R. E. Woods, Digital Image Processing (McGraw-Hill, 2002).

Yang, R.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

Ye, M.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

Yu, H.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

Zhang, Q.

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

ACM Trans. Graph.

D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, ACM Trans. Graph. 24, 536 (2005).
[CrossRef]

IEEE Sens. J.

S. Foix, G. Alenya, and C. Torras, IEEE Sens. J. 11, 1917 (2011).
[CrossRef]

IEEE Trans. Instrum. Meas.

S. Hussmann, A. Hermanski, and T. Edeler, IEEE Trans. Instrum. Meas. 60, 1682 (2011).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell.

R. T. Frankot and R. Chellappa, IEEE Trans. Pattern Anal. Mach. Intell. 10, 439 (1988).
[CrossRef]

Opt. Eng.

R. J. Woodham, Opt. Eng. 19, 191139 (1980).
[CrossRef]

M. Frank, M. Plaue, H. Rapp, U. Köthe, B. Jähne, and F. A. Hamprecht, Opt. Eng. 48, 013602 (2009).
[CrossRef]

Other

Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 2472–2479.

S.-J. Kim, B. Kang, J. Kim, K. Lee, C.-Y. Kim, and K. Kim, in Proceedings of IEEE Conference on Solid-State Circuits Conference Digest of Technical Papers (IEEE, 2012), pp. 396–398.

M. Wiedemann, M. Sauer, F. Driewer, and K. Schilling, in Proceedings of the 17th IFAC World Congress (IFAC, 2008), pp. 689–694.

D. Falie and V. Buzuloiu, in Proceedings of International Symposium on Signals, Circuits and Systems (IEEE, 2007), pp. 229–232.

R. C. Gonzalez and R. E. Woods, Digital Image Processing (McGraw-Hill, 2002).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (7)

Fig. 1.
Fig. 1.

Example of the loss in depth details. (a) Intensity image and (b) depth map captured by (c) ToF camera. (d) Normalized depth map in the area inside red dotted lines in (b). Depth details on the relivo (e.g., dragon in this figure) cannot be observed in the depth map.

Fig. 2.
Fig. 2.

Schematic diagrams of (a) a previous work and (b) the proposed one to acquire depth and normal maps.

Fig. 3.
Fig. 3.

Phase and intensity image acquisitions. Sixteen subphase images are captured with variations in gate pulse timing and light emission pattern. Each four-phase images and four-intensity images are produced from the subphase images, respectively.

Fig. 4.
Fig. 4.

Offset removal. (a) Depth map (DPS) produced by PS. (b) Offset (DOF) estimated as a quadratic polynomial surface. (c) Offset-removed depth map (DDS).

Fig. 5.
Fig. 5.

(a) Depth maps produced by ToF (left; DToF) and PS (right; DDS). (b) Fourier spectrums (FToF, FDS) from (a). (d) A Fourier spectrum (left; FNew) and a depth map (right; DNew) combined from (b) with the weight maps (c).

Fig. 6.
Fig. 6.

Depth maps produced by (a)–(c) a laser scanner, (d)–(f) the conventional ToF, and (g)–(i) proposed method. Depth maps in the second and third rows were produced by using the same ToF camera. Conventional ToF depth maps are de-noised.

Fig. 7.
Fig. 7.

Depth maps and normal maps with respect to various angular extents of light sources. The angular extent is changed by moving the location of the captured object. The numbers from first to forth row are the distance, angular extent, integration time, and the image size, respectively. Depth maps [(a)–(e); conventional, (k)–(o); proposed] and normal maps [(f)–(j)] are produced by using the same ToF camera. All the images are normalized to enhance their contrast for visualization.

Tables (1)

Tables Icon

Table 1. Performance Comparison of a Conventional ToF and the Proposed Methoda

Equations (10)

Equations on this page are rendered with MathJax. Learn more.

Qϕ(x)=X{A,B,C,D}Qϕ,X(x)
IX(x)=ϕ{0°,90°,180°,270°}Qϕ,X(x),
φ(x)=arctan(Q270°(x)Q90°(x)Q0°(x)Q180°(x))
DToF(x)=c2fmφ(x)2π,
N(x)=I(x)·pinv(L)
I(x)=[IA(x)IB(x)IC(x)ID(x)]
L=[010110100000],
DDS(x)=DPS(x)DOF(x).
FNew(x)=W(x)·FToF(x)+(1W(x))·FDS(x)
W(x)={1if dist(x,xc)<Dth0otherwise,

Metrics