Abstract

Three-dimensional imaging using Time-of-flight (ToF) sensors is rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, the current generation of ToF cameras suffers from low spatial resolution due to physical fabrication limitations. In this paper, we propose CS-ToF, an imaging architecture to achieve high spatial resolution ToF imaging via optical multiplexing and compressive sensing. Our approach is based on the observation that, while depth is non-linearly related to ToF pixel measurements, a phasor representation of captured images results in a linear image formation model. We utilize this property to develop a CS-based technique that is used to recover high resolution 3D images. Based on the proposed architecture, we developed a prototype 1-megapixel compressive ToF camera that achieves as much as 4× improvement in spatial resolution and 3× improvement for natural scenes. We believe that our proposed CS-ToF architecture provides a simple and low-cost solution to improve the spatial resolution of ToF and related sensors.

© 2017 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Exploiting sparsity in time-of-flight range acquisition using a single time-resolved sensor

Ahmed Kirmani, Andrea Colaço, Franco N. C. Wong, and Vivek K. Goyal
Opt. Express 19(22) 21485-21507 (2011)

Three-dimensional single-pixel compressive reflectivity imaging based on complementary modulation

Wen-Kai Yu, Xu-Ri Yao, Xue-Feng Liu, Long-Zhen Li, and Guang-Jie Zhai
Appl. Opt. 54(3) 363-367 (2015)

Supervised preserving projection for learning scene information based on time-of-flight imaging sensor

Yi Jiang, Yong Liu, Yunqi Lei, and Qicong Wang
Appl. Opt. 52(21) 5279-5288 (2013)

References

  • View by:
  • |
  • |
  • |

  1. J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.
  2. K. Gallo and G. Assanto, “Vision based obstacle detection for wheeled robots,” in International Conference on Control, Automation and Systems (2008), pp. 1587–1592.
  3. G. A. Howland, D. J. Lum, M. R. Ware, and J. C. Howell, “Photon counting compressive depth mapping,” Opt. Express 21(20), 23822–23837 (2013).
    [Crossref] [PubMed]
  4. L. R. Bissonnette, Lidar: Range-Resolved Optical Remote Sensing of the Atmosphere (Springer, 2005).
  5. C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo, “Time-of-flight cameras and microsoft kinect,” (Springer, 2012).
    [Crossref]
  6. S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: a survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
    [Crossref]
  7. R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001).
    [Crossref]
  8. Microsoft Kinect Sensor (The second verison), https://developer.microsoft.com/en-us/windows/kinect , (2016).
  9. TI ToF sensor, https://www.ti.com/sensing-products/optical-sensors/3d-time-of-flight/overview.html , (2016).
  10. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003).
  11. C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.
  12. F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
    [Crossref]
  13. A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
    [Crossref]
  14. F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
    [Crossref]
  15. F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
    [Crossref] [PubMed]
  16. K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.
  17. S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.
  18. S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.
  19. R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).
  20. G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
    [Crossref] [PubMed]
  21. C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.
  22. A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.
  23. L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.
  24. J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.
  25. J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
    [Crossref]
  26. X. Song, Y. Dai, and X. Qin, “Deep Depth Super-Resolution: Learning Depth Super-Resolution using Deep Convolutional Neural Network,” arXiv, (2016).
  27. D. W. Davies, “Spatially multiplexed infrared camera,” J. Opt. Soc. Am. 65(6), 707–711 (1975).
    [Crossref]
  28. R. N. Ibbett, D. Aspinall, and J. F. Grainger, “Real-time multiplexing of dispersed spectra in any wavelength region,” Appl. Opt. 7(6), 1089–1094 (1968).
    [Crossref] [PubMed]
  29. N. J. A. Sloane, T. Fine, P. G. Phillips, and M. Harwit, “Codes for multiplex spectrometry,” Appl. Opt. 8(10), 2103–2106 (1969).
    [Crossref] [PubMed]
  30. J. A. Decker and M. Harwit, “Experimental operation of a Hadamard spectrometer,” Appl. Opt. 8, 2552 (1969).
    [Crossref] [PubMed]
  31. E. D. Nelson and M. L. Fredman, “Hadamard spectroscopy,” J. Opt. Soc. Am. 60(12), 1664–1669 (1970).
    [Crossref]
  32. Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
    [Crossref] [PubMed]
  33. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
    [Crossref]
  34. G. A. Howland, P. B. Dixon, and J. C. Howell, “Photon-counting compressive sensing laser radar for 3D imaging,” Appl. Opt. 50(31), 5917–5920 (2011).
    [Crossref] [PubMed]
  35. A. Kirmani, A. Colaço, F. N. C. Wong, and V. K. Goyal, “Exploiting sparsity in time-of-flight range acquisition using a single time-resolved sensor,” Opt. Express 19(22), 21485–21507 (2011).
    [Crossref] [PubMed]
  36. A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).
  37. A. C. Sankaranarayanan, C. Studer, and R. G. Baraniu, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. ICCP (IEEE, 2012).
  38. J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens: A scalable architecture for video compressive sensing,” in Proc. ICCP (IEEE, 2015).
  39. H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.
  40. M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
    [Crossref]
  41. M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).
  42. J. M. Bioucas-Dias and M. A. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
    [Crossref] [PubMed]
  43. Y. Endo, T. Shimobaba, T. Kakue, and T. Ito, “GPU-accelerated compressive holography,” Opt. Express 24(8), 8437–8445 (2016).
    [Crossref] [PubMed]
  44. S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
    [Crossref]
  45. D. Mittleman, “Sensing with terahertz radiation,” Springer85, (2013).
  46. J. H. Ender, “On compressive sensing applied to radar,” Signal Processing 90(5), 1402–1414 (2010).
    [Crossref]
  47. D. Scharstein and R. Szeliski, “High-accuracy stereo depth maps using structured light,” in Proc. CVPR (IEEE, 2003), pp. I-I.

2016 (2)

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
[Crossref]

Y. Endo, T. Shimobaba, T. Kakue, and T. Ito, “GPU-accelerated compressive holography,” Opt. Express 24(8), 8437–8445 (2016).
[Crossref] [PubMed]

2015 (3)

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
[Crossref] [PubMed]

2014 (2)

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

2013 (3)

G. A. Howland, D. J. Lum, M. R. Ware, and J. C. Howell, “Photon counting compressive depth mapping,” Opt. Express 21(20), 23822–23837 (2013).
[Crossref] [PubMed]

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

2012 (1)

S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
[Crossref]

2011 (3)

2010 (1)

J. H. Ender, “On compressive sensing applied to radar,” Signal Processing 90(5), 1402–1414 (2010).
[Crossref]

2008 (1)

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

2007 (2)

Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
[Crossref] [PubMed]

J. M. Bioucas-Dias and M. A. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

2001 (1)

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001).
[Crossref]

1975 (1)

1970 (1)

1969 (2)

1968 (1)

Alenya, G.

S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: a survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Askeland, J.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Aspinall, D.

Assanto, G.

K. Gallo and G. Assanto, “Vision based obstacle detection for wheeled robots,” in International Conference on Control, Automation and Systems (2008), pp. 1587–1592.

Baraniu, R. G.

A. C. Sankaranarayanan, C. Studer, and R. G. Baraniu, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. ICCP (IEEE, 2012).

Baraniuk, R.

A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).

Baraniuk, R. G.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Barsi, C.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

Becker, J.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Belhumeu, P. N.

Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
[Crossref] [PubMed]

Bhandari, A.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

Bioucas-Dias, J. M.

J. M. Bioucas-Dias and M. A. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

Bissonnette, L. R.

L. R. Bissonnette, Lidar: Range-Resolved Optical Remote Sensing of the Atmosphere (Springer, 2005).

Cha, S.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Chellappa, R.

A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).

Chen, H.

H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.

Chou, C.-C.

J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.

Colaço, A.

Cortelazzo, G. M.

C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo, “Time-of-flight cameras and microsoft kinect,” (Springer, 2012).
[Crossref]

Cossairt, O.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Dai, Y.

X. Song, Y. Dai, and X. Qin, “Deep Depth Super-Resolution: Learning Depth Super-Resolution using Deep Convolutional Neural Network,” arXiv, (2016).

Davenport, M. A.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Davies, D. W.

Davis, J.

S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.

C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.

Decker, J. A.

Dixon, P. B.

Dolson, J.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Dorrington, A.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

Duarte, M. F.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Eisemann, M.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Ender, J. H.

J. H. Ender, “On compressive sensing applied to radar,” Signal Processing 90(5), 1402–1414 (2010).
[Crossref]

Endo, Y.

Evangelidis, G. D.

G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
[Crossref] [PubMed]

Feris, R.

J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.

Feris, R. S.

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
[Crossref]

Figueiredo, M. A.

J. M. Bioucas-Dias and M. A. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

Fine, T.

Foix, S.

S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: a survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Fredman, M. L.

Gallo, K.

K. Gallo and G. Assanto, “Vision based obstacle detection for wheeled robots,” in International Conference on Control, Automation and Systems (2008), pp. 1587–1592.

Garbe, C. S.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Gong, J.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Goyal, V. K.

Grainger, J. F.

Gregson, J.

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

Gupta, M.

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens: A scalable architecture for video compressive sensing,” in Proc. ICCP (IEEE, 2015).

Han, S.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Hansard, M.

G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
[Crossref] [PubMed]

Hartley, R.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003).

Harwit, M.

Heide, F.

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Heidrich, W.

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Held, D.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Horaud, R.

G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
[Crossref] [PubMed]

Howell, J. C.

Howland, G. A.

Huang, X.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Hullin, M.

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

Hullin, M. B.

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Ibbett, R. N.

Ito, T.

Kadambi, A.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.

Kagawa, K.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Kakue, T.

Kammel, S.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Kawahito, S.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Kelly, K. F.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Khwaja, A.

S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
[Crossref]

Kim, S.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Kim, T.-C.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Kim, W.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Kirmani, A.

Kodama, M.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Kolb, A.

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Kolter, J. Z.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Kondermann, D.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Kutulakos, K.

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Kutulakos, K. N.

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

Lange, R.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001).
[Crossref]

Langer, D.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Laska, J. N.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Lee, S.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Lenzen, F.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Levinson, J.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Li, F.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Lum, D. J.

Ma, J.

S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
[Crossref]

Magnor, M.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Martin, J.

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

Matsuda, N.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Meister, S.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Min, D.-K.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Mittleman, D.

D. Mittleman, “Sensing with terahertz radiation,” Springer85, (2013).

Mutto, C. D.

C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo, “Time-of-flight cameras and microsoft kinect,” (Springer, 2012).
[Crossref]

Nair, R.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Nayar, S. K.

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
[Crossref] [PubMed]

Nelson, E. D.

Noh, Y.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

O’Toole, M.

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Pan, Z.

C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.

Park, H.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

Phillips, P. G.

Pink, O.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Pratt, V.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Qin, X.

X. Song, Y. Dai, and X. Qin, “Deep Depth Super-Resolution: Learning Depth Super-Resolution using Deep Convolutional Neural Network,” arXiv, (2016).

Raskar, R.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.

Ruhl, K.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Salman Asif, M.

H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.

Sankaranarayanan, A. C.

H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.

J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens: A scalable architecture for video compressive sensing,” in Proc. ICCP (IEEE, 2015).

A. C. Sankaranarayanan, C. Studer, and R. G. Baraniu, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. ICCP (IEEE, 2012).

A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).

Schafer, H.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

Scharstein, D.

D. Scharstein and R. Szeliski, “High-accuracy stereo depth maps using structured light,” in Proc. CVPR (IEEE, 2003), pp. I-I.

Schechner, Y. Y.

Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
[Crossref] [PubMed]

Schuon, S.

S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.

Seitz, P.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001).
[Crossref]

Shi, B.

A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.

Shimobaba, T.

Sloane, N. J. A.

Sokolsky, M.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Song, X.

X. Song, Y. Dai, and X. Qin, “Deep Depth Super-Resolution: Learning Depth Super-Resolution using Deep Convolutional Neural Network,” arXiv, (2016).

Stanek, G.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Stavens, D.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Streeter, L.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

Studer, C.

A. C. Sankaranarayanan, C. Studer, and R. G. Baraniu, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. ICCP (IEEE, 2012).

Sun, M.-T.

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
[Crossref]

J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.

Sun, T.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Szeliski, R.

D. Scharstein and R. Szeliski, “High-accuracy stereo depth maps using structured light,” in Proc. CVPR (IEEE, 2003), pp. I-I.

Taamazyan, V.

A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.

Takasawa, T.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Takhar, D.

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

Teichman, A.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Theobalt, C.

S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.

Thru, S.

S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.

Thrun, S.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Ti, C.

C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.

Torras, C.

S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: a survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Turaga, P.

A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).

Usui, T.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Veeraraghavan, A.

H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.

Walton, M.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Wang, J.

J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens: A scalable architecture for video compressive sensing,” in Proc. ICCP (IEEE, 2015).

Ware, M. R.

Werling, M.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

Wetzstein, G.

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

Whyte, R.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

Wong, F. N. C.

Xiao, L.

F. Heide, L. Xiao, A. Kolb, M. Hullin, and W. Heidrich, “Imaging in scattering media using correlation image sensors and sparse convolutional coding,” Opt. Express 22(2), 26338–26350 (2014).
[Crossref] [PubMed]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

Xie, J.

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
[Crossref]

J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.

Yang, R.

C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.

Yasutomi, K.

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

Yeh, C.

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

Yu, S.

S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
[Crossref]

Zanuttigh, P.

C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo, “Time-of-flight cameras and microsoft kinect,” (Springer, 2012).
[Crossref]

Zisserman, A.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003).

ACM Trans. Graph. (5)

F. Heide, M. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. 32(4), 45 (2013).
[Crossref]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. 32(6), 167 (2013).
[Crossref]

F. Heide, W. Heidrich, M. Hullin, and G. Wetzstein, “Doppler time-of-flight imaging,” ACM Trans. Graph. 34(4), 36 (2015).
[Crossref]

M. Gupta, S. K. Nayar, M. B. Hullin, and J. Martin, “Phasor imaging: A generalization of correlation-based time-of-flight imaging,” ACM Trans. Graph. 34(5), 156 (2015).
[Crossref]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5D transient analysis of global light transport,” ACM Trans. Graph. 33(4), 87 (2014).

Appl. Opt. (4)

IEEE J. Quantum Electron. (1)

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001).
[Crossref]

IEEE Sens. J. (1)

S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: a survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

IEEE Signal Process. Mag. (1)

M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008).
[Crossref]

IEEE Trans. Image Process. (2)

J. M. Bioucas-Dias and M. A. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process. 25(1), 428–438 (2016).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

G. D. Evangelidis, M. Hansard, and R. Horaud, “Fusion of range and stereo data for high-resolution scene-modeling,” IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2178–2192 (2015).
[Crossref] [PubMed]

Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeu, “Multiplexing for optimal lighting,” IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007).
[Crossref] [PubMed]

J. Opt. Soc. Am. (2)

Opt. Express (4)

Signal Processing (2)

S. Yu, A. Khwaja, and J. Ma, “Compressed sensing of complex-valued data,” Signal Processing 92(2), 357–362 (2012).
[Crossref]

J. H. Ender, “On compressive sensing applied to radar,” Signal Processing 90(5), 1402–1414 (2010).
[Crossref]

Other (23)

D. Scharstein and R. Szeliski, “High-accuracy stereo depth maps using structured light,” in Proc. CVPR (IEEE, 2003), pp. I-I.

D. Mittleman, “Sensing with terahertz radiation,” Springer85, (2013).

K. Yasutomi, T. Usui, S. Han, M. Kodama, T. Takasawa, K. Kagawa, and S. Kawahito, “A time-of-flight image sensor with sub-mm resolution using draining only modulation pixels,” in Proc. Int. Image Sensor Workshop (2013), pp. 357–360.

S. Kim, S. Cha, H. Park, J. Gong, Y. Noh, W. Kim, S. Lee, D.-K. Min, W. Kim, and T.-C. Kim, “Time of flight image sensor with 7um pixel and 640× 480 resolution,” in IEEE Symposium on VLSI Technology (2013), T146–T147.

S. Schuon, C. Theobalt, J. Davis, and S. Thru, “High-quality scanning using time-of-flight depth superresolution,” in Proc. CVPR Workshops (IEEE, 2008), pp. 1–7.

R. Nair, K. Ruhl, F. Lenzen, S. Meister, H. Schafer, C. S. Garbe, M. Eisemann, M. Magnor, and D. Kondermann, “A survey on time-of-flight stereo fusion,” Springer, 105–127 (2013).

L. R. Bissonnette, Lidar: Range-Resolved Optical Remote Sensing of the Atmosphere (Springer, 2005).

C. D. Mutto, P. Zanuttigh, and G. M. Cortelazzo, “Time-of-flight cameras and microsoft kinect,” (Springer, 2012).
[Crossref]

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in IEEE Intelligent Vehicles Symposium (2011), pp. 163–168.

K. Gallo and G. Assanto, “Vision based obstacle detection for wheeled robots,” in International Conference on Control, Automation and Systems (2008), pp. 1587–1592.

Microsoft Kinect Sensor (The second verison), https://developer.microsoft.com/en-us/windows/kinect , (2016).

TI ToF sensor, https://www.ti.com/sensing-products/optical-sensors/3d-time-of-flight/overview.html , (2016).

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003).

C. Yeh, N. Matsuda, X. Huang, F. Li, M. Walton, and O. Cossairt, “A Streamlined Photometric Stereo Framework for Cultural Heritage,” in Proc. ECCV (Springer, 2016), pp. 738–752.

A. C. Sankaranarayanan, P. Turaga, R. Baraniuk, and R. Chellappa, “Compressive acquisition of dynamic scenes,” in Proc. ECCV (Springer, 2010).

A. C. Sankaranarayanan, C. Studer, and R. G. Baraniu, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. ICCP (IEEE, 2012).

J. Wang, M. Gupta, and A. C. Sankaranarayanan, “LiSens: A scalable architecture for video compressive sensing,” in Proc. ICCP (IEEE, 2015).

H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in Proc. CVPR (IEEE, 2015), pp. 2358–2366.

X. Song, Y. Dai, and X. Qin, “Deep Depth Super-Resolution: Learning Depth Super-Resolution using Deep Convolutional Neural Network,” arXiv, (2016).

C. Ti, R. Yang, J. Davis, and Z. Pan, “Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor,” in Proc. CVPR (IEEE, 2015), pp. 4334–4342.

A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in Proc. CVPR (IEEE, 2015), pp. 3370–3378.

L. Xiao, F. Heide, M. O’Toole, A. Kolb, M. B. Hullin, K. Kutulakos, and W. Heidrich, “Defocus deblurring and superresolution for time-of-flight depth cameras,” in Proc. CVPR (IEEE, 2015), pp. 2376–2384.

J. Xie, C.-C. Chou, R. Feris, and M.-T. Sun, “Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering,” in Proc. ICME (IEEE, 2014), pp. 1–6.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1 CS-ToF architecture: The light from the laser diode hits the object and is reflected and imaged on the DMD. Then, the DMD-modulated image is re-imaged at the ToF sensor plane via a relay lens. The laser diode, DMD, and ToF camera are controlled and synchronized by a computer.
Fig. 2
Fig. 2 ToF depth imaging (assume single depth): The computer sends out two signals: m(t) to control the laser diode and r(tψ) as reference to ToF sensor. The reflection from object (apm(tϕp)) is collected by ToF pixels, and then correlates with the reference signal (r(tψ)) to generate the camera’s output.
Fig. 3
Fig. 3 CS-ToF prototype system with components of the system are highlighted.
Fig. 4
Fig. 4 System Calibration: Calibration is performed by displaying an array of impulses on the DMD and measuring the sensor response for each individual impulse in the array. The response is then placed in the corresponding location in C. We traverse every DMD-sensor pixel pair to complete the matrix C.
Fig. 5
Fig. 5 Pixel scanning of a resolution target: (a) shows original low-resolution ToF measurement of the resolution chart target. (b) shows the pixel-wise scanning for the resolution target. Color boxes mark corresponding insets.
Fig. 6
Fig. 6 Real-world experiment setups: (a) Conceptual diagram of the resolution target experiment. (b) Photo of the resolution targets used. (c) Conceptual diagram of the natural scene experiment. (d) Photo of the natural scene
Fig. 7
Fig. 7 Intensity reconstruction of resolution charts: (a), (b), (c), and (d) show the original LR ToF intensity image, HR CS-ToF reconstruction results with no compression, 0.6 and 0.25 compression ratios, respectively. Fine patterns on resolution chart and the center of Siemens Star are shown in the insets. Ground truth intensity of the insets, taken with a 12-MP camera, are displayed on the left.
Fig. 8
Fig. 8 Phase reconstruction for a natural scene: (a), (b), (c), and (d) show LR ToF phase image and HR CS-ToF reconstruction phase images using no compression, 0.6 and 0.25 compression ratios, respectively. Colorbars show the depth information with unit of meter. A portion of the far resolution chart and the white board behind is also shown in the insets (a1–d1) with their corresponding photograph (marked with green box) in the left. Red arrows point out the boundary of two planes at different depths in (a1) and the corresponding photograph. Leaves and their branches on ”toy tree” are shown in the insets (a2–d2) with their corresponding photograph (marked with red box) in the left. Close-up images of (a2–d2) are further shown in (a3–d3).
Fig. 9
Fig. 9 Intensity reconstruction for a natural scene: (a), (b), (c), and (d) show LR ToF intensity image and HR CS-ToF reconstruction intensity image using no compression, 0.6 and 0.25 compression ratios, respectively. Fine patterns on the toy tree and the metal star are shown in the insets ((a1–d1, a2–d2)) with their corresponding photographs on the left (marked with the green box, and the red box). Note the screw on the metal star (marked with the red dashed circle) and the tip of the metal star (marked with the red arrow).
Fig. 10
Fig. 10 Scene projected on DMD plane with white field illumination: (a). The scene on DMD with ToF camera placed at the back focal plane of the relay lens. (c). The same scene on DMD with ToF camera slightly defocused. Color boxes represent insets in (a) and (c).
Fig. 11
Fig. 11 The quantification of depth accuracy for CS-ToF: (a). The photograph of the 3D scene for the simulation experiments. (b). The ground truth depth for the 3D scene. (c). The bicubic interpolation of LR ToF measurement depth with 25 dB Gaussian noise added in the system. (d), (e) show the HR CS-ToF depth iamges with 0.6 and 0.2 compression ratios, respectively. (25 dB Gaussian noise is added in the measurements). (f) shows the depth values along the red lines in (b–e) with 30dB SNR Gaussian noise in the measurements. (g) shows the depth values on the same pixels with (f) with 25dB SNR Gaussian noise added. (h) shows the depth values on the same pixels with (f) with 20dB SNR Gaussian noise added.

Tables (1)

Tables Icon

Table 1 RMSE of LR ToF measurement depth with bicubic interpolation and HR CS-ToF reconstruction depth with respect to the grouth truth depth.

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

B ( p , ψ ) = t = 0 T a p m ( t ϕ p ) r ( t ψ ) d t
m ( t ) = o m + a m cos ( ω t )
r ( t ) = o r + a r cos ( ω t ψ )
a ( p ) = [ B ( p , 3 π / 2 ) B ( p , π / 2 ) ] 2 + [ B ( p , π ) B ( p , 0 ) ] 2 2
ϕ ( p ) = arctan ( B ( p , 3 π / 2 ) B ( p , π / 2 ) B ( p , π ) B ( p , 0 ) ) .
x = a s e i ϕ s
y = a e i ϕ
y = CMx = Ax
y = Ax [ α 1 e i ϕ 1 α M e i ϕ M ] = [ C 11 C 1 N C M 1 C M N ] [ M 1 M N ] I [ α 1 e i ϕ 1 α N e i ϕ N ]
[ y 1 y 2 y T ] = [ A 1 x A 2 x A T x ] = [ A 1 A 2 A T ] x
x ^ = argmin x 1 2 y Ax 2 + λ Φ ( x )
Φ ( x ) = TV ( x ) = i | ( G u ( x i ) | 2 + | G v ( x i ) | 2

Metrics