Abstract

We analyze multi-bounce propagation of light in an unknown hidden volume and demonstrate that the reflected light contains sufficient information to recover the 3D structure of the hidden scene. We formulate the forward and inverse theory of secondary scattering using ideas from energy front propagation and tomography. We show that using Fresnel approximation greatly simplifies this problem and the inversion can be achieved via a backpropagation process. We study the invertibility, uniqueness and choices of space-time-angle dimensions using synthetic examples. We show that a 2D streak camera can be used to discover and reconstruct hidden geometry. Using a 1D high speed time of flight camera, we show that our method can be used recover 3D shapes of objects “around the corner”.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
    [CrossRef]
  2. S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, “A theory of inverse light transport,” in Proc. of IEEE ICCV (2005), Vol. 2, pp. 1440–1447.
  3. S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
    [CrossRef]
  4. A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.
  5. R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
    [CrossRef]
  6. S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.
  7. R. Raskar and J. Davis, “5D time-light transport matrix: What can we reason about scene properties,” MIT Technical Report (2008), http://hdl.handle.net/1721.1/67888 .
  8. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
    [CrossRef]
  9. D. Needell and J. Tropp, “CoSaMP: Iterative signal recovery from incomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.26, 301–321 (2009).
    [CrossRef]
  10. Hamamatsu, “Hamamatsu streak camera tutorial,” http://learn.hamamatsu.com/tutorials/java/streakcamera/ .
  11. D. Forsyth and J. Ponce, Computer Vision, a Modern Approach (Prentice Hall, 2002).
  12. E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
    [CrossRef] [PubMed]
  13. B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
    [CrossRef]

2011

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

2010

S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.

2009

D. Needell and J. Tropp, “CoSaMP: Iterative signal recovery from incomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.26, 301–321 (2009).
[CrossRef]

2008

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

2006

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

2005

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

2004

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Atcheson, B.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Bardagjy, A.

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

Bawendi, M. G.

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

Bradley, D.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Chen, B.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Couch, G.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Davis, J.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.

Ferrin, T.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Forsyth, D.

D. Forsyth and J. Ponce, Computer Vision, a Modern Approach (Prentice Hall, 2002).

Garg, G.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Goddard, T.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Greenblatt, D.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Grossberg, M. D.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

Gupta, O.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

Heidrich, W.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Horowitz, M.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Huang, C.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Hutchison, T.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.

Ihrke, I.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Kirmani, A.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.

Krishnan, G.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

Kutulakos, K. N.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, “A theory of inverse light transport,” in Proc. of IEEE ICCV (2005), Vol. 2, pp. 1440–1447.

Lensch, H. P. A.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Levoy, M.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Liu, S.

S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.

Magnor, M.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Marschner, S. R.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Matsushita, Y.

S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, “A theory of inverse light transport,” in Proc. of IEEE ICCV (2005), Vol. 2, pp. 1440–1447.

Meng, E.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Nayar, S. K.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

Needell, D.

D. Needell and J. Tropp, “CoSaMP: Iterative signal recovery from incomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.26, 301–321 (2009).
[CrossRef]

Ng, T.

S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.

Pandharkar, R.

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

Pettersen, E.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Ponce, J.

D. Forsyth and J. Ponce, Computer Vision, a Modern Approach (Prentice Hall, 2002).

Raskar, R.

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.

Seidel, H.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Seitz, S. M.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, “A theory of inverse light transport,” in Proc. of IEEE ICCV (2005), Vol. 2, pp. 1440–1447.

Sen, P.

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

Tevs, A.

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Tropp, J.

D. Needell and J. Tropp, “CoSaMP: Iterative signal recovery from incomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.26, 301–321 (2009).
[CrossRef]

Veeraraghavan, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

Velten, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

Willwacher, T.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

ACM Trans. Graphics

P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. A. Lensch, “Dual photography,” ACM Trans. Graphics24(2), 745–755 (2005).
[CrossRef]

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graphics25, 935–944 (2006).
[CrossRef]

B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley, M. Magnor, and H. Seidel, “Time-resolved 3d capture of non-stationary gas flows,” ACM Trans. Graphics27, 1–9 (2008).
[CrossRef]

Appl. Comput. Harmon. Anal.

D. Needell and J. Tropp, “CoSaMP: Iterative signal recovery from incomplete and inaccurate samples,” Appl. Comput. Harmon. Anal.26, 301–321 (2009).
[CrossRef]

J. Comput. Chem.

E. Pettersen, T. Goddard, C. Huang, G. Couch, D. Greenblatt, E. Meng, and T. Ferrin, “UCSF Chimera–a visualization system for exploratory research and analysis,” J. Comput. Chem.25, 1605–1612 (2004).
[CrossRef] [PubMed]

Nat. Commun.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging,” Nat. Commun.3, 745 (2011).
[CrossRef]

Proc. of CVPR

R. Pandharkar, A. Velten, A. Bardagjy, M. G. Bawendi, and R. Raskar, “Estimating motion and size of moving non-line-of-sight objects in cluttered environments,” in Proc. of CVPR (2011), pp. 265–272.
[CrossRef]

Proc. of ECCV

S. Liu, T. Ng, and Y. Matsushita, “Shape from second-bounce of light transport,” in Proc. of ECCV (2010), pp. 280–293.

Other

R. Raskar and J. Davis, “5D time-light transport matrix: What can we reason about scene properties,” MIT Technical Report (2008), http://hdl.handle.net/1721.1/67888 .

Hamamatsu, “Hamamatsu streak camera tutorial,” http://learn.hamamatsu.com/tutorials/java/streakcamera/ .

D. Forsyth and J. Ponce, Computer Vision, a Modern Approach (Prentice Hall, 2002).

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in Proc. of IEEE ICCV (2009), pp. 159–166.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, “A theory of inverse light transport,” in Proc. of IEEE ICCV (2005), Vol. 2, pp. 1440–1447.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1
Fig. 1

Forward Model. (Left) The laser illuminates the surface S and each point sS generates a energy front. The spherical energy front contributes to a hyperbola in the space-time streak photo, IR. (Right) Spherical energy fronts propagating from a point create a hyperbolic space-time curve in streak photo.

Fig. 2
Fig. 2

A space time transform on a raw streak photo allows us to convert a 4 segment problem into a sequence of 2 segment problems. The toy scene is a small 1cm×1cm patch creating a prominent (blurred) hyperbola in thewarped photo. Backprojection creates a low frequency residual but simple thresholding recovers the patch geometry.

Fig. 3
Fig. 3

The top left figure shows streak images being generated by near field sources. On bottom left we see effect when this sources travel farther away, The rightmost figure depicts how we can analytically predict single sources using multiple sensor laser combinations. Notice how the accuracy is effected if lasers shift.

Fig. 4
Fig. 4

Simulated reconstruction using CoSAMP. While CoSAMP promises to perform far superior on a perfectly calibrated system, it is outperformed by backprojection on the current data due to calibration inaccuracies.

Fig. 5
Fig. 5

Reconstruction of a scene consisting of a big disk, a triangle and a square at different depth. (Left) Ground truth. (Middle) Reconstruction, front view. (Right) Reconstruction, side view. Note that the disk is only partially reconstructed, and the square is rounded of, while the triangle is recovered very well. This illustrates the diminishing resolution in directions parallel to the receiver plane towards the borders of the field of view. The blue planes indicate the ground truth. The gray ground planes and shadows have been added to help visualization.

Fig. 6
Fig. 6

Reconstruction of a planar object in an unknown plane in 3D. (Left) The object. (Middle Left) 2D Projection of the filtered heatmap. (Middle Right) A 3D visualization of the filtered heatmap. (Right) Reconstruction using sparsity based methods. The gray ground plane has been added to aid visualization.

Fig. 7
Fig. 7

Reconstruction of a wooden man, painted white. Center - reconstruction using simple back projection based methods. Right - reconstruction using sparse reconstruction methods.

Fig. 8
Fig. 8

Depiction of our reconstruction algorithm for a scene consisting of two birds in different planes. From top left to bottom right: (a) Photographs of the input models.(b) 9 out of 33 streak images used for reconstruction. (c) The raw (unfiltered) backprojection.(d) The filtered backprojection,(e,f) after taking a second derivative. 3D renderings in Chimera.

Fig. 9
Fig. 9

The laser beam (red) is split to provide a syncronization signal for the camera (dotted red) and an attenuated reference pulse (orange) to compensate for synchronization drifts and laser intensity fluctiations. The main laser beam is directed to a wall with a steering mirror and the returned third bounce is captured by the streak camera. An Occluder inserted at the indicated position does not significantly change the collected image.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

I R ( w , t ) = I C ( H ( w ) , t L B C w ) .
I R ( w , t ) = S τ 1 π r c 2 δ ( r c t + τ ) I S ( s , τ ) d τ d 2 s
I R ( w , t ) = S I 1 π r c 2 1 π r l 2 δ ( t r c r l ) d 2 s
t r l = r c = ( x u ) 2 + ( y v ) 2 + z ( x , y ) 2
I R ( w , t , L ) = S I 1 π r c 2 1 π r l 2 δ ( t r c r l ) d 2 s = 3 1 π r c 2 1 π r l 2 δ ( t r c r l ) I δ S ( x ) d 3 x = 3 1 π r c 2 1 π r l 2 δ ( t r c r l ) W ( x ) d 3 x
c t = | v L | + | v w | + | w C | .
H ( v ) = p ( | v w | | v L | ) α I p .
H f = ( 3 ) 2 H .
H f > λ loc M loc + λ glob M glob

Metrics