Abstract

Ultrafast sources and detectors have been used to record the time-resolved scattering of light propagating through macroscopic scenes. In the context of computational imaging, decomposition of this transient light transport (TLT) is useful for applications, such as characterizing materials, imaging through diffuser layers, and relighting scenes dynamically. Here, we demonstrate a method of convolutional sparse coding to decompose TLT into direct reflections, inter-reflections, and subsurface scattering. The method relies on the sparsity composition of the time-resolved kernel. We show that it is robust and accurate to noise during the acquisition process.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. S. Colak, D. Papaioannou, G. ’t Hooft, M. Van der Mark, H. Schomberg, J. Paasschens, J. Melissen, and N. Van Asten, Appl. Opt. 36, 180 (1997).
    [CrossRef]
  2. S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
    [CrossRef]
  3. D. Reddy, R. Ramamoorthi, and B. Curless, in Proceedings of the ECCV (Springer, 2012), pp. 596–610.
  4. S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, in Proceedings of the ICCV (IEEE, 2005), pp. 1440–1447.
  5. N. Abramson, Opt. Lett. 3, 121 (1978).
    [CrossRef]
  6. D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
    [CrossRef]
  7. J. Busck and H. Heiselberg, Appl. Opt. 43, 4705 (2004).
    [CrossRef]
  8. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
    [CrossRef]
  9. N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
    [CrossRef]
  10. D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.
  11. M. B. A. Velten and R. Ramesh, Imaging Systems and Applications (Optical Society of America, 2011).
  12. B. A. Olshausen and D. J. Fieldt, Vis. Res. 37, 3311 (1997).
    [CrossRef]
  13. Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
    [CrossRef]
  14. E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).
  15. S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
    [CrossRef]
  16. Y. Li and S. Osher, Inverse Probl. Imaging 3, 487 (2009).
  17. S. V. Vaseghi, Advanced Digital Signal Processing and Noise Reduction (Wiley, 2008).

2013 (1)

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

2012 (1)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

2011 (1)

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

2009 (1)

Y. Li and S. Osher, Inverse Probl. Imaging 3, 487 (2009).

2008 (1)

E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).

2006 (1)

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

2004 (1)

2001 (1)

S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
[CrossRef]

1997 (2)

1991 (1)

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

1978 (1)

’t Hooft, G.

Abramson, N.

Agrawal, A.

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

Bala, K.

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

Bawendi, M. G.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

Boyd, S. P.

E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).

Busck, J.

Candes, E. J.

E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).

Chang, W.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Chen, S. S.

S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
[CrossRef]

Colak, S.

Curless, B.

D. Reddy, R. Ramamoorthi, and B. Curless, in Proceedings of the ECCV (Springer, 2012), pp. 596–610.

Dai, Q.

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Deng, Y.

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Donoho, D. L.

S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
[CrossRef]

Fieldt, D. J.

B. A. Olshausen and D. J. Fieldt, Vis. Res. 37, 3311 (1997).
[CrossRef]

Flotte, T.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Fujimoto, J. G.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Gregory, K.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Grossberg, M. D.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

Gupta, O.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

Hee, M. R.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Heiselberg, H.

Hu, S.

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Huang, D.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Krishnan, G.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

Kutulakos, K. N.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, in Proceedings of the ICCV (IEEE, 2005), pp. 1440–1447.

Li, Y.

Y. Li and S. Osher, Inverse Probl. Imaging 3, 487 (2009).

Lin, C. P.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Liu, R.

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Matsushita, Y.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, in Proceedings of the ICCV (IEEE, 2005), pp. 1440–1447.

Melissen, J.

Naik, N.

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

Nayar, S. K.

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

O’Toole, M.

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

Olshausen, B. A.

B. A. Olshausen and D. J. Fieldt, Vis. Res. 37, 3311 (1997).
[CrossRef]

Osher, S.

Y. Li and S. Osher, Inverse Probl. Imaging 3, 487 (2009).

Paasschens, J.

Papaioannou, D.

Puliafito, C. A.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Ramamoorthi, R.

D. Reddy, R. Ramamoorthi, and B. Curless, in Proceedings of the ECCV (Springer, 2012), pp. 596–610.

Ramesh, R.

M. B. A. Velten and R. Ramesh, Imaging Systems and Applications (Optical Society of America, 2011).

Raskar, R.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

Reddy, D.

D. Reddy, R. Ramamoorthi, and B. Curless, in Proceedings of the ECCV (Springer, 2012), pp. 596–610.

Saunders, M. A.

S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
[CrossRef]

Schomberg, H.

Schuman, J. S.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Seitz, S. M.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, in Proceedings of the ICCV (IEEE, 2005), pp. 1440–1447.

Stinson, W. G.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Swanson, E. A.

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

Van Asten, N.

Van der Mark, M.

Vaseghi, S. V.

S. V. Vaseghi, Advanced Digital Signal Processing and Noise Reduction (Wiley, 2008).

Veeraraghavan, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

Velten, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

Velten, M. B. A.

M. B. A. Velten and R. Ramesh, Imaging Systems and Applications (Optical Society of America, 2011).

Wakin, M. B.

E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).

Willwacher, T.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

Wu, D.

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

Zhang, Z.

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Zhao, S.

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

ACM Trans. Graph. (2)

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala, ACM Trans. Graph. 30, 171 (2011).
[CrossRef]

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, ACM Trans. Graph. 25, 935 (2006).
[CrossRef]

Appl. Opt. (2)

IEEE Trans. Neural Netw. Learn. Syst. (1)

Y. Deng, Q. Dai, R. Liu, Z. Zhang, and S. Hu, IEEE Trans. Neural Netw. Learn. Syst. 24, 383 (2013).
[CrossRef]

Inverse Probl. Imaging (1)

Y. Li and S. Osher, Inverse Probl. Imaging 3, 487 (2009).

J. Fourier Anal. Appl. (1)

E. J. Candes, M. B. Wakin, and S. P. Boyd, J. Fourier Anal. Appl. 14, 877 (2008).

Nat. Commun. (1)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[CrossRef]

Opt. Lett. (1)

Science (1)

D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, Science 254, 1178 (1991).
[CrossRef]

SIAM Rev. (1)

S. S. Chen, D. L. Donoho, and M. A. Saunders, SIAM Rev. 43, 129 (2001).
[CrossRef]

Vis. Res. (1)

B. A. Olshausen and D. J. Fieldt, Vis. Res. 37, 3311 (1997).
[CrossRef]

Other (5)

D. Wu, M. O’Toole, A. Velten, A. Agrawal, and R. Raskar, in Proceedings of CVPR (IEEE, 2012), pp. 366–373.

M. B. A. Velten and R. Ramesh, Imaging Systems and Applications (Optical Society of America, 2011).

S. V. Vaseghi, Advanced Digital Signal Processing and Noise Reduction (Wiley, 2008).

D. Reddy, R. Ramamoorthi, and B. Curless, in Proceedings of the ECCV (Springer, 2012), pp. 596–610.

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos, in Proceedings of the ICCV (IEEE, 2005), pp. 1440–1447.

Supplementary Material (4)

» Media 1: MOV (3273 KB)     
» Media 2: MOV (2558 KB)     
» Media 3: MOV (4106 KB)     
» Media 4: MOV (3407 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1.
Fig. 1.

(a) TLT measurement system. (b) Streak image and time profile.

Fig. 2.
Fig. 2.

Forward model of time profile. (a) Schematic of TLT system. (b) Input laser pulse. (c) Scene interaction effects. (d) Time-resolved PSF of the system. (e) Time profile of a pixel Q .

Fig. 3.
Fig. 3.

Decomposition of time profile into its components. (a)–(c) Time profiles for three simulated pixels. Inset: decomposed components. d , s , r 2 , and r 3 , respectively, denote the decomposed direct component, sss, the second bounce, and the third bounce. (a) Time profile containing only d and s . (b) Time profile containing d , s , and r 2 . (c) Time profile containing d , s , r 2 , and r 3 . (d) Decomposition fidelity compared with GradBM under three types of noise.

Fig. 4.
Fig. 4.

Decomposition result on real TLT data. (a) Photo of Scene 1 and corresponding TLT data integrated over time. (b) Time profiles of point 1 and corresponding ConvSCD and GradBM decompositions. (c) Decomposed frames of d and s of ConvSCD (Media 1). (d) Decomposed frames of d and s of GradBM (Media 2). (Denotation is the same as that in Fig. 3.)

Fig. 5.
Fig. 5.

Decomposition result on real TLT data. (a) Photo of Scene 2 and corresponding TLT data integrated over time. (b) Time profiles of point 2 and corresponding ConvSCD and GradBM decompositions. (c) Decomposed frames of s , d , r 2 , and r 3 of ConvSCD (Media 3). (d) Decomposed frames of s , d , and r 2 of GradBM (Media 4). (Denotation is the same as that in Fig. 3.)

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

p ( t ) = c ( t ) * e ( t ) * i ( t ) + n ( t ) ,
p ( t ) = { r = 0 R a r δ ( t t r ) + b δ ( t t s ) * [ e α t u ( t ) ] * } * e t 2 / ( 2 σ 0 2 ) + n ( t ) .
p = n = 1 N ( k n ( 1 ) * z n ( 1 ) ) + m = 1 M ( k m ( 2 ) * z m ( 2 ) ) + n .
min n z n ( 1 ) l s + m z m ( 2 ) l s s . t . p = n ( k n ( 1 ) * z n ( 1 ) ) + m ( k m ( 2 ) * z m ( 2 ) ) + n , n 2 < ε
min λ p n ( k n ( 1 ) * z n ( 1 ) ) m ( k m ( 2 ) * z m ( 2 ) ) 2 + n z n ( 1 ) l s + m z m ( 2 ) l s ,

Metrics