Abstract

The single-shot compressed ultrafast photography (CUP) camera is the fastest receive-only camera in the world. In this Letter, we introduce an external CCD camera and a space- and intensity-constrained (SIC) reconstruction algorithm to improve the image quality of CUP. The external CCD camera takes a time-unsheared image of the dynamic scene. Unlike the previously used unconstrained algorithm, the proposed algorithm incorporates both spatial and intensity constraints based on the additional prior information provided by the external CCD camera. First, a spatial mask is extracted from the time-unsheared image to define the zone of action. Next, an intensity threshold is determined based on the similarity between the temporally projected image of the reconstructed datacube and the time-unsheared image. Both simulation and experimental studies show that the SIC reconstruction improves the spatial resolution, contrast, and general quality of the reconstructed image.

© 2016 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Bound constrained bundle adjustment for reliable 3D reconstruction

Yuanzheng Gong, De Meng, and Eric J. Seibel
Opt. Express 23(8) 10771-10785 (2015)

Image reconstruction using spectroscopic and hyperspectral information for compressive terahertz imaging

Zhimin Xu and Edmund Y. Lam
J. Opt. Soc. Am. A 27(7) 1638-1646 (2010)

High spatio-temporal resolution video with compressed sensing

Roman Koller, Lukas Schmid, Nathan Matsuda, Thomas Niederberger, Leonidas Spinoulas, Oliver Cossairt, Guido Schuster, and Aggelos K. Katsaggelos
Opt. Express 23(12) 15992-16007 (2015)

References

  • View by:
  • |
  • |
  • |

  1. P. W. W. Fuller, J. Imaging Sci. 57, 293 (2009).
    [Crossref]
  2. Invisible Vision Ltd, http://www.invisiblevision.com/pdf/UBSi_(1Bn_fps_Camera).pdf .
  3. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
    [Crossref]
  4. C. B. Schaffer, N. Nishimura, E. N. Glezer, A. M. T. Kim, and E. Mazur, Opt. Express 10, 196 (2002).
    [Crossref]
  5. G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
    [Crossref]
  6. Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).
  7. K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
    [Crossref]
  8. K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
    [Crossref]
  9. B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.
  10. H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
    [Crossref]
  11. L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
    [Crossref]
  12. J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
    [Crossref]
  13. J. M. Bioucas-Dias and M. A. T. Figueiredo, IEEE Trans. Image Process. 16, 2992 (2007).
    [Crossref]
  14. N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.
  15. A. Beck and M. Teboulle, IEEE Trans. Image Process. 18, 2419 (2009).
    [Crossref]

2015 (2)

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

2014 (3)

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

2012 (1)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

2009 (3)

P. W. W. Fuller, J. Imaging Sci. 57, 293 (2009).
[Crossref]

K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
[Crossref]

A. Beck and M. Teboulle, IEEE Trans. Image Process. 18, 2419 (2009).
[Crossref]

2007 (1)

J. M. Bioucas-Dias and M. A. T. Figueiredo, IEEE Trans. Image Process. 16, 2992 (2007).
[Crossref]

2002 (1)

1999 (1)

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

Barsi, C.

B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.

Bawendi, M. G.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

Beck, A.

A. Beck and M. Teboulle, IEEE Trans. Image Process. 18, 2419 (2009).
[Crossref]

Bioucas-Dias, J. M.

J. M. Bioucas-Dias and M. A. T. Figueiredo, IEEE Trans. Image Process. 16, 2992 (2007).
[Crossref]

Buller, G. S.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Chang, Y.-Y.

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

Downer, M. C.

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

Faccio, D.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Figueiredo, M. A. T.

J. M. Bioucas-Dias and M. A. T. Figueiredo, IEEE Trans. Image Process. 16, 2992 (2007).
[Crossref]

Fuller, P. W. W.

P. W. W. Fuller, J. Imaging Sci. 57, 293 (2009).
[Crossref]

Gao, L.

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

Gariepy, G.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Glezer, E. N.

Goda, K.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
[Crossref]

Gupta, O.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

Hai, P.

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

Henderson, R.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Heshmat, B.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.

Heya, M.

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

Hirosawa, K.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Horisaki, R.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Iwasaki, A.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Jalali, B.

K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
[Crossref]

Joshi, M.

N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.

Kannari, F.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Kim, A. M. T.

Krstajic, N.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Leach, J.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Li, C.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

Li, Z.

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

Liang, J.

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

Liao, H.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Mazur, E.

Miyanaga, N.

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

More, S.

N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.

Nakagawa, K.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Nakamura, A.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Nakasuji, M.

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

Nishimura, N.

Oishi, Y.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Phansalkar, N.

N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.

Raskar, R.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.

Sabale, A.

N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.

Sakuma, I.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Satat, G.

B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.

Schaffer, C. B.

Shiraga, H.

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

Teboulle, M.

A. Beck and M. Teboulle, IEEE Trans. Image Process. 18, 2419 (2009).
[Crossref]

Thomson, R. R.

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Tsia, K. K.

K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
[Crossref]

Tsukamoto, A.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Ushida, T.

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Veeraraghavan, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

Velten, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

Wang, L. V.

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

Wang, X.

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

Willwacher, T.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

Zgadzaj, R.

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

IEEE Trans. Image Process. (2)

J. M. Bioucas-Dias and M. A. T. Figueiredo, IEEE Trans. Image Process. 16, 2992 (2007).
[Crossref]

A. Beck and M. Teboulle, IEEE Trans. Image Process. 18, 2419 (2009).
[Crossref]

J. Imaging Sci. (1)

P. W. W. Fuller, J. Imaging Sci. 57, 293 (2009).
[Crossref]

Nat. Commun. (3)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Nat. Commun. 3, 745 (2012).
[Crossref]

G. Gariepy, N. Krstajić, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, Nat. Commun. 6, 6021 (2015).
[Crossref]

Z. Li, R. Zgadzaj, X. Wang, Y.-Y. Chang, and M. C. Downer, Nat. Commun. 5, 3085 (2014).

Nat. Photonics (1)

K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, Nat. Photonics 8, 695 (2014).
[Crossref]

Nature (2)

K. Goda, K. K. Tsia, and B. Jalali, Nature 458, 1145 (2009).
[Crossref]

L. Gao, J. Liang, C. Li, and L. V. Wang, Nature 516, 74 (2014).
[Crossref]

Opt. Express (1)

Rev. Sci. Instrum. (1)

H. Shiraga, M. Nakasuji, M. Heya, and N. Miyanaga, Rev. Sci. Instrum. 70, 620 (1999).
[Crossref]

Sci. Rep. (1)

J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, Sci. Rep. 5, 15504 (2015).
[Crossref]

Other (3)

N. Phansalkar, S. More, A. Sabale, and M. Joshi, in International Conference on Communications and Signal Processing (ICCSP) (IEEE, 2011), p. 218.

Invisible Vision Ltd, http://www.invisiblevision.com/pdf/UBSi_(1Bn_fps_Camera).pdf .

B. Heshmat, G. Satat, C. Barsi, and R. Raskar, in CLEO: Science and Innovations (Optical Society of America, 2014), paper STu3E.7.

Supplementary Material (2)

NameDescription
» Visualization 1: MP4 (175 KB)      Frame-by-frame comparison of the two reconstructed datacubes.
» Visualization 2: MP4 (97 KB)      Frame-by-frame comparison of the two reconstructed datacubes.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1.
Fig. 1.

Operation principle of the space- and intensity-constrained (SIC) reconstruction for compressed ultrafast photography (CUP).

Fig. 2.
Fig. 2.

Results of the numerical simulation. (a) Root-mean-square error (RMSE) as the similarity metric versus tested threshold values. (b) Fifth frame of the ground truth dynamic scene. (c) As (b), but shows the TwIST reconstructed result. (d) As (b) and (c), but shows the SIC reconstructed result. (e) Change in average normalized intensity across ten frames of the two reconstructed datacubes.

Fig. 3.
Fig. 3.

System schematic. Inset: detailed illustration of the principle of the streak camera. DMD, digital micromirror device; MCP, microchannel plate; PS, phosphor screen; V, sweep voltage. Equipment details: camera lens, Nikon, f = 18 55    mm ; DMD, Texas Instruments, LightCrafter 3000; external CCD camera, Point Gray, GS3-U3-28S4M-C; internal CCD, Hamamatsu, ORCA-R2; objective lens, Olympus, UPLSAPO 4 × ; streak camera, Hamamatsu, C7700; tube lens, Thorlabs, AC254-150-A.

Fig. 4.
Fig. 4.

Car model experimental results. (a) Temporally projected image of the TwIST-reconstructed datacube. (b) As (a), but showing the datacube from the SIC reconstruction. A frame-by-frame comparison of the two reconstructed datacubes is shown in Visualization 1. (c) Signal evolution at the circled pixels in (a) and (b). (d) An x y t volumetric rendering of the TwIST-reconstructed datacube. (e) As (d), but showing the SIC-reconstructed datacube.

Fig. 5.
Fig. 5.

Propagating picosecond laser pulse results. (a) TwIST-reconstructed frame at t = 170    ps . (b) SIC-reconstructed frame at t = 170    ps . White edges in both panels indicate the zone of action identified by the external CCD camera, which was not used in the TwIST reconstruction method. A frame-by-frame comparison of the two reconstructed datacubes is shown in Visualization 2. (c) Image profiles along the indicated lines labeled in (a) and (b).

Equations (3)

Equations on this page are rendered with MathJax. Learn more.

x ^ = arg min x { 0.5 TSC ( x ) y 2 2 + λ x TV } ,
x s = arg min x M , x > s { 0.5 TSC ( x ) y 2 2 + λ x TV } .
s ^ = arg min s [ 1 / N · T ( x s ) y 0 2 2 ] 1 / 2 ,

Metrics