Abstract

A compressive high-speed stereo imaging system is reported. The system is capable of reconstructing 3D videos at a frame rate 10 times higher than the sampling rate of the imaging sensors. An asymmetric configuration of stereo imaging system has been implemented by including a high-speed spatial modulator in one of the binocular views, and leaving the other view unchanged. We have developed a two-step reconstruction algorithm to recover the irradiance and depth information of the high-speed scene. The experimental results have demonstrated high-speed video reconstruction at 800fps from 80fps measurements. The reported compressive stereo imaging method does not require active illumination, offering a robust yet inexpensive solution to high-speed 3D imaging.

© 2017 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Motion-oriented high speed 3-D measurements by binocular fringe projection using binary aperiodic patterns

Shijie Feng, Qian Chen, Chao Zuo, Tianyang Tao, Yan Hu, and Anand Asundi
Opt. Express 25(2) 540-559 (2017)

High spatio-temporal resolution video with compressed sensing

Roman Koller, Lukas Schmid, Nathan Matsuda, Thomas Niederberger, Leonidas Spinoulas, Oliver Cossairt, Guido Schuster, and Aggelos K. Katsaggelos
Opt. Express 23(12) 15992-16007 (2015)

Three-dimensional single-pixel compressive reflectivity imaging based on complementary modulation

Wen-Kai Yu, Xu-Ri Yao, Xue-Feng Liu, Long-Zhen Li, and Guang-Jie Zhai
Appl. Opt. 54(3) 363-367 (2015)

References

  • View by:
  • |
  • |
  • |

  1. D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006).
    [Crossref]
  2. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
    [Crossref] [PubMed]
  3. X. Yuan and S. Pang, “Structured illumination temporal compressive microscopy,” Biomed. Opt. Express 7(3), 746–758 (2016).
    [Crossref] [PubMed]
  4. Y. Sun, X. Yuan, and S. Pang, “High-speed compressive range imaging based on active illumination,” Opt. Express 24(20), 22836–22846 (2016).
    [Crossref] [PubMed]
  5. D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel compressive camera for high speed imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 329–336.
    [Crossref]
  6. Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
    [Crossref]
  7. J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
    [Crossref] [PubMed]
  8. X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in International Conference on Image Processing (IEEE, 2016), pp. 2539 - 2543.
    [Crossref]
  9. J. M. Bioucas-Dias and M. A. Figueiredo, “A New TwIst: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
    [Crossref] [PubMed]
  10. X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
    [Crossref]
  11. Z. Wang, L. Spinoulas, K. He, L. Tian, O. Cossairt, A. K. Katsaggelos, and H. Chen, “Compressive holographic video,” Opt. Express 25(1), 250–262 (2017).
    [Crossref] [PubMed]
  12. S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
    [Crossref]
  13. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
    [Crossref]
  14. T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012).
    [Crossref] [PubMed]
  15. B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
    [Crossref]
  16. L. I. Rudin and S. Osher, “Total variation based image restoration with free local constraints,” in Proceedings - International Conference on Image Processing (IEEE, 1994), pp. 31–35.
    [Crossref]
  17. X. Yuan and S. Pang, “Compressive video microscope via structured illumination,” in International Conference on Image Processing (IEEE, 2016), pp. 1589–1593.
  18. G. Heiko Hirschmüller, “Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognition (2005), pp. 807–814.
  19. V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2001), pp. 508–515.
  20. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000).
    [Crossref]
  21. Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004).
    [Crossref] [PubMed]
  22. V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004).
    [Crossref] [PubMed]
  23. F. Gamboa and E. Gassiat, “Bayesian methods and maximum entropy for ill-posed inverse problems,” Ann. Stat. 25(1), 328–350 (1997).
    [Crossref]
  24. X. Yuan, Y. Sun, and S. Pang, “Compressive video sensing with side information,” Appl. Opt. 56(10), 2697–2704 (2017).
    [Crossref] [PubMed]
  25. X. Yuan, Y. Sun, and S. Pang, “Video compressed imaging using side information,” Proc. SPIE 10222, 102220I (2017).
    [Crossref]
  26. L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
    [Crossref]
  27. J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
    [Crossref] [PubMed]
  28. X. Yuan, X. Liao, P. Llull, D. Brady, and L. Carin, “Efficient patch-based approach for compressive depth imaging,” Appl. Opt. 55(27), 7556–7564 (2016).
    [Crossref] [PubMed]

2017 (4)

Z. Wang, L. Spinoulas, K. He, L. Tian, O. Cossairt, A. K. Katsaggelos, and H. Chen, “Compressive holographic video,” Opt. Express 25(1), 250–262 (2017).
[Crossref] [PubMed]

X. Yuan, Y. Sun, and S. Pang, “Compressive video sensing with side information,” Appl. Opt. 56(10), 2697–2704 (2017).
[Crossref] [PubMed]

X. Yuan, Y. Sun, and S. Pang, “Video compressed imaging using side information,” Proc. SPIE 10222, 102220I (2017).
[Crossref]

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

2016 (5)

2014 (2)

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
[Crossref]

2013 (1)

2012 (1)

T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012).
[Crossref] [PubMed]

2011 (1)

S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

2010 (1)

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

2007 (1)

J. M. Bioucas-Dias and M. A. Figueiredo, “A New TwIst: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

2006 (1)

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006).
[Crossref]

2004 (2)

Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004).
[Crossref] [PubMed]

V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004).
[Crossref] [PubMed]

2000 (1)

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

1997 (1)

F. Gamboa and E. Gassiat, “Bayesian methods and maximum entropy for ill-posed inverse problems,” Ann. Stat. 25(1), 328–350 (1997).
[Crossref]

Alenyà, G.

S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Archibald, J.

B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
[Crossref]

Bioucas-Dias, J. M.

J. M. Bioucas-Dias and M. A. Figueiredo, “A New TwIst: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

Bishop, T. E.

T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012).
[Crossref] [PubMed]

Boykov, Y.

Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004).
[Crossref] [PubMed]

Brady, D.

Brady, D. J.

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

Carin, L.

X. Yuan, X. Liao, P. Llull, D. Brady, and L. Carin, “Efficient patch-based approach for compressive depth imaging,” Appl. Opt. 55(27), 7556–7564 (2016).
[Crossref] [PubMed]

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
[Crossref]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

Chellappa, R.

D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel compressive camera for high speed imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 329–336.
[Crossref]

Chen, H.

Chen, Y.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Cossairt, O.

Donoho, D. L.

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006).
[Crossref]

Favaro, P.

T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012).
[Crossref] [PubMed]

Fernandez, S.

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

Figueiredo, M. A.

J. M. Bioucas-Dias and M. A. Figueiredo, “A New TwIst: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

Foix, S.

S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Gamboa, F.

F. Gamboa and E. Gassiat, “Bayesian methods and maximum entropy for ill-posed inverse problems,” Ann. Stat. 25(1), 328–350 (1997).
[Crossref]

Gao, L.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Gassiat, E.

F. Gamboa and E. Gassiat, “Bayesian methods and maximum entropy for ill-posed inverse problems,” Ann. Stat. 25(1), 328–350 (1997).
[Crossref]

Gu, J.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

Gupta, M.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

He, K.

Hitomi, Y.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

Katsaggelos, A. K.

Kittle, D.

Kolmogorov, V.

Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004).
[Crossref] [PubMed]

V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004).
[Crossref] [PubMed]

V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2001), pp. 508–515.

Lee, D. J.

B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
[Crossref]

Li, H.

X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
[Crossref]

Liang, J.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Liao, X.

X. Yuan, X. Liao, P. Llull, D. Brady, and L. Carin, “Efficient patch-based approach for compressive depth imaging,” Appl. Opt. 55(27), 7556–7564 (2016).
[Crossref] [PubMed]

X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
[Crossref]

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

Lillywhite, K.

B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
[Crossref]

Llado, X.

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

Llull, P.

Ma, C.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Mitsunaga, T.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

Nayar, S. K.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

Pang, S.

Pribanic, T.

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

Reddy, D.

D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel compressive camera for high speed imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 329–336.
[Crossref]

Salvi, J.

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

Sapiro, G.

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

Spinoulas, L.

Sun, Y.

Tian, L.

Tippetts, B.

B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
[Crossref]

Torras, C.

S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

Veeraraghavan, A.

D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel compressive camera for high speed imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 329–336.
[Crossref]

Wang, L. V.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Wang, Z.

Xu, Q.

Yang, J.

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

Yuan, X.

X. Yuan, Y. Sun, and S. Pang, “Video compressed imaging using side information,” Proc. SPIE 10222, 102220I (2017).
[Crossref]

X. Yuan, Y. Sun, and S. Pang, “Compressive video sensing with side information,” Appl. Opt. 56(10), 2697–2704 (2017).
[Crossref] [PubMed]

Y. Sun, X. Yuan, and S. Pang, “High-speed compressive range imaging based on active illumination,” Opt. Express 24(20), 22836–22846 (2016).
[Crossref] [PubMed]

X. Yuan and S. Pang, “Structured illumination temporal compressive microscopy,” Biomed. Opt. Express 7(3), 746–758 (2016).
[Crossref] [PubMed]

X. Yuan, X. Liao, P. Llull, D. Brady, and L. Carin, “Efficient patch-based approach for compressive depth imaging,” Appl. Opt. 55(27), 7556–7564 (2016).
[Crossref] [PubMed]

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
[Crossref] [PubMed]

X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in International Conference on Image Processing (IEEE, 2016), pp. 2539 - 2543.
[Crossref]

X. Yuan and S. Pang, “Compressive video microscope via structured illumination,” in International Conference on Image Processing (IEEE, 2016), pp. 1589–1593.

Zabih, R.

V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004).
[Crossref] [PubMed]

V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2001), pp. 508–515.

Zhang, Z.

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

Zhu, L.

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 3(7), 694–697 (2016).
[Crossref]

Ann. Stat. (1)

F. Gamboa and E. Gassiat, “Bayesian methods and maximum entropy for ill-posed inverse problems,” Ann. Stat. 25(1), 328–350 (1997).
[Crossref]

Appl. Opt. (2)

Biomed. Opt. Express (1)

IEEE Sens. J. (1)

S. Foix, G. Alenyà, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” IEEE Sens. J. 11(9), 1917–1926 (2011).
[Crossref]

IEEE Trans. Image Process. (2)

J. M. Bioucas-Dias and M. A. Figueiredo, “A New TwIst: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration,” IEEE Trans. Image Process. 16(12), 2992–3004 (2007).
[Crossref] [PubMed]

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Video compressive sensing using gaussian mixture models,” IEEE Trans. Image Process. 23(11), 4863–4878 (2014).
[Crossref] [PubMed]

IEEE Trans. Inf. Theory (1)

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (4)

T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012).
[Crossref] [PubMed]

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000).
[Crossref]

Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004).
[Crossref] [PubMed]

V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004).
[Crossref] [PubMed]

J. Real-Time Image Process. (1)

B. Tippetts, D. J. Lee, K. Lillywhite, and J. Archibald, “Review of stereo vision algorithms and their suitability for resource-limited systems,” J. Real-Time Image Process. 11(1), 5–25 (2016).
[Crossref]

Opt. Express (3)

Optica (1)

Pattern Recognit. (1)

J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010).
[Crossref]

Proc. SPIE (1)

X. Yuan, Y. Sun, and S. Pang, “Video compressed imaging using side information,” Proc. SPIE 10222, 102220I (2017).
[Crossref]

Sci. Adv. (1)

J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017).
[Crossref] [PubMed]

SIAM J. Imaging Sci. (1)

X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted-l2,1 minimization with applications to model-based compressive sensing,” SIAM J. Imaging Sci. 7(2), 797–823 (2014).
[Crossref]

Other (7)

L. I. Rudin and S. Osher, “Total variation based image restoration with free local constraints,” in Proceedings - International Conference on Image Processing (IEEE, 1994), pp. 31–35.
[Crossref]

X. Yuan and S. Pang, “Compressive video microscope via structured illumination,” in International Conference on Image Processing (IEEE, 2016), pp. 1589–1593.

G. Heiko Hirschmüller, “Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognition (2005), pp. 807–814.

V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” in Proc. IEEE International Conference on Computer Vision (ICCV) (2001), pp. 508–515.

D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2C2: Programmable pixel compressive camera for high speed imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 329–336.
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2011), pp. 287–294.
[Crossref]

X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in International Conference on Image Processing (IEEE, 2016), pp. 2539 - 2543.
[Crossref]

Supplementary Material (2)

NameDescription
» Visualization 1       A 10 frame high-speed video of a backward-moving ball
» Visualization 2       An 800fps 3D video is reconstructed from 80fps measurements shown in Fig.4 in the main paper.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (4)

Fig. 1
Fig. 1

System schematic (a) On the left-view optical path, Camera 1 records low-speed measurement IL. On the right-view optical path, high-speed right-view scene FRH are encoded by the DMD with N distinct patterns M. The coded right-view scene is then relayed by lens Lrelay and recorded by Camera 2 within the exposure time to form the right-view measurement IR. (b) A photo of the setup.

Fig. 2
Fig. 2

The flow chart of the reconstruction algorithm. The high-speed scene FRH is reconstructed from the modulated measurement IR using video compressive sensing inversion algorithm, TwIST. Then, the high-speed depth maps are estimated from IL and FRH by our one-to-N correspondence algorithm based on Graph Cut.

Fig. 3
Fig. 3

Reconstruction of a backward-moving ball. (a) Measurements from two optical paths in our system. The different traces of the motion blurs indicate the varying depth of the moving ball. (b) 1th, 6th and 10th fames of reconstructed high-speed video from single measurement IR. (c) 1th, 6th and 10th fames of reconstructed high-speed depth map. (d) An overlay plot of 10 depth maps within a single exposure. The gradient of the color implies that the ball is moving away from our imaging system. (See Visualization 1).

Fig. 4
Fig. 4

Reconstruction of an 800fps high-speed scene with two flying “shurikens”. (a) Stereo imaging measurements. (b) Selected frames of reconstructed 800fps video. (c) High-speed 3D scene. The rectangular shuriken rotated about 30 degrees within the exposure time, and the triangular shuriken rotated about 20 degrees.

Equations (14)

Equations on this page are rendered with MathJax. Learn more.

I R ( i,j )= n=1 N F RH ( i,j,n )M( i,j,n ) ,
I L ( i,j )= n=1 N F LH ( i,j,n ),
( F ^ RH , H ^ )=arg min F RH ,H I R n=1 N F RH M + I L n=1 N H F RH ,
( F ^ RH , H ^ )=arg min F RH ,H I R n=1 N F RH M +λΦ( F RH )+ I L n=1 N H F RH +κΩ(H),
F ^ RH =arg min F RH I R n=1 N F RH M +λΦ( F RH ),
I L ( i,j )= n=1 N F RH ( iD( i,j,n ),j,n ) ,
Z( i,j,n )= f c b D(i,j,n) ,
D ^ =arg min D E data (D)+ E regularizer (D),
E data (D)= i,j | I L (i,j) n F RH ( iD(i,j,n),j,n ) | .
E regularizer = E occlusion + E uniqueness + E smoothness
E occlusion = K occ N occ ,
E uniqueness = K uniqueness T(D),
E smoothness = s,pΠ, D s D p V ,
dZ= Z 2 f c b dD,

Metrics