Abstract

We present a hybrid camera system that combines optical designs with computational processing to achieve content-adaptive high-resolution hyperspectral video acquisition. In particular, we record two video streams: one high-spatial resolution RGB video and one low-spatial resolution hyperspectral video in which the recorded points are dynamically selected using a spatial light modulator (SLM). Then through video-frame registration and a spatio-temporal spreading of the co-located spectral/RGB information, video with high spatial and spectral resolution is produced. The sampling patterns on the SLM are generated on-the-fly according to the scene content, which fully exploits the self-adaptivity of the hybrid camera system. With an experimental prototype, we demonstrate significantly improved accuracy and efficiency as compared to the state-of-the-art.

© 2014 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. W. M. Porter and H. T. Enmark, Proc. SPIE 834, 22 (1985).
  2. R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
    [CrossRef]
  3. N. Gat, Proc. SPIE 4056, 50 (2000).
  4. M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
    [CrossRef]
  5. W. R. Johnson, D. W. Wilson, and G. Bearman, Appl. Opt. 45, 1898 (2006).
    [CrossRef]
  6. N. A. Hagen and E. L. Dereniak, Appl. Opt. 47, F85 (2008).
    [CrossRef]
  7. W. R. Johnson, D. W. Wilson, and G. Bearman, Opt. Lett. 30, 1464 (2005).
    [CrossRef]
  8. R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
    [CrossRef]
  9. M. E. Gehm, R. John, R. Willett, T. Schultz, and D. Brady, Opt. Express 15, 14013 (2007).
    [CrossRef]
  10. A. Wagadarikar, R. John, R. Willett, and D. Brady, Appl. Opt. 47, B44 (2008).
    [CrossRef]
  11. A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, Opt. Express 17, 6368 (2009).
    [CrossRef]
  12. D. Kittle, D. Marks, and D. Brady, Opt. Eng. 51, 071403 (2012).
    [CrossRef]
  13. A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
    [CrossRef]
  14. X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).
  15. X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.
  16. University of Joensuu Color Group, http://www.uef.fi/fi/spectral/spectral-image-database (2005).
  17. C. Nvidia, Compute Unified Device Architecture Programming Guide (NVIDIA Corp., 2007).

2012 (2)

R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
[CrossRef]

D. Kittle, D. Marks, and D. Brady, Opt. Eng. 51, 071403 (2012).
[CrossRef]

2011 (1)

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

2009 (2)

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, Opt. Express 17, 6368 (2009).
[CrossRef]

2008 (2)

2007 (1)

2006 (2)

W. R. Johnson, D. W. Wilson, and G. Bearman, Appl. Opt. 45, 1898 (2006).
[CrossRef]

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

2005 (1)

2000 (1)

N. Gat, Proc. SPIE 4056, 50 (2000).

1995 (1)

R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
[CrossRef]

1985 (1)

W. M. Porter and H. T. Enmark, Proc. SPIE 834, 22 (1985).

Anderson, M. E.

R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
[CrossRef]

Basedow, R. W.

R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
[CrossRef]

Bearman, G.

Beaven, S.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Bodkin, A.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Brady, D.

Cao, X.

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.

Carmer, D. C.

R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
[CrossRef]

Dai, Q.

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.

Daly, J.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Dereniak, E. L.

Du, H.

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

Enmark, H. T.

W. M. Porter and H. T. Enmark, Proc. SPIE 834, 22 (1985).

Fukuda, H.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Gat, N.

N. Gat, Proc. SPIE 4056, 50 (2000).

Gehm, M. E.

Habel, R.

R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
[CrossRef]

Hagen, N. A.

Haneishi, H.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Iwama, R.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

John, R.

Johnson, W. R.

Kanazawa, H.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Kishimoto, J.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Kittle, D.

D. Kittle, D. Marks, and D. Brady, Opt. Eng. 51, 071403 (2012).
[CrossRef]

Kudenov, M.

R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
[CrossRef]

Lin, S.

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.

Marks, D.

D. Kittle, D. Marks, and D. Brady, Opt. Eng. 51, 071403 (2012).
[CrossRef]

Norton, A.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Nvidia, C.

C. Nvidia, Compute Unified Device Architecture Programming Guide (NVIDIA Corp., 2007).

Ohyama, N.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Pitsianis, N.

Porter, W. M.

W. M. Porter and H. T. Enmark, Proc. SPIE 834, 22 (1985).

Schultz, T.

Sheinis, A.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Sun, X.

Tong, X.

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.

Tsuchida, M.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Wagadarikar, A.

Weinheimer, J.

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Willett, R.

Wilson, D. W.

Wimmer, M.

R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
[CrossRef]

Yamaguchi, M.

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

Appl. Opt. (3)

Comput. Graph. Forum (1)

R. Habel, M. Kudenov, and M. Wimmer, Comput. Graph. Forum 31, 449 (2012).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, IEEE Trans. Pattern Anal. Mach. Intell. 33, 2423 (2011).

Opt. Eng. (1)

D. Kittle, D. Marks, and D. Brady, Opt. Eng. 51, 071403 (2012).
[CrossRef]

Opt. Express (2)

Opt. Lett. (1)

Proc. SPIE (5)

W. M. Porter and H. T. Enmark, Proc. SPIE 834, 22 (1985).

R. W. Basedow, D. C. Carmer, and M. E. Anderson, Proc. SPIE 2480, 258 (1995).
[CrossRef]

N. Gat, Proc. SPIE 4056, 50 (2000).

M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, Proc. SPIE, 6062, 60620G (2006).
[CrossRef]

A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, Proc. SPIE 7334, 73340H (2009).
[CrossRef]

Other (3)

X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI (2011), pp. 297–304.

University of Joensuu Color Group, http://www.uef.fi/fi/spectral/spectral-image-database (2005).

C. Nvidia, Compute Unified Device Architecture Programming Guide (NVIDIA Corp., 2007).

Supplementary Material (2)

» Media 1: MOV (4073 KB)     
» Media 2: MOV (2495 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1.
Fig. 1.

Overview of our proposed system: optical design and hyperspectral data reconstruction process. While capturing the high-spatial-resolution RGB video (left) and the low-spatial-resolution hyperspectral video (right), the high-resolution spectra video stream is reconstructed on-the-fly (Media 1).

Fig. 2.
Fig. 2.

Spectrum information spreading. In our proposed method, we distribute the captured spectrum information of the sampled points to others. By means of spreading, we are describing the information dissemination during this spectrum reconstruction.

Fig. 3.
Fig. 3.

Illustration of our iterative pattern-computing algorithm. First row: RGB video frames; second row: whiter pixels indicate higher uncertainty; third row: pixels with more intense color means higher weight, which leads to more dense sampling; the last row shows the dynamically changing SLM pattern, in which the white areas allow light passing and represent spectral samples (Media 1).

Fig. 4.
Fig. 4.

(a) Accuracy comparison. Accuracy improvement at the beginning is because our content-adaptive pattern can sample scene points containing more information. Improvements in the following frames also benefit from our temporal multiplexing spectrum-reconstruction algorithm. (b) Targeted spectral acquisition by annotating ROI.

Fig. 5.
Fig. 5.

Our prototype and hyperspectral capture results with the proposed system (Media 2).

Equations (3)

Equations on this page are rendered with MathJax. Learn more.

sijn=cr,g,bkΩGσr(dkr)Gσs(dks)Gσt(dkt)ρkc(wcsk)kΩGσr(dkr)Gσs(dks)Gσt(dkt),
weightijn=kΩ¯cr,g,bGσr(dkr)Gσs(dks)Gσt(dkt)ρkcukn,
uij(n+1)=1cr,g,bkΩGσr(dkr)Gσs(dks)Gσt(dkt)ρkc·μkkΩGσr(dkr)Gσs(dks)Gσt(dkt),

Metrics