Abstract

Hyperspectral video acquisition is a trade-off between spectral and temporal resolution. We present an algorithm for recovering dense hyperspectral video of dynamic scenes from a few measured multispectral bands per frame using optical flow and sparse coding. Different set of bands are measured in each video frame and optical flow is used to register them. Optical flow errors are corrected by exploiting sparsity in the spectra and the spatial correlation between images of a scene at different wavelengths. A redundant dictionary of atoms is learned that can sparsely approximate training spectra. The restoration of correct spectra is formulated as an 1 convex optimization problem that minimizes a Mahalanobis-like weighted distance between the restored and corrupt signals as well as the restored signal and the median of the eight connected neighbours of the corrupt signal such that the restored signal is a sparse linear combination of the dictionary atoms. Spectral restoration is followed by spatial restoration using a guided dictionary approach where one dictionary is learned for measured bands and another for a band that is to be spatially restored. By constraining the sparse coding coefficients of both dictionaries to be the same, the restoration of corrupt band is guided by the more reliable measured bands. Experiments on real data and comparison with an existing volumetric image denoising technique shows the superiority of our algorithm.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframe image estimation for coded aperture snapshot spectral imagers,” Appl. Opt. 49, 6824–6833 (2010).
    [CrossRef] [PubMed]
  2. M. Shankar, N. Pitsianis, and D. Brady, “Compressive video sensors using multichannel imagers,” Appl. Opt. 49, B9–B17 (2010).
    [CrossRef] [PubMed]
  3. A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady, “Video rate spectral imaging using a coded aperture snapshot spectral imager,” Opt. Express 17, 6368–6388 (2009).
    [CrossRef] [PubMed]
  4. Y. Pati, R. Rexaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition,” in Proceedings of the 27th Asilomar Conference on Signals, Systems, and Computers (IEEE, 1993), 40–44.
    [CrossRef]
  5. R. Tibshirani, “Regression shrinkage and selection via the Lasso,” J. R. Stat. Soc. Ser. B 58, 267–288 (1996).
  6. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
    [CrossRef]
  7. M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process. 15, 3736–3745 (2006).
    [CrossRef] [PubMed]
  8. M. Elad and G. Sapiro, “Sparse representation for color image restoration,” IEEE Trans. Image Process. 17, 53–69 (2008).
    [CrossRef] [PubMed]
  9. H. Othman and S. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Trans. Geosci. Remote Sens. 44, 397–408 (2006).
    [CrossRef]
  10. S. Bourguignon, D. Mary, and E. Slezak, “Sparsity-based denoising of hyperspectral astrophysical data with colored noise: Application to the MUSE instrument,” in 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE, 2010), 1–4.
    [CrossRef]
  11. G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Proceedings of the 13th Scandinavian Conference on Image Analysis (Springer, 2003), 363–370.
  12. J. Parkkinen, J. Hallikainen, and T. Jaaskelainen, “Characteristic spectra of munsell colors,” J. Opt. Soc. Am. A 6, 318–322 (1989).
    [CrossRef]
  13. J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).
  14. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
    [CrossRef]
  15. J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
    [CrossRef]
  16. P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

2011

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

2010

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).

D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframe image estimation for coded aperture snapshot spectral imagers,” Appl. Opt. 49, 6824–6833 (2010).
[CrossRef] [PubMed]

M. Shankar, N. Pitsianis, and D. Brady, “Compressive video sensors using multichannel imagers,” Appl. Opt. 49, B9–B17 (2010).
[CrossRef] [PubMed]

2009

2008

M. Elad and G. Sapiro, “Sparse representation for color image restoration,” IEEE Trans. Image Process. 17, 53–69 (2008).
[CrossRef] [PubMed]

2006

H. Othman and S. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Trans. Geosci. Remote Sens. 44, 397–408 (2006).
[CrossRef]

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
[CrossRef]

M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process. 15, 3736–3745 (2006).
[CrossRef] [PubMed]

2004

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

1996

R. Tibshirani, “Regression shrinkage and selection via the Lasso,” J. R. Stat. Soc. Ser. B 58, 267–288 (1996).

1989

Aharon, M.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
[CrossRef]

M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process. 15, 3736–3745 (2006).
[CrossRef] [PubMed]

Bourguignon, S.

S. Bourguignon, D. Mary, and E. Slezak, “Sparsity-based denoising of hyperspectral astrophysical data with colored noise: Application to the MUSE instrument,” in 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE, 2010), 1–4.
[CrossRef]

Brady, D.

Bruckstein, A.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
[CrossRef]

Choi, K.

Efron, B.

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

Elad, M.

M. Elad and G. Sapiro, “Sparse representation for color image restoration,” IEEE Trans. Image Process. 17, 53–69 (2008).
[CrossRef] [PubMed]

M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process. 15, 3736–3745 (2006).
[CrossRef] [PubMed]

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
[CrossRef]

Farnebäck, G.

G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Proceedings of the 13th Scandinavian Conference on Image Analysis (Springer, 2003), 363–370.

Hallikainen, J.

Hastie, T.

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

Huang, T.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

Jaaskelainen, T.

Johnstone, I.

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

Kikuchi, H.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

Kittle, D.

Krishnaprasad, P.

Y. Pati, R. Rexaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition,” in Proceedings of the 27th Asilomar Conference on Signals, Systems, and Computers (IEEE, 1993), 40–44.
[CrossRef]

Ma, Y.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

Mairal, J.

J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).

Mary, D.

S. Bourguignon, D. Mary, and E. Slezak, “Sparsity-based denoising of hyperspectral astrophysical data with colored noise: Application to the MUSE instrument,” in 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE, 2010), 1–4.
[CrossRef]

Muramatsu, S.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

Ndajah, P.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

Othman, H.

H. Othman and S. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Trans. Geosci. Remote Sens. 44, 397–408 (2006).
[CrossRef]

Parkkinen, J.

Pati, Y.

Y. Pati, R. Rexaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition,” in Proceedings of the 27th Asilomar Conference on Signals, Systems, and Computers (IEEE, 1993), 40–44.
[CrossRef]

Pitsianis, N.

Ponce, J.

J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).

Qian, S.

H. Othman and S. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Trans. Geosci. Remote Sens. 44, 397–408 (2006).
[CrossRef]

Rexaiifar, R.

Y. Pati, R. Rexaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition,” in Proceedings of the 27th Asilomar Conference on Signals, Systems, and Computers (IEEE, 1993), 40–44.
[CrossRef]

Sapiro, G.

J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).

M. Elad and G. Sapiro, “Sparse representation for color image restoration,” IEEE Trans. Image Process. 17, 53–69 (2008).
[CrossRef] [PubMed]

Shankar, M.

Slezak, E.

S. Bourguignon, D. Mary, and E. Slezak, “Sparsity-based denoising of hyperspectral astrophysical data with colored noise: Application to the MUSE instrument,” in 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE, 2010), 1–4.
[CrossRef]

Sun, X.

Tibshirani, R.

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

R. Tibshirani, “Regression shrinkage and selection via the Lasso,” J. R. Stat. Soc. Ser. B 58, 267–288 (1996).

Wagadarikar, A.

Watanabe, H.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

Wright, J.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

Yang, J.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

Yukawa, M.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

Ann. Stat.

B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” Ann. Stat. 32, 407–499 (2004).
[CrossRef]

Appl. Opt.

IEEE Trans. Geosci. Remote Sens.

H. Othman and S. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Trans. Geosci. Remote Sens. 44, 397–408 (2006).
[CrossRef]

IEEE Trans. Image Process.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861–2873 (2010).
[CrossRef]

M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process. 15, 3736–3745 (2006).
[CrossRef] [PubMed]

M. Elad and G. Sapiro, “Sparse representation for color image restoration,” IEEE Trans. Image Process. 17, 53–69 (2008).
[CrossRef] [PubMed]

IEEE Trans. Signal Process.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006).
[CrossRef]

Int. J. Circuits, Systems and Signal Process.

P. Ndajah, H. Kikuchi, M. Yukawa, H. Watanabe, and S. Muramatsu, “An investigation on the quality of denoised images,” Int. J. Circuits, Systems and Signal Process. 5, 423–434 (2011).

J. Mach. Learn. Res.

J. Mairal, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res. 11, 19–60 (2010).

J. Opt. Soc. Am. A

J. R. Stat. Soc. Ser. B

R. Tibshirani, “Regression shrinkage and selection via the Lasso,” J. R. Stat. Soc. Ser. B 58, 267–288 (1996).

Opt. Express

Other

Y. Pati, R. Rexaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition,” in Proceedings of the 27th Asilomar Conference on Signals, Systems, and Computers (IEEE, 1993), 40–44.
[CrossRef]

S. Bourguignon, D. Mary, and E. Slezak, “Sparsity-based denoising of hyperspectral astrophysical data with colored noise: Application to the MUSE instrument,” in 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE, 2010), 1–4.
[CrossRef]

G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in Proceedings of the 13th Scandinavian Conference on Image Analysis (Springer, 2003), 363–370.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1
Fig. 1

Sample bands at 690, 650, 620, 610, 600, 590, 580, 540, 520, 510, 500, 490, 480 and 440 nm of a hyperspectral image cube. Each band is rendered as it would be seen by a human eye.

Fig. 2
Fig. 2

An illustration of hyperspectral video. There are three hyperspectral frames each with five bands.

Fig. 3
Fig. 3

(a) RGB image of the scene in Fig. 1. (b) Five bands (60nm apart) are sensed in each frame with a between-frame offset of 10nm. Six consecutive frames cover 30 bands (430–720nm) for example the first frame (row) comprises 430, 490, 550, 610, 670nm bands.

Fig. 4
Fig. 4

Third band (rendered as gray scale images) from six consecutive frames of a dynamic scene with static background and a moving block in the foreground. Notice the varying texture which makes it challenging to calculate optical flow between frames/bands.

Fig. 5
Fig. 5

Optical flow in the horizontal direction for the scene in Fig. 4 represented as 3D plots. The x,y directions correspond to the image dimensions and the vertical direction corresponds to horizontal displacement between frames. The frame bands used to calculate the optical flow are written under each plot. The bottom right plot shows optical flow calculated between five band spatio-spectral images of consecutive hyperspectral frames.

Fig. 6
Fig. 6

(a) Ordering of bands in a spatio-spectral image. Each set of 3 × 3 pixels is formed by ordering the corresponding pixels of the 5 bands. The corner pixels are interpolated from the nearest pixels in the same set excluding the center pixel. (b) A scene patch at 550nm rendered as a gray scale image. (c) A spatio-spectral image of the same patch constructed from 5 bands i.e. 430, 490, 550, 610 and 670nm.

Fig. 7
Fig. 7

A 540nm band (left) is sequentially registered from frame 6 to 1 using inter-band optical flow (center) and spatio-spectral image-based optical flow (right).

Fig. 8
Fig. 8

Optical flow errors lead to incorrect spectral reflectance curves at many pixels (see above examples). Using the proposed technique, the correct spectral reflectance can be recovered.

Fig. 9
Fig. 9

Left: A 550nm band registered sequentially from frame 1 to frame 6. Center: The errors propagated from optical flow are corrected by spectral restoration. Right: Ground truth 550nm band acquired with frame 6.

Fig. 10
Fig. 10

Spectral restoration of a 490nm band sequentially registered from five frames apart. Some errors can be noticed around the boundaries of the moving blocks which are removed by the spatial restoration.

Fig. 11
Fig. 11

Magnified views of the (left most middle part of the) three images in Fig. 10. After spectral+spatial restoration (middle image), the boundaries are better recovered and the image has more resemblance to the ground truth (right image).

Fig. 12
Fig. 12

(a) Hyperspectral camera setup. (b) Transmittance of the LCTF. (c) Quantum efficiency of the camera CCD. (d) Spectral curve of the halogen light.

Fig. 13
Fig. 13

Comparison with K-SVD denoising. A 560nm band, registered from 5 frames apart is restored with (a) spectral restoration, (b) spectral + spatial restoration, (d) K-SVD volumetric denoising, (e) spectral restoration + KSVD volumetric denoising and (f) K-SVD volumetric denoising + spectral restoration. Measured ground truth is in (c).

Tables (3)

Tables Icon

Table 1 RMSE of restored hyperspectral frames from measured ground truth. Frames registered with optic flow give better restoration accuracy compared to cubic interpolation.

Tables Icon

Table 2 RMSE of recovered hyperspectral frames w.r.t. the number of iterations.

Tables Icon

Table 3 Comparison with the volumetric K-SVD algorithm under different configurations. λ: proposed spectral restoration only, λ +G: proposed spectral+spatial restoration, KSVD: K-SVD volumetric denoising [6]. The overall best performance is achieved by the proposed spectral+spatial restoration λ +G.

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

x^Dα^where
α^=minαα0s.t.Dαx22ɛ,
oralternativelyα^=argminα{Dαx22+γα0}.
α^=argminα{Dαx22+γ1α1},
{A^,X^}=argminA,X{ijDαijRijX22+ijγijαij0+γ2XY22}.
K=(Jp000Jp000Jp),whereJpisap×pmatrixofones.
(δx,δy)=minx,yI(x,y)I(x+δx,y+δy)22.
H^ij=Dλα^ij
α^ij=argminαij{Dλαijsij22+γ1αij1},
α^ij=argminαij{W(Dλαijsij)22+γ1αij1},
s˜ij=(1γ2)sij+γ2s¯ij,
α^ij=argminαij{W(Dλαijs˜ij)22+γ1αij1},
{β^ij,D^sm,D^se}=argminβij,Dsm,Dse{DsmβijRijHm22+γ4DseβijRijHe22+γ3βij1},
{β^ij,D^s}=argminβij,Ds{DsβijRijH22+γ3βij1}.
RMSE=1uvni=1uj=1vk=1n(HijkrHijkg)2,

Metrics