Abstract

Hybrid-resolution multispectral imaging is a framework to acquire multispectral images through a reconstruction procedure using two types of measurement data with different spatial and spectral resolutions. In this paper, we propose a new method for such a framework on the basis of a full-resolution RGB image and the data obtained from an image sensor with a multispectral filter array (MSFA). In the proposed method, a small region of each image band is reconstructed as a linear combination of RGB images, where the weighting coefficients are determined using MSFA data. The effectiveness of the proposed approach is shown by simulations using spectral images of natural scenes.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
    [CrossRef]
  2. B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE 3963, 3–14 (2000).
  3. H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39(35), 6621–6632 (2000).
    [CrossRef] [PubMed]
  4. J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
    [CrossRef]
  5. M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
    [CrossRef]
  6. L. Gao, R. T. Kester, N. Hagen, and T. S. Tkaczyk, “Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express 18(14), 14330–14344 (2010).
    [CrossRef] [PubMed]
  7. R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
    [CrossRef]
  8. J. Brauers and T. Aach, “A color filter array based multispectral camera,” presented at the 12. Workshop Farbbildverarbeitung, Ilmenau, Germany, 5–6 Oct. 2006.
  9. Y. Monno, M. Tanaka, and O. Masatoshi, “Multispectral demosaicking using adaptive kernel upsampling,” in Proceedings of 18th IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, 2011), 3218–3221.
  10. F. H. Imai and R. S. Berns, “High-resolution multi-spectral image archives: A hybrid approach,” in Proceedings of 6th Color Imaging Conference, (Society for Imaging Science and Technology, 1998), 224–227.
  11. Y. Murakami, K. Ietomi, M. Yamaguchi, and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. 46(28), 7068–7082 (2007).
    [CrossRef] [PubMed]
  12. Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. 48(11), 2188–2202 (2009).
    [CrossRef] [PubMed]
  13. O. Kohonen, “Multiresolution-based pansharpening in spectral color images,” in Proceedings of 5th European Conference on Colour in Graphics, Imaging, and Vision, (Society for Imaging Science and Technology, 2010), 535–540.
  14. Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011).
    [CrossRef]
  15. R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Institute of Electrical and Electronics Engineers, 2011), 2329–2336.
  16. D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
    [CrossRef] [PubMed]
  17. W. K. Pratt and C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. 15(1), 73–75 (1976).
    [CrossRef] [PubMed]

2011 (2)

R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
[CrossRef]

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011).
[CrossRef]

2010 (1)

2009 (1)

2008 (1)

M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
[CrossRef]

2007 (1)

2004 (1)

D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
[CrossRef] [PubMed]

2002 (1)

J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
[CrossRef]

2000 (2)

1999 (1)

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

1976 (1)

Amano, K.

D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
[CrossRef] [PubMed]

Brettel, H.

J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
[CrossRef]

Foster, D. H.

D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
[CrossRef] [PubMed]

Gao, L.

Hagen, N.

Haneishi, H.

Hardeberg, J. Y.

R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
[CrossRef]

J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
[CrossRef]

Hasegawa, T.

Hauta-Kasari, M.

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

Hideaki, H.

M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
[CrossRef]

Hill, B.

B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE 3963, 3–14 (2000).

Hosoi, A.

Ietomi, K.

Kester, R. T.

Khan, R.

R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
[CrossRef]

Mancill, C. E.

Miyake, Y.

Miyazawa, K.

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

Murakami, Y.

Nascimento, S. M. C.

D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
[CrossRef] [PubMed]

Ohyama, N.

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011).
[CrossRef]

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. 48(11), 2188–2202 (2009).
[CrossRef] [PubMed]

M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
[CrossRef]

Y. Murakami, K. Ietomi, M. Yamaguchi, and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. 46(28), 7068–7082 (2007).
[CrossRef] [PubMed]

Parkkinen, J.

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

Pratt, W. K.

Schmitt, F.

J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
[CrossRef]

Shrestha, R.

R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
[CrossRef]

Tkaczyk, T. S.

Toyooka, S.

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

Tsumura, N.

Yamaguchi, M.

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011).
[CrossRef]

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. 48(11), 2188–2202 (2009).
[CrossRef] [PubMed]

M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
[CrossRef]

Y. Murakami, K. Ietomi, M. Yamaguchi, and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. 46(28), 7068–7082 (2007).
[CrossRef] [PubMed]

Yokoyama, Y.

Appl. Opt. (4)

J. Imaging Sci. Technol. (1)

M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008).
[CrossRef]

J. Opt. Soc. Am. (1)

M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999).
[CrossRef]

J. Opt. Soc. Am. A. (1)

Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011).
[CrossRef]

Opt. Eng. (1)

J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002).
[CrossRef]

Opt. Express (1)

Proc. SPIE (2)

B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE 3963, 3–14 (2000).

R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011).
[CrossRef]

Vis. Neurosci. (1)

D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004).
[CrossRef] [PubMed]

Other (5)

J. Brauers and T. Aach, “A color filter array based multispectral camera,” presented at the 12. Workshop Farbbildverarbeitung, Ilmenau, Germany, 5–6 Oct. 2006.

Y. Monno, M. Tanaka, and O. Masatoshi, “Multispectral demosaicking using adaptive kernel upsampling,” in Proceedings of 18th IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, 2011), 3218–3221.

F. H. Imai and R. S. Berns, “High-resolution multi-spectral image archives: A hybrid approach,” in Proceedings of 6th Color Imaging Conference, (Society for Imaging Science and Technology, 1998), 224–227.

O. Kohonen, “Multiresolution-based pansharpening in spectral color images,” in Proceedings of 5th European Conference on Colour in Graphics, Imaging, and Vision, (Society for Imaging Science and Technology, 2010), 535–540.

R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Institute of Electrical and Electronics Engineers, 2011), 2329–2336.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1
Fig. 1

Conceptual diagram of hybrid-resolution multispectral imaging using multispectral filter array (MSFA).

Fig. 2
Fig. 2

An example of a configuration for a one-shot camera for hybrid-resolution multispectral imaging.

Fig. 3
Fig. 3

Hyperspectral images used in the simulations (presented as color images).

Fig. 4
Fig. 4

Configurations of 16-band imaging systems compared in the simulations.

Fig. 5
Fig. 5

Spectral sensitivities of R, G, and B color filters (left), 16 narrow-band color filters (middle), and four color filters for RG1G2B (right) used in simulation.

Fig. 6
Fig. 6

Simulation results: NRMSE of 16-band images.

Fig. 7
Fig. 7

Enlarged 70 × 70-pixel region of 16-band image of Scene3. For this presentation, three-band images were allotted to RGB channels. The corresponding band numbers are indicated at the top.

Fig. 8
Fig. 8

Enlarged 70 × 70-pixel region of 16-band image of Scene6. For this presentation, three-band images were allotted to RGB channels. The corresponding band numbers are indicated at the top.

Fig. 9
Fig. 9

Correspondence between principal components derived from original and reconstructed 16-band images. Ideally, matrix becomes an identity matrix.

Equations (21)

Equations on this page are rendered with MathJax. Learn more.

R = ( r 1 T r N T ) .
f i = H M S T r i ,
F = R H M S ,
F = ( f 1 T f N T ) .
F = ( f ˜ 1 f ˜ K ) ,
c k = S k f ˜ k ,
G = ( g R , g G , g B ) = R H R G B ,
r i q = 1 3 b i q u q j ,
[ R ] j [ B ] j U j ,
U j = ( u 1 j T u 2 j T u 3 j T ) .
[ F ] j = [ R ] j H M S ,
[ G ] j = [ R ] j H R G B .
[ F ] j [ B ] j U j H M S ,
[ G ] j [ B ] j U j H R G B .
[ B ] j [ G ] j ( U j H R G B ) 1 .
[ F ] j [ G ] j ( U j H R G B ) 1 U j H M S = [ G ] j A j ,
[ f ˜ k ] j [ G ] j a k j ,
[ c k ] j = [ S k f ˜ k ] j .
[ c k ] j [ S k G ] j a j k = D k j a j k .
a ^ k j = [ D k j T D k j ] 1 D k j T [ c k ] j .
[ f ˜ ^ k ] j = [ G ] j a ^ k j .

Metrics