## Abstract

Hybrid-resolution multispectral imaging is a framework to acquire multispectral images through a reconstruction procedure using two types of measurement data with different spatial and spectral resolutions. In this paper, we propose a new method for such a framework on the basis of a full-resolution RGB image and the data obtained from an image sensor with a multispectral filter array (MSFA). In the proposed method, a small region of each image band is reconstructed as a linear combination of RGB images, where the weighting coefficients are determined using MSFA data. The effectiveness of the proposed approach is shown by simulations using spectral images of natural scenes.

©2012 Optical Society of America

## 1. Introduction

Multispectral imaging techniques are utilized as an effective tool to measure and analyze targets noninvasively. In addition, the utilization of multispectral images has been expected in the field of color image reproduction with high fidelity [1–5]. However, multispectral image acquisition requires bulky imaging systems or time-consuming processes that include data scanning along either the wavelength or spatial axis. As a result, the applications are limited to several typical fields, such as remote sensing and microscopic imaging. Recently, snapshot spectral imaging techniques have been developed, which directly acquire a spectral image in a single exposure (e.g., see [6]). However, thus far, the image quality is insufficient compared to that obtained by a conventional scanning-type imaging system. To utilize multispectral imaging techniques in wider applications, compact and one-shot imaging instruments that realize high accuracy are strongly anticipated.

Image sensors based on color filter arrays (CFAs) are widely used to capture color images, and interpolation techniques for each type of CFA are vigorously investigated. Recently, multispectral image acquisition techniques using the multispectral filter array (MSFA) were proposed as an extension of the CFA-based method [7–9]. However, such an approach is inappropriate when the number of spectral bands is large (e.g., more than ten bands), because the spatial sampling becomes too sparse to reconstruct spectral images by interpolation-based methods.

On the other hand, a hybrid-resolution spectral imaging scheme has been proposed as a new type of spectral imaging technique [10–15]. In this scheme, spectral images are estimated from two types of data with different spatial and spectral resolutions: for instance, high-resolution RGB images and corresponding low-resolution spectral data. Although the data obtained by an MSFA-type image sensor can be regarded as a kind of low-resolution spectral data, a hybrid-resolution approach has not yet been applied.

This paper proposes a new scheme for hybrid-resolution multispectral imaging based on high-resolution RGB images and the data obtained by an MSFA-type image sensor with narrow-band color filters. Because the previously proposed methods presuppose low-resolution spectral data measured at the same position for all bands, this paper presents a new reconstruction method that can be applied to the data of an MSFA.

Subsection 2.1 first introduces the data acquisition model of the proposed hybrid-resolution multispectral imaging, and Subsection 2.2 then presents a low-dimensional model of multispectral images. On the basis of the low-dimensional model, Subsection 2.3 proposes the reconstruction method for multispectral images. Section 3 shows the simulation results, and Section 4 gives the conclusions.

## 2. Method

This paper proposes a hybrid-resolution multispectral imaging, which produces a *K*-band multispectral image from a set of RGB images and the data of an MSFA with narrow-band filters of *K* colors (Fig. 1
). These data can be obtained in one shot, if a camera is configured with four image sensors (three sensors for RGB and one for an MSFA) accompanied with a four-channel separation prism, as shown in Fig. 2
. The data acquisition model and reconstruction method for multispectral images are presented below.

#### 2.1 Data acquisition model

Let us first introduce the data acquisition model, which is assumed in the proposed method. For simplicity, the original spectral image is treated below as discrete signals.

Let **r*** _{i}* be an

*L*-dimensional column vector representing the spectral reflectance function of the original spectral reflectance image at pixel

*i*, where

*L*is the number of spectral samples and $1\le i\le {N}_{x}\times {N}_{y}=N$, with

*N*the number of pixels. The original spectral reflectance image is represented by an $N\times L$ matrix,

First, the ideal *K*-band multispectral image is defined. The *K*-band multispectral data corresponding to **r*** _{i}* is represented by a

*K*-dimensional column vector,

**H**

*is an $L\times K$ matrix that represents the spectral sensitivity of the multispectral imaging system. The whole multispectral image is represented byWhere*

_{MS}If we define ${\tilde{f}}_{k}$as the *k*th *N*-dimensional column vector of **F**, another representation of **F** is

*k*th image band of the multispectral image. Note that the

*K*-band multispectral image defined by Eqs. (2)-(5) is not measurement data of the proposed method, but for an ideal one.

Let us now introduce the representation of the data of an MSFA. The data obtained at the pixels assigned to *k*th band color filters is

**c**

*is an*

_{k}*M*-dimensional column vector,

_{k}**S**

*is an ${M}_{k}\times N$ matrix that represents sampling pixels for*

_{k}*k*th band color filters, and

*M*is the number of pixels assigned to

_{k}*k*th band color filters.

Next is a model for RGB image capturing. Color imaging devices can be modeled as linear systems if the nonlinearity of the system is adequately corrected. Hence, the three-band RGB image is represented by

where**G**is a $N\times 3$ matrix, each

**g**is a one-dimensional representation of a channel of the RGB image, and

**H**

*is an $L\times 3$ system matrix comprising the spectral characteristics of the camera and the illumination spectrum.*

_{RGB}From {**c**_{1},…,**c*** _{K}*} and

**G**, we want to reconstruct the ideal

*K*-band image

**F**.

#### 2.2 Localized low-dimensional model for multispectral images

Before presenting the reconstruction method, let us introduce a localized low-dimensional model for multispectral images to serve as the basis of the proposed reconstruction method.

A widely used approach is the representation of spectral reflectance functions by a linear combination of a relatively small number of basis functions. The required number of basis functions depends on the target data and the accuracy of the representation. If we consider the spectral reflectance functions included by a relatively small region of a spectral image, the variation of those spectral reflectance functions is limited. As a result, it is expected that only a small number of basis functions should be needed to represent the spectral reflectance functions in that region.

Based on this assumption, we derive a model for multispectral images in relation to RGB images. The model is a kind of low-dimensional approximation of multispectral images, which works within localized regions. Now we select a region, indexed by *j*, in which any spectrum can be approximately represented by a linear combination of three *L*-dimensional spectral basis vectors ${u}_{q}^{j}$ as

*b*is a weighting coefficient. The representation for the whole

_{iq}*j*th region by Eq. (8) is written aswhere []

*is an operator to select the*

_{j}*j*th region from a whole image,

**B**is an$N\times 3$ matrix that consists of elements

*b*, and

_{ij}Using the operator []* _{j}*, Eqs. (3) and (7) are rewritten as

Substituting Eq. (9) into Eqs. (11) and (12), we have

Except for the cases in which the 3 × 3 matrix **UH*** _{RGB}* is singular,

Substituting Eq. (15) into Eq. (13),

**A**

*= (*

_{j}**U**

_{j}**H**

*)*

_{RGB}^{−1}

**U**

_{j}**H**

*. By taking the*

_{MS}*k*th column vectors from [

**F**]

*and*

_{j}**A**

*, we havewhere*

_{j}**a**

*is the*

_{kj}*k*th column vector of

**A**

*.*

_{j}Equation (17) indicates that any image band of a multispectral image can be approximately represented by a linear combination of the corresponding region of the RGB images, if the region is selected to satisfy both Eq. (8) and the nonsingularity condition on **UH*** _{RGB}*.

#### 2.3 Multispectral image reconstruction method

In the proposed method, multispectral images are divided into regions, each of which satisfies Eq. (8) as far as possible. Then, the multispectral images are reconstructed independently for every region by using the MSFA observation. The reconstruction is based on the model of Eq. (17), where the three-dimensional vector **a*** _{kj}* is derived for each region and each band.

By referring to Eq. (6), the MSFA data for this region is represented by

If *m _{kj}* is the number of the pixels assigned to the

*k*th color filter included in this region, [

**c**

*]*

_{k}*is an m*

_{j}*-dimensional column vector. By substituting Eq. (17) and setting ${D}_{kj}={\left[{S}_{k}G\right]}_{j}$, we have*

_{kj}Because we have both [**c*** _{k}*]

*and*

_{j}**D**

*as measurement data, the weighting coefficients*

_{kj}**a**

*can be derived by minimizing the error between the sides of Eq. (19). If a simple linear regression analysis is applied, we have*

_{kj}Then, the *k*th image band of this region is estimated as

By repeating the calculation in Eq. (21) for all bands and all regions, the whole multispectral image is reconstructed.

Noise was ignored in the observation model represented by Eqs. (6) and (7). In actual, there should be random noise on the measurement data in real imaging systems. If noise is nonnegligible, some regularization methods should be introduced into the regression to derive the weighting coefficients. With an appropriate regularization method, the proposed method works properly, which will be presented in another paper. In addition, there may be the error caused by a misalignment among four image sensors. This kind of error should be removed before reconstructing multispectral images because the assumption of Eq. (17) does not hold with this error. The observation of a test target can be used to remove the misalignment.

To end this subsection, let us discuss how to determine the regions described above. The smaller the size of a region, the higher the probability that the region includes only spectra with similar features. As a result, the condition in Eq. (8) is easily satisfied. However, as the size becomes smaller (i.e., as *m _{kj}* becomes smaller), the regression to derive Eq. (20) becomes unstable. Therefore, these opposite influences should be considered in deciding the size of the regions. Regardless of the size, there are several ways to divide images into regions. A simple method is that an image be divided horizontally and vertically into small blocks. Because neighboring pixels in multispectral images are more likely to have similar spectral characteristics, simple block division can be effective. Another method is that the pixels with similar RGB values be collected into a single region.

## 3. Simulation results

Three hyperspectral images described by Foster et al. [16] were used for simulations. Because each band of the original hyperspectral images was captured in a time sequence, there are slight misalignments between image bands, especially in the regions of fluttering leaves. By eliminating the effect of these misalignments in the simulations, parts of the original images were cut out and resized. The resized images are shown in Fig. 3 as color images. The image of Scene3 consists of $256\times 256$ pixels, sampled every 10 nm from 400 nm to 720 nm (33 dimensions). The image of Scene5 consists of $480\times 320$ pixels, sampled every 10 nm from 400 nm to 710 nm (32 dimensions). The image of Scene6 consists of $480\times 320$ pixels, sampled every 10 nm from 400 nm to 720 nm (33 dimensions). For Scene 5, data for 720 nm were originally excluded, because the signal at this wavelength was excessively noisy.

In the simulations, data were assumed to have been captured by four image sensors with the same number of pixels as the hyperspectral images. Then, 16-band images with the same numbers of pixels were reconstructed. Three systems, including the proposed approach, were compared (Fig. 4 ).

For the proposed hybrid-resolution approach, three image sensors were assigned to RGB, and the remaining one 16-band MSFA-type image sensor (RGB + 16FA) was assigned. The spectral sensitivity of the RGB channels was assumed to be that of typical HDTV video cameras. The spectral sensitivity functions of the 16 narrow-band color filters were those obtained by approximately dividing the wavelength range spanned by the RGB sensitivities into 16 equally. The spectral sensitivities of them are shown in Fig. 5 (left and middle panels). The image was divided horizontally and vertically into 16 × 16-pixel regions, and the reconstruction in Eqs. (20) and (21) was applied to each region. The size of the regions was approximately selected at the smallest possible under the condition that the regression be performed stably.

For comparison, two different types of imaging system were simulated. One of these was a 4-band imaging system called RG1G2B. The four image sensors were assigned to R, short-pass G, high-pass G, and B channels. The spectral sensitivities of the four color filters are shown in the right panel of Fig. 5. From the 4-band image data, 16-band data were reconstructed pixel by pixel through Wiener estimation with the spectral correlation matrix formulated on the basis of a Markov process [17]. In another system, called 4FA × 4, every image sensor was a four-band MSFA-type image sensor, but with different color filters; i.e., the first MSFA image sensor was assigned to the color filters of bands 1–4, the second was assigned to the color filters of bands 5–8, and so on. Since each image band was sampled every four pixels, each image band was independently reconstructed by linear interpolation in the spatial domain.

For reference, ideal 16-band images were generated using the spectral sensitivities of the 16 narrow-band color filters of the MSFA. These are called the original 16-band images below.

Figure 6 shows the normalized root-mean-square error (NRMSE) for each image band of the 16-band images reconstructed by the three methods. The error of 4FA × 4 is generally higher than others across the bands, while the error of the proposed system is generally lower. The error of RG1G2B is high in several specific image bands. We can see the effectiveness of the proposed method from these results.

To reveal the spatial features of the reconstructed 16-band images, enlarged 70 × 70-pixel regions of Scene3 and Scene6 are shown in Figs. 7
and 8
, respectively. Three-band images selected from a 16-band image were allotted to RGB channels and rendered as color images. The selected band numbers are indicated at the top of each figure. In the images by 4FA × 4 we can see apparent spatial degradation, including blur and color artifacts. Conversely, in the images by RGB + 16FA (proposed approach) and RG1G2B there is no such spatial degradation. A closer look at the images by RG1G2B shows a slight color change in the images of (11,6,1) and (15,10,5) in Fig. 7, which indicates the existence of error in the corresponding spectral components. As for RGB + 16FA, a distinct error in the form of a spurious magenta line arises in the image of (12,7,2) in Fig. 8. We detect such errors only in three regions, including this one in the image of Scene6, which is expected to be improved by introducing the regularization techniques into the deriving of **a*** _{jk}*. Except for this kind of error, there is no perceived error in the reconstruction of images by the proposed RGB + 16FA.

To see the spectral features contained in the reconstructed 16-band images, principal components of 16-band images were evaluated. The principal components of the original 16-band image, **V** = (**v**_{1},…,**v**_{16}), can be derived as the eigenvectors of **FF*** ^{T}*. These are sixteen 16-dimensional vectors ordered by the magnitudes of the corresponding eigenvalues. In a similar way, the principal components of the 16-band images reconstructed by RGB + 16FA, RG1G2B, and 4FA × 4 were derived; i.e.,

**V**

_{RGB + 16FA},

**V**

_{RG1G2B}, and

**V**

_{4FA × 4}, respectively. Figure 9 shows

**V**

^{T}**V**

_{RGB + 16FA},

**V**

^{T}**V**

_{RG1G2B}, and

**V**

^{T}**V**

_{4FAx4}in matrix form, where the range of 0–1 is shown in 8-bit grayscale. Because of the orthonormality of principal components, the matrix becomes an identity matrix when the principal components derived from the reconstructed 16-band image are equivalent to

**V**.

In the case of RG1G2B, we can derive only four principal components, as the information on principal components with orders higher than fourth is completely lost. As for 4FA × 4, we cannot see any correspondence between two sets of principal components at orders higher than third. Conversely, for RGB + 16FA the power is concentrated along the diagonal, which indicates that the spectral features are well reconstructed.

## 4. Conclusions

In this paper, we proposed a new scheme for hybrid-resolution multispectral imaging using the data of an MSFA-type image sensor as low-resolution spectral data. By reconstructing a multispectral image as a linear combination of RGB images, the degradation in spatial resolution can be suppressed despite the low sampling ratio of each image band. In addition, since the weighting coefficients for the linear combination are derived for each small region, it was confirmed that spectral features of more than three dimensions can be reconstructed. For further improvement, the following topics will be investigated in the future: a selection method for the region in the reconstruction, a more appropriate or more advanced method of regression to derive the weighting coefficients, including regularization techniques, and verification of the localized low-dimensional model for multispectral images.

## Acknowledgments

This work was supported by a kakenhi Grant-in-Aid for Scientific Research (20-40108 and 23135509) from Japan Society for the Promotion of Science.

## References and links

**1. **M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. **16**(10), 2352–2362 (1999). [CrossRef]

**2. **B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE **3963**, 3–14 (2000).

**3. **H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. **39**(35), 6621–6632 (2000). [CrossRef] [PubMed]

**4. **J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. **41**(10), 2532–2548 (2002). [CrossRef]

**5. **M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. **52**(1), 010201 (2008). [CrossRef]

**6. **L. Gao, R. T. Kester, N. Hagen, and T. S. Tkaczyk, “Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express **18**(14), 14330–14344 (2010). [CrossRef] [PubMed]

**7. **R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE **7875**, 787503, 787503-9 (2011). [CrossRef]

**8. **J. Brauers and T. Aach, “A color filter array based multispectral camera,” presented at the 12. Workshop Farbbildverarbeitung, Ilmenau, Germany, 5–6 Oct. 2006.

**9. **Y. Monno, M. Tanaka, and O. Masatoshi, “Multispectral demosaicking using adaptive kernel upsampling,” in *Proceedings of 18th IEEE International Conference on Image Processing* (Institute of Electrical and Electronics Engineers, 2011), 3218–3221.

**10. **F. H. Imai and R. S. Berns, “High-resolution multi-spectral image archives: A hybrid approach,” in *Proceedings of 6th Color Imaging Conference*, (Society for Imaging Science and Technology, 1998), 224–227.

**11. **Y. Murakami, K. Ietomi, M. Yamaguchi, and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. **46**(28), 7068–7082 (2007). [CrossRef] [PubMed]

**12. **Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. **48**(11), 2188–2202 (2009). [CrossRef] [PubMed]

**13. **O. Kohonen, “Multiresolution-based pansharpening in spectral color images,” in *Proceedings of 5th European Conference on Colour in Graphics, Imaging, and Vision*, (Society for Imaging Science and Technology, 2010), 535–540.

**14. **Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. **28**(7), 1470–1481 (2011). [CrossRef]

**15. **R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in *Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),* (Institute of Electrical and Electronics Engineers, 2011), 2329–2336.

**16. **D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. **21**(3), 331–336 (2004). [CrossRef] [PubMed]

**17. **W. K. Pratt and C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. **15**(1), 73–75 (1976). [CrossRef] [PubMed]