1. Introduction

Over the years, the use of fluorescent materials has increased in our daily lives. All kinds of everyday objects are made of such materials, including paint, dye, paper, plastic, and cloth. The usefulness of fluorescence is based on the effect that the visual appearance of the object surface is improved compared to a reflective surface based on non-fluorescent reflection. Because of fluorescent emissions, many fluorescent surfaces appear brighter and more vivid with respect to the original color of the surface.

The fluorescent characteristics are well-described in terms of their bispectral radiance factor. The radiance factor is a function of two wavelength variables: the excitation wavelength of incident light and the emission/reflection wavelength. The bispectral radiance factor can be summarized as a Donaldson matrix [1], which is an illuminant independent matrix representing the bispectral radiance factor of a target object. The bispectral radiance factor can be measured using two monochromators [2, 3]. However, the two-monochromator method is time-consuming and expensive. Thus, its use is confined to the laboratory and it is not available in ordinary imaging systems.

Although there have been many papers related to reflectance and fluorescent spectral recovery using an imaging system, most of them do not consider estimation of the Donaldson matrix. Rather, they separate fluorescent emission and reflectance (e.g., see [4–7]). Some approaches towards Donaldson matrix estimation are found in a limited number of papers [8–11].

It is usually considered that control of the illuminant spectrum is required in order to distinguish the reflected and luminescent photons. For example, Fu et al. [8] described a method that uses nine colored light in the visible range (400 to 700 nm) to recover the reflectance spectrum and the relative spectrum of emission and absorption from color camera data. Blasinski et al. [9] presented a framework that can estimate the reflectance spectrum and the absolute emission and absorption spectra in two fluorophores. This method is based on an imaging system consisting of many narrowband light sources and transmission filters attached to a camera. These methods used a linear basis function approximation to represent three spectral functions of reflectance, emission, and excitation. However, the selection of basis functions is not easy because the three spectral functions have unique spectral features. Increasing the number of basis functions may lead to an improvement of the spectral approximation, but that will result in an increased number of narrowband light sources. It should be noted that increasing the number of unknown parameters does not result in reliable spectral estimation. The optimal selection of basis functions, illuminant spectrum, and camera filtration is still a problem. Suo et al. [10] described an imaging system that uses programmable spectral filters placed in both sides of light source and camera. Although this approach could yield a direct estimation of the Donaldson matrix, the spectral sampling was too rough to use in practical fluorescent analysis.

Tominaga et al. [11] presented an approach using a spectral imaging system where only two illuminants with different spectra are projected onto an object without controlling the spectral shapes. The Donaldson matrix was modeled with high spectral resolution. This had the possibility to be used for several fluorescence analyses, such as mutual illumination [12], appearance reconstruction [13], and texture analysis [14]. The principle of the Donaldson matrix estimation was based on that the difference between the total radiance factors observed under the two illuminants. This difference was caused only by the luminescent radiance component due to fluorescent emission. However, the method was a graphical approach for finding spectral differences between the two total spectral radiance factors. Thus, the estimation results were often unreliable and unstable, especially for low intensity light sources.

The present paper proposes a stable and reliable method to estimate the bispectral Donaldson matrices for spectral imaging data of fluorescent objects. We suppose a general image acquisition system that allows more than two illuminant projections. The Donaldson matrix is modeled as a 71 × 60 array in spectrally high dimension. Excitation is defined in a 350 to 700 nm wavelength range, and emission range is defined a 400 to 700 nm wavelength range.

The estimation problem is solved as an optimization problem, where the residual error of the observations acquired by the spectral imaging system is minimized. We estimate the spectral functions of reflectance, emission, and excitation in a direct way without using basis functions. The output visible wavelength is segmented into two types of wavelength ranges: one consisting of only reflection and another consisting of both reflection and emission. In the first range, the reflectance is straightforward estimated; in the second range, the reflection and emission spectra are estimated iteratively. The excitation spectrum is estimated using a physical model. The emission wavelength range is determined based on error minimization. The usefulness of the proposed method is examined in experiments using different fluorescent objects and illuminants. We show how precisely the algorithm estimates the Donaldson matrix of a fluorescent object with a fluorophore. The effective selection of illuminants is discussed with the presented experiments. Finally, a successful application to spectral analysis and reconstruction of a fluorescent image is demonstrated using a real scene, which includes different fluorescent and non-fluorescent materials.

2. Bispectral modeling

2.1 Donaldson matrix for a fluorescent object

A Donaldson matrix $D({\lambda }_{em},{\lambda }_{ex})$ represents the bispectral radiance factor of a fluorescent object as a two-variable function of the excitation wavelength ${\lambda }_{em}$ and the emission/reflection wavelength ${\lambda }_{ex}$. The excitation wavelength range for all fluorescent materials starts from about 330 to 350 nm (see [15,16]). Because most light sources used in everyday life contain some ultraviolet (UV) component that contributes to fluorescent emission, the excitation range in this paper is set to $350\le {\lambda }_{ex}\le 700$ nm. As our spectral imaging system operates in the visible wavelength range, the emission/reflection range is set to$400\le {\lambda }_{em}\le 700$ nm.

The Donaldson matrix is decomposed into two components of the reflected radiance factor $D_R({\lambda }_{em},{\lambda }_{ex})$ by light reflection, and the luminescent radiance factor$D_L({\lambda }_{em},{\lambda }_{ex})$ by fluorescent emission. The matrix $D_R({\lambda }_{em},{\lambda }_{ex})$ is diagonal and has values only at ${\lambda }_{em}={\lambda }_{ex}$. This corresponds to surface spectral reflectance$S(\lambda )$ because monochrome light reflected from a surface has the same wavelength as the incident light. The matrix $D_L({\lambda }_{em},{\lambda }_{ex})$ has values only in the off-diagonal of satisfying ${\lambda }_{em}>{\lambda }_{ex}$ because the luminescent energy is emitted at a longer wavelength than each excitation wavelength due to the Stokes shift [17]. In this study, the luminescent radiance factor is separated into a multiplication of excitation and emission wavelength components as $D_L({\lambda }_{em},{\lambda }_{ex})=\alpha ({\lambda }_{em})\beta ({\lambda }_{ex})$. The two functions $\alpha ({\lambda }_{em})$ and $\beta ({\lambda }_{ex})$ are the emission and excitation spectra, respectively. The chromaticity invariance of fluorescence emission [8, 9, 11] is derived from this separation property. The above multiplication of $\alpha ({\lambda }_{em})$ and $\beta ({\lambda }_{ex})$ implies that one of two spectra can be arbitrarily rescaled. Therefore, we assume that the excitation spectrum is normalized such that ${\displaystyle {\int }_{350}^{700}\beta ({\lambda }_{ex})d{\lambda }_{ex}}=1$.

A discrete form of the Donaldson matrix with the above properties can be represented in an N × M matrix as

 

Fig. 1 Donaldson matrix obtained from a pink sample containing an orange fluorescent color.

Download Full Size | PDF

2.2 Observation model

We consider the matte surface of a fluorescent object without specularity. Let $E(\lambda )$ be the illuminant spectrum of a light source. The spectral radiances observed from the surface under this illuminant are described as a sum of the diffuse reflection component and the luminescent component as follows:

Suppose that n light sources with different spectral-power distributions are available. When the same fluorescent object surface is illuminated with each of these light sources, the observations can be described as

3. Estimation method

For computational simplicity, the observation equation is equivalently described in vector notation as

3.1 Estimation of only reflectance

The observations are described simply as

3.2 Estimation of both reflectance and emission

In the wavelength range${\lambda }_1\le \lambda \le {\lambda }_2$, we have

 

Fig. 2 Luminescent efficiency curve that represents the average curve of the efficiencies for different fluorescent samples. Thin curves represent the efficiencies for 12 different fluorescent objects. The bold curve and the broken curves represent the average curve and the standard deviation curves of $\pm 1\sigma$ from the average, respectively [11].

Download Full Size | PDF

We note that the reflectance $S(\lambda )$ is nested in Eqs. (11) and (13), thus $S(\lambda )$ and $\alpha (\lambda )$ cannot be determined directly. We adopt an iterative procedure. The initial condition of $\widehat{S}(\lambda )$ is set to a constant spectrum in the excitation range. The estimates $\widehat{S}(\lambda )$ and $\widehat{\alpha }(\lambda )$ are updated using Eqs. (11)-(13) at each iteration step. This procedure is repeated until the residual error ${\Vert y(\lambda )-\widehat{y}(\lambda )\Vert }^2$ becomes sufficiently small over the entire wavelength range. Note that the relationship in Eq. (13) is available only for estimation of $\beta (\lambda )$ in $400\le \lambda$ due to observation limitations. The spectral curve of $\beta (\lambda )$ in $350\le \lambda <400$ is estimated by interpolation based on the estimates of $\widehat{\beta }(\lambda )$ in $400\le \lambda$ and the terminal condition of$\widehat{\beta }(350)=0$.

3.3 Determination of the emission range

We determine the optimal wavelength range (${\lambda }_1$,${\lambda }_2$) for effective fluorescent emission. A pair of (${\lambda }_1$,${\lambda }_2$) are two-dimensional variables. We determine the optimal values that minimize the average residual error $J=\overline{{\Vert y(\lambda )-\widehat{y}(\lambda )\Vert }^2}$ over the entire visible wavelength range. The emission spectrum $\alpha (\lambda )$ has a single peak as shown in Fig. 3. We judge the unimodal property of $\alpha (\lambda )$ during the estimation procedure. Let ${\lambda }_{\text{p}}$be the peak wavelength of the emission spectrum. It is obvious that the slope of $\alpha (\lambda )$ should be positive when ${\lambda }_1\le \lambda <{\lambda }_{\text{p}}$ and negative when ${\lambda }_{\text{p}}<\lambda \le {\lambda }_{\text{2}}$.

 

Fig. 3 Unimodality of the emission spectrum.

Download Full Size | PDF

In Fig. 3, the unimodality was suggested as we had only considered a single fluorescent material in this study. The proposed method only considers a single fluorescent material which could only produce a unimodal emission spectrum. We do not recommend the proposed method for materials with multimodal fluorescence.

4. Experiments

A series of experiments using different fluorescent objects and illuminants was conducted to show the usefulness of the proposed method. The analysis includes precise estimation of the Donaldson matrix, effective selection of illuminants, and application to appearance decomposition and reconstruction. Figure 4 shows the spectral imaging system used in experiments, which consisted of a monochrome CCD camera with 12-bit dynamic range and Peltier cooling (QImaging, Retiga 1300), a VariSpec liquid crystal tunable filter, an IR-cut filter, and a personal computer. Figure 5 shows the total spectral sensitivity functions of the imaging system, including filter transmittances and camera sensitivity. The spectral images were captured at 5 nm intervals in the visible wavelength range, thus each captured image was represented in an array of 61-dimensional vectors. Since the total sensitivities depend on wavelength as shown in Fig. 5, the camera was adjusted so that the captured image intensity was equal in every wavelength channel. Let$R_i(\lambda )$ (i = 1, 2, …, 61) be the spectral sensitivity function of the i-th sensor of the spectral imaging system and $T_i$ (i = 1, 2, …, 61) be the exposure time of the sensor. The exposure time of the camera was determined to satisfy the following relationship for every channel:

 

Fig. 4 Spectral imaging system used in experiments.

Download Full Size | PDF

 

Fig. 5 Total spectral sensitivity functions of the imaging system.

Download Full Size | PDF

Figure 6 shows the spectral-power distributions for the following light sources: (1) an incandescent lamp (Iwasaki, PRF-300W), (2) an artificial sunlight lamp (SERIC, SOLAX 100W), (3) a photographic flood daylight lamp (Iwasaki, PRF-350WD), and (4) a white LED lamp (Hitachi, LDA11D-G/100C). Each illuminant spectral-power distribution was represented in an array of 71-dimensional vectors. Because the light sources used in this study emit unpolarized light, we do not suspect the occurrence of fluorescence polarization. However, since the VariSpec liquid crystal tunable filter is polarization-sensitive, it is possible in principle to sense polarization effects when an object is illuminated with a strong polarized light source.

 

Fig. 6 Illuminant spectral-power distributions of four light sources.

Download Full Size | PDF

In this paper, all gathered images and the analysis results are displayed in sRGB color images in order to reproduce accurate color appearance in a calibrated display device. To this end, all spectral images were first transformed into the CIE-XYZ images by using the color-matching functions, and were then converted to the sRGB images.

4.1 Estimation accuracy

The test samples were chosen from general objects that one can easily find in daily life. Figure 7 shows the observed images of (a) a pink sample and (b) a green sample under illumination with four different light sources. We made these samples by pasting cut sheets (product name: Lumino Sheet, manufacturer; Okina Inc.) on a board with dimensions of about 5 cm by 5 cm. Unfortunately, the manufacturers of the fluorescent objects did not provide us the details of the fluorescent substances included in the test objects. A portion of these images was used for estimation.

 

Fig. 7 Observed images of two fluorescent samples of (a) pink and (b) green under four light sources.

Download Full Size | PDF

The estimation algorithm was run by iteratively changing the emission wavelength range (${\lambda }_1$,${\lambda }_2$) while searching for the minimum average residual error J. The iterative algorithm converged after about three iterations in every case. Figure 8 depicts the average residual error for the pink sample as a function of ${\lambda }_1$ and ${\lambda }_2$. The unimodal property of the emission spectrum was satisfied in this range. Error minimization was achieved with J = 3.5 at ${\lambda }_1$ = 565 nm and ${\lambda }_2$ = 700 nm. Thus, the emission range was determined to be 565 to 700 nm. The estimated spectral curves of reflection, emission, and excitation for the pink sample are depicted in Figs. 9(a)-9(c). It is shown in Figs. 9(a) and 9(b) that the object color of this sample is pink, while the fluorescent color is orange. The Donaldson matrix constructed with the estimated spectral curves is shown previously in Fig. 1.

 

Fig. 8 Average residual error for the pink sample as a function of parameters ${\lambda }_1$ and ${\lambda }_2$.

Download Full Size | PDF

 

Fig. 9 Estimated spectral curves of (a) reflection, (b) emission, and (c) excitation for the pink sample.

Download Full Size | PDF

Next, the estimation results for the green sample are shown in Figs. 10-12. The residual error was minimized with J = 12.9 at ${\lambda }_1$ = 480 nm and ${\lambda }_2$ = 660 nm in Fig. 10, and the emission range was determined to be 480 to 660 nm. The estimated spectral curves in Fig. 11 and the Donaldson matrix in Fig. 12 suggest that the fluorescent material emits more saturated green light than the original green color of the object because the emission range is narrower than the reflectance range.

 

Fig. 10 Average residual error for the green sample as a function of parameters ${\lambda }_1$ and ${\lambda }_2$.

Download Full Size | PDF

 

Fig. 11 Estimated spectral curves of (a) reflection, (b) emission, and (c) excitation for the green sample.

Download Full Size | PDF

 

Fig. 12 Estimated Donaldson matrix for the green sample.

Download Full Size | PDF

For comparison, Fig. 13 shows the Donaldson matrices obtained by the previous method in [11], where two light sources (1) and (2) were used to illuminate the same fluorescent samples. The average residual error was J = 47.3 and J = 68.2 for the pink and green samples, respectively. These values are much larger than those using the present method. The estimated reflectance curves exhibit an unusual discontinuity, as shown in Fig. 13(b).

 

Fig. 13 Donaldson matrices for (a) pink and (b) green samples obtained by the previous method [11], where two light sources (1) and (2) were used to illuminate the same samples.

Download Full Size | PDF

4.2 Illuminant selection

We investigated how the illuminant spectra influenced the estimation performance. Since a brighter light source with higher power distribution is more effective for estimating the Donaldson matrix, the illuminants in Fig. 6 were normalized to have the same power. Figure 14 shows a set of illuminant spectral curves with $\Vert e_i\Vert =\text{constant}$ (i = 1, 2, 3, and 4), which were used in a numerical simulation for bispectral estimation. The same estimation algorithm was executed for different illuminant combinations. The performance indices in typical combinations for the pink sample are as follows:

 

Fig. 14 Illuminant spectral curves normalized to the same power.

Download Full Size | PDF

4.3 Application to appearance decomposition and reconstruction

Since the proposed method makes estimation of the precise Donaldson matrix at every pixel point in a scene containing different fluorescent objects, we can consider various appearance analyses applied to a fluorescence scene. Since the Donaldson matrix is spectrally constructed with the reflection component and the luminescent component as $D=D_{\text{R}}+D_{\text{L}}$, the image observed with illuminant e is decomposed into $D_{\text{R}}e$ and $D_{\text{L}}e$. The appearance of the reflection component depends directly on the illuminant e. Although the intensity depends on the illuminant, the luminescent component always has the same spectral composition as the emission spectrum. We should note that, once we estimate the Donaldson matrices for the fluorescent objects, we can spectrally reconstruct the appearance of the same object with an arbitrary illuminant. In other words, we can produce the spectral image required to predict the relighting result of the same object by using an arbitrary illuminant.

Figure 15 shows the observed images of a scene with different objects under illumination with (1) incandescent light and (2) sunlight. The scene contains six pencils, two erasers, and a white plate. The fluorescent pencils were Textsurfer® dry triangular highlighter pencils produced by Steadtler, and the non-fluorescent pencils were Color pencils produced by Tombo. The erasers were Resare produced by Kokuyo. The white plate was a white calibration plate (Konica-Minolta CRA43) made of a glass coated fine ceramic, whose chief ingredient was aluminum (III) oxide (Al₂0₃). The Donaldson matrix was estimated at each pixel using the proposed algorithm. Then, the observed images were decomposed into their reflection and luminescent components based on the estimated matrices, as shown in Figs. 16 and 17. It is seen that the objects with fluorescence are clearly separated from the non-fluorescent objects. Figure 17 shows only the fluorescent objects of three pencils and two erasers. The fluorescent objects in Fig. 17 (a) appear darker than the objects in Fig. 17 (b) because the incandescent light has less energy in the shorter wavelength range. Note that the fluorescent colors in Fig. 17 (a) have the same chromaticities as in Fig. 17 (b).

 

Fig. 15 Observed images of a scene with different objects under two light sources. ((a): incandescent and (b): sunlight).

Download Full Size | PDF

 

Fig. 16 Appearance of the reflection component. ((a): incandescent and (b): sunlight).

Download Full Size | PDF

 

Fig. 17 Appearance of the luminescent component. ((a): incandescent and (b): sunlight).

Download Full Size | PDF

We used the illuminant spectral-power distributions of a white LED and a RGB mixing LED shown in Figs. 18(a) and 18(b) for appearance reconstruction. The two light sources have the same color temperature of about 6500 K. Figure 19 demonstrates the appearance reconstruction results of the same scene by illuminating with the two LED lights. The white plates in Figs. 19(a) and 19(b) present the same color appearance under the two illuminants because the sources have the same color temperature. It is interesting to observe that the appearances of the yellow pen and the green pen are distinguishable between the two figures. When Figs. 19(a) and 19(b) are compared, one can see in Fig. 19(a) that the green appears emphasized so that the yellow pen shifts to green and the green pen is saturated, compared with in Fig. 19(b). This is because the white LED illuminant has a much larger amount of shorter wavelength components that contribute to green fluorescent emission. Thus, the appearance of fluorescent objects is strongly affected by the illuminant spectral curve. The proposed method is useful for various purposes including these spectral analysis and spectral reconstruction of a fluorescent object scene.

 

Fig. 18 Illuminant spectral-power distributions of (a) white LED and (b) RGB mixing LED, which have the same color temperature of about 6500 K.

Download Full Size | PDF

 

Fig. 19 Appearance reconstruction results of the same scene by illuminating the two LED lights. ((a): white LED and (b): RGB mixing LED).

Download Full Size | PDF

4.4 Useful and practical applications

The developed algorithm and multispectral imaging system can have various applications in computer vision, image processing, and computer graphics [20], as well as many other fields including biology, medical diagnosis, food science, material science, and art inspection. In the field of biology, after discovery of green fluorescent protein by Shimomura et al. [21], many studies have been conducted on chlorophyll fluorescence analysis [22]. Fluorescence imaging provides a useful tool to investigate the photosynthetic activities of plant leaves with chlorophyll [23], and to identify plants and characterize their state of health in remote sensing [24].

In medical diagnosis, fluorescence imaging provides an important diagnostic method [25]. For instance, fluorescence spectroscopy of skin shows significant differences between normal and tumorous tissue, both from the point of view of detecting endogenous signals and of using of exogenous fluorescent markers [26]. Combining the multispectral reflection mode imaging with the fluorescence imaging in a single instrument lends strength to the proposed technique. Therefore, if the S/N (signal-to-noise) ratio is high, the proposed technique could be applied to discriminate pathological tissues from normal tissues in biomedicine by using an ordinary light source. Reconstruction may be possible if the fluorescence component of the signal and the intrinsic sample fluorescence are spectrally distinguishable. Since the bispectral characteristics in the proposed technique could be estimated at each pixel point, two-dimensional distribution and visualization of fluorescence is possible.

In food science, the analysis based on an excitation-emission matrix (EEM), called a fluorescent fingerprint, has been successfully applied to quality evaluation of different types of food [27, 28]. In material science applications, bispectral spectrophotometry is used for colorimetry and radiance evaluation of fluorescent printing materials [29, 30]. A similar fluorescent analysis is also used for evaluation of groundwater and wastewater [31]. In art inspection, a noninvasive technique based on fluorescent spectroscopy is used to characterize the painting materials and evaluate the conservation state [15].

The accuracy of the Donaldson matrix estimation depends on the imaging geometry. In this paper, we consider the light source to be located in the vertical direction with respect to the object surface, and the camera in front observes the illuminated surface. Since the object surface is a Lambertian diffuser, the observed signal intensities depend on the incidence angle of light rather than the viewing angle. As the incidence angle increases toward the grazing angle, the reflection and emission signal intensities decrease, leading to decreased estimation accuracy. The accuracy also depends on the curvature of the surface in a similar way. If the surface roughness is low, the estimation results for a rough surface show that the performance is similar to the results from a smooth surface.

5. Conclusions

In this paper, we have proposed a method to estimate the bispectral Donaldson matrices of fluorescent objects using a spectral imaging system. This is the first attempt to use multiple ordinary light sources where continuous spectral-power distributions were projected sequentially to the object surfaces without controlling the illuminant spectral shape. The problem of estimating the Donaldson matrices was solved as an optimization problem, where the residual error of the observations acquired by the spectral imaging system is minimized. We estimated the spectral functions of reflection, emission, and excitation at each wavelength without using a basis function approximation.

To improve the estimation efficiency, the output visible range was segmented into two wavelength ranges. One consisted only of reflection, and another consisted of both reflection and emission. An iterative estimation algorithm was developed based on this wavelength segmentation and the physical excitation model. The optimal fluorescent emission range was determined based on the error minimization method. We examined the usefulness of the proposed method in experiments using different fluorescent objects and illuminants. We showed the estimation accuracy of the Donaldson matrices, discussed the effective selection of illuminants used, and demonstrated an application to spectral analysis and reconstruction of a fluorescent image.

The proposed method has several useful features. The algorithm obtained here is simple and stable, which is optimized in the sense that the estimates are determined to minimize the residual error of the observations. The Donaldson matrix estimation procedure is more direct and does not require external control over the illuminant spectra, nor does it require a basis approximation of spectral functions. The Donaldson matrices were represented as a spectrally high-dimension 71 × 61 array with an excitation range of 350–700 nm and emission range of 400–700 nm. Because of these useful features, we can easily extend the present method to wider spectral ranges. If the Donald matrices are measured directly by the two-monochromator method, the reliability and accuracy can be verified clearer. Therefore, we suggest that further studies should include a comparison with the direct measurement results of the Donald matrices.