This tutorial offers an introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research. The main types of hyperspectral cameras are described together with procedures for image acquisition, postprocessing, and calibration for either radiance or reflectance data. Image transformations are defined for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Several example applications are also presented. These include calculating the color properties of scenes, such as gamut volume and metamerism, and analyzing the utility of color in observer tasks, such as identifying surfaces under illuminant changes. The effects of noise and uncertainty are considered in both image acquisition and color vision applications.
Published by The Optical Society under the terms of the Creative Commons Attribution 4.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.
Hyperspectral imaging combines spatial and spectral data in such a way that each pixel of an image of a scene or object represents a continuous radiance or reflectance spectrum . Terminology in spectral imaging is not standardized , but can be indicative. Thus hyperspectral imaging differs from multispectral imaging, where spectral sampling takes place over multiple nonadjacent wavelength bands or is relatively coarse within a band so that some spectral information is lost. In turn, multispectral imaging differs from conventional trichromatic or RGB color imaging, where spectral sampling is reduced to just three components, from the long-, medium-, and short-wavelength regions of the visible spectrum.
Hyperspectral imaging is relevant to research in vision, especially vision in natural scenes, because hyperspectral images capture the full spatial and spectral properties of the light signal, thereby preserving the relevant degrees of freedom. In the main, these signals are spatially and spectrally complex . Figure 1 shows radiance spectra reflected from individual pixels in a hyperspectral image of a flower scene.
Natural scenes are interpreted broadly to include both undeveloped and developed land cover [6,7]. Specifically, scenes may be predominantly vegetated, containing woodland, shrubland, herbaceous vegetation, and cultivated land, or predominantly nonvegetated, containing barren land, urban development, as well as farm outbuildings and painted or treated surfaces . Importantly, they embody the everyday visual environment, unlike some arrays and tableaux used in laboratory studies.
In addition to enabling visual function to be realistically analyzed and computationally modeled in the natural world, hyperspectral imaging allows more accurate testing of colorimetric and color appearance formulas and more faithful rendering of scene data on color display devices, for both laboratory research and its practical applications.
Unavoidably, the technical literature spans multiple disciplines. The aim, therefore, of this tutorial is to provide an integrated introduction to terrestrial and close-range hyperspectral imaging and some of its uses in human color vision research.
Section 2 describes the main types of hyperspectral cameras and procedures for image acquisition, postprocessing, and calibration. Section 3 compares spectral radiance and reflectance images and the effects of real and simulated illumination changes. Sections 4 and 5 present image transformations for colorimetric representations, color rendering, and cone receptor and postreceptor coding. Sections 6 and 7 contain example applications. These include the calculation of color gamut, color constancy, and metamerism, and an analysis of the utility of color in tasks requiring an observer to distinguish surfaces or identify them under illuminant changes. Notation is guided by [9–13] but amended where necessary for clarity or consistency with previous work. As with other tutorials , no attempt is made at a comprehensive review of the field or to report novel research findings. Citations may not reflect priority.
General introductions to hyperspectral imaging can be found in Refs. [15–17] and to color vision and colorimetry in Refs. [18–22]. Other sources are cited at points in the text. Advice on different aspects of camera calibration is available in Refs. [23–30], and material on photon transfer, radiometry, and reflectance in Refs. [9,10,31,32]. Image quality assessment that takes into account visual performance is considered in Ref. . Applications concerning artwork and vision are explored in Refs. [34–36]. A list of close-range hyperspectral image datasets relevant to vision research is available in Ref. ; see also Ref. . Various hyperspectral image-compression techniques are proposed in Refs. [39–41].
The color images and associated data in this article are from hyperspectral images acquired with an in-house wavelength-scanning hyperspectral camera, which provided images of size over a spectral range 400–720 nm in 10 nm steps . Apart from material in Section 2, however, where different camera types are described, none of this tutorial is specific to any one imaging system.
2. HYPERSPECTRAL IMAGING
A hyperspectral image can be treated as a three-dimensional set of data, sometimes called a datacube or hypercube, with values representing a spectroradiometric quantity, such as spectral radiance or spectral reflectance, indexed by spatial coordinates , and wavelength . A two-dimensional section of the datacube, known as a slice, can be defined for a particular passband center wavelength , or for a particular spatial coordinate or , to within the sensor resolution of the camera. Figure 2 shows an array of wavelength slices. Each is a grayscale intensity image at wavelengths of making up the hyperspectral radiance image of the flower scene in Fig. 1. The different spectral reflecting properties of the surfaces are evident, with, e.g., the flowers on the left of the scene appearing brighter in the grayscale images from around 600 nm onward.
Depending on the hyperspectral camera, images can be acquired a slice at a time or in a single operation. Significant postprocessing of the raw signal is required either internally by embedded hardware or externally by the user in order to reduce the effects of noise and to obtain accurate radiance or reflectance estimates.
A. Hyperspectral Cameras
In the present context, which hyperspectral cameras are the most suitable for scene imaging? Systems based on wavelength scanning, line scanning, and non-scanning, single-shot imaging are all popular [16,17,42]. Non-scanning systems offer greater efficiencies than scanning systems  but may entail compromises in resolution.
Active hyperspectral imaging, such as terrestrial laser scanning, delivers only reflectance data .
1. Wavelength Scanning
In wavelength-scanning imaging, also known as area or focal-plane scanning, or front staring imaging, the incident light passes through a tunable spectral filter placed in front of the camera lens or between the camera lens and camera sensor.
Spectral sampling may be achieved with a fixed number of mechanically interchangeable bandpass filters [45–47], or with an electronically tunable liquid crystal or acousto-optical filter or an interferometer [17,42,48,49]. The image is constructed either directly, one wavelength at a time, or indirectly from the interferogram.
Wavelength-scanning systems have the advantage that with well-corrected optics, they preserve the geometry of the scene or object during spectral sampling, and exposure time can be adjusted with wavelength according to the spectral properties of the scene. They have the disadvantage that acquisition time can be prolonged at low light levels and that spatial resolution is limited by the size of the sensor array. Additionally, scene movement may generate chromatic artifacts in the rendered image.
2. Line Scanning
With a line-scanning or pushbroom hyperspectral camera, the incident light is focused by a lens onto a slit aperture from which the light is spectrally dispersed by a grating or other device  and then focused onto the camera sensor. The image of the scene is thus constructed a row (or equivalently a column) at a time, and at every point along the row, a complete spectrum, indexed by wavelength , is recorded . Successive rows, indexed by spatial coordinate , may be acquired with a moving mirror or by rotating the whole camera .
Line-scanning systems have the advantage that spatial sampling in the other spatial dimension is, in principle, unlimited, but the disadvantage that asymmetric sampling may lead to distortion . Exposure time cannot be adjusted with wavelength in the same way as with wavelength-scanning systems, and large variations in incident radiance with wavelength can prove a challenge . Line-scanning systems are also vulnerable to scene movement.
3. Single Shot
With a single-shot or snapshot hyperspectral camera, the incident light is divided simultaneously over multiple spectrally selective sensor arrays, or it is optically coded and spectrally dispersed over a single sensor array, or imaged on a single spectrally patterned sensor array, analogous to the Bayer filter mosaic in a conventional color camera [2,54,55].
Single-shot systems have the advantage that complete spatial and spectral data are acquired simultaneously, but the disadvantage that there is a tradeoff between the number of spatial and spectral samples according to how the incident light is divided or coded [2,43,54,56,57]. As the number of spectral samples increases, the number of sensor elements allocated to a particular wavelength generally decreases. The alternative of increasing the size of the sensor array can lead to longer integration or readout times. Other relevant issues include spatial and spectral aliasing [2,58], demosaicing artifacts [56,59], and, with spectrally patterned arrays, the consistency of wavelength tuning across corresponding sensor elements.
B. Image Acquisition
The workflow for image acquisition outdoors depends on the choice of hyperspectral camera, viewing geometry, and whether spectral radiance or reflectance data are required. The following steps are typical for a spectroradiometrically uncalibrated scanning system [8,16,25,32,50,60,61]:
- 1. Adjust the camera position and focus for the scene (or region or object).
- 2. To aid later spectral calibration (Sections 2.C.5, 2.C.6), introduce into the scene or field of view one or more small, spectrally neutral, reference surfaces, e.g., a barium sulfate plug or a Munsell gray sphere or plate , as in Fig. 1, or a multi-color chart (e.g., ColorChecker, X-Rite Pantone, Grand Rapids MI, USA) for additional control measurements [62,63]. Ensure that the reference surface is oriented correctly with respect to the camera axis and incident illumination. Insert or identify any structures in the scene that can provide fiducial marks for later spatial calibration and image registration in postprocessing (Section 2.C.4).
- 3. Adjust the aperture of the camera, and, if necessary, estimate the exposure timing in a trial acquisition sequence, avoiding point specularities.
- 4. Acquire a hyperspectral image of the scene.
- 5. If the image is to be processed as a spectral radiance image rather than a reflectance image, record the spectral radiance reflected from the reference surface or surfaces in the scene or field of view with a calibrated spectroradiometer or telespectroradiometer
- 6. Acquire a hyperspectral image of a uniform, neutral, matte, flat, reflecting plate placed in front of the camera to cover the field of view. This flat-field image is used to estimate fixed-pattern noise (Section 2.C.1), although its effects are more systematic than random. As a spatial measurement, it is distinct from the spectral measurements with the reference surface in steps 2 and 5. Flat-field images can also be obtained in the laboratory from a full spectrum source with an isotropic diffuser placed in front of the camera or with an integrating sphere [32,64].
- 7. Acquire a hyperspectral image with the lens covered but with the same exposure timing as for the scene image and at the same temperature. This dark-field image is used to estimate the combined effects of dark-current noise, bias, and readout noise.
- 8. If necessary, repeat the scene image acquisition to reduce uncorrelated image noise by subsequent averaging. Photon noise can be a major source of error. Both flat-field and dark-field image acquisitions should also be repeated several times . The plate used for flat-fielding may be moved a little between acquisitions to ensure the uniformity of the estimate.
Image sequences should be examined for defects, most often movement artifacts and sensor element saturation resulting from transient point specularities. Defective sequences may need to be discarded. If the imaging conditions are constant, then common flat-field images and common dark-field images may be used. Details of the scene and imaging conditions should be recorded as part of the acquisition documentation.
C. Image Postprocessing
After acquisition, images obtained with an uncalibrated system need to be corrected for noise errors and other systematic errors and then calibrated spatially to give angular subtense and spectrally to give spectral radiance or reflectance . Correcting for achromatic geometric errors and spatial blurring are considered in Refs. [65,66] and for spectral blurring in Refs. [67,68]. Notation here differs somewhat from that in Refs. [10,13].
1. Noise Errors
Errors due to noise can be divided into non-imaging and imaging errors. Non-imaging noise errors include camera dark-current noise, which increases with exposure duration and temperature; bias, which provides a constant offset; and readout noise. The main imaging noise error is fixed-pattern noise, which includes non-uniformity in the sensor array response, i.e., pixel-to-pixel variation in quantum efficiency, gain, and dust [10,31,69], and off-axis vignetting, i.e., darkening at the corners and edges of the image [27,64]. These and other sources of noise are analyzed in Refs. [28,31,32].
Let be the scene image intensity indexed by spatial coordinates , , and wavelength ; let be the corresponding flat-field image intensity; and let be the corresponding dark-field image intensity (independent estimates of dark-current noise, bias, and readout noise are usually unneeded ). The approximately corrected scene image intensity is then given [32,70] by32,69], and any departures from linearity can be accommodated by adding correction terms to Eq. (1).
Fluctuations in the recorded values of may lead to the corrected scene image intensity in Eq. (1) taking on numerically small, negative values at some points. These values may be set either to zero or to some mean estimate derived from other areas of the image.
A correction for stray light or light scatter within the camera may also be applied [29,71,72], but in practice, it is necessary only if there are locally very bright reflecting or emissive surfaces within the scene.
2. Wavelength Errors
Errors in the nominal passband center wavelength setting of the system can be quantified with a calibrated precision monochromator. Errors in the passband center wavelength across the field of view can be quantified by repeating a flat-field measurement in which the illumination is from spectral line sources . Both kinds of wavelength error ought to be small and capable of being accommodated by corrections to Eq. (1).
3. Spectral Sampling and Out-of-Band Transmission
The wavelength sampling interval of the imaging system is usually larger than the passband width, so sampling can be approximated by a delta function . Alternatively, a routine correction can be applied for bandpass error [13,73].
Spectral leakage, i.e., transmission at wavelengths outside the selected passband of the system [10,25,64], may need to be considered. It may be extremely small, several orders of magnitude less than maximum transmission , yet, in principle, its effects can be cumulative at long wavelengths where the radiant energy of the sun is greater, vegetated surfaces reflect more strongly (the “red edge”), and the quantum efficiency of the camera sensor is higher. A correction for spectral leakage can be made as follows.
Let be the spectral transmission of the system as a function of wavelength at each passband center wavelength . The in-band portion of this function can be measured with a calibrated precision monochromator, but the residual out-of-band transmission may need other methods. Let and be the respective image spectral intensities at the point before and after spectral sampling. Then,2) can be solved by matrix methods to yield the required estimate of , with due attention paid to the accuracy of the numerical data for and the stability of the solution.
In practice, with an infrared-blocking filter in place, the correction is usually very small and may be omitted under good imaging conditions. Nevertheless, one or more calibrated surfaces that reflect predominantly at long wavelengths, e.g., Munsell 10YR , may be included in the scene to monitor the extent of long-wavelength leakage.
4. Image Registration
Longitudinal chromatic aberration of the optical system and consequent chromatic image blur are reduced with achromatic or apochromatic lenses . Even so, lateral chromatic aberration, i.e., variation in optical image size with wavelength, may still be visible at the extremes of the spectrum, resulting in misalignments of several pixels across the full image width. Figure 3, top left, shows a color image of a stone cottage rendered from a hyperspectral radiance image with an enlarged copy of the square area marked in white, bottom left. The color fringing on the drainpipe is obvious, but also detectable on the branches of trees and the walls of buildings.
The principle of image registration is to take one wavelength slice as a reference or base image, usually from the middle of the spectrum, and apply operations such as scaling and translation to the other wavelength slices to optimize the alignment, quantified, e.g., by an areal measure such as normalized cross correlation . Figure 3, top right, shows the effect of registration. An enlarged copy of the square area marked in white is shown bottom right. The color fringing is eliminated.
As an alternative to areal registration, image features such as edges or corners may be identified and local edge fitting applied to define control points for a single scaling and translation operation . Non-Euclidean transformations may also be applied .
The method of registration needs to be chosen with reference to the composition of the scene, since regions of the image may reverse contrast from one part of the spectrum to another. The effect is evident in Fig. 2. At 520 nm, the flower on the lower right of the scene appears darker than the leaf beneath it in the grayscale image, but at 550 nm, it appears brighter, with the result that the correlation between the slices changes sign. In these parts of the spectrum, regions, edges, and corners may lose their identity or shift spatially.
The problem of aligning wavelength slices differing significantly in structure can be avoided by applying registration iteratively, each slice being compared with the next , though with the risk that errors may accumulate across the spectrum.
Whichever method is chosen, the edge of the registered image may need to be trimmed to eliminate pixel artifacts with zero or replicated padded values.
5. Spectral Radiance Calibration
A spectral radiance image can be obtained from the corrected scene image by exploiting flat-field data (Section 2.B), but the calculation requires additional measurements and contains uncertainties. A more robust way to proceed is to use the spectral radiance data recorded independently from the reference surface or surfaces in the scene or field of view (Section 2.B).
Suppose that is the spectral radiance recorded from a reference surface at a point as a function of wavelength . Then, for the selected viewing geometry, the spectral radiance at the point is estimated by26]. It can be improved by using data from several reference surfaces . Suppose that are the spectral radiances recorded from these surfaces at the points . Perform an th-order regression of on , where . Thus, if and the estimated regression coefficients are and , then 1) is correct.
6. Spectral Reflectance Calibration
The procedure for obtaining a spectral reflectance image from the corrected scene image is analogous to that for obtaining a spectral radiance image. But only the spectral reflectance at a point on the reference surface is required as a function of wavelength , not its spectral radiance . For the selected viewing geometry, the spectral reflectance at the point is estimated by3), may amplify image errors at short wavelengths, when the incident illumination is weak. But as with spectral radiance estimates, this one-point calibration can be improved by using data from several reference surfaces and performing an th-order regression of spectral reflectances on . If and the estimated regression coefficients are and , then 1) is correct.
7. Spatial Subsampling
Even after registration, data from adjacent pixels in the hyperspectral image of a scene are likely to be correlated because of the limited resolution of the camera optical system. The point-spread function (PSF), which in general varies with wavelength, provides a useful guide. It can be estimated by methods using point imaging , edge imaging , and random-noise imaging . If the PSF is much larger than the pixel spacing, the spectral radiance image or spectral reflectance image may need to be spatially subsampled before being used in analysis and modeling.
3. RADIANCE AND REFLECTANCE ESTIMATES
Given the procedural similarities in calibrating for spectral radiance and spectral reflectance, which of the two representations is the more appropriate? It depends on how the data are used, the differences between real and simulated illumination changes, and the variations in natural reflected spectra.
A. Effective Reflectances and Global Illuminants
Recall from Section 2.C.6 that a spectral reflectance image , indexed by spatial coordinates , and wavelength , is obtained by scaling the corrected scene image by the spectral reflectance of one or more reference surfaces in the scene. It is strictly an effective reflectance in the sense that it represents the scene for the selected viewing geometry as if it consists of planar Lambertian reflecting surfaces illuminated by a spatially uniform or global illuminant , which need not coincide with the true illumination on the scene .
More precisely, suppose, again, that is the spectral radiance recorded at a point on a reference surface in the scene or field of view (Section 2.C.5) and that is the known spectral reflectance of that surface (Section 2.C.6). Then, the global illuminant is defined by3) or indirectly through Eq. (4), since the outcome is the same. Effective spectra are sometimes referred to as apparent  or equivalent [83,84].
Any nonlinear contributions to Eq. (4) from processes such as mutual reflection and fluorescence are assumed to be negligible. Mutual reflection, where light reflected from one surface illuminates a second , also referred to as mutual illumination or interreflection , depends on the proximity and orientation of the surfaces in the scene [87,88]. Fluorescence, which is commonly associated with vegetation , accounts for perhaps 4%–15% of the nominal reflectance in the 670–690 nm region of the spectrum . Both contributions, which are difficult to predict for the general case, are probably outweighed by other factors, as illustrated in Section 3.B.
These specific nonlinearities aside, the reflected radiance is the result of a fundamentally more complicated interaction than that summarized by Eq. (4), since it involves the bidirectional reflectance distribution function (BRDF) [9,91]. With an abuse of notation, let be the incident spectral irradiance at the point in the direction at wavelength , where the polar and azimuthal angles and are defined with respect to a fixed directional coordinate system. Likewise, let be the bidirectional spectral reflectance distribution function at in the direction at wavelength , for the viewing direction , where the points of incidence and reflection are assumed to be the same . The reflected spectral radiance in the direction is then given by
The role of direction of incidence and position in and is nontrivial. In the product of effective quantities in Eq. (4), the variation in the incident spectral irradiance is absorbed into the effective spectral reflectance, whose magnitude may exceed unity at some points because the reference surface is oriented with its normal to the camera and at an angle to the sun. These unreal values of lose their significance when multiplied by in Eq. (4) to form the reflected spectral radiance .
It is emphasized that in practice the effective spectral reflectance recorded from a physical matte object whose spectral reflectance is nominally constant across its surface can still vary markedly from pixel to pixel. In addition to photon and acquisition noise identified in Sections 2.B and 2.C, there are multiple other sources of noise associated with physical surfaces under natural illumination. These sources are enumerated in Section 3.C.
B. Real and Simulated Illumination Changes
Whether a reflected spectral radiance can be represented as a product of a global illuminant and an effective spectral reflectance, as in Eq. (4), depends on the separability of the integrand in Eq. (5), i.e., with a further abuse of notation, whether can be approximated as the product of spatial and spectral factors:5) for the reflected spectral radiance becomes 4). The approximation can be usefully tested with two kinds of natural illumination change, one mainly spectral, the other mainly geometric.
1. Spectral Illumination Changes
An example of where spectral illumination effects dominate is shown in Figs. 4(a) and 4(b). The color images are rendered from time-lapse hyperspectral radiance images of a rock face acquired in the late afternoon, about 25 min apart. Although there are small differences in the distribution of shadows in (a) and (b), the obvious change is in the shift of the mean scene color towards red, reflecting the change in prevailing illumination.
The color of the illumination is usually identified with its correlated color temperature in kelvin [13,92], where 4000 K typifies the yellow-orange of the setting sun and 25,000 K the blue of the north or polar sky. For the scene in Fig. 4, the spectrum of the direct illumination on the scene can be calculated from the hyperspectral image radiance at the small neutral matte Munsell N7 rectangular plate at the bottom right of the scene, arrowed in (a). The correlated color temperature of the incident light changes from 4180 K at 18:15 in (a) to 3350 K at 18:40 in (b). For later reference, notional average daylight D65 has a correlated color temperature of approximately 6500 K , which on a reciprocal color temperature scale , falls roughly midway between 4000 K and 25,000 K.
Can the reflected spectral radiances in Figs. 4(a) and 4(b) be adequately represented as products, as in Eq. (4), with a common effective spectral reflectance? Suppose that and are global illuminant spectra with correlated color temperatures 4180 K and 3350 K, respectively, and that and are the corresponding spectral radiances reflected from the scene. Suppose further that is an effective spectral reflectance defined by . The approximation for is then given by4(c) shows the color image rendered from according to Eq. (6). This simulated image and the real one in Fig. 4(b) are closely similar, though there are small differences in the shadows. The normalized root mean square deviation between the spectral radiances is 1.9%, where normalization is with respect to the range.
2. Geometric Illumination Changes
More frequently, it is geometric effects that dominate natural illumination changes, as illustrated in Figs. 4(d) and 4(e). The color images are rendered from time-lapse hyperspectral radiance images of the same rock face as in (a) and (b) but acquired at two earlier times of day. The correlated color temperature of the light incident on the rectangular plate at the bottom right of the scene, arrowed in (a), decreases from 5700 K at 13:21 in (d) to 5510 K at 15:15 in (e). The spectral difference is negligible, but there are marked changes in the distribution of shadows.
Can the reflected spectral radiances in Figs. 4(d) and 4(e), like those in Figs. 4(a) and 4(b), still be adequately represented as products with a common effective spectral reflectance? The analysis follows Section 3.B.1. Suppose that and are global illuminant spectra with correlated color temperatures 5700 K and 5510 K, respectively, and that and are the corresponding reflected spectral radiances. Again, is defined by .
Figure 4(f) shows the color image rendered from the approximation for according to Eq. (6). The simulated image in (f) and the real one in (e) are now clearly different. The normalized root mean square deviation between the spectral radiances is 13.7%, about seven times larger than in (b) and (c).
Although the assumption of a common effective spectral reflectance for the two images is here unsafe, some local statistical properties may nonetheless be preserved .
C. Variations in Natural Spectra
Reflected spectra recorded outdoors may show much greater variation than in the laboratory for several reasons:
- 3. Natural illumination is inconstant. In a clear sky with no visible cloud, the primary source of variation is solar elevation, but unobservable cirrus clouds and aerosols can be highly inhomogeneous, leading to additional variation in the solar beam. Average minute-to-minute temporal fluctuations in irradiance can be on the order of 0.1% around midday and much more around sunrise and sunset (data provided by A. R. D. Smedley and A. R. Webb, University of Manchester). The angular and spectral distributions of light from the sun and sky are also very different , and the balance between the direct beam and diffuse radiation and the radiance distribution across the sky change with solar elevation. Nearer to the scene being imaged, object occlusions, mutual reflections between surfaces, and transilluminance all contribute to further variation in the incident light .
On the other hand, in the laboratory, reflection spectra are typically recorded from relatively homogenous materials with a non-imaging device such as a reflection spectrophotometer, which averages over extended areas under controlled illumination. Where materials and imaging conditions coincide, however, closely similar spectral estimates ought to be obtainable from a correctly calibrated hyperspectral imaging system and a non-imaging spectroradiometer [8, Fig. 1].
4. COLORIMETRIC REPRESENTATION
For colorimetric calculations, psychophysical measurements, image display, or other related purposes, hyperspectral radiance images may be mapped into a variety of color spaces. How, then, to decide on which space? Factors to consider include the effect of different scene illuminants in the space, the uniformity of the space, i.e., whether equal distances correspond to equal color differences, and adjustments for the characteristics of any display device. Color spaces modeled on physiological data are discussed in Section 5.
In colorimetric applications, it is conventional to generate an intermediate representation expressed in CIE XYZ tristimulus values [21,22]. Thus, let be the CIE XYZ  color-matching functions for the 2° standard observer, and let be a spectral radiance image, indexed by spatial coordinates , and wavelength . The tristimulus values , , at each point or pixel are given by21].
There is generally no easy relationship between the RGB values of a scene recorded by a color camera and the corresponding CIE XYZ values. The camera spectral sensitivities would need to be linear transformations of the CIE color-matching functions, i.e., meet the Luther condition [21,106].
To simplify notation, the explicit dependence of , , on position (, ) is suppressed.
A. Compensating for Different Illuminations
Given a trichromatic representation of a scene, a color image, it is possible to compensate approximately for a pure spectral change in illumination by transforming the representation. The transformation, traditionally called a chromatic adaptation transform, transforms the colors of the scene under one illuminant, referred to as the test, to the corresponding colors under another illuminant, the reference, so that, ideally, they have the same color appearance [21,22,107].
Historically, chromatic adaptation transforms are based on von Kries scaling [22,92]. The idea is to compensate for a shift in the color of the illumination, e.g., towards long wavelengths, by reducing the gain of the appropriate cone receptor class, here the long-wavelength-sensitive cones. This adjustment is independent of any adjustments to the medium- and short-wavelength-sensitive cones. In matrix notation, it is a diagonal matrix transformation .
In practice, with color images represented by their CIE XYZ tristimulus values, additional transformations are needed to convert the tristimulus values to cone responses or some sharpened version  for von Kries scaling and then back again. Such a transformation is incorporated into the standard chromatic adaptation transform CMCCAT2000, which takes as parameters the tristimulus values of the test and reference illuminants and , along with some other optional parameters . Routines for performing chromatic adaptation transforms are available from several sources, e.g., Ref. .
The reason that chromatic adaptation can compensate only approximately for illuminant changes is that for most scenes, the number of degrees of freedom for surface reflectance and illumination spectra is greater than for their trichromatic representations. Chromatic adaptation transforms can, though, be optimized for particular sets of reflecting surfaces. These are often manufactured sets, such as textiles and Munsell papers . But unlike hyperspectral reflectance data, they do not capture the distributions of spectral reflectances in the natural world.
B. Representing Hyperspectral Images in Uniform Color Spaces
Although CIE XYZ tristimulus values allow the equality of image colors to be decided under the same conditions, equal differences in tristimulus values need not correspond to equal differences in discriminability or color appearance. What is required is a uniform color space, i.e., one in which equally spaced points correspond to equal perceptual color differences [92,111]. Standard examples include the approximately uniform color space CIELAB  and the more uniform space CIECAM02 , used here as a color space, not an appearance model. The coordinates of CIELAB space are , , , where represents lightness, ranging from 0 to 100, and and represent nominally red–green and yellow–blue chromatic components, respectively. The corresponding coordinates of CIECAM02 space are [21,22].
It is straightforward to map a hyperspectral radiance image into CIELAB space. Let , , be the tristimulus values at each point and be the tristimulus values of the scene illuminant at the reference surface, the illuminant being assumed global. The transformation of , , to CIELAB values with respect to is defined  as follows:
As an illustration, the CIELAB hue angles of the selected samples from the flower scene in Fig. 1 are 140° and 114° for the green leaves, left and top; 49° for the orange flower, left; 305° (i.e., ) for the purple flower; and 71° for the yellow flower, right. Further discussion can be found in Refs. [21,22].
Somewhat inconveniently, CIELAB space is not suited to comparing differences between surfaces under illuminants other than average daylight [13,22]. The reason is that in Eq. (8) the illuminant appears in a quotient of tristimulus values, whereas to accurately accommodate chromatic adaptation, the quotient should be of cone responses (Section 4.A). The formula delivers what is sometimes called the wrong von Kries transformation .
A better procedure [13,22] for comparing color differences under changes in illuminant is to first apply an accurate chromatic adaptation transform with respect to some standard reference illuminant (not to be confused with the actual illuminant at the reference surface in the scene) and then transform the result to CIELAB coordinates. The chromatic adaptation transform CMCCAT2000 is used in this analysis, but recall from Section 4.A that chromatic adaptation can compensate only approximately for illuminant changes.
As before, let , , be the tristimulus values at each point and be the tristimulus values of the scene illuminant at the reference surface. Then:
- 1. Let be the tristimulus values of the reference illuminant, e.g., an equi-energy illuminant or standard daylight with correlated color temperature of 6500 K.
- 2. Apply a chromatic adaptation transform such as CMCCAT2000 with parameters and to transform , , to corresponding values under the reference illuminant.
- 3. Transform to CIELAB values with respect to .
The procedure is the same for any other scene illuminant. The separate issue of the approximate uniformity of CIELAB space can be accommodated by replacing the Euclidean metric with a more accurate color-difference formula, e.g., CIEDE2000 [13,112,113].
For CIECAM02 space, color differences are defined by the Euclidean metric, i.e., , or some other orthogonal combination . But the comparison of color differences across scene illuminants is simpler than with CIELAB space because an accurate chromatic adaptation transform is built into CIECAM02 space. Its uniformity is also greater than that of CIELAB space, although it too can be improved .
C. Color Rendering Hyperspectral Images
To present a spectral radiance image on a display device requires a color space for rendering and some knowledge of the display device . Color rendering for image illustration can be treated as a form of spectral visualization .
With the default RGB color space sRGB , rendering proceeds as follows. First, radiance values are converted to CIE XYZ tristimulus values according to Eq. (7), but rescaled to range from 0 to 1 rather than the default from 0 to 100. Next, sRGB tristimulus values are obtained by linear transformation:
A nonlinear correction can be applied to compensate approximately for the input–output function of the display device, sometimes referred to as gamma encoding , despite its ambiguity . A typical approximate correction has the form21,116,119]. A key distinction between these transformations and those in Section 4.B is that color rendering transformations depend on the characteristics of the display device.
Figure 5, left, shows an sRGB rendering of a hyperspectral radiance image of a yellow flower (adapted from ). The image appears dark because the exposure timing used during image acquisition was limited by the specular highlight from the shiny leaf at the top right of the scene. To improve appearance, the sRGB levels of the image can be clipped to the level of a less specular but still bright region, such as the arrowed area on the light-gray sphere in the left image [121,122], and then scaled. Thus if is the maximum of the sRGB values in the arrowed area, the clipped and scaled sRGB values at each point are given by5, right. The detail in the darker areas is now much clearer. Clipping affects about 1% of the pixels in this scene. Clipping can, however, lead to artifacts, and more nuanced approaches exploit gamut-mapping algorithms to preserve the relationships between colors  and tone-mapping algorithms to preserve the detail in both light and dark regions .
5. RECEPTOR AND POSTRECEPTOR CODING
Estimating cone receptor and postreceptor responses to a spectral radiance image is more complicated than mapping points into a color space. The receptor mosaic is spatially non-uniform; preceptor absorption in the ocular media varies between individuals and with age; and chromatic adaptation and postreceptor coding take place at multiple levels. Calculations may, though, be simplified with the use of standardized observer data and viewing conditions .
As with colorimetric representations, there is generally no easy relationship between the RGB image of a scene recorded by a color camera and the corresponding receptor or postreceptor responses, standardized or not.
To simplify notation, the explicit dependence of cone signals on position (, ) is suppressed.
A. Receptor Responses
For a 2° standard observer , the calculation of normalized cone excitations is exactly analogous to the calculation of CIE XYZ tristimulus values (Section 4), since both are transforms of the color-matching functions.
Let be a spectral radiance image, indexed by spatial coordinates , and wavelength . If are the long-, medium-, and short-wavelength-sensitive cone spectral sensitivities, measured at the cornea, i.e., incorporating preceptor absorption, then at each point , the corresponding cone excitations are given by123]. Cone excitations may instead be calculated by linear transformation of the corresponding XYZ tristimulus values [19,21].
Individual differences in observer preceptor effects may be accommodated by replacing cone spectral sensitivities measured at the cornea by pigment spectral sensitivities combined with individual lens absorption and macular pigment data, as described in Refs. [124,125].
Because of the variation in receptor density and receptor type across the retina, assumptions may need to be made about how the gaze of the observer samples the spectral radiance image. For vision outside the central fovea, the intrusion of rod signals [126,127] and melanopsin-generated signals from intrinsically photosensitive retinal ganglion cells [128,129] may also need to be considered.
B. Postreceptor Transformations
The cone excitations estimated at each point may be transformed in several ways, according to the requirements of the simulation or analysis and whether cone-opponency or color-opponency is relevant .
Given the large overlap in long- and medium-wavelength-sensitive cone spectral sensitivities, a frequent computational objective is to reduce redundancy. Other objectives may be to optimize chromatic adaptation or to maximize the information that can be extracted from scenes. The postreceptor transformations are often assumed to be linear, albeit sometimes preceded by a nonlinear stage .
An early study by Buchsbaum and Gottschalk  provided an influential model . The postreceptor transformation is designed to linearly decorrelate cone signals , , in response to random combinations of monochromatic spectra. Cone spectral sensitivities are from the Vos-Walraven fundamentals , which differ a little from those due to Stockman and Sharpe [124,125]. The result is an achromatic response , a red–green chromatic response , and a yellow–blue chromatic response ; thus,
In Ref. , the postreceptor transformation maximally constrains responses to narrow bands of wavelengths. These sharpened spectral sensitivities improve the action of von Kries scaling in compensating for the effects of illuminant changes with Munsell reflectance spectra. The cone fundamentals are derived from [134–136]. The postreceptor responses are given by137], the postreceptor transformation maximizes the Shannon information available from natural scenes under simulated daylight changes. Responses are assumed to be subjected to von Kries scaling and values are therefore normalized to a mean of 1.0. The cone fundamentals are from [124,125]. The postreceptor responses are given by 138,139]. Still, with variable coefficients, they enable performance to be optimized with respect to specific functional objectives.
By contrast, simpler, fixed, postreceptor transformations may be used to explore possible psychophysical implications of the data, e.g., by mapping cone excitations into a particular color space such as the MacLeod–Boynton chromaticity diagram  or the color space due to Derrington, Krauskopf, and Lennie , as illustrated in Refs. [18,138]. The same caveat about neural correlates applies .
6. PROPERTIES OF SCENE COLORS
Some of the most basic color properties of a scene have to do with the existence and stability of colors, e.g., how many distinguishable colors it contains, the constancy of its colors under changes in illumination, and the presence of aliased colors, i.e., metamerism.
Since color is used here as a proxy for spectral reflectance or radiance, there is no reference to the spatial characteristics of the scene, at least explicitly, though procedures intended to compensate for illumination changes may well do so. Critically, these color properties set a least upper bound on what observers can infer about scenes from surface color alone.
Elsewhere, interest may be in color phenomena that do relate to spatial characteristics. For example, an extension of CIELAB known as S-CIELAB [142,143] incorporates preprocessing by the approximate spatial contrast sensitivity functions of achromatic and chromatic visual pathways. Some of the relationships between the chromatic and spatial properties of natural scenes are analyzed in Refs. [144–147].
The material in this section is based partly on .
A. Number of Distinguishable Colors
Because natural spectra come from a restricted population of reflecting surfaces and illuminants, the number of distinguishable colors in any one scene is likely to be very much smaller than the maximum of 1.7–2.3 million associated with an idealized population, the theoretical object color solid [149–152].
How, then, to estimate the number of distinguishable colors, N say, in a given scene? With an approximately uniform color space such as CIELAB or CIECAM02 and a nominal observer threshold color difference, say, an estimate of can be obtained from a hyperspectral radiance image, or from a reflectance image in combination with a global illuminant, as follows:
- 1. Map each point of the scene into the color space.
- 2. Segment the color space into unit cells, most simply cubes of side equal to the threshold interval .
- 3. Count the number of occupied cells.
Crucially, this estimate of takes no account of how often each color appears or where it comes from in the scene. All that is required is that a cell in the color space is occupied [35,153]. The number of occupied cells defines the number of points that on average are separated by at least .
The estimate of does, of course, depend on the metric used to define distances in the color space, the size of the discrimination threshold , and the method used to define the unit cell. In traditional color discrimination measurements, representative values for are typically 1.0 for CIELAB [154,155] and 0.5–0.6 with the more accurate color-difference formula CIEDE2000 [156,157]. In practical applications, though, representative values may be set several times larger. Unit cells may be cubes, spheres, or dodecahedra , and the distribution of cells need not form a regular array . More realistically, thresholds may be defined probabilistically .
Incidentally, nomenclature in this area varies. Along with colors being said to be distinguishable , they may also be called distinct , discernible [149,159], and discriminable [148,161]. Their description here as distinguishable is for compatibility with more general applications in information theory .
Based on 50 close-up and distant hyperspectral radiance images of rural and urban scenes, the average number of distinguishable colors per scene is found to be approximately , for spherical unit cells and the color-difference formula CIEDE2000 with discrimination threshold . This average number is almost an order of magnitude smaller than with the theoretical object color solid. Similar calculations can be performed with other kinds of hyperspectral images, e.g., of artworks .
Where illumination can be controlled, the number of distinguishable colors can be evaluated under different illuminants, e.g., incandescent, fluorescent, and metal halide lamps, and LED sources , which can be optimized for observers with color deficiency .
B. Gamut Volume
The range or gamut of the colors in a scene can be quantified by the number of colors distinguishable by an observer. This measure is dimensionless but dependent on the size of the threshold color difference , as explained in Section 6.A. More commonly, however, the color gamut is expressed as a volume in a color space . Other issues need then to be addressed such as whether points are treated as members of a discrete or continuous set and whether the boundary of the set should form a convex hull. Several estimation methods are available [122,164].
Values of the gamut volume also depend on the dimensions of the chosen color space, typically CIELAB or CIECAM02 space . In CIECAM02 space, gamut volume is measured in cubic CIECAM02 units .
Given an independent estimate of the gamut volume for some scene, the number of distinguishable colors may be estimated not by the cell-counting method in Section 6.A, but by dividing by the volume of the unit cell, in the same color space; i.e., if the unit cell is a cube of side , then7.A.
C. Color Constancy
Color constancy refers to the effect whereby the perceived or apparent color of a surface remains constant despite changes in the intensity and spectral composition of the illumination [120,166,167]. This definition is not the same as that sometimes used in computational color science, where it refers to the estimation of a representative illuminant on the scene [84,168], i.e., a global illuminant. The difference between real illumination changes and their approximation by global illuminants is discussed in Section 3.B.
1. Colorimetric Approaches
Consider the scene in Fig. 6. It shows a color rendering of a hyperspectral reflectance image of a terrace with flowers under a 4000 K daylight, top left, and under a 6500 K daylight, top right. Enlarged copies of the square areas marked in white are shown bottom left and bottom right. Suppose that under the 4000 K daylight, the sample point on the flower, arrowed in yellow, bottom left, has CIE tristimulus values and that under the 6500 K daylight, the same point arrowed in yellow, bottom right, has tristimulus values (the white arrowed points are discussed later). Suppose, next, that as the result of some color-constancy operation applied to a trichromatic representation of the scene under the 4000 K daylight, the sample point takes on tristimulus values under the 6500 K daylight. In general, these values only approximate the true values . To quantify the error (Section 4.B), transform and to corresponding CIELAB values and with respect to the 6500 K daylight and evaluate the color difference between and , possibly with a color-difference formula such as CIEDE2000. The color difference is sometimes called the color inconstancy index .
How good is the color constancy of the sample point on the flower if the constancy operation is supplied by the chromatic adaptation transform CMCCAT2000? The CIELAB color difference under the 6500 K daylight is 2.2, which is larger than the representative CIELAB threshold value of 1.0. With the color-difference formula CIEDE2000, a more accurate estimate of is 1.6, which is also larger than the corresponding representative threshold value of 0.5–0.6 (Section 6.A). Color constancy with CMCCAT2000 is therefore good under a moderate illuminant change, but not perfect.
More usually, the color difference is converted into a dimensionless color constancy index CI by scaling by the distance, say, between the two illuminants in the same color space and subtracting from unity , i.e., . Perfect constancy corresponds to an index of unity, and the complete absence of constancy corresponds to an index of zero. With the chromatic adaptation transform CMCCAT2000 and color-difference formula CIEDE2000, the color constancy index of the sample point on the flower is 0.92. With CIECAM02 space, it is 0.91.
Although the scene and its contents are not typical of those used in psychophysical measurements, these index values are compatible with the upper limit recorded for observers with a variety of experimental methods, materials, and illuminant changes [120,167].
To extend measures of color constancy from individual surfaces or points to the whole scene, global statistics may be deployed such as the mean and median of the distribution of index values.
2. Preserving Identity
Instead of using colorimetric measures of scene color constancy, there is a more utilitarian way of proceeding, namely, by measuring how far color constancy allows points to be identified by their color under a change in illuminant .
Consider, again, the sample point on the flower under the 4000 K daylight, arrowed in yellow in Fig. 6, bottom left. As already explained, under the 6500 K daylight, the same point arrowed in yellow, bottom right, has tristimulus values , and that with chromatic adaptation provided by CMCCAT2000, the estimated tristimulus values differ from the true values . But the color difference might be acceptable if were closer to than the tristimulus values of any other points in the scene—in other words, if constitutes a nearest-neighbor match according to some color-difference formula. In that event, color can still be used to identify the sample point uniquely despite the inaccuracy of the constancy operation. Notice that it is not the size of the color difference that counts but the risk of confusion.
Color constancy can then be quantified by the number of points in the scene that preserve their identity under the change in illuminant, i.e., the number of illuminant-invariant points. For a given constancy operation, this number obviously varies with the scene and the illuminant change.
A way to estimate the number of illuminant-invariant points is developed in Section 7.C.
Metamerism refers to the effect whereby two or more lights appear the same but have different spectral compositions [13,92]. It can arise in several forms, though the most important in practice is associated with surfaces. Metameric surfaces have different spectral reflectances but match in color under some illuminant, i.e., have the same tristimulus values. Since the match is likely to fail under some other illuminant , it follows that surface color is not a reliable guide to material identity. As with several other color phenomena, metamerism is a consequence of the number of degrees of freedom for reflectance and illumination spectra being greater than for their trichromatic representations.
The extent to which spectral reflectances are classified as the same or different is an empirical issue. Here, metameric surfaces are taken to be those for which color differences are subthreshold under one illuminant and suprathreshold by a certain multiple of threshold—the criterion degree of metamerism—under another illuminant .
The proportion or relative frequency of metameric surfaces in a scene may be calculated by taking a random sample of pairs of points from a hyperspectral reflectance image. Notice that the sample takes no account of what physically or semantically defines a surface, e.g., the flowers in Fig. 1 or the stone cottage in Fig. 3. Therefore, points from the same object in shadow and in direct illumination are treated distinctly. This situation differs from that described in Section 3.B in that pairs of points are considered as isolated samples undergoing a change in illuminant spectrum [8, Appendix A]. That said, omitting shadowed regions in these calculations seems to have little effect on relative frequency estimates [8, Section 3.F].
Two kinds of relative frequencies may be defined, both with respect to a nominal discrimination threshold. The relative frequency of metamerism is the proportion of times a pair of points chosen at random are indiscriminable under one illuminant but discriminable under another illuminant. The conditional relative frequency of metamerism is the proportion of times a pair of points chosen at random from pairs of points that are indiscriminable under one illuminant are discriminable under another illuminant.
How should these frequencies be estimated for a scene? Given a hyperspectral reflectance image, choose a pair of global illuminants, say daylights of 25,000 K and 4000 K. Choose an approximately uniform color space, CIELAB or CIECAM02, and a nominal threshold color difference . With CIELAB space, apply a chromatic adaptation transform such as CMCCAT2000 to first transform tristimulus values to a reference illuminant such as a 6500 K daylight, as explained in Section 4.B. Then proceed  as follows:
- 1. Choose a random sample of, say, pairs of points in the scene (the symbol N is used generically).
- 2. From this sample, find the number of pairs whose color differences under the 25,000 K daylight are less than the threshold .
- 3. From this subsample, find the number of pairs whose color differences under the 4000 K daylight are greater than or equal to (or some multiple, say, of , the criterion degree of metamerism).
- 4. The relative frequency of metamerism is .
- 5. The conditional relative frequency of metamerism is .
Each of these frequencies is a probability estimate and subject to sampling error. Even so, based on 50 close-up and distant hyperspectral reflectance images of urban and rural scenes under 25,000 K and 4000 K daylights, CIEDE2000 color difference thresholds and 1.0, and criterion degrees of metamerism , the relative frequency of metamerism appears to be low, from about to , depending on the criterion. In contrast, the conditional relative frequency is very much higher, from about to , sufficiently high to influence visual judgments. Predictably, daylights that have closer correlated color temperatures yield lower relative frequencies .
That there should be so large a difference between relative frequency and conditional relative frequency is to be expected, for in a scene with a moderately large gamut of colors, any two surfaces chosen at random are unlikely to match. Thus, for the 50 natural scenes used in Ref. , the median CIEDE2000 color difference between randomly chosen points under a 6500 K daylight is about 19.0, which is much greater than the threshold color difference of 0.5 or 1.0. On the other hand, given a pair of points that do match, i.e., have a color difference less than , they may well fail to match with a change in illuminant.
Broadly similar estimates of the relative frequency of metamerism are obtained by sampling from pooled collections of individual reflectance spectra taken from different natural and manufactured materials illuminated by natural and artificial lights .
7. UTILITY OF COLOR
Basic color properties of a scene such as the number of distinguishable colors and their constancy under changes in illumination do not depend on how often different reflecting surfaces appear, as long as the colors exist and are stable. Yet for color to be of practical use to an observer in distinguishing surfaces and identifying them under illumination changes, their differing relative frequencies do need to be taken into account.
The reason is fundamental. Relative frequencies determine to a greater or lesser extent the composition of any random sample of points from a scene and therefore their distinguishability. How distinguishability should be quantified can be framed in different ways , even without introducing cognitive factors . For the present purposes, however, information-theoretic methods  offer a fruitful guide to what is possible, in particular, the relationship between the number of distinguishable colors in a scene under constant illumination, the number of surfaces or points that can be distinguished by virtue of their color, and the number of points that can be identified under a change in illumination.
To simplify the exposition, technical notions are introduced only when necessary.
A. Entropy and Effective Gamut Volume
Consider the scene in Fig. 6, top right. Histogram estimates of the relative frequency distributions of its CIECAM02 color components are plotted in Fig. 7. The plot on the left is for the lightness component , in the middle for the red–green component , and on the right for the yellow–blue component , with the arrows indicating the maximum value of each component. All three histograms are peaked with the long tails in the positive direction for and due to the orange of the flowers.
These three histograms can be viewed as the marginal estimates of an underlying probability density function of a continuous random variable , say. The color triplets are the individual instances of . The uncertainty in may be quantified by an information-theoretic measure known as the Shannon differential entropy , defined  thus:176]. Other kinds of information measures are described in Ref. .
Methods of evaluating the integral in Eq. (10) are described in Section 7.D. It is emphasized that estimates of the differential entropy thereby obtained refer to the underlying continuous distributions, not the discrete hyperspectral samples that represent them.
Given a differential entropy , its logarithmic inverse (otherwise known as its exponential or antilogarithm) can be interpreted as the volume that defines the effective support set size of the random variable or, equivalently, of its probability density function [162,178,179]. In the present context, it defines the effective volume of the color gamut, which, as explained in Section 7.B, may be used to estimate the number of points that can be distinguished by their color. Still, as a consistency check, it is reasonable to ask whether for a suitable probability density function in Eq. (10) the definition of effective gamut volume coincides with the conventional definition of gamut volume.
Suppose that the colors in a scene are equally probable, unlike the colors in the scene in Fig. 6, top right. The marginal distributions, in particular, are then flat, by contrast with those in Fig. 7. Let be the gamut volume. Since the value in Eq. (10) is constant over all , its integral is unity, and so . Equation (10) then simplifies to
Since the differential entropy of an arbitrary distribution on a bounded set is always less than or equal to the differential entropy of a uniform distribution, the effective gamut volume is always less than or equal to the gamut volume.
As with gamut volume, the effective gamut volume depends on the dimensions of the color space. For the scene in Fig. 6, top right, its differential entropy is approximately 12.5 bits, corresponding to an effective CIECAM02 gamut volume of approximately , much smaller than the conventional CIECAM02 gamut volume of , calculated by empirically flattening the probability density function .
B. Number of Distinguishable Surfaces
Differential entropy provides a way to take into account the relative frequencies of colors in a scene. But how does it enable an estimate to be made of the number of surfaces or points that can be distinguished by their color?
As before, treat the CIECAM02 color triplets as instances of a three-dimensional continuous random variable, say, and the corresponding observer responses as instances of another three-dimensional continuous random variable, say. Notation here differs from . The amount of information that provides about is given  by the mutual information, written . It can be defined in several equivalent ways. The following uses the differential entropies and and the differential entropy of the distribution of and taken jointly:162].
Importantly, there is a simple relationship between mutual information and the approximate number of distinguishable points that can be sampled from a scene [162,178], and therefore retain their identity in the presence of uncertainty or internal noise in the observer (or color camera). This number, say, is given by the inverse logarithm of , i.e.,12), but it can be simplified. Suppose that observer internal noise is represented by a three-dimensional continuous random variable such that . With a hard threshold defined by the interval , the probability density function of is the uniform distribution on a cube with side (alternatives include the uniform distribution on a ball and a three-dimensional Gaussian noise distribution ). Providing that is not too large, then to a good approximation , the mutual information in Eq. (12) can be written as 7.A, the differential entropy is the logarithm of the effective gamut volume, and, by the same kind of calculation that led to Eq. (11), the differential entropy is the logarithm of the volume of the unit cell . Substituting these values into Eq. (14) gives 13) yields the following for the number of distinguishable points: 9). Each is the quotient of a gamut volume by the volume of a unit cell. Indeed Eq. (9) is a special case of Eq. (15) in which the differential entropy is derived for a flattened version of the probability density function , as in Section 7.A.
The number of distinguishable points can now be estimated for the scene in Fig. 6, top right. From Section 7.A, the differential entropy of the colors is approximately 12.5 bits, corresponding to an effective CIECAM02 gamut volume of approximately . If this volume is divided by the volume of the unit cell , with , then by Eq. (15), the number of distinguishable points is approximately . This is much smaller than the estimate of obtained by taking its CIECAM02 gamut volume of (Section 7.A) and dividing by the same volume of the unit cell .
These estimates are for only one scene. Values for 50 natural scenes are given in Ref. [148, Fig. 4].
In other applications, differential entropy is used to predict the relative frequency and magnitude of metamerism across scenes and the changes in relative color cues or color relations between surfaces under changes in illuminant [148,180]. In its discrete form , entropy is used to classify observers’ color-naming behavior with individual colored surfaces . Mutual information is additionally used to estimate the relationship between differences in luminance and differences in color within natural scenes  and to estimate the optimum spectral locations of cone photopigments .
C. Number of Illuminant-Invariant Points
A utilitarian way of measuring color constancy in a scene is outlined in Section 6.C.2, namely, as the number of points that preserve their identity under a change in illuminant. But how to estimate this number? Consider once more the scene in Fig. 6. One approach is to perform an exhaustive nearest-neighbor search for color matches, i.e., by counting all those points whose transformed CIE tristimulus values are closer to their true values than for any other points in the scene, with respect to some color-difference formula. But the result depends on the chromatic adaptation transform. Different transforms define different nearest-neighbor matches; moreover, nearest-neighbor matches themselves may not always be optimum .
Instead, information-theoretic methods may be used in a way similar to that for estimating the number of distinguishable points under constant illumination described in Section 7.B. Such an approach provides a least upper bound on all possible chromatic adaptation transforms and more generally on any color-constancy operation.
Again treat the CIECAM02 color triplets as instances of a three-dimensional continuous random variable, say, under the first illuminant and as a different three-dimensional continuous random variable, say, under the second illuminant. Suppose that and are the corresponding probability density functions of and . The amount of information that retains about is measured by the mutual information . In Section 7.B, it is used to determine the number of points that preserve their identity in the presence of observer internal noise. Here, it determines the number of points that retain their identity in the presence of what might be thought of as illuminant-change noise. This number, say, is given by the inverse logarithm of , i.e.,6, top, under 4000 K and 6500 K daylights, the estimated mutual information , calculated as in Sections 7.A and 7.B, is approximately 21.9 bits, corresponding to approximately illuminant-invariant points.
As before, these estimates are for one scene. Means and standard deviations for 50 natural scenes under different daylight illuminants are given in Ref. [137, Fig. 3(a)].
Although does not depend on linear transformations of the color space, it does depend on the type of color space, since the relationship between one space and another is usually nonlinear. In fact, with the chromatic adaptation transform CMCCAT2000 and color-difference formula CIEDE2000, the CIELAB estimate of for the scene in Fig. 6 is approximately 22.4 bits, just 0.5 bits larger than with CIECAM02. The corresponding number of illuminant-invariant points is approximately .
These estimates of the number of illuminant-invariant points for the scene in Fig. 6 are large but plausible, since they exclude observer uncertainty. To help interpret the magnitude of the illuminant-change noise, it can be expressed in terms of equivalent Gaussian variables . Suppose that the noise is represented by a three-dimensional continuous random variable , such that . Suppose further that the components of and of are distributed normally and independently with respective common variances and . Then, as shown in Ref. , the mutual information . It follows that . Therefore, given that is approximately 21.9 bits, the equivalent Gaussian signal-to-noise amplitude ratio is approximately 158, or, conversely, the percentage noise amplitude is about 0.63%. This value is smaller than measures of observer internal noise such as the Fechner fraction, which is on the order of 2% .
Suppose, now, that the estimate of the number of illuminant-invariant points includes observer uncertainty, as in Section 7.B. The mutual information between and the observer response to decreases, since there are now two sources of noise that are largely independent of each other, one due to an illuminant change and one due to the observer, i.e.,
Thus, to continue with the example of the scene in Fig. 6, top, suppose that the probability density function for observer internal noise is the uniform distribution on a cube of side . Then, the estimated mutual information , calculated as in Sections 7.A and 7.B, falls to approximately 11.8 bits, corresponding to approximately illuminant-invariant points, three orders of magnitude smaller than without observer uncertainty. Similar estimates are obtained by replacing the uniform distribution for observer internal noise by a three-dimensional Gaussian noise distribution with the same covariance.
In addition to spectral changes, non-spectral changes in illumination, e.g., changes in the distribution of shadows, specularities, and mutual reflections, along with other uncertainties (Section 3.C) can be interpreted as sources of noise, leading to further reductions in and in . The complexities of environmental illumination may also affect observer performance in other ways .
Notwithstanding the usefulness of mutual information and its equivalents, can colorimetric measures such as the color constancy index provide a similar guide to identifying surfaces or points under illuminant changes? Unfortunately, global statistics based on the mean or median of point-wise color differences do not suffice. The reason is evident with the scene in Fig. 6. With the chromatic adaptation transform CMCCAT2000, the tristimulus values of the sample point arrowed in yellow, bottom left, are transformed to the values , which, as noted in Section 6.C.1, differ from the true values of the sample point, arrowed in yellow, bottom right. These estimates are actually closer to the tristimulus values of each of the two points arrowed in white (and to the tristimulus values of many other points not indicated in the image), according to the color-difference formula CIEDE2000. A more detailed example is described in Ref. .
These kinds of erroneous matches occur with other points in the scene and with other chromatic adaptation transforms. Patently, identifying a point by its color involves not only the color difference between its estimated and true tristimulus values and , or their equivalents, but also the color differences between and the tristimulus values of all the other points in the scene.
Although mutual information sets a least upper bound on the number of illuminant-invariant points in a scene, it does not automatically follow that it is possible to find a chromatic adaptation transform that can reach that bound, even with illumination changes that are purely spectral. In practice [137,175], performance tends to fall several bits below the least upper bound. Locally weighted polynomial regression may offer a better description , though it risks overfitting .
Numerical estimates of differential entropy may be obtained directly from probability density functions, e.g., from empirical histograms. Examples of histogram estimates of the marginal distributions are shown in Fig. 7. But using histograms in place of the unknown probability density function in Eq. (10) can lead to marked biases in the estimates . Fortunately, differential entropy can be estimated from nearest-neighbor statistics, a safer approach than constructing empirical histograms. The Kozachenko–Leonenko th-nearest-neighbor estimator [187,188] is asymptotically unbiased and converges reasonably rapidly with increasing sample size. Its convergence is improved with an offset method, details of which are given in Ref. . The essence of the method is to decompose the estimate into two components: one the mutual information between equivalent Gaussian variables with known variance–covariance structure, and the other an offset obtained by applying the nearest-neighbor estimator to normalized versions of the variables.
8. COMMENT AND CONCLUSION
Terrestrial and close-range hyperspectral imaging makes possible the realistic analysis and modeling of color vision in the environment. Cone receptor and postreceptor responses can be estimated, scene color statistics calculated, and the utility of color assessed in tasks requiring an observer to distinguish surfaces or identify them under changes in illumination. In addition, colorimetric formulas can be tested and color rendering transformations optimized.
Caution does, though, need to be exercised in securing images and in analyzing them:
- 1. To be valid, hyperspectral images should be linearly related to the physical quantities they represent. The workflow for image acquisition, processing, and calibration demands careful, accurate management.
- 2. The choice of spectral radiance or reflectance images leads to different tradeoffs. Spectral reflectance data allow a wider range of questions to be addressed but entail more assumptions or constraints than spectral radiance data.
- 3. In modeling receptor responses to images, variations in receptor density and type may need to be considered, along with the intrusion of signals from rods and intrinsically photosensitive retinal ganglion cells.
- 4. In assessing the utility of color in different observer tasks, average measures of scene properties such as the mean or median of color signals may be less relevant than information measures such as Shannon differential entropy. But estimating differential entropy with little or no bias depends on the goodness of the chosen numerical estimator.
More generally, each spectral radiance or reflectance image constitutes a single sample from the real world in which reflecting properties may vary from point to point and reflected light from instant to instant. Inferences drawn from such data are inevitably probabilistic.
Similar caveats apply to extensions of hyperspectral imaging, including stereo, goniometric, and video hyperspectral imaging. Constraints on acquisition time may force reductions in the spatial or spectral sampling density available to the imaging system. Nevertheless, preserving the relevant degrees of freedom associated with natural scene data remains a prerequisite.
Engineering and Physical Sciences Research Council (EPSRC) (EP/F023669/1, EP/B000257/1, EP/E056512/1, GR/R39412/01).
We are grateful to S. M. C. Nascimento for collaborating in the acquisition and processing of the hyperspectral images used in this study and to the Confraria do Bom Jesus do Monte for permission to acquire images on their grounds. We thank A. R. D. Smedley and A. R. Webb for advice on surface radiation. We also thank J. Hernández Andrés, J. M. M. Linhares, I. Marín-Franch, J. L. Nieves, and S. Westland for critical reading of the manuscript. Some of the hyperspectral images used in this tutorial are available at http://personalpages.manchester.ac.uk/staff/d.h.foster/ and http://online.uminho.pt/pessoas/smcn/.
1. H. F. Grahn and P. Geladi, eds., Techniques and Applications of Hyperspectral Image Analysis (Wiley, 2007).
2. N. Hagen and M. W. Kudenov, “Review of snapshot spectral imaging technologies,” Opt. Eng. 52, 090901 (2013). [CrossRef]
3. L. Arend, “Environmental challenges to color constancy,” in Human Vision and Electronic Imaging VI, B. E. Rogowitz and T. N. Pappas, eds. (SPIE, 2001), pp. 392–399.
4. Munsell Color Company, Munsell Book of Color–Matte Finish Collection (Munsell Color Corporation, 1976).
5. J. Hernández-Andrés, J. Romero, and J. L. Nieves, “Color and spectral analysis of daylight in southern Europe,” J. Opt. Soc. Am. A 18, 1325–1335 (2001). [CrossRef]
6. UNESCO, International Classification and Mapping of Vegetation (UNESCO, 1973).
7. Federal Geographic Data Committee, “Vegetation classification standard,” U.S. Geological Survey, FGDC-STD-005, 1997.
8. D. H. Foster, K. Amano, S. M. C. Nascimento, and M. J. Foster, “Frequency of metamerism in natural scenes,” J. Opt. Soc. Am. A 23, 2359–2372 (2006). [CrossRef]
9. F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W. Ginsberg, and T. Limperis, Geometrical considerations and nomenclature for reflectance (Institute for Basic Standards/National Bureau of Standards, 1997).
10. C. L. Wyatt, V. Privalsky, and R. Datla, Recommended practice: symbols, terms, units and uncertainty analysis for radiometric sensor calibration (National Institute of Standards and Technology, U.S. Department of Commerce Technology Administration, 1998).
11. W. L. Wolfe, “Glossary and fundamental constants,” in Handbook of Optics, Volume 1, Geometrical and Physical Optics, Polarized Light, Components and Instruments, M. Bass, ed., 3rd ed. (McGraw Hill, 2010), pp. xxix–xxxiv.
12. CIE, “A colour appearance model for colour management systems: CIECAM02,” CIE Central Bureau, CIE Publication 159:2004, 2004.
13. CIE, “Colorimetry, 4th Edition,” CIE Central Bureau, CIE Publication 015:2018, 2018.
14. F. Gori and P. S. Carney, “Introducing JOSA A tutorials: editorial,” J. Opt. Soc. Am. A 32, ED3 (2015). [CrossRef]
15. V. C. Coffey, “Hyperspectral imaging for safety and security,” Opt. Photon. News 26, 28–33 (2015).
16. T. H. Kurz and S. J. Buckley, “A review of hyperspectral imaging in close range applications,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLI-B5, 865–870 (2016). [CrossRef]
17. Q. Li, X. He, Y. Wang, H. Liu, D. Xu, and F. Guo, “Review of spectral imaging technology in biomedical engineering: achievements and challenges,” J. Biomed. Opt. 18, 100901 (2013). [CrossRef]
18. S. K. Shevell, ed., The Science of Color, 2nd ed. (Elsevier, 2003).
19. D. H. Brainard and A. Stockman, “Colorimetry,” in Handbook of Optics. Volume III. Vision and Vision Optics, M. Bass, ed., 3rd ed. (McGraw Hill, 2010), pp. 10.1–10.56.
20. D. H. Foster, “Chromatic function of the cones,” in Encyclopedia of the Eye, D. A. Dartt, ed. (Academic, 2010), pp. 266–274.
21. S. Westland, C. Ripamonti, and V. Cheung, Computational Colour Science Using Matlab, 2nd ed. (Wiley, 2012).
22. M. D. Fairchild, Color Appearance Models, 2nd ed. (Wiley, 2013).
23. K. Barnard and B. Funt, “Camera characterization for color research,” Color Res. Appl. 27, 152–163 (2002). [CrossRef]
24. V. C. Cardei and B. Funt, “Color correcting uncalibrated digital images,” J. Imaging Sci. Technol. 44, 288–378 (2000).
25. J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41, 2532–2548 (2002). [CrossRef]
26. P. Geladi, J. Burger, and T. Lestlander, “Hyperspectral imaging: calibration problems and solutions,” Chemom. Intell. Lab. Syst. 72, 209–217 (2004). [CrossRef]
27. M. Brady and G. E. Legge, “Camera calibration for natural image studies and vision research,” J. Opt. Soc. Am. A 26, 30–42 (2009). [CrossRef]
28. M. A. López-Álvarez, J. Hernández-Andrés, J. Romero, J. Campos, and A. Pons, “Calibrating the elements of a multispectral imaging system,” J. Imaging Sci. Technol. 53, 031102 (2009). [CrossRef]
29. N. Ekpenyong, Hyperspectral Imaging: Calibration and Applications with Natural Scenes (School of Electrical and Electronic Engineering University of Manchester, 2013).
30. P. Ziemer, “Design and implementation of a multispectral imaging system,” M.Sc. thesis (Department of Computer Science and Information Science, Universität Konstanz, 2013).
31. J. R. Janesick, Photon Transfer DN → λ (SPIE, 2007).
32. S. B. Howell, Handbook of CCD Astronomy, 2nd ed., Cambridge Observing Handbooks for Research Astronomers (Cambridge University, 2006).
33. S. Le Moan and P. Urban, “Image-difference prediction: from color to spectral,” IEEE Trans. Image Process. 23, 2058–2068 (2014). [CrossRef]
34. J. M. M. Linhares, P. D. A. Pinto, and S. M. C. Nascimento, “Color rendering of art paintings under CIE illuminants for normal and color deficient observers,” J. Opt. Soc. Am. A 26, 1668–1677 (2009). [CrossRef]
35. C. Montagner, J. M. M. Linhares, M. Vilarigues, and S. M. C. Nascimento, “Statistics of colors in paintings and natural scenes,” J. Opt. Soc. Am. A 33, A170–A177 (2016). [CrossRef]
36. S. M. C. Nascimento, J. M. M. Linhares, C. Montagner, C. A. R. João, K. Amano, C. Alfaro, and A. Bailão, “The colors of paintings and viewers’ preferences,” Vis. Res. 130, 76–84 (2017). [CrossRef]
37. H. A. Khan, S. Mihoubi, B. Mathon, J.-B. Thomas, and J. Y. Hardeberg, “HyTexiLa: high resolution visible and near infrared hyperspectral texture images,” Sensors 18, 2045 (2018). [CrossRef]
38. R. Ennis, F. Schiller, M. Toscani, and K. R. Gegenfurtner, “Hyperspectral database of fruits and vegetables,” J. Opt. Soc. Am. A 35, B256–B266 (2018). [CrossRef]
39. K. S. Babu, V. Ramachandran, K. K. Thyagharajan, and G. Santhosh, “Hyperspectral image compression algorithms—a review,” in International Conference on Artificial Intelligence and Evolutionary Algorithms in Engineering Systems (ICAEES), L. P. Suresh, S. S. Dash, and B. K. Panigrahi, eds. (Springer, 2015), Vol. 2, pp. 127–138.
40. M. Paul, R. Xiao, J. B. Gao, and T. Bossomaier, “Reflectance prediction modelling for residual-based hyperspectral image coding,” PLoS ONE 11, e0161212 (2016). [CrossRef]
41. R. Dusselaar and M. Paul, “Hyperspectral image compression approaches: opportunities, challenges, and future directions: discussion,” J. Opt. Soc. Am. A 34, 2170–2180 (2017). [CrossRef]
42. M. Zucco, M. Pisani, V. Caricato, and A. Egidi, “A hyperspectral imager based on a Fabry–Perot interferometer with dielectric mirrors,” Opt. Express 22, 1824–1834 (2014). [CrossRef]
43. L. Gao and L. V. Wang, “A review of snapshot multidimensional optical imaging: measuring photon tags in parallel,” Phys. Rep. 616, 1–37 (2016). [CrossRef]
44. T. Hakala, J. Suomalainen, S. Kaasalainen, and Y.-W. Chen, “Full waveform hyperspectral LiDAR for terrestrial laser scanning,” Opt. Express 20, 7119–7127 (2012). [CrossRef]
45. F. H. Imai and R. S. Berns, “Spectral estimation using trichromatic digital cameras,” in International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives (Society of Multispectral Imaging of Japan, 1999), pp. 42–49.
46. E. M. Valero, J. L. Nieves, S. M. C. Nascimento, K. Amano, and D. H. Foster, “Recovering spectral data from natural scenes with an RGB digital camera and colored filters,” Color Res. Appl. 32, 352–360 (2007). [CrossRef]
47. J. L. Nieves, E. M. Valero, J. Hernández-Andrés, and J. Romero, “Recovering fluorescent spectra with an RGB digital camera and color filters using different matrix factorizations,” Appl. Opt. 46, 4144–4154 (2007). [CrossRef]
48. F. H. Imai, M. R. Rosen, and R. S. Berns, “Comparison of spectrally narrow-band capture versus wide-band with a priori sample analysis for spectral reflectance estimation,” in Eighth Color Imaging Conference: Color Science and Engineering: Systems, Technologies and Applications (Society for Imaging Science and Technology, 2000), pp. 234–241.
49. J. M. Lerner, N. Gat, and E. Wachman, “Approaches to spectral imaging hardware,” Curr. Protoc. Cytom. 53, 12–20 (2010). [CrossRef]
50. J. Eckhard, T. Eckhard, E. M. Valero, J. L. Nieves, and E. G. Contreras, “Outdoor scene reflectance measurements using a Bragg-grating-based hyperspectral imager,” Appl. Opt. 54, D15–D24 (2015). [CrossRef]
51. D. Bannon, “Hyperspectral imaging: cubes and slices,” Nat. Photonics 3, 627–629 (2009). [CrossRef]
52. M. Vakalopoulou and K. Karantzalos, “Automatic descriptor-based co-registration of frame hyperspectral data,” Remote Sens. 6, 3409–3426 (2014). [CrossRef]
53. P. Jerram, D. Burt, D. Morris, T. Eaton, and M. Fryer, “Design of image sensors for hyperspectral applications,” in Sensors, Systems, and Next-Generation Satellites XIII, R. Meynart, S. P. Neeck, and H. Shimoda, eds. (SPIE, 2009), pp. 74741E1.
54. L. Galvis, H. Arguello, and G. R. Arce, “Coded aperture design in mismatched compressive spectral imaging,” Appl. Opt. 54, 9875–9882 (2015). [CrossRef]
55. X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q.-H. Dai, L. Carin, and D. J. Brady, “Computational snapshot multispectral cameras: toward dynamic capture of the spectral world,” IEEE Signal Process. Mag. 33(5), 95–108 (2016). [CrossRef]
56. X.-B. Wang, P. J. Green, J.-B. Thomas, J. Y. Hardeberg, and P. Gouton, “Evaluation of the colorimetric performance of single-sensor image acquisition systems employing colour and multispectral filter array,” in Computational Color Imaging, CCIW, A. Trémeau, R. Schettini, and S. Tominaga, eds. (Springer, 2015), pp. 181–191.
57. C. V. Correa, C. A. Hinojosa, G. R. Arce, and H. Arguello, “Multiple snapshot colored compressive spectral imager,” Opt. Eng. 56, 041309 (2016). [CrossRef]
58. L.-L. Qian, Q.-B. Lü, M. Huang, Q.-S. Cai, and B. Xiang-li, “Effect of keystone on coded aperture spectral imaging,” Optik 127, 686–689 (2016). [CrossRef]
59. R. Zhen and R. L. Stevenson, “Image demosaicing,” in Color Image and Video Enhancement, M. E. Celebi, M. Lecca, and B. Smolka, eds. (Springer, 2015), pp. 13–22.
60. A. Mansouri, F. S. Marzani, and P. Gouton, “Development of a protocol for CCD calibration: application to a multispectral imaging system,” Int. J. Robot. Autom. 20, 94–100 (2005).
61. H. Liang, “Advances in multispectral and hyperspectral imaging for archaeology and art conservation,” Appl. Phys. A 106, 309–323 (2012). [CrossRef]
62. M. R. Pointer, G. G. Attridge, and R. E. Jacobson, “Practical camera characterization for colour measurement,” Imaging Sci. J. 49, 63–80 (2001). [CrossRef]
63. V. Cheung and S. Westland, “Methods for optimal color selection,” J. Imaging Sci. Technol. 50, 481–488 (2006). [CrossRef]
64. J. M. Palmer, “Radiometry and photometry: units and conversions,” in Handbook of Optics, Volume II, Design, Fabrication, and Testing; Sources and Detectors; Radiometry and Photometry, M. Bass, ed., 3rd ed. (McGraw Hill, 2010), pp. 36.1–36.19.
65. T. Luhmann, C. Fraser, and H.-G. Maas, “Sensor modelling and camera calibration for close-range photogrammetry,” ISPRS J. Photogramm. Remote Sens. 115, 37–46 (2016). [CrossRef]
66. J. Salvi, X. Armangué, and J. Batlle, “A comparative review of camera calibrating methods with accuracy evaluation,” Pattern Recogn. 35, 1617–1635 (2002). [CrossRef]
67. T. F. Blake, S. C. Cain, and M. E. Goda, “Enhancing the resolution of spectral images from the advanced electro-optical system spectral imaging sensor,” Opt. Eng. 46, 057001 (2007). [CrossRef]
68. F. Toadere, “Simulating the functionality of a digital camera pipeline,” Opt. Eng. 52, 102005 (2013). [CrossRef]
69. S. Watanabe, T. Takahashi, and K. Bennett, “Quantitative evaluation of the accuracy and variance of individual pixels in a scientific CMOS (sCMOS) camera for computational imaging,” Proc. SPIE 10071, 100710Z (2017). [CrossRef]
70. J. Burger and P. Geladi, “Hyperspectral NIR image regression, part I: calibration and correction,” J. Chemom. 19, 355–363 (2005). [CrossRef]
71. P. A. Jansson and R. P. Breault, “Correcting color-measurement error caused by stray light in image scanners,” in Sixth Color and Imaging Conference: Color Science, Systems, and Applications (Society for Imaging Science and Technology, 1998), pp. 69–73.
72. S. Helling, “Improvement of multispectral image capture by compensating for stray light,” in CGIV 2006, 3rd European Conference on Colour in Graphics, Imaging, and Vision (Society for Imaging Science and Technology, 2006), pp. 458–462.
73. H. S. Fairman, “An improved method for correcting radiance data for bandpass error,” Color Res. Appl. 35, 328–333 (2010). [CrossRef]
74. N. Gat, “Imaging spectroscopy using tunable filters: a review,” Proc. SPIE 4056, 50–64 (2000). [CrossRef]
75. R. Kingslake and R. B. Johnson, Lens Design Fundamentals, 2nd ed. (Academic/SPIE, 2010).
76. B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21, 977–1000 (2003). [CrossRef]
77. T. Eckhard, J. Eckhard, E. M. Valero, and J. L. Nieves, “Nonrigid registration with free-form deformation model of multilevel uniform cubic B-splines: application to image registration and distortion correction of spectral image cubes,” Appl. Opt. 53, 3764–3772 (2014). [CrossRef]
78. L. Laaksonen, E. Claridge, P. Fält, M. Hauta-Kasari, H. Uusitalo, and L. Lensu, “Comparison of image registration methods for composing spectral retinal images,” Biomed. Signal Process. Control 36, 234–245 (2017). [CrossRef]
79. D. G. Abdelsalam, M. Stanislas, and S. Coudert, “CCD or CMOS camera calibration using point spread function,” Proc. SPIE 9234, 92340Z (2014). [CrossRef]
80. E. Buhr, S. Günther-Kohfahl, and U. Neitzel, “Simple method for modulation transfer function determination of digital imaging detectors from edge images,” Proc. SPIE 5030, 877–884 (2003). [CrossRef]
81. J. Brauers, C. Seiler, and T. Aach, “Direct PSF estimation using a random noise target,” Proc. SPIE 7537, 75370B (2010). [CrossRef]
82. J. R. Miller, S. C. Jain, N. T. O’Neill, W. R. McNeil, and K. P. B. Thomson, “Interpretation of airborne spectral reflectance measurements over Georgian Bay,” Remote Sens. Environ. 6, 183–200 (1977). [CrossRef]
83. D. H. Brainard and L. T. Maloney, “Surface color perception and equivalent illumination models,” J. Vis. 11(5), 1 (2011). [CrossRef]
84. H. A. Khan, J.-B. Thomas, J. Y. Hardeberg, and O. Laligant, “Illuminant estimation in multispectral imaging,” J. Opt. Soc. Am. A 34, 1085–1098 (2017). [CrossRef]
85. B. V. Funt, M. S. Drew, and J. Ho, “Color constancy from mutual reflection,” Int. J. Comput. Vis. 6, 5–24 (1991). [CrossRef]
86. S. Tominaga, K. Kato, K. Hirai, and T. Horiuchi, “Spectral image analysis of mutual illumination between florescent objects,” J. Opt. Soc. Am. A 33, 1476–1487 (2016). [CrossRef]
87. B. V. Funt and M. S. Drew, “Color space analysis of mutual illumination,” IEEE Trans. Pattern Anal. Mach. Intell. 15, 1319–1326 (1993). [CrossRef]
88. R. Deeb, D. Muselet, M. Hebert, and A. Tremeau, “Interreflections in computer vision: a survey and an introduction to spectral infinite-bounce model,” J. Math. Imaging Vis. 60, 661–680 (2018). [CrossRef]
89. H. K. Lichtenthaler and U. Rinderle, “The role of chlorophyll fluorescence in the detection of stress conditions in plants,” CRC Crit. Rev. Anal. Chem. 19, S29–S85 (1988). [CrossRef]
90. P. K. E. Campbell, E. M. Middleton, L. A. Corp, and M. S. Kim, “Contribution of chlorophyll fluorescence to the apparent vegetation reflectance,” Sci. Total Environ. 404, 433–439 (2008). [CrossRef]
91. G. Schaepman-Strub, M. E. Schaepman, T. H. Painter, S. Dangel, and J. V. Martonchik, “Reflectance quantities in optical remote sensing—definitions and case studies,” Remote Sens. Environ. 103, 27–42 (2006). [CrossRef]
92. G. Wyszecki and W. S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd ed. (Wiley, 1982).
93. D. H. Foster, K. Amano, and S. M. C. Nascimento, “Time-lapse ratios of cone excitations in natural scenes,” Vis. Res. 120, 45–60(2016). [CrossRef]
94. A. Gijsenij, R. Lu, and T. Gevers, “Color constancy for multiple light sources,” IEEE Trans. Image Process. 21, 697–707 (2012). [CrossRef]
95. L. Gu, C. P. Huynh, and A. Robles-Kelly, “Segmentation and estimation of spatially varying illumination,” IEEE Trans. Image Process. 23, 3478–3489 (2014). [CrossRef]
96. R. C. Love, Surface Reflection Model Estimation from Naturally Illuminated Image Sequences (School of Computer Studies, University of Leeds, 1997).
97. J. H. McClendon, “The micro-optics of leaves. I. Patterns of reflection from the epidermis,” Am. J. Bot. 71, 1391–1397 (1984). [CrossRef]
98. D. A. Sims and J. A. Gamon, “Relationships between leaf pigment content and spectral reflectance across a wide range of species, leaf structures and developmental stages,” Remote Sens. Environ. 81, 337–354 (2002). [CrossRef]
99. D. Lee, Nature’s Palette: The Science of Plant Color (University of Chicago, 2007).
100. J. B. Clark and G. R. Lister, “Photosynthetic action spectra of trees. II. The relationship of cuticle structure to the visible and ultraviolet spectral properties of needles from four coniferous species,” Plant Physiol. 55, 407–413 (1975). [CrossRef]
101. J. H. McClendon and L. Fukshansky, “On the interpretation of absorption spectra of leaves. I. Introduction and the correction of leaf spectra for surface reflection,” Photochem. Photobiol. 51, 203–210 (1990). [CrossRef]
102. J. H. McClendon and L. Fukshansky, “On the interpretation of absorption spectra of leaves. II. The non-absorbed ray of the sieve effect and the mean optical pathlength in the remainder of the leaf,” Photochem. Photobiol. 51, 211–216 (1990). [CrossRef]
103. S. M. C. Nascimento, K. Amano, and D. H. Foster, “Spatial distributions of local illumination color in natural scenes,” Vis. Res. 120, 39–44 (2016). [CrossRef]
104. R. Heylen, M. Parente, and P. Gader, “A review of nonlinear hyperspectral unmixing methods,” IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 7, 1844–1868 (2014). [CrossRef]
105. J. M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader, and J. Chanussot, “Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based approaches,” IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 5, 354–379 (2012). [CrossRef]
106. D. K. Prasad and L. Wenhe, “Metrics and statistics of frequency of occurrence of metamerism in consumer cameras for natural scenes,” J. Opt. Soc. Am. A 32, 1390–1402 (2015). [CrossRef]
107. R. Bala, G. Finlayson, and C. Lee, “Computational color imaging,” in Handbook of Convex Optimization Methods in Imaging Science, V. Monga, ed. (Springer, 2017), pp. 43–70.
108. H. Terstiege, “Chromatic adaptation: a state-of-the-art report,” J. Color Appearance 1, 19–23, 40 (1972).
109. G. D. Finlayson, M. S. Drew, and B. V. Funt, “Spectral sharpening: sensor transformations for improved color constancy,” J. Opt. Soc. Am. A 11, 1553–1563 (1994). [CrossRef]
110. C. Li, M. R. Luo, B. Rigg, and R. W. G. Hunt, “CMC 2000 chromatic adaptation transform: CMCCAT2000,” Color Res. Appl. 27, 49–58 (2002). [CrossRef]
111. K. A. G. Smet, M. A. Webster, and L. A. Whitehead, “A simple principled approach for modeling and understanding uniform color metrics,” J. Opt. Soc. Am. A 33, A319–A331 (2016). [CrossRef]
112. M. R. Luo, G. Cui, and B. Rigg, “The development of the CIE 2000 colour-difference formula: CIEDE2000,” Color Res. Appl. 26, 340–350 (2001). [CrossRef]
113. M. Melgosa, R. Huertas, and R. S. Berns, “Performance of recent advanced color-difference formulas using the standardized residual sum of squares index,” J. Opt. Soc. Am. A 25, 1828–1834 (2008). [CrossRef]
114. M. R. Luo, G. Cui, and C. Li, “Uniform colour spaces based on CIECAM02 colour appearance model,” Color Res. Appl. 31, 320–330 (2006). [CrossRef]
115. C. Li, Z. Li, Z. Wang, Y. Xu, M. R. Luo, G. Cui, M. Melgosa, M. H. Brill, and M. Pointer, “Comprehensive color solutions: CAM16, CAT16, and CAM16-UCS,” Color Res. Appl. 42, 703–718 (2017). [CrossRef]
116. E. Reinhard, E. A. Khan, A. O. Akyüz, and G. M. Johnson, Color Imaging: Fundamentals and Applications (A. K. Peters, 2008).
117. S. Le Moan, A. Mansouri, Y. Voisin, and J. Y. Hardeberg, “A constrained band selection method based on information measures for spectral image color visualization,” IEEE Trans. Geosci. Remote Sens. 49, 5104–5115 (2011). [CrossRef]
118. IEC, “Colour management in multimedia systems–Part 2: colour management, Part 2.1: default RGB colour space–sRGB,” International Electrotechnical Commission, IEC/4WD 61966-2-1, 1998.
119. W. Cowan, “Displays for vision research,” in Handbook of Optics. Volume III. Vision and Vision Optics, M. Bass, ed., 3rd ed. (McGraw Hill, 2010), pp. 22.1–22.41.
120. D. H. Foster, “Color constancy,” Vis. Res. 51, 674–700 (2011). [CrossRef]
121. W. K. Pratt, Digital Image Processing, 4th ed. (Wiley, 2007).
122. J. Morovič, Color Gamut Mapping (Wiley, 2008).
123. A. Stockman and L. T. Sharpe, “Cone spectral sensitivities and color matching,” in Color Vision: From Genes To Perception, K. R. Gegenfurtner and L. T. Sharpe, eds. (Cambridge University, 1999), pp. 53–87.
124. A. Stockman, L. T. Sharpe, and C. Fach, “The spectral sensitivity of the human short-wavelength sensitive cones derived from thresholds and color matches,” Vis. Res. 39, 2901–2927 (1999). [CrossRef]
125. A. Stockman and L. T. Sharpe, “The spectral sensitivities of the middle- and long-wavelength-sensitive cones derived from measurements in observers of known genotype,” Vision Res. 40, 1711–1737 (2000). [CrossRef]
126. T. D. Lamb, “Why rods and cones?” Eye 30, 179–185 (2016). [CrossRef]
127. A. J. Zele and D. Cao, “Vision under mesopic and scotopic illumination,” Front. Psychol. 5, 1594, 1–15 (2015). [CrossRef]
128. D. M. Dacey, H.-W. Liao, B. B. Peterson, F. R. Robinson, V. C. Smith, J. Pokorny, K.-W. Yau, and P. D. Gamlin, “Melanopsin-expressing ganglion cells in primate retina signal colour and irradiance and project to the LGN,” Nature 433, 749–754 (2005). [CrossRef]
129. P. A. Barrionuevo and D. C. Cao, “Contributions of rhodopsin, cone opsins, and melanopsin to postreceptoral pathways inferred from natural image statistics,” J. Opt. Soc. Am. A 31, A131–A139(2014). [CrossRef]
130. B. R. Conway, R. T. Eskew, P. R. Martin, and A. Stockman, “A tour of contemporary color vision research,” Vis. Res. 151, 2–6 (2018). [CrossRef]
131. D. L. Ruderman, T. W. Cronin, and C.-C. Chiao, “Statistics of cone responses to natural images: implications for visual coding,” J. Opt. Soc. Am. A 15, 2036–2045 (1998). [CrossRef]
132. G. Buchsbaum and A. Gottschalk, “Trichromacy, opponent colours coding and optimum colour information transmission in the retina,” Proc. R. Soc. London B 220, 89–113 (1983). [CrossRef]
133. T.-W. Lee, T. Wachtler, and T. J. Sejnowski, “Color opponency is an efficient representation of spectral properties in natural scenes,” Vis. Res. 42, 2095–2103 (2002). [CrossRef]
134. J. J. Vos and P. L. Walraven, “On the derivation of the foveal receptor primaries,” Vis. Res. 11, 799–818 (1970). [CrossRef]
135. D. M. Mount and S. Arya, “ANN: a library for approximate nearest neighbor searching, version 1.1.2,” (University of Maryland, 2010).
136. S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu, “An optimal algorithm for approximate nearest neighbor searching in fixed dimensions,” J. ACM 45, 891–923 (1998). [CrossRef]
137. D. H. Foster, I. Marín-Franch, K. Amano, and S. M. C. Nascimento, “Approaching ideal observer efficiency in using color to retrieve information from natural scenes,” J. Opt. Soc. Am. A 26, B14–B24 (2009). [CrossRef]
138. S. K. Shevell and P. R. Martin, “Color opponency: tutorial,” J. Opt. Soc. Am. A 34, 1099–1108 (2017). [CrossRef]
139. R. Shapley and M. J. Hawken, “Color in the cortex: single- and double-opponent cells,” Vision Res. 51, 701–717 (2011). [CrossRef]
140. D. I. A. MacLeod and R. M. Boynton, “Chromaticity diagram showing cone excitation by stimuli of equal luminance,” J. Opt. Soc. Am. 69, 1183–1186 (1979). [CrossRef]
141. A. M. Derrington, J. Krauskopf, and P. Lennie, “Chromatic mechanisms in lateral geniculate nucleus of macaque,” J. Physiol. 357, 241–265 (1984). [CrossRef]
142. X. Zhang, D. A. Silverstein, J. E. Farrell, and B. A. Wandell, “Color image quality metric S-CIELAB and its application on halftone texture visibility,” in IEEE COMPCON 97 (IEEE, 1997), pp. 44–48.
143. G. M. Johnson and M. D. Fairchild, “A top down description of S-CIELAB and CIEDE2000,” Color Res. Appl. 28, 425–435 (2003). [CrossRef]
144. C. A. Párraga, G. Brelstaff, T. Troscianko, and I. R. Moorehead, “Color and luminance information in natural scenes,” J. Opt. Soc. Am. A 15, 563–569 (1998). [CrossRef]
145. I. Fine, D. I. A. MacLeod, and G. M. Boynton, “Surface segmentation based on the luminance and color statistics of natural scenes,” J. Opt. Soc. Am. A 20, 1283–1291 (2003). [CrossRef]
146. E. Doi, T. Inue, T.-W. Lee, T. Wachtler, and T. J. Sejnowski, “Spatiochromatic receptive field properties derived from information-theoretic analyses of cone mosaic responses to natural scenes,” Neural Comput. 15, 397–417 (2003). [CrossRef]
147. E. Provenzi, J. Delon, Y. Gousseau, and B. Mazin, “On the second order spatiochromatic structure of natural images,” Vis. Res. 120, 22–38 (2016). [CrossRef]
148. D. H. Foster, “The Verriest lecture: color vision in an uncertain world,” J. Opt. Soc. Am. A 35, B192–B201 (2018). [CrossRef]
149. M. R. Pointer and G. G. Attridge, “The number of discernible colours,” Color Res. Appl. 23, 52–54 (1998). [CrossRef]
150. F. Martínez-Verdú, E. Perales, E. Chorro, D. de Fez, V. Viqueira, and E. Gilabert, “Computation and visualization of the MacAdam limits for any lightness, hue angle, and light source,” J. Opt. Soc. Am. A 24, 1501–1515 (2007). [CrossRef]
151. M. Flinkman, H. Laamanen, P. Vahimaa, and M. Hauta-Kasari, “Number of colors generated by smooth nonfluorescent reflectance spectra,” J. Opt. Soc. Am. A 29, 2566–2575 (2012). [CrossRef]
152. J. Morovic, V. Cheung, and P. Morovic, “Why we don’t know how many colors there are,” in CGIV 2012, 6th European Conference on Colour in Graphics, Imaging, and Vision (Society for Imaging Science and Technology, 2012), pp. 49–53.
153. I. Marín-Franch and D. H. Foster, “Number of perceptually distinct surface colors in natural scenes,” J. Vis. 10(9), 9 (2010). [CrossRef]
154. B. Hill, T. Roger, and F. W. Vorhagen, “Comparative analysis of the quantization of color spaces on the basis of the CIELAB color-difference formula,” ACM Trans. Graph. 16, 109–154 (1997). [CrossRef]
155. W. S. Mokrzycki and M. Tatol, “Colour difference ΔE–a survey,” Mach. Graph. Vis. 20, 383–411 (2011).
156. P.-L. Sun and J. Morovic, “Inter-relating colour difference metrics,” in Tenth Color Imaging Conference: Color Science and Engineering Systems, Technologies, Applications (Society for Imaging Science and Technology, 2002), pp. 55–60.
157. S. Westland, University of Leeds (personal communication, 2018).
158. Y. Li, S. Westland, Q. Pan, and V. Cheung, “Methods to assess the relative number of discernible colors for displays,” in 22nd Color and Imaging Conference (Society for Imaging Science and Technology, 2014), pp. 151–154.
159. J. M. M. Linhares, P. D. Pinto, and S. M. C. Nascimento, “The number of discernible colors in natural scenes,” J. Opt. Soc. Am. A 25, 2918–2924 (2008). [CrossRef]
160. D. L. MacAdam, “Note on the number of distinct chromaticities,” J. Opt. Soc. Am. 37, 308–309 (1947). [CrossRef]
161. A. Akbarinia and K. R. Gegenfurtner, “Color metamerism and the structure of illuminant space,” J. Opt. Soc. Am. A 35, B231–B238 (2018). [CrossRef]
162. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. (Wiley, 2006).
163. J. M. M. Linhares, P. E. R. Felgueiras, P. D. Pinto, and S. M. C. Nascimento, “Colour rendering of indoor lighting with CIE illuminants and white LEDs for normal and colour deficient observers,” Ophthalmic Physiolog. Opt. 30, 618–625 (2010). [CrossRef]
164. A. M. Bakke, I. Farup, and J. Y. Hardeberg, “Evaluation of algorithms for the determination of color gamut boundaries,” J. Imaging Sci. Technol. 54, 050502 (2010). [CrossRef]
165. P. Urban, “Gamut volume,” in Encyclopedia of Color Science and Technology, M. R. Luo, ed. (Springer, 2016), pp. 676–678.
166. H. E. Smithson, “Sensory, computational and cognitive components of human colour constancy,” Philos. Trans. R. Soc. B. 360, 1329–1346 (2005). [CrossRef]
167. C. Witzel and K. R. Gegenfurtner, “Color perception: objects, constancy, and categories,” Annu. Rev. Vis. Sci. 4, 475–499 (2018). [CrossRef]
168. A. Gijsenij, T. Gevers, and J. van de Weijer, “Computational color constancy: survey and experiments,” IEEE Trans. Image Process. 20, 2475–2489 (2011). [CrossRef]
169. M. R. Luo and R. W. G. Hunt, “A chromatic adaptation transform and a colour inconstancy index,” Color Res. Appl. 23, 154–158 (1998). [CrossRef]
170. L. E. Arend Jr., A. Reeves, J. Schirillo, and R. Goldstein, “Simultaneous color constancy: papers with diverse Munsell values,” J. Opt. Soc. Am. A 8, 661–672 (1991). [CrossRef]
171. D. H. Brainard, “Color constancy in the nearly natural image. 2. Achromatic loci,” J. Opt. Soc. Am. A 15, 307–325 (1998). [CrossRef]
172. Y. Ling and A. Hurlbert, “Role of color memory in successive color constancy,” J. Opt. Soc. Am. A 25, 1215–1226 (2008). [CrossRef]
173. J. Roca-Vila, C. A. Parraga, and M. Vanrell, “Chromatic settings and the structural color constancy index,” J. Vis. 13 (4), 3 (2013). [CrossRef]
174. J. L. Nieves and J. Romero, “Heuristic analysis influence of saliency in the color diversity of natural images,” Color Res. Appl. 43, 713–725 (2018). [CrossRef]
175. I. Marín-Franch and D. H. Foster, “Estimating Information from image colors: an application to digital cameras and natural scenes,” IEEE Trans. Pattern Anal. Mach. Intell. 35, 78–91 (2013). [CrossRef]
176. G. J. Klir, Uncertainty and Information: Foundations of Generalized Information Theory (Wiley, 2006).
177. C. Arndt, Information Measures: Information and its Description in Science and Engineering (Springer, 2001).
178. D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge University, 2003).
179. M. Grendar, “Entropy and effective support size,” Entropy 8, 169–174 (2006). [CrossRef]
180. G. Feng and D. H. Foster, “Predicting frequency of metamerism in natural scenes by entropy of colors,” J. Opt. Soc. Am. A 29, A200–A208 (2012). [CrossRef]
181. N. Fider and N. L. Komarova, “Quantitative study of color category boundaries,” J. Opt. Soc. Am. A 35, B165–B183 (2018). [CrossRef]
182. A. Lewis and L. Zhaoping, “Are cone sensitivities determined by natural color statistics?” J. Vis. 6 (3), 285–302 (2006). [CrossRef]
183. A. Lapidoth, “Nearest neighbor decoding for additive non-Gaussian noise channels,” IEEE Trans. Inf. Theory 42, 1520–1529 (1996). [CrossRef]
184. T. Morimoto and H. E. Smithson, “Discrimination of spectral reflectance under environmental illumination,” J. Opt. Soc. Am. A 35, B244–B255 (2018). [CrossRef]
185. D. H. Foster and K. Żychaluk, “Is there a better non-parametric alternative to von Kries scaling?” in CGIV 2008, 4th European Conference on Colour in Graphics, Imaging, and Vision (Society for Imaging Science and Technology, 2008), pp. 41–44.
186. R. Steuer, J. Kurths, C. O. Daub, J. Weise, and J. Selbig, “The mutual information: detecting and evaluating dependencies between variables,” Bioinformatics 18, S231–S240 (2002). [CrossRef]
187. L. F. Kozachenko and N. N. Leonenko, “Sample estimate of the entropy of a random vector,” Probl. Inf. Transm. 23, 95–101(1987).
188. M. N. Goria, N. N. Leonenko, V. V. Mergel, and P. L. Novi Inverardi, “A new class of random vector entropy estimators and its applications in testing statistical hypotheses,” J. Nonparametr. Stat. 17, 277–297 (2005). [CrossRef]