This paper discusses a number of open problems in color constancy theory whose correct solution is a prerequisite for the theory of the phenomenon. Solutions employing suitable visually meaningful versus physically meaningful basis functions (principal components) are examined. In the former case the starting point is an estimate of the first derivative of the reflectance (illuminant), essential for defining color, instead of an estimate of the reflectance (illuminant), as in the latter. Conceptual consequences are discussed. Mathematical and physical constraints are identified. We compare the results of theories that do or do not ignore them. The following questions are considered. (1) Do unique solutions of the estimation problem exist everywhere in the object-color solid belonging to the illuminant? (2) Are they physically meaningful, i.e., at least nonnegative? (3) Are they representative for reflectance and spectral distribution functions? (4) What role plays metamerism?
© 2014 Optical Society of America
The visual system is a signal detection, discrimination, and processing system. Its perceptual variables exhibit color (i.e., appearance to the generic human observer) constancy, the phenomenon that the appearance of a scene is qualitatively preserved across illuminants. Somehow, the visual system manages to construct almost illuminant-independent variables from illuminant-dependent input. A simple consideration sheds some light on the phenomenon.
Let be the reflectance of some patch, the illuminant, the equi-energy spectrum equal to unity for all , chosen as the reference illuminant, and a linear superposition of the CIE color-matching functions, normalized such that the integral of over the visual range equals unity, for example (bar tacitly understood), , or , with , , and the tristimulus values of . The integral over the visual range of over the same integral of , called a von Kries type quotient, is obviously strictly independent of the amplitude of . Numerical analysis  is aware that such quotients, almost illuminant-independent if is almost constant, are often weakly dependent on other illuminant parameters, for example, the temperature when is a blackbody radiator. Numerical simulation confirms this. Von Kries type quotients are employed in the definition of CIE space , apparently rather successfully. If is a cone sensitivity we have a classic von Kries quotient arising in the von Kries hypothesis and the associated coefficient rule. Attempts to experimentally verify the hypothesis have failed ; see Section 3.
Despite this fact, Forsyth  concluded from numerical work in machine vision that the coefficient rule has much to recommend it if sensitivities are narrowband. “Narrowband” is a relative concept. A theorem [4,5] states that a von Kries type quotient equals the integral of apart from an illuminant-dependent error, which is small if, in a precisely defined sense, is narrow compared to and , both dimensionless, a fact trivially true if one of the latter is constant as a function of (width infinity) or if is a -function (width zero). Despite its counterintuitive nature, the -approximation is of conceptual importance: if (and only if ) it applies is perfect color constancy possible. Cone sensitivities are narrow functions but narrower, nonnegative functions exist . The analysis reveals not only the physical basis of color constancy but also when it can fail. Reflectance and spectral distribution functions associated with saturated colors are not broad in the definition of the theorem explaining color constancy failures as observed . Empiricism and theory show that von Kries type quotients constitute a reasonable starting point for color constancy theory, attempting to construct visual variables with smaller illuminant- dependent error. In this paper, and are desaturated. Occasionally, results for saturated are stated.
It is conceivable that expressions other than the previous von Kries type quotient can be found with improved illuminant-independent behavior. The expressions must be independent of the amplitude of the illuminant, restricting them to appropriate rational functions of tristimulus values. Other approaches have observed that color constant variables can be constructed from estimates of [and , needed for that purpose], determined by the tristimulus values , , of [and , , of ], if equal or close enough to the actual . Equality occurs with probability of almost zero. “Close enough” has no chance of applying unless the estimate appears “natural.” The definition of “close enough” then must ensure that the propagation of the estimation error remains small in a later definition of lightness and color, zero for the constant reflectance, and hence dependent on an estimate of . It is to be distinguished from the first derivative of the estimate of due to the amplification of the estimation error by differentiation. Unfortunately, the definition of a representative, physically meaningful estimate and the notion “close enough,” although intuitively clear, are hard problems. Metamerism, always indicative of a flaw if ignored, causes an estimate uniquely constructed from tristimulus values, even if “close enough” to , to differ, sometimes considerably, from its metamers, in agreement with the qualitative nature of color constancy. The definition of an estimate needs a decision about the appropriate basis functions. In view of the different nature of reflectances belonging to desaturated and saturated colors, no single set of basis functions exists that works for all tristimulus values in the object-color belonging to an estimate of . Reflectances of saturated colors, essentially smoothed versions of Schrödinger’s optimal colors, vanish on certain intervals, all within the visual range so that, at least, similar basis functions are needed. In the linear parts of the theory the expansion coefficients satisfy a set of linear equations. We need proof that the determinant of the matrix never vanishes: a model of the visual system must always work, not often or mostly, a demand that, although severe, is generally accepted in the development of signal processing systems. Symmetric, positive definite matrices possess the required property . A restriction to such matrices has consequences for the choice of the basis functions; see Section 2.B. Finally, it must be shown that variables constructed from the estimate exhibit better color constant behavior than von Kries type quotients. We repeat the proof [5,9], temporarily (see Section 2.D) accepting that the chromaticity coordinates of the actual illuminant are a priori provided as in CTV, or by specular reflection, or otherwise, e.g., by the background in the experiment; these are, in fact, assumptions that underlie the von Kries theory as well. The construction replaces von Helmholtz’s unconscious judgment  as to how an object would look in white light, i.e., the canonical illuminant , the reference in this paper, by well-defined signal processing. Other approaches to the color constancy problem then must show that their solution always exists, is physically meaningful (see Section 2.A), and is still better. The human visual system performs similarly, better or worse, in each case, raising interesting questions; see Section 3.
Wyszecki and Stiles , p. 173, remark that “the science of color has not advanced far enough to deal with this problem (various visual phenomena that contribute significantly to color perception) quantitatively.” Thus, it is a good question  whether the situation today is better and, if not, what is the prevalent challenge. The appearance of a patch with reflectance under is a function of the tristimulus values of and , and, possibly, other variables (surround). The von Kries approach employs three von Kries type quotients. Variables in color constancy theory need the three von Kries type quotients and the chromaticity coordinates of . Thus, color science must find three nonlinear functions of at least five variables with optimal discrimination properties that are, hopefully, able to describe lightness and color. This urges modesty. As a first step, we could replace in empirical formulas  von Kries type quotients by theoretical color constant variables and see what happens—provided that color constancy theory is sound. However, some approaches do not differentiate between meaningless and meaningful quantities; see Section 2.A. Matrices are assumed not proved to be nonsingular; see Section 2.B. Assumptions seem untenable from the physical and mathematical point of view; see Section 2.C. Information lacking in some models is gathered by means of methods that are apparently not applied by the visual system; see Section 2.D. With the benefit of hindsight, these problems can possibly be solved within the adopted framework. They constitute as many open problems to adherents.
However, of predominant importance, since then color constancy theory takes a different course, are two conceptual problems. First, it must be decided whether basis functions in estimates of and should be visually or physically meaningful, e.g., principal components and phases of daylight. Visually meaningful basis functions are appropriately defined by means of the color-matching functions. The data processing uses symmetric, positive definite matrices whose inversion is trivial and, hence, not beyond the power of a biological system. They ensure that the sets of linear equations defining estimates always have a unique solution; see Section 2.B. In this framework we discern desaturated  and different types of saturated colors . Nonnegative estimates, if is concerned, are obtained by a linear analysis in the former case and by nonlinear methods in the latter, in agreement with mathematical predictions; see Section 2.A. The ensuing visual representation of the “world” differs from the physical one, which is an old philosophical issue; see Section 2.D. Actually, is estimated. Next, its integration and a suitable integration constant yields the estimate of with basis functions as the integrals of the functions in the representation of and the function equal to unity for all . This order affords a decisive advantage. Since color conveys information about , constancy of color benefits from its direct estimation, preferably by a least-squares fit; see Section 2.B. Integration suppresses the illuminant-dependent estimation error. The estimate and the function to be estimated, if smooth, are close. On the other hand, if first is estimated, the required estimate of must be obtained by differentiation, amplifying estimation error; see Section 2.C for examples.
Sections 2.A–2.D can be read independently. In Section 2.B Eqs. (3a) and (3b), the interim summary and discussion of prior results are necessary and, hopefully, sufficient for understanding the role played by the two approaches to the color constancy problem. The passage proves von Helmholtz’s conjecture  and can be read independently. See Section 3 for experimentally verifiable theoretical predictions. In view of the extensive bibliography in Foster , only supplementary references are provided.
2. OPEN PROBLEMS
Reflectance and spectral distribution functions are nonnegative functions of . They satisfy quadratic models, extensively discussed in the literature . The result does not rule out every linear model, but then adds nonnegativity as a constraint. Failure to comply invalidates a lot of even elementary results. Furthermore, it states that a theory based on linear models alone is incomplete. Actually, reflectance functions of saturated colors are often cut-off filters, zero on a certain interval . For example , CIE test color 12 (strong red) is approximately zero on the interval [490 and 545 nm]. In the estimation of such functions from their tristimulus values, not only the coefficients of the basis functions but also the wavelengths of the interval must be determined, obviously a nonlinear problem, and solved by iteration . Linear theory benefits from the powerful methods of linear algebra at the expense of a later proof that the mathematical result is physically meaningful. Readers are recommended to beforehand make sure that authors paid the price before completing the paper. If not, the omission can perhaps be remedied in hindsight, an always welcome revelation when solutions are physically meaningful. Nonlinearity causes a jump in the problem’s complexity, growing steeply with the number of parameters. A proof of non-negativity seems hopeless for any model with four or more parameters. Using three principal components, Troost and de Weert  constructed 2734 “reflectances” from Munsell , , values under CIE standard illuminant . More than 50% of patches had to be rejected because of failure to be between 0 and 1 for all . The thesis has been proposed that any reflectance can be expressed as a weighted sum of principal components, dependent on the author, i.e., possesses a finite model, see Section 2.C. Perfect recovery of is deemed possible [11,18]; see Section 2.D. If , this applies at most for that satisfy conditions a priori excluding the physically meaningless “reflectances” rejected by Troost and de Weert by hindsight. Conditions could exclude all in certain domains of the object-solid belonging to , e.g., saturated colors, or certain types of reflectance, e.g., nonsmooth . This is the dominant difficulty. An estimate of chooses and suitable basis functions. Nonnegative estimates, for all , are constructed from tristimulus values in domains in the object color solid, corresponding to desaturated  and saturated  colors; see Section 2.B, at the expense of a possibly large illuminant-dependent estimation error. Tristimulus values of desaturated colors are located in the so-called principal domain , where a linear analysis suffices. Methods are found to suppress the error; see Eqs. (3); not fully, however, but better than for von Kries type quotients. An estimate of is similarly constructed.
B. Physically versus Visually Meaningful Basis Functions
Like orthogonal polynomials, principal components form a complete set of physically meaningful functions. Any function that is like can be written as a generally infinite series with appropriate coefficients. A finite expansion with tristimulus values under an estimate of equal to those of , is called an estimate of , if its values are between 0 and 1. The definition of a similar nonnegative estimate of , determined by the tristimulus values of , needs, first, a decision about the appropriate basis functions. The three cone sensitivities are linearly independent on (, ), with and the begin and end points of the visual range. If desired, we can construct mutually orthogonal functions (apply the Gram–Schmidt method). Functions orthogonal to the color-matching functions multiplied by are called metameric blacks  under . Together, the functions constitute a complete set of visually meaningful basis functions. Any estimate of [estimate of ] can be written as a linear superposition of the cone sensitivities multiplied by [estimate of ] plus some metameric black under [idem]. An unjustifiable restriction of to phases of daylight has been removed. For example, blackbody radiators also possess excellent color rendering properties , which a valid theory should be able to explain. It can be proved that representations are invariant for a nonsingular transformation of the color-matching functions. For practical purposes, we may employ the CIE functions and tristimulus values.
Consider estimates of and of to be calculated, respectively, from its cone signals or tristimulus values , , , and from those of , corresponding to , , of . The three physically meaningful basis functions are daylight phases [if ] or principal components [if is concerned]. On the other hand, the visually meaningful ones are the cone sensitivities, multiplied by either or in case of or , both dimensionless. In all cases, the coefficients , , 2, 3 of the basis functions satisfy an inhomogeneous set of linear equations with, on the left-hand side, the given th cone signal of either or . On the right-hand side, the matrix elements equal the integral over the visual range of the th visually meaningful basis function and the th basis function, chosen either visually or physically meaningful. Recall from linear algebra that the solution of the set exists and is unique if the determinant of the matrix is nonzero. If it vanishes, either no solution exists or, if it exists, it is not unique. If this situation is to be avoided, the determinant must not vanish. This is trivial for the visually meaningful representation of and . The matrix involved is symmetric and positive definite . Its determinant, equal to the product of its positive eigenvalues, is nonzero. If this holds true when the phases of daylight or principal components are concerned, its proof seems nontrivial. In more sophisticated approaches  the same problem occurs. For the sake of the argument, we grant that the determinant is nonzero. The homogeneous system of linear equations has only the solution zero so that a metameric black involving three principal components is zero for all . For the evaluation of the matrix elements involving physically meaningful basis functions, the human visual system must dispose of “knowledge of the physical world” that then must reside in memory. This leads to problems connected with the development of color constant vision in infants, who start with an empty memory . The same argument motivates the choice of as the reference illuminant instead of , for example.
These visually meaningful estimates  break down when the cone sensitivities tend to -functions because the mentioned matrix elements for , essentially the integral of the product of two -functions with the same peak, diverge. Nonnegativity is still to be proved. The estimates, consisting of the three peaked cone sensitivities, are not representative of actual functions, e.g., the constant reflectance. For smoothest estimates of , its second derivative is a weighted sum of the visual basis functions, replacing by the smoothest estimate of , with three coefficients to be determined, see Eq. (4) of Ref.  or Eq. (A4) in Appendix A with , replacing by and, for practical purposes, the cone sensitivities by the CIE color-matching functions. The solution of the inhomogeneous differential equation involved whose first derivative vanishes at both ends of the visual range (two boundary conditions), defines new visually meaningful basis functions and, next, leads to a inhomogeneous set of linear equations for its coefficients with symmetric, positive definite matrix in Eq. (9) of Ref. , hence nonsingular. Its solution , , if is constant, always exists and is independent of the amplitude of . It is between 0 and 1 for tristimulus values , , in the principal domain of the object color solid belonging to for all , similarly constructed, see Eq. (16) of Ref. . See Fig. 2 of Ref.  for examples of .
Actually, and are estimated and, next, and by integration and a suitable integration constant. We prove that is a least-squares fit to . In Eqs. (A2)–(A9) of Appendix A with replace , , , , , , , , and by, respectively, , , , , , , , , and . From Eq. (A9) we haveA6). Since at and at there. Next, minimize, as a function of and , 2) coincides with Eq. (A11), mutatis mutandis. is obtained by integrating from to with appropriate integration constant [see Eq. (16) of Ref. ], and is linear in its tristimulus values , , . The three basis functions are these integrals of the two functions on the right-hand side of Eq. (1) and the function equal to unity for all . The least-squares fit guarantees that is representative of , in general. Since they are metameric under , they intersect at the, at least three, zeroes of blacks [2,21] , 539, and 608 nm. On (449, 539) and (539, 608) the mean square difference of and cannot be large if the mean square difference of and is sufficiently small by Wirtinger’s inequality . On (449, 608) and are then close. Near the ends of visual range and in general differ as the latter’s boundary values are unknown. In the definition of [see Eqs. (A7a) and (A6)], when , this error propagates favorably since the color-matching functions are zero at these ends. Thus, the error introduced by replacing by is small, if is smooth. If , is a least-squares fit to , by the prior proof. Since like disposes of two parameters only, a least-squares fit is indispensable for constancy of color . In general, , especially at the ends of the visual range. Since the error propagates favorably, the mean square difference of and is almost minimal. On (449, 608) and cannot much differ if that difference is sufficiently small, as discussed above. Examples in Fig. 2 of Ref.  show that the terms and are, in general, not small since and , unknown to the visual system, assume arbitrary values between 0 and 1. We suppress the resulting error.
We have the basic equation on (, ):A7), simplified in Eq. (A9), w(λ) = 1 and Eq. (16) of Ref. . The other basis function equals unity for all λ. The illuminant-dependent coefficients are uniquely determined by the tristimulus values , , of equal to those of ; is satisfied if , , is in the principal domain  of the object-color solid belonging to , a condition satisfied for desaturated . The residual estimation error , zero for constant , is small on (449, 608) if is smooth, but, in general, large at the ends of the visual range. Perceptual variables are functions of color constant visual variables. The latter are defined by the left-hand side of the following equation with some narrow . We have 5]. Consider over the integral of , equal to the integral of . Since the von Kries type quotients belonging to , and , are equal, the error in Eq. (3b) equals, apart from a minus sign, the first term. See Eq. (21) of Ref.  with and replaced by and . A theorem (see Eq. (12) of Ref. ), shows that the error depends on, among other things, and , small for smooth and . Hence, the visual variable defined by the integral on the left of Eq. (3b), for constant , is almost illuminant-independent, with error smaller than in the case of the von Kries type quotient. If it is virtually zero, the left-hand side of Eq. (3b) recovers from tristimulus values , , of the image as it would have appeared if it had been taken under , the first integral on the right. If so, this proves von Helmholtz’s conjecture [3,9,10], defining “white light.” Equations (3) also apply to saturated . Substitute , derived in that case .
Strongly different metameric possess the same estimate . Their well-known mismatch  under causes the residual error in Eq. (3b) to vary among them. Although it is appreciably smaller than in the von Kries case, in general, the experiment [see Eqs. (7)–(9)], could reveal that it is not small enough. However, corresponds to the simplest choice from a class of smoothest reflectances involving a weight function , yielding similar but not the same results . Another choice [see Eq. (A10)] could lead to a smaller error in practice or better mimic the human visual system.
Alternative approaches [18,11] claim that in Eq. (3a) can be modeled by a finite weighted sum of physical functions, replacing , with, essentially, everywhere. If the parameters can be uniquely determined (see Section 2.D) the claim ensures that the representation is between 0 and 1 for all . It inherits the property from the actual , making the work of Section 2.A superfluous. This obviously crucial claim is discussed next.
C. Finite Models versus Estimates
Physically meaningful basis functions, e.g., orthogonal principal components or polynomials are illuminant-independent. A given sum with three expansion coefficients such that its values are between 0 and 1 can be interpreted as a reflectance . The expansion coefficients are uniquely determined by its cone signals. We have constructed an estimate of that coincides with itself. We then say that possesses a finite model. This example is of interest for showing the correctness of an algorithm  that exactly recovers its expansion coefficients and those of by multiple views (see Section 2.D), not necessarily the correctness of the underlying mathematical idea. Actually, the issue is whether all or ‘“most” a priori given reflectances in the field of view possess a finite model. Any such trivially possesses a finite model, consisting of , supplemented by orthogonal basis functions from the chosen set. If is not orthogonal to them, applying the Gram–Schmidt method achieves this. However, we are interested in the validity of the thesis: there exists a common basis such that any possesses an model, with , dependent on the author, apart from a small amount of residual error, at most, i.e., Eq. (3a) applies with for all . Note the interchange of logical quantifiers, always urging prudence. From “all events  possess an (event-dependent) cause [finite model],” it does not follow “there exists a common cause for all events.” This is an important physical and astronomical discovery of the past century, the “Big Bang.”
Recall from mathematics : for any an integer , dependent on , exists such that the sum of the first or less terms from its series expansion, convergent on some interval (, ), is equal or close to it. In the latter case, this depends on its rate of uniform convergence on (, ), permitting term-by-term integration. Thus the thesis assumes that reflectances constitute a special kind of function such that is independent of . It is also tacitly assumed that the visual interval (, ) is part of all intervals (, ), defined by the physical properties of the different materials corresponding to . This is temporarily accepted. The following two examples show that if a reflectance satisfies a finite, for simplicity, model in some set of basis functions on (, ) it need not possess a similar finite model in another set of basis functions or the other way around. Apparently, the claimed common set of basis functions, if it exists, is hard to find.
Define , mapping (, ) on (, 1), and consider the two following sets of complete, orthogonal basis functions: (1) Legendre polynomials  , e.g. , and (2) Fourier series; basis functions 1, and , . In both systems the constant function equals the first basis function and therefore trivially satisfies the finite , 1, model, as does the function in system (1). On (, 1) we have the Fourier series , also easily verified directly, for :
The latter representation is not finite in system (2) and even converges slowly. Similarly, the function satisfies the finite model in system (2). We have the, also not finite, representation in system (1) for :24], It can be proved that this series converges fast, yet this convergence speed is of little help.
If is not equal to a finite weighted sum of chosen basis functions, the expansion coefficients could be optimally chosen, e.g., such that the estimate is a least-squares fit to , perhaps yielding a sufficiently small amount of residual error as the thesis allows. Since the basis functions of sets (1) and (2) are orthogonal, the truncated series of order affords this least-squares fit . The basis functions are, respectively , , 1, 2 and  1, , . Equation (4) is reduced to5) yields
Although the two simple examples render the prospects of the thesis bleak, they do not rule out that a “special” common finite basis exists. If reflectances , , very large compared to , , all possess a common finite model in some set of basis functions on a common interval (, ), defined by the physics of the different materials with reflectance , the correlation function, defined for , on that interval,28] show that the first four eigenfunctions are not linearly dependent; therefore, . If an analysis would show that linear dependence occurs for, e.g., , the original basis functions can be replaced by principal components, without loss of generality.
Unfortunately, the analysis is little rewarding unless the set is an unbiased sample from all possible reflectances. A rough calculation  suggests that metamerism is a quite common phenomenon for desaturated . Reflectances from color atlases are usually smooth. However, in practice, metameric, nonsmooth functions also occur, as lamp manufacturers are aware. We could modify by replacing any single by a suitable sum of its metamers, rendering the set unbiased, and start again.
As four principal components define a metameric black, an model of is reducible to a weighted sum of three principle components and () metameric blacks under . In principle, multiple views under and, at least, one additional illuminant , all represented by a common, finite , e.g., , model, can recover all coefficients; see Section 2.D. Unless and strongly differ, a black under is almost black under . It is then doubtful whether its coefficient can reliably be recovered. If the sum of blacks is considered as noise, an acceptable thesis, in this framework, could be: on (, ) any is represented by an estimate involving three principal components apart from a residual error , i.e., the sum of blacks under , not small compared to the estimate in general. This thesis leads to an equation similar to Eq. (3a) with replaced by the present estimate with one crucial exception: we cannot evade finding conditions on which the estimate of is between 0 and 1 for all (see Section 2.A) as is done by claiming that possesses a finite () model, that inherits this property from the actual . Prospects seem bleak; see Section 2.A.
D. Lack of Information
The tristimulus values of from patches, equal to the tristimulus values of the product of the estimates of and , do not enable us to separately determine the tristimulus values of the former estimates and the chromaticity coordinates of the latter. The chromaticity coordinates of are provided by the camera responses on perfect white in CTV or by specular reflection (hampering discrimination) in color constancy or, in the experiment, by the background. They and the coefficients in a finite model of (see Section 2.C) can be perfectly recovered from multiple views of the same scene under different illuminants , granting assumptions. Consider two views. The information from the first view is in memory. The information from the second view is provided by the observation. The algorithm  yields the desired data. Preparing for a third view, we must refresh the information in memory. Two possibilities arise: either (1) the first view is preserved in memory and the second view is lost, or (2) the second view overwrites the first view in memory. In case (1), a return to the first view (= third view) violates the algorithm requiring two different views. Thus we have case (2): the return interchanges the two views and, since the algorithm is symmetric for that interchange, the corresponding information (equal in the second and third view) processing yields the same output, the identical appearance of the scene in the second and third views. Consider the appearance of an article of dress, first viewed in a shop with poor lighting and next viewed outdoors in daylight. The proposal predicts that, upon return to the shop (third view), the latter appearance is preserved, at variance with this author’s observation. If this is generally agreed, the proposal is, in any case, not applied by the human visual system.
Economical data processing can free information for the determination of the chromaticity coordinates of . Consider the light from a scene in three-dimensional space and its image on the retina. Regions on which continuously varies within a few just-noticeable differences as a function of space variables are called patches with tristimulus values equal to their mean value. Distinct borders belong to different patches. The traveling salesman algorithm finds the shortest circuit such that the salesman visits cities (patches) once and returns to the starting city (patch). We use a perceptually meaningful metric, expressing distances in terms of just-noticeable differences. By deleting the largest edge in the circuit we obtain a chain of minimally distinct patches. A patch and its successor define, in total, () differences of triplets of tristimulus values associated with reflectance differences or different illumination conditions (due to surface curvature, stray light, etc.). One triplet of tristimulus values is freed for the determination of the chromaticity coordinates of , i.e., the white point. Unfortunately, the complexity of the traveling salesman problem increases fast with . In any event, differences are essential for the discrimination problem so that a decision on the proper definition must be deferred until then. The estimate of is obtained by replacing in the theory by , achieving that the estimate of . Color constant visual signals follow from Eq. (3b). We start with the hypothesis that the illumination conditions do not vary across the scene, to be revoked if results are inconsistent or “odd.” “Inconsistent” means that not all free choices of the triplet (patch) yield the same chromaticity coordinates of .
Unfortunately, if and are the actual reflectance of some patch and the actual illuminant, respectively, the light incident on the eye can be written for any fixed for all as the product of a phenomenological reflectance and phenomenological illuminant , corresponding to physically different scenes, visually indiscernible from the actual one. In particular, any patch with reflectance can serve as a white point corresponding to the phenomenological for all , and similar illuminant . “Reality,” i.e., the appearance of the world, is then a construct of the observer’s mind (which designates in some way the white patch; hence. the illuminant), a philosophical thesis (Bishop Berkeley’s famous dictum “esse est percipi”) also defended by Kant and influential among German physiologists at the end of the 19th century .
There exist many visually indiscernible physical worlds but only one observer able to define a common framework. In the basic Eqs. (1)–(3) of Ref. , replace by and by . Their solution is the smoothest estimate of , a weighted sum of the same visually meaningful basis functions as in the case for all (see Eq. (4) of Ref. ), provided that the estimate of is a weighted sum of the visually meaningful basis functions of Section 2.B . If the tristimulus values of were known, the present coefficients and in Eq. (2) of this paper and the integration constant could be calculated as before and, next, the present coefficients of by imposing, as in the case :2]. In practice, optimal colors are seldom present in the field of view. Theoretically, they are ubiquitous. For any a filter exists, up to a multiplicative factor, metameric under , i.e., we have 2]. is its luminance equal to the value of the integral on the left for . is the maximum attainable luminance, i.e., the value of the integral on the right-hand side. Equation (6b) is then an identity. The cases and provide two nonlinear equations for the coefficients , and , up to a multiplicative constant, to be solved on the condition for all . Although they are difficult, optimal filters , i.e., and , are easily constructed numerically. The entire analysis is no more than an application of the principle that  can be estimated from given tristimulus values if , e.g., an optimal color, only dependent on two parameters and excluding metamerism,  is known. If the choices of patches do not yield the same answer, we must drop the hypothesis that the illumination is the same across the image and partition it accordingly.
3. EXPERIMENT AND THEORETICAL PREDICTIONS
Color constancy theory in Section 2.B assigns predominant importance to the estimation of , determining color. Its estimate determines its constancy; see Section 2.B. Wyszecki and Stiles  state that opponent color matching, except unique red, is independent of luminance. Both theory and experiment suggest that disentanglement of color and luminance matching is possible. If so, we focus on constancy of color with the advantage that matches of and variables  can be determined with great precision. An experiment that falsifies this part of the theory falsifies it entirely. Theoretical considerations about the limitations of color constancy shed light on the problem.
The starting point is the discussion of asymmetric matching in Wyszecki and Stiles . In the interpretation of the experiment concerning human color constancy, we compare the following:
- (1) The appearance of some under , defined by the tristimulus values , , of . The tristimulus values , , of are provided by the background.
- (2) The corresponding experimental asymmetric appearance match under the reference illuminant , defined by its tristimulus values , , . The tristimulus values of are , , (Wyszecki and Stiles  mentions caveats).
- (3) The appearance of under , determined by the calculated tristimulus values of .
We have from Eq. 3 (5.12.1) of Ref. , stating that if an asymmetric match exists, a nonsingular matrix exists such that7) is in the object-color solid of , if , , are the tristimulus values of , for some reflectance . If so, the vector elements on the right-hand side of Eq. (7) are von Kries type quotients. In an experiment that uses paper matches, this condition is satisfied. If some patch with reflectance preserves its appearance under when viewed under , then up to a metameric black that vanishes under . If we may ignore the error over the integral of for desaturated, smooth and [see the discussion of Eq. (3b)] we have for narrow , , 2, 3, 2] for the hypothesis lacks, as the theoretical, relative error , although zero for the constant reflectance, can amount to some 10%. The color constancy result in Eq. (3b) yields 8); see the discussion of Eq. (3b). Recall from Eq. (3a) that is a weighted sum of the function equal to unity for all λ and two visually meaningful basis functions dependent on the estimate ; see Eq. (16) of Ref. . Numerical integration on the left of Eq. (9) can be carried out. Substitution of the coefficients in the representation of , i.e., an illuminant-dependent, weighted sum of von Kries type quotients , , and or similar cone signals, shows that the left-hand sides of Eqs. (7) and (9) are of the same form. The elements of , in general nondiagonal, depend on the chromaticity coordinates of , equal to those of the actual . They specify “adaptation conditions.”21] with (misprint) on the left-hand side of Eq. (A10). It can be proved that these values of and make a least-squares fit to if and almost so if not. Comparison of Figs. (1) and (7) of Ref.  shows a considerable improvement of the fit, understood from the discussion in Section 2.B. Unfortunately, is no estimate of since it seems hopeless to determine and such that is satisfied; see Section 2.A. For heuristic purposes, occasional failures of this condition can be ignored. For variable and , Eq. (10) defines a representative sample of reflectances with error in Eq. (3a). In Eq. (3b) this error is suppressed but, perhaps, not sufficiently in practice.
If so, Eq. (10) is able to correct a systematic error; vanishes at both and , while this seldom happens for in practice. Furthermore, is smoother than any actual it estimates. A slightly less smooth estimate is more representative for in practice; is orthogonal to and , nonzero at and , respectively. The integral of is the sum of two quadratic expressions. It suffices to construct a smooth estimate whose first derivatives are nonzero at both and , corresponding to nonzero and in Eq. (9). Recall from the discussion of Eqs. (1)–(3) of Ref.  that another class of smoothest estimates is defined by a weight function , multiplying there. The simplest choice for all yields in this paper. It is proved in Appendix A that a visually meaningful can be constructed that tends sufficiently fast to zero at and and thereby achieves that the first derivative of the associated estimate is nonzero there; see Eqs. (A5) and (A8). Since the improved estimate is metameric to , it is of the form of with one important exception: as before, we are able to find the slightly different conditions on which it is between 0 and 1 for all .
In its youth, any field of science faces the necessity of finding a consistent, mathematically and physically correct basis. This paper discusses the issue for color constancy. Since it is a qualitative phenomenon, theory can achieve, at most, that color constancy errors, e.g., due to metamerism, usually are small and random. Numerical simulations can establish this. Unfortunately, the issue complicates the experiment that, next, must decide whether the theory adequately mimics the behavior of the human visual system. Theoretically optimal, artificial systems may be of independent interest. Faites vos jeux (choose and play)!
In this appendix we construct for a given [estimate, e.g., of the] illuminant with tristimulus values , , , the smoothest reflectance associated with a weight function for all , except, possibly, at and . The special case yields in this paper. We are interested in a visually meaningful that tends sufficiently fast to zero at and , thereby achieving that is nonzero there; see Eq. (A5). We also show that all data processing can be carried out by a trivial inversion of symmetric, positive definite matrices instead of a such-like matrices, as previously. The needed generalization of Eqs. (1)–(3) of Ref.  readsA4), A5), at A7a), A7a) enforces at if tends to zero at in a similar way as the functions do. Suppose that, for , A6) and integration by parts yields for , and similarly, , 3:
The zero of at has multiplicity . Consider the determinant , first row , , ; second row (), (), (); third row (), (), (), and the determinant equal to , except that in the first row we have , , . Define (see Eq. (B1) of Ref. )
The integral of is unity. The right-hand side of Eq. (A9) equals multiplied by a function, only dependent on chromaticity coordinates, that possesses a simple zero at and, hence, tends to zero in a way similar to for . Hence, we achieve at and, similarly, at . Evaluation of the determinants shows that equals the narrowest, nonnegative function  constructed by Yule :A7) are determined by a set of linear equations. Multiply Eq. (A7b) by the functions and , whose integral over the visual range is zero. Integration on the left-hand side and Eq. (A2) yields and , respectively. Integration by parts in the second term yields
Next, we multiply Eq. (A7b) by and integrate over the visual range. By Eq. (A2), the left-hand side is . On the right-hand side, the first term is . The second term is calculated by using integration by parts, applying Eq. (A7b) and eliminating the coefficients by Eq. (A11). Define coefficients such that
Similarly, we obtain, with ,
The signal processing needs no more than (the trivial inversion of) matrices. The coefficients and define the principal domain, comprising , , in Eq. (A2) such that Eq. (A1) is satisfied for desaturated .
REFERENCES AND NOTES
1. C. E. Fröberg, Introduction to Numerical Analysis (Addison-Wesley, 1966), p. 297.
2. G. Wyszecki and W. S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae (Wiley, 1982).
3. D. A. Forsyth, “A novel algorithm for color constancy,” Int. J. Comput. Vis. 5, 5–35 (1990). [CrossRef]
4. C. van Trigt, “Illuminant-dependence of von Kries type quotients,” Int. J. Comput. Vis. 61, 5–30 (2005). [CrossRef]
5. C. van Trigt, “Von Kries versus color constancy,” Color Res. Appl.35, 164–183 (2010). A(λ)E(λ) and S(λ)/E(λ) in the present paper are denoted there by A(λ) and S(λ). [CrossRef]
6. J. A. C. Yule, Principles of Color Reproduction (Wiley, 1967).
7. H. E. Ives, “The relation between the color of the illuminant and the color of the illuminated object,” Trans. Illum. Eng. Soc. 7, 62–72 (1912).
8. G. Polya and G. Szegö, Aufgaben und Lehrsätze aus der Analysis, Band II (Springer-Verlag, 1964), aufgabe 63, p. 114.
9. C. van Trigt, “Linear models in color constancy theory,” J. Opt. Soc. Am. A 24, 2684–2691 (2007). [CrossRef]
10. D. Jameson and L. M. Hurvich, “Essay concerning colour constancy,” Ann. Rev. Psychol. 40, 1–22 (1989). [CrossRef]
11. D. A. Foster, “Color constancy,” Vis. Res. 51, 674–700 (2011). [CrossRef]
12. C. van Trigt, “Smoothest reflectance functions. I. Definition and main results,” J. Opt. Soc. Am. A 7, 1891–1904 (1990). [CrossRef]
13. C. van Trigt, “Smoothest reflectance functions. II. Complete results,” J. Opt. Soc. Am. A 7, 2208–2222 (1990). [CrossRef]
14. G. Szegö, Orthogonal Polynomials (American Mathematical Society, 1967).
15. J. N. Lythgoe, The Ecology of Vision (Clarendon, 1997).
16. CIE, Method of measuring and specifying colour rendering properties of light sources (Bureau Central de la CIE, 1974).
17. J. M. Troost and C. M. M. de Weert, “Techniques for simulating object color under changing illuminant conditions on electronic displays,” Color Res. Appl. 17, 316–327 (1992). [CrossRef]
18. G. Iverson and M. D’Zmura, “Criteria for color constancy in trichromatic bilinear models,” J. Opt. Soc. Am. A 11, 1970–1975 (1994). [CrossRef]
19. J. I. Dannemiller, “Computational approaches to color constancy, adaptive and ontogenetic considerations,” Psychol. Rev. 96, 255–266 (1989). [CrossRef]
20. J. B. Cohen and W. E. Kappauf, “Metameric color stimuli, fundamental metamers and Wyszecki’s metameric blacks,” Am. J. Psychol. 95, 537–564 (1982). [CrossRef]
21. C. van Trigt, “Metameric blacks and estimating reflectance,” J. Opt. Soc. Am. A 11, 1003–1024 (1994). [CrossRef]
22. D. S. Mitrinovic, J. E. Pecaric, and A. M. Fink, Inequalities Involving Functions and Their Integrals and Derivatives (Kluwer Academic, 1991). Wirtinger’s inequality occurs in two versions.
23. E. T. Whittaker and G. N. Watson, A Course of Modern Analysis (Cambridge University, 1962).
24. A. Erdelyi, W. Magnus, F. Oberhettinger, and F. G. Tricomi, Higher Transcendental Functions II (McGraw-Hill, 1953), p. 178.
25. Reference , p. 161, example 1 on interval (, ).
27. M. D’Zmura, “Color constancy: surface color from changing illumination,” J. Opt. Soc. Am. A 9, 490–493 (1992), Table 1. [CrossRef]
28. J. P. S. Parkkinen, J. Hallikainen, and T. Jaaskelainen, “Characteristic spectra of Munsell colors,” J. Opt. Soc. Am. A 6, 318–322 (1989), Fig. 1. [CrossRef]
29. G. Wyszecki and W. S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae (Wiley, 1982), p. 196.
30. H. Lang, “Color vision theories in the nineteenth century Germany between idealism and empiricism,” Color Res. Appl. 12, 270–281 (1987). [CrossRef]
31. L. M. Hurvich, Color Vision (Sinauer, 1981).
32. E. Brunswick, “Zur Entwicklung der Albedowahrneming,” Z. Psychol. 64, 216–227 (1928).