Abstract

The relative performance of color constancy algorithms is evaluated. We highlight some problems with previous algorithm evaluation and define more appropriate testing procedures. We discuss how best to measure algorithm accuracy on a single image as well as suitable methods for summarizing errors over a set of images. We also discuss how the relative performance of two or more algorithms should best be compared, and we define an experimental framework for testing algorithms. We reevaluate the performance of six color constancy algorithms using the procedures that we set out and show that this leads to a significant change in the conclusions that we draw about relative algorithm performance as compared with those from previous work.

© 2006 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. D. H. Brainard, W. A. Brunt, and J. M. Speigle, "Color constancy in the nearly natural image. 1. Asymmetric matches," J. Opt. Soc. Am. A 14, 2091-2110 (1997).
    [CrossRef]
  2. D. H. Brainard, "Color constancy in the nearly naturalimage. 2. Achromatic loci," J. Opt. Soc. Am. A 15, 307-325 (1998).
    [CrossRef]
  3. M. Lucassen, "Quantitative studies of color constancy," Ph.D. thesis (Utrecht University, 1993).
  4. L. Arend and A. Reeves, "Simultaneous color constancy," J. Opt. Soc. Am. A 3, 1743-1751 (1986).
    [CrossRef] [PubMed]
  5. E. H. Land, "The retinex theory of color vision," Sci. Am. 237, 108-129 (1977).
    [CrossRef] [PubMed]
  6. G. Buchsbaum, "A spatial processor model for object colour perception," J. Franklin Inst. 310, 1-26 (1980).
    [CrossRef]
  7. R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.
  8. L. T. Maloney and B. A. Wandell, "Color constancy: a method for recovering surface spectral reflectance," J. Opt. Soc. Am. A 3, 29-33 (1986).
    [CrossRef] [PubMed]
  9. S. A. Shafer, "Using color to separate reflection components," Color Res. Appl. 10, 210-218 (1985).
    [CrossRef]
  10. Hsien-Che Lee, "Method for computing scene-illuminant chromaticity from specular highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 340-347.
  11. G. Healey, "Estimating spectral reflectance using highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 335-339.
  12. D. A. Forsyth, "A novel algorithm for colour constancy," Int. J. Comput. Vis. 5, 5-36 (1990).
    [CrossRef]
  13. G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
    [CrossRef]
  14. K. Barnard, "Practical colour constancy," Ph.D. thesis (Simon Fraser University, School of Computing Science, 2000).
  15. B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.
  16. G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
    [CrossRef]
  17. K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
    [CrossRef]
  18. K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
    [CrossRef]
  19. http://www.cs.sfu.ca/~colour/data/colourlowbarconstancylowbartestlowbarimages/index.html.
  20. R. W. G. Hunt, The Reproduction of Colour, 5th ed. (Fountain, 1995).
  21. G. Strang, Linear Algebra and Its Applications (Saunders, 1988).
  22. R. V. Hogg and E. A. Tanis, Probability and Statistical Inference (Prentice Hall, 2001).
  23. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).
  24. B. K. P. Horn, Robot Vision (MIT Press, 1986).
  25. G. D. Finlayson and Ruixia Xu, "Convex programming colour constancy," in Proceedings of Workshop on Color and Photometric Methods in Computer Vision (IEEE, 2003).
  26. V. C. Cardei, B. Funt, and K. Barnard, "Estimating the scene illuminant chromaticity by using a neural network," J. Opt. Soc. Am. A 19, 2374-2386 (2002).
    [CrossRef]
  27. K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

2003 (1)

G. D. Finlayson and Ruixia Xu, "Convex programming colour constancy," in Proceedings of Workshop on Color and Photometric Methods in Computer Vision (IEEE, 2003).

2002 (4)

V. C. Cardei, B. Funt, and K. Barnard, "Estimating the scene illuminant chromaticity by using a neural network," J. Opt. Soc. Am. A 19, 2374-2386 (2002).
[CrossRef]

G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
[CrossRef]

K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
[CrossRef]

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

2001 (2)

G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
[CrossRef]

R. V. Hogg and E. A. Tanis, Probability and Statistical Inference (Prentice Hall, 2001).

2000 (2)

K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

K. Barnard, "Practical colour constancy," Ph.D. thesis (Simon Fraser University, School of Computing Science, 2000).

1998 (2)

B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.

D. H. Brainard, "Color constancy in the nearly naturalimage. 2. Achromatic loci," J. Opt. Soc. Am. A 15, 307-325 (1998).
[CrossRef]

1997 (1)

1995 (1)

R. W. G. Hunt, The Reproduction of Colour, 5th ed. (Fountain, 1995).

1993 (1)

M. Lucassen, "Quantitative studies of color constancy," Ph.D. thesis (Utrecht University, 1993).

1992 (3)

Hsien-Che Lee, "Method for computing scene-illuminant chromaticity from specular highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 340-347.

G. Healey, "Estimating spectral reflectance using highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 335-339.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

1990 (1)

D. A. Forsyth, "A novel algorithm for colour constancy," Int. J. Comput. Vis. 5, 5-36 (1990).
[CrossRef]

1988 (1)

G. Strang, Linear Algebra and Its Applications (Saunders, 1988).

1987 (1)

R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.

1986 (3)

1985 (1)

S. A. Shafer, "Using color to separate reflection components," Color Res. Appl. 10, 210-218 (1985).
[CrossRef]

1980 (1)

G. Buchsbaum, "A spatial processor model for object colour perception," J. Franklin Inst. 310, 1-26 (1980).
[CrossRef]

1977 (1)

E. H. Land, "The retinex theory of color vision," Sci. Am. 237, 108-129 (1977).
[CrossRef] [PubMed]

Arend, L.

Barnard, K.

V. C. Cardei, B. Funt, and K. Barnard, "Estimating the scene illuminant chromaticity by using a neural network," J. Opt. Soc. Am. A 19, 2374-2386 (2002).
[CrossRef]

K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
[CrossRef]

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

K. Barnard, "Practical colour constancy," Ph.D. thesis (Simon Fraser University, School of Computing Science, 2000).

K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.

Brainard, D. H.

Brunt, W. A.

Buchsbaum, G.

G. Buchsbaum, "A spatial processor model for object colour perception," J. Franklin Inst. 310, 1-26 (1980).
[CrossRef]

Cardei, V.

K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
[CrossRef]

Cardei, V. C.

Coath, A.

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

Finlayson, G. D.

G. D. Finlayson and Ruixia Xu, "Convex programming colour constancy," in Proceedings of Workshop on Color and Photometric Methods in Computer Vision (IEEE, 2003).

G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
[CrossRef]

G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
[CrossRef]

Flannery, B. P.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

Forsyth, D. A.

D. A. Forsyth, "A novel algorithm for colour constancy," Int. J. Comput. Vis. 5, 5-36 (1990).
[CrossRef]

Funt, B.

K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
[CrossRef]

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

V. C. Cardei, B. Funt, and K. Barnard, "Estimating the scene illuminant chromaticity by using a neural network," J. Opt. Soc. Am. A 19, 2374-2386 (2002).
[CrossRef]

K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.

Gershon, R.

R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.

Healey, G.

G. Healey, "Estimating spectral reflectance using highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 335-339.

Hogg, R. V.

R. V. Hogg and E. A. Tanis, Probability and Statistical Inference (Prentice Hall, 2001).

Hordley, S.

G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
[CrossRef]

Hordley, S. D.

G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
[CrossRef]

Horn, B. K.

B. K. P. Horn, Robot Vision (MIT Press, 1986).

Hubel, P.

G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
[CrossRef]

Hubel, P. M.

G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
[CrossRef]

Hunt, R. W.

R. W. G. Hunt, The Reproduction of Colour, 5th ed. (Fountain, 1995).

Jepson, A. D.

R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.

Land, E. H.

E. H. Land, "The retinex theory of color vision," Sci. Am. 237, 108-129 (1977).
[CrossRef] [PubMed]

Lee, Hsien-Che

Hsien-Che Lee, "Method for computing scene-illuminant chromaticity from specular highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 340-347.

Lucassen, M.

M. Lucassen, "Quantitative studies of color constancy," Ph.D. thesis (Utrecht University, 1993).

Maloney, L. T.

Martin, L.

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.

Press, W. H.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

Reeves, A.

Shafer, S. A.

S. A. Shafer, "Using color to separate reflection components," Color Res. Appl. 10, 210-218 (1985).
[CrossRef]

Speigle, J. M.

Strang, G.

G. Strang, Linear Algebra and Its Applications (Saunders, 1988).

Tanis, E. A.

R. V. Hogg and E. A. Tanis, Probability and Statistical Inference (Prentice Hall, 2001).

Teukolsky, S. A.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

Tsotsos, J. K.

R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.

Vetterling, W. T.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

Wandell, B. A.

Xu, Ruixia

G. D. Finlayson and Ruixia Xu, "Convex programming colour constancy," in Proceedings of Workshop on Color and Photometric Methods in Computer Vision (IEEE, 2003).

Color Res. Appl. (2)

S. A. Shafer, "Using color to separate reflection components," Color Res. Appl. 10, 210-218 (1985).
[CrossRef]

G. D. Finlayson, S. Hordley, and P. Hubel, "Illuminant estimation for object recognition," Color Res. Appl. 27, 260-270 (2002).
[CrossRef]

IEEE Trans. Image Process. (2)

K. Barnard, V. Cardei, and B. Funt, "A comparison of computational color constancy algorithms. I: Methodology and experiments with synethetic images," IEEE Trans. Image Process. 11, 972-984 (2002).
[CrossRef]

K. Barnard, L. Martin, A. Coath, and B. Funt, "A comparison of computational color constancy algorithms. II: Experiments with image data," IEEE Trans. Image Process. 11, 985-996 (2002).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

G. D. Finlayson, S. D. Hordley, and P. M. Hubel, "Color by correlation: a simple, unifying framework for color constancy," IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209-1221 (2001).
[CrossRef]

Int. J. Comput. Vis. (1)

D. A. Forsyth, "A novel algorithm for colour constancy," Int. J. Comput. Vis. 5, 5-36 (1990).
[CrossRef]

J. Franklin Inst. (1)

G. Buchsbaum, "A spatial processor model for object colour perception," J. Franklin Inst. 310, 1-26 (1980).
[CrossRef]

J. Opt. Soc. Am. A (5)

Sci. Am. (1)

E. H. Land, "The retinex theory of color vision," Sci. Am. 237, 108-129 (1977).
[CrossRef] [PubMed]

Other (14)

M. Lucassen, "Quantitative studies of color constancy," Ph.D. thesis (Utrecht University, 1993).

R. Gershon, A. D. Jepson, and J. K. Tsotsos, "From [R,G,B] to surface reflectance: computing color constant descriptors in images," in Proceeding of the International Joint Conference on Artificial Intelligence, Milan, Italy (IEEE, 1987), Vol. 2, pp. 755-758.

Hsien-Che Lee, "Method for computing scene-illuminant chromaticity from specular highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 340-347.

G. Healey, "Estimating spectral reflectance using highlights," in Color, G.E.Healey, S.A.Shafer, and L.B.Wolff, eds. (Jones and Bartlett, 1992), pp. 335-339.

http://www.cs.sfu.ca/~colour/data/colourlowbarconstancylowbartestlowbarimages/index.html.

R. W. G. Hunt, The Reproduction of Colour, 5th ed. (Fountain, 1995).

G. Strang, Linear Algebra and Its Applications (Saunders, 1988).

R. V. Hogg and E. A. Tanis, Probability and Statistical Inference (Prentice Hall, 2001).

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C: The Art of Scientific Computing (Cambridge U. Press, 1992).

B. K. P. Horn, Robot Vision (MIT Press, 1986).

G. D. Finlayson and Ruixia Xu, "Convex programming colour constancy," in Proceedings of Workshop on Color and Photometric Methods in Computer Vision (IEEE, 2003).

K. Barnard, L. Martin, and B. Funt, "Colour by correlation in a three dimensional colour space," in Proceedings of the 6th European Conference on Computer Vision (Springer, 2000), pp. 275-289.

K. Barnard, "Practical colour constancy," Ph.D. thesis (Simon Fraser University, School of Computing Science, 2000).

B. Funt, K. Barnard, and L. Martin, "Is machine colour constancy good enough?" in Proceedings of 5th European Conference on Computer Vision (Springer, 1998), pp. 455-459.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1
Fig. 1

(Color online) (top left) Histogram of r-chromaticity errors for the Max-RGB algorithm (1000 images, each with eight surfaces), (top right) histogram of Euclidean distance in chromaticity space for the same algorithm and images, (bottom left) histogram of angular errors for the same algorithm and images, (bottom right) normal-quantile plot for the angular errors from the previous plot.

Fig. 2
Fig. 2

(Color online) RMS angular error for each of the six algorithms tested in experiment 1 as a function of ( log 2 ) number of surfaces in an image.

Fig. 3
Fig. 3

(Color online) Median angular error for each of the six algorithms tested in experiment 1 as a function of ( log 2 ) number of surfaces in an image.

Fig. 4
Fig. 4

(Color online) Median angular error together with 95% confidence intervals for the three best algorithms tested in experiment 1 as a function of ( log 2 ) number of surfaces in an image.

Fig. 5
Fig. 5

(Color online) Median angular error for each of the six algorithms tested in experiment 2 as a function of ( log 2 ) number of surfaces in an image.

Tables (10)

Tables Icon

Table 1 Experiment 1: Rankings by Number of Surfaces per Image Using RMS and Median Angular Error in Sensor Space

Tables Icon

Table 2 Experiment 1: Rankings over All 6000 Images Based on Angular Error in Sensor Space

Tables Icon

Table 3 Experiment 1: Rankings over All 6000 Images Based on Angular Error in XYZ Space

Tables Icon

Table 4 Experiment 1: Rankings over All 6000 Images Based on Chromaticity Error in XYZ Space

Tables Icon

Table 5 Overall Median Angular Error for the Six Algorithms in the Two Synthetic Image Experiments

Tables Icon

Table 6 Experiment 2: Rankings over All 6000 Images Based on Angular Error in Sensor Space

Tables Icon

Table 7 RMS, Median, and Mean Angular Error Computed in Sensor Space over the 321 Real Images

Tables Icon

Table 8 Experiment 1: Rankings over All 321 Real Images Based on Angular Error in Sensor Space

Tables Icon

Table 9 RMS, Median, and Mean Angular Error Computed in XYZ Space over the 321 Real Images

Tables Icon

Table 10 Experiment 1: Rankings over All 321 Real Images Based on Angular Error in XYZ Space

Equations (25)

Equations on this page are rendered with MathJax. Learn more.

p k = ω S ( λ ) E o ( λ ) Q k ( λ ) d λ ,
E ̂ o ( λ ) = C ( P o ) .
p ̂ w o = ( p ̂ w , 1 o p ̂ w , 2 o p ̂ w , 3 o ) t = C w ( P o ) .
p c D ̂ c , o p o ,
D ̂ c , o = C D ( P o , E c ( λ ) ) .
p ̂ o = ( D ̂ c , o ) 1 p c = [ C D ( P o , E c ( λ ) ) ] 1 p c .
c ̂ w o = ( c ̂ w , 1 o c ̂ w , 2 o ) t = C c ( P o ) .
c 1 = p 1 p 1 + p 2 + p 3 , c 2 = p 2 p 1 + p 2 + p 2 , c 3 = p 3 p 1 + p 2 + p 3 .
q = ( c 1 c 2 1 c 1 c 2 ) t .
e E u c = ( c ̂ w , 1 o c w , 1 o ) 2 + ( c ̂ w , 2 o c w , 2 o ) 2 ,
e A n g = acos ( q w o t q ̂ w o q w o q ̂ w o ) .
x = M p .
e c 1 = c w , 1 o c ̂ w , 1 o , e c 2 = c w , 2 o c ̂ w , 2 o
RMSE = 1 N i = 1 N e i 2 ,
H 0 : p = 0.5 , the medians of the two distributions are the same .
H 1 : p < 0.5 , algorithm A has a lower median than that for algorithm B .
D = max < x < C A ( x ) C B ( x ) ,
H 0 : C A ( x ) = C B ( x ) , the two distribution are the same .
H 1 : C A ( x ) < C B ( x ) , errors for algorithm A are lower than those for algorithm B ,
P ( D > observed ) = Q K S ( N c + 0.12 + 0.11 N c ) ,
Q K S ( y ) = 2 j = 1 ( 1 ) j 1 e 2 j 2 y 2
p 1 0 = j = 1 M e 0 ( λ j ) s ( λ j ) q 1 ( λ j ) ,
p 2 0 = j = 1 M e 0 ( λ j ) s ( λ j ) q 2 ( λ j ) ,
p 3 0 = j = 1 M e 0 ( λ j ) s ( λ j ) q 3 ( λ j ) ,
p ̂ w , k o = p w , k c p m , k o p m , k c , k = 1 , 2 , 3 ,

Metrics