Manuel Melgosa, Rafael Huertas, and Roy S. Berns, "Performance of recent advanced color-difference formulas using the standardized residual sum of squares index," J. Opt. Soc. Am. A 25, 1828-1834 (2008)
The standardized residual sum of squares (STRESS) index was used to reevaluate four experimental datasets employed during the development of CIEDE2000, the current CIE recommended color-difference formula. This index enables statistical inferences not achievable by other metrics used commonly for performance evaluation. It was found that CIEDE2000 was statistically superior at a 95% confidence level to either CIE94, the previous recommended equation by the CIE, or the simple Euclidean distance in CIELAB, . Recent formulas based on the CIECAM02 color-appearance space and chroma-compressed variants of CIELAB were also evaluated and found to have only slightly reduced performance compared with CIEDE2000. These formulas have the advantage of simplicity and easier interpretation when used for quantifying color accuracy. Finally, each experimental dataset was evaluated separately rather than weight averaged as used during the development of CIEDE2000. Significant differences were found between datasets, suggesting that combining datasets may obscure important differences and that the practice of parameter optimization during formula development using combined data is likely suboptimal.
You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
Number of Color Pairs, Weighting Factors, Degrees of Freedom, and Confidence Intervals (95% Confidence Level) for the Five Experimental Datasets Analyzed in this Papera
When the squared ratios of the STRESS values from two color-difference formulas are in these intervals, the formulas are not significantly different [6, 10].
18 for the incorrect RIT-DuPont dataset, which employed 156 color pairs.
Table 2
STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the Corrected COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]a
COM-Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIELAB
43.93
42.46
40.09
33.42
51.71
CIE94
32.07
33.88
30.57
20.31
31.94
CIE94-Chroma
46.14
45.08
40.74
35.77
53.66
CIE94-Hue
35.80
37.62
36.76
CIEDE2000
27.49
29.55
19.25
19.47
30.22
CIEDE2000-Lightness
29.19
31.50
27.56
27.38
CIEDE2000-Chroma
45.18
43.54
39.66
36.84
53.04
CIEDE2000-Hue
33.86
35.57
24.38
23.05
35.42
CIEDE2000-Rot.Term
29.54
31.14
21.59
33.88
CIEDE2000-Gray
28.89
31.13
21.75
Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.
Table 3
STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas for the Incorrect and Correct RIT-DuPont and COM-Weighted Datasetsa
Incorrect RIT-DuPont
Correct RIT-DuPont
Incorrect COM-Weighted
Correct COM-Weighted
CIELAB
26.52
33.42
44.28
43.93
CIE94
18.42
20.31
32.52
32.07
CIE94-Chroma
27.01
35.77
46.44
46.14
CIE94-Hue
36.26
35.80
CIEDE2000
18.85
19.47
27.85
27.49
CIEDE2000-Lightness
29.65
29.19
CIEDE2000-Chroma
28.47
36.84
45.38
45.18
CIEDE2000-Hue
23.05
34.24
33.86
CIEDE2000-Rot.Term
29.91
29.54
CIEDE2000-Gray
15.52
29.29
28.89
The incorrect data were employed at CIEDE2000 development [3] as well as in [1]. Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.
Table 4
STRESS Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]a
COM- Weighted
BFD-P
Leeds
RIT- DuPont
Witt
CIEDE2000
27.49
29.55
19.25
19.47
30.22
CMC
30.64
30.89
24.71
27.28
35.29
DIN99d
29.24
31.70
22.76
CAM02-SCD
28.46
22.13
24.42
CAM02-UCS
29.08
30.96
24.59
OSA-GP
29.72
27.41
24.29
Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level.
Table 5
STRESS Values for the BFD-P Dataset and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulasa
BFD-P
BFD-D65
BFD-M
BFD-C
CIEDE2000
29.55
24.09
35.23
29.08
DIN99d
31.70
26.30
CAM02-SCD
26.56
CAM02-UCS
30.96
28.00
OSA-GP
27.63
Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level. The confidence intervals at this confidence level are [0.917;1.091], [0.846;1.183], and [0.757;1.321] for BFD-D65, BFD-M, and BFD-C, respectively.
Table 6
PF/3 Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the COM-Weighted Dataset and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]
COM-Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIELAB
55.71
56.31
47.23
33.79
70.94
CIE94
37.72
42.96
33.35
20.98
42.33
CIE94-Chroma
57.75
59.26
47.65
35.80
73.97
CIE94-Hue
41.91
47.45
34.54
23.15
47.84
CIEDE2000
32.11
37.31
22.02
19.56
38.78
CIEDE2000-Lightness
33.57
39.71
29.72
19.11
35.32
CIEDE2000-Chroma
55.97
55.87
45.14
37.29
72.29
CIEDE2000-Hue
38.88
43.68
27.81
24.05
45.05
CIEDE2000-Rot.Term
35.00
39.02
23.70
20.97
44.95
CIEDE2000-Gray
33.47
39.09
24.38
17.76
40.26
Table 7
PF/3 Values for Incorrect and Correct RIT-DuPont and COM-Weighted Datasets [4] Using CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions)
Incorrect RIT-DuPont
Correct RIT-DuPont
Incorrect COM-Weighted
Correct COM-Weighted
CIELAB
30.59
33.79
56.15
55.71
CIE94
20.11
20.98
38.26
37.72
CIE94-Chroma
31.72
35.80
58.15
57.75
CIE94-Hue
22.38
23.15
42.53
41.91
CIEDE2000
19.34
19.56
32.67
32.11
CIEDE2000-Lightness
18.70
19.11
34.24
33.57
CIEDE2000-Chroma
33.39
37.29
56.34
55.97
CIEDE2000-Hue
23.71
24.05
39.56
38.88
CIEDE2000-Rot.Term
20.60
20.97
35.45
35.00
CIEDE2000-Gray
16.85
17.76
34.01
33.47
Table 8
PF/3 Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]
COM- Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIEDE2000
32.11
37.31
22.02
19.56
38.78
CMC
37.50
39.90
27.34
28.09
47.07
DIN99d
34.31
39.34
26.07
21.54
40.10
CAM02-SCD
33.67
38.05
25.46
24.85
39.09
CAM02-UCS
34.30
39.27
27.92
21.90
39.52
OSA-GP
35.32
38.09
30.09
24.19
41.76
Table 9
PF/3 Values for BFD-P and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas
BFD-P
BFD-D65
BFD-M
BFD-C
CIEDE2000
37.31
29.77
51.95
36.28
DIN99d
39.34
32.09
53.23
37.10
CAM02-SCD
38.05
32.01
50.29
37.84
CAM02-UCS
39.27
33.55
51.15
39.61
OSA-GP
38.09
32.52
49.25
37.31
Tables (9)
Table 1
Number of Color Pairs, Weighting Factors, Degrees of Freedom, and Confidence Intervals (95% Confidence Level) for the Five Experimental Datasets Analyzed in this Papera
When the squared ratios of the STRESS values from two color-difference formulas are in these intervals, the formulas are not significantly different [6, 10].
18 for the incorrect RIT-DuPont dataset, which employed 156 color pairs.
Table 2
STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the Corrected COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]a
COM-Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIELAB
43.93
42.46
40.09
33.42
51.71
CIE94
32.07
33.88
30.57
20.31
31.94
CIE94-Chroma
46.14
45.08
40.74
35.77
53.66
CIE94-Hue
35.80
37.62
36.76
CIEDE2000
27.49
29.55
19.25
19.47
30.22
CIEDE2000-Lightness
29.19
31.50
27.56
27.38
CIEDE2000-Chroma
45.18
43.54
39.66
36.84
53.04
CIEDE2000-Hue
33.86
35.57
24.38
23.05
35.42
CIEDE2000-Rot.Term
29.54
31.14
21.59
33.88
CIEDE2000-Gray
28.89
31.13
21.75
Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.
Table 3
STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas for the Incorrect and Correct RIT-DuPont and COM-Weighted Datasetsa
Incorrect RIT-DuPont
Correct RIT-DuPont
Incorrect COM-Weighted
Correct COM-Weighted
CIELAB
26.52
33.42
44.28
43.93
CIE94
18.42
20.31
32.52
32.07
CIE94-Chroma
27.01
35.77
46.44
46.14
CIE94-Hue
36.26
35.80
CIEDE2000
18.85
19.47
27.85
27.49
CIEDE2000-Lightness
29.65
29.19
CIEDE2000-Chroma
28.47
36.84
45.38
45.18
CIEDE2000-Hue
23.05
34.24
33.86
CIEDE2000-Rot.Term
29.91
29.54
CIEDE2000-Gray
15.52
29.29
28.89
The incorrect data were employed at CIEDE2000 development [3] as well as in [1]. Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.
Table 4
STRESS Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]a
COM- Weighted
BFD-P
Leeds
RIT- DuPont
Witt
CIEDE2000
27.49
29.55
19.25
19.47
30.22
CMC
30.64
30.89
24.71
27.28
35.29
DIN99d
29.24
31.70
22.76
CAM02-SCD
28.46
22.13
24.42
CAM02-UCS
29.08
30.96
24.59
OSA-GP
29.72
27.41
24.29
Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level.
Table 5
STRESS Values for the BFD-P Dataset and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulasa
BFD-P
BFD-D65
BFD-M
BFD-C
CIEDE2000
29.55
24.09
35.23
29.08
DIN99d
31.70
26.30
CAM02-SCD
26.56
CAM02-UCS
30.96
28.00
OSA-GP
27.63
Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level. The confidence intervals at this confidence level are [0.917;1.091], [0.846;1.183], and [0.757;1.321] for BFD-D65, BFD-M, and BFD-C, respectively.
Table 6
PF/3 Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the COM-Weighted Dataset and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]
COM-Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIELAB
55.71
56.31
47.23
33.79
70.94
CIE94
37.72
42.96
33.35
20.98
42.33
CIE94-Chroma
57.75
59.26
47.65
35.80
73.97
CIE94-Hue
41.91
47.45
34.54
23.15
47.84
CIEDE2000
32.11
37.31
22.02
19.56
38.78
CIEDE2000-Lightness
33.57
39.71
29.72
19.11
35.32
CIEDE2000-Chroma
55.97
55.87
45.14
37.29
72.29
CIEDE2000-Hue
38.88
43.68
27.81
24.05
45.05
CIEDE2000-Rot.Term
35.00
39.02
23.70
20.97
44.95
CIEDE2000-Gray
33.47
39.09
24.38
17.76
40.26
Table 7
PF/3 Values for Incorrect and Correct RIT-DuPont and COM-Weighted Datasets [4] Using CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions)
Incorrect RIT-DuPont
Correct RIT-DuPont
Incorrect COM-Weighted
Correct COM-Weighted
CIELAB
30.59
33.79
56.15
55.71
CIE94
20.11
20.98
38.26
37.72
CIE94-Chroma
31.72
35.80
58.15
57.75
CIE94-Hue
22.38
23.15
42.53
41.91
CIEDE2000
19.34
19.56
32.67
32.11
CIEDE2000-Lightness
18.70
19.11
34.24
33.57
CIEDE2000-Chroma
33.39
37.29
56.34
55.97
CIEDE2000-Hue
23.71
24.05
39.56
38.88
CIEDE2000-Rot.Term
20.60
20.97
35.45
35.00
CIEDE2000-Gray
16.85
17.76
34.01
33.47
Table 8
PF/3 Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]
COM- Weighted
BFD-P
Leeds
RIT-DuPont
Witt
CIEDE2000
32.11
37.31
22.02
19.56
38.78
CMC
37.50
39.90
27.34
28.09
47.07
DIN99d
34.31
39.34
26.07
21.54
40.10
CAM02-SCD
33.67
38.05
25.46
24.85
39.09
CAM02-UCS
34.30
39.27
27.92
21.90
39.52
OSA-GP
35.32
38.09
30.09
24.19
41.76
Table 9
PF/3 Values for BFD-P and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas