Manuel Melgosa, Rafael Huertas, and Roy S. Berns, "Performance of recent advanced color-difference formulas using the standardized residual sum of squares index," J. Opt. Soc. Am. A 25, 1828-1834 (2008)

The standardized residual sum of squares (STRESS) index was used to reevaluate four experimental datasets employed during the development of CIEDE2000, the current CIE recommended color-difference formula. This index enables statistical inferences not achievable by other metrics used commonly for performance evaluation. It was found that CIEDE2000 was statistically superior at a 95% confidence level to either CIE94, the previous recommended equation by the CIE, or the simple Euclidean distance in CIELAB, $\mathrm{\Delta}{E}_{ab}^{*}$. Recent formulas based on the CIECAM02 color-appearance space and chroma-compressed variants of CIELAB were also evaluated and found to have only slightly reduced performance compared with CIEDE2000. These formulas have the advantage of simplicity and easier interpretation when used for quantifying color accuracy. Finally, each experimental dataset was evaluated separately rather than weight averaged as used during the development of CIEDE2000. Significant differences were found between datasets, suggesting that combining datasets may obscure important differences and that the practice of parameter optimization during formula development using combined data is likely suboptimal.

You do not have subscription access to this journal. Citation lists with outbound citation links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Article level metrics are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Number of Color Pairs, Weighting Factors, Degrees of Freedom, and Confidence Intervals (95% Confidence Level) for the Five Experimental Datasets Analyzed in this Paper^{
a
}

When the squared ratios of the STRESS values from two color-difference formulas are in these intervals, the formulas are not significantly different [6, 10].
18 for the incorrect RIT-DuPont dataset, which employed 156 color pairs.

Table 2

STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the Corrected COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]^{
a
}

COM-Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIELAB

43.93

42.46

40.09

33.42

51.71

CIE94

32.07

33.88

30.57

20.31

31.94

CIE94-Chroma

46.14

45.08

40.74

35.77

53.66

CIE94-Hue

35.80

37.62

$30.{71}^{*}$

$22.{40}^{*}$

36.76

CIEDE2000

27.49

29.55

19.25

19.47

30.22

CIEDE2000-Lightness

29.19

31.50

27.56

$18.{92}^{*}$

27.38

CIEDE2000-Chroma

45.18

43.54

39.66

36.84

53.04

CIEDE2000-Hue

33.86

35.57

24.38

23.05

35.42

CIEDE2000-Rot.Term

29.54

31.14

21.59

$20.{56}^{*}$

33.88

CIEDE2000-Gray

28.89

31.13

21.75

$17.{75}^{*}$

$31.{41}^{*}$

Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.

Table 3

STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas for the Incorrect and Correct RIT-DuPont and COM-Weighted Datasets^{
a
}

Incorrect RIT-DuPont

Correct RIT-DuPont

Incorrect COM-Weighted

Correct COM-Weighted

CIELAB

26.52

33.42

44.28

43.93

CIE94

18.42

20.31

32.52

32.07

CIE94-Chroma

27.01

35.77

46.44

46.14

CIE94-Hue

$20.{40}^{*}$

$22.{40}^{*}$

36.26

35.80

CIEDE2000

18.85

19.47

27.85

27.49

CIEDE2000-Lightness

$17.{92}^{*}$

$18.{92}^{*}$

29.65

29.19

CIEDE2000-Chroma

28.47

36.84

45.38

45.18

CIEDE2000-Hue

$21.{84}^{*}$

23.05

34.24

33.86

CIEDE2000-Rot.Term

$19.{75}^{*}$

$20.{56}^{*}$

29.91

29.54

CIEDE2000-Gray

15.52

$17.{75}^{*}$

29.29

28.89

The incorrect data were employed at CIEDE2000 development [3] as well as in [1]. Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.

Table 4

STRESS Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]^{
a
}

COM- Weighted

BFD-P

Leeds

RIT- DuPont

Witt

CIEDE2000

27.49

29.55

19.25

19.47

30.22

CMC

30.64

30.89

24.71

27.28

35.29

DIN99d

29.24

31.70

22.76

$20.{91}^{*}$

$30.{06}^{*}$

CAM02-SCD

28.46

$29.{93}^{*}$

22.13

24.42

$30.{27}^{*}$

CAM02-UCS

29.08

30.96

24.59

$21.{27}^{*}$

$30.{46}^{*}$

OSA-GP

29.72

$30.{14}^{*}$

27.41

24.29

$32.{19}^{*}$

Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level.

Table 5

STRESS Values for the BFD-P Dataset and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas^{
a
}

BFD-P

BFD-D65

BFD-M

BFD-C

CIEDE2000

29.55

24.09

35.23

29.08

DIN99d

31.70

26.30

$37.{36}^{*}$

$28.{79}^{*}$

CAM02-SCD

$29.{93}^{*}$

26.56

$33.{78}^{*}$

$30.{34}^{*}$

CAM02-UCS

30.96

28.00

$34.{33}^{*}$

$32.{10}^{*}$

OSA-GP

$30.{14}^{*}$

27.63

$33.{14}^{*}$

$30.{09}^{*}$

Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level. The confidence intervals at this confidence level are [0.917;1.091], [0.846;1.183], and [0.757;1.321] for BFD-D65, BFD-M, and BFD-C, respectively.

Table 6

PF/3 Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the COM-Weighted Dataset and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]

COM-Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIELAB

55.71

56.31

47.23

33.79

70.94

CIE94

37.72

42.96

33.35

20.98

42.33

CIE94-Chroma

57.75

59.26

47.65

35.80

73.97

CIE94-Hue

41.91

47.45

34.54

23.15

47.84

CIEDE2000

32.11

37.31

22.02

19.56

38.78

CIEDE2000-Lightness

33.57

39.71

29.72

19.11

35.32

CIEDE2000-Chroma

55.97

55.87

45.14

37.29

72.29

CIEDE2000-Hue

38.88

43.68

27.81

24.05

45.05

CIEDE2000-Rot.Term

35.00

39.02

23.70

20.97

44.95

CIEDE2000-Gray

33.47

39.09

24.38

17.76

40.26

Table 7

PF/3 Values for Incorrect and Correct RIT-DuPont and COM-Weighted Datasets [4] Using CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions)

Incorrect RIT-DuPont

Correct RIT-DuPont

Incorrect COM-Weighted

Correct COM-Weighted

CIELAB

30.59

33.79

56.15

55.71

CIE94

20.11

20.98

38.26

37.72

CIE94-Chroma

31.72

35.80

58.15

57.75

CIE94-Hue

22.38

23.15

42.53

41.91

CIEDE2000

19.34

19.56

32.67

32.11

CIEDE2000-Lightness

18.70

19.11

34.24

33.57

CIEDE2000-Chroma

33.39

37.29

56.34

55.97

CIEDE2000-Hue

23.71

24.05

39.56

38.88

CIEDE2000-Rot.Term

20.60

20.97

35.45

35.00

CIEDE2000-Gray

16.85

17.76

34.01

33.47

Table 8

PF/3 Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]

COM- Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIEDE2000

32.11

37.31

22.02

19.56

38.78

CMC

37.50

39.90

27.34

28.09

47.07

DIN99d

34.31

39.34

26.07

21.54

40.10

CAM02-SCD

33.67

38.05

25.46

24.85

39.09

CAM02-UCS

34.30

39.27

27.92

21.90

39.52

OSA-GP

35.32

38.09

30.09

24.19

41.76

Table 9

PF/3 Values for BFD-P and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas

BFD-P

BFD-D65

BFD-M

BFD-C

CIEDE2000

37.31

29.77

51.95

36.28

DIN99d

39.34

32.09

53.23

37.10

CAM02-SCD

38.05

32.01

50.29

37.84

CAM02-UCS

39.27

33.55

51.15

39.61

OSA-GP

38.09

32.52

49.25

37.31

Tables (9)

Table 1

Number of Color Pairs, Weighting Factors, Degrees of Freedom, and Confidence Intervals (95% Confidence Level) for the Five Experimental Datasets Analyzed in this Paper^{
a
}

When the squared ratios of the STRESS values from two color-difference formulas are in these intervals, the formulas are not significantly different [6, 10].
18 for the incorrect RIT-DuPont dataset, which employed 156 color pairs.

Table 2

STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the Corrected COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]^{
a
}

COM-Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIELAB

43.93

42.46

40.09

33.42

51.71

CIE94

32.07

33.88

30.57

20.31

31.94

CIE94-Chroma

46.14

45.08

40.74

35.77

53.66

CIE94-Hue

35.80

37.62

$30.{71}^{*}$

$22.{40}^{*}$

36.76

CIEDE2000

27.49

29.55

19.25

19.47

30.22

CIEDE2000-Lightness

29.19

31.50

27.56

$18.{92}^{*}$

27.38

CIEDE2000-Chroma

45.18

43.54

39.66

36.84

53.04

CIEDE2000-Hue

33.86

35.57

24.38

23.05

35.42

CIEDE2000-Rot.Term

29.54

31.14

21.59

$20.{56}^{*}$

33.88

CIEDE2000-Gray

28.89

31.13

21.75

$17.{75}^{*}$

$31.{41}^{*}$

Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.

Table 3

STRESS Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas for the Incorrect and Correct RIT-DuPont and COM-Weighted Datasets^{
a
}

Incorrect RIT-DuPont

Correct RIT-DuPont

Incorrect COM-Weighted

Correct COM-Weighted

CIELAB

26.52

33.42

44.28

43.93

CIE94

18.42

20.31

32.52

32.07

CIE94-Chroma

27.01

35.77

46.44

46.14

CIE94-Hue

$20.{40}^{*}$

$22.{40}^{*}$

36.26

35.80

CIEDE2000

18.85

19.47

27.85

27.49

CIEDE2000-Lightness

$17.{92}^{*}$

$18.{92}^{*}$

29.65

29.19

CIEDE2000-Chroma

28.47

36.84

45.38

45.18

CIEDE2000-Hue

$21.{84}^{*}$

23.05

34.24

33.86

CIEDE2000-Rot.Term

$19.{75}^{*}$

$20.{56}^{*}$

29.91

29.54

CIEDE2000-Gray

15.52

$17.{75}^{*}$

29.29

28.89

The incorrect data were employed at CIEDE2000 development [3] as well as in [1]. Bold is employed for the results found with the complete CIE formulas: CIELAB, CIE94, and CIEDE2000. The asterisks indicate no significant differences between complete (CIE94 or CIEDE2000) and reduced formulas, at the 95% confidence level.

Table 4

STRESS Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]^{
a
}

COM- Weighted

BFD-P

Leeds

RIT- DuPont

Witt

CIEDE2000

27.49

29.55

19.25

19.47

30.22

CMC

30.64

30.89

24.71

27.28

35.29

DIN99d

29.24

31.70

22.76

$20.{91}^{*}$

$30.{06}^{*}$

CAM02-SCD

28.46

$29.{93}^{*}$

22.13

24.42

$30.{27}^{*}$

CAM02-UCS

29.08

30.96

24.59

$21.{27}^{*}$

$30.{46}^{*}$

OSA-GP

29.72

$30.{14}^{*}$

27.41

24.29

$32.{19}^{*}$

Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level.

Table 5

STRESS Values for the BFD-P Dataset and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas^{
a
}

BFD-P

BFD-D65

BFD-M

BFD-C

CIEDE2000

29.55

24.09

35.23

29.08

DIN99d

31.70

26.30

$37.{36}^{*}$

$28.{79}^{*}$

CAM02-SCD

$29.{93}^{*}$

26.56

$33.{78}^{*}$

$30.{34}^{*}$

CAM02-UCS

30.96

28.00

$34.{33}^{*}$

$32.{10}^{*}$

OSA-GP

$30.{14}^{*}$

27.63

$33.{14}^{*}$

$30.{09}^{*}$

Bold is employed for the results found with the last CIE-recommended formula, CIEDE2000. The asterisks indicate no significant differences with respect to CIEDE2000 at the 95% confidence level. The confidence intervals at this confidence level are [0.917;1.091], [0.846;1.183], and [0.757;1.321] for BFD-D65, BFD-M, and BFD-C, respectively.

Table 6

PF/3 Values Computed for CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions) Color-Difference Formulas Using the COM-Weighted Dataset and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]

COM-Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIELAB

55.71

56.31

47.23

33.79

70.94

CIE94

37.72

42.96

33.35

20.98

42.33

CIE94-Chroma

57.75

59.26

47.65

35.80

73.97

CIE94-Hue

41.91

47.45

34.54

23.15

47.84

CIEDE2000

32.11

37.31

22.02

19.56

38.78

CIEDE2000-Lightness

33.57

39.71

29.72

19.11

35.32

CIEDE2000-Chroma

55.97

55.87

45.14

37.29

72.29

CIEDE2000-Hue

38.88

43.68

27.81

24.05

45.05

CIEDE2000-Rot.Term

35.00

39.02

23.70

20.97

44.95

CIEDE2000-Gray

33.47

39.09

24.38

17.76

40.26

Table 7

PF/3 Values for Incorrect and Correct RIT-DuPont and COM-Weighted Datasets [4] Using CIELAB, CIE94 (with Its Two Reduced Versions), and CIEDE2000 (with Its Five Reduced Versions)

Incorrect RIT-DuPont

Correct RIT-DuPont

Incorrect COM-Weighted

Correct COM-Weighted

CIELAB

30.59

33.79

56.15

55.71

CIE94

20.11

20.98

38.26

37.72

CIE94-Chroma

31.72

35.80

58.15

57.75

CIE94-Hue

22.38

23.15

42.53

41.91

CIEDE2000

19.34

19.56

32.67

32.11

CIEDE2000-Lightness

18.70

19.11

34.24

33.57

CIEDE2000-Chroma

33.39

37.29

56.34

55.97

CIEDE2000-Hue

23.71

24.05

39.56

38.88

CIEDE2000-Rot.Term

20.60

20.97

35.45

35.00

CIEDE2000-Gray

16.85

17.76

34.01

33.47

Table 8

PF/3 Values Computed for CIEDE2000 and Other Recent Advanced Color-Difference Formulas Using the Experimental COM-Weighted Dataset [3] and Its Four Individual Datasets: BFD-P [16], Leeds [17], RIT-DuPont [5], and Witt [18]

COM- Weighted

BFD-P

Leeds

RIT-DuPont

Witt

CIEDE2000

32.11

37.31

22.02

19.56

38.78

CMC

37.50

39.90

27.34

28.09

47.07

DIN99d

34.31

39.34

26.07

21.54

40.10

CAM02-SCD

33.67

38.05

25.46

24.85

39.09

CAM02-UCS

34.30

39.27

27.92

21.90

39.52

OSA-GP

35.32

38.09

30.09

24.19

41.76

Table 9

PF/3 Values for BFD-P and Its Three Subsets (BFD-D65, BFD-M, and BFD-C) for CIEDE2000 and Four Recent Advanced Color-Difference Formulas