David M. Rouse, Sheila S. Hemami, Romuald Pépion, and Patrick Le Callet, "Estimating the usefulness of distorted natural images using an image contour degradation measure," J. Opt. Soc. Am. A 28, 157-188 (2011)
Quality estimators aspire to quantify the perceptual resemblance, but not the usefulness, of a distorted image when compared to a reference natural image. However, humans can successfully accomplish tasks (e.g., object identification) using visibly distorted images that are not necessarily of high quality. A suite of novel subjective experiments reveals that quality does not accurately predict utility (i.e., usefulness). Thus, even accurate quality estimators cannot accurately estimate utility. In the absence of utility estimators, leading quality estimators are assessed as both quality and utility estimators and dismantled to understand those image characteristics that distinguish utility from quality. A newly proposed utility estimator demonstrates that a measure of contour degradation is sufficient to accurately estimate utility and is argued to be compatible with shape-based theories of object perception.
Timothy D. Dixon, Eduardo Fernández Canga, Stavri G. Nikolov, Tom Troscianko, Jan M. Noyes, C. Nishan Canagarajah, and Dave R. Bull J. Opt. Soc. Am. A 24(12) B125-B135 (2007)
You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
Quantized DCT coefficients according to the lossy JPEG image compression standard. Parameterized by JPEG quality parameter .
Increasing decreases the level of distortion.
Quantized discrete wavelet transform coefficients using quantization step-sizes specified by the DCQ strategy for a target encoding bitrate, R.
Increasing R decreases the level of distortion.
BLOCK
Replace each block of pixels by their average and quantize this average pixel value using the quantization parameter .
Decreasing decreases the level of distortion.
TS
TS with limited disruption to image edges. Parameterize by TS parameter γ.
Decreasing γ decreases the level of distortion.
TS (i.e., TS distortions) plus high-pass filtering. Parameterize by TS parameter γ.
Decreasing γ decreases the level of distortion.
The relationship between the distortion parameter and the level of distortion is described for each distortion. For a reference image subjected to one distortion type, utility and quality are assumed to exhibit a monotonically, nondecreasing relationship with decreasing distortion level.
Table 2
Results Summarizing the Relationship between Perceived Quality and Perceived Utilitya
Factor
Image Subset
n
r
ρ
τ
RMSE
OR
All
163
0.909
0.919
0.750
14.2
0.925
0.58
Quality region
Low quality
72
0.819
0.791
0.606
12.4
0.812
0.58
Medium quality
63
0.620
0.625
0.458
0.627
0.67
High quality
28
0.603
0.583
0.402
8.7
0.614
0.32
Distortion
JPEG
39
0.931
0.938
0.795
11.2
0.939
0.62
BLOCKS
6
0.228
0.116
0.138
6.3
0.221
0.00
42
0.953
0.953
0.825
11.5
0.955
0.45
TS
38
0.963
0.934
0.769
11.0
0.957
0.50
38
0.884
0.868
0.690
0.894
0.71
Reference image
Airplane (set 1)
18
0.981
0.976
0.905
6.0
0.986
0.28
Backhoe (set 1)
16
0.968
0.945
0.812
7.7
0.972
0.31
Guitarist (set 1)
21
0.940
0.966
0.865
8.5
0.977
0.43
Jackolanterns (set 1)
18
0.953
0.975
0.892
7.4
0.974
0.22
Boy and cat (set 2)
16
0.936
0.895
0.740
12.9
0.949
0.56
Caged birds (set 2)
13
0.950
0.945
0.821
11.9
0.942
0.54
Pianist (set 2)
21
0.912
0.943
0.823
11.9
0.950
0.33
Skier (set 2)
19
0.907
0.942
0.826
12.9
0.945
0.42
Train (set 2)
21
0.924
0.927
0.794
11.8
0.951
0.48
Sets of references
Set 1
73
0.940
0.948
0.800
10.8
0.954
0.47
Set 2
90
0.893
0.895
0.714
16.1
0.909
0.64
Each row corresponds to a subset of n images either spanning a particular range of quality or corresponding to a particular distortion. The Pearson linear correlation r, the Spearman rank correlation ρ, and the Kendall rank correlation τ are computed between the perceived quality and perceived utility scores. The RMSE and the OR were computed using the utility scores and the mapped [i.e., Eq. (1)] quality scores. denotes the Pearson linear correlation after applying the mapping. For the correlation statistics and OR, bold values are statistically equivalent to the largest value for a subset of images (excluding All). Bold RMSE values are statistically larger than the other subsets based on ANOVA.
Table 3
Statistics Summarizing the Performance of Estimators as Utility Estimatorsa
Estimating Perceived Utility
Correlation Measures
Accuracy Measures
Estimator
Recognition Detection Accuracy
ρ
τ
r
RMSE
OR
Skew/Kurt
Spectral slope
β
0.729
0.751
0.535
0.730
25.6
0.748
64.4
Signal fidelity measures
PSNR
0.768
0.520
0.422
0.414
34.1
0.859
57.3
0.792
0.521
0.404
0.211
36.6
0.877
38.2
Estimators based on HVS properties
WSNR
0.766
0.485
0.372
0.415
34.0
0.847
57.6
NQM
0.796
0.509
0.401
0.422
33.9
0.847
54.1
VSNR
0.790
0.530
0.436
0.541
31.5
0.742
83.9
C4
0.830
0.661
0.517
0.651
28.4
0.785
75.9
Estimators based on hypothesized HVS objectives
SSIM
0.924
0.862
0.682
0.845
20.0
0.595
55.2
MS-SSIM
0.935
0.731
0.585
0.652
28.4
0.828
66.4
VIF
0.978
0.959
0.821
0.943
12.4
0.595
26.6
1
VIF*
0.973
0.928
0.768
0.924
14.3
0.497
41.1
0.850
Proposed utility estimators
0.980
0.951
0.804
0.924
14.3
0.564
33.6
0.398
0.980
0.937
0.785
0.935
13.3
0.454
39.1
0.472
0.979
0.956
0.816
0.923
14.4
0.583
33.0
0.296
0.980
0.959
0.821
0.911
15.4
0.577
33.4
0.073
0.980
0.958
0.817
0.902
16.2
0.601
34.0
0.016
0.981
0.947
0.794
0.901
16.3
0.601
34.5
0.008
The recognition detection accuracy is the probability that an unrecognizable image and a recognizable image are correctly distinguished. The Pearson (linear) correlation coefficient r, the Spearman rank correlation coefficient ρ, the Kendall rank correlation τ, the RMSE, the OR, and the resolving power are reported when the estimates are compared with the perceived utility scores for test images with perceived utility exceeding ( test images). Italicized p values for the BFL test () indicate that the residual variance is statistically equivalent to that of VIF. The skewness and kurtosis of the residuals are italicized when the JB test indicates that the residuals belong to a Gaussian distribution (see Section 6). Except for the skewness and kurtosis statistics, optimal values appear in bold with statistically equivalent values italicized.
Table 4
Statistics Summarizing the Performance of Objective Estimators as Quality Estimatorsa
Correlation Measures
Accuracy Measures
Estimator
ρ
τ
r
RMSE
OR
Skew/Kurt
Spectral slope
β
0.518
0.331
0.585
0.895
0.835
1.902
Signal fidelity measures
PSNR
0.598
0.477
0.656
0.833
0.506
1.949
0.627
0.480
0.401
1.011
0.881
2.413
Estimators based on HVS properties
WSNR
0.582
0.443
0.648
0.841
0.823
2.052
NQM
0.600
0.461
0.666
0.823
0.831
1.911
VSNR
0.607
0.466
0.738
0.745
0.794
1.760
C4
0.822
0.636
0.832
0.615
0.808
1.600
Estimators based on hypothesized HVS objectives
SSIM
0.870
0.696
0.883
0.519
0.700
2.517
MS-SSIM
0.713
0.561
0.603
0.850
0.864
1.918
VIF
0.929
0.774
0.950
0.345
0.531
0.828
0.13
VIF*
0.938
0.799
0.959
0.313
0.568
1.056
1
Proposed utility estimators
0.932
0.780
0.885
0.515
0.786
2.076
0.914
0.746
0.934
0.394
0.568
1.020
0.35
0.935
0.784
0.875
0.535
0.778
2.256
0.937
0.789
0.860
0.563
0.765
2.405
0.940
0.796
0.855
0.572
0.782
2.291
0.946
0.810
0.855
0.572
0.757
2.254
The Pearson (linear) correlation coefficient r, the Spearman rank correlation coefficient ρ, the Kendall rank correlation τ, the RMSE, the OR, and the resolving power are reported when the estimates are compared with the perceived quality scores for all test images (). Italicized p values corresponding to the BFL test () indicate that the residual variance is statistically equivalent to that of VIF*. The skewness and kurtosis of the residuals are italicized when the JB test indicated that the residuals belong to a Gaussian distribution (see Section 6). Except for the skewness and kurtosis statistics, optimal values appear in bold with statistically equivalent values italicized.
Quantized DCT coefficients according to the lossy JPEG image compression standard. Parameterized by JPEG quality parameter .
Increasing decreases the level of distortion.
Quantized discrete wavelet transform coefficients using quantization step-sizes specified by the DCQ strategy for a target encoding bitrate, R.
Increasing R decreases the level of distortion.
BLOCK
Replace each block of pixels by their average and quantize this average pixel value using the quantization parameter .
Decreasing decreases the level of distortion.
TS
TS with limited disruption to image edges. Parameterize by TS parameter γ.
Decreasing γ decreases the level of distortion.
TS (i.e., TS distortions) plus high-pass filtering. Parameterize by TS parameter γ.
Decreasing γ decreases the level of distortion.
The relationship between the distortion parameter and the level of distortion is described for each distortion. For a reference image subjected to one distortion type, utility and quality are assumed to exhibit a monotonically, nondecreasing relationship with decreasing distortion level.
Table 2
Results Summarizing the Relationship between Perceived Quality and Perceived Utilitya
Factor
Image Subset
n
r
ρ
τ
RMSE
OR
All
163
0.909
0.919
0.750
14.2
0.925
0.58
Quality region
Low quality
72
0.819
0.791
0.606
12.4
0.812
0.58
Medium quality
63
0.620
0.625
0.458
0.627
0.67
High quality
28
0.603
0.583
0.402
8.7
0.614
0.32
Distortion
JPEG
39
0.931
0.938
0.795
11.2
0.939
0.62
BLOCKS
6
0.228
0.116
0.138
6.3
0.221
0.00
42
0.953
0.953
0.825
11.5
0.955
0.45
TS
38
0.963
0.934
0.769
11.0
0.957
0.50
38
0.884
0.868
0.690
0.894
0.71
Reference image
Airplane (set 1)
18
0.981
0.976
0.905
6.0
0.986
0.28
Backhoe (set 1)
16
0.968
0.945
0.812
7.7
0.972
0.31
Guitarist (set 1)
21
0.940
0.966
0.865
8.5
0.977
0.43
Jackolanterns (set 1)
18
0.953
0.975
0.892
7.4
0.974
0.22
Boy and cat (set 2)
16
0.936
0.895
0.740
12.9
0.949
0.56
Caged birds (set 2)
13
0.950
0.945
0.821
11.9
0.942
0.54
Pianist (set 2)
21
0.912
0.943
0.823
11.9
0.950
0.33
Skier (set 2)
19
0.907
0.942
0.826
12.9
0.945
0.42
Train (set 2)
21
0.924
0.927
0.794
11.8
0.951
0.48
Sets of references
Set 1
73
0.940
0.948
0.800
10.8
0.954
0.47
Set 2
90
0.893
0.895
0.714
16.1
0.909
0.64
Each row corresponds to a subset of n images either spanning a particular range of quality or corresponding to a particular distortion. The Pearson linear correlation r, the Spearman rank correlation ρ, and the Kendall rank correlation τ are computed between the perceived quality and perceived utility scores. The RMSE and the OR were computed using the utility scores and the mapped [i.e., Eq. (1)] quality scores. denotes the Pearson linear correlation after applying the mapping. For the correlation statistics and OR, bold values are statistically equivalent to the largest value for a subset of images (excluding All). Bold RMSE values are statistically larger than the other subsets based on ANOVA.
Table 3
Statistics Summarizing the Performance of Estimators as Utility Estimatorsa
Estimating Perceived Utility
Correlation Measures
Accuracy Measures
Estimator
Recognition Detection Accuracy
ρ
τ
r
RMSE
OR
Skew/Kurt
Spectral slope
β
0.729
0.751
0.535
0.730
25.6
0.748
64.4
Signal fidelity measures
PSNR
0.768
0.520
0.422
0.414
34.1
0.859
57.3
0.792
0.521
0.404
0.211
36.6
0.877
38.2
Estimators based on HVS properties
WSNR
0.766
0.485
0.372
0.415
34.0
0.847
57.6
NQM
0.796
0.509
0.401
0.422
33.9
0.847
54.1
VSNR
0.790
0.530
0.436
0.541
31.5
0.742
83.9
C4
0.830
0.661
0.517
0.651
28.4
0.785
75.9
Estimators based on hypothesized HVS objectives
SSIM
0.924
0.862
0.682
0.845
20.0
0.595
55.2
MS-SSIM
0.935
0.731
0.585
0.652
28.4
0.828
66.4
VIF
0.978
0.959
0.821
0.943
12.4
0.595
26.6
1
VIF*
0.973
0.928
0.768
0.924
14.3
0.497
41.1
0.850
Proposed utility estimators
0.980
0.951
0.804
0.924
14.3
0.564
33.6
0.398
0.980
0.937
0.785
0.935
13.3
0.454
39.1
0.472
0.979
0.956
0.816
0.923
14.4
0.583
33.0
0.296
0.980
0.959
0.821
0.911
15.4
0.577
33.4
0.073
0.980
0.958
0.817
0.902
16.2
0.601
34.0
0.016
0.981
0.947
0.794
0.901
16.3
0.601
34.5
0.008
The recognition detection accuracy is the probability that an unrecognizable image and a recognizable image are correctly distinguished. The Pearson (linear) correlation coefficient r, the Spearman rank correlation coefficient ρ, the Kendall rank correlation τ, the RMSE, the OR, and the resolving power are reported when the estimates are compared with the perceived utility scores for test images with perceived utility exceeding ( test images). Italicized p values for the BFL test () indicate that the residual variance is statistically equivalent to that of VIF. The skewness and kurtosis of the residuals are italicized when the JB test indicates that the residuals belong to a Gaussian distribution (see Section 6). Except for the skewness and kurtosis statistics, optimal values appear in bold with statistically equivalent values italicized.
Table 4
Statistics Summarizing the Performance of Objective Estimators as Quality Estimatorsa
Correlation Measures
Accuracy Measures
Estimator
ρ
τ
r
RMSE
OR
Skew/Kurt
Spectral slope
β
0.518
0.331
0.585
0.895
0.835
1.902
Signal fidelity measures
PSNR
0.598
0.477
0.656
0.833
0.506
1.949
0.627
0.480
0.401
1.011
0.881
2.413
Estimators based on HVS properties
WSNR
0.582
0.443
0.648
0.841
0.823
2.052
NQM
0.600
0.461
0.666
0.823
0.831
1.911
VSNR
0.607
0.466
0.738
0.745
0.794
1.760
C4
0.822
0.636
0.832
0.615
0.808
1.600
Estimators based on hypothesized HVS objectives
SSIM
0.870
0.696
0.883
0.519
0.700
2.517
MS-SSIM
0.713
0.561
0.603
0.850
0.864
1.918
VIF
0.929
0.774
0.950
0.345
0.531
0.828
0.13
VIF*
0.938
0.799
0.959
0.313
0.568
1.056
1
Proposed utility estimators
0.932
0.780
0.885
0.515
0.786
2.076
0.914
0.746
0.934
0.394
0.568
1.020
0.35
0.935
0.784
0.875
0.535
0.778
2.256
0.937
0.789
0.860
0.563
0.765
2.405
0.940
0.796
0.855
0.572
0.782
2.291
0.946
0.810
0.855
0.572
0.757
2.254
The Pearson (linear) correlation coefficient r, the Spearman rank correlation coefficient ρ, the Kendall rank correlation τ, the RMSE, the OR, and the resolving power are reported when the estimates are compared with the perceived quality scores for all test images (). Italicized p values corresponding to the BFL test () indicate that the residual variance is statistically equivalent to that of VIF*. The skewness and kurtosis of the residuals are italicized when the JB test indicated that the residuals belong to a Gaussian distribution (see Section 6). Except for the skewness and kurtosis statistics, optimal values appear in bold with statistically equivalent values italicized.