Abstract

Accurate quality assessment of fused images, such as combined visible and infrared radiation images, has become increasingly important with the rise in the use of image fusion systems. We bring together three approaches, applying two objective tasks (local target analysis and global target location) to two scenarios, together with subjective quality ratings and three computational metrics. Contrast pyramid, shift-invariant discrete wavelet transform, and dual-tree complex wavelet transform fusion are applied, as well as levels of JPEG2000 compression. The differing tasks are shown to be more or less appropriate for differentiating among fusion methods, and future directions pertaining to the creation of task-specific metrics are explored.

© 2007 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |

  1. A. Toet, J. IJspeert, A. M. Waxman, and M. Aguilar, "Fusion of visible and thermal imagery improves situational awareness," Displays 18, 85-95 (1997).
    [CrossRef]
  2. E. A. Essock, J. S. McCarley, M. J. Sinai, and J. K. DeFord, "Human perception of sensor-fused imagery," in Interpreting Remote Sensing Imagery: Human Factors, R.R.Hoffman and A.B.Markman, eds. (CRC, 2001), pp. 137-182.
  3. T. D. Dixon, E. F. Canga, S. G. Nikolov, T. Troscianko, J. M. Noyes, D. R. Bull, and C. N. Canagarajah, "Quality assessment of false-colored fused images," J. Soc. Inf. Disp. 14, 883-894 (2006).
    [CrossRef]
  4. T. D. Dixon, E. F. Canga, J. M. Noyes, T. Troscianko, S. G. Nikolov, D. R. Bull, and C. N. Canagarajah, "Methods for the assessment of fused images," ACM Trans. Appl. Percept. 3, 309-332 (2006).
    [CrossRef]
  5. G. Qu, D. Zhang, and P. Yan, "Information measure for performance of image fusion," Electron. Lett. 38, 313-315 (2002).
    [CrossRef]
  6. V. S. Petrovic and C. S. Xydeas, "Sensor noise effects on signal-level image fusion performance," Inf. Fusion 4, 167-183 (2003).
    [CrossRef]
  7. G. Piella, "New quality measures for image fusion," in Proceedings of the 7th International Conference on Information Fusion (Academic, 2004), pp. 542-546.
  8. H. Li, B. S. Manjunath, and S. K. Mitra, "Multisensor image fusion using the wavelet transform," CVGIP: Graph. Models Image Process. 57, 235-245 (1995).
    [CrossRef]
  9. A. Toet, "Multiscale contrast enhancement with applications to image fusion," Opt. Eng. (Bellingham) 31, 1026-1031 (1992).
    [CrossRef]
  10. O. Rockinger, "Image sequence fusion using a shift-invariant wavelet transform," in Proceedings of the IEEE International Conference on Image Processing (IEEE, 1997), pp. 288-291.
    [CrossRef]
  11. N. G. Kingsbury, "Complex wavelets for shift invariant analysis and filtering of signals," Appl. Comput. Harmon. Anal. 10, 234-253 (2001).
    [CrossRef]
  12. T. Wilson, S. Rogers, and L. R. Meyers, "Perceptual based hyperspectral image fusion using multispectral analysis," Opt. Eng. (Bellingham) 34, 3154-3164 (1995).
    [CrossRef]
  13. S. G. Nikolov, P. Hill, D. R. Bull, and C. N. Canagarajah, "Wavelets for image fusion," in Wavelets in Signal and Image Analysis, A.Petrosian and F.Meyer, eds. (Academic, 2001), pp. 213-244.
  14. D. Taubman and M. Marcellin, JPEG2000: Image Compression Fundamentals, Standards and Practice (Academic, 2002).
    [CrossRef]
  15. D. Santa-Cruz, R. Grosbois, and T. Ebrahimi, "JPEG 2000 performance evaluation and assessment," Signal Process. 17, 113-130 (2002).
  16. A. N. Skodras, C. A. Christopoulos, and T. Ebrahimi, "JPEG2000: the upcoming still image compression standard," Pattern Recogn. Lett. 22, 1337-1345 (2001).
    [CrossRef]
  17. A. M. Eskiciouglu, "Quality measurement for monochrome compressed images in the past 25 years," in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE, 2000), pp. 1907-1910.
  18. A. Loza, T. D. Dixon, E. F. Canga, S. G. Nikolov, D. R. Bull, C. N. Canagarajah, J. M. Noyes, and T. Troscianko, "Methods of fused image analysis and assessment," in Advances and Challenges in Multisensor Data and Information Processing, NATO Security through Science Series (IOS Press, 2007).
  19. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error measurement to structural similarity," IEEE Trans. Image Process. 13, 600-612 (2004).
    [CrossRef] [PubMed]
  20. Comité Consultatif International Radio (CCIR), Method for the Subjective Assessment of the Quality of Television Pictures, CCIR Recommendation 500-3 (CCIR, 1986).
  21. Radiocummunication Sector (ITU-R), Methodology for the Subjective Assessment of the Quality of Television Picture, ITU-R Recommendation BT.500-10 (ITU-R, 2000).
  22. N. Anderson, "Algebraic models in perception," in Handbook of Perception: Psychophysical Judgement and Measurement, E.Carterette and M.Friedman, eds. (Academic, 1974), pp. 216-298.
  23. L. L. Thurstone, "A law of comparative judgement," Psychol. Rev. 34, 273-286 (1927).
    [CrossRef]
  24. B. C. Hansen and E. A. Essock, "The relationship between human perceptual performance and the physical attributes of night vision imagery," in Trends in Experimental Psychology Research, D.T.Rosen, R.S.Kozak, G.K.Carlson, M.R.Tyler, and S.V.Joist, eds. (Nova, 2005), pp. 37-66.
  25. T. D. Dixon, S. G. Nikolov, J. J. Lewis, J. Li, E. F. Canga, J. M. Noyes, T. Troscianko, D. R. Bull, and C. N. Canagarajah, "Assessment of fused videos using scanpaths: a comparison of data analysis methods," Spatial Vis. 20, 437-466 (2007).
    [CrossRef]
  26. Cedrus, SuperLab Pro Version 2.0 (1999), availeble at http://wwwSuperlab.com/.
  27. K. Tiippana, R. Nasanen, and J. Rovamo, "Contrast matching of two-dimensional compound gratings," Vision Res. 34, 1157-1163 (1994).
    [CrossRef] [PubMed]
  28. A. van Dijk and J. Martens, "Subjective quality assessment of compressed images," Signal Process. 58, 235-252 (1997).
    [CrossRef]
  29. F. M. Felisberti, J. A. Solomon, and M. J. Morgan, "The role of target salience in crowding," Prog. Aerosp. Sci. 34, 823-833 (2005).
  30. Y. Zheng, E. A. Essock, B. C. Hansen, and A. M. Haun, "A new metric based on extended spatial frequency and its application to DWT based fusion algorithms," Inf. Fusion 8, 177-192 (2007).
    [CrossRef]

2007

T. D. Dixon, S. G. Nikolov, J. J. Lewis, J. Li, E. F. Canga, J. M. Noyes, T. Troscianko, D. R. Bull, and C. N. Canagarajah, "Assessment of fused videos using scanpaths: a comparison of data analysis methods," Spatial Vis. 20, 437-466 (2007).
[CrossRef]

Y. Zheng, E. A. Essock, B. C. Hansen, and A. M. Haun, "A new metric based on extended spatial frequency and its application to DWT based fusion algorithms," Inf. Fusion 8, 177-192 (2007).
[CrossRef]

2006

T. D. Dixon, E. F. Canga, S. G. Nikolov, T. Troscianko, J. M. Noyes, D. R. Bull, and C. N. Canagarajah, "Quality assessment of false-colored fused images," J. Soc. Inf. Disp. 14, 883-894 (2006).
[CrossRef]

T. D. Dixon, E. F. Canga, J. M. Noyes, T. Troscianko, S. G. Nikolov, D. R. Bull, and C. N. Canagarajah, "Methods for the assessment of fused images," ACM Trans. Appl. Percept. 3, 309-332 (2006).
[CrossRef]

2005

F. M. Felisberti, J. A. Solomon, and M. J. Morgan, "The role of target salience in crowding," Prog. Aerosp. Sci. 34, 823-833 (2005).

2004

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error measurement to structural similarity," IEEE Trans. Image Process. 13, 600-612 (2004).
[CrossRef] [PubMed]

2003

V. S. Petrovic and C. S. Xydeas, "Sensor noise effects on signal-level image fusion performance," Inf. Fusion 4, 167-183 (2003).
[CrossRef]

2002

G. Qu, D. Zhang, and P. Yan, "Information measure for performance of image fusion," Electron. Lett. 38, 313-315 (2002).
[CrossRef]

D. Santa-Cruz, R. Grosbois, and T. Ebrahimi, "JPEG 2000 performance evaluation and assessment," Signal Process. 17, 113-130 (2002).

2001

A. N. Skodras, C. A. Christopoulos, and T. Ebrahimi, "JPEG2000: the upcoming still image compression standard," Pattern Recogn. Lett. 22, 1337-1345 (2001).
[CrossRef]

N. G. Kingsbury, "Complex wavelets for shift invariant analysis and filtering of signals," Appl. Comput. Harmon. Anal. 10, 234-253 (2001).
[CrossRef]

1997

A. Toet, J. IJspeert, A. M. Waxman, and M. Aguilar, "Fusion of visible and thermal imagery improves situational awareness," Displays 18, 85-95 (1997).
[CrossRef]

A. van Dijk and J. Martens, "Subjective quality assessment of compressed images," Signal Process. 58, 235-252 (1997).
[CrossRef]

1995

T. Wilson, S. Rogers, and L. R. Meyers, "Perceptual based hyperspectral image fusion using multispectral analysis," Opt. Eng. (Bellingham) 34, 3154-3164 (1995).
[CrossRef]

H. Li, B. S. Manjunath, and S. K. Mitra, "Multisensor image fusion using the wavelet transform," CVGIP: Graph. Models Image Process. 57, 235-245 (1995).
[CrossRef]

1994

K. Tiippana, R. Nasanen, and J. Rovamo, "Contrast matching of two-dimensional compound gratings," Vision Res. 34, 1157-1163 (1994).
[CrossRef] [PubMed]

1992

A. Toet, "Multiscale contrast enhancement with applications to image fusion," Opt. Eng. (Bellingham) 31, 1026-1031 (1992).
[CrossRef]

1927

L. L. Thurstone, "A law of comparative judgement," Psychol. Rev. 34, 273-286 (1927).
[CrossRef]

ACM Trans. Appl. Percept.

T. D. Dixon, E. F. Canga, J. M. Noyes, T. Troscianko, S. G. Nikolov, D. R. Bull, and C. N. Canagarajah, "Methods for the assessment of fused images," ACM Trans. Appl. Percept. 3, 309-332 (2006).
[CrossRef]

Appl. Comput. Harmon. Anal.

N. G. Kingsbury, "Complex wavelets for shift invariant analysis and filtering of signals," Appl. Comput. Harmon. Anal. 10, 234-253 (2001).
[CrossRef]

CVGIP: Graph. Models Image Process.

H. Li, B. S. Manjunath, and S. K. Mitra, "Multisensor image fusion using the wavelet transform," CVGIP: Graph. Models Image Process. 57, 235-245 (1995).
[CrossRef]

Displays

A. Toet, J. IJspeert, A. M. Waxman, and M. Aguilar, "Fusion of visible and thermal imagery improves situational awareness," Displays 18, 85-95 (1997).
[CrossRef]

Electron. Lett.

G. Qu, D. Zhang, and P. Yan, "Information measure for performance of image fusion," Electron. Lett. 38, 313-315 (2002).
[CrossRef]

IEEE Trans. Image Process.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error measurement to structural similarity," IEEE Trans. Image Process. 13, 600-612 (2004).
[CrossRef] [PubMed]

Inf. Fusion

Y. Zheng, E. A. Essock, B. C. Hansen, and A. M. Haun, "A new metric based on extended spatial frequency and its application to DWT based fusion algorithms," Inf. Fusion 8, 177-192 (2007).
[CrossRef]

V. S. Petrovic and C. S. Xydeas, "Sensor noise effects on signal-level image fusion performance," Inf. Fusion 4, 167-183 (2003).
[CrossRef]

J. Soc. Inf. Disp.

T. D. Dixon, E. F. Canga, S. G. Nikolov, T. Troscianko, J. M. Noyes, D. R. Bull, and C. N. Canagarajah, "Quality assessment of false-colored fused images," J. Soc. Inf. Disp. 14, 883-894 (2006).
[CrossRef]

Opt. Eng. (Bellingham)

T. Wilson, S. Rogers, and L. R. Meyers, "Perceptual based hyperspectral image fusion using multispectral analysis," Opt. Eng. (Bellingham) 34, 3154-3164 (1995).
[CrossRef]

A. Toet, "Multiscale contrast enhancement with applications to image fusion," Opt. Eng. (Bellingham) 31, 1026-1031 (1992).
[CrossRef]

Pattern Recogn. Lett.

A. N. Skodras, C. A. Christopoulos, and T. Ebrahimi, "JPEG2000: the upcoming still image compression standard," Pattern Recogn. Lett. 22, 1337-1345 (2001).
[CrossRef]

Prog. Aerosp. Sci.

F. M. Felisberti, J. A. Solomon, and M. J. Morgan, "The role of target salience in crowding," Prog. Aerosp. Sci. 34, 823-833 (2005).

Psychol. Rev.

L. L. Thurstone, "A law of comparative judgement," Psychol. Rev. 34, 273-286 (1927).
[CrossRef]

Signal Process.

A. van Dijk and J. Martens, "Subjective quality assessment of compressed images," Signal Process. 58, 235-252 (1997).
[CrossRef]

D. Santa-Cruz, R. Grosbois, and T. Ebrahimi, "JPEG 2000 performance evaluation and assessment," Signal Process. 17, 113-130 (2002).

Spatial Vis.

T. D. Dixon, S. G. Nikolov, J. J. Lewis, J. Li, E. F. Canga, J. M. Noyes, T. Troscianko, D. R. Bull, and C. N. Canagarajah, "Assessment of fused videos using scanpaths: a comparison of data analysis methods," Spatial Vis. 20, 437-466 (2007).
[CrossRef]

Vision Res.

K. Tiippana, R. Nasanen, and J. Rovamo, "Contrast matching of two-dimensional compound gratings," Vision Res. 34, 1157-1163 (1994).
[CrossRef] [PubMed]

Other

Cedrus, SuperLab Pro Version 2.0 (1999), availeble at http://wwwSuperlab.com/.

B. C. Hansen and E. A. Essock, "The relationship between human perceptual performance and the physical attributes of night vision imagery," in Trends in Experimental Psychology Research, D.T.Rosen, R.S.Kozak, G.K.Carlson, M.R.Tyler, and S.V.Joist, eds. (Nova, 2005), pp. 37-66.

Comité Consultatif International Radio (CCIR), Method for the Subjective Assessment of the Quality of Television Pictures, CCIR Recommendation 500-3 (CCIR, 1986).

Radiocummunication Sector (ITU-R), Methodology for the Subjective Assessment of the Quality of Television Picture, ITU-R Recommendation BT.500-10 (ITU-R, 2000).

N. Anderson, "Algebraic models in perception," in Handbook of Perception: Psychophysical Judgement and Measurement, E.Carterette and M.Friedman, eds. (Academic, 1974), pp. 216-298.

A. M. Eskiciouglu, "Quality measurement for monochrome compressed images in the past 25 years," in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE, 2000), pp. 1907-1910.

A. Loza, T. D. Dixon, E. F. Canga, S. G. Nikolov, D. R. Bull, C. N. Canagarajah, J. M. Noyes, and T. Troscianko, "Methods of fused image analysis and assessment," in Advances and Challenges in Multisensor Data and Information Processing, NATO Security through Science Series (IOS Press, 2007).

S. G. Nikolov, P. Hill, D. R. Bull, and C. N. Canagarajah, "Wavelets for image fusion," in Wavelets in Signal and Image Analysis, A.Petrosian and F.Meyer, eds. (Academic, 2001), pp. 213-244.

D. Taubman and M. Marcellin, JPEG2000: Image Compression Fundamentals, Standards and Practice (Academic, 2002).
[CrossRef]

O. Rockinger, "Image sequence fusion using a shift-invariant wavelet transform," in Proceedings of the IEEE International Conference on Image Processing (IEEE, 1997), pp. 288-291.
[CrossRef]

E. A. Essock, J. S. McCarley, M. J. Sinai, and J. K. DeFord, "Human perception of sensor-fused imagery," in Interpreting Remote Sensing Imagery: Human Factors, R.R.Hoffman and A.B.Markman, eds. (CRC, 2001), pp. 137-182.

G. Piella, "New quality measures for image fusion," in Proceedings of the 7th International Conference on Information Fusion (Academic, 2004), pp. 542-546.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1
Fig. 1

All target locations superimposed onto a single frame of the UN Camp sequence.

Fig. 2
Fig. 2

All experiment 1 stimuli conditions, including target locations 1 [(a), (d), (g)], 3 [(b), (e), (h)], and 6 [(c), (f), (i)].

Fig. 3
Fig. 3

Merged image showing the whole clearing, including target location 1 (left and right).

Fig. 4
Fig. 4

All experiment 2 stimuli conditions, including target locations 1 [right: (a), (d), (g)], 6 [left: (b), (e), (h)], and 1 [left: (c), (f), (i)].

Fig. 5
Fig. 5

Collapsed means of experiment 1A for reaction times.

Fig. 6
Fig. 6

Means of experiment 1A for reaction times.

Fig. 7
Fig. 7

Means of experiment 1B for subjective rating.

Fig. 8
Fig. 8

Means of experiment 1C for computational metrics.

Fig. 9
Fig. 9

Collapsed means of experiment 2A for reaction times.

Fig. 10
Fig. 10

Means of experiment 2A for reaction times.

Fig. 11
Fig. 11

Means of experiment 2B for subjective rating.

Fig. 12
Fig. 12

Means of experiment 2C for computational metrics.

Equations (3)

Equations on this page are rendered with MathJax. Learn more.

I F A ( f , a ) = a , b p F A ( f , a ) log p F A ( f , a ) p F ( f ) p A ( a ) ,
I F B ( f , b ) = a , b p F B ( f , b ) log p F B ( f , b ) p F ( f ) p B ( b ) ,
Q p A B F ( n , m ) = n = 1 N m = 1 M Q A F ( n , m ) w A ( n , m ) + Q B F ( n , m ) w B ( n , m ) n = 1 N m = 1 M w A ( n , m ) + w B ( n , m ) .

Metrics