Abstract

Human observers localize events in the world by using sensory signals from multiple modalities. We evaluated two theories of spatial localization that predict how visual and auditory information are weighted when these signals specify different locations in space. According to one theory (visual capture), the signal that is typically most reliable dominates in a winner-take-all competition, whereas the other theory (maximum-likelihood estimation) proposes that perceptual judgments are based on a weighted average of the sensory signals in proportion to each signal’s relative reliability. Our results indicate that both theories are partially correct, in that relative signal reliability significantly altered judgments of spatial location, but these judgments were also characterized by an overall bias to rely on visual over auditory information. These results have important implications for the development of cue integration and for neural plasticity in the adult brain that enables humans to optimally integrate multimodal information.

© 2003 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. B. E. Stein, M. A. Meredith, The Merging of the Senses (MIT Press, Cambridge, Mass, 1993).
  2. R. Held, “Shifts in binaural localization after prolonged exposure to atypical combinations of stimuli,” Am. J. Psychol. 68, 526–548 (1955).
    [CrossRef] [PubMed]
  3. H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
    [CrossRef]
  4. R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980).
    [CrossRef] [PubMed]
  5. E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995).
    [CrossRef] [PubMed]
  6. M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998).
    [PubMed]
  7. J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Norwell, Mass, 1990).
  8. Z. Ghahramani, Computation and Psychophysics of Sensorimotor Integration, Ph.D. dissertation (Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Mass., 1995).
  9. R. A. Jacobs, “Optimal integration of texture and motion cues to depth,” Vision Res. 39, 3621–3629 (1999).
    [CrossRef]
  10. M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002).
    [CrossRef] [PubMed]
  11. D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).
  12. R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961).
    [CrossRef]
  13. G. C. Goodwin, K. S. Sin, Adaptive Filtering Prediction and Control (Prentice-Hall, Englewood Cliffs, N.J., 1984).
  14. A. L. Yuille, H. H. Bülthoff, “Bayesian decision theory and psychophysics,” in Perception as Bayesian Inference, D. C. Knill, W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 123–161.
  15. F. Wightman, Department of Psychology, University of Wisconsin, Madison, Wisconsin (personal communication,2000).
  16. J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
    [CrossRef] [PubMed]
  17. S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
    [CrossRef] [PubMed]
  18. A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).
  19. S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
    [CrossRef] [PubMed]
  20. S. Wu, S. Amari, “Neural implementation of Bayesian inference in population codes,” in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, Z. Ghahramani, eds. (MIT Press, Cambridge, Mass., 2002).
  21. L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
    [CrossRef]
  22. S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001).
    [CrossRef] [PubMed]
  23. H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
    [CrossRef]
  24. A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

2002 (3)

M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002).
[CrossRef] [PubMed]

D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).

H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
[CrossRef]

2001 (2)

S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001).
[CrossRef] [PubMed]

S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
[CrossRef] [PubMed]

2000 (2)

L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
[CrossRef]

A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).

1999 (2)

S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
[CrossRef] [PubMed]

R. A. Jacobs, “Optimal integration of texture and motion cues to depth,” Vision Res. 39, 3621–3629 (1999).
[CrossRef]

1998 (1)

M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998).
[PubMed]

1997 (1)

J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
[CrossRef] [PubMed]

1995 (1)

E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995).
[CrossRef] [PubMed]

1980 (1)

R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980).
[CrossRef] [PubMed]

1969 (1)

H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
[CrossRef]

1961 (1)

R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961).
[CrossRef]

1955 (1)

R. Held, “Shifts in binaural localization after prolonged exposure to atypical combinations of stimuli,” Am. J. Psychol. 68, 526–548 (1955).
[CrossRef] [PubMed]

Amari, S.

S. Wu, S. Amari, “Neural implementation of Bayesian inference in population codes,” in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, Z. Ghahramani, eds. (MIT Press, Cambridge, Mass., 2002).

Banks, M. S.

M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002).
[CrossRef] [PubMed]

Brainard, M. S.

M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998).
[PubMed]

E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995).
[CrossRef] [PubMed]

Bucy, R. S.

R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961).
[CrossRef]

Bülthoff, H. H.

A. L. Yuille, H. H. Bülthoff, “Bayesian decision theory and psychophysics,” in Perception as Bayesian Inference, D. C. Knill, W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 123–161.

Carlin, J. B.

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

Clark, J. J.

J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Norwell, Mass, 1990).

Dayan, P.

A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).

Deneve, S.

S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
[CrossRef] [PubMed]

S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
[CrossRef] [PubMed]

Ernst, M. O.

M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002).
[CrossRef] [PubMed]

Fine, I.

H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
[CrossRef]

Gelman, A.

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

Ghahramani, Z.

Z. Ghahramani, Computation and Psychophysics of Sensorimotor Integration, Ph.D. dissertation (Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Mass., 1995).

Goodwin, G. C.

G. C. Goodwin, K. S. Sin, Adaptive Filtering Prediction and Control (Prentice-Hall, Englewood Cliffs, N.J., 1984).

Hay, J. C.

H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
[CrossRef]

Held, R.

R. Held, “Shifts in binaural localization after prolonged exposure to atypical combinations of stimuli,” Am. J. Psychol. 68, 526–548 (1955).
[CrossRef] [PubMed]

Jacobs, R. A.

R. A. Jacobs, “Optimal integration of texture and motion cues to depth,” Vision Res. 39, 3621–3629 (1999).
[CrossRef]

Kalman, R. E.

R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961).
[CrossRef]

Kamitani, Y.

L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
[CrossRef]

Knill, D. C.

D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).

Knudsen, E. I.

M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998).
[PubMed]

E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995).
[CrossRef] [PubMed]

Latham, P. E.

S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
[CrossRef] [PubMed]

S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
[CrossRef] [PubMed]

MacLeod, D. I. A.

H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
[CrossRef]

Meredith, M. A.

B. E. Stein, M. A. Meredith, The Merging of the Senses (MIT Press, Cambridge, Mass, 1993).

Norman, J. F.

J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
[CrossRef] [PubMed]

Perotti, V. J.

J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
[CrossRef] [PubMed]

Pick, H. L.

H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
[CrossRef]

Pouget, A.

S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
[CrossRef] [PubMed]

A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).

S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
[CrossRef] [PubMed]

Rubin, D. B.

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

Saunders, J.

D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).

Shams, L.

S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001).
[CrossRef] [PubMed]

L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
[CrossRef]

Shimojo, S.

S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001).
[CrossRef] [PubMed]

L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
[CrossRef]

Sin, K. S.

G. C. Goodwin, K. S. Sin, Adaptive Filtering Prediction and Control (Prentice-Hall, Englewood Cliffs, N.J., 1984).

Smallman, H. S.

H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
[CrossRef]

Stein, B. E.

B. E. Stein, M. A. Meredith, The Merging of the Senses (MIT Press, Cambridge, Mass, 1993).

Stern, H. S.

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

Tittle, J. S.

J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
[CrossRef] [PubMed]

Warren, D. H.

R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980).
[CrossRef] [PubMed]

H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
[CrossRef]

Welch, R. B.

R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980).
[CrossRef] [PubMed]

Wightman, F.

F. Wightman, Department of Psychology, University of Wisconsin, Madison, Wisconsin (personal communication,2000).

Wu, S.

S. Wu, S. Amari, “Neural implementation of Bayesian inference in population codes,” in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, Z. Ghahramani, eds. (MIT Press, Cambridge, Mass., 2002).

Yuille, A. L.

A. L. Yuille, H. H. Bülthoff, “Bayesian decision theory and psychophysics,” in Perception as Bayesian Inference, D. C. Knill, W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 123–161.

J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Norwell, Mass, 1990).

Zemel, R. S.

A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).

Am. J. Psychol. (1)

R. Held, “Shifts in binaural localization after prolonged exposure to atypical combinations of stimuli,” Am. J. Psychol. 68, 526–548 (1955).
[CrossRef] [PubMed]

Annu. Rev. Neurosci. (1)

E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995).
[CrossRef] [PubMed]

Curr. Opin. Neurobiol. (1)

S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001).
[CrossRef] [PubMed]

J. Basic Eng. Ser. D (1)

R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961).
[CrossRef]

J. Exp. Psychol. Hum. Percept. Perform. (1)

J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997).
[CrossRef] [PubMed]

J. Neurosci. (1)

M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998).
[PubMed]

J. Vision (1)

D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).

Nat. Neurosci. (1)

S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999).
[CrossRef] [PubMed]

Nature (2)

M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002).
[CrossRef] [PubMed]

L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000).
[CrossRef]

Nature Neurosci. (1)

S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001).
[CrossRef] [PubMed]

Nature Rev. Neurosci. (1)

A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).

Percept. Psychophys (1)

H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969).
[CrossRef]

Psychol. Bull. (1)

R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980).
[CrossRef] [PubMed]

Vision Res. (2)

R. A. Jacobs, “Optimal integration of texture and motion cues to depth,” Vision Res. 39, 3621–3629 (1999).
[CrossRef]

H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002).
[CrossRef]

Other (8)

A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

S. Wu, S. Amari, “Neural implementation of Bayesian inference in population codes,” in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, Z. Ghahramani, eds. (MIT Press, Cambridge, Mass., 2002).

B. E. Stein, M. A. Meredith, The Merging of the Senses (MIT Press, Cambridge, Mass, 1993).

J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Norwell, Mass, 1990).

Z. Ghahramani, Computation and Psychophysics of Sensorimotor Integration, Ph.D. dissertation (Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Mass., 1995).

G. C. Goodwin, K. S. Sin, Adaptive Filtering Prediction and Control (Prentice-Hall, Englewood Cliffs, N.J., 1984).

A. L. Yuille, H. H. Bülthoff, “Bayesian decision theory and psychophysics,” in Perception as Bayesian Inference, D. C. Knill, W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 123–161.

F. Wightman, Department of Psychology, University of Wisconsin, Madison, Wisconsin (personal communication,2000).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (7)

Fig. 1
Fig. 1

Optimal model of sensory integration based on MLE theory. (a) Visual and auditory signals are equally reliable indicators of event location. (b) Visual signal is a more reliable indicator of event location.

Fig. 2
Fig. 2

Schematic illustration of single-modality and multimodality trials. The standard stimulus is shown on the left and the comparison stimulus is on the right. For simplicity, the comparison stimulus is shown only at one of the seven possible locations at which it could appear. (a) Auditory-only trial. (b) Visual-only trial. (c) Visual–auditory trial.

Fig. 3
Fig. 3

Results for one subject on the auditory-only trials. The horizontal axis shows the comparison locations (in degrees of visual angle away from the center of the workspace), and the vertical axis shows the percentage of trials in which the subject judged the comparison stimulus as depicting an event located to the right of the event depicted in the standard stimulus. The curve fitted to the data points is a cumulative normal distribution.

Fig. 4
Fig. 4

Results for one subject on the visual-only trials. The solid and dashed curves are cumulative normal distributions fitted to the data points in the lowest-noise and highest-noise conditions, respectively.

Fig. 5
Fig. 5

Results for one subject on the visual–auditory trials. The solid and dashed curves are cumulative normal distributions fitted to the data points in the lowest-noise and highest-noise conditions, respectively.

Fig. 6
Fig. 6

Average PSE over all ten subjects on the visual–auditory trials. The horizontal axis represents the visual noise level (1, lowest level; 5, highest level), and the vertical axis gives the average PSE in degrees of visual angle (the error bars give the standard errors of the means).

Fig. 7
Fig. 7

Average visual weights over all ten subjects on the visual–auditory trials. The horizontal axis represents the visual noise level (1, lowest level; 5, highest level), and the vertical axis gives the average visual weight (the error bars give the standard errors of the means).

Equations (6)

Equations on this page are rendered with MathJax. Learn more.

L*=wvLv*+waLa*,
wv=1/σv21/σv2+1/σa2 andwa=1/σa21/σv2+1/σa2
p(R|μ, σ2)=t=1Tptrt(1-pt)1-rt,
Lc=wvLvc+waLac,
Ls=wvLvs+waLas,
pt=p(rt=1|wv, wa)=11+exp[-(Lc-Ls)/τ],

Metrics