Abstract

It is argued that important aspects of early and middle level visual coding may be understood as resulting from basic geometric processing of the visual input. The input is treated as a hypersurface defined by image intensity as a function of two spatial coordinates and time. Analytical results show how the Riemann curvature tensor R of this hypersurface represents speed and direction of motion. Moreover, the results can predict the selectivity of MT neurons for multiple motions and for motion in a direction along the optimal spatial orientation. Finally, a model based on integrated R components predicts global-motion percepts related to the barber-pole illusion.

© Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |

  1. H. R. Wilson and J. R. Bergen, "A four mechanisms model for threshold spatial vision," Vision Research 19 , 19-33 (1979 ).
    [CrossRef] [PubMed]
  2. A. B. Watson, "Detection and recognition of simple spatial forms," in Physical and biological processing of images, O. J. Braddick and A. C. Sleigh, eds. (Springer-Verlag, Berlin, 1983).
    [CrossRef]
  3. A. B. Watson and A. J. Ahumada, Jr., "Model of human visual-motion sensing," J. Opt. Soc. Am. A 2, 322-342 (1985).
    [CrossRef] [PubMed]
  4. E. H. Adelson and J. R. Bergen, "The Plenoptic Function and the Elements of Early Vision," in Computational Models of Visual Processing, M. Landy and J. A. Movshon, eds. (MIT Press, Cambridge, MA, 1991).
  5. C. Zetzsche and E. Barth, "Fundamental limits of linear filters in the visual processing of two-dimensional signals," Vision Research 30, 1111-1117 (1990).
    [CrossRef] [PubMed]
  6. C. Zetzsche, E. Barth, and B. Wegmann, "The importance of intrinsically two-dimensional image features in biological vision and picture coding," in Digital images and human vision, A. B. Watson, ed. (MIT Press, Cambridge, MA, 1993).
  7. R. M. Haralick, L. T. Watson, and T. J. Laffey, "The topographic primal sketch," International J. of Robotic Research 2, 50-72 (1983).
    [CrossRef]
  8. P. J. Besl and R. C. Jain, "Segmentation through variable-order surface fitting," IEEE Trans. Pattern Anal. Mach. Intell. 10, 167-192 (1988).
    [CrossRef]
  9. J. Shi and C. Tomasi, "Good features to track," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 593-600 (1994).
  10. J. J. Koenderink and A. J. v. Doorn, "Representation of local geometry in the visual system," Biol. Cybern. 55, 367-375 (1987).
    [CrossRef] [PubMed]
  11. D. H. Hubel and T. N. Wiesel, "Receptive fields and functional architecture of monkey striate cortex," J. Physiol. 195, 215-243 (1968).
    [PubMed]
  12. J. Y. Lettvin, H. R. Maturana, W. S. McCulloch, and W. H. Pitts, "What the frog's eye tells the frog's brain," Proceedings IRE 47, 1940-1951 (1959).
    [CrossRef]
  13. G. A. Orban, Neuronal operations in the visual cortex, (Springer, Heidelberg, 1984).
    [CrossRef]
  14. E. Peterhans and R. von der Heydt, "Functional organization of area V2 in the alert macaque," European Journal of Neuroscience 5, 509-24 (1993).
    [CrossRef] [PubMed]
  15. J. B. Levitt, D. C. Kiper, and J. A. Movshon, "Receptive fields and functional architecture of macaque V2," J Neurophysiol 71, 2517-42 (1994).
    [PubMed]
  16. C. Yu and D. M. Levi, "End stopping and length tuning in psychophysical spatial filters," J. Opt. Soc. Am. A 14, 2346-54 (1997).
    [CrossRef]
  17. A. Dobbins, S. W. Zucker, and M. S. Cynader, "Endstopping and curvature," Vision Res 29, 1371-87 (1989).
    [CrossRef] [PubMed]
  18. F. Heitger, L. Rosenthaler, R. von der Heydt, E. Peterhans, and O. Kubler, "Simulation of neural contour mechanisms: from simple to end-stopped cells," Vision Res 32, 963-81 (1992).
    [CrossRef] [PubMed]
  19. H. R. Wilson and W. A. Richards, "Mechanisms of contour curvature discrimination," J. Opt. Soc. Am. A 6, 106- 115 (1989).
    [CrossRef] [PubMed]
  20. S. P. Liou and R. C. Jain, "Motion detection in spatio-temporal space," Computer Vision, Graphics, and Image Processing 45, 227-250 (1989).
    [CrossRef]
  21. C. Zetzsche and E. Barth, "Direct detection of flow discontinuities by 3D-curvature operators," Pattern Recognition Letters 12, 771--779 (1991).
    [CrossRef]
  22. C. Zetzsche, E. Barth, and J. Berkmann, "Spatio-temporal curvature measures for flow field analysis," Geometric Methods in Computer Vision, B. Vemuri Ed. SPIE 1590, 337--350 (1991).
  23. M. P. Do Carmo, Riemannian Geometry, (Birkh�user, Boston, 1992).
  24. S. Weinberg, Gravitation and Cosmology, (Wiley and Sons, New York, 1972).
  25. B. Schutz, A first course in general relativity, (Cambridge University Press, Cambridge, 1985).
  26. E. Barth, T. Caelli, and C. Zetzsche, "Image encoding, labelling and reconstruction from differential geometry," CVGIP:GRAPHICAL MODELS AND IMAGE PROCESSING 55, 428--446 (1993).
    [CrossRef]
  27. C. Mota and J. Gomes, "Curvature Operators in Geometric Image Processing," presented at Brasilian Symposium On Computer Graphics and Image Processing, (Campinas, Brazil, 1999).
  28. E. Barth, C. Zetzsche, and G. Krieger, "Curvature measures in visual information processing," Open Systems and Information Dynamics 5, 25-39 (1998).
    [CrossRef]
  29. E. Barth, Riemann-tensor motion analysis, (2000), http://www.visionscience.com/vsDemos.html .
  30. O. Tretiak and L. Pastor, "Velocity estimation from image sequences with second order differential operators," presented at Proc. 7th Int. Conf. Pattern Recognition, (Montreal, Canada, 1984).
  31. T. S. Huang and A. N. Netravali, "Motion and structure from feature correspondence: a review," Proceedings of the IEEE 82, 252-268 (1994).
    [CrossRef]
  32. E. Barth, "Spatio-temporal curvature and the visual coding of motion," in Neural Computation (NC'2000), vol. 1404-093, H. Bothe and R. Rojas, eds. (ICSC Academic Press, Berlin, 2000).
  33. H. Hau�ecker and H. Spies, "Motion," in Handbook of Computer Vision and Applications, B. Jahne, H. Hau�ecker, and P. Geissler, eds., 1999).
  34. S. J. Nowlan and T. J. Sejnowski, "A selection model for motion processing in area MT of primates," J Neurosci 15, 1195-214 (1995).
    [PubMed]
  35. S. Wuerger, R. Shapley, and N. Rubin, ""On the visually perceived direction of motion" by Hans Wallach: 60 years later," Perception 25, 1317-1367 (1996).
    [CrossRef]
  36. F. L. Kooi, "Local direction of edge motion causes and abolishes the barberpole illusion," Vision Res 33, 2347-51 (1993).
    [CrossRef] [PubMed]
  37. E. Barth, C. Zetzsche, and I. Rentschler, "Intrinsic two-dimensional features as textons," J. Opt. Soc. Am. A Opt Image Sci Vis 15, 1723-32 (1998).
    [CrossRef] [PubMed]
  38. T. D. Albright, "Direction and orientation selectivity of neurons in visual area MT of the macaque," J. Neurophysiol 52, 1106-30 (1984).
    [PubMed]
  39. G. H. Recanzone, R. H. Wurtz, and U. Schwarz, "Responses of MT and MST neurons to one and two moving objects in the receptive field," J Neurophysiol 78, 2904-15 (1997).
  40. E. P. Simoncelli and D. J. Heeger, "A model of neuronal responses in visual area MT," Vision Res 38, 743-61 (1998).
    [CrossRef] [PubMed]
  41. E. Barth and A. B. Watson, "Nonlinear spatio-temporal model based on the geometry of the visual input," Investigative Ophthalmology and Visual Science 39, S2110 (1998).

Other (41)

H. R. Wilson and J. R. Bergen, "A four mechanisms model for threshold spatial vision," Vision Research 19 , 19-33 (1979 ).
[CrossRef] [PubMed]

A. B. Watson, "Detection and recognition of simple spatial forms," in Physical and biological processing of images, O. J. Braddick and A. C. Sleigh, eds. (Springer-Verlag, Berlin, 1983).
[CrossRef]

A. B. Watson and A. J. Ahumada, Jr., "Model of human visual-motion sensing," J. Opt. Soc. Am. A 2, 322-342 (1985).
[CrossRef] [PubMed]

E. H. Adelson and J. R. Bergen, "The Plenoptic Function and the Elements of Early Vision," in Computational Models of Visual Processing, M. Landy and J. A. Movshon, eds. (MIT Press, Cambridge, MA, 1991).

C. Zetzsche and E. Barth, "Fundamental limits of linear filters in the visual processing of two-dimensional signals," Vision Research 30, 1111-1117 (1990).
[CrossRef] [PubMed]

C. Zetzsche, E. Barth, and B. Wegmann, "The importance of intrinsically two-dimensional image features in biological vision and picture coding," in Digital images and human vision, A. B. Watson, ed. (MIT Press, Cambridge, MA, 1993).

R. M. Haralick, L. T. Watson, and T. J. Laffey, "The topographic primal sketch," International J. of Robotic Research 2, 50-72 (1983).
[CrossRef]

P. J. Besl and R. C. Jain, "Segmentation through variable-order surface fitting," IEEE Trans. Pattern Anal. Mach. Intell. 10, 167-192 (1988).
[CrossRef]

J. Shi and C. Tomasi, "Good features to track," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 593-600 (1994).

J. J. Koenderink and A. J. v. Doorn, "Representation of local geometry in the visual system," Biol. Cybern. 55, 367-375 (1987).
[CrossRef] [PubMed]

D. H. Hubel and T. N. Wiesel, "Receptive fields and functional architecture of monkey striate cortex," J. Physiol. 195, 215-243 (1968).
[PubMed]

J. Y. Lettvin, H. R. Maturana, W. S. McCulloch, and W. H. Pitts, "What the frog's eye tells the frog's brain," Proceedings IRE 47, 1940-1951 (1959).
[CrossRef]

G. A. Orban, Neuronal operations in the visual cortex, (Springer, Heidelberg, 1984).
[CrossRef]

E. Peterhans and R. von der Heydt, "Functional organization of area V2 in the alert macaque," European Journal of Neuroscience 5, 509-24 (1993).
[CrossRef] [PubMed]

J. B. Levitt, D. C. Kiper, and J. A. Movshon, "Receptive fields and functional architecture of macaque V2," J Neurophysiol 71, 2517-42 (1994).
[PubMed]

C. Yu and D. M. Levi, "End stopping and length tuning in psychophysical spatial filters," J. Opt. Soc. Am. A 14, 2346-54 (1997).
[CrossRef]

A. Dobbins, S. W. Zucker, and M. S. Cynader, "Endstopping and curvature," Vision Res 29, 1371-87 (1989).
[CrossRef] [PubMed]

F. Heitger, L. Rosenthaler, R. von der Heydt, E. Peterhans, and O. Kubler, "Simulation of neural contour mechanisms: from simple to end-stopped cells," Vision Res 32, 963-81 (1992).
[CrossRef] [PubMed]

H. R. Wilson and W. A. Richards, "Mechanisms of contour curvature discrimination," J. Opt. Soc. Am. A 6, 106- 115 (1989).
[CrossRef] [PubMed]

S. P. Liou and R. C. Jain, "Motion detection in spatio-temporal space," Computer Vision, Graphics, and Image Processing 45, 227-250 (1989).
[CrossRef]

C. Zetzsche and E. Barth, "Direct detection of flow discontinuities by 3D-curvature operators," Pattern Recognition Letters 12, 771--779 (1991).
[CrossRef]

C. Zetzsche, E. Barth, and J. Berkmann, "Spatio-temporal curvature measures for flow field analysis," Geometric Methods in Computer Vision, B. Vemuri Ed. SPIE 1590, 337--350 (1991).

M. P. Do Carmo, Riemannian Geometry, (Birkh�user, Boston, 1992).

S. Weinberg, Gravitation and Cosmology, (Wiley and Sons, New York, 1972).

B. Schutz, A first course in general relativity, (Cambridge University Press, Cambridge, 1985).

E. Barth, T. Caelli, and C. Zetzsche, "Image encoding, labelling and reconstruction from differential geometry," CVGIP:GRAPHICAL MODELS AND IMAGE PROCESSING 55, 428--446 (1993).
[CrossRef]

C. Mota and J. Gomes, "Curvature Operators in Geometric Image Processing," presented at Brasilian Symposium On Computer Graphics and Image Processing, (Campinas, Brazil, 1999).

E. Barth, C. Zetzsche, and G. Krieger, "Curvature measures in visual information processing," Open Systems and Information Dynamics 5, 25-39 (1998).
[CrossRef]

E. Barth, Riemann-tensor motion analysis, (2000), http://www.visionscience.com/vsDemos.html .

O. Tretiak and L. Pastor, "Velocity estimation from image sequences with second order differential operators," presented at Proc. 7th Int. Conf. Pattern Recognition, (Montreal, Canada, 1984).

T. S. Huang and A. N. Netravali, "Motion and structure from feature correspondence: a review," Proceedings of the IEEE 82, 252-268 (1994).
[CrossRef]

E. Barth, "Spatio-temporal curvature and the visual coding of motion," in Neural Computation (NC'2000), vol. 1404-093, H. Bothe and R. Rojas, eds. (ICSC Academic Press, Berlin, 2000).

H. Hau�ecker and H. Spies, "Motion," in Handbook of Computer Vision and Applications, B. Jahne, H. Hau�ecker, and P. Geissler, eds., 1999).

S. J. Nowlan and T. J. Sejnowski, "A selection model for motion processing in area MT of primates," J Neurosci 15, 1195-214 (1995).
[PubMed]

S. Wuerger, R. Shapley, and N. Rubin, ""On the visually perceived direction of motion" by Hans Wallach: 60 years later," Perception 25, 1317-1367 (1996).
[CrossRef]

F. L. Kooi, "Local direction of edge motion causes and abolishes the barberpole illusion," Vision Res 33, 2347-51 (1993).
[CrossRef] [PubMed]

E. Barth, C. Zetzsche, and I. Rentschler, "Intrinsic two-dimensional features as textons," J. Opt. Soc. Am. A Opt Image Sci Vis 15, 1723-32 (1998).
[CrossRef] [PubMed]

T. D. Albright, "Direction and orientation selectivity of neurons in visual area MT of the macaque," J. Neurophysiol 52, 1106-30 (1984).
[PubMed]

G. H. Recanzone, R. H. Wurtz, and U. Schwarz, "Responses of MT and MST neurons to one and two moving objects in the receptive field," J Neurophysiol 78, 2904-15 (1997).

E. P. Simoncelli and D. J. Heeger, "A model of neuronal responses in visual area MT," Vision Res 38, 743-61 (1998).
[CrossRef] [PubMed]

E. Barth and A. B. Watson, "Nonlinear spatio-temporal model based on the geometry of the visual input," Investigative Ophthalmology and Visual Science 39, S2110 (1998).

Supplementary Material (4)

» Media 1: MOV (179 KB)     
» Media 2: MOV (168 KB)     
» Media 3: MOV (147 KB)     
» Media 4: MOV (144 KB)     

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1.
Fig. 1.

(180 K) The movie shows the six Riemann-tensor components for a square (shown on the left) that appears and moves in different directions. The arrangement of the components is as in Eq. (2).

Fig. 2.
Fig. 2.

(196 K) The movie shows a moving square with colored circles (red, green, blue for intrinsic dimensions 3, 2, 1) indicating the intrinsic dimension of different movie regions.

Fig. 3.
Fig. 3.

(2*148 K) Two movies showing the Barber-pole illusion (left) and the Kooi effect (right). While in both cases the grating moves horizontally to the left behind the gray aperture, we see it moving down left in the direction of the oblique aperture in left movie. If the shape of the aperture is changed as in the right movie, the grating is seen to move horizontally. [Media 3] [Media 4]

Fig. 4.
Fig. 4.

Simulation results for the two movies in Fig. 3. Note that small changes in the shape of the aperture change the resulting global direction of motion.

Fig. 5.
Fig. 5.

Data by Albright [38] are shown in the right columns for direction (top) and orientation (bottom) selectivity of macaque MT neurons (that are selective to motion along the preferred spatial orientation) and simulation results in the left columns - see text.

Fig. 6.
Fig. 6.

Data by Recanzone et al. illustrating the selectivity of MT neurons to multiple motions are shown on the right and simulation results obtained as in Fig. 5 (but for a rotation by 135 degree) on the left. As indicated by the arrows on the left, blue is chosen for the case of a single moving dot, red for the case with an additional dot moving opposite to the preferred direction, and green for the case with an additional dot moving along the preferred direction.

Tables (1)

Tables Icon

Table 1. Summary of correspondences between motions and curvatures.

Equations (8)

Equations on this page are rendered with MathJax. Learn more.

( x , y , t , f ( x , y , t ) )
R 2121 = f xx f yy f xy 2 1 + f x 2 + f y 2 + f 1 2 ; R 3131 = f xx f tt f xt 2 1 + f x 2 + f y 2 + f t 2 ; R 3232 = f yy f tt f yt 2 1 + f x 2 + f y 2 + f t 2 ;
R 3231 = f xy f tt f xt f yt 1 + f x 2 + f y 2 + f 1 2 ; R 3121 = f xx f yt f xt f xy 1 + f x 2 + f y 2 + f t 2 ; R 3221 = f xy f yt f yy f xt 1 + f x 2 + f y 2 + f t 2 .
f : f ( x tv cos θ , y tv sin θ )
R 3221 R 2121 = R 3232 R 3221 = R 3231 R 3121 = ν cos ( θ ) ; R 3232 R 2121 = ν 2 cos ( θ ) 2 ;
R 3121 R 2121 = R 3131 R 3121 = R 3231 R 3221 = ν sin ( θ ) ; R 3131 R 2121 = ν 2 sin ( θ ) 2 .
R 3131 R 3231 = R 3121 R 3221 = R 3231 R 3232 = tan ( θ ) ; R 3131 R 3232 = tan ( θ ) 2 .
R 3131 + R 3232 = v 2 R 2121 .

Metrics