Abstract

The utility of using inertial data for the structure-from-motion (SfM) problem is addressed. We show how inertial data can be used for improved noise resistance, reduction of inherent ambiguities, and handling of mixed-domain sequences. We also show that the number of feature points needed for accurate and robust SfM estimation can be significantly reduced when inertial data are employed. Cramér–Rao lower bounds are computed to quantify the improvements in estimating motion parameters. A robust extended-Kalman-filter-based SfM algorithm using inertial data is then developed to fully exploit the inertial information. This algorithm has been tested by using synthetic and real image sequences, and the results show the efficacy of using inertial data for the SfM problem.

© 2001 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. R. Tsai, T. Huang, “Estimating 3-D motion parameters of a rigid planar patch. 1,” IEEE Trans. Acoust. Speech Signal Process. ASP-29, 1147–1152 (1981).
    [CrossRef]
  2. T. J. Broida, R. Chellappa, “Estimation of object motion parameters from noisy image,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 90–99 (1986).
    [CrossRef]
  3. T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
    [CrossRef]
  4. T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
    [CrossRef]
  5. J. Oliensis, “A critique of structure from motion algorithms,” (NEC Research Institute, Princeton, N.J., 2000), www.neci.nj.com/ homepages/oliensis/poleiccv.ps .
  6. C. Jerian, R. Jain, “Structure from motion: a critical analysis of methods,” IEEE Trans. Syst. Man Cybern. 21, 572–588 (1991).
    [CrossRef]
  7. P. Narayanan, P. Rander, T. Kanade, “Constructing virtual worlds using dense stereo,” in Proceedings of the 6th International Conference on Computer Vision (Narosa, New Delhi, 1998), pp. 3–10.
  8. T. Viéville, O. Faugeras, “Computation of inertial information on a robot,” in Proceedings of the Fifth International Symposium on Robotics Research, H. Miura, S. Arimoto, eds. (MIT, Cambridge, Mass., 1989), pp. 57–65.
  9. T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).
  10. T. Mukai, N. Ohnishi, “The recovery of object shape and camera motion using a sensing system with a video camera and a gyro sensor,” in Proceedings of the 7th International Conference on Computer Vision (IEEE Computer Society, Los Alamitos, Calif., 1999), pp. 411–417.
  11. S. Neill, “Synchronous data sampling system for image/data fusion,” in Proceedings of the 3rd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1999), pp. 107–109.
  12. J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.
  13. A. Azarbayejani, A. Pentland, “Recursive estimation of motion, structure, and focal length,” IEEE Trans. Pattern Anal. Mach. Intell. 17, 562–575 (1995).
    [CrossRef]
  14. A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic, New York, 1970).
  15. H. Poor, An Introduction to Signal Detection and Estimation (Springer-Verlag, New York, 1988).
  16. G. Young, R. Chellappa, “Statistical analysis of inherent ambiguities in recovering 3-D motion from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 995–1013 (1992).
    [CrossRef]
  17. T. Broida, R. Chellappa, “Performance bounds for estimating three-dimensional motion parameters from a sequence of noisy images,” J. Opt. Soc. Am. A 6, 879–889 (1989).
    [CrossRef]
  18. Z. Zhang, “Determining the epipolar geometry and its uncertainty: a review,” (French National Institute for Research in Computer Science and Control, Paris, 1996).
  19. R. Tsai, T. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 13–27 (1984).
    [CrossRef]
  20. G. Adiv, “Inherent ambiguities in recovering 3-D motion and structure from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 11, 477–489 (1989).
    [CrossRef]
  21. K. Daniilidis, H. Nagel, “The coupling of rotation and translation in motion estimation of planar surfaces,” in Proceedings of the 8th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1993), pp. 188–193.
  22. D. Lawton, “Processing translational motion sequences,” Comput. Vision Graph. Image Process. 22, 116–144 (1983).
    [CrossRef]
  23. G. Adiv, “Determining 3-D motion and structure from optical flow generated by several moving objects,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-7, 384–401 (1985).
    [CrossRef]
  24. J. Oliensis, “A multi-frame structure-from-motion algorithm under perspective projection,” Int. J. Comput. Vision 34, 1–30 (1999).
    [CrossRef]
  25. C. Tomasi, J. Shi, “Good features to track,” in Proceedings of the 9th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1994), pp. 593–600.

1999 (2)

T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
[CrossRef]

J. Oliensis, “A multi-frame structure-from-motion algorithm under perspective projection,” Int. J. Comput. Vision 34, 1–30 (1999).
[CrossRef]

1995 (1)

A. Azarbayejani, A. Pentland, “Recursive estimation of motion, structure, and focal length,” IEEE Trans. Pattern Anal. Mach. Intell. 17, 562–575 (1995).
[CrossRef]

1992 (1)

G. Young, R. Chellappa, “Statistical analysis of inherent ambiguities in recovering 3-D motion from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 995–1013 (1992).
[CrossRef]

1991 (1)

C. Jerian, R. Jain, “Structure from motion: a critical analysis of methods,” IEEE Trans. Syst. Man Cybern. 21, 572–588 (1991).
[CrossRef]

1990 (1)

T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
[CrossRef]

1989 (2)

T. Broida, R. Chellappa, “Performance bounds for estimating three-dimensional motion parameters from a sequence of noisy images,” J. Opt. Soc. Am. A 6, 879–889 (1989).
[CrossRef]

G. Adiv, “Inherent ambiguities in recovering 3-D motion and structure from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 11, 477–489 (1989).
[CrossRef]

1986 (1)

T. J. Broida, R. Chellappa, “Estimation of object motion parameters from noisy image,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 90–99 (1986).
[CrossRef]

1985 (1)

G. Adiv, “Determining 3-D motion and structure from optical flow generated by several moving objects,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-7, 384–401 (1985).
[CrossRef]

1984 (1)

R. Tsai, T. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 13–27 (1984).
[CrossRef]

1983 (1)

D. Lawton, “Processing translational motion sequences,” Comput. Vision Graph. Image Process. 22, 116–144 (1983).
[CrossRef]

1981 (1)

R. Tsai, T. Huang, “Estimating 3-D motion parameters of a rigid planar patch. 1,” IEEE Trans. Acoust. Speech Signal Process. ASP-29, 1147–1152 (1981).
[CrossRef]

Adiv, G.

G. Adiv, “Inherent ambiguities in recovering 3-D motion and structure from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 11, 477–489 (1989).
[CrossRef]

G. Adiv, “Determining 3-D motion and structure from optical flow generated by several moving objects,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-7, 384–401 (1985).
[CrossRef]

Audren, J.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Azarbayejani, A.

T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
[CrossRef]

A. Azarbayejani, A. Pentland, “Recursive estimation of motion, structure, and focal length,” IEEE Trans. Pattern Anal. Mach. Intell. 17, 562–575 (1995).
[CrossRef]

Broida, T.

Broida, T. J.

T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
[CrossRef]

T. J. Broida, R. Chellappa, “Estimation of object motion parameters from noisy image,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 90–99 (1986).
[CrossRef]

Buffa, M.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Chandrashekhar, S.

T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
[CrossRef]

Chellappa, R.

G. Young, R. Chellappa, “Statistical analysis of inherent ambiguities in recovering 3-D motion from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 995–1013 (1992).
[CrossRef]

T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
[CrossRef]

T. Broida, R. Chellappa, “Performance bounds for estimating three-dimensional motion parameters from a sequence of noisy images,” J. Opt. Soc. Am. A 6, 879–889 (1989).
[CrossRef]

T. J. Broida, R. Chellappa, “Estimation of object motion parameters from noisy image,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 90–99 (1986).
[CrossRef]

Daniilidis, K.

K. Daniilidis, H. Nagel, “The coupling of rotation and translation in motion estimation of planar surfaces,” in Proceedings of the 8th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1993), pp. 188–193.

Dong, M. J.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Facao, P.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Faugeras, O.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

T. Viéville, O. Faugeras, “Computation of inertial information on a robot,” in Proceedings of the Fifth International Symposium on Robotics Research, H. Miura, S. Arimoto, eds. (MIT, Cambridge, Mass., 1989), pp. 57–65.

Hotz, B.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Howe, R.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Huang, T.

R. Tsai, T. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 13–27 (1984).
[CrossRef]

R. Tsai, T. Huang, “Estimating 3-D motion parameters of a rigid planar patch. 1,” IEEE Trans. Acoust. Speech Signal Process. ASP-29, 1147–1152 (1981).
[CrossRef]

Jain, R.

C. Jerian, R. Jain, “Structure from motion: a critical analysis of methods,” IEEE Trans. Syst. Man Cybern. 21, 572–588 (1991).
[CrossRef]

Jazwinski, A. H.

A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic, New York, 1970).

Jebara, T.

T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
[CrossRef]

Jerian, C.

C. Jerian, R. Jain, “Structure from motion: a critical analysis of methods,” IEEE Trans. Syst. Man Cybern. 21, 572–588 (1991).
[CrossRef]

Kaiser, W. J.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Kanade, T.

P. Narayanan, P. Rander, T. Kanade, “Constructing virtual worlds using dense stereo,” in Proceedings of the 6th International Conference on Computer Vision (Narosa, New Delhi, 1998), pp. 3–10.

Lawton, D.

D. Lawton, “Processing translational motion sequences,” Comput. Vision Graph. Image Process. 22, 116–144 (1983).
[CrossRef]

Mathieu, H.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Mukai, T.

T. Mukai, N. Ohnishi, “The recovery of object shape and camera motion using a sensing system with a video camera and a gyro sensor,” in Proceedings of the 7th International Conference on Computer Vision (IEEE Computer Society, Los Alamitos, Calif., 1999), pp. 411–417.

Nagel, H.

K. Daniilidis, H. Nagel, “The coupling of rotation and translation in motion estimation of planar surfaces,” in Proceedings of the 8th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1993), pp. 188–193.

Narayanan, P.

P. Narayanan, P. Rander, T. Kanade, “Constructing virtual worlds using dense stereo,” in Proceedings of the 6th International Conference on Computer Vision (Narosa, New Delhi, 1998), pp. 3–10.

Neill, S.

S. Neill, “Synchronous data sampling system for image/data fusion,” in Proceedings of the 3rd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1999), pp. 107–109.

Ohnishi, N.

T. Mukai, N. Ohnishi, “The recovery of object shape and camera motion using a sensing system with a video camera and a gyro sensor,” in Proceedings of the 7th International Conference on Computer Vision (IEEE Computer Society, Los Alamitos, Calif., 1999), pp. 411–417.

Oliensis, J.

J. Oliensis, “A multi-frame structure-from-motion algorithm under perspective projection,” Int. J. Comput. Vision 34, 1–30 (1999).
[CrossRef]

J. Oliensis, “A critique of structure from motion algorithms,” (NEC Research Institute, Princeton, N.J., 2000), www.neci.nj.com/ homepages/oliensis/poleiccv.ps .

Ortolf, J. M.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Pentland, A.

T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
[CrossRef]

A. Azarbayejani, A. Pentland, “Recursive estimation of motion, structure, and focal length,” IEEE Trans. Pattern Anal. Mach. Intell. 17, 562–575 (1995).
[CrossRef]

Poor, H.

H. Poor, An Introduction to Signal Detection and Estimation (Springer-Verlag, New York, 1988).

Rander, P.

P. Narayanan, P. Rander, T. Kanade, “Constructing virtual worlds using dense stereo,” in Proceedings of the 6th International Conference on Computer Vision (Narosa, New Delhi, 1998), pp. 3–10.

Robert, L.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Romann,

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Seshia, A.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Shi, J.

C. Tomasi, J. Shi, “Good features to track,” in Proceedings of the 9th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1994), pp. 593–600.

Tomasi, C.

C. Tomasi, J. Shi, “Good features to track,” in Proceedings of the 9th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1994), pp. 593–600.

Tsai, R.

R. Tsai, T. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 13–27 (1984).
[CrossRef]

R. Tsai, T. Huang, “Estimating 3-D motion parameters of a rigid planar patch. 1,” IEEE Trans. Acoust. Speech Signal Process. ASP-29, 1147–1152 (1981).
[CrossRef]

Viéville, T.

T. Viéville, O. Faugeras, “Computation of inertial information on a robot,” in Proceedings of the Fifth International Symposium on Robotics Research, H. Miura, S. Arimoto, eds. (MIT, Cambridge, Mass., 1989), pp. 57–65.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

Young, G.

G. Young, R. Chellappa, “Statistical analysis of inherent ambiguities in recovering 3-D motion from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 995–1013 (1992).
[CrossRef]

Yung, K. G.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Zemany, P.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

Zhang, Z.

Z. Zhang, “Determining the epipolar geometry and its uncertainty: a review,” (French National Institute for Research in Computer Science and Control, Paris, 1996).

Comput. Vision Graph. Image Process. (1)

D. Lawton, “Processing translational motion sequences,” Comput. Vision Graph. Image Process. 22, 116–144 (1983).
[CrossRef]

IEEE Signal Process. Mag. (1)

T. Jebara, A. Azarbayejani, A. Pentland, “3-D structure from 2D motion,” IEEE Signal Process. Mag. 16(3), 66–84 (1999).
[CrossRef]

IEEE Trans. Acoust. Speech Signal Process. (1)

R. Tsai, T. Huang, “Estimating 3-D motion parameters of a rigid planar patch. 1,” IEEE Trans. Acoust. Speech Signal Process. ASP-29, 1147–1152 (1981).
[CrossRef]

IEEE Trans. Aerosp. Electron. Syst. (1)

T. J. Broida, S. Chandrashekhar, R. Chellappa, “Recursive estimation of 3-D kinematics and structure from a noisy monocular image sequence,” IEEE Trans. Aerosp. Electron. Syst. 26, 639–656 (1990).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (6)

R. Tsai, T. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 13–27 (1984).
[CrossRef]

G. Adiv, “Inherent ambiguities in recovering 3-D motion and structure from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 11, 477–489 (1989).
[CrossRef]

T. J. Broida, R. Chellappa, “Estimation of object motion parameters from noisy image,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 90–99 (1986).
[CrossRef]

G. Adiv, “Determining 3-D motion and structure from optical flow generated by several moving objects,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-7, 384–401 (1985).
[CrossRef]

A. Azarbayejani, A. Pentland, “Recursive estimation of motion, structure, and focal length,” IEEE Trans. Pattern Anal. Mach. Intell. 17, 562–575 (1995).
[CrossRef]

G. Young, R. Chellappa, “Statistical analysis of inherent ambiguities in recovering 3-D motion from a noisy flow field,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 995–1013 (1992).
[CrossRef]

IEEE Trans. Syst. Man Cybern. (1)

C. Jerian, R. Jain, “Structure from motion: a critical analysis of methods,” IEEE Trans. Syst. Man Cybern. 21, 572–588 (1991).
[CrossRef]

Int. J. Comput. Vision (1)

J. Oliensis, “A multi-frame structure-from-motion algorithm under perspective projection,” Int. J. Comput. Vision 34, 1–30 (1999).
[CrossRef]

J. Opt. Soc. Am. A (1)

Other (12)

K. Daniilidis, H. Nagel, “The coupling of rotation and translation in motion estimation of planar surfaces,” in Proceedings of the 8th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1993), pp. 188–193.

C. Tomasi, J. Shi, “Good features to track,” in Proceedings of the 9th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Los Alamitos, Calif., 1994), pp. 593–600.

Z. Zhang, “Determining the epipolar geometry and its uncertainty: a review,” (French National Institute for Research in Computer Science and Control, Paris, 1996).

A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic, New York, 1970).

H. Poor, An Introduction to Signal Detection and Estimation (Springer-Verlag, New York, 1988).

P. Narayanan, P. Rander, T. Kanade, “Constructing virtual worlds using dense stereo,” in Proceedings of the 6th International Conference on Computer Vision (Narosa, New Delhi, 1998), pp. 3–10.

T. Viéville, O. Faugeras, “Computation of inertial information on a robot,” in Proceedings of the Fifth International Symposium on Robotics Research, H. Miura, S. Arimoto, eds. (MIT, Cambridge, Mass., 1989), pp. 57–65.

T. Viéville, Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P. Facao, O. Faugeras, J. Audren, “Autonomous navigation of a mobile robot using inertial and visual cues,” in Intelligent Robots and Systems, M. Kikode, T. Sato, K. Tatsuno, eds. (Publisher unknown, Yokohama, 1993).

T. Mukai, N. Ohnishi, “The recovery of object shape and camera motion using a sensing system with a video camera and a gyro sensor,” in Proceedings of the 7th International Conference on Computer Vision (IEEE Computer Society, Los Alamitos, Calif., 1999), pp. 411–417.

S. Neill, “Synchronous data sampling system for image/data fusion,” in Proceedings of the 3rd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1999), pp. 107–109.

J. M. Ortolf, P. Zemany, W. J. Kaiser, M. J. Dong, K. G. Yung, R. Howe, A. Seshia, “Microsensors for army applications,” in Proceedings of the 2nd Annual Fedlab Symposium (U.S. Army Research Laboratory, Adelphi, Md., 1998), pp. 93–98.

J. Oliensis, “A critique of structure from motion algorithms,” (NEC Research Institute, Princeton, N.J., 2000), www.neci.nj.com/ homepages/oliensis/poleiccv.ps .

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1
Fig. 1

Kalman-filter-based SfM algorithm for fusing feature correspondences and inertial data.

Fig. 2
Fig. 2

Imaging model of a moving camera.

Fig. 3
Fig. 3

Performance comparison of the R-D and A-P algorithms under various noise levels. Plots (a), (c), and (e) are the feature trajectories corrupted by AWGN with STDs of 0.5, 2, and 4 pixels, respectively, and plots (b), (d), and (f) are the corresponding motion estimates (translation along the X axis). It can be observed that when the noise level increases, the performance of the A-P algorithm deteriorates much faster than that of the R-D algorithm, which is affected only slightly by the tracking noise level.

Fig. 4
Fig. 4

CRLBs for the X translation with (dotted curves) and without (dashed curves) using inertial rate data. CRLBs with three levels of feature tracking errors are shown here. It can be observed that although the CRLBs in all cases increase with the increase in tracking noise, the CRLBs with rate data are affected only very slightly while the CRLBs without rate data increase very rapidly.

Fig. 5
Fig. 5

(σt, σin) pairs that generate the same CRLBs as the standard CRLB produced by (1 pixel, ∞). The camera moves with a uniform translational velocity of (-0.1, 0.1, 0)T unit length per second and a uniform rotational rate of (0.4, 0.3, 0.1)T rad/s. The abscissa (horizontal axis) is the STD of the inertial data σin, ranging from 0.01 to 1 rad/s. The ordinate (vertical axis) is the STD of the tracking noise, ranging from 0 to 10 pixels. Plots (a)–(f) are for the translation and global rotation about the X, Y, and Z axes, respectively.

Fig. 6
Fig. 6

Camera ego-motion estimates using feature correspondences with mismatched feature points. Plots (a), (b), and (c) are the estimates of translation along the X, Y, and Z axes, respectively. Plots (d), (e), and (f) are the estimates of global rotation about the X, Y, and Z axes, respectively.

Fig. 7
Fig. 7

(a) Camera ego-motion estimates using only seven feature points. Compared with the estimate shown in Fig. 3(d), where 13 features were used, this estimate shows that the A-P algorithm drifts away from the ground truth quickly when only seven features were tracked although the tracking noise level was the same (AWGN with STD of 2 pixels). However, the results obtained using the R-D algorithm are still very close to the ground truth and hardly affected by the reduction in the number of feature points.

Fig. 8
Fig. 8

Depth estimates using a mixed-domain sequence. Plots (a)–(l) are the depth estimates of feature points 1–12, respectively.

Fig. 9
Fig. 9

Camera ego-motion estimates using a mixed-domain sequence. Plots (a), (b), and (c) are the estimates of translation along the X, Y, and Z axes, respectively. Plots (d), (e), and (f) are the estimates of global rotation about the X, Y, and Z axes, respectively.

Fig. 10
Fig. 10

Feature points and their trajectories tracked through a translation sequence collected in experiment 1. The bar code containing inertial rate data can be seen at the bottom of both frames.

Fig. 11
Fig. 11

Camera ego-motion estimates using a translational sequence collected in experiment 1. Plots (a), (b), and (c) are the estimates of translation along the X, Y, and Z axes, respectively. Plots (d), (e), and (f) are the estimates of global rotation about the X, Y, and Z axes, respectively.

Fig. 12
Fig. 12

Feature points and their trajectories tracked through a sequence collected in experiment 2.

Fig. 13
Fig. 13

Camera ego-motion estimates using a sequence collected in experiment 2. Plots (a), (b), and (c) are the estimates of translation along the X, Y, and Z axes, respectively. Plots (d), (e), and (f) are the estimates of global rotation about the X, Y, and Z axes, respectively.

Tables (2)

Tables Icon

Table 1 3D Structure of Feature Points

Tables Icon

Table 2 Ground Truth and Two Estimates of Feature Point Structure

Equations (54)

Equations on this page are rendered with MathJax. Learn more.

ω˜x=ωx+nx,
ω˜y=ωy+ny,
ω˜z=ωz+nz,
Tk+1=Tk+nT,
Ωk+1=Ωk+nΩ,
R(Ψ, τ)=n12+(1-n12)ηn1n2(1-η)+n3ζn1n3(1-η)-n2ζn1n2(1-η)-n3ζn22+(1-n22)ηn2n3(1-η)+n1ζn1n3(1-η)+n2ζn2n3(1-η)-n1ζn32+(1-n32)η,
R(Ψk+1)=R(Ωk, tk+1-tk)R(Ψk);
Ψk+1=ϕ2 sin ϕ(r23-r32, r31-r13, r12-r21)T,
XkcYkcZkc=R(Ψk)XYZ-Tk.
u˜k=Xkc1+βZkc+ηx,v˜k=Ykc1+βZkc+ηy,
X=(1+αβ)(lu+bu),
Y=(1+αβ)(lv+bv),
Z=α,
xk=(Tk, Ωk, bu(1), bv(1), α1 ,, αL-1, bu(L), bv(L))T.
Hk=u1xk,v1xk,,uLxk,vLxk,ΩkxkT,
ulTk=1D(-1, 0, βul)T,vlTk=1D(0, -1, βvl)T,
ulΩk=1DRΩk,x(1)-βulRΩk,x(3)RΩk,y(1)-βulRΩk,y(3)RΩk,z(1)-βulRΩk,z(3)Rk-1μ000μ0001S,
vΩk=1DRΩk,x(2)-βvlRΩk,x(3)RΩk,y(2)-βvlRΩk,y(3)RΩk,z(2)-βvlRΩk,z(3)Rk-1μ000μ0001S,
D=1-βTk,z+βR(3)[(1+αlβ)(bu(l)+lu), (1+αlβ)(bv(l)+lv),αl]T,
S=[bu(l)+lu, bv(l)+lv, αl]T,
μ=1+αlβ.
(ul, vl)[(bu(l), bv(l), αl)T]=1Dμ000μ0β(bu(l)+lu)β(bv(l)+lv)1×R(1)-ulβR(3)R(2)-vlβR(3)T.
V=E[(χˆ-χ)(χˆ-χ)T]J-1,
J=E ln f(z|χ)χ ln f(z|χ)χTχ,
H=h1(χ)χ,h2(χ)χ,,hn(χ)χ,
J=HU-1HT.
J=i=1n1σi2hi(χ)χhi(χ)χT.
Ji=J(zi)=1σi2hi(χ)χhi(χ)χT,
J=i=1nJi=i=1nJ(zi),
χ=({αi}i=1L-1, T2, Ψ2 ,, TK, ΨK).
uχ=,uαl,..,uTkT,uΨkT ,T,
vχ=,vαl,..,vTkT,vΨkT, T,
uΨk=1DRx(1)-βuRx(3)Ry(1)-βuRy(3)Rz(1)-βuRz(3)μ000μ0001S,
vΨk=1DRx(2)-βvRx(3)Ry(2)-βvRy(3)Rz(2)-βvRz(3)μ000μ0001S,
Ωk=1Δt(Ψk+1-Ψk),
CRLB(σt, σin)=ΔS.
u(x, y)=ut(x, y)+ur(x, y),
v(x, y)=vt(x, y)+vr(x, y),
ut(x, y)=(xtz-tx)1z(x, y),
vt(x, y)=(ytz-ty)1z(x, y),
ur(x, y)=xyωx+(1+x2)ωy+yωz,
vr(x, y)=(1+y2)ωx-xyωy-xωz,
u˜(x, y)=u(x, y)+nu,v˜(x, y)=v(x, y)+nv,
u˜r(x, y)=xyω˜x-(1+x2)ω˜y+yω˜z=ur(x, y)+nu,r,
v˜r(x, y)=vr(x, y)+nv,r,
nu,r=xynx-(1+x2)ny+ynz,
nv,r=(1+x2)nx-xyny+xnz
σu,r2=(xy)2σx2+(1+x2)2σy2+y2σz2,
σv,r2=(xy)2σy2+(1+y2)2σx2+x2σz2.
u˜t(x, y)=u˜(x, y)-u˜r(x, y)=ut(x, y)+nu,t,
v˜t(x, y)=vt(x, y)+nv,t,
σu,t2=σu2+(xy)2σx2+(1+x2)2σy2+y2σz2,
σv,t2=σv2+(xy)2σy2+(1+y2)2σx2+x2σz2.
C(x)=[h0-h(x)]TR-1[h0-h(x)],

Metrics