Abstract

One major research issue associated with 3D perception by robotic systems is the creation of efficient sensor systems that can generate dense range maps reliably. A visual sensor system for robotic applications is developed that is inherently equipped with two types of sensor, an active trinocular vision and a passive stereo vision. Unlike in conventional active vision systems that use a large number of images with variations of projected patterns for dense range map acquisition or from conventional passive vision systems that work well on specific environments with sufficient feature information, a cooperative bidirectional sensor fusion method for this visual sensor system enables us to acquire a reliable dense range map using active and passive information simultaneously. The fusion algorithms are composed of two parts, one in which the passive stereo vision helps active vision and the other in which the active trinocular vision helps the passive one. The first part matches the laser patterns in stereo laser images with the help of intensity images; the second part utilizes an information fusion technique using the dynamic programming method in which image regions between laser patterns are matched pixel-by-pixel with help of the fusion results obtained in the first part. To determine how the proposed sensor system and fusion algorithms can work in real applications, the sensor system is implemented on a robotic system, and the proposed algorithms are applied. A series of experimental tests is performed for a variety of configurations of robot and environments. The performance of the sensor system is discussed in detail.

© 2008 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. G. N. DeSouza and A. C. Kak, “Vision for mobile robot navigation: a survey,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 237-267 (2002).
  2. O. Faugeras, Three Dimensional Computer Vision: a Geometric Viewpoint (MIT Press, 1993).
  3. N. Ayache, Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception (MIT Press, 1991).
  4. P. Weckesser and R. Dillmann, “Modeling unknown environments with a mobile robot,” Rob. Auton. Syst. 23, 293-300(1998).
    [CrossRef]
  5. H. R. Everett, Sensors for Mobile Robots: Theory and Application (AK Peters, 1995).
  6. J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
    [CrossRef]
  7. C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
    [CrossRef]
  8. P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
    [CrossRef]
  9. V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).
  10. M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
    [CrossRef]
  11. I. S. Chang and R. H. Park, “Segmentation based on fusion of range and intensity images using robust trimmed methods,” Pattern Recogn. 34, 1951-1962 (2001).
    [CrossRef]
  12. K. Umeda, K. Ikushima, and T. Arai, “Fusion of range image and intensity image for 3D shape recognition,” in 1996 IEEE International Conference on Robotics and Automation (IEEE1996), Vol. 1, pp. 680-685.
  13. K. Tate and Z. N. Li, “Depth map construction from range-guided multiresolution stereo matching,” IEEE Trans. Syst. Man Cybern. 24, 134-144 (1994).
  14. M. Y. Kim and H. S. Cho, “An active trinocular vision system for sensing mobile robot navigation environments,” Sens. Actuators A 125, 192-209 (2006).
  15. R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, 1995).
  16. M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)
  17. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vision 47, 7-42 (2002)
    [CrossRef]
  18. S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Comput. Vision 35, 269-293 (1999)
    [CrossRef]
  19. H. J. Zimmermann, Fuzzy Set Theory and Its Applications (Kluwer Adademic, 1991).
  20. http://www.intel.com/technology/computing/opencv/

2006 (1)

M. Y. Kim and H. S. Cho, “An active trinocular vision system for sensing mobile robot navigation environments,” Sens. Actuators A 125, 192-209 (2006).

2003 (1)

M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)

2002 (3)

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vision 47, 7-42 (2002)
[CrossRef]

G. N. DeSouza and A. C. Kak, “Vision for mobile robot navigation: a survey,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 237-267 (2002).

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

2001 (1)

I. S. Chang and R. H. Park, “Segmentation based on fusion of range and intensity images using robust trimmed methods,” Pattern Recogn. 34, 1951-1962 (2001).
[CrossRef]

1999 (2)

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Comput. Vision 35, 269-293 (1999)
[CrossRef]

1998 (2)

P. Weckesser and R. Dillmann, “Modeling unknown environments with a mobile robot,” Rob. Auton. Syst. 23, 293-300(1998).
[CrossRef]

J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
[CrossRef]

1997 (1)

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

1994 (2)

M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
[CrossRef]

K. Tate and Z. N. Li, “Depth map construction from range-guided multiresolution stereo matching,” IEEE Trans. Syst. Man Cybern. 24, 134-144 (1994).

Abdulghafour, M.

M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
[CrossRef]

Abidi, M. A.

M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
[CrossRef]

Arai, T.

K. Umeda, K. Ikushima, and T. Arai, “Fusion of range image and intensity image for 3D shape recognition,” in 1996 IEEE International Conference on Robotics and Automation (IEEE1996), Vol. 1, pp. 680-685.

Ayache, N.

N. Ayache, Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception (MIT Press, 1991).

Batlle, J.

J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
[CrossRef]

Birchfield, S.

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Comput. Vision 35, 269-293 (1999)
[CrossRef]

Brown, M. Z.

M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)

Burschka, D.

M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)

Chandra, T.

M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
[CrossRef]

Chang, I. S.

I. S. Chang and R. H. Park, “Segmentation based on fusion of range and intensity images using robust trimmed methods,” Pattern Recogn. 34, 1951-1962 (2001).
[CrossRef]

Chen, C. S.

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

Chiang, C. C.

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

Cho, H. S.

M. Y. Kim and H. S. Cho, “An active trinocular vision system for sensing mobile robot navigation environments,” Sens. Actuators A 125, 192-209 (2006).

DeSouza, G. N.

G. N. DeSouza and A. C. Kak, “Vision for mobile robot navigation: a survey,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 237-267 (2002).

Dias, P.

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

Dillmann, R.

P. Weckesser and R. Dillmann, “Modeling unknown environments with a mobile robot,” Rob. Auton. Syst. 23, 293-300(1998).
[CrossRef]

Everett, H. R.

H. R. Everett, Sensors for Mobile Robots: Theory and Application (AK Peters, 1995).

Faugeras, O.

O. Faugeras, Three Dimensional Computer Vision: a Geometric Viewpoint (MIT Press, 1993).

Gongalves, J. G. M.

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Hager, G. D.

M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)

Hogg, D.

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Hung, Y. P.

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

Ikushima, K.

K. Umeda, K. Ikushima, and T. Arai, “Fusion of range image and intensity image for 3D shape recognition,” in 1996 IEEE International Conference on Robotics and Automation (IEEE1996), Vol. 1, pp. 680-685.

Jain, R.

R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, 1995).

Kak, A. C.

G. N. DeSouza and A. C. Kak, “Vision for mobile robot navigation: a survey,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 237-267 (2002).

Kasturi, R.

R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, 1995).

Kim, M. Y.

M. Y. Kim and H. S. Cho, “An active trinocular vision system for sensing mobile robot navigation environments,” Sens. Actuators A 125, 192-209 (2006).

Li, Z. N.

K. Tate and Z. N. Li, “Depth map construction from range-guided multiresolution stereo matching,” IEEE Trans. Syst. Man Cybern. 24, 134-144 (1994).

M. Gongalves, J. G.

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

Mouaddib, E.

J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
[CrossRef]

Ng, K.

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Park, R. H.

I. S. Chang and R. H. Park, “Segmentation based on fusion of range and intensity images using robust trimmed methods,” Pattern Recogn. 34, 1951-1962 (2001).
[CrossRef]

Salvi, J.

J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
[CrossRef]

Scharstein, D.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vision 47, 7-42 (2002)
[CrossRef]

Schunck, B. G.

R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, 1995).

Sequeira, V.

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Szeliski, R.

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vision 47, 7-42 (2002)
[CrossRef]

Tate, K.

K. Tate and Z. N. Li, “Depth map construction from range-guided multiresolution stereo matching,” IEEE Trans. Syst. Man Cybern. 24, 134-144 (1994).

Tomasi, C.

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Comput. Vision 35, 269-293 (1999)
[CrossRef]

Umeda, K.

K. Umeda, K. Ikushima, and T. Arai, “Fusion of range image and intensity image for 3D shape recognition,” in 1996 IEEE International Conference on Robotics and Automation (IEEE1996), Vol. 1, pp. 680-685.

Vaz, F.

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

Weckesser, P.

P. Weckesser and R. Dillmann, “Modeling unknown environments with a mobile robot,” Rob. Auton. Syst. 23, 293-300(1998).
[CrossRef]

Wolfart, E.

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Wu, J. L.

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

Zimmermann, H. J.

H. J. Zimmermann, Fuzzy Set Theory and Its Applications (Kluwer Adademic, 1991).

Control Eng. Pract. (1)

M. A. Abidi, M. Abdulghafour, and T. Chandra, “Fusion of visual and range features using fuzzy logic,” Control Eng. Pract. 2, 833-847 (1994).
[CrossRef]

IEEE Trans. Pattern Anal. Mach. Intell. (2)

M. Z. Brown, D. Burschka, and G. D. Hager, “Advances in computational stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 993-1008 (2003)

G. N. DeSouza and A. C. Kak, “Vision for mobile robot navigation: a survey,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 237-267 (2002).

IEEE Trans. Syst. Man Cybern. (1)

K. Tate and Z. N. Li, “Depth map construction from range-guided multiresolution stereo matching,” IEEE Trans. Syst. Man Cybern. 24, 134-144 (1994).

Image Vision Comput. (1)

C. S. Chen, Y. P. Hung, C. C. Chiang, and J. L. Wu, “Range data acquisition using color structured lighting and stereo vision,” Image Vision Comput. 15,445-456 (1997).
[CrossRef]

Int. J. Comput. Vision (2)

D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” Int. J. Comput. Vision 47, 7-42 (2002)
[CrossRef]

S. Birchfield and C. Tomasi, “Depth discontinuities by pixel-to-pixel stereo,” Int. J. Comput. Vision 35, 269-293 (1999)
[CrossRef]

ISPRS J. Photogramm. Remote Sens. (1)

V. Sequeira, K. Ng, E. Wolfart, J. G. M. Gongalves, and D. Hogg, “Automated reconstruction of 3D models from real environment,” ISPRS J. Photogramm. Remote Sens. 54, 1-22 (1999).

Pattern Recog. (1)

J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recog. 31, 963-982 (1998).
[CrossRef]

Pattern Recogn. (1)

I. S. Chang and R. H. Park, “Segmentation based on fusion of range and intensity images using robust trimmed methods,” Pattern Recogn. 34, 1951-1962 (2001).
[CrossRef]

Rob. Auton. Syst. (1)

P. Weckesser and R. Dillmann, “Modeling unknown environments with a mobile robot,” Rob. Auton. Syst. 23, 293-300(1998).
[CrossRef]

Rob. Autono. Syst. (1)

P. Dias, V. Sequeira, J. G. M. Gongalves, and F. Vaz, “Automatic registration of laser reflectance and colour intensity images for 3D reconstruction,” Rob. Autono. Syst. 39, 157-168 (2002).
[CrossRef]

Sens. Actuators A (1)

M. Y. Kim and H. S. Cho, “An active trinocular vision system for sensing mobile robot navigation environments,” Sens. Actuators A 125, 192-209 (2006).

Other (7)

R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, 1995).

K. Umeda, K. Ikushima, and T. Arai, “Fusion of range image and intensity image for 3D shape recognition,” in 1996 IEEE International Conference on Robotics and Automation (IEEE1996), Vol. 1, pp. 680-685.

H. J. Zimmermann, Fuzzy Set Theory and Its Applications (Kluwer Adademic, 1991).

http://www.intel.com/technology/computing/opencv/

H. R. Everett, Sensors for Mobile Robots: Theory and Application (AK Peters, 1995).

O. Faugeras, Three Dimensional Computer Vision: a Geometric Viewpoint (MIT Press, 1993).

N. Ayache, Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception (MIT Press, 1991).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (13)

Fig. 1
Fig. 1

(a)  Autonomous mobile robot LCARIII and (b)  its sensor head for 3D environment perception.

Fig. 2
Fig. 2

Multistripe laser pattern projected onto a scene.

Fig. 3
Fig. 3

Line features acquired for three active trinocular vision images and their correspondence matching.

Fig. 4
Fig. 4

Algorithmic flow chart for 3D information extraction of laser line features in active trinocular vision.

Fig. 5
Fig. 5

Accumulator array used for correspondence matching between line features with l × m × n size.

Fig. 6
Fig. 6

Conceptual diagram of an approach fusing information from two sensor parts: (a) solving ambiguous correspondences between laser line features using stereo intensity images and (b) acquiring 3D dense depth map from stereo intensity images with the assistance of range data from the active trinocular sensor.

Fig. 7
Fig. 7

Proposed method of assigning penalty values in dynamic programming in the case of fusing laser information with stereo information

Fig. 8
Fig. 8

Experimental result for test environment 1: (a) environment composed of a ball, a cup, and a textured plane; (b) left image from stereo vision; (c) left image from active trinocular vision; (d) range image from general dynamic programming; (e) range image from the proposed fusion algorithm.

Fig. 9
Fig. 9

Experimental result for test environment 2: (a) environment composed of an untextured box and a textured plane, (b) left image from stereo vision, (c) left image from active trinocular vision, (d) range image from general dynamic programming, (e) range image from the proposed fusion algorithm.

Fig. 10
Fig. 10

Experimental result for test environment 3: (a) environment composed of a ball, cups, and an untextured plane; (b) left image from stereo vision; (c) left image from active trinocular vision; (d) range image from general dynamic programming; (e) range image from the proposed fusion algorithm.

Fig. 11
Fig. 11

Experimental result for test environment 4: (a) environment composed of two untextured boxes and an untextured plane, (b) left image from stereo vision, (c) left image from active trinocular vision, (d) range image from general dynamic programming, (e) range image from the proposed fusion algorithm.

Fig. 12
Fig. 12

Error analysis with variation of the penalty value in the proposed fusion algorithm: (a) average of disparity error and (b) standard deviation of disparity error.

Fig. 13
Fig. 13

Error analysis for all experimental cases with variations of the occlusion penalty and the match reward in dynamic programming (contour plot): (a) average of disparity error and (b) standard deviation of disparity error.

Tables (1)

Tables Icon

Table 1 Disparity Error Statistics between Ground Truth and Experimental Result

Equations (7)

Equations on this page are rendered with MathJax. Learn more.

A ( i , j , k ) = A ( i , j , k ) + 1.
A ( i , j , k ) = A ( i , j , k ) + 1 N .
C ( i , j ) = { [ A ( i , j , k ) , k ] | max [ A ( i , j , k ) ] f o r     k = 1 , , l } ,
J ( m ) = N oc λ oc N m λ r + i = 1 N m d ( x i , y i )
Δ B ( i , j , k ) = ( u , v ) W [ I 1 ( u , v ) I ¯ 1 ] [ I 2 ( x + u , y + v ) I ¯ 2 ] ( u , v ) W [ I 1 ( u , v ) I ¯ 1 ] 2 · ( u , v ) W [ I 2 ( x + u , y + v ) I ¯ 2 ] 2
i f C ( i , j ) > T sub   and if   B ( i , j , k ) > T sub 2 , then line pair   ( i , j )   is confirmed.
E = u , v | disp real ( u , v ) disp measured ( u , v ) | ,

Metrics