Abstract

With increasing capability and complexity of surgical interventions, intra-operative visualization is becoming an important part of a surgical environment. This paper reviews some of our recent progress in the intelligent use of pre- and intra-operative data for enhanced surgical navigation and motion compensated visualization. High fidelity augmented reality (AR) with enhanced 3D depth perception is proposed to provide effective surgical guidance. To cater for large scale tissue deformation, real-time depth recovery based on stereo disparity and eye gaze tracking is introduced. This allows the development of motion compensated visualization for improved visual perception and for facilitating motion adaptive AR displays. The discussion of the paper is focused on how to ensure perceptual fidelity of AR and the need for real-time tissue deformation recovery and modeling, as well as the importance of incorporating human perceptual factors in surgical displays.

© 2008 IEEE

PDF Article

References

  • View by:
  • |
  • |

  1. R. H. Taylor, D. Stoianovici, "Medical robotics in computer-integrated surgery," IEEE Trans. Robot. Autom. 19, 765-781 (2003).
  2. R. J. Damiano, "Robotics in cardiac surgery: The emperor's new clothes," J., Thoracic & Cardiovasc. Surgery 134, 559-561 (2007).
  3. H. Delingette, X. Pennec, L. Soler, J. Marescaux, N. Ayache, "Computational models for image-guided robot-assisted and simulated medical interventions," Proc. IEEE 94, 1678-1688 (2006).
  4. T. M. Peters, "Image-guidance for surgical procedures," Phys. in Med. Biol. 51, R505-R540 (2006).
  5. S. Vogt, A. Khamene, F. Sauer, "Reality augmentation for medical procedures: System architecture, single camera marker tracking, and system evaluation," Int. J. Comput. Vision 70, 179-190 (2006).
  6. T. Sielhorst, C. Bichlmeier, S. M. Heining, N. Navab, "Depth perception—A major issue in medical AR: Evaluation study by twenty surgeons," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2006) pp. 364-372.
  7. M. Lerotic, A. J. Chung, G. Mylonas, G. Z. Yang, "pq -space based non-photorealistic rendering for augmented reality," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 102-109.
  8. T. Okatani, K. Deguchi, "Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection centre," Computer Vision and Image Understanding 66, 119-131 (1997).
  9. F. Deligianni, A. J. Chung, G. Z. Yang, "Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration," Computer Aided Surgery 9, 215-226 (2004).
  10. B. K. P. Horn, M. J. Brooks, Shape from Shading (The MIT Press, 1989).
  11. H. U. Rashid, P. Burger, "Differential algorithm for the determination of shape from shading using a point light source," Image and Vision Computing 10, 119-127 (1992).
  12. R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge Univ. Press, 2000).
  13. J. Heikkila, O. Silven, "A four-step camera calibration procedure with implicit image correction," IEEE Comput. Soc. Conf. on Computer Vision and Pattern Recogn. (1997) pp. 1106-1112.
  14. Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330-1334 (2000).
  15. D. Stoyanov, A. Darzi, G. Z. Yang, "Laparoscope self-calibration for robotic assisted minimally invasive surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2005) pp. 114-121.
  16. S. Baker, T. Sim, T. Kanade, "When is the shape of a scene unique given its light-field: A fundamental theorem of 3D vision?," IEEE Trans. Pattern Anal. Mach. Intell. 25, 100-109 (2003).
  17. D. Stoyanov, A. Darzi, G. Z. Yang, "Dense 3d depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2004) pp. 41-48.
  18. W. W. Lau, N. A. Ramey, J. Corso, N. V. Thakor, G. D. Hager, "Stereo-based endoscopic tracking of cardiac surface deformation," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2004) pp. 494-501.
  19. D. Stoyanov, A. Darzi, G. Z. Yang, "A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery," Computer Aided Surgery 10, 199-208 (2005).
  20. K. Mori, D. Deguchi, J. Sugiyama, Y. Suenagaa, J. Toriwakia, C. R. M. Jr, H. Takabatake, H. Natori, "Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images," Medical Image Analysis 6, 321-336 (2002).
  21. D. Burschka, M. Li, R. Taylor, G. D. Hager, "Scale-invariant registration of monocular endoscopic images to ct-scans for sinus surgery," International Conference on Medical Image Computing and Computer Assisted Intervention (2004) pp. 413-421.
  22. M. Gröger, T. Ortmaier, W. Sepp, G. Hirzinger, "Tracking local motion on the beating heart," SPIE Med. Imag. Conf. (2002) pp. 233-241.
  23. D. Stoyanov, G. P. Mylonas, F. Deligianni, A. Darzi, G. Z. Yang, "Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2005) pp. 139-146.
  24. B. Lucas, T. Kanade, "An iterative image registration technique with an application to stereo vision," Int. Joint Conf. Artif. Intell. (1981) pp. 674-679.
  25. M. Hu, G. P. Penney, P. J. Edwards, M. Figl, D. J. Hawkes, "3D reconstruction of internal organ surfaces for minimal invasive surgery," Int. Conf. on Med. Image Computing and Computer Assisted Intervention (2007) pp. 68-77.
  26. M. Gröger, T. Ortmaier, W. Sepp, G. Hirzinger, "Reconstruction of image structure in presence of specular reflections," DAGM-Symp. on Pattern Recogn. (2001) pp. 53-60.
  27. D. Stoyanov, G. Z. Yang, "Removing specular reflection components for robotic assisted laparoscopic surgery," IEEE Int. Conf. on Image Process. (2005) pp. 350-359.
  28. D. Stoyanov, G. Z. Yang, "Stabilization of image motion for robotic assisted beating heart surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 417-424.
  29. G. P. Mylonas, D. Stoyanov, A. Darzi, G. Z. Yang, "Assessment of perceptual quality for gaze-contingent motion stabilization in robotic assisted minimally invasive surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 660-667.
  30. H. Togami, "Effects on visual search performance of individual differences in fixation time and number of fixations," Ergonomics 27, 789-799 (1984).
  31. A. J. Chung, F. Deligianni, P. Shah, A. Wells, M. Elhelw, G. Z. Yang, "Patient specific bronchoscopy visualisation through BRDF estimation and disocclusion correction," IEEE Trans. Med. Imag. 25, 503-513 (2006).
  32. M. ElHelw, M. Nicolaou, A. Chung, G. Z. Yang, M. S. Atkins, "A gaze-based framework for investigating the perception of visual realism in simulated scenes," ACM Trans. Appl. Perception 5, (2008).
  33. X. P. Hu, L. Dempere-Marco, G. Z. Yang, "Hot spot detection based on feature space representation of visual search," IEEE Trans. Med. Imag. 22, 1152-1162 (2003).
  34. L. Dempere-Marco, X. P. Hu, S. L. S. Macdonald, S. M. Ellis, D. M. Hansell, G. Z. Yang, "The use of visual search for knowledge gathering in image decision support," IEEE Trans. Med. Imag. 21, 741-754 (2002).
  35. G. Z. Yang, L. Dempre-Marco, X. P. Hu, A. Rowe, "Visual search: Psychophysical models and practical applications," Image and Vision Computing 20, 291-305 (2002).
  36. A. James, D. Viera, B. P. L. Lo, A. Darzi, G. Z. Yang, "Eye-gaze driven surgical workflow segmentation," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 110-117.
  37. G. P. Mylonas, A. Darzi, G. Z. Yang, "Gaze-contingent control for minimally invasive robotic surgery," Computer Aided Surgery 11, 256-266 (2006).

2008 (1)

M. ElHelw, M. Nicolaou, A. Chung, G. Z. Yang, M. S. Atkins, "A gaze-based framework for investigating the perception of visual realism in simulated scenes," ACM Trans. Appl. Perception 5, (2008).

2007 (1)

R. J. Damiano, "Robotics in cardiac surgery: The emperor's new clothes," J., Thoracic & Cardiovasc. Surgery 134, 559-561 (2007).

2006 (5)

H. Delingette, X. Pennec, L. Soler, J. Marescaux, N. Ayache, "Computational models for image-guided robot-assisted and simulated medical interventions," Proc. IEEE 94, 1678-1688 (2006).

T. M. Peters, "Image-guidance for surgical procedures," Phys. in Med. Biol. 51, R505-R540 (2006).

S. Vogt, A. Khamene, F. Sauer, "Reality augmentation for medical procedures: System architecture, single camera marker tracking, and system evaluation," Int. J. Comput. Vision 70, 179-190 (2006).

A. J. Chung, F. Deligianni, P. Shah, A. Wells, M. Elhelw, G. Z. Yang, "Patient specific bronchoscopy visualisation through BRDF estimation and disocclusion correction," IEEE Trans. Med. Imag. 25, 503-513 (2006).

G. P. Mylonas, A. Darzi, G. Z. Yang, "Gaze-contingent control for minimally invasive robotic surgery," Computer Aided Surgery 11, 256-266 (2006).

2005 (1)

D. Stoyanov, A. Darzi, G. Z. Yang, "A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery," Computer Aided Surgery 10, 199-208 (2005).

2004 (1)

F. Deligianni, A. J. Chung, G. Z. Yang, "Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration," Computer Aided Surgery 9, 215-226 (2004).

2003 (3)

R. H. Taylor, D. Stoianovici, "Medical robotics in computer-integrated surgery," IEEE Trans. Robot. Autom. 19, 765-781 (2003).

S. Baker, T. Sim, T. Kanade, "When is the shape of a scene unique given its light-field: A fundamental theorem of 3D vision?," IEEE Trans. Pattern Anal. Mach. Intell. 25, 100-109 (2003).

X. P. Hu, L. Dempere-Marco, G. Z. Yang, "Hot spot detection based on feature space representation of visual search," IEEE Trans. Med. Imag. 22, 1152-1162 (2003).

2002 (3)

L. Dempere-Marco, X. P. Hu, S. L. S. Macdonald, S. M. Ellis, D. M. Hansell, G. Z. Yang, "The use of visual search for knowledge gathering in image decision support," IEEE Trans. Med. Imag. 21, 741-754 (2002).

G. Z. Yang, L. Dempre-Marco, X. P. Hu, A. Rowe, "Visual search: Psychophysical models and practical applications," Image and Vision Computing 20, 291-305 (2002).

K. Mori, D. Deguchi, J. Sugiyama, Y. Suenagaa, J. Toriwakia, C. R. M. Jr, H. Takabatake, H. Natori, "Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images," Medical Image Analysis 6, 321-336 (2002).

2000 (1)

Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330-1334 (2000).

1997 (1)

T. Okatani, K. Deguchi, "Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection centre," Computer Vision and Image Understanding 66, 119-131 (1997).

1992 (1)

H. U. Rashid, P. Burger, "Differential algorithm for the determination of shape from shading using a point light source," Image and Vision Computing 10, 119-127 (1992).

1984 (1)

H. Togami, "Effects on visual search performance of individual differences in fixation time and number of fixations," Ergonomics 27, 789-799 (1984).

ACM Trans. Appl. Perception (1)

M. ElHelw, M. Nicolaou, A. Chung, G. Z. Yang, M. S. Atkins, "A gaze-based framework for investigating the perception of visual realism in simulated scenes," ACM Trans. Appl. Perception 5, (2008).

Computer Aided Surgery (3)

G. P. Mylonas, A. Darzi, G. Z. Yang, "Gaze-contingent control for minimally invasive robotic surgery," Computer Aided Surgery 11, 256-266 (2006).

F. Deligianni, A. J. Chung, G. Z. Yang, "Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration," Computer Aided Surgery 9, 215-226 (2004).

D. Stoyanov, A. Darzi, G. Z. Yang, "A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery," Computer Aided Surgery 10, 199-208 (2005).

Computer Vision and Image Understanding (1)

T. Okatani, K. Deguchi, "Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection centre," Computer Vision and Image Understanding 66, 119-131 (1997).

Ergonomics (1)

H. Togami, "Effects on visual search performance of individual differences in fixation time and number of fixations," Ergonomics 27, 789-799 (1984).

IEEE Trans. Med. Imag. (3)

A. J. Chung, F. Deligianni, P. Shah, A. Wells, M. Elhelw, G. Z. Yang, "Patient specific bronchoscopy visualisation through BRDF estimation and disocclusion correction," IEEE Trans. Med. Imag. 25, 503-513 (2006).

X. P. Hu, L. Dempere-Marco, G. Z. Yang, "Hot spot detection based on feature space representation of visual search," IEEE Trans. Med. Imag. 22, 1152-1162 (2003).

L. Dempere-Marco, X. P. Hu, S. L. S. Macdonald, S. M. Ellis, D. M. Hansell, G. Z. Yang, "The use of visual search for knowledge gathering in image decision support," IEEE Trans. Med. Imag. 21, 741-754 (2002).

IEEE Trans. Pattern Anal. Mach. Intell. (2)

S. Baker, T. Sim, T. Kanade, "When is the shape of a scene unique given its light-field: A fundamental theorem of 3D vision?," IEEE Trans. Pattern Anal. Mach. Intell. 25, 100-109 (2003).

Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330-1334 (2000).

IEEE Trans. Robot. Autom. (1)

R. H. Taylor, D. Stoianovici, "Medical robotics in computer-integrated surgery," IEEE Trans. Robot. Autom. 19, 765-781 (2003).

Image and Vision Computing (2)

H. U. Rashid, P. Burger, "Differential algorithm for the determination of shape from shading using a point light source," Image and Vision Computing 10, 119-127 (1992).

G. Z. Yang, L. Dempre-Marco, X. P. Hu, A. Rowe, "Visual search: Psychophysical models and practical applications," Image and Vision Computing 20, 291-305 (2002).

Int. J. Comput. Vision (1)

S. Vogt, A. Khamene, F. Sauer, "Reality augmentation for medical procedures: System architecture, single camera marker tracking, and system evaluation," Int. J. Comput. Vision 70, 179-190 (2006).

J., Thoracic & Cardiovasc. Surgery (1)

R. J. Damiano, "Robotics in cardiac surgery: The emperor's new clothes," J., Thoracic & Cardiovasc. Surgery 134, 559-561 (2007).

Medical Image Analysis (1)

K. Mori, D. Deguchi, J. Sugiyama, Y. Suenagaa, J. Toriwakia, C. R. M. Jr, H. Takabatake, H. Natori, "Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images," Medical Image Analysis 6, 321-336 (2002).

Phys. in Med. Biol. (1)

T. M. Peters, "Image-guidance for surgical procedures," Phys. in Med. Biol. 51, R505-R540 (2006).

Proc. IEEE (1)

H. Delingette, X. Pennec, L. Soler, J. Marescaux, N. Ayache, "Computational models for image-guided robot-assisted and simulated medical interventions," Proc. IEEE 94, 1678-1688 (2006).

Other (18)

T. Sielhorst, C. Bichlmeier, S. M. Heining, N. Navab, "Depth perception—A major issue in medical AR: Evaluation study by twenty surgeons," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2006) pp. 364-372.

M. Lerotic, A. J. Chung, G. Mylonas, G. Z. Yang, "pq -space based non-photorealistic rendering for augmented reality," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 102-109.

B. K. P. Horn, M. J. Brooks, Shape from Shading (The MIT Press, 1989).

D. Burschka, M. Li, R. Taylor, G. D. Hager, "Scale-invariant registration of monocular endoscopic images to ct-scans for sinus surgery," International Conference on Medical Image Computing and Computer Assisted Intervention (2004) pp. 413-421.

M. Gröger, T. Ortmaier, W. Sepp, G. Hirzinger, "Tracking local motion on the beating heart," SPIE Med. Imag. Conf. (2002) pp. 233-241.

D. Stoyanov, G. P. Mylonas, F. Deligianni, A. Darzi, G. Z. Yang, "Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2005) pp. 139-146.

B. Lucas, T. Kanade, "An iterative image registration technique with an application to stereo vision," Int. Joint Conf. Artif. Intell. (1981) pp. 674-679.

M. Hu, G. P. Penney, P. J. Edwards, M. Figl, D. J. Hawkes, "3D reconstruction of internal organ surfaces for minimal invasive surgery," Int. Conf. on Med. Image Computing and Computer Assisted Intervention (2007) pp. 68-77.

M. Gröger, T. Ortmaier, W. Sepp, G. Hirzinger, "Reconstruction of image structure in presence of specular reflections," DAGM-Symp. on Pattern Recogn. (2001) pp. 53-60.

D. Stoyanov, G. Z. Yang, "Removing specular reflection components for robotic assisted laparoscopic surgery," IEEE Int. Conf. on Image Process. (2005) pp. 350-359.

D. Stoyanov, G. Z. Yang, "Stabilization of image motion for robotic assisted beating heart surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 417-424.

G. P. Mylonas, D. Stoyanov, A. Darzi, G. Z. Yang, "Assessment of perceptual quality for gaze-contingent motion stabilization in robotic assisted minimally invasive surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 660-667.

A. James, D. Viera, B. P. L. Lo, A. Darzi, G. Z. Yang, "Eye-gaze driven surgical workflow segmentation," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2007) pp. 110-117.

R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge Univ. Press, 2000).

J. Heikkila, O. Silven, "A four-step camera calibration procedure with implicit image correction," IEEE Comput. Soc. Conf. on Computer Vision and Pattern Recogn. (1997) pp. 1106-1112.

D. Stoyanov, A. Darzi, G. Z. Yang, "Laparoscope self-calibration for robotic assisted minimally invasive surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2005) pp. 114-121.

D. Stoyanov, A. Darzi, G. Z. Yang, "Dense 3d depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2004) pp. 41-48.

W. W. Lau, N. A. Ramey, J. Corso, N. V. Thakor, G. D. Hager, "Stereo-based endoscopic tracking of cardiac surface deformation," Int. Conf. on Medical Image Computing and Computer Assisted Intervention (2004) pp. 494-501.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.