Open Access
Add to BibSonomyAbstract
Intra-operative optical coherence tomography (OCT) requires a display technology which allows surgeons to visualize OCT data without disrupting surgery. Previous research and commercial intrasurgical OCT systems have integrated heads-up display (HUD) systems into surgical microscopes to provide monoscopic viewing of OCT data through one microscope ocular. To take full advantage of our previously reported real-time volumetric microscope-integrated OCT (4D MIOCT) system, we describe a stereoscopic HUD which projects a stereo pair of OCT volume renderings into both oculars simultaneously. The stereoscopic HUD uses a novel optical design employing spatial multiplexing to project dual OCT volume renderings utilizing a single micro-display. The optical performance of the surgical microscope with the HUD was quantitatively characterized and the addition of the HUD was found not to substantially effect the resolution, field of view, or pincushion distortion of the operating microscope. In a pilot depth perception subject study, five ophthalmic surgeons completed a pre-set dexterity task with 50.0% (SD = 37.3%) higher success rate and in 35.0% (SD = 24.8%) less time on average with stereoscopic OCT vision compared to monoscopic OCT vision. Preliminary experience using the HUD in 40 vitreo-retinal human surgeries by five ophthalmic surgeons is reported, in which all surgeons reported that the HUD did not alter their normal view of surgery and that live surgical maneuvers were readily visible in displayed stereoscopic OCT volumes.
© 2016 Optical Society of America
1. Introduction
Optical coherence tomography (OCT) is an interferometric imaging modality that generates volumetric, micron-resolution images of human ocular tissue [1]. As a result, OCT has revolutionized retinal diagnostics [2–5] and is quickly becoming prominent in imaging and biometry of the anterior segment [6–9]. Since then, OCT has been extended to a wide variety of applications including in vivo imaging with motion compensation [10, 11]. The development of handheld OCT (HHOCT) systems allowed for imaging of supine, anesthetized patients and therefore intraoperative ophthalmic imaging for visualizing microarchitectural surgical alterations [7, 12–17]. However, HHOCT probes required displacement of the surgical microscope away from the patient, which could only occur during pauses in surgery for imaging. To achieve concurrent OCT and surgical microscope imaging, we and others have developed prototype spectral-domain microscope-integrated OCT (SD-MIOCT) systems [18–24]. These prototypes seamlessly integrated the OCT optical path into the surgical microscope and allow for imaging of live surgery. Due to limited A-scan acquisition rates in previous MIOCT systems, real-time imaging was restricted to B-scans, therefore visualization of complete surgical maneuvers and dynamic surgical tool/tissue interactions remained challenging. For comprehensive imaging of dynamic ophthalmic surgery, we have recently developed a swept-source volumetric imaging through time (4D) microscope-integrated OCT (MIOCT) system that acquires, processes, and renders volumetric surgical OCT data in real time at up to 10 volumes/second [25–27].
Real-time intraoperative OCT feedback requires that the surgeon must be able to view the OCT images without looking away from the surgical binoculars and disrupting surgery. First generation MIOCT systems displayed acquired images on a monitor that was not visible to the surgeon during the operation [28–30]. To the best of our knowledge, there are two current commercial MIOCT systems available with built-in heads-up displays (HUD): iOCT (Haag Streit Surgical; Wedel, Germany) [31] and the RESCAN 700 (Carl Zeiss Meditec, Jena, Germany) [32]. A prototype MIOCT system including a built-in HUD was also reported by a group at the Cleveland Clinic [33, 34]. These three MIOCT systems provide real time intraoperative OCT feedback to the surgeon, all of which either project B-scans into a single microscope ocular (RESCAN 700 and the Cleveland Clinic prototype), or project the same B-scan image into both microscope oculars (iOCT). While this approach is sufficient for projecting 2D images, monoscopic vision limits the depth perception and stereopsis of 3D images, including OCT volumes. Recently, several stereoscopic viewing devices including head mounted displays (HMD) [35] and 3D monitors such as the TrueVision commercial system (TrueVision 3D Surgical; Santa Barbara, CA) [36, 37] have been reported for intraoperative applications. However, compared to surgical microscopes, these external stereoscopic viewing devices have not yet been widely adopted for ophthalmic surgery.
Artificial stereoscopy was invented by Sir Charles Wheatstone in 1838 [38]. The fundamental principle is to display an object from one angle to the left eye and to display the same object from a slightly different angle to the right eye. The viewer’s brain fuses the two images to create the stereoscopic effect. In the next 178 years, many three dimensional display technologies for HUD have been developed. The standard approach is to use a pair of micro displays to generate slightly different images to both eyes [39, 40]. However, each micro display requires a separate optical path and thus results in bulky optics and high costs. Other approaches have been reported to display stereoscopic images using only one display combined with polarizing filters [41] or active shutter glasses [42]. However, these methods require viewers to wear special glasses and are therefore incompatible with current microsurgical disciplines which employ stereo microscopes. Another method which has been used is to place a prism between two mirrors in front of observer’s eyes in order to direct half of a single display screen to each respective eye [43]. While the principle of spatially multiplexing sections of a single display into both eyes is very useful, the optical implementations in prior work was too bulky for integration into modern surgical microscopes.
This work presents a novel and compact design for displaying stereoscopic images utilizing a single organic light-emitting diode (OLED) micro display for MIOCT. We integrate our custom stereoscopic HUD with our previously reported 4D MIOCT system [44] for stereoscopic visualization of volumetric OCT data via the microscope binoculars. Using this system, we quantitatively characterize the utility of real-time stereoscopic volumetric OCT feedback using the HUD in guiding model eye surgical maneuvers. Finally, we apply the stereoscopic HUD in conjunction with a surgeon-controlled foot pedal to visualize and control the rendering perspective of stereoscopic OCT volumes in 40 live human ophthalmic surgeries.
2. Methods
2.1 Microscope integrated stereoscopic HUD design
Our stereoscopic HUD (Fig. 1(A)) was designed to be mounted onto a Leica M844 ophthalmic surgical microscope (Leica; Wetzlar, Germany; see Fig. 2(B)). The optics of the surgical microscope binocular head (Fig. 1(A), green box) and HUD unit (Fig. 1(A), red box) were modeled in a ray tracing software (Zemax, LLC; Kirkland, WA). Lens A and the lenses inside Leica Binocular head were modeled as paraxial lenses since their exact prescription is proprietary.
Fig. 1 (A) Optical design of novel microscope-integrated HUD system mounted on a surgical microscope. (B) Illustration of images seen through oculars. The black circles represent the microscope ocular FOVs. The red squares represent the micro-display image relayed to viewers’ eyes. The green and yellow regions represent the images directed to the observer’s right and left eyes, respectively.
Fig. 2 Mechanical design of novel microscope-integrated HUD system. (A) Photograph of the HUD and MIOCT system in use during surgery. (B) Photograph of HUD (white box in red border) mounted on a surgical microscope attached with a custom SSOCT sample arm (marked in yellow box). (C) Mechanical design of the HUD. (D) Photograph of 3D printed HUD prototype. Scale bars are 10mm.
Intrasurgical OCT images displayed on a high contrast OLED micro display (SVGA050, OLIGHTEK Co.,Ltd.; Yunna, China) were first relayed to an intermediate plane (FOV = 21 mm) by a telescope comprised of a 50 mm focal length lens (Edmund Optics Inc.; Barrington, NJ) and a 190 mm focal length lens inside the Leica binocular head. The OLED display had a resolution of 800 × 600, a viewing area of 10.13 mm × 7.61 mm, pixel pitch of 12.6 µm, contrast ratio of 5200:1, and brightness 171 Cd/m2. The use of an OLED display allowed completely transparent black pixels (contrast = 5200:1) to ensure that operating microscope view through the HUD was not compromised. The focal length of the Leica lens was empirically determined by illuminating it with collimated 633 nm light from a Helium-neon laser and calculating the focal distance using a beam profiler. The intermediate image was relayed to the observer’s eye with a 10x microscope eyepiece. The focus of lens A was adjusted carefully so that the virtual image of the OLED was parfocal with the virtual image of the surgical field before the observer’s eye. Two beam splitters (BS025, Thorlabs, Inc.; Newton, NJ; Reflection: Transmission = 10:90) coupled the HUD and surgical microscope optical axes. Thus, 10% of the light from the HUD’s OLED display was reflected by the first beam splitter cube, directing the light into the left ocular. The remaining (90%) light transmitted by the first beam splitter was reflected (10%) by a second splitter, directing the light into the right ocular. The beam splitters were tilted by 1.5° relative to the rest of the optical elements. The angles of both tilted beam splitters were calibrated carefully so that the left half of the display was directed to the right ocular while the right half of the display was directed to the left ocular. The optimum tilt angle was experimentally determined with a protractor as 1.5 ± 0.25°, which agreed with the value predicted by the Zemax model. The tilted beam splitters also resulted in a slight lateral displacement of collimated light in the infinity space prior to the 190 mm FL lens due to refraction at the glass/air interfaces. The lateral shift due to a 1.5° tilt was calculated to be 0.23 mm and did not cause any observable vignetting at the biocular stops.
A custom mechanical enclosure for the HUD was designed in Solidworks (Fig. 2(C)). The Leica binocular head was mounted directly to the HUD. A male dovetail connector mounted to the bottom of the HUD allowed for quick attachment of the HUD on the surgical microscope (Fig. 2(A) and 2(B)). The HUD interfaced with the SS-MIOCT control computer via USB and HDMI, and enabled real-time acquisition, processing, and stereoscopic HUD display of volumetric images in real time (up to 10 volumes/sec). Custom, GPU-based OCT software enabled simultaneous rendering of two separate views of each volume onto separate areas of the micro display [45]. The two renderings were rotated by 9° relative to each other to create the stereoscopic effect. The volume intended for the left eye (Fig. 1(B), yellow square) was displayed in the right half of the micro display, which was only visible to the left eye. Similarly, the OCT volume intended for the right eye (Fig. 1(B), green square) was displayed in the left half of the micro display, which was only visible to the right eye. This rotational offset between the two rendered volumes and the positions of the two images was calibrated experimentally so that two images could be fused by the observers comfortably [46]. Furthermore, the surgeon controlled the perspective of the stereoscopic OCT volumes with a rotatable commercial foot joystick (PC042, Mayflash, Guangdong, China) in real time during surgery.
2.2 Optical characterization of stereoscopic HUD
To characterize the HUD performance, a camera with a 1280 by 1024 resolution and a pixel pitch of 5.3 µm (Point Grey Research, Inc.; Richmond, Canada) and an imaging lens (Edmund Optics Inc.; Barrington, NJ) was attached to the Leica M844 surgical microscope’s eyepiece with a custom 3D printed mount to capture images through the microscope oculars. The effect of the HUD on the operating microscope view, resolution, field of view (FOV) and distortion both with and without the HUD was measured with a USAF resolution test target, a caliper, and a grid chart, respectively in the surgical field at 10x total magnification of the microscope. This magnification was chosen due to the prevalence of this setting for ophthalmic surgery. The distortion of the HUD’s optical system was also characterized with the same grid chart which was displayed on the HUD’s OLED micro display and captured by a camera through the microscope oculars. The SMIA TV [25] distortion in the captured grid chart images was then analyzed with a commercial image quality software (Imatest LCC; Boulder, CO).
2.3 Depth perception pilot study
To investigate the ability of the stereoscopic HUD to enhance depth perception in real time 3D OCT guidance of model eye surgical maneuvers, five ophthalmic surgeons from the Duke Eye Center were recruited to participate in a depth perception pilot study. Five plastic beads (diameter: ~0.5 mm, Fig. 3(B)) placed on a cardboard base were imaged with the MIOCT/HUD system. A 0.5 mm diameter pencil lead divided the cardboard into left and right sides. The microscope view was blocked and the subjects were asked to use 25 gauge surgical forceps (Pinnacle 360°, Synergetics; O'Fallon, MO, Fig. 3(A)) to transfer beads from the left side to the right side of the divider using only the real-time rendered OCT volumes (Fig. 3(C)), using either monoscopic or stereoscopic vision, with all other surgical illumination turned off. The procedures were recorded with cameras attached to the assistant scope of the microscope. Before the study, the subjects were randomized to either monoscopic or stereoscopic OCT and were allowed 30 minutes to practice the task. When subjects were ready for testing, they were asked to touch the top of the dividing lead with the forceps, which marked the beginning of the study and the start point of the first transfer. The subjects then reached out to the first bead (closest to the subject) to grab and transfer it over the center lead to the right side. The end point for a successful transfer was defined as when the bead touched the right side of the cardboard. An unsuccessful transfer was defined as any beads exiting the field of view, or the subject giving up after five minutes. After the first transfer, the subjects were instructed to move the forceps from right to left to reach the second bead. The start point for the next transfer was defined as when the forceps passed the center lead from right to left side while the end point was the same as before. Five beads in total were transferred from left to right side of the cardboard during each trial. The time to transfer each bead and the success rate (number of successfully transferred beads/total number of beads) were assessed by a masked independent grader using the recorded videos.
Fig. 3 Depth perception test. (A) Photograph of 25 gauge pinnacle forceps. (B) Photograph of the sample used in the study. (C) Sample image of OCT volume recorded through one of the microscope oculars during the study. Scale bar represents 1mm.
2.4 HUD evaluation during live 4DMIOCT porcine eye and human surgical imaging
To prepare surgeons to use the Duke 4D MIOCT/HUD system in human vitreoretinal surgeries, the system was tested in 10 wet lab-based cadaveric porcine eye studies before translation into the human operating room. Five ophthalmic surgeons from the Duke Eye Center participated these studies. Simulated retinal surgical maneuvers including brushing with a surgical scraper (Synergetics; O'Fallon, MO) and membrane peeling with surgical forceps (Synergetics; O'Fallon, MO), were performed in cadaveric porcine eyes. During these simulated retinal surgeries, standard 3-port vitrectomy was utilized: the first port was used to insert the infusion line, the second port was used to insert a fiber-optic endoilluminator, and the third port was used to insert the surgical instruments. Retinal structures were visualized through a disposable flat surgical contact lens (Alcon; Ft. Worth, TX). Stereoscopic, 4D MIOCT volumetric renders of porcine eye surgery were acquired and displayed via the HUD in real time. According to each surgeon’s preference, the locations of displayed OCT images were adjusted inside the oculars to allow surgeons an unobstructed view of the surgical field and of the real time stereoscopic OCT data simultaneously. The B scan location was depicted by a white square border in the OCT volume and could be changed arbitrarily by a trained technician. A foot joystick was utilized so that surgeons could control the perspective of OCT volumes to inspect data sets from different angles. In order to record the surgeons’ view inside the oculars during the simulated surgeries, a pair of cameras (Point Grey Research, Inc., Richmond, Canada) were attached to the oculars of an operator’s assistant’s scope with two custom mounts. During all wet lab-based simulated surgeries, each surgeon was asked for verbal feedback on different display formats of the OCT and the microscope field as well as the image quality of the HUD including stereopsis, contrast and visibility of the OCT images.
To investigate the utility of the stereoscopic HUD in vivo, the MIOCT/HUD system was evaluated by five ophthalmic surgeons from the Duke Eye Center in 40 human vitreoretinal surgeries. All human participant studies were performed under protocols approved by the Duke University Health System Institutional Review Board. The studies included macular surgery for ERM, macular hole, lamellar hole, and/or retinal detachment. Additionally, the surgical instruments used and imaged during these studies included forceps (23g, 25g and 27g; Synergetics; O'Fallon, MO), loop (23g and 25g; Alcon; Ft. Worth, TX), soft tip cannula (25g; Synergetics; O'Fallon, MO), scraper (25g; Synergetics; O'Fallon, MO) and needle (41g; Synergetics; O'Fallon, MO). MIOCT imaging was performed on patients with standard 3-port vitrectomy and visualization of retina was achieved with a disposable flat contact lens (Alcon; Ft. Worth, TX). Real-time MIOCT B scans and volumetric data set were captured, processed and visualized by surgeons using the stereo HUD during live human retinal surgeries. The OCT images displayed via the HUD were duplicated on a separate small monitor, through which a trained technician adjusted the brightness, locations and sizes of the images displayed inside the oculars based on the surgeon’s preference before or during the surgery. The distance between the OCT volumetric stereo pair was optimized by the technician before the surgery to achieve comfortable stereoscopic vision. The foot joystick was set up to allow surgeons a real-time control of the perspective of OCT volumetric data set and each surgeon found a preferred perspective that was set as the “home perspective” in surgery. Verbal feedback was requested from surgeons regarding the utility of display formats and HUD image quality.
3. Results
3.1 Stereoscopic HUD optical characterization results
The grid charts placed in the surgical field of the microscope with or without the HUD were imaged through microscope oculars (Fig. 4(A), 4(B)) at 10x total magnification. Quantitatively characterized resolution, FOV and SMIA TV distortion of the Leica M844 surgical microscope with and without HUD are reported in Table 1. From the data and images, it is clear that the addition of HUD in the optical path of the microscope had no effect on resolution or FOV, and increased the pincushion distortion of the microscope by 0.09%. The distortion of the HUD optical system was also characterized by the same grid chart pattern displayed on the OLED micro-display (Fig. 4(C)) and was found to be 1.21%, which was 0.26% smaller than that of the surgical microscope (1.47%). The resolution of the HUD was limited by the pixel pitch of the OLED micro display, which was 12.6µm and the angular resolution subtended by viewers’ eyes was calculated to be 0.11°.
Fig. 4 Images captured through oculars when (A) a grid chart was placed in the microscope surgical field without HUD mounted, (B) the grid chart was placed in the microscope surgical field with HUD mounted and (C) the grid chart image was displayed the HUD’s OLED micro display.
Table 1. Comparison of resolution, FOV and distortion between microscope with HUD and without HUD.
3.2 Depth perception pilot study results
Five ophthalmic surgeons completed the assigned depth perception study tasks using both stereoscopic and monoscopic OCT volumes to guide their maneuvers. For all subjects, their task performance improved in stereoscopic OCT vision either in success rate or transfer time (Fig. 5(A), 5(B)). The average improved success rate and reduced time are shown in box plots (Fig. 5(C), 5(D)). These figures show that, on average, the subjects achieved 50.0% (SD = 37.3%) higher success rate and 35.0% (SD = 24.8%) less time in transferring each bead in stereoscopic OCT vision compared to monoscopic OCT vision.
Fig. 5 Depth perception pilot study results. (A) Success rate; (B) Average time spent in successfully transferring each bead; (C) Improved success rate in stereoscopic OCT vision; (D) Reduced time in stereoscopic OCT vision.
3.3 Intraoperative stereoscopic HUD visualization with MIOCT
Through the wet lab-based cadaveric porcine eye studies, the five ophthalmic surgeons familiarized themselves with the new display format inside the oculars and successfully used the MIOCT/HUD system to perform surgical maneuvers including brushing and peeling retinal membranes. Visualization 1 shows the surgeon’s view upon using surgical forceps to grab and peel retinal membranes during a cadaveric porcine eye surgery. The time required to calculate, render and display stereoscopic OCT volumes (16 B scans with 300 A lines in each B scan) was 40 ± 2 ms, achieving real time stereoscopic display. A typical display format of MIOCT B scans, volumes and microscope view during wet lab-based studies and live human surgeries was shown in Fig. 6.
Fig. 6 4D-MIOCT data displayed on HUD during a porcine eye operation. Images were captured with two cameras attached to the oculars of an assistant scope (Visualization 1).
Five ophthalmic surgeons provided feedback on the HUD and its use during simulated surgeries in the wet lab and five surgeons provided the same during and after use in human surgery. In the wet lab and in human surgeries, all surgeons reported preferring (1) to place OCT images in the peripheral regions which allowed an unobstructed central microscope view; (2) rather than viewing the OCT from the microscope’s top-down perspective, to visualize the OCT volume from an angled side perspective ranging between 30 to 90 degrees to the vertical and (3) to change rotational orientation to examine specific features or steps in surgery, but to otherwise maintain the rotation aligned with the real world. Preferences for the location of the B scan versus volume in the HUD varied across surgeons. During both wet lab and human vitreoretinal surgery, the ophthalmic surgeons reported that the HUD did not restrict or alter their normal view of surgery, and that the stereoscopic OCT volumes rendered in the HUD during retinal surgery were readily visible. Surgeons also did not appreciate any difference in brightness between left and right ocular images. They reported that they preferred stereoscopic to monoscopic visualization of MIOCT volumes for improved depth perception in the wet lab testing. After adjusting to their preferred angled side perspective view of the OCT volume, surgeons infrequently rotated the volume while maneuvering.
MIOCT volumetric movies were captured and visualized by surgeons using the stereo HUD during live human retinal surgery. Two stereo MIOCT volumetric images show stereoscopic visualization of the surgeon peeling the internal limiting membrane (ILM) (yellow arrows) away from underlying retina with surgical forceps (white arrows) in Fig. 7. The MIOCT/HUD systems enabled enhanced surgical visualization using stereo 3D OCT data rendered at a different perspective controlled by a surgeon through a foot-operated joystick. Figure 7(A) depicts one surgeon controlled viewing angle and rotation, which during surgery could be changed to the alternate angle and rotation seen in Fig. 7(B). Each stereo perspective reveals different information on the relationship between the forceps tips and the membrane. This is unlike the surgical microscope view that has limited depth perception. The video of stereoscopic vision (Visualization 2) shows visualization of the spatial structure of peeled membranes from the front, back, left and right perspectives (yellow arrows). The spatial position of the surgical forceps relative to both the retinal surface and the peeled membranes was enhanced in terms of depth with stereoscopic vision (Visualization 2) compared with the video of monoscopic vision (Visualization 3) when viewed from these four perspectives.
Fig. 7 4D-MIOCT stereoscopic volumes seen from different perspectives (Stereoscopic version: Visualization 2; Monoscopic version: Visualization 3). The white arrow points to the surgical forceps. The yellow arrow points to the internal limiting membrane (ILM).
4D MIOCT real-time volumetric imaging observed through the HUD enabled visualization of surgical peeling of ILM from different perspectives. Figure 8 illustrates a real time, stereoscopic MIOCT volume time series projected in the microscope oculars during ILM peeling in vitreoretinal surgeries (Visualization 4 and Visualization 5). In the OCT volumes, surgeons were able to view the forceps grasping and peeling ILM and the resulting deformation of the underlying retina surface. The spatial structure of the peeled ILM and the spatial interaction between the retinal surface and the surgical forceps were not clear in the monoscopic vision of the OCT video excerpts (Visualization 5), while stereoscopic vision provided surgeons the spatial information (visible elevated and scrolled sheet of tissue separated from the inner retinal surface) and could potentially help them locate the margin of the ILM peel (Visualization 4).
Fig. 8 4D-MIOCT stereoscopic images during human vitreoretinal surgery (Stereoscopic version: Visualization 4; Monoscopic version: Visualization 5). The surgeon used a surgical forceps to peel off the unwanted membrane. The white arrow points to the surgical forceps. The yellow arrow points to the internal limiting membrane (ILM).
4D MIOCT and HUD systems also enabled visualization of a surgical loop (white arrow) scraping against retinal membrane around a macular hole (yellow arrow) from different perspectives as illustrated in Fig. 9. In OCT volumes, the epiretinal membrane (ERM) material scraped from the retinal surface was readily visible on the tip of the surgical loop. Compared to monoscopic OCT vision (Visualization 7), stereoscopic OCT images (Visualization 6) provided more accurate depth information of the surgical loop relative to the rest tissues, which could potentially help surgeons better locate the unwanted membranes.
Fig. 9 4D-MIOCT stereoscopic images during human macular surgery (Stereoscopic version: Visualization 6; Monoscopic version: Visualization 7). The surgeon used a surgical loop to initiate retina. The white arrow points to the surgical loop. The yellow arrow points to a macular hole.
4. Discussion
We have shown that the integration of a novel stereoscopic HUD into an ophthalmic surgical microscope did not change microscope’s optical performance in terms of FOV, resolution or distortion. The optical performance of the HUD was excellent in terms of distortion and resolution. The angular resolution of the HUD subtended to viewers’ eyes was 0.11°, limited by micro display’s pixel pitch. Unlike previously reported HUDs which only project monoscopic images, the novel HUD displays stereoscopic images through both oculars and thus increased depth perception in OCT images during simulated surgeries and human vitreoretinal surgeries.
In our depth perception pilot study, we noted that surgeons performed the required tasks faster and more accurately using stereoscopic OCT compared to monoscopic OCT. The most likely reason for the difference in performance between two conditions is that in monoscopic OCT vision, the subjects had to reach each bead with forceps through trial and error due to poor depth perception in the OCT images. The enhanced depth perception in stereoscopic OCT allowed the surgeons to perceive the relative distances of objects in space and thus to localize the beads in three dimension faster and more accurately. We posit that the ability to view volumetric OCT data stereoscopically may confer a similar advantage to that conferred by stereoscopic versus monoscopic surgical microscopes.
Stereopsis is one of the strongest depth cues in which human brains reconstruct distance according to lateral displacement between two points perceived by left and right eyes [47]. As two points are displayed on a 3D viewing device, both eyes can be fixated at one point through vergence while focusing on the display plane. Visual comfort strongly depends on the relationship between vergence and accommodation, which tends to be maintained within human eyes’ depth of focus (DOF) [46]. In our stereoscopic HUD, the rotational offset between left and right eye OCT volumes was set to 9°. The vergence distance, defined as the distance from observer’s eyes to simulated points in stereoscopic OCT volumes, was between 235.5mm (4.25 D) and 266mm (3.76 D) while the focal distance, defined as the distance from observer’s eyes to the physical display, was 250mm (4 D). Therefore, the vergence distance was between + 0.25 D and −0.24 D relative to the display plane, which falls within human eyes’ DOF ( ± 0.3 D) [46] and thereby justifies the visual comfort experienced by surgeons.
The ability to view the surgical field from different perspectives is one of the MIOCT/HUD system’s most important advantages over a standard surgical microscope. All surgeon users preferred to orient the OCT volumes in an angled side perspective, which is orthogonal to the microscope’s top-down perspective. We believe that the side perspective provided by the OCT volumes helped surgeons visualize the distance between the tip of the surgical instrument and the retinal surface and membranes below. This was demonstrated through the surgeons’ selection of this stereo view in the depth perception study and in feedback from four of five surgeons in human surgery. Such depth information may not be as evident in the microscope’s top-down perspective.
Since surgeons are more familiar with the standard microscope view, it is possible that visualization from alternative perspectives, especially with rotation relative to the real world, may be disorienting during surgery. We believe that with experience, surgeons may be able to gain new intuition from this newly available information, particularly by recognizing the appearance and orientation of surgical instruments within the MIOCT field of view. Nonetheless, to prevent disorientation, in this prototype instrument we included the capability for the surgeon to rapidly return to a “home perspective” using a button on the foot joystick.
Although intrasurgical OCT systems remain in active development, advantages already discovered using such systems in ophthalmic surgeries including visualization of 3D micro-scale retinal structures (e.g. peeled ILM in Fig. 8 and macular hole in Fig. 9) and interactions between surgical tools and tissues from different perspectives. Therefore, MIOCT may improve surgeon’s understanding of surgical maneuvers’ impact on retinal tissues in real time and help surgeons adjust maneuvers accordingly. Since ophthalmic tissues and surgical maneuvers are 3 dimensional, 2D images such as OCT B scans (cross-sectional images) may be hard to interpret in real time while OCT volumes can provide intuitive intrasurgical information to guide surgeons. Since monoscopic vision of such OCT volumes limits the depth perception, we believe that stereoscopic HUD technology will be necessary for surgeons to optimally view OCT data sets acquired in next-generation intrasurgical OCT systems.
Besides the microscope-integrated stereoscopic HUD described in this report, other stereoscopic viewing devices such as external 3D monitors have been developed for ophthalmic surgeries which have the potential to display stereoscopic surgical field and OCT imagery simultaneously. For example, the commercial TrueVision 3D Surgical display system (Santa Barbara, CA) incorporates a polarization based stereoscopic ultra-high-definition monitor to display the surgical field. The transition from a stereo microscope to an external 3D monitor may substantially increase the field of view to surgeons, improve the ergonomics of intraocular surgery, and enable visualization of the retina using lower-intensity illumination [48]. However, the resolution of the microscope field image in the commercial system was limited by the 3D monitor’s total number of pixels, which would need to be improved to match the optical resolution capability of the surgical microscope.
5. Conclusion
To our knowledge, this work presents the first microscope integrated stereoscopic HUD used for intrasurgical OCT systems to allow surgeons view the OCT data and the surgical field through the oculars concurrently. Using spatial multiplexing, the HUD was designed to project stereo images into both oculars simultaneously utilizing only one micro-display. The optical performance of the surgical microscope with the HUD was quantitatively characterized and the addition of the HUD did not compromise the operating microscope view in terms of resolution (12.40 µm), field of view (21 mm) and pincushion distortion (1.56%). Compared to monoscopic OCT vision, stereoscopic OCT vision was found to improve success rate by 50.0% (SD = 37.3%) and reduce time by 35.0% (SD = 24.8%) on average in the depth perception pilot study. The ophthalmic surgeons who used the HUD reported that the HUD provided more information on surgical maneuvers without altering normal microscope field. These studies indicate the potential of the stereoscopic HUD in MIOCT-guided ophthalmic surgeries.
Acknowledgments
This research was supported by NIH Biomedical Research Partnership Grant R01-EY023039, “Intraoperative OCT Guidance of Intraocular Surgery” (Izatt/Toth MPI), NCRR UL1 RR024128-01 and R21 EY019411.
References and links
1. D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, “Optical coherence tomography,” Science 254, 1178 (1991).
2. C. A. Puliafito, M. R. Hee, C. P. Lin, E. Reichel, J. S. Schuman, J. S. Duker, J. A. Izatt, E. A. Swanson, and J. G. Fujimoto, “Imaging of Macular Diseases with Optical Coherence Tomography,” Ophthalmology 102(2), 217–229 (1995). [CrossRef] [PubMed]
3. M. R. Hee, C. A. Puliafito, C. Wong, J. S. Duker, E. Reichel, J. S. Schuman, E. A. Swanson, and J. G. Fujimoto, “Optical Coherence Tomography of Macular Holes,” Ophthalmology 102(5), 748–756 (1995). [CrossRef] [PubMed]
4. M. R. Hee, C. A. Puliafito, C. Wong, J. S. Duker, E. Reichel, B. Rutledge, J. S. Schuman, E. A. Swanson, and J. G. Fujimoto, “Quantitative assessment of macular edema with optical coherence tomography,” Arch. Ophthalmol. 113(8), 1019–1029 (1995). [CrossRef] [PubMed]
5. M. R. Hee, C. R. Baumal, C. A. Puliafito, J. S. Duker, E. Reichel, J. R. Wilkins, J. G. Coker, J. S. Schuman, E. A. Swanson, and J. G. Fujimoto, “Optical Coherence Tomography of Age-related Macular Degeneration and Choroidal Neovascularization,” Ophthalmology 103(8), 1260–1270 (1996). [CrossRef] [PubMed]
6. J. A. Izatt, M. R. Hee, E. A. Swanson, C. P. Lin, D. Huang, J. S. Schuman, C. A. Puliafito, and J. G. Fujimoto, “Micrometer-scale resolution imaging of the anterior eye in vivo with optical coherence tomography,” Arch. Ophthalmol. 112(12), 1584–1589 (1994). [CrossRef] [PubMed]
7. S. Radhakrishnan, A. M. Rollins, J. E. Roth, S. Yazdanfar, V. Westphal, D. S. Bardenstein, and J. A. Izatt, “Real-time optical coherence tomography of the anterior segment at 1310 nm,” Arch. Ophthalmol. 119(8), 1179–1185 (2001). [CrossRef] [PubMed]
8. M. Zhao, A. N. Kuo, and J. A. Izatt, “3D refraction correction and extraction of clinical parameters from spectral domain optical coherence tomography of the cornea,” Opt. Express 18(9), 8923–8936 (2010). [CrossRef] [PubMed]
9. R. P. McNabb, S. Farsiu, S. S. Stinnett, J. A. Izatt, and A. N. Kuo, “Optical Coherence Tomography Accurately Measures Corneal Power Change from Laser Refractive Surgery,” Ophthalmology 122(4), 677–686 (2015). [CrossRef] [PubMed]
10. K. V. Vienola, B. Braaf, C. K. Sheehy, Q. Yang, P. Tiruveedhula, D. W. Arathorn, J. F. de Boer, and A. Roorda, “Real-time eye motion compensation for OCT imaging with tracking SLO,” Biomed. Opt. Express 3(11), 2950–2963 (2012). [CrossRef] [PubMed]
11. O. Carrasco-Zevallos, D. Nankivil, B. Keller, C. Viehland, B. J. Lujan, and J. A. Izatt, “Pupil tracking optical coherence tomography for precise control of pupil entry position,” Biomed. Opt. Express 6(9), 3405–3419 (2015). [CrossRef] [PubMed]
12. S. A. Boppart, B. E. Bouma, C. Pitris, G. J. Tearney, J. G. Fujimoto, and M. E. Brezinski, “Forward-imaging instruments for optical coherence tomography,” Opt. Lett. 22(21), 1618–1620 (1997). [CrossRef] [PubMed]
13. P. N. Dayani, R. Maldonado, S. Farsiu, and C. A. Toth, “Intraoperative use of handheld spectral domain optical coherence tomography imaging in macular surgery,” Retina 29(10), 1457–1468 (2009). [CrossRef] [PubMed]
14. P. B. Knecht, C. Kaufmann, M. N. Menke, S. L. Watson, and M. M. Bosch, “Use of Intraoperative Fourier-Domain Anterior Segment Optical Coherence Tomography During Descemet Stripping Endothelial Keratoplasty,” Am. J. Ophthalmol. 150(3), 360–365 (2010). [CrossRef] [PubMed]
15. R. S. Maldonado, R. V. O’Connell, N. Sarin, S. F. Freedman, D. K. Wallace, C. M. Cotten, K. P. Winter, S. Stinnett, S. J. Chiu, J. A. Izatt, S. Farsiu, and C. A. Toth, “Dynamics of Human Foveal Development after Premature Birth,” Ophthalmology 118(12), 2315–2325 (2011). [CrossRef] [PubMed]
16. F. LaRocca, D. Nankivil, S. Farsiu, and J. A. Izatt, “Handheld simultaneous scanning laser ophthalmoscopy and optical coherence tomography system,” Biomed. Opt. Express 4(11), 2307–2321 (2013). [CrossRef] [PubMed]
17. D. Nankivil, G. Waterman, F. LaRocca, B. Keller, A. N. Kuo, and J. A. Izatt, “Handheld, rapidly switchable, anterior/posterior segment swept source optical coherence tomography probe,” Biomed. Opt. Express 6(11), 4516–4528 (2015). [CrossRef] [PubMed]
18. G. Geerling, M. Müller, C. Winter, H. Hoerauf, S. Oelckers, H. Laqua, and R. Birngruber, “Intraoperative 2-dimensional optical coherence tomography as a new tool for anterior segment surgery,” Arch. Ophthalmol. 123(2), 253–257 (2005). [CrossRef] [PubMed]
19. Y. K. Tao, J. P. Ehlers, C. A. Toth, and J. A. Izatt, “Intraoperative spectral domain optical coherence tomography for vitreoretinal surgery,” Opt. Lett. 35(20), 3315–3317 (2010). [CrossRef] [PubMed]
20. L. B. Lee and S. K. Srivastava, “Intraoperative spectral-domain optical coherence tomography during complex retinal detachment repair,” Ophthalmic Surg. Lasers Imaging 42, e71–e74 (2011). [PubMed]
21. L. De Benito-Llopis, J. S. Mehta, R. I. Angunawela, M. Ang, and D. T. H. Tan, “Intraoperative Anterior Segment Optical Coherence Tomography: A Novel Assessment Tool during Deep Anterior Lamellar Keratoplasty,” Am. J. Ophthalmol. 157(2), 334–341 (2014). [CrossRef] [PubMed]
22. J. P. Ehlers, J. Goshe, W. J. Dupps, P. K. Kaiser, R. P. Singh, R. Gans, J. Eisengart, and S. K. Srivastava, “Determination of feasibility and utility of microscope-integrated optical coherence tomography during ophthalmic surgery: the DISCOVER Study RESCAN Results,” JAMA Ophthalmol. 133(10), 1124–1132 (2015). [CrossRef] [PubMed]
23. J. Migacz, O. Carrasco-Zevallos, P. Hahn, A. Kuo, C. Toth, and J. Izatt, “Intraoperative Retinal Optical Coherence Tomography,” in Optical Coherence Tomography, W. Drexler and J. G. Fujimoto, eds. (Springer International Publishing, 2015), pp. 1771–1796.
24. O. Carrasco-Zevallos, B. Keller, C. Viehland, L. Shen, G. Waterman, B. Todorich, C. Shieh, P. Hahn, S. Farsiu, A. N. Kuo, C. A. Toth, and J. A. Izatt, “Live volumetric (4D) visualization and guidance of in vivo human ophthalmic microsurgery with intra-operative optical coherence tomography,” (Submitted).
25. O. Carrasco-Zevallos, B. Keller, C. Viehland, L. Shen, G. Waterman, C. Chukwurah, P. Hahn, A. N. Kuo, C. A. Toth, and J. A. Izatt, “Real-time 4D Stereoscopic Visualization of Human Ophthalmic Surgery with Swept-Source Microscope Integrated Optical Coherence Tomography,” Invest. Ophthalmol. Vis. Sci. 56, 4085 (2015).
26. P. Hahn, O. Carrasco-Zevallos, D. Cunefare, J. Migacz, S. Farsiu, J. A. Izatt, and C. A. Toth, “Intrasurgical Human Retinal Imaging With Manual Instrument Tracking Using a Microscope-Integrated Spectral-Domain Optical Coherence Tomography Device,” Transl. Vis. Sci. Technol. 4(4), 1 (2015). [PubMed]
27. N. D. Pasricha, C. Shieh, O. M. Carrasco-Zevallos, B. Keller, J. A. Izatt, C. A. Toth, and A. N. Kuo, “Real-Time Microscope-Integrated OCT to Improve Visualization in DSAEK for Advanced Bullous Keratopathy,” Cornea 34(12), 1606–1610 (2015). [CrossRef] [PubMed]
28. P. Hahn, J. Migacz, R. O’Connell, R. S. Maldonado, J. A. Izatt, and C. A. Toth, “The use of optical coherence tomography in intraoperative ophthalmic imaging,” Ophthalmic Surg. Lasers Imaging 42(4Suppl), S85–S94 (2011). [CrossRef] [PubMed]
29. S. Binder, C. I. Falkner-Radler, C. Hauger, H. Matz, and C. Glittenberg, “Feasibility of intrasurgical spectral-domain optical coherence tomography,” Retina 31(7), 1332–1336 (2011). [CrossRef] [PubMed]
30. P. Steven, C. Le Blanc, K. Velten, E. Lankenau, M. Krug, S. Oelckers, L. M. Heindl, U. Gehlsen, G. Hüttmann, and C. Cursiefen, “Optimizing descemet membrane endothelial keratoplasty using intraoperative optical coherence tomography,” JAMA Ophthalmol. 131(9), 1135–1142 (2013). [CrossRef] [PubMed]
31. S. Siebelmann, P. Steven, D. Hos, G. Hüttmann, E. Lankenau, B. Bachmann, and C. Cursiefen, “Advantages of microscope-integrated intraoperative online optical coherence tomography: usage in Boston keratoprosthesis type I surgery,” J. Biomed. Opt. 21(1), 016005 (2016). [CrossRef]
32. J. P. Ehlers, P. K. Kaiser, and S. K. Srivastava, “Intraoperative optical coherence tomography using the RESCAN 700: preliminary results from the DISCOVER study,” Br. J. Ophthalmol. 98(10), 1329–1332 (2014). [CrossRef] [PubMed]
33. Y. K. Tao, S. K. Srivastava, and J. P. Ehlers, “Microscope-integrated intraoperative OCT with electrically tunable focus and heads-up display for imaging of ophthalmic surgical maneuvers,” Biomed. Opt. Express 5(6), 1877–1885 (2014). [CrossRef] [PubMed]
34. J. P. Ehlers, S. K. Srivastava, D. Feiler, A. I. Noonan, A. M. Rollins, and Y. K. Tao, “Integrative advances for OCT-guided ophthalmic surgery and intraoperative OCT: microscope integration, surgical instrumentation, and heads-up display surgeon feedback,” PLoS One 9(8), e105224 (2014). [CrossRef] [PubMed]
35. K. Kihara, Y. Fujii, H. Masuda, K. Saito, F. Koga, Y. Matsuoka, N. Numao, and K. Kojima, “New three-dimensional head-mounted display system, TMDU-S-3D system, for minimally invasive surgery application: procedures for gasless single-port radical nephrectomy,” Int. J. Urol. 19, 886–890 (2012).
36. M. D. Heath and A. A. Cohen-Gadol, “Intraoperative stereoscopic 3D video imaging: pushing the boundaries of surgical visualisation and applications for neurosurgical education,” Br. J. Neurosurg. 26(5), 662–667 (2012). [CrossRef] [PubMed]
37. C. Eckardt and E. B. Paulo, “Heads-up surgery for vitreoretinal procedures: an experimental and clinical study,” Retina 36(1), 137–147 (2016). [CrossRef] [PubMed]
38. C. Wheatstone, “Contributions to the physiology of vision.–Part the first. On some remarkable, and hitherto unobserved, phenomena of binocular vision,” Philos. Trans. R. Soc. Lond. 128(0), 371–394 (1838). [CrossRef]
39. C. E. Rash, M. B. Russo, T. R. Letowski, and E. T. Schmeisser, “Helmet-mounted displays: Sensation, perception and cognition issues,” (DTIC Document, 2009).
40. N. S. Holliman, N. A. Dodgson, G. E. Favalora, and L. Pockett, “Three-dimensional displays: a review and applications analysis,” IEEE Trans. on Broadcasting 57(2), 362–371 (2011). [CrossRef]
41. J. A. Trias, “Real-time high-resolution 3-D large-screen display using laser-activated liquid crystal light valves,” (Google Patents, 1986).
42. M. Gross, S. Würmlin, M. Naef, E. Lamboray, C. Spagno, A. Kunz, E. Koller-Meier, T. Svoboda, L. V. Gool, S. Lang, K. Strehlke, A. V. Moere, and O. Staadt, “blue-c: a spatially immersive display and 3D video portal for telepresence,” in ACM SIGGRAPH 2003 Papers, (ACM, San Diego, California, 2003), pp. 819–827.
43. D. B. Corbin, “Stereo head mounted display using a single display device,” (Google Patents, 2001).
44. O. Carrasco-Zevallos, B. Keller, C. Viehland, P. Hahn, A. N. Kuo, P. J. DeSouza, C. A. Toth, and J. A. Izatt, “Real-time 4D visualization of surgical maneuvers with 100kHz swept-source Microscope Integrated Optical Coherence Tomography (MIOCT) in model eyes,” Invest. Ophthalmol. Vis. Sci. 55, 1633 (2014).
45. C. Viehland, B. Keller, O. M. Carrasco-Zevallos, D. Nankivil, L. Shen, S. Mangalesh, D. T. Viet, A. N. Kuo, C. A. Toth, and J. A. Izatt, “Enhanced volumetric visualization for real time 4D intraoperative ophthalmic swept-source OCT,” Biomed. Opt. Express (Submitted to).
46. D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks, “Vergence-accommodation conflicts hinder visual performance and cause visual fatigue,” J. Vis. 8(3), 33 (2008). [CrossRef] [PubMed]
47. P. Didyk, T. Ritschel, E. Eisemann, K. Myszkowski, and H.-P. Seidel, “A perceptual model for disparity,” in ACM Transactions on Graphics (TOG), (ACM, 2011), 96.
48. Y. S. Modi and J. P. Ehlers, “Heads-up Vitreoretinal Surgery: Emerging Technology in Surgical Visualization,” Retinal Physician 13, 26–29 (2016).
