Open Access
Add to BibSonomy
Abstract
Laser ablation is an effective treatment modality. However, current laser scanners suffer from laser defocusing when scanning targets at different depths in a 3D surgical scene. This study proposes a deep learning-assisted 3D laser steering strategy for minimally invasive surgery that eliminates laser defocusing, increases working distance, and extends scanning range. An optofluidic laser scanner is developed to conduct 3D laser steering. The optofluidic laser scanner has no mechanical moving components, enabling miniature size, lightweight, and low driving voltage. A deep learning-based monocular depth estimation method provides real-time target depth estimation so that the focal length of the laser scanner can be adjusted for laser focusing. Simulations and experiments indicate that the proposed method can significantly increase the working distance and maintain laser focusing while performing 2D laser steering, demonstrating the potential for application in minimally invasive surgery.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Minimally invasive surgery allows access to internal anatomy through natural orifices or small external incisions. Lasers can be used to perform surgical procedures in minimally invasive surgery, such as ablating, cutting, and excising tissue without or with small incisions [1]. The laser is projected on the target in a non-contact manner for surgical operations, minimizing the risk of contamination and infection [2]. Small vessels can be sealed to avoid dressing or suturing due to the thermal rise caused by the laser at the surgical site [3]. The high temperature also facilitates incisional sterility. However, the requirement for a straight working path of the rigid optical components limits the application of laser surgery within a confined environment. For example, suspension laryngoscopes that project laser energy to the laryngeal lesions require adequate laser beam exposure [4]. Currently, lasers with shorter wavelengths that can be coupled to flexible optical fibers, such as 445 nm and 808 nm, are adopted to enhance access to targets and achieve effective tissue dissection [5–8].
Flexible optical fibers have been used to deliver the laser to the target site, allowing laser energy to be projected into confined spaces through minimally invasive surgery [9]. Transoral laser surgery for head and neck cancer has been reported using flexible optical fibers in combination with the da Vinci robot arm [10,11]. However, the laser emitted from the fiber tip diverges rapidly; thus, the contact between the fiber and the tissue should be maintained to achieve effective ablation, resulting in a lack of scanning ability, uneven ablation, and fiber damage [12]. Although a scanning-capable laser scanner has been developed [13], manipulating the fiber tip results in a limited working distance of 1 mm.
Combining a flexible fiber and a fiber collimator is currently popular when designing laser scanners [14–16]. Fang et al. [14] designed a hydraulically driven laser scanner with a working distance of 15 mm that manipulated the laser beam by bending the optical fiber back and forth. Acemoglu et al. [15] built an electromagnetically driven laser scanner with a working distance of 30 mm by manipulating the optical fiber with a miniature coil. York et al. [16] developed a piezoelectric-driven laser scanner with a working distance of 25 mm that reflected a collimated beam using two mirrors. However, to prevent laser divergence, the working distance of previous laser scanners was fixed to the working distance of the optical components and could not be adjusted at will. According to endoscopic datasets SCARED [17], Hamlyn [18], and SERV-CT [19], the maximum depth in surgical endoscopic scenarios reaches 180 mm [20]. A laser scanner with a large working distance can pave the way for a wider range of surgical applications, potentially from endoluminal to laparoscopy.
Maintaining the laser focusing on targets at different depths in minimally invasive surgery facilitates uniform and complete tissue ablation. The general method is to use a plano-convex lens with a fixed focal length to achieve laser focusing. For example, Kundrat et al. [21] achieved laser collimation and subsequent focusing through two plano-convex lenses with a diameter of 7.5 mm. The lenses were coated with anti-reflective coatings, and the focal length was fixed at 20 mm. Regarding automatic laser focusing, Geraldes et al. [22] developed a hydraulically driven MEMS varifocal mirror with a size of 3 mm × 4.24 mm, which can adjust the focal length without moving the endoscope tip. The device can autofocus over a focal length of 12 mm to 52 mm. Schoob et al. [23] used a 3-axis mechanical scanning module to drive a lens system to achieve laser focusing by moving back and forth in the depth direction.
Measuring the target depth in real-time and adjusting the focal length of the laser scanner accordingly are feasible to achieve automatic laser focusing in robot-assisted surgery. At present, deep learning-based monocular depth estimation methods have achieved impressive performance in autonomous driving [24]. Contrary to the sparse depth map of lidar, monocular depth estimation can provide a dense depth map from a single image [25]. Applying deep learning-based monocular depth estimation methods to surgical scenarios to measure the target depth is natural. However, different from the brightness constancy assumption of previous depth estimation methods, endoscopic scenarios suffer from sparse and uneven distribution of key points due to inter-frame brightness fluctuations, lack of features, specular reflections, and tissue deformation [20,26]. In addition, collecting the ground truth of the endoscopic image is time-consuming and expensive, and the default values are inevitable [27]. Generalized results for patients with specific endoscopic textures, shapes, and colors without a number of ground truths are also difficult to obtain [28]. Therefore, a real-time monocular depth estimation method with high generalization capability under endoscopic scenarios is essential to maintain laser focusing.
This study proposes an intelligent 3D laser steering method using an optofluidic laser scanner capable of addressing the challenge of laser defocusing when scanning targets at different depths in a 3D surgical scene. As indicated in Table 1, compared with previous studies, the first contribution of this work is designing a low voltage-actuated laser scanner based on optofluidic technology for 3D laser steering. The designed optofluidic laser scanner has a miniature size of 10 mm × 11 mm and a light weight of ∼2 g without mechanical moving components, enabling installation on the end effector of a continuum endoscope. The second contribution is developing a 3D laser steering strategy based on deep learning. The deep learning method enables real-time monocular depth estimation, providing the focal length to the laser scanner for laser focusing. The third contribution is achieving repeatability of 30.8 µm while increasing the working distance to 180 mm, thereby satisfying the requirements of laser targeting accuracy (< 1 mm) [15,29] for clinical laser surgery. Finally, due to the continuous laser focusing ability, the proposed laser scanner has a large working distance, thus increasing the scanning range and maximum speed. It is assumed that previous laser scanners have great potential to achieve superior performance when paired with a liquid lens for continuous laser focusing. Simulations and experiments are performed to demonstrate the effectiveness of the deep learning-assisted 3D laser steering strategy.
Table 1. Comparison With State of The Art
2. Methods
2.1 System overview
The proposed 3D laser steering system is illustrated in Fig. 1 (a). A visible laser beam is emitted from the laser source and then transmitted to the fiber collimator through the flexible optical fiber. A fiber collimator is inserted into the endoscope’s working channel to deliver the laser to the scanner. The designed optofluidic laser scanner, installed on the end effector of the endoscope, is used to perform 3D laser steering in an endoscopic environment. Figure 1 (b), an endoscopic image from the SCARED dataset, shows the endoscopic environment. Figure 1 (c) shows the predicted depth map using the deep learning method. The three points, za, zb and zc, express different depths when laser scanning, in which zb is the projection point of the optofluidic laser scanner. According to the depth map, the height difference of the scanning path za to zc reaches approximately 70 mm, which poses a challenge for accurate laser ablation. Therefore, we designed the 3D laser steering device, as shown in Fig. 1 (d) and (e). The optofluidic laser scanner with a diameter of 10 mm and a length of 11 mm consists of a liquid prism for 2D laser steering and a liquid lens for laser focusing in the depth direction.
Fig. 1. The 3D laser steering device. (a) Schematic of the laser steering system. (b) Endoscopic image from the SCARED dataset. (c) Predicted depth map using deep learning method. (d) The optofluidic laser scanner consisting of a liquid prism and a liquid lens. (e) Schematic of the optofluidic laser scanner on the end effector of an endoscope.
The workflow of the developed 3D laser steering strategy is demonstrated in Fig. 2. When performing laser surgery in a 3D endoscopic environment, operators initially search for the target through the endoscopic image. After aiming the endoscope at the target, the depth of the endoscopic image is estimated using a deep learning method. Then, the 3D coordinates of the target point are sent to the computer. The computer, working as a controller, sends the control signal to the drivers of the optofluidic laser scanner. The optofluidic laser scanner comprises a liquid lens and a liquid prism, in which a commercial liquid lens (Optotune, EL-3-10) is used for laser focusing with an operating voltage of 1 V. The 2D laser steering on the projection plane is achieved with a self-made liquid prism that is actuated by a 1 kHz AC voltage of less than 15 V provided by the prism driver L298N and Arduino MEGA2560. The liquid lens and prism have a response time of 4 ms and 200 ms, respectively [30,31]. As a result, laser energy is focused and automatically projected to the defined ablation area by the optofluidic laser scanner.
2.2 Optofluidic laser scanner
2.2.1 Working principle of the liquid lens
The optofluidic laser scanner achieves 3D laser steering with a liquid lens for laser focusing and a liquid prism for laser scanning. The liquid lens used has a diameter of 10 mm and a length of 4 mm, whose cavity is filled with an optical fluid sealed with an elastic polymer membrane. The electromagnetic actuator in the lens applies pressure on the lens cavity by using current, thereby adjusting the curvature of the lens and finally controlling the optical power of the lens. As a result, the focal length of the liquid lens, which is equal to 1 m divided by the optical power, is adjusted.
Figure 3 demonstrates the schematics and simulations when applying different currents to the liquid lens. When the maximum driving current of +120 mA is applied to the liquid lens, a maximum optical power of +25 dpt is achieved, corresponding to a focal length of 40 mm. Similarly, when a driving current of 100 mA is applied to the liquid lens, an optical power of +20 dpt and a focal length of 50 mm are achieved. In our experiment, the current is set as I (mA), and the focal length f (mm) of the liquid lens can be approximately expressed as follows [30]:
in which the laser is focused when I > 0; the laser is diverged when I < 0, as shown in Fig. 3 (c). The divergent laser allows a smooth transition from cutting to coagulation, avoiding gross bleeding [9].Fig. 3. Schematics and simulations of the liquid lens for laser focusing. (a) Mechanism of the liquid lens when applied +120 mA. (b) Mechanism of the liquid lens when applied +100 mA. (c) Mechanism of the liquid lens when applied -120 mA.
2.2.2 Working principle of the liquid prism
The focused laser from the liquid lens becomes incident into the liquid prism for laser scanning on the projection plane. The proposed liquid prism performs laser steering based on electrowetting on dielectric (EWOD), enabling compact structure, low driving voltage, and high stability [32]. EWOD balances the interfacial tension of the solid-oil-water phase by applying voltages, thus getting different contact angles. Figure 4 (a) illustrates the schematic of the liquid prism, which is filled with transparent, immiscible, and density-matched oil and water with different refractive indices. Four pieces of indium tin oxide (ITO) glasses with a width of 6 mm, a height of 5 mm, and a thickness of 1 mm are staggered and spliced to form the side wall of the liquid prism. The top of the prism is bare glass, and the bottom is ITO glass, with an internal width of 3 mm and a height of 5 mm. By combining the four contact angles on the side walls, the liquid prism changes the tilt angle of the oil-water interface to refract the laser. The liquid prism is driven with an AC voltage due to the benefits of reducing contact angle hysteresis, suppressing charge trapping at the liquid-solid interface, and delaying contact angle saturation [33]. This study promotes the dielectric layer by incorporating 5 wt.% nano Al2O3 particles (Macklin, Shanghai, China) with a particle size of 30 nm [34], [35]. The laser beam is refracted twice as it passes through the liquid prism, resulting in laser deflection. Based on optical refraction, the laser steering angle δ (°) can be described by the tilt angle φ (°) of the oil-water interface as
Fig. 4. Schematic of the liquid prism for 2D laser steering. (a) Mechanism of the liquid prism, in which U1, U2, U3 and U4 are the driving voltages. (b) Relationship between the contact angle and the voltage.
Figure 4 (b) presents the relationship between the voltage and the contact angle when applying a voltage to one ITO electrode. The contact angle decreases from 160° to 65° when the voltage increases from 0 to 15 V. Besides, the moving posture of the designed liquid prism has no impact on the contact angle due to the density-matched oil and water [31]. Therefore, the four contact angles can be matched to achieve programmable laser steering by adjusting the voltages on the ITO electrodes. Moreover, the laser spot position can be described in polar coordinates as (δ, ψ, z), where δ is the laser steering angle, ψ is the rotation angle, and z is the depth [37]. Based on (2) and (3), the spot position on the projection plane is determined by the voltages applied to the liquid prism [38].
2.2.3 Optical simulation
Figure 5 illustrates the simulations of 3D laser steering using the optofluidic laser scanner. The liquid lens maintains laser focusing with an optical power of +20 dpt at the focal length of 50 mm in Fig. 5 (a), (b), and (c), whereas the liquid prism performs laser scanning from the positive y-axis to the negative y-axis with a maximum liquid interface tilt angle of 20°. Similarly, the liquid lens can adjust the focal length to focus on targets at different depths. Figure 5 (d) illustrates that the laser beam can be focused at a focal length of 40 mm with an optical power of +25 dpt when scanning the positive x-axis. The simulations demonstrate the feasibility of the proposed 3D laser steering strategy.
Fig. 5. Simulations of 3D laser steering using the optofluidic laser scanner. (a) Positive y-axis steering with a focal length of 50 mm. (b) No steering with a focal length of 50 mm. (c) Negative y-axis steering with a focal length of 50 mm. (d) Positive x-axis steering with a focal length of 40 mm.
2.3 Deep learning method
To achieve automatic laser focusing on targets at different depths, this study adopts a deep learning-based monocular depth estimation method to measure the target depth, based on which the focal length of the liquid lens is adjusted. A deep learning architecture, dense prediction transformer (DPT) [39], is used to address the lack of texture and brightness fluctuations in endoscopic environments. DPT assembles tokens from the dense vision transformer into images of various resolutions and combines them into full-resolution predictions with a convolutional decoder. DPT achieves depth estimation with strong generalization and global coherence by training on 1.4 million images.
Figure 6 demonstrates the qualitative depth evaluations of DPT on the SCARED, Hamlyn, and SERV-CT datasets. The ground truth in the SCARED dataset is sparse, lacking many depth values, whereas the depth estimation map using DPT for a single image is dense. Compared with the ground truth, DPT presents excellent depth estimation performance in the three datasets.
Moreover, quantitative experiments are performed to evaluate the DPT depth estimation results with 7459 images of the SCARED dataset, 5816 images of the Hamlyn dataset, and 16 images of the SERV-CT dataset. Table 2 lists the absolute relative difference (Abs-rel), the squared relative difference (Sq-rel), the root mean square error (RMSE), the logarithm RMSE (RMSE log), and the accuracy under a threshold (A1 = 1.25, A2 = 1.252, A3 = 1.253) of the quantitative experiments. DPT has the best performance in the Hamlyn dataset with an Abs-rel of 0.075 and an RMSE of 9.199, whereas DPT has the largest Abs-rel of 0.109 and RMSE of 13.281 in the SERV-CT dataset. The impact of the predicted depth error on laser spot size is negligible due to the excellent performance of DPT on depth estimation. In our experiments, when running on an Intel i9-12900 H CPU and an Nvidia RTX 3060 GPU, DPT realizes real-time depth estimation with a frame rate of 15 Hz.
Table 2. Quantitative Depth Estimations of DPT on The SCARED, Hamlyn, and SERV-CT Datasetsa
Therefore, the depth information provided by the deep learning method can be used as the focal length of the liquid lens to achieve autofocus. Meanwhile, the operator can also use the depth information to determine the optimal working distance under different surgical scenarios. Compared with endoscopic surgery that relies on the operator's experience to estimate depth, the deep learning method provides a new scheme for reliable clinical surgery.
3. Experiment results
3.1 Laser focusing of the liquid lens
The proposed optofluidic laser scanner uses a liquid lens for laser focusing in the designed 3D laser steering strategy. We initially tested the laser focusing ability of the liquid lens. The minimum spot that can be obtained theoretically is the Airy disk [40], expressed as
where D1 is the Airy disk diameter, λ is the wavelength, f is the focal length, and D is the aperture diameter. The Airy disk diameter increases linearly with the focal length. However, achieving the minimum spot diameter through focusing is difficult due to non-ideal laser beam quality. Therefore, we measured the observed spot diameter before and after focusing to demonstrate the laser focusing ability of the liquid lens in this section, which is feasible in previous studies [13–16]. The experiment setup is shown in Fig. 1 (a), where the liquid lens was affixed to the end effector of an endoscope. An incident beam with a diameter of 2 mm was emitted from a fiber collimator. A CMOS camera with a high resolution of 10.4 µm/pixel to 24 µm/pixel was used to record the unfocused and focused spot. Then, ImageJ software was used to measure the spot size. At each focal length, the spot size was measured three times.Figure 7 (a) illustrates the laser spot when the depth is 50 mm, 100 mm, and 180 mm without and with the liquid lens. Without the liquid lens, the laser beam gradually diverges as the depth increases, and the spot diameter increases accordingly. The observed spot size is presented in Fig. 7 (b). As shown by the red line in Fig. 7 (b), when the focal length is 40 mm, the observed spot diameter is 0.64 mm. When the focal length is 180 mm, the observed spot diameter reaches a maximum value of 1.41 mm. Considering that when performing laser ablation in a non-contact mode, an observed spot size of 2 mm can achieve effective tissue ablation [13], the focused spot size is acceptable. The ratio of the unfocused spot area to the focused spot area varies from 22.06 to 40.23, reflecting the liquid lens's excellent focusing ability.
Fig. 7. Laser focusing test. (a) The laser spot at different depths without and with the liquid lens, respectively. (b) Observed spot diameters at different focal lengths. The f is the focal length. Diameter(f-10) and Diameter(f + 10) represent the laser spot diameter at the (f-10) mm and (f + 10) mm from the liquid lens, with the focal length of the liquid lens set to f.
Moreover, the depth of focus ZF can be expressed by beam waist length [41]:
where ω is the outgoing beam waist radius, approximately equal to λf /D. The depth of focus increases with the focal length. When focal length is 140 mm, the depth of focus reaches 20.0 mm, indicating that working at a long focal length facilitates the reduction of the focusing error caused by depth estimation.Since the liquid lens can achieve continuous laser focusing when provided with depth information, and the focusing error of the liquid lens depends on the accuracy of deep learning-based depth estimation methods, we studied the effect of the depth estimation error on the focused spot size to demonstrate the focusing capability of the liquid lens. With the focal length of the liquid lens to f, Fig. 7 (b) presents the spot diameters in the three states of f and f ± 10 mm from the lens, respectively. When the distance from the target to the liquid lens is f, it is the focused state, as shown by the red line; when the distance from the target to the liquid lens is f-10 mm, the spot diameter is Diameter (f-10), as shown by the green line; when the distance from the target to the liquid lens is f + 10 mm, the spot diameter is Diameter (f + 10), as shown by the orange line. The difference between the focused spot diameter and the spot diameter at f-10 mm and f + 10 mm is the focusing error, between 0.01 mm and 0.09 mm. Considering that the depth estimation error of the proposed deep learning method is less than 9.3 mm in over 13000 images of SCARED and Hamlyn datasets, it indicates that the focusing error caused by depth estimation is negligible.
3.2 2D laser scanning of the liquid prism
After investigating the laser focusing ability of the optofluidic laser scanner along the axis direction, 2D laser scanning on the projection plane was studied with a liquid prism. The proposed liquid prism refracts the laser spot to different sites by adjusting the tilt angle of the oil-water interface, which is determined by the driving voltages. Figure 8 illustrates the working status of the liquid prism. Initially, the driving voltage is 0 V, and the water assumes a convex shape with an obtuse contact angle due to the hydrophobic layer inside the prism, as shown in Fig. 8 (a). When U1 = U2 = U3 = U4 = 8 V, as shown in Fig. 8 (b), a horizontal liquid interface is achieved, enabling the laser beam to project to the origin directly. Figure 8 (c) and (d) demonstrate the liquid interface when conducting x-axis and y-axis laser steering, respectively. Figure 8 (e) and (f) illustrate the liquid interface when performing laser steering in two diagonal directions. The liquid prism can achieve a flat and slippery liquid interface by combining the four contact angles on the sidewalls. Therefore, the designed liquid prism enables 2D laser scanning by switching different liquid interface states.
Fig. 8. Working status of the liquid prism for 2D laser steering. (a) Initial state with U1 = U2 = U3 = U4 = 0 V. (b) Horizontal liquid interface with U1 = U2 = U3 = U4 = 8 V. (c) x-axis laser steering with U1 = 15 V, U2 = 2 V, and U3 = U4 = 8 V. (d) y-axis laser steering with U1 = U2 = 8 V, and U3 = 15 V, U4 = 2 V. (e) Diagonal laser steering with U1 = U3 = 15 V, and U2 = U4 = 2 V. (f) Diagonal laser steering with U1 = U4 = 2 V, and U2 = U3 = 15 V.
Figure 9 demonstrates the 2D laser scanning paths when placing the liquid prism at a working distance of 180 mm. The spot locations on the projection plane were extracted by image processing. The repeatability is calculated as the root mean square error of successive cycles from the first cycle. Maximum error is measured as the maximum absolute distance between the scanning points in successive cycles from the first cycle. Table 3 illustrates the repeatability and maximum error of the laser scanning paths in vertical, horizontal, and two diagonal directions. The repeatability difference in each direction of the laser scanning paths results from the fabrication errors of the liquid prism [42]. The designed low voltage-actuated liquid prism achieves repeatability of as low as 30.8 µm at a 180 mm working distance. The scanning lengths of diagonal lines A and B are 28.4 and 30.7 mm, respectively, indicating a scanning range of 30.7 mm by 28.4 mm for the optofluidic laser scanner. Moreover, considering that a 1-2 mm resection margin is demanded to ensure that no cancer cells remain at the margin of the resected tissue [15], the maximum repeatability of 48.3 µm and the maximum error of 272.4 µm satisfy the clinical requirement for laser targeting accuracy (< 1 mm). Therefore, the experiments demonstrate the 2D laser steering ability of the optofluidic laser scanner with high accuracy at a working distance of 180 mm. Notably, the cubic liquid prism is manually fabricated, and the scanning path is not a straight line due to manufacturing errors.
Fig. 9. 2D laser scanning paths using the liquid prism at a working distance of 180 mm. (a) Horizontal and vertical directions. (b) Diagonal directions.
Table 3. Repeatability and Maximum Error of 2D Laser Scanning Paths
Figure 10 shows the qualitative comparison of laser spot sizes at a depth of 180 mm for 2D laser steering and 3D laser steering. Without the liquid lens for laser focusing, the laser beam in 2D laser steering diverges severely at a depth of 180 mm, as shown in Fig. 10 (a). By contrast, a significant decrease in laser spot size was observed due to the liquid lens for laser focusing in 3D laser steering, as shown in Fig. 10 (b). The comparison indicates the effectiveness of 3D laser steering using the optofluidic laser scanner.
Fig. 10. Qualitative comparison of laser spot sizes at a depth of 180 mm for 2D laser steering and 3D laser steering. (a) Grayscale images of 2D laser steering with a liquid prism for laser scanning but without a liquid lens for laser focusing. (b) Grayscale images of 3D laser steering with a liquid lens for laser focusing and a liquid prism for laser scanning.
3.3 Phantom demonstration
Figure 11 uses a pork phantom to demonstrate the workflow of the proposed deep learning-assisted 3D laser steering strategy in minimally invasive surgery. Figure 11 (a) shows the endoscopic image of the pork phantom, where points A and B are the start and end points of the laser scanning path, respectively. Figure 11 (b) demonstrates the depth map of the pork phantom predicted by the deep learning-based monocular depth estimation method. The predicted depth map accurately reflects the distance from the phantom surface to the endoscopic camera but with a distorted predicted depth at the edge of the image. Figure 11 (c) and (d) indicate the 3D laser steering on the pork phantom from point A to point B.
Fig. 11. 3D laser steering on a pork phantom. (a) Endoscopic image of a pork phantom. (b) Predicted depth map of the pork phantom. (c) and (d) 3D laser steering from point A to point B. (e) Predicted depth and the resulting liquid lens driving current versus scanning length from point A to point B.
Due to the quantitative corresponding of laser focal length to the driving current of the liquid lens, considering the scattering of visible light in the tissue, we recorded the driving current of the liquid lens, as shown in Fig. 11 (e). The blue curve donates the predicted depth on scanning path AB, which is considered the focal length of the liquid lens. The red curve in Fig. 11 (e) presents the relationship between the scanning length from point A to point B and the driving current of the liquid lens, illustrating the laser focusing state. The different driving currents indicate that the optofluidic laser scanner achieves laser focusing by adjusting the focal length based on the predicted depth while performing laser scanning. Through the corresponding relationship between current and focal length (Eq. (1)), we can verify that the liquid lens performs laser focusing. Therefore, the experiment demonstrates the workflow of using the deep learning method for laser focusing with the liquid lens while performing laser scanning.
4. Discussion
Previous studies have demonstrated the excellent power handling ability of the liquid devices. For example, [43] tested the liquid devices using a continuous wave of 4 W at 1550 nm for laser communications; [44] achieves laser marking using the liquid lens with a high-power laser. In this study, the liquid lens has an optical damage threshold of 1 kW/cm2, which can withstand high-power surgical lasers.
In addition, the transmission of the liquid device at different wavelengths can be increased by coating. The liquid lens can be coated to maintain the transmission of visible to infrared light (400 nm to 2500 nm) at about 95% [30]. The liquid lens has a power loss of about 5%. For the liquid prism, reflection loss occurs when the laser passes through the oil-water and water-air interfaces. When the tilt angle of the liquid interface is 25°, the reflection loss reaches the maximum of 2.8% according to the Fresnel equation [36]. Moreover, the optical fiber and collimator should be matched according to the wavelength and power of the laser source. This study uses a fiber collimator with a maximum power of 300 mW and a working wavelength of 650 nm, which can be replaced easily.
Although we tested the laser scanner at the maximum depth in the existing endoscopic datasets (180 mm), the working distance can be adjusted according to different working spaces due to the excellent focusing ability of the liquid lens. The proposed laser scanner has a large working distance, indicating a remarkable adaptability to different endoscopic sceneries.
The designed 3D laser steering system can be further enhanced in the following aspects: First, a liquid lens with greater focusing ability can achieve a shorter focal length, potentially allowing a wider range of biomedical applications. Second, fabricating the liquid prism by standard micromachining processes can reduce the manufacturing error and facilitate miniaturization. When the laser spot is visible, the closed-loop control method is also significant for correcting nonlinearities and manufacturing errors. Third, a more accurate depth estimation algorithm with robust adaptability to the positional displacement resulting from tissue movement during surgical procedures facilitates precise laser focusing. For example, the recently emerged neural radiance fields method [45] has the potential to achieve better depth estimation and 3D reconstruction in minimally invasive surgery.
5. Conclusion
This study proposes a deep learning-assisted 3D laser steering strategy using an optofluidic laser scanner for minimally invasive surgery. The optofluidic laser scanner uses a liquid lens for laser focusing and a liquid prism for 2D laser scanning, capable of addressing the laser defocusing when scanning targets at different depths in a 3D surgical scene. The deep learning-based monocular depth estimation method provides the focal length for laser focusing in real time. The proposed low voltage-actuated optofluidic laser scanner has a large working distance and scanning range with excellent repeatability. Experiments were carried out to verify the laser focusing ability and 2D laser steering ability of the optofluidic laser scanner. The optofluidic laser scanner achieves laser focusing at a working distance of 180 mm. The repeatability of 2D laser steering reached 30.8 µm at a depth of 180 mm, satisfying the clinical laser targeting accuracy (< 1 mm) and implying the reliability and stability of the optofluidic laser scanner. In future work, we will combine depth imaging, such as OCT imaging, MRI, etc., to further explore the application of the optofluidic laser scanner in laser surgery.
Funding
University Grants Committee (C1134-20G, CityU 11211421); National Natural Science Foundation of China (U20A20194); Science and Technology Foundation of Shenzhen City (SGDX2020110309300502).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data is available upon reasonable request for the author.
References
1. E. Khalkhal, M. Rezaei-Tavirani, M. R. Zali, et al., “The evaluation of laser application in surgery: a review article,” J Lasers Med Sci 10(5), S104–S111 (2019). [CrossRef]
2. C. Y. Hsiao, S.-C. Yang, A. Alalaiwe, et al., “Laser ablation and topical drug delivery: a review of recent advances,” Expert Opin. Drug Delivery 16(9), 937–952 (2019). [CrossRef]
3. S. Parker, M. Cronshaw, E. Anagnostaki, et al., “Current concepts of laser-oral tissue interaction,” Dentistry Journal 8(3), 61 (2020). [CrossRef]
4. B. Benjamin and C. E. Lindholm, “Systematic direct laryngoscopy: the lindholm laryngoscopes,” Ann Otol Rhinol Laryngol 112(9), 787–797 (2003). [CrossRef]
5. B. J. Miller, A. Abdelhamid, Y. Karagama, et al., “Applications of office-based 445 nm blue laser transnasal flexible laser surgery: a case series and review of practice,” Ear Nose Throat J 100(1_suppl), 105S–112S (2021). [CrossRef]
6. B. Balouch, R. Garabet, P. J. Maxwell, et al., “The safety and efficacy of the 445-nm blue laser for operative management of benign nonvascular laryngeal lesions,” J Voice (2023).
7. D. Agop-Forna, C. Topoliceanu, N. Forna, et al., “Laser applications in oral surgery,” Annals of the Academy of Romanian Scientists Series on Biological Sciences 10(2), 48–55 (2021). [CrossRef]
8. F. Angiero, L. Parma, R. Crippa, et al., “Diode laser (808 nm) applied to oral soft tissue lesions: a retrospective study to assess histopathological diagnosis and evaluate physical damage,” Lasers Med Sci 27(2), 383–388 (2012). [CrossRef]
9. M. Karaman, T. Gün, B. Temelkuran, et al., “Comparison of fiber delivered CO(2) laser and electrocautery in transoral robot assisted tongue base surgery,” Eur Arch Otorhinolaryngol 274(5), 2273–2279 (2017). [CrossRef]
10. S. C. Desai, C.-K. Sung, D. W. Jang, et al., “Transoral robotic surgery using a carbon dioxide flexible laser for tumors of the upper aerodigestive tract,” Laryngoscope 118(12), 2187–2189 (2008). [CrossRef]
11. C. A. Solares and M. Strome, “Transoral robot-assisted CO2 laser supraglottic laryngectomy: experimental and clinical data,” Laryngoscope 117(5), 817–820 (2007). [CrossRef]
12. S. Patel, R. Milind, K. Stefan, et al., “Endoscopic laser scalpel for head and neck cancer surgery,” in Photonic Therapeutics and Diagnostics VIII, (2012).
13. M. Zhao, T. J. C. Oude Vrielink, A. A. Kogkas, et al., “LaryngoTORS: a novel cable-driven parallel robotic system for transoral laser phonosurgery,” IEEE Robot. Autom. Lett. 5(2), 1516–1523 (2020). [CrossRef]
14. G. Fang, M. K. Chow, J. D. L. Ho, et al., “Soft robotic manipulator for intraoperative MRI-guided transoral laser microsurgery,” Sci. Robot. 6(57), eabg5575 (2021). [CrossRef]
15. A. Acemoglu, D. Pucci, L. S. Mattos, et al., “Design and control of a magnetic laser scanner for endoscopic microsurgeries,” IEEE/ASME Trans. Mechatron. 24(2), 527–537 (2019). [CrossRef]
16. P. A. York, R. Peña, D. Kent, et al., “Microrobotic laser steering for minimally invasive surgery,” Sci. Robot. 6(50), eabd5476 (2021). [CrossRef]
17. M. Allan, J. Mcleod, C. Wang, et al., “Stereo correspondence and reconstruction of endoscopic data challenge,” arXiv, arXiv:2101.01133 (2021). [CrossRef]
18. D. Stoyanov, M. V. Scarzanella, Philip Pratt, et al., “Real-time stereo reconstruction in robotically assisted minimally invasive surgery,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervention, 275–282 (2010).
19. P. J. E. Edwards, D. Psychogyios, S. Speidel, et al., “SERV-CT: A disparity dataset from cone-beam CT for validation of endoscopic 3D reconstruction,” Med. Image Anal. 76, 102302 (2022). [CrossRef]
20. S. Shao, Z. Pei, W. Chen, et al., “Self-Supervised monocular depth and ego-Motion estimation in endoscopy: Appearance flow to the rescue,” Med. Image Anal. 77, 102338 (2022). [CrossRef]
21. I. Gannot, “Endoluminal non-contact soft tissue ablation using fiber-based Er:YAG laser delivery,” in Optical Fibers and Sensors for Medical Diagnostics and Treatment Applications XVI, (2016).
22. A. Geraldes, P. Fiorini, and L. S. Mattos, “Auto-Focusing System for Endoscopic Laser Surgery based on a Hydraulic MEMS Varifocal Mirror,” International Conference on Advanced Robotics (ICAR), (2019).
23. A. Schoob, D. Kundrat, L. Kleingrothe, et al., “Tissue surface information for intraoperative incision planning and focus adjustment in laser surgery,” Int J Comput Assist Radiol Surg 10(2), 171–181 (2015). [CrossRef]
24. C. Godard, O. M. Aodha, M. Firman, et al., “Digging into self-supervised monocular depth estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 3828–3838 (2019).
25. X. Dong, M. A. Garratt, S. G. Anavatti, et al., “Towards Real-Time Monocular Depth Estimation for Robotics: A Survey,” IEEE Trans. Intell. Transport. Syst. 23(10), 16940–16961 (2022). [CrossRef]
26. R. Wei, B. Li, H. Mo, et al., “Distilled visual and robot kinematics embeddings for metric depth estimation in monocular scene reconstruction,” IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 8072–8077 (2022).
27. X. Liu, A. Sinha, M. Ishii, et al., “Dense depth estimation in monocular endoscopy with self-supervised learning methods,” IEEE Trans. Med. Imaging 39(5), 1438–1447 (2020). [CrossRef]
28. K. B. Ozyoruk, G. I. Gokceler, T. L. Bobrow, et al., “EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos,” Med. Image Anal. 71, 102058 (2021). [CrossRef]
29. D. Kundrat, R. Graesslin, A. Schoob, et al., “Preclinical performance evaluation of a robotic endoscope for non-contact laser surgery,” Ann Biomed Eng 49(2), 585–600 (2021). [CrossRef]
30. “Optotune Company. EL-3-10-lens.” [Online] Available: https://www.optotune.com/el-3-10-lens.
31. C. Zhang, H. Mo, L. Zheng, et al., “Design of a liquid-driven laser scanner with low voltage based on liquid-infused membrane,” Advanced Intelligent Systems 5, 2200259 (2023). [CrossRef]
32. J. Cheng and C.-L. Chen, “Adaptive beam tracking and steering via electrowetting-controlled liquid prism,” Appl. Phys. Lett. 99(19), 191108 (2011). [CrossRef]
33. X. Li, H. Tian, J. Shao, et al., “Decreasing the Saturated Contact Angle in Electrowetting-on-Dielectrics by Controlling the Charge Trapping at Liquid-Solid Interfaces,” Adv. Funct. Mater. 26(18), 2994–3002 (2016). [CrossRef]
34. Y.-P. Zheng, J.-X. Zhang, Q. Li, et al., “The Influence of High Content Nano-Al2O3 on the Properties of Epoxy Resin Composites,” Polym.-Plast. Technol. Eng. 48(4), 384–388 (2009). [CrossRef]
35. F. Wang, M. Liu, C. Liu, et al., “Light-induced charged slippery surfaces,” Sci. Adv. 8(27), eabp9369 (2022). [CrossRef]
36. G. Chen, Y. Wang, D. Jiang, et al., “A programmable compound prism powered by triboelectric nanogenerator for highly efficient solar beam steering,” Nano Energy 80, 105524 (2021). [CrossRef]
37. D. Sauter, “Simultaneous beam steering and shaping using a single-interface optofluidic component,” Journal of Optical Microsystems, 1 (2021).
38. J. Lee, J. Lee, Y. H. Won, et al., “Image stitching using an electrowetting-based liquid prism with a fabrication method,” Opt. Express 29(2), 729–739 (2021). [CrossRef]
39. R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in Proc. IEEE Int. Conf. Comput. Vis., 12179–12188 (2021).
40. D. Yu and H. Tan, “Engineering Optics,” Mechanical Industry Press, 247–262 (2000).
41. Y. Chen and J. Wang, “Laser Principle,” Zhejiang University Press, 206–211 (1992).
42. C. Liu, D. Wang, Q.-H. Wang, et al., “Multifunctional optofluidic lens with beam steering,” Opt. Express 28(5), 7734–7745 (2020). [CrossRef]
43. S. Kacker, “Optical Performance and Prototyping of a Liquid Lens Laser Communications Transceiver,” Massachusetts Institute of Technology (2022).
44. D. Stadler and H.-Y. Lo, “3D laser processing with tunable lens technology,” International Congress on Applications of Lasers & Electro-Optics (2016).
45. B. Mildenhall, P. P. Srinivasan, M. Tancik, et al., “NERF: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM 65(1), 99–106 (2022). [CrossRef]
