Open Access
Add to BibSonomy
Abstract
Partial aperture imaging is a combination of two different techniques; coded aperture imaging and imaging through an aperture that is only a part of the complete disk, commonly used as the aperture of most imaging systems. In the present study, the partial aperture is a ring where the imaging through this aperture resolves small details of the observed scene similarly to the full disk aperture with the same diameter. However, unlike the full aperture, the annular aperture enables using the inner area of the ring for other applications. In this study, we consider the implementation of this special aperture in medical imaging instruments, such as endoscopes, for imaging internal cavities in general and of the human body in particular. By using this annular aperture, it is possible to transfer through the internal open circle of the ring other elements such as surgical tools, fibers and illumination devices. In the proposed configuration, light originated from a source point passes through an annular coded aperture and creates a sparse, randomly distributed, intensity dot pattern on the camera plane. A combination of the dot patterns, each one recorded only once, is used as the point spread hologram of the imaging system. The image is reconstructed digitally by cross correlation between the object intensity response and the point spread hologram.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Endoscopy is an important medical diagnosis and treatment tool for many body cavities such as the colon, esophagus, abdomen, uterus, lungs and stomach. In various medical areas, endoscopy provides an alternative for open surgery, faster recovery and less post-operative pain and discomfort. With minimally invasive surgery like laser lithotripsy, an endoscopic system is used as a high-resolution imaging setup which helps surgeons operate on delicate organs with high precision. The word endoscopy originated from the Greek language and means “to examine within” [1]. Therefore, apart from medical applications, endoscopy also finds its application in nondestructive observation of internal volumes of structures, machines, or systems. However, with all the advantages of this technology, endoscopy also faces some considerable limitations, mainly because of the limited size of the lateral cross-section of the observed cavity. The use of surgical tools in parallel with the vision system adds several limitations on the optical performance of the endoscope. The proposed method in this paper can dramatically improve a single incision via laparoscopic surgery [2], and can also be applied to robotic surgery [3]. Obstruction of the field of view, restrictions on the endoscope’s maximal possible numerical aperture and boundaries on the depth of field are some of the additional limitations. To control the endoscope’s depth of field, a liquid crystal based tunable lens has been integrated into an endoscopic system [4]. Digital holography has been integrated with endoscopic system to add 3D imaging capabilities [5]. The maximal numerical aperture of an endoscope operating in parallel with other devices is considered in this study.
Endoscopes with high resolution imaging capability is of great importance for optimized interactions in medical procedures. The surgical part of the system, may consist of a fiber delivering laser radiation, a high-intensity focused ultrasound surgical applicator, a radio-frequency device or a mechanical tool. Each of them can be used for tumor ablation or resection. In addition, there is an illumination component. Since the internal size of an organ under treatment is limited, additional instruments force engineers to decrease the aperture diameter of the imaging system, and hence the image resolution and the field of view are reduced. Alternatively, if the imaging system has an annular aperture with the maximal diameter for a given organ, the maximal resolution is guaranteed, and the space surrounded by the annular aperture is free to be used for the other required instruments. A scheme of the new imaging concept is shown in Fig. 1, where the annular imaging aperture is implemented on the periphery of the distal tip of endoscope. The internal area of the ring is utilized by a light source, illuminating fiber and surgical tools. The image sensor is positioned behind the annular aperture. Figure 1 is a conceptual representation of our new imaging idea and depending upon the application, more tools can be introduced into the system. The main goal of this study is to develop a new applicator, with superior features, by integrating optical and digital image processing tools. To a large extent, this is now possible thanks to currently available computers and state-of-the-art electro-optical devices, including spatial light modulators (SLMs) and digital cameras.
An imaging method which combines two different concepts; coded aperture imaging and incoherent digital holography, has been recently proposed [6]. The new imaging concept is dubbed interferenceless coded aperture correlation holography (I-COACH) [6,7]. In I-COACH, a one-time calibration procedure is required, in which a library of pattern responses to a point object positioned at various axial locations are created. Each complex-valued response function is termed point spread hologram (PSH). Following the calibration stage, the intensity response of a 3D object, placed within the axial boundaries of the library, is recorded. The object can be considered as a collection of point objects distributed on the 3D object shape. Therefore, the object response, termed object hologram (OH), is a collection of the PSHs, distributed according to the shape of the object. Once the complex hologram of the object is recorded into the computer, any axial plane of the observed object space is finally reconstructed by a digital 2D cross-correlation between the OH and the corresponding PSH from the library. Modulating the object beam by a coded phase mask (CPM) [8,9], and cross-correlating the object with the point responses, yield similar images, with the similar qualities and resolution as the images of conventional lens-based imaging systems.
Following the initial I-COACH, other versions [7,10–13] have been developed in order to improve various features of imaging. Among the advanced I-COACH systems published in 2017 there is one [10] that can image 3D scene with only annular aperture rather than with the complete circular aperture, commonly used as a standard of most imaging systems. More importantly, the imaging through the annular aperture resolves small details of the observed scene similarly to the full disk aperture. Annular aperture is a special case of a more general partial aperture and hence the system has been termed as partial aperture imaging system (PAIS). In this study, we propose to implement the annular aperture in medical imaging instruments, such as endoscopes for imaging of internal cavities of the human body (i.e. gastrointestinal tracts, colon, uterus and stomach). We term the proposed system endoscopic I-COACH (EI-COACH). The PAIS system [10] and the proposed EI-COACH both use annular aperture, but PAIS is developed for far field imaging and EI-COACH is developed for near-field imaging. Another major difference between PAIS and EI-COACH is the nature of the PSH. The PSH of PAIS is continuous chaotic pattern, whereas the PSH of EI-COACH is a sparse collection of randomly distributed dots. This advanced PSH makes EI-COACH more power efficient system with higher signal-to-noise ratio (SNR) than PAIS.
EI-COACH system is, to the best of our knowledge, the first attempt to design a practical endoscope with a partial aperture, but with performance as close as possible to that of a full aperture system with the same external diameter. The main benefit of this system is that part of the aperture originally used for imaging can be now used for other purposes and can be occupied by other devices. However, the system also has limitations that should be overcome in a future research. In order to avoid from a strong background noise, two camera shots with two independent CPMs should be taken for every image. The method is thoroughly described in Section 2 and the experimental results are presented in Section 3.
2. Methodology
The operation principles of EI-COACH are described in this section with the use of an optical setup that imitates the endoscopic setting. The optical configuration simulating the EI-COACH is shown in Fig. 2(a). The annular CPM is generated in the computer and displayed on the SLM. Since we do not have an annular SLM and we investigate annular aperture of a changeable width, the SLM used in our experiments is a conventional rectangle matrix of M×N pixels that just simulates the annular SLM. The internal area of the SLM surrounded by the annular CPM is used to divert unwanted light by displaying diffractive optical element (DOE). The DOE contains the quadratic phase function with linear phase function both focus the unwanted light away from the sensor. Therefore, only the light passing through the annular CPM arrives at the sensor. The size ratio between the areas of the DOE and CPM is changed in order to test the system performance under various widths of the annular CPM.
In the calibration stage of the system, light emitted from a point source is modulated by the SLM which comprises pseudorandom annular CPM and a diffractive lens of focal length f. The light scattered from the annular CPM is projected on the sensor plane by the annular diffractive lens to satisfy the Fourier-transform relations between the CPM and the sensor planes. If the CPM is removed from the optical configuration, the system becomes a direct imaging system with a diffractive lens and annular aperture with constant phase. The resultant images of any object obtained directly on the camera from this direct imaging system are used as a reference in the comparison with the results of the proposed EI-COACH system.
The light source is quasi-monochromatic, spatially incoherent and the object is illuminated in the mode of critical illumination. Hence, the system is treated as a spatially incoherent system with linear space-invariant relations between the 2D intensity patterns on the object and on the camera planes. Formally, for a point spread function (PSF) IPSF and object intensity distribution O, The intensity on the camera plane is IC=ν[zs/zh](O⁎IPSF), where ‘⁎’ indicates convolution,$\nu$ is a scaling operator defined by ν[a]f(x)= f(ax), zh and zs are the distances of SLM-camera and of object plane-SLM, respectively. In the case of a single lens imaging of a point source, the PSF of the system recorded by the camera is the squared magnitude of the scaled Fourier transform of the lens aperture [14]. In the case of Fig. 2, the lens aperture is the annular CPM. Hence, IPSF=|ν[1/zh]ℱ{C(r)}|2, where, ℱ{˙} indicates two-dimensional Fourier transform, r=(x,y) and C(r) is the transmission function of the annular CPM. The image of the object is reconstructed by a decorrelation from the OH, and in order to minimize the reconstruction noise various techniques are applied. First, to avoid background level over the reconstruction plane, we generate bipolar holograms for both the PSH and the OH. This is done by recording, in each case, two independent intensity distributions and considering each hologram as the difference between these distributions. The independence between the intensity distributions is achieved by using two independent CPMs, C1(r) and C2(r). Therefore, the PSH is given by hPSH=IPSF,1-IPSF,2, where IPSF,k=|ν[1/zh]ℱ{Ck(r)}|2 and k=1,2. The OH is given by,
Each of the two annular CPMs is synthesized in the computer using a modified version of the Gerchberg-Saxton algorithm (GSA) [16], shown schematically in Fig. 3. In GSA, an initial ring-shaped random phase mask is Fourier transformed from the CPM plane to the sensor plane. On the sensor plane, the magnitude distribution is replaced with the chosen pattern of IPSF, whereas the phase distribution remains unchanged. The resulting complex amplitude is inversely Fourier transformed to the CPM plane, and the magnitude distribution is replaced with the ring-shaped aperture. This iterative process continues till the generated intensity profile converges to satisfy the constraints.
The last item in this section is the shape of the intensity response IPSF recoded by the camera for a point object in the input. In direct imaging, a point image is recorded by the camera for any point object, and hence ideally the entire input power is concentrated in the point image that is assumed to be of the size of a single camera pixel. On the other hand, in the original I-COACH [6], the response to a point is a chaotic intensity pattern of M×M camera pixels. Therefore, in average the intensity per pixel is lower by M 2 than the response of direct imaging. Since the electronic noise of the camera per pixel is independent of the imaging method, the SNR of the camera in I-COACH is also lower by M 2. To increase the SNR of I-COACH the sparse I-COACH (SI-COACH) [12] has been recently proposed, in which the intensity response IPSF is a pattern of K<<M sparse randomly distributed dots. In the case of SI-COACH the camera SNR is lower by a factor of K relative to the direct imaging. If the camera’s SNR is the only important parameter, then it be wise to minimize K. However, minimizing K decreases the complexity of IPSF and as a consequence increases the background noise on the image reconstruction plane. In other words, when K is reduced toward 1 the ratio between the maximum peak to the maximum of the sidelobe in the autocorrelation function hPSH⊗hPSH is reduced, and hence the level of the background noise is increased. Therefore, we experimentally search for the optimal K in order to get minimum of noise in both the camera and the reconstructed image. The dots spread over a constraint area on the image sensor. Since EI-COACH is developed to integrate it with other devices and tools, only about 4% of camera area at the center are used as the constraint area of the dots.
3. Experiments
The experimental study of EI-COACH was carried out using the setup shown schematically in Fig. 4. For recording the intensity response of a point object, a 15$\mu m$ pinhole was illuminated by a HeNe laser (λ=632.8 nm). Although the EI-COACH is an incoherent imaging system, the light beyond the pinhole is spatially coherent under the assumption of a point-like pinhole. Therefore, if the wavelength of the HeNe laser and the incoherent source are close, it is legitimate to use the laser for the PSH only, and to benefit from its power superiority over the incoherent source. Light diffracted from the pinhole was polarized to the active orientation of the SLM (Holoeye PLUTO, 1920 × 1080 pixels, 8 µm pixel pitch, phase-only modulation). The phase pattern displayed on the SLM was obtained by modulo-2π phase addition of the CPM with the diffractive lens of f = 15cm focal length. Since only 1080×1080 pixels of the SLM are used to display the CPM, the maximum CPM diameter is 1080×8µm=8.64mm. A beamsplitter was used to reflect back the modulated light coming from the SLM toward a digital camera (PCO.Edge 5.5 CMOS, pixel pitch=6.5$\mu m$, 2560×2160 pixel). The camera was at 29cm away from the center of beamsplitter and the gap between the center of beamsplitter (Thorlabs BS013 50:50 Non-Polarizing Beamsplitter Cube, 400 - 700 nm, 1”) and the SLM was 3cm. Therefore, the distance between the SLM and the camera was Zh=32cm. The MATLAB software was used for simulation and for processing the data recorded by the camera.
For the object hologram, the optical setup was the same as for the PSH, but the object was illuminated by a LED in order to maintain the spatially incoherence needed for appropriate operation of EI-COACH. A LED (Thorlabs LED 635L, 170mW,$\; {\lambda _C} = 635nm,\; {\Delta }\lambda = 15nm$) was mounted at a distance of 11cm from lens L (d=2.5 cm, fo=7cm) and critically illuminated the object. Unlike previous experiments of PAIS [10,13], the targets in these experiments are light reflective and hence the illumination mode is similar to the case of typical endoscopes. 3rd group of USAF resolution chart was used as an object in this study. Bipolar OH was recorded by following the same procedure of recoding the bipolar PSH. The diffractive lens was displayed on the SLM along with the CPM and the target was at a distance of zs=28cm from the SLM. Therefore, the NA and minimal resolved size were ∼0.0154 and (0.61$\lambda /NA$) ∼25$\mu m$, respectively.
Regarding the working distance of zs=28cm which is longer than of the typical commercial endoscopes, it should be noted that the proposed system is a proof of concept and it is compared with conventional imaging system under the same conditions and with the same distances between the various components. The main principle that the system must satisfy is imaging of the object on the camera in absence of the coded aperture mask. Under this condition, there are several ways to shorten the working distance from the SLM to the target. One option is to display on the SLM a diffractive lens with a shorter focal length. The minimum focal length is f = Dp/λ, where D is the diameter of the lens (SLM) and p is the pixel size. For the current SLM (p=8μm, D=8.64 mm) and λ=635 nm, f=10.9 cm. Assuming D is dictated by the size of the examined cavity, the options to shorten f are to use a longer wavelength or to use SLM with smaller pixel size. For a given wavelength and pixel size, there is still the option of adding sequentially several annular lenses, such that the overall imaging has shorter working distance. These kind of improvements will be probably explored in the future.
We started the experiment by looking for the optimal number of dots in the intensity response IPSF, whereas the visibility of the reconstructed images of a full aperture I-COACH system has been used as the figure of merit. The number of dots from each CPM was varying between 5 to 20 dots in the interval of 5 dots and between 20 to 70 dots in the interval of 10 dots. Note that the number of total dots of each PSH is double the number of dots created from each CPM.
Visibility of the 6th element of the 3rd group was used as the figure of merit for choosing the optimal number of dots. Figures 5(a) and 5(b) shows two phase masks (CPM × diffractive lens), each of which was synthesized to yield an intensity pattern of 5 dots. The central portions of the two intensity responses IPSF, which composite the PSH, appear in Figs. 5(c) and 5(d). Their corresponding object responses are shown in Figs. 5(e) and 5(f). The bipolar PSH and OH are shown in Figs. 5(g) and 5(h), respectively. Reconstruction result generated by cross-correlation between PSH and OH is shown in Fig. 5(i), whereas Fig. 5(j) shows the direct imaging result obtained by displaying only the diffractive lens on the SLM. From the intensity plots of the horizontal and vertical gratings of the 6th element shown in Fig. 5(i), in comparison to the direct imaging plots shown in Fig. 5(j), the visibility values of EI-COACH intensity curves are higher than of the direct imaging curves. Results of the reconstructions for different dot numbers, with the average intensity cross-section of the 6th element, are shown in Fig. 6. Figure 7 shows the visibility graph versus the dot numbers. Apparently, the reconstruction results with 10 dots gives the best visibility over other reconstruction results. Therefore, a 10-dot pattern is chosen for the rest of the experiments with the EI-COACH systems.
Fig. 5. Reconstruction results of I-COACH (full aperture) system with 5 dots. (a-b) Central part of CPMs with the diffractive lens, (c-d) The two parts of the PSH, (e-f) The two parts of the object hologram, (g) Bipolar PSH, (h) Bipolar Object hologram, (i) I-COACH reconstruction with average intensity plots of the 6th element, (j) Direct imaging result with average intensity plots of the 6th element.
Fig. 6. Reconstruction results by different dot number along with average visibility plot of the horizontal and vertical gratings of 6th element shown in the red boxes. The numbers in red color in each box represent the number of sparse dots in each PSH.
The MTF of I-COACH system is theoretically calculated by Eq. (3) based on the computed two CPMs. For verification purposes, we also calculated the experimental MTF, equal to the magnitude of the Fourier transform of the measured PSH. In case of incoherent direct imaging, the MTF is the magnitude of the autocorrelation of the aperture function. To calculate the experimental MTF of direct imaging, the impulse response is experimentally recorded and Fourier transformed, where the MTF is the magnitude of this Fourier transform. Same techniques are further used to calculate MTFs of the entire annular aperture-based imaging systems. The simulation and experimental results of MTF curves of direct imaging with various annular aperture sizes are shown in Figs. 8(a) and 8(b), respectively.
Fig. 8. MTF curves of direct imaging system with varying annular aperture. (a) Simulation results, (b) Experimental results
As mentioned above, reconstruction with 10 dots yields the best visibility in comparison to other dot numbers. Therefore, the 10 dots pattern was chosen for the experiments with EI-COACH systems. Annular ring CPMs with width of 100-400 pixels, in a step of 100 pixels were synthesized by the GSA and tested. MTF plots of EI-COACH system with various ring size were calculated and compared with the MTF of the direct imaging system with the same annular apertures. MTF plots of EI-COACH with different ring aperture calculated by theoretical and experimental methods are shown in Figs. 9(a) and 9(b), respectively. They demonstrate that with the same annular aperture on average, EI-COACH suppresses the signals in the higher-than-zero spatial frequencies less than the direct imaging.
Fig. 9. MTF curves of EI-COACH with different annular ring size in comparison to direct imaging with the same annular apertures. (a) Simulation results, (b) Experimental results
For additional comparison between the methods, structural similarity index (SSIM) has been used [17]. SSIM is a reference based relative comparison method, hence the image obtained from full-aperture direct imaging is used as the reference. Reconstruction results of the annular aperture EI-COACH and direct imaging are compared with full-aperture direct imaging results. Because the image plane contained unwanted background noise, DC noise was removed from all the direct imaging results before calculating the SSIM index given by,
Fig. 13. Images of a dice with full aperture and with 300-pixel annular aperture by direct and EI-COACH system
Among the tested cases, the annular aperture of 100-pixel width leaves maximum internal area for the non-imaging devices and its values of the structure similarity and visibility are reasonably accepted. Therefore, we extend the investigation in the EI-COACH system with the ring width of 100 pixels. First, it is verified that the number of dots for optimal reconstruction results is indeed 10 dots. Visibility and SSIM index of reconstructed images from 5 to 50 sparse dots are experimentally checked. Reconstruction results up to 19 dots are shown in Fig. 14. Visibility and SSIM plots of all reconstruction results are shown in Figs. 15 and 16, respectively. The SNR is also calculated for 100-pixel annular width EI-COACH system and the product of SNR with visibility is used to decide optimal number of dots. Figures 17 and 18 show the plots of SNR and product of SNR with visibility in 100-pixel EI-COACH system, respectively. Though SSIM index shows that reconstruction result of 10 dots has lower structure similarity, it has highest SNR value and product of SNR and visibility.
Fig. 14. Reconstruction results of EI-COACH of 100-pixel annular ring aperture with different number of sparse dot pattern
Fig. 18. Product of SNR and visibility for different number of dot pattern in 100-pixel EI-COACH system
4. Summary and conclusion
EI-COACH system has been investigated in this study. The annular aperture of the system with the outer radius of R gives a resolution performance similar to a full disk aperture with the same outer radius R. This high-resolution feature is achieved due to the unique MTF with relatively high transparency in the non-zero frequencies. In order to improve the power efficiency and the SNR obtained with the previous I-COACH systems, we have synthesized the CPMs such that the system PSF is a pattern of sparse dots. The penalty of using a response of dots is the loss of the capability to image 3D scene with a single CPM. This capability remains as a challenge for future research of advanced versions of EI-COACH systems.
Funding
European Commission (777222) for ATTRACT project; Israel Science Foundation (1669/16); Israel Ministry of Science and Technology.
Acknowledgments
The authors thank Mr. Mani R. Rai for fruitful discussions.
Disclosures
The authors declare no conflicts of interest.
References
1. H. Zeng, Diagnostic Endoscopy (CRC Press, 2013).
2. A. G. Barutcu, D. Klein, M. Kilian, M. Biebl, R. Raakow, J. Pratschke, and J. Raakow, “Long-term follow-up after single-incision laparoscopic surgery,” Surg. Endosc., 126–132 (2019).
3. B. S. Peters, P. R. Armijo, C. Krause, S. A. Choudhury, and D. Oleynikov, “Review of emerging surgical robotic technology,” Surg. Endosc. 32(4), 1636–1655 (2018). [CrossRef]
4. H.-S. Chen and Y.-H. Lin, “An endoscopic system adopting a liquid crystal lens with an electrically tunable depth-of-field,” Opt. Express 21(15), 18079–18088 (2013). [CrossRef]
5. E. Kolenovic, W. Osten, R. Klattenhoff, S. Lai, C. von Kopylow, and W. Jüptner, “Miniaturized digital holography sensor for distal three-dimensional endoscopy,” Appl. Opt. 42(25), 5167–5172 (2003). [CrossRef]
6. A. Vijayakumar and J. Rosen, “Interferenceless coded aperture correlation holography–a new technique for recording incoherent digital holograms without two- wave interference,” Opt. Express 25(12), 13883–13896 (2017). [CrossRef]
7. J. Rosen, V. Anand, M. R. Rai, S. Mukherjee, and A. Bulbul, “Review of 3D imaging by coded aperture correlation holography (COACH),” Appl. Sci. 9(3), 605 (2019). [CrossRef]
8. W. Chi and N. George, “Optical imaging with phase-coded aperture,” Opt. Express 19(5), 4294–4300 (2011). [CrossRef]
9. W. Chi and N. George, “Phase-coded aperture for optical imaging,” Opt. Commun. 282(11), 2110–2117 (2009). [CrossRef]
10. A. Bulbul, A. Vijayakumar, and J. Rosen, “Partial aperture imaging by systems with annular phase coded masks,” Opt. Express 25(26), 33315–33329 (2017). [CrossRef]
11. S. Mukherjee, A. Vijayakumar, M. Kumar, and J. Rosen, “3D imaging through scatterers with interferenceless optical system,” Sci. Rep. 8(1), 1134 (2018). [CrossRef]
12. M. R. Rai and J. Rosen, “Noise suppression by controlling the sparsity of the point spread function in interferenceless coded aperture correlation holography (I-COACH),” Opt. Express 27(17), 24311–24323 (2019). [CrossRef]
13. A. Bulbul, A. Vijayakumar, and J. Rosen, “Superresolution far-field imaging by coded phase reflectors distributed only along the boundary of synthetic apertures,” Optica 5(12), 1607–1616 (2018). [CrossRef]
14. J. W. Goodman, Introduction to Fourier Optics (Roberts and Company, 2005).
15. J. L. Horner and P. D. Gianino, “Phase-only matched filtering,” Appl. Opt. 23(6), 812–816 (1984). [CrossRef]
16. R. W. Gerchberg and W. O. Saxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik (Stuttg) 35(2), 237–246 (1972).
17. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process. 13(4), 600–612 (2004). [CrossRef]
18. M. R. Rai, A. Vijayakumar, and J. Rosen, “Non-linear adaptive three-dimensional imaging with interferenceless coded aperture correlation holography (I-COACH),” Opt. Express 26(14), 18143–18154 (2018). [CrossRef]
