Dynamic biological systems present challenges to existing three-dimensional (3D) optical microscopes because of their continuous temporal and spatial changes. Most techniques are rigid in adapting the acquisition parameters over time, as in confocal microscopy, where a laser beam is sequentially scanned at a predefined spatial sampling rate and pixel dwell time. Such lack of tunability forces a user to provide scan parameters, which may not be optimal, based on the best assumption before an acquisition starts. Here, we developed volumetric Lissajous confocal microscopy to achieve unsurpassed 3D scanning speed with a tunable sampling rate. The system combines an acoustic liquid lens for continuous axial focus translation with a resonant scanning mirror. Accordingly, the excitation beam follows a dynamic Lissajous trajectory enabling sub-millisecond acquisitions of image series containing 3D information at a sub-Nyquist sampling rate. By temporal accumulation and/or advanced interpolation algorithms, the volumetric imaging rate is selectable using a post-processing step at the desired spatiotemporal resolution for events of interest. We demonstrate multicolor and calcium imaging over volumes of tens of cubic microns with 3D acquisition speeds of 30 Hz and frame rates up to 5 kHz.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Confocal laser scanning microscopy (CLSM) has arguably become the de facto standard for the three-dimensional (3D) characterization of biological processes at the sub-cellular level [1–3]. Based on the raster scanning of a laser beam for illumination, CLSM provides optical sectioning with synchronous multi-channel imaging. In addition, these confocal architectures are compatible with super-resolution microscopy and advanced spectroscopic techniques such as fluorescence correlation spectroscopy (FCS) and fluorescence lifetime imaging [4–9]. However, despite the complete toolset that CLSM represents, a central question in this imaging modality is how to select the main scanning parameters (number of pixels, pixel dwell time and scanned volume) to maximize the spatiotemporal information retrieved from a sample. While the answer is straightforward when characterizing fixed samples or slowly varying processes—we can always use long pixel dwell times and a large number of pixels to attain high signal to noise ratio (SNR) and spatial resolution—problems arise when imaging fast phenomena or rapidly evolving systems. In these time-sensitive cases, one is typically faced with the dilemma of having to sacrifice either spatial or temporal resolution. This is common in most imaging technologies—reducing the number of sampled points increases imaging speed, albeit with a loss of spatial resolution [10,11]—but in CLSM two aspects can further aggravate such a tradeoff. First, current confocal systems are unsuitable for fast 3D scanning. By using resonant mirrors, lateral (X,Y) beam scanning at microsecond timescales can be routinely achieved, but axial focus translation (Z) remains significantly slower [12–14]. Thus, and despite scanning optimization approaches, 3D sampling is unevenly performed—fast along XY, slow along Z— greatly limiting volumetric imaging rates. Secondly, even if the desired sampling rate and optimal tradeoff between imaging speed and resolution can be selected, it must remain fixed for the entirety of an acquisition. In other words, the rigid scanning method of confocal microscopes impedes adapting the scanning parameters as samples evolve over time. As a result, confocal microscopes remain sub-optimal tools to investigate the dynamics of fast processes or multiscale systems that occur over different temporal scales.
Volumetric Lissajous confocal microscopy addresses the limitations encountered with existing CLSM systems. By integrating an ultrasound-driven liquid lens (TAG lens) into a confocal workstation equipped with a resonant mirror, we achieve three-dimensional beam scanning at kHz rates. This high speed comes with the caveat of sub-optimal sampling—the scan trajectories correspond to dynamic Lissajous patterns that sample only a fraction of the voxels within a volume. However, by properly selecting the TAG lens driving frequency, the trajectories can be rendered dynamic, enabling trajectory variations over time. As a result, each volumetric scan samples different voxels, and by accumulating multiple scans, the spatial resolution of the reconstructed 3D image can be improved. This represents a paradigm shift in laser scanning microscopy techniques—information is continuously acquired, and a post-processing step allows the volumetric imaging rate to be selected based on the desired spatial or temporal resolution. In addition, Lissajous scanning is compatible with inpainting algorithms used to fill the missing information due to sub-Nyquist sampling, helping to restore spatial resolution without sacrificing time. We present a detailed description and characterization of the technique and demonstrate its potential for fast 3D live-cell imaging by capturing the calcium signals of neuronal networks from an in vitro brain model.
2. Principle and implementation of volumetric Lissajous confocal microscopy
Our approach uses two harmonically oscillating systems to achieve high-speed 3D scanning. A liquid lens driven by ultrasound (TAG lens) at ∼456 kHz is used for z-focus scanning and a resonant mirror at ∼8 kHz for x-scanning. Volumetric scanning is completed with a galvo mirror system to translate the beam along the y-axis linearly (e.g. at 30 Hz for 512 pixels, which makes it the slow axis in our microscope). As shown in Fig. 1(a), the resulting 3D scan trajectories correspond to Lissajous patterns exhibiting two key features. First, a large increase in laser scanning speed is achieved, up to 45-fold faster for our experimental conditions, compared with conventional scanning methods with a single resonant mirror (see Appendix and Fig. 6). Thus, for the same acquisition time, a higher amount of information from the sample can be retrieved. Even if Lissajous trajectories have been reported in different microscopy techniques, including atomic force microscopy or two-photon microscopy [11,15–18], they have been limited to 2D scans along the xy plane. Instead, our system extends their use into 3D imaging utilizing the superior scanning speed in the z-axis. Secondly, the Lissajous patterns can be adjusted by tuning the TAG lens driving frequency (Fig. 1(b)). Notably, not only can the shape of the trajectory be controlled, but also its temporal characteristics enabling the generation of static or dynamic patterns. The dynamic patterns, as illustrated in Fig. 1(b), can increase the number of sampled voxels over multiple scans, providing a unique method to correlate spatial sampling with imaging rate. Note that the number of Lissajous scans needed to retrieve information from an entire volume is typically between 10 and 100, depending on factors such as z-scanning frequency, the number of lines sampled across the y axis, or the axial range.
Volumetric Lissajous confocal microscopy is moderately simple to implement, in terms of optics and electronics, in any commercial confocal system with a resonant mirror. The main modification involves inserting the TAG lens at the conjugate plane of the back focal plane of the microscope objective lens. In our experiments, we placed the TAG lens and a relay lens system between the tube lens and the scan lens of the microscope, as illustrated in Fig. 1(c). Ideally, to avoid aberrations and/or reduction in the field of view, the TAG lens should be placed before the scanning mirrors, but this position was not physically accessible in our commercial microscope. One key aspect when implementing the Lissajous microscope with the TAG lens is synchronizing the scanned excitation with the detection. In previous studies, this task was typically accomplished with photon-counting detectors combined with fast acquisition cards. However, the relatively high frequency of the TAG lens required electronics significantly faster than those commonly employed. Moreover, due to the short pixel dwell time, the dynamic range in photon flux became limited [19,20]. To ease implementation, we designed a three-step approach that obviates the need for fast electronics, as shown in Fig. 1(d). The Lissajous pattern is first reconstructed, line by line, by taking into account the clock signals of the TAG lens (directly fed into channel 1 of the acquisition card of the microscope) enabling the axial position corresponding to each pixel to be defined. This requires an initial calibration step, as described in Methods. The fluorescence signal of the sample, simultaneously detected in channel 2, is then analyzed, and the axial information intrinsically contained for each pixel is decoded. By sorting the photons into an arbitrary number of optical sections (Nyquist criterion and size of the PSF to be considered), a 3D image can be reconstructed.
3.1 Optical performance of the microscope
Initially, we characterized the optical performance of the Lissajous microscope by imaging 100 nm fluorescent nanospheres volumetrically dispersed in 1% agarose gel using a 40x (1.15 NA) objective lens. To address any loss of spatial resolution caused by sub-optimal sampling and obtain a high SNR, we averaged the Lissajous trajectories over 250 scans. The sample information was continuously acquired, and a z-stack containing 32 optical sections (this number can be arbitrarily selected) was computed using a post-processing step. Figure 2(a) shows the reconstructed 3D image, in which the axial position (depth) of the spheres has been color-coded. The nanospheres can be distinguished within an axial range (Δz) of 70 µm, about 30 times the native depth of field of the objective. The axial range depends on several factors such as the optical magnification of the system, the driving frequency and voltage of the TAG lens, and the objective lens used [21,22]. For example, a 20x (0.8 NA) objective lens realized a 200 µm axial range at the same imaging conditions as the 40x objective (Fig. 7). By benchmarking the image against those obtained with a traditional piezo z-stage (Fig. 2(b)) we can validate the suitability of Lissajous scans and the reconstruction process for optical microscopy. There is an excellent match between the two images with only small discrepancies at the bottom and the top of the axial range, where the PSF of the Lissajous scan is longer, as to be expected from aberrations induced by the TAG lens at these positions . Even if we did not perform a full analysis of photobleaching, we did not observe significant differences between the two methods. In fact, the shortest pixel dwell time of the Lissajous scanning and the effective “stroboscopic illumination” can be mitigating factors for photobleaching . It is also worth noting that, in current experiments, we did only compensate for the not uniform sampling density caused by the Lissajous scanning along the x axis, but we could have extended this feature to the z axis by using a non-linear pixel clock. While it could result in an increase in the intensity of the optical sections located at the axial extrema of the scanned volume, increased aberrations at these points partially compensated for this phenomenon .
A more quantitative analysis of the spatial resolution of our microscope can be obtained by measuring its point spread function (PSF) at different focal planes using the nanospheres, as illustrated in Fig. 2(c). For all planes the shape of the PSF is straight and symmetrical along the optical axis, with only small distortions close to the extrema of the axial range. This trend is also seen in the intensity line profiles of the PSFs (Fig. 2(d)). Indeed, the lateral and axial full width at the half maximum (FWHM) of the PSF remain practically constant within ± 20 µm of the native plane of the objective but becomes larger beyond this range (Fig. 2(e)). Note the clear asymmetry in spatial resolution between the x-y and z axis — while the average FWHM of the lateral PSF is about 0.4 µm, in agreement with the 0.37 µm measured with the standard piezo, the axial PSF is about 4 times longer, 8 µm vs 2.2 µm (see Appendix and Fig. 8). The effective elongation of the PSF along the z-axis is not a fundamental limit of the technique, but rather a drawback of our simplified implementation. Since the electronic card of our commercial microscope has a limited number of time channels, for each x-scan, the collected photons can only be sorted into 1024 time windows or pixels. Although, in a regular confocal microscope, it is sufficient to fully reconstruct a 2D image, in our case the focus is continuously scanned along the z-axis and each time window can contain information of multiple focal planes depending on the axial range or frequency of the TAG (fTAG) lens (Fig. 8(a)). Therefore, an increased number of time windows could provide the optimal pixel sampling. For instance, the conditions used here (Δz=70 µm, fTAG = 457 kHz, and 40 ns time window) resulted in the merging of information of up to 4 µm along the z axis in a single pixel. For Δz = 18 µm, the integration of axial data is reduced to 1 µm, thus explaining the increase in z-resolution experimentally observed in this case (Fig. 8(b)). Nevertheless, the optical performance of our microscope could be easily improved by decreasing the number of extrema per x-line, i.e. by employing a smaller fTAG value (Fig. 9) or by increasing the number of time windows using one of several commercial microscopes with resonant mirror systems capable of up to 4000 pixels per line.
Notably, our Lissajous confocal microscope preserves the core advantages of confocal microscopes, including the possibility of simultaneous multi-color imaging. Figure 3 shows images of a murine brain slice captured with our system after integration over 30 scans. In this case, neurons, blood vessels and nuclei were labeled with three different dyes. The sample was simultaneously irradiated with three lasers and the corresponding information was captured with three separate point detectors. The reconstructed sections at different focal planes, over an axial range of 70 µm, and the corresponding axial projection image based on a smooth manifold extract  and 3D rendered volumes reveal an overall good image quality. The different brain components are clearly distinguished with negligible crosstalk between the channels (see Visualization 1). These results also indicate that chromatic aberration effects are not significant within the used wavelength range (500–700 nm), in agreement with previous works on TAG-enabled systems .
3.2 Instant 3D imaging with tunable spatiotemporal resolution
The key features of our microscope are 3D beam scanning at high speed and sub-Nyquist sampling, together with time-varying spatial resolution. These two aspects are highlighted in Fig. 4, where 3D images of neurons from a murine brain slice rendered by our Lissajous microscope are compared with a conventional raster scanning/piezo stage microscope. Only one 3D-scan (one frame) is needed for the Lissajous image to obtain volumetric information, with neurons clearly discernable across the entire 3D imaging space. This result is in striking contrast to the image obtained with the typical confocal system—even if the same acquisition time and laser intensity were used, only one optical section is captured. The restriction for a single acquisition is that our image does have a lower SNR, and the neuronal networks appear to be disconnected due to the sparse sampling. Remarkably, however, by accumulating the number of scans the Lissajous images exhibit an increased spatial sampling and, consequently, a higher spatial resolution and SNR. Indeed, after accumulating 5 scans, more neuronal processes become recognizable, and after 10 scans, the neuronal networks extending in the 3D space become entirely traceable. Integration over 30 scans results in images where all points have been sampled. The temporal evolution of the volumetric information retrieved with the Lissajous microscope is drastically different from that observed in a conventional confocal system. Conventional methods require sequentially capturing multiple focal planes, a time-consuming task due to the need for mechanical z-focus translation. As such, the acquisition is directional (from bottom to top in current experiment), and after 30 scans (or frames) only half the volume of interest has been imaged, with the remaining volume being totally unknown. This comparison between the two microscope modalities manifests the unprecedented flexibility of Lissajous scanning in terms of selecting the acquisition parameters. As 3D structural information is immediately accessed and the accumulation of multiple scans enhances spatial resolution, scanning parameters do not need to be selected before launching the acquisition, and the required spatiotemporal resolution can be selected after. Note that, in the current implementation, the axial resolution of the Lissajous trajectory is degraded, but diffraction-limited axial resolution can be attained by increasing the number of time windows.
3.3 Fast volumetric imaging of calcium transients
Traditional confocal microscopes face several issues when monitoring the dynamics of fast events. In essence, their 3D imaging speed might not be sufficient to characterize the process under study, it can be challenging to determine the optimal tradeoff between spatial and temporal resolution and, even if such optimal conditions are found, they may change over time. In these instances, volumetric Lissajous confocal microscopy can offer a competitive advantage. To prove this point, we recorded the calcium activity of murine neuronal networks from a 3D neuron culture resembling an in vitro brain tissue model. We stained the cells with the fluorophore Fluo-4 AM, which is sensitive to local calcium concentrations (see Methods section). As calcium events have a typical duration of milliseconds and propagate through complex networks distributed across the three dimensions of space, they are incredibly challenging to image in commercial confocal systems. Figure 5 shows images and data obtained from a 60 × 30 × 50 µm3 volume of the neuronal culture over a total of 35 seconds. Each 3D scan took 33 ms, thus capturing a total of 1050 scans. As expected, integrating the information retrieved over multiple scans leads to higher spatial sampling and thus higher resolution (Fig. 5(a), top row), albeit with an inevitable loss of temporal resolution. Interestingly, it is possible to bypass this problem by using advanced image processing algorithms. Specifically, the information from the non-sampled voxels can be considered as an inpainting problem, where images that exhibit lost or deteriorated parts are reconstructed. Several inpainting methods are currently available, based either on machine learning  or compressed sensing , among others. In these experiments, we used an iterative model-based inpainting algorithm already used in microscopy [16,29]. As presented in Fig. 5(a), bottom row, the inpainted image after 1 single scan shows the soma of a neuron and several additional structures distributed over the entire scanned volume with high contrast (arrows in the Fig. image). Note that from the image counterpart prior to inpainting, the soma of the neuron is barely distinguishable. After 3 scans, the inpainted image shows most of the structures that were only visible after averaging 100 scans (a ground-truth image, Fig. 5(b)). After 5 scans, structural details in the image appeared barely distinguishable when compared to images formed with 100 scans, including the vertical dark lines indicated by arrowheads. To obtain similar results without inpainting, it is necessary to integrate more than 10 scans. Therefore, inpainting offers a feasible route to enhance image quality without sacrificing temporal resolution (Fig. 10).
In the study of neuronal networks, the dynamics of calcium transients, i.e. local variations in calcium flux which lead to important cellular functions for neuronal development and physiology, are analyzed at different regions over a large volume . Figure 5(c) shows intensity plots (ΔF/F) at four regions of interests (ROIs) at different z-positions (note the ROIs in Fig. 5(b))—the peaks and subsequent decays indicate typical variations of local calcium concentrations caused by neuronal signal transmissions. The magnified plots at two time points show that the system time resolution was high enough to discriminate that the fluorescence signal at ROI2 started increasing just before it rose at ROI1. These temporal variations of the calcium concentrations are visualized in Fig. 5(d) where three images acquired at 33 milliseconds intervals show that the signal intensity increases in different regions at different time points. At t1 = 0 ms, the neuron exhibited relatively low signal and at t2 = 33 ms the fluorescence signal increased at the region indicated with an arrow, and at t3 = 66 ms the signal increased at a dendrite-like structure (arrowhead). Such signal variations at different time points were resolved over the entire 3D space (Visualization 2 without inpainting but 10 scan accumulations, Visualization 3 with inpainting), and this indicates that Lissajous confocal microscopy may be a powerful tool in the study of functional connectivity of neuronal circuits.
3.4 Ultrafast x-z imaging
Our volumetric Lissajous confocal microscope has the particular feature of having the slow scanning oriented along the y-axis. Even if it were possible to add a resonant mirror for this axis, the expected gain in scanning speed would significantly increase the systems complexity due to the additional synchronization and faster electronics needed. Alternatively, our current configuration reduces 3D scan time by limiting the travel range of the galvo mirror by having fewer pixels along the y-direction. This original method of scanning could find applications in imaging flow cytometry , particle imaging velocimetry  or other instances where fast x-z imaging is preferred. As an extreme example, we imaged an object moving along the z-direction with a static galvo mirror, acquiring 2D sections across the x-z plane. Here, the x-z scan was completed in only 62 µs. Figure 11 shows the x-z images obtained of an oscillating mirror characterized at 5300 and 1600 frames per second. Such ultrafast speed enables object displacements at 1.7 mm/s to be distinguished. As such, the initial acceleration of the object, the posterior advance at constant speed, and the final deceleration can be temporally resolved. Arguably, the lower SNR of normal fluorescence samples would impede the use of this speed for bio-imaging. In this case, temporal accumulation would be required, and consequently, sample drift or sample movements could be an issue. This is a common problem in all 3D microscopes. Still, continuous advances toward more sensitive detectors and brighter dyes are increasing the photon budget of optical microscopes, rendering methods for fast beam scanning more relevant than ever .
Combining an ultrasound varifocal lens with a resonant mirror leads to a novel type of confocal microscope where the laser beam is scanned following dynamic Lissajous trajectories across a volume. Continuous interrogation of the specimen at sub-microsecond time scales and sub-Nyquist sampling enables minimizing the light dose exposure and optimizing the data content based on application. In this way, instant or accumulated observation of large volumes can be achieved in a post-processing step, resulting in 3D images with a user-selectable spatiotemporal resolution. Such control enables unique features, such as adaptive integration time, namely dynamically selecting the optimal trade-off between temporal resolution and spatial resolution/SNR. Additionally, the system preserves most of the key features of conventional CLSM, including high-speed detection, which makes it suitable for FCS over multiple points .
Optical imaging techniques capable of capturing continuous changes of living specimen at sub-cellular resolution have been a long-term quest in science. Several strategies have been successfully developed, but they typically involve a tradeoff between spatial or temporal resolution. As our results demonstrate, volumetric Lissajous confocal microscopy provides a unique flexibility for selecting, at any given instance, the optimal conditions for characterizing biological phenomena. The new microscope opens the door to monitoring rapidly evolving processes or events that occur over temporally or spatially varying scales. We anticipate that coupling the technique with parallelized detection methods  or nonlinear excitation [19,35] will lead to improved signal-to-noise ratio, spatial resolution or penetration depth, helping to reconstruct complex 3D phenomena with maximal detail.
5.1 Implementation of the volumetric Lissajous confocal microscope
All experiments used a commercial confocal microscope equipped with a resonant scanning mirror (Nikon A1R, Nikon Instruments, Japan), four excitation continuous-wave lasers (405 nm, 488 nm, 561 nm, 640 nm), and four photomultiplier tubes (two normal and two GaAsP PMTs). The typical resonant frequency of the resonant mirror was 7929 Hz. In all experiments, the confocal pinhole was set to 1.5 Airy units (in the case of multi-color excitation, the Airy unit was calculated for the shortest excitation wavelength) and the pixel size was selected to be smaller than the Nyquist sampling rate for each objective lens. Unless otherwise specified, the resonant scan direction was bi-directional. The image size along the x-direction was always set to 1024 pixels (the maximum available for the current system) and the pixel along y-direction was typically 512 pixels but modified for each experiment. To implement the fast axial scanning in this commercial system, we detached the scan head from the microscope body and inserted a pair of relay lenses (f = 60 mm and f = 75 mm at fTAG = 457 kHz (Thorlabs, Inc., USA), and f = 200 mm and f = 200mm at fTAG = 142 kHz (Nikon Instruments, Japan)) between the tube lens and scan lens. Then, we placed a TAG lens (TAG lens 2.0, TAG Optics Inc., USA) between the relay lenses at a conjugate plane of both the scanning mirrors and the back focal plane of the objective lens. The TAG lens served as a resonant axial scanner that, unless indicated, was driven at a frequency of 457 kHz using the driving kit provided by TAG Optics. To avoid aberrations caused by the Bessel-like refractive index profile of the TAG lens [35,36], the entrance pupil of the TAG lens was physically blocked with a 1.7 mm aperture. Experiments were performed using either a 40x water immersion (Apo LWD 40x WI λS DIC N2, NA 1.15, Nikon Instruments, Japan), a 20x air (Plan Apo VC 20x DIC N2, NA 0.75, Nikon Instruments, Japan), or 60x water (Plan Apo IR 60x WI DIC N2, NA 1.27, Nikon Instruments, Japan) objective lens. The limited size of the TAG lens aperture reduced the beam diameter of the excitation laser, and the filling factor at the back aperture of the objective lens was approx. 60% for 40x objective and 45% for 20x objective lens. This problem could have been solved by placing the TAG lens before the scan head and expanding the beam diameter [19,35]. However, in our current commercial microscope, this was not possible. The magnification factor caused by the difference of focal length between the relay lenses caused a smaller field of view, with the pixel size calibrated accordingly. When the TAG lens is off, the microscope operates as a standard confocal system. In this case, we used a piezo scanner (P-736 ZRN, Physik Instrumente, Germany) for axial sample translation. For the experiments with 40x objective lens and static specimen, the applied laser power, measured at the objective lens position, was a few tens of microwatts in all wavelength for Lissajous scan, ranging from 1 to 3 times higher than the standard piezo scan. For the calcium dynamics imaging, the applied laser power was 50 µW. For the experiment with 20x objective lens, the laser power was 140 µW, which was approx. 3 times higher than the standard piezo scan.
5.2 Image acquisition and processing
To reconstruct an image, it is necessary to know the x-y-z position of the focused beam at any time instance. This typically requires a fast acquisition card. Instead, we designed a scheme that obviates the need for fast electronics and can be used with the traditional acquisition card of a commercial confocal microscope. We sent the clock signal generated by the TAG lens driver (adjusted to indicate when the induced refractive index was at a minimum) to the input of the electronic board of one of the PMT channels (channel 1), located in the microscope controller box of the Nikon microscope. The commercial software of the microscope generated a black image with some bright pixels corresponding to the TAG lens clock signal (named “reference image” hereafter). Note that such an image corresponded to a Lissajous trajectory, in which each pixel has an associated unknown z-position. We then proceeded to reconstruct the Lissajous trajectory, thus decoding the relationship between the pixel and z-position, using a custom-written Matlab code. For each line, a Lissajous pattern was fitted to match its minima positions to each peak pixel (TAG clock signal) in the reference image (Fig. 1(d)). Once the Lissajous trajectory was reconstructed, the same code enabled sorting the information into a z-stack with an arbitrary number of axial planes. Importantly, the axial scan range produced by the TAG lens depends on the driving voltage amplitude, frequency or microscope objective used . This value was obtained before starting each experiment by recording the reflected light from a mirror as it was axially translated.
Once the z-stack was retrieved, we processed the images using ImageJ. To improve visualization, we adjusted image contrast, and in some instances, we used a Bandpass Filter for removing the vertical lines and a Gaussian Blur for removing noise. Quantitative data analysis was performed on raw images without post-processing. For the measurement of FWHM values of the nanospheres, Gaussian fitting was used on the intensity line profile of each nanosphere. Image inpainting was performed using an existing open-source Matlab library (Plug and Play priors, available at https://engineering.purdue.edu/∼bouman/Plug-and-Play/). Briefly, the inpainting algorithm applies two-dimensional Shepard interpolation and the alternating direction method of multipliers.
5.3 Sample preparation
3D beads sample
The sample consisting of 3D fluorescence nanospheres was prepared by diluting yellow-green 100 nm fluorescent beads (FluoSpheres Carboxylate-Modified Microspheres, 0.1 µm F8803, ThermoFisher Scientific, USA) into water at a concentration of 3 × 109 particles/mL. The solution was mixed with 1% agarose solution (Invitrogen Ultrapure Low Melting Point Agarose, Invitrogen Corp., USA), and put on a microscope coverglass.
The optically cleared brain slice was prepared following the protocols described in earlier studies [9,37]. Briefly, the tissue was fixed with 4% paraformaldehyde, sectioned to 500 μm, labeled with Alexa Fluor488 wheat germ agglutinin for the blood vessels, Alexa Fluor 633 anti-tyrosine hydroxylase antibody mainly labeling the dopaminergic neurons, and SYTOX Orange labels for the nuclei, and mounted in the RapiClear 1.52 reagent (SunJin lab Co., Taiwan).
3D neuronal culture for calcium imaging
Primary hippocampal neurons from postnatal C57BL/6 J mice (P0), isolated as previously described , were obtained from sacrificed animals respecting the 3R principle, in accordance with the guidelines established by the European Community Council (Directive 2010/63/EU). 3D neuronal cultures in alginate were prepared as already reported [39,40]. Briefly, a cell suspension [107 neurons/mL] in 0.15% (w/v) sodium alginate (Pronova UP LVG, Novamatrix, Norway) was loaded into donut-shaped (3 mm inner diameter) 0.8% agarose (Sigma-Aldrich) molds which were surrounded by Neurobasal medium (Invitrogen) supplemented with 10 mM CaCl2 (Sigma-Aldrich) to induce alginate gelation. After 30 min, the gelation solution was replaced by Neurobasal medium without additional CaCl2. Half of the medium was changed once a week until calcium imaging was performed. For imaging the spatiotemporal dynamics of the calcium activity, after 21 days in vitro (DIVs), 3D neuronal cultures were loaded with the fluorescent calcium indicator Fluo-4 AM (5 µM) (F14201, Thermo Fisher Scientific), in extracellular recording solution (95 mM NaCl, 5 mM KCl, 1.8 mM CaCl2, 0.8 mM MgCl2, 1 mM NaH2P04-2H20, 23 mM NaHCO3, 10 mM HEPES and 10 mM Glucose (Sigma-Aldrich), pH 7.3), for 15 min. Thereafter, the calcium indicator was washed away, and neuron cultures were maintained in the standard recording solution during imaging.
Lissajous trajectory length in xz plane
Combining a resonant mirror (x-axis) with the TAG lens (z-axis) enables translating the laser focus across the xz plane. Because the horizontal scanned range (Δx) is fixed, the overall length of the trajectory depends on the frequency and axial scan range of the TAG lens (Δz). As shown in Fig. 6 (red line), when the TAG lens is switched-off the scan trajectory corresponds to a single line in the xz plane –the focus remains at the native focal plane of the lens and the trajectory length is simply (Δx). When the TAG lens is switched-on, the trajectory length increases due to the continuous axial focus translation. Specifically, the trajectory forms a Lissajous pattern according to the system of parametric equations described in Fig. 1(a). This effect significantly increases the number of voxels scanned per one stroke of a resonant mirror, and consequently, scanning speed. At conditions herein, the trajectory lenght for the Lissajous scan is 45-fold longer than the traditional single-line scan obtained with resonant mirror.
Volumetric imaging beyond the typical piezo range
Typical piezo stages used in commercial microscopes cover a scan range of 100–200 µm. When the depth of the imaged sample exceeds this range, one is forced to use an alternative translation method, which is typically a linear mechanical actuator that moves the objective lens. This alternative is often slower than the piezo stage, and thus the advantage of Lissajous scan becomes more noticeable. Figure 7 shows images of blood vessels from a brain slice captured with a 20x objective lens and over an axial range of 215 µm. The acquisition by our Lissajous scan (top row) took approx. 2 seconds, while the acquisition with the objective translation (bottom row) took 30 seconds, which is 15-fold longer than the Lissajous scan.
Optical sectioning capability over different axial scan ranges
The optical sectioning performance of the Lissajous microscope depends on the number of time windows into which the information is divided. To satisfy the Nyquist sampling, the number of time windows should be selected based on the axial scanned range (Δz) or TAG lens frequency (fTAG). However, in our current implementation, the number of time windows is limited to 1024 at most. Therefore, the effective voxel size in our system depends on Δz and fTAG, as shown in Fig. 8(a) and 9(a), respectively. When Δz is large, the available time windows are not enough to sample the entire Δz at the Nyquist frequency. Figure 8(b) shows a comparison of the axial PSFs in the xz-plane at the same imaging conditions (40x obj, 457 kHz), but with different Δz values. When Δz = 18 µm, the full width at half maximum (FWHM) of the axial PSF was 3.4 µm, close to the value of the piezo scan. The FWHM values increase with Δz — for a scan range of 24 µm, 32 µm, 59 µm, the axial FWHMs were 3.9, 4.5, and 6.7 µm, respectively. Note that the TAG axial scan is continuous and thus each optical slice in a reconstructed 3D stack is a projection of a small volume, so we do not lose the information of the sample along the axial direction. This contrasts with standard piezo scanning, in which increasing the step size and reducing the axial sampling rate results in some of the planes being skipped and consequent loss of information.
Lissajous imaging with an increased number of time windows
In order to increase the number of time windows and thus improve the optical sectioning, one can simply reduce the oscillation frequency of the TAG lens or the axial scanned range. This decreases the number of extrema per a fixed number of time windows, resulting in a higher number of time windows per extrema. Figure 9(a) shows images of a fluorescent nanosphere (left most) acquired by Lissajous scanning with Δz = 9 µm, fTAG = 142 kHz, and a 60x objective lens (NA1.27), and its image counterpart acquired by the piezo scan. The lateral and axial FWHMs of the PSF were 0.37 µm and 1.5 µm by Lissajous scanning, while the corresponding values by the piezo scan were 0.34 µm and 1.4 µm, respectively. The values of the two axial FWHMs are in a good agreement, confirming that with a sufficient number of time windows the optical resolution of our strategy is not degraded. The images in Fig. 9(b) provide further supports on this—a 3D stack of a flower pollen at five different z-positions show that optical sectioning by Lissajous scanning (top row) is identical to that achieved with a traditional piezo scan (bottom row).
Restoration of information via image inpainting
Here, we evaluate the quality of the image inpainting by comparing structures of the neuron between the three images, namely the ground-truth (100 scans accumulated from T=0 to T=3 second), inpainted image with one scan (T = 0), and the raw image with one scan (T = 0). Multiple structures over the axial range visible in the ground-truth appear in the inpainted image while in the raw image they are difficult to perceive. The line profiles shows that structures are well restored in the inpainted image in comparison with the ground-truth, while they are not well visible in the raw image. Note that due to the local intensity variation over time, some strucutures are visible only in the ground-truth image.
Fast imaging of a moving object
We imaged a fast-moving silicon mirror to demonstrate fast cross-sectional imaging. Specifically, we placed the mirror on top of a piezo stage scanner and imaged it with our volumetric Lissajous confocal microscope. We then acquired a time-lapse sequence by translating the mirror along the z-axis. Importantly, the movement of the piezo stage was completely independent from the image acquisition.
In Fig. 11(a), we show line-images of the mirror displacement at two different imaging rates, namely 5.3 kHz and 1.6 kHz. Note that, even if the images were inpainted, several gaps are present due to the poor sampling. In any case, the axial position of the moving mirror could be retrieved over time (Visualization 5 & Visualization 6), and consequently, the mirror trajectory was reconstructed (Fig. 11(b)). From this information, we obtained a mirror translation velocity of 1.23 mm/sec at the cruising region.
Nvidia GPU grant program (Quadro P6000).
We thank Dr. Contestabile for sharing the staining protocol of the 3D brain model, SunJin lab Co. for the kind donation of the 3-color brain slice, and Dr. Andrea Barberis, Dr. Tiziana Ravasenga and Dr. Alice Polenghi for the murine hippocampal neurons. We also acknowledge Dr. Vicidomini and Dr. Koho for useful discussions, Marco Scotto for his technical assistance, and Nikon Japan for advice on system modifications. We especially thank Dr. Anthony for English proofreading, and Prof. Bouman for insights into the inpainting algorithm. Dr. Duocastella is a Serra Húnter fellow.
The authors declare no conflicts of interest.
1. D. L. Coutu, K. D. Kokkaliaris, L. Kunz, and T. Schroeder, “Multicolor quantitative confocal imaging cytometry,” Nat. Methods 15(1), 39–46 (2018). [CrossRef]
2. X. Long, J. Colonell, A. M. Wong, R. H. Singer, and T. Lionnet, “Quantitative mRNA imaging throughout the entire Drosophila brain,” Nat. Methods 14(7), 703–706 (2017). [CrossRef]
3. A. J. Schain, R. A. Hill, and J. Grutzendler, “Label-free in vivo imaging of myelinated axons in health and disease with spectral confocal reflectance microscopy,” Nat. Med. 20(4), 443–449 (2014). [CrossRef]
4. C. Eggeling, C. Ringemann, R. Medda, G. Schwarzmann, K. Sandhoff, S. Polyakova, V. N. Belov, B. Hein, C. von Middendorff, A. Schonle, and S. W. Hell, “Direct observation of the nanoscale dynamics of membrane lipids in a living cell,” Nature 457(7233), 1159–1162 (2009). [CrossRef]
5. T. Laviv, B. B. Kim, J. Chu, A. J. Lam, M. Z. Lin, and R. Yasuda, “Simultaneous dual-color fluorescence lifetime imaging with novel red-shifted fluorescent proteins,” Nat. Methods 13(12), 989–992 (2016). [CrossRef]
6. K. Fujita, M. Kobayashi, S. Kawano, M. Yamanaka, and S. Kawata, “High-resolution confocal microscopy by saturated excitation of fluorescence,” Phys. Rev. Lett. 99(22), 228105 (2007). [CrossRef]
7. J. Dreier, M. Castello, G. Coceano, R. Cáceres, J. Plastino, G. Vicidomini, and I. Testa, “Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo,” Nat. Commun. 10(1), 556 (2019). [CrossRef]
8. D. Magde, E. Elson, and W. W. Webb, “Thermodynamic Fluctuations in a Reacting System—Measurement by Fluorescence Correlation Spectroscopy,” Phys. Rev. Lett. 29(11), 705–708 (1972). [CrossRef]
9. M. Castello, G. Tortarolo, M. Buttafava, T. Deguchi, F. Villa, S. Koho, L. Pesce, M. Oneto, S. Pelicci, L. Lanzanó, P. Bianchini, C. J. R. Sheppard, A. Diaspro, A. Tosi, and G. Vicidomini, “A robust and versatile platform for image scanning microscopy enabling super-resolution FLIM,” Nat. Methods 16(2), 175–178 (2019). [CrossRef]
10. V. Iyer, T. M. Hoogland, and P. Saggau, “Fast Functional Imaging of Single Neurons Using Random-Access Multiphoton (RAMP) Microscopy,” J. Neurophysiol. 95(1), 535–545 (2006). [CrossRef]
11. W. Göbel, B. M. Kampa, and F. Helmchen, “Imaging cellular network dynamics in three dimensions using fast 3D laser scanning,” Nat. Methods 4(1), 73–79 (2007). [CrossRef]
12. R. T. Borlinghaus, “MRT letter: High speed scanning has the potential to increase fluorescence yield and to reduce photobleaching,” Microsc. Res. Tech. 69(9), 689–692 (2006). [CrossRef]
13. Y. Wu, X. Wu, R. Lu, J. Zhang, L. Toro, and E. Stefani, “Resonant Scanning with Large Field of View Reduces Photobleaching and Enhances Fluorescence Yield in STED Microscopy,” Sci. Rep. 5(1), 14766 (2015). [CrossRef]
14. E. J. Botcherby, C. W. Smith, M. M. Kohl, D. Debarre, M. J. Booth, R. Juskaitis, O. Paulsen, and T. Wilson, “Aberration-free three-dimensional multiphoton imaging of neuronal activity at kHz rates,” Proc. Natl. Acad. Sci. 109(8), 2919–2924 (2012). [CrossRef]
15. T. Tuma, J. Lygeros, V. Kartik, A. Sebastian, and A. Pantazi, “High-speed multiresolution scanning probe microscopy based on Lissajous scan trajectories,” Nanotechnology 23(18), 185501 (2012). [CrossRef]
16. S. Z. Sullivan, R. D. Muir, J. A. Newman, M. S. Carlsen, S. Sreehari, C. Doerge, N. J. Begue, R. M. Everly, C. A. Bouman, and G. J. Simpson, “High frame-rate multichannel beam-scanning microscopy based on Lissajous trajectories,” Opt. Express 22(20), 24224 (2014). [CrossRef]
17. H. Lin, C. S. Liao, P. Wang, N. Kong, and J. X. Cheng, “Spectroscopic stimulated Raman scattering imaging of highly dynamic specimens through matrix completion,” Light: Sci. Appl. 7(5), 17179 (2018). [CrossRef]
18. D. Y. Kim, K. Hwang, J. Ahn, Y.-H. Seo, J.-B. Kim, S. Lee, J.-H. Yoon, E. Kong, Y. Jeong, S. Jon, P. Kim, and K.-H. Jeong, “Lissajous Scanning Two-photon Endomicroscope for In vivo Tissue Imaging,” Sci. Rep. 9(1), 1–8 (2019). [CrossRef]
19. L. Kong, J. Tang, J. P. Little, Y. Yu, T. Lämmermann, C. P. Lin, R. N. Germain, and M. Cui, “Continuous volumetric imaging via an optical phase-locked ultrasound lens,” Nat. Methods 12(8), 759–762 (2015). [CrossRef]
20. H. Har-Gil, L. Golgher, S. Israel, D. Kain, O. Cheshnovsky, M. Parnas, and P. Blinder, “PySight: plug and play photon counting for fast continuous volumetric intravital microscopy,” Optica 5(9), 1104 (2018). [CrossRef]
21. G. Sancataldo, L. Scipioni, T. Ravasenga, L. Lanzanò, A. Diaspro, A. Barberis, and M. Duocastella, “Three-dimensional multiple-particle tracking with nanometric precision over tunable axial ranges,” Optica 4(3), 367 (2017). [CrossRef]
22. M. Duocastella, G. Vicidomini, and A. Diaspro, “Simultaneous multiplane confocal microscopy using acoustic tunable lenses,” Opt. Express 22(16), 19293–19301 (2014). [CrossRef]
23. A. Mermillod-Blondin, E. McLeod, and C. B. Arnold, “High-speed varifocal imaging with a tunable acoustic gradient index of refraction lens,” Opt. Lett. 33(18), 2146 (2008). [CrossRef]
24. G. Donnert, C. Eggeling, and S. W. Hell, “Major signal increase in fluorescence microscopy through dark-state relaxation,” Nat. Methods 4(1), 81–86 (2007). [CrossRef]
25. S. Kang, M. Duocastella, and C. B. Arnold, “Variable optical elements for fast focus control,” Nat. Photonics 14(9), 533–542 (2020). [CrossRef]
26. A. Shihavuddin, S. Basu, E. Rexhepaj, F. Delestro, N. Menezes, S. M. Sigoillot, E. Del Nery, F. Selimi, N. Spassky, and A. Genovesio, “Smooth 2D manifold extraction from 3D image stack,” Nat. Commun. 8(1), 15554 (2017). [CrossRef]
27. J. Xie, L. Xu, and E. Chen, “Image Denoising and Inpainting with Deep Neural Networks,” in Advances in Neural Information Processing Systems (Curran Associates, Inc., 2012), pp. 1–9.
28. H. Shen, X. Li, L. Zhang, D. Tao, and C. Zeng, “Compressed sensing-based inpainting of aqua moderate resolution imaging spectroradiometer band 6 using adaptive spectrum-weighted sparse bayesian dictionary learning,” IEEE Trans. Geosci. Remote Sens. 52(2), 894–906 (2014). [CrossRef]
29. S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, “Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation,” IEEE Trans. Comput. Imaging 2, 1 (2016). [CrossRef]
30. T. H. Murphy, L. A. Blatter, W. G. Wier, and J. M. Baraban, “Spontaneous synchronous synaptic calcium transients in cultured cortical neurons,” J. Neurosci. 12(12), 4834–4845 (1992). [CrossRef]
31. L. Kong, J. Tang, and M. Cui, “Multicolor multiphoton in vivo imaging flow cytometry,” Opt. Express 24(6), 6126 (2016). [CrossRef]
32. T. H. Chen, J. T. Ault, H. A. Stone, and C. B. Arnold, “High-speed axial-scanning wide-field microscopy for volumetric particle tracking velocimetry,” Exp. Fluids 58(5), 41 (2017). [CrossRef]
33. H. Mikami, J. Harmon, H. Kobayashi, S. Hamad, Y. Wang, O. Iwata, K. Suzuki, T. Ito, Y. Aisaka, N. Kutsuna, K. Nagasawa, H. Watarai, Y. Ozeki, and K. Goda, “Ultrafast confocal fluorescence microscopy beyond the fluorescence lifetime limit,” Optica 5(2), 117 (2018). [CrossRef]
34. M. A. Digman and E. Gratton, “Lessons in Fluctuation Correlation Spectroscopy,” Annu. Rev. Phys. Chem. 62(1), 645–668 (2011). [CrossRef]
35. S. Piazza, P. Bianchini, C. Sheppard, A. Diaspro, and M. Duocastella, “Enhanced volumetric imaging in 2-photon microscopy via acoustic lens beam shaping,” J. Biophotonics 11(2), e201700050 (2018). [CrossRef]
36. M. Duocastella, G. Sancataldo, P. Saggau, P. Ramoino, P. Bianchini, and A. Diaspro, “Fast Inertia-Free Volumetric Light-Sheet Microscope,” ACS Photonics 4(7), 1797–1804 (2017). [CrossRef]
37. A. Gomariz, P. M. Helbling, S. Isringhausen, U. Suessbier, A. Becker, A. Boss, T. Nagasawa, G. Paul, O. Goksel, G. Székely, S. Stoma, S. F. Nørrelykke, M. G. Manz, and C. Nombela-Arrieta, “Quantitative spatial analysis of haematopoiesis-regulating stromal cells in the bone marrow microenvironment by 3D microscopy,” Nat. Commun. 9(1), 2532 (2018). [CrossRef]
38. E. M. Petrini, T. Ravasenga, T. J. Hausrat, G. Iurilli, U. Olcese, V. Racine, J. B. Sibarita, T. C. Jacob, S. J. Moss, F. Benfenati, P. Medini, M. Kneussel, and A. Barberis, “Synaptic recruitment of gephyrin regulates surface GABA A receptor dynamics for the expression of inhibitory LTP,” Nat. Commun. 5(1), 3921 (2014). [CrossRef]
39. G. Palazzolo, N. Broguiere, O. Cenciarelli, H. Dermutz, and M. Zenobi-Wong, “Ultrasoft Alginate Hydrogels Support Long-Term Three-Dimensional Functional Neuronal Networks,” Tissue Eng. - Part A 21(15-16), 2177–2185 (2015). [CrossRef]
40. G. Palazzolo, M. Moroni, A. Soloperto, G. Aletti, G. Naldi, M. Vassalli, T. Nieus, and F. Difato, “Fast wide-volume functional imaging of engineered in vitro brain tissues,” Sci. Rep. 7(1), 8499 (2017). [CrossRef]