1. INTRODUCTION

Microscopes can be naturally divided into two categories, those that are based on parallelized data acquisition with a camera, and those that are based on sequential data acquisition with a scanner. Accordingly, this review is divided into two parts, involving camera-based and scanner-based microscope strategies. Several redundancies will become apparent, where strategies appear quite similar across both categories. These redundancies should come as no surprise, since they stem directly from the formal equivalence between camera- and scanning-based microscopes resulting from the Helmholtz reciprocity theorem [1]. In other words, a component in the detection path of a camera-based microscope very often needs simply be transposed to the illumination path in a scanning-based microscope (or vice versa) to achieve the same result. And what at first glance might appear to be very different techniques are, upon further examination, simply variations of one another. An effort has been made here to cluster the various techniques to make their similarities more apparent.

2. CAMERA-BASED STRATEGIES

The function of a camera-based fluorescence microscope is almost always to project the fluorescence from a 2D plane in the sample (object plane) onto the 2D plane of the camera sensor (image plane), presumably with some magnification. Imaging is said to be “good” when the object and image planes are conjugate to one another, and the image reveals high-resolution information about the sample that is in focus. But this information is in a transverse 2D plane. What about the third axial dimension? How is it possible to retrieve high-resolution image information from sample structures that are out of focus?

A. Scanned Focus

The most obvious way to perform volumetric imaging is to acquire many 2D images in sequence, each revealing in-focus information from a different object plane. This can be achieved, for example, by physically translating the sample itself in the axial direction, or the camera, or an optical element in between such as the objective. But such physical translation of large objects is inherently slow because of the inertia involved. Recent advances in this strategy have come from reducing or eliminating altogether the inertia associated with focus scanning. Example optical elements that can perform fast scanning are electrically tunable lenses (ETLs) [2], which, when miniaturized, can operate at kilohertz rates [3]. Even faster electrical tuning can be achieved with a tunable acoustic gradient index of refraction (or TAG) lens [4], though at the cost of a reduced aperture size with increasing speed. As an alternative to a transmittive lens, a reflecting component can be used, such as a liquid crystal spatial light modulator (SLM) [5], or a deformable microelectromechanical system (MEMS) [6,7]. To maintain system telecentricity (i.e., magnification independent of defocus), the above elements should be placed in a pupil plane of the microscope, which generally involves the addition of relay optics. Nevertheless, imaging aberrations can arise, especially as the microscope objective is operated farther and farther from its nominal working distance. In principle, SLMs or MEMS can be programmed to compensate for such aberrations. A simpler solution involves directly canceling these aberrations by a technique of remote focusing [8], which makes use of relay optics to duplicate an object plane with unit magnification. In this last technique, a scanning reflector is placed in an object plane, instead of a pupil plane. Because the scanning reflector can be small in size (typically a voice-coil-mounted micromirror), this technique can also be fast, though it does suffer from a loss in fluorescence collection efficiency caused by the use of a beam splitter.

The scanned-focus techniques described above provide multiple 2D images obtained from different focal planes. A drawback of these techniques, which is endemic to all standard fluorescence microscopes, is that the 2D images are not optically sectioned. That is, they are not devoid of out-of-focus background. Such background can be numerically removed post hoc using techniques such as structured illumination microscopy (SIM; see [9] for a recent review). However, SIM requires the acquisition of even more raw images, which typically reduces acquisition speed by an order of magnitude or more and can be prohibitively time consuming when applied to volumetric imaging.

An alternative to numerically removing background post hoc is to prevent the generation of background in the first place. This is the key idea behind light sheet fluorescence microscopy (see [10] for a recent review). Provided the sample is optically clear, light sheet microscopy can provide high-resolution background-free 2D images over a large field of view (FOV) [11]. Scanned-focus imaging can then be performed by axially translating the sample (which is slow) or by axially scanning the light sheet (which can be fast). However, this last approach involves the synchronized co-registration of the detection focal plane with the light field plane, which, while amenable to the use of ETLs [12] or TAG lenses [13], becomes technically involved when high accuracy is required [14]. Of course, a difficulty with light sheet microscopy is that it requires side-on illumination, which, depending on the sample, may not always be feasible. To circumvent this problem, geometries have been devised where the same objective is used to both deliver the light sheet and collect the resulting fluorescence [15,16]. Moreover, the co-registration of the light sheet and focal plane becomes automatically synchronized when using the same scanning optics [17–19] (e.g., Fig. 1), achieving volumetric imaging rates of the order of 10 Hz. Finally, a drawback of pupil clipping in the detection optics can be mitigated by the use of very high NA relay optics [20].

 

Fig. 1. By scanning an oblique light sheet (blue), and descanning the resultant fluorescence (green, yellow orange) with the same mechanical scanner, the light sheet and (oblique) focal planes are automatically co-registered with no other moving parts. The stationary intermediate oblique image plane is then projected onto a downstream camera. Reproduced with permission from [18], 2015, Springer Nature. Alternative scan strategies that maintain a fixed light sheet tilt are described in [17,19].

Download Full Size | PDF

A light sheet can also be generated by two-photon excitation [21], which becomes advantageous if the sample is not quite clear. Indeed, two-photon excitation even provides an alternative method for generating a light sheet based on temporal focusing [22,23]. This more esoteric method is power inefficient, generally restricting it to small FOVs. On the other hand, it provides essentially instantaneous transverse scanning and has the convenience of operating in a head-on illumination configuration. Fast, volumetric imaging can then be achieved by focus scanning, for example with an ETL [24].

B. Multi-Focus

The scanned focus approaches described above involve acquiring a stack of images, typically a z-stack, sequentially in time. We turn now to a multi-focus approach where these images are acquired simultaneously. The most straightforward way to achieve this is with multiple cameras, each conjugated to a different object plane [25]. With the use of an oblique sample geometry, this simple approach can be extended to light sheet imaging [26]. Alternatively, different focal planes can be distributed onto a same camera, which can be achieved by combining lenses of different focusing strengths into a same pupil. Such a technique requires the splitting of the light field into multiple components, which can be conveniently achieved with a single diffractive optical element (DOE) that performs both the splitting and focusing of the field components [27] (or, for more versatility, a SLM [28]). Additional steps can even be taken to correct for the chromatic aberrations that are inherent with DOEs [29] (see Fig. 2). This same DOE-based approach can also be applied to light sheet microscopy [30].

 

Fig. 2. Axially distributed focal planes in the primary image space (conjugate to the sample) are transversely distributed onto a camera plane (final image) with the use of a multi-focus grating (MFG). An additional chromatic correction grating (CDG) and prism assembly compensate for chromatic dispersion, leading to an aberration corrected multi-focus microscope. Reproduced with permission from [29], 2013, Springer Nature.

Download Full Size | PDF

The above techniques involve the use of multiple lenses in the fluorescence detection path, either refractive or diffractive. But there is a different way to perform simultaneous multi-focus imaging by exploiting parallax. This involves not only detecting the positional distribution of the fluorescence light, but also its directional distribution. A four-dimensional function that encapsulates both distributions is the light field [31], or equivalently the plenoptic function [32] (ray-optic variants of the Wigner distribution [33]). The projection of the light field onto a single camera frame necessarily entails a compromise in spatial resolution to accommodate some degree of directional resolution. The most common implementation of light field microscopy involves placing a microlens array in a plane conjugate to the object, followed by a camera in a pupil plane (or Fourier plane). In this manner, the camera sensor is segmented into a grid of “super-pixels,” each of which records the local directional distribution of the detected fluorescence. The price paid is a loss in spatial resolution, which has been degraded to the size of a super-pixel (typically the size of a microlens). The gain that comes from directional resolution, however, is an ability to numerically synthesize images that are focused at different depths in the sample. In other words, what was achieved directly with the multiple lens approaches above, can now be achieved post hoc. The numerical refocusing of images can be performed with a simple shearing algorithm [31], or through more sophisticated approaches involving deconvolution [34], which can be aided by wavefront coding [35,36]. Light field microscopy is effective at providing volumetric images in optically clear samples [37], where it has been generalized to multi-view imaging for isotropic resolution [38] (see Fig. 3). In samples that are not so clear, image reconstruction becomes more difficult and requires a priori information, such as an assumption of sample sparsity [39]. A miniaturized version of this has even been applied to freely moving mice [40] (with the caveat that its accuracy was confirmed only with planar fluorescent source distributions rather than volumetric distributions).

 

Fig. 3. Objective configuration of an orthogonal-view light field microscope (blue, excitation; green fluorescence detection with microlens arrays not shown). Each view separately leads to poor axial resolution, whereas both views together lead to isotropic resolution (PSF) upon image reconstruction. Adapted with permission from [38], 2019, Springer Nature.

Download Full Size | PDF

But other variants of light field microscopy are possible (see [41] for review). For example, the microlens array can be placed in a Fourier plane while the camera is placed in an object plane (sometimes called integral imaging), more or less inverting their roles in encoding positional versus directional information (e.g., [42]). Finally, it should be mentioned that pseudo-3D visualizations of objects can be obtained directly by techniques of view synthesis [43,44], circumventing the need for volumetric object reconstruction altogether.

C. Extended Focus

So far, we have considered approaches that provide images focused at different depths, acquired sequentially or simultaneously. We turn now to a different approach that does not provide focused imaging at any depth at all. Such an approach involves extending the focus of a microscope in the axial direction, rather than scanning it. In other words, such a microscope produces 2D images where all depths are superposed upon one another (as opposed to distributed), and all axial resolution is lost. Such extended depth of focus (EDOF) imaging can be achieved by taking the scanned focus approach to its extreme such that the focal sweep is performed within a camera exposure time rather than across multiple exposure times [45] (which presumes, of course, that the sweep mechanism is faster yet than the speed of the camera [7,13]). The resulting EDOF images are peculiar in that they are simultaneously in- and out-of-focus, with an attendant point spread function (PSF) that is nearly space invariant in all three dimensions and amenable to deconvolution [46–48]. Nevertheless, a problem with EDOF imaging is that the contribution of out-of-focus background increases with increasing EDOF range, leading to increased background noise that hampers effective deconvolution. This problem can be alleviated by reducing the generation of background with 3D targeted illumination [49].

An extended focus can also be generated with no moving parts, by modulating the wavefront of the fluorescence in either amplitude or phase. The resulting PSF is thus “engineered” to be axially extended. To preserve PSF translational invariance, the wavefront modulator should be placed in the detection pupil plane. Example modulators are annular masks [50], (incoherent) superpositions of annular masks [51,52], Fresnel zone plates [53], logarithmic masks [54,55], etc. These example masks are radially symmetric, meaning that they lead to radially symmetric PSFs (as does the focal sweep method described above). Such PSFs have the advantage that they produce images that are readily interpretable even in their raw form. The same cannot be said for asymmetric pupil masks, such as cubic phase masks [56,57], or even random diffusers [58,59], which require the additional step of deconvolution. But asymmetric masks have their own advantages. Because of asymmetries in the resultant engineered PSFs, there can be a coupling between the axial position of an object and the transverse position of its image, which can facilitate depth ranging from a single image [60]. PSFs optimized for this purpose can be very strange indeed [61,62], and even made broadband [63]. Note that such depth ranging can be rendered more robust with two-shot imaging with different wavefront-coded pupils [64,65] (generalized to multi-shot [66]), or even with opposing illumination ramps [67], but this undermines somewhat the speed advantage of EDOF imaging.

EDOF approaches can also be applied to light sheet microscopy. For example, phase/amplitude masks can be placed in the illumination pupil to extend the range of a light sheet [68], or to produce what is known as a lattice light sheet of exceptionally fine resolution [69] (see Fig. 4). Alternatively, masks can be placed in the detection pupil, allowing the light sheet to be broadened (and hence also extended in range) [70,71]. This last strategy was used in [72], where the detection path was aberrated by a simple refractive-index block.

 

Fig. 4. Different masks applied to the illumination pupil of a (swept-beam) light sheet microscope. A, a Gaussian beam at the pupil leads to a Gaussian beam at the sample that, when swept, becomes a light sheet. B, an annular beam at the pupil leads to a Bessel beam light sheet. C, a square lattice at the pupil optimizes the confinement to the central Bessel plane. D, a hexagonal lattice optimizes the overall light sheet axial resolution. Reproduced with permission from [69], 2014, AAAS.

Download Full Size | PDF

A final EDOF approach is mentioned here because it as intriguing as it is difficult to classify [73,74]. By imaging the Fourier plane (as opposed to the object plane) of a phase-shifting rotational shear interferometer, and performing SIM-like computations, an image with in principle limitless DOF can be numerically reconstructed.

3. SCANNING-BASED STRATEGIES

We turn now to volumetric imaging strategies based on scanning microscopes. One of the key advantages of scanning microscopes is that they enable the possibility of optical sectioning even in head-on illumination configurations, either by physically blocking fluorescence background (as in confocal microscopy), or by not generating background in the first place (as in multi-photon microscopy), thereby providing high signal contrast. However, such microscopes generally rely on the scanning of a single focal probe volume throughout the sample, which can be slow, particularly in three dimensions. An obvious way to address this problem is to develop faster scanning mechanisms, and indeed this direct approach is perhaps the most promising for the advancement of scanning-based volumetric microscopy. However, before considering this approach we turn to an alternative compromise strategy, which, while not providing volumetric imaging per se, provides quasi-volumetric imaging. Specifically, we turn to EDOF approaches based on PSF engineering.

A. PSF Engineering

As before, the point of PSF engineering is to increase acquisition speed. In the case of camera-based microscopes, quasi-volumetric images are obtained from a single camera frame, without the need for axial scanning. In the same manner, in the case of scanning-based microscopes, one less scan direction becomes required. Several examples of EDOF imaging can be found in multi-photon microscopy, where the laser focus is stretched from a Gaussian to a Bessel focus either dynamically [75] or statically [76–80] (e.g., Fig. 5). Another example makes use of simultaneous multi-focus excitation distributed axially [81,82]. In these examples, because of the symmetric nature of the engineered PSF, axial resolution is lost. On the other hand, for samples that are fixed in space and vary only in time, it can be argued that such axial resolution is not necessary, since it can be obtained separately either pre or post hoc. Moreover, the segmentation of dynamic objects (e.g., active neurons) can in practice be performed post hoc using statistical methods [83], provided one has acquired sufficiently long time sequences.

 

Fig. 5. A, scanning a standard Gaussian focus (yellow) in the x–y plane probes structures in a thin optical section, whereas scanning a Bessel focus (orange) in the x–y plane probes structures throughout a 3D volume. B, a Bessel beam can be created by inserting a SLM and mask to achieve annular illumination in the pupil plane of a two-photon microscope. Reproduced with permission from [78], 2017, Springer Nature.

Download Full Size | PDF

Nevertheless, in some applications, some degree of axial resolution may be desirable, even if only a capacity for depth ranging. To this end, asymmetric PSFs that couple axial to transverse spatial information can be useful. For example, two-photon microscopy, where the PSF is defined by the excitation beam, can achieve depth ranging with twin-beam stereoscopy [84,85]. Similarly, confocal microscopy, where the PSF is defined by both excitation and detection, can achieve depth ranging with combined wavefront engineering and the use of an array detector [86,87] (which can be parallelized for higher speed [88]). A drawback of these methods is that they all require some form of image deconvolution to disentangle the depth information. A more straightforward approach is to perform multi-plane imaging directly (i.e., computation-free) with the use of a micromirror array [89] or axially distributed reflecting pinholes [90] (see Fig. 6). These last techniques are confocal microscopes that provide optically sectioned image stacks rather than single images, using the same laser power and with no penalty in speed.

 

Fig. 6. Multi-Z confocal imaging can be achieved by underfilling the illumination pupil while utilizing the full detection pupil and detecting the resultant fluorescence with a series of axially distributed reflecting pinholes. Here, four optically sectioned are acquired simultaneously. More can be acquired sequentially with the addition of an ETL. Adapted with permission from [90], 2019, OSA.

Download Full Size | PDF

B. Fast 3D Scanning

We turn now to strategies where axial information is acquired not instantaneously, as it was above, but instead sequentially in time, allowing time to serve as the dimension into which spatial information is encoded. In other words, we revert to the standard strategy of focal spot scanning, but here in 3D rather than simply 2D. The most recent developments in this regard have been based on increases in scanning speed.

One approach comes from recognizing that fluorescent objects of interest are often sparsely distributed, meaning that fast scanning can be achieved by visiting only those objects, while skipping over the spaces in between. Rapid switching from one target to another can be achieved, for example, with acousto-optic deflectors (AODs), which are free of inertia. By exploiting the fact that AODs can be used to focus laser beams in addition to deflecting them [91], such targeted scanning can even be performed in 3D with two-photon excitation [92–95]. However, a drawback of this approach is that it is restricted to objects (here, neurons) that remain fixed in time, making it susceptible to motion artifacts that cannot be corrected a posteriori. Moreover, it requires an initial calibration step to determine where the objects of interest are located in the first place.

For object distributions that move or distort over time, or are unknown a priori, space-filling scanning strategies are preferred. An early attempt at performing this at high speed involved transverse spiral scanning and sweeping the objective of a two-photon microscope as fast as possible [96]. More recently, much faster axial scanning has been demonstrated with ETLs [97] or TAG [98] lenses, or voice-coil-based remote focusing [99,100]. But space filling at high resolution quickly becomes problematic, since it is intrinsically time consuming and leads to the proliferation of unwieldy amounts of data. For fast, large-scale imaging, an intentional sacrifice in resolution may be required, and even perfectly acceptable (e.g., [90,101,102]). Alternatively, one can perform fast scanning that is not exactly space filling by resorting once more to discrete multi-plane imaging, as demonstrated in confocal microscopy with a TAG lens [103], or in two-photon microscopy with an ETL [104] or fast AOD switching [105].

Finally, with the advent of fast detector technologies, a new technique for multi-plane imaging has emerged that is near-instantaneous while nevertheless continuing to allow the separation of different planes in time. This technique, applicable to multi-photon microscopy, involves splitting the excitation laser beam into independently time-delayed beamlets that can be focused to different depths within the sample. Lower limits to the time delays between the beamlets is not even imposed by the detector speeds (which nowadays are fast), but rather by the finite lifetime of the fluorescent indicators, which is typically a few nanoseconds. Since its original demonstration [106], this technique has gained popularity [107–109], which will likely grow with the continued development of low-repetition rate lasers for ultradeep imaging (e.g., [110]). For example, this technology has been applied with combined two- and three-photon microscopy, to rapidly probe volumes more than a cubic millimeter in mouse brain [111] (see Fig. 7). But the complexity of these devices increases with the number of imaging planes. To solve this problem, a “reverberation loop” can be used to split the excitation beam into an arbitrary number of increasingly time-delayed beamlets [112] (see Fig. 8). In addition to being simple, this last technique has the benefit of automatically matching the beamlet power to the imaging depth in tissue.

 

Fig. 7. Schematic of a hybrid microscope that provides simultaneous two- (red) and three- (blue) photon excitation, using a custom-built laser. Resulting fluorescence is in green. This microscope includes a time multiplexing module for near-instantaneous four-plane two-photon imaging, a temporal focusing module to tailor the spatial resolution for video-rate scanning, and a remote focusing module for axial scanning of the four-plane stack. Reproduced with permission from [111], 2019, Elsevier.

Download Full Size | PDF

 

Fig. 8. Arbitrary number of focal planes can be near-instantaneously captured by inserting a reverberation loop in the excitation beam of a multi-photon microscope [112]. A lens (or equivalent) in the loop causes each subsequent beamlet to focus to increasingly shallower depths all the way to the sample surface. With a 50:50 loop beam splitter, half the laser power targets the deepest plane and the other half targets all the other planes, while producing roughly equal fluorescence power per plane.

Download Full Size | PDF

4. OUTLOOK

This review of volumetric microscopy strategies will most certainly become obsolete within a few years, given the rapid progress in new camera and scanning technologies. For example, the point of the PSF engineering approaches described above is that they obviate the need for axial scanning. This need becomes less pressing with the development of ever faster scanning techniques. Hardware is already available that enables volumetric microscopy at scales up to a cubic millimeter and speeds up to video rate in a direct manner (i.e., computation-free). At these scales, another problem usually intervenes that is much more fundamental than the hardware itself, which is the problem of signal-to-noise ratio. Indeed, the shorter the amount of time one can allocate to the detection of signal, the greater the relative noise, shot or otherwise. This problem of SNR is particularly restrictive in the case of fluorescence, where maximum signals are limited by fluorescence lifetimes, or, as is more often the case, by constraints related to photobleaching or photodamage. Ultimately, these are the issues that will have to be addressed if progress in volumetric imaging is to continue. Computational techniques based on a priori sample information will likely be useful in this regard, as will the development of indicators that are brighter and more robust, or not based on fluorescence at all.