Light field microscopy has been proposed as a new high-speed volumetric computational imaging method that enables reconstruction of 3-D volumes from captured projections of the 4-D light field. Recently, a detailed physical optics model of the light field microscope has been derived, which led to the development of a deconvolution algorithm that reconstructs 3-D volumes with high spatial resolution. However, the spatial resolution of the reconstructions has been shown to be non-uniform across depth, with some z planes showing high resolution and others, particularly at the center of the imaged volume, showing very low resolution. In this paper, we enhance the performance of the light field microscope using wavefront coding techniques. By including phase masks in the optical path of the microscope we are able to address this non-uniform resolution limitation. We have also found that superior control over the performance of the light field microscope can be achieved by using two phase masks rather than one, placed at the objective’s back focal plane and at the microscope’s native image plane. We present an extended optical model for our wavefront coded light field microscope and develop a performance metric based on Fisher information, which we use to choose adequate phase masks parameters. We validate our approach using both simulated data and experimental resolution measurements of a USAF 1951 resolution target; and demonstrate the utility for biological applications with in vivo volumetric calcium imaging of larval zebrafish brain.
© 2014 Optical Society of America
Light field microscopy, first presented by Levoy et. al in 2006 and 2009 [1, 2] and further improved upon by Broxton et. al in 2013 , is a method for single-snapshot volumetric imaging that employs a microlens array in the optical path of a fluorescence microscope. Unlike the widefield microscope, which records the spatial distribution of light collected by the microscope objective, the light field microscope (LFM) records both the spatial and angular distribution of the light. Captured light field images can be processed to reconstruct a full three dimensional (weakly scattering) volume or a single two dimensional z plane in the volume. In  we presented a reconstruction method that uses an inverse-problem framework with a wave-optics optical model of the LFM and solved it using 3D deconvolution. We showed that unlike traditional microscopy, in which high resolution can only be achieved for a small range of z depths around the native focus plane of the microscope (its depth of field), the volume reconstructed from the captured light field image of the LFM preserves significant portion of lateral spatial resolution at each z plane; even over a hundred microns away from the native object plane.
However, as we showed in , whether reconstructing a full volume or only a two dimensional z plane in that volume from a light field image, the achievable lateral resolution of the LFM is not uniform across z depths. Figure 1(b) shows qualitatively the resolution that can be achieved with a 20× 0.5NA water-dipping objective and a 125μm-pitch f/20 microlens array, and a more detailed analysis of the resolution is given in section 4.1 and Fig. 5 of . Our analysis showed that at the center z planes of the imaged volume, over a z range spanning 10 μm around the native object plane of the LFM, the achievable lateral resolution is significantly degraded compared with that of a widefield fluorescence microscope at its native object plane (Fig. 1(a), left-most column). Farther away from the native object plane the achievable spatial resolution degrades gradually. However, the LFM working range, over which the lateral resolution is high, remains much larger compared with a widefield fluorescence microscope. This non-uniformity of resolution hinders practical use of the LFM for some applications that require localization and measurement of fine 3D structures in a sample that is spread over a large range of z depths. In [4, 5], the authors demonstrate the use of LFM to image neural activity in a transparent larval zebrafish. In supplemental video 6 of , reconstructed neurons located at the native object plane of the LFM show up as large uniform square patches compared to neurons at other z depths, which are better resolved.
In this paper we propose a novel extension to the LFM that aims to mitigate the nonuniformity of lateral resolution across depth by placing phase masks in the optical path of the LFM, in order to produce a more uniform performance across depth. This technique, called wavefront coding, shapes the point spread function of the microscope and is a natural extension to traditional light field microscopy. It can be easily incorporated into the optical model and light field deconvolution framework we proposed in . We propose two types of designs: (1) placing a single phase mask at the back focal plane of the microscope’s objective (we call this the objective mask), and (2) combining this objective mask with an array of smaller phase masks positioned at the native image plane of the microscope, as an integral part of the microlens array (microlenses masks). We also show that properly-designed masks can improve the LFM resolution profile for reconstructing both 2D planes (off the object native plane) and 3D volumes and create either a more uniform resolution profile across z depths. Our design solves the low-resolution problem around the native object plane, and can even create a resolution profile that favors the native object plane over other depths. Our data suggests a general design principle for our wavefront coded LFM - using an objective phase mask to control the size of the point spread function (PSF) at the native image plane that is sampled by the microlenses; and using microlenses phase masks to shape the PSF at the detector plane so that high frequencies can be resolved over a wide range of depths.
Since wavefront coding techniques were first proposed by Cathey and Dowski  phase masks have been designed and employed by researchers in various imaging modalities. In microscopy, phase-only elements have been placed in the back focal plane of a microscope’s objective, mainly for the purpose of extending the depth of field of the microscope when imaging samples that are spread over a wide range of depths [7–10], but also for the purpose of reducing the depth of field, allowing for better optical sectioning .
In the context of integral imaging, Castro et al. proposed to add an array of quartic phase masks to an integral imaging acquisition system made of a microlens array and a detector, in order to improve each microlens depth of field . The three-dimensional reconstruction method requires to convolve each microlens image with a single inverse filter to remove the mask blur, and then the volume is reconstructed using a pinhole ray-optics model. However, the proposed method does not aim to reconstruct above the resolution of the microlens array, nor does it consider a complex optical system such as a camera or a microscope, where diffraction takes place.
A light field image could also be captured together with a widefield image, the latter providing high-resolution information about the native object plane as proposed in . This method requires splitting the incoming light between two cameras or capturing the light field and wide-field images sequentially, which is less suitable for imaging dynamic phenomena. Another approach to improve the low resolution at the native object plane is to divide the light captured by the objective into two optical paths with a known path difference between them, and place two microlens arrays (potentially with different focal lengths) side by side in front of the detector . However this method has the drawback that it sacrifices half the sensor area.
It is worth noting that there are some degrees of freedom in the design and use of the LFM that can help mitigate the problem of non-uniform resolution at the expense of other properties. For example, by varying the microlens array pitch a tradeoff between the peak resolution of the LFM and the resolution farther away from the native focal plane can be achieved - smaller pitch would result in higher peak resolution in the vicinity of the native object plane and worse lateral resolution farther away from it. It is also possible to avoid the low resolution at the native object plane by focusing the LFM outside the volume of interest in the sample. However, this sacrifices half of the microscope’s working range.
The paper is organized as follows: in Section 2 we present an extension to the optical model we presented in  for the wavefront coded fluorescence LFM that includes phase masks. This model is then used to generate simulated results and to deconvolve experimental data. In Section 3 we derive a performance metric for the LFM based on Fisher information. The metric is independent of the type of reconstruction algorithm used and is well correlated with standard resolution measurement methods. We demonstrate that it can be a useful tool for designing phase masks and optimizing LFM performance. Finally, in Section 4 we present both experimental results using a fabricated glass objective phase mask, as well as simulated results for other proposed microscope configurations, including microlenses phase masks.
2. Optical model
The reconstruction of a 2D plane or a 3D volume from a light field image requires solving an inverse problem of the form
In  we also presented a physical-optics model of light propagation through the LFM, based on wave optics theory. The model assumes a fluorescence microscope in which the objective and tube lens are set up as a doubly-telecentric 4-f system where the back focal plane of the objective coincides with the tube lens’s front focal plane. The wavefront at the native image plane of the microscope, generated by a point source in the volume, is calculated using scalar diffraction theory for a circular aperture objective  (which is accurate for low-NA objectives with an aperture diameter much larger than the wavelength of incoming light, for which polarization effects can be neglected). This wavefront is then multiplied by the transmission function of the microlens array and the result is propagated a distance equal to one microlens focal length to the detector plane where an intensity image is recorded. In order to solve the inverse problem, the optical model is discretized and measurement noise is modeled to have a Poisson distribution (i.e. mostly shot noise).
In this work we extend this optical model to include a phase mask in the back focal plane of the microscope’s objective, and phase masks at the aperture of the microlenses of the microlens array. Our model takes advantage of the explicit Fourier transform relationship between parallel planes in the microscope, and is fast to compute numerically. This is especially important, as we later define a performance metric, based on the forward optical model, to search for suitable phase masks that improve the performance of the LFM.
We define our coordinate system so that its origin is located on the optical axis at the front focal plane of the objective, which we also refer to as the native object plane. Positive z values are defined towards the objective. x and y are spatial coordinates on planes perpendicular to the optical axis. A point source with electric field amplitude A, located at p = [xp, yp, zp]T in the volume (marked by the red dot in Fig. 2) emits a spherical wavefront that propagates to the front focal plane of the objective. According to the Huygens-Fresnel integral  and assuming r ≫ λ, at the front focal plane of the objective we have
The objective and tube-lens in a well-corrected, doubly-telecentric microscope can be modeled as 4-f system, as shown in Fig. 2. The objective’s focal length, denoted fobj can be calculated from the known magnification M and the tube lens focal length ftl15]
Under the Fresnel approximation, the wavefront at the objective’s front focal plane U0 (x, y, p), multiplied by the objective’s pupil function and the wavefront at the back focal plane of the objective form a Fourier transform pair (up to a complex scaling factor)
In a 4-f telecentric microscope configuration, the back focal plane of the objective coincides with the front focal plane of the tube lens, and therefore the wavefront at the back focal plane of the tube lens , is a scaled Fourier transform of17], multiplied by the phase function ϕobj (x, y, Θobj).
Our original design of the LFM features a microlens array at the native image plane of the microscope (where the detector is positioned in a widefield fluorescence microscope). As we described at length in , the microlens array acts as a sensing element that enables the recording of both spatial and angular information about the light distribution in the imaged sample. We extend the optical model to include an arbitrary phase function in each microlens aperture. Denoting Tse (x, y, Θμlens) the general transmission function of the sensing element, Pse (x, y) the aperture function and Qse (x, y, Θμlens) the phase function, the transmission function takes on the form of a rectangular grid, with added phase component that multiplies the quadratic phase of each microlens
Tse (x, y, Θμlens) multiplies the incident wavefront . For convenience, we define Θ = [Θobj, Θμlens]T and we haveFig. 2, equals to propagated forward the distance between the sensing element and the detector which is equal to one microlens focal length fμlens in our case
Finally, the resulting incoherent light field PSF at the detector plane is given by the squared modulus of the coherent PSF3]. In this work we use the same formulation and reconstruction techniques presented there. Since the addition of phase masks to the optical model merely changes the intensity distribution in the light field PSF, compared with the light field PSF of a standard LFM (i.e. the system matrix H in Eq. (1)), our proposed extension of the LFM comes at only a minor increase in computational complexity over the existing light field deconvolution technique.
3. Analysis of the performance of the LFM based on Fisher information
Characterizing the performance of an LFM is not a trivial task. Common methods such as measuring the full-width-half-max of the optical system’s PSF or using a resolution target (such as a US Air Force 1951 test target) to measure the achievable resolution cannot be applied to the light field image directly. Rather, these methods must be applied to the image or volume reconstructed from the light field image (see ) and are therefore become sensitive to the choice of reconstruction algorithm and any priors the algorithm applies. In addition, reconstructions are computation intensive and thus time consuming. For these reasons, it is inconvenient to rely on them when designing objective or microlenses phase masks for enhancing the LFM performance. It is therefore beneficial to develop a performance metric that is independent of the reconstruction but correlates well with resolution measurement methods.
We propose to characterize the performance of our wavefront coded LFM by calculating the Fisher information of the optical system. This is similar to the approach used in , in which the Fisher information was used to analyze the sensitivity of a cubic phase mask to misfocus. Subsequently, the Fisher information was also used in design and analysis of other phase masks for enhancing the depth of field of imaging systems [18–20]. In these works, the phase mask performance is analyzed and optimized by calculating the Fisher information of the optical system’s PSF with respect to the Zernike defocus coefficient W20. The Fisher information was used to measure how the PSF changes with defocus. A phase mask that minimizes the Fisher information results in a PSF that is insensitive to defocus, thus granting the optical system extended depth of field. In contrast, in  the authors designed a phase-amplitude mask that results in a rotating PSF that yields a high Fisher information with respect to defocus. This allowed them to estimate the depth of a two-dimensional thin object from the rotation angle of the PSF via a deconvolution process. In our setting, we will see that the Fisher information can measure the sensitivity of the light field PSF to the position of a point source in the volume and serve as the basis for a metric that the calculates the performance of the microscope.
We adapt the Fisher information-based approach to our needs by defining a measure of how the light field PSF changes when the position of a point source translates slightly in the volume. Formally, consider the light field PSF h (x, y, p, Θ) observed at the detector plane. Normalized properly, h (x, y, p, Θ) can be regarded as a two-dimensional probability density function, de-fined over the detector area, which depends on the unknown parameters vector p. Therefore, we can calculate the Fisher information matrix ℐ (p, Θ) with respect to the unknown point source position p, from the observed light field PSF
High Fisher information values suggest that the light field PSF is sensitive to the exact position of the point source in the volume - i.e. slight shifts in the point source position in the volume result in a significant, detectable change in the light field PSF recorded on the detector. And vice versa - low Fisher information values mean that the recorded light field PSF appears the same on the detector, regardless of small shifts in the point source position in the volume. In this case, the imaging system suffers from inherent ambiguity with respect to the point source position that will hinder reconstruction methods that try to localize the point source in the volume.
The proposed Fisher information matrix depends only on the optical model of the system, irrespective of which reconstruction algorithm is employed. It is also relatively fast to compute numerically - it requires calculating three Fourier transforms and four matrix multiplications per object point (this depends on the exact elements in the optical path), and therefore can be used for both analysis and synthesis purposes.
The proposed performance metric was used to design the combination of phase masks proposed in section 4.4 - the parameters of the phase masks Θ were chosen to maximize a performance metric that is based on the Fisher information matrix in Eq. (2). The performance metric is a weighted sum of the Fisher information matrix elements
In Fig. 8(e) and Fig. 8(f) we show a comparison between the lateral resolution limit measured on reconstructed planes of a simulated USAF 1951 resolution target and the proposed performance metric for different configurations of phase masks in our wavefront coded LFM, respectively. To generate the performance metric plot we set ωij to consider only the elements of the Fisher information matrix that correspond to lateral resolution (in the directions of the axes xpxp, ypyp and the diagonals xpyp and xpyp). The values ℐij (p, Θ) were calculated for a point source on the optical axis of the microscope for different z planes. The derivative was calculated numerically using a finite difference approximation with a step of 0.5 μm (slightly below the smallest feature on the USAF 1951 target). The comparison confirms that the performance metric we propose correlates well with actual resolution measurement on reconstructed USAF 1951 data.
4. Experimental and simulation results
In this section we present simulated and experimental results that demonstrate the performance of our wavefront coded LFM. We begin with experimental results using an objective phase mask that we fabricated. In section 4.1 we discuss our experimental setup and in section 4.2 we show experimental results with only a cubic phase mask placed at the objective’s back focal plane, and using a standard microlens array with no phase masks. We compare the lateral resolution of the standard LFM to our wavefront coded LFM and discuss the reasons for the difference in performance. We also demonstrate experimentally the improvement in resolution when reconstructing full volumes in section 4.3 - both for a synthetic sample and for a biological application. In section 4.4 we show simulation results of additional configurations of our wavefront coded LFM, including phase masks in the microlenses apertures. We discuss the merits of different configurations and compare their respective resolution performance across depth. Finally, in section 4.5 we give a ray-space analysis of our wavefront coded LFM with two phase masks configurations. The analysis provides more insight on how the use of phase masks affects sampling of spatial and angular information of the LFM.
4.1. The experimental setup
In order to test the performance of our wavefront coded LFM we altered our light field microscope setup described in  to include a physical phase mask in the objective back focal plane. It is not straightforward to access the objective’s actual back focal plane in our setup so we added relay optics to create a 1:1 conjugate back focal plane where we can easily place a phase mask. Figure 3 shows an image of our setup and a schematic diagram of the optical path. A sample may be trans-illuminated from below by a diffused halogen light source (that emits incoherent light) and through a 0.9NA condenser lens (Nikon) arranged in Köhler-illumination, or epi-illuminated through the objective using a xenon arc lamp at 488 nm wavelength. Note that the condenser NA has to be equal or larger than the objective’s NA so that light completely fills the objective’s back aperture and the resolution is therefore not limited by the illumination (in this case, the number of the pixels under a microlens that collect useful data about the sample is maximized). The sample is imaged by a 20× 0.5NA water immersion objective (Nikon) through a 535 nm fluorescence emission filter (Semrock 535/50) and through a tube lens, L1 (Nikon 200mm). The physical phase mask is placed between a pair of relay lenses, L2 and L3 (3” Edmund Optics achromatic doublets, f=200 mm) which together form a 1:1 4-f system. We chose to match their focal length to that of the tube lens to create a 1:1 image of the back focal plane of the objective. The exact position of the glass phase mask (PM) is determined according to the location of the conjugate objective’s back focal plane (note that if the microscope is not exactly telecentric, as is the case in our setup, this plane may be different than the back focal plane of L2). The mask is mounted on a stage that allows for XY translation and rotation about the optical axis for accurate alignment. L3 is placed one focal length after the phase mask. The remainder of the light path is the standard path used in : a microlens array (MLA, RPC 125 μm pitch, f/20) is positioned at the native image plane of the microscope, with lenses L4 and L5 (Nikon 50mm f/1.4 and Nikon 35 mm f/2, arranged in back-to-back configuration) used to relay the detector (DET, Retiga 4000R) plane to precisely one microlens focal length behind the microlens array. This optical setup effectively implements the optical path shown in Fig. 2 (without the microlens phase masks). Note that the pair of relay optics L2, L3 and L4, L5 are not modeled of our optical model since we assume they are high quality optics and since we use only the center part of these large lenses to minimize aberrations. Therefore, the relay optics have very little effect on the light field PSF. We note however, that the choice of focal lengths for L4 and L5 does affect the number of detector pixels under each microlens, as explained in . In our configuration we chose to have 25 × 25 pixels under each microlens.
We chose to manufacture and test the well known cubic phase mask since it is relatively easy to fabricate and due to its promising performance in simulations. The cubic mask, made by RPC Photonics, was fabricated by depositing a 20 mm × 20 mm polymer (with refractive index n = 1.52 for emission wavelength of λ = 535 nm) on BK7 glass substrate and etching the cubic shape onto the polymer. The phase function it implements is p(x, y) = 117 (x3 + y3), where the spatial coordinates x and y are normalized to the square mask size. The free parameter α = 117 was chosen to take advantage of the entire depth range achievable by the fabrication process over the 20 mm × 20 mm mask area (the depth limitation of the process was 75 μm). Note that the mask area is larger than the diameter of the back aperture of the 20× 0.5NA objective we use (the back aperture diameter dobj = 2 fobjNA  is equal to 10 mm in the case of our 20× 0.5NA objective). Since our phase mask has width and height twice that size, it can also accommodate other objectives - particularly ones with relatively low magnification and high NA that have larger back aperture diameter.
To test the accuracy of the fabrication we measured the profile of the mask using a coherence correlation interferometer (CCI HD, Taylor Hobson). Figure 4(a) shows the measured mask sag and the back aperture diameter of the microscope objective. The maximal surface error compared to the design was measured to be 500 nm over the entire mask area and 300 nm inside the objective’s back aperture diameter.
4.2. Experimental results with a single phase mask - resolution analysis
Before presenting the resolution measurements with the objective cubic phase mask, it is beneficial to study light field PSFs for a point source at the native object plane to gain insight into the increase in resolution at that plane, attributed to the phase mask. Figures 4(b) and 4(c) show the effect of a small lateral translation on the PSFs at the native image plane and on the light field PSFs at the detector plane, with and without the cubic phase mask, respectively. The cubic PSF shape at the native image plane spans an area of about 150 μm×150 μm and is much larger than the diffraction spot of the standard LFM. It is spread by the microlens array and forms a complicated diffraction pattern on the detector that spans four microlenses. This light field PSF is more sensitive to the exact position of the point source on the native focus plane than the light field PSF without the phase mask. When the point source is translated 2 μm laterally, the diffraction pattern changes significantly. The diffraction spot of the standard LFM on the other hand, is spread by a single microlens to a circular disk on the detector plane. Shifting the point source position laterally causes only a slight change in the light field PSF - since the shifted small diffraction spot on the native image plane still lies within the numerical aperture of a single microlens, it produces in a very similar disk pattern on the detector plane. Therefore, the cubic phase mask reduces the ambiguity in localizing a point source on the native object plane.
To confirm the improvement in lateral resolution in actual experiments we used the same procedure as in  where we imaged a high resolution USAF 1951 test target (Max Levy DA052) at each of 201 z depths spanning 200 μm, and reconstructed the image of each target, restricting the light emission to voxels at the known z depth of the target in each image. We chose to reconstruct each image with a spatial sampling period 16 times smaller than the native sampling period of the LFM, as determined by the microlens array pitch (or microlenses spacing) divided by the objective magnification (its reciprocal is referred to herein as the native resolution of the LFM). The resulting reconstructions of the resolution target (Figs. 5(a) and 5(c)) were then analyzed using a similar contrast metric as in  to produce the resulting modulus transfer function (MTF) plot shown in Figs. 5(b) and 5(d). Each z plane containing a USAF 1951 pattern was registered to a ground truth image and the contrast was calculated for each of the target’s regions of interest according toFigs. 5(b) and 5(d) the green curve in shows the limiting resolution of the LFM taken to be the 10% contrast cut-off frequency.
Using the LFM with no phase masks, we see several limitations. Over a large range of depths around the native object plane of the LFM the achievable spatial resolution fluctuates significantly, with certain z depths (−15 μm off the native object plane) reaching high spatial resolution, while other z depths (around −25 μm off the native object plane) only reaching lower spatial resolution. In particular, the lateral spatial resolution of the LFM at z depths very close to the native object plane (the z-depth the microscope is focused at ±5 μm around it) is significantly degraded compared with that of a traditional fluorescence microscope. Here, resolution is limited by the fact that the detector pixels collect aliased, redundant information about the spatial position of an object in the volume, as explained in . In the example shown in Fig. 5, a LFM with a 20× objective and 125 μm pitch microlens array yields a maximal attainable frequency of 80lp/mm at the native object plane. At these z depths, objects with fine details cannot be reconstructed with high fidelity. Moreover, even at z depths where high spatial resolution is achieved, lower spatial frequencies are degraded, resulting in inconsistent performance. For example, note the lower contrast at frequencies around 200 lp/mm, between −10 μm to −20 μm off the native object plane. Farther away from the native object plane, the achievable spatial resolution degrades gradually, limiting the effective working range of the microscope (even though it is still much larger compared with a widefield fluorescence microscope).
With the cubic phase mask however, many of these limitations are overcome. The resolution is significantly more uniform around the native object plane and is also slightly improved farther away from it. Moreover, the MTF frequency response at each depth is more monotonic decreasing with frequency. This more uniform resolution profile comes at the expense of peak resolution performance - the peak resolution when using a cubic phase mask reaches only about 64% of the peak resolution of the LFM when no phase mask is used.
4.3. Volumetric reconstructions with the cubic phase mask
We conclude the experimental results by comparing three-dimensional volumes reconstructed from light fields that were captured with and without our experimental objective cubic phase mask. Unlike the planar reconstruction, in volumetric reconstructions, the volume is rendered without any prior knowledge about the z planes where the is specimen is located. Moreover, unlike a USAF 1951 target, in the examples we give the specimen is not planar but is spread over a large range of depths. We imaged with our 20× 0.5NA objective and reconstructed the volumes with a sampling period 4 times smaller than the native sampling period. Figures 6(a) and 6(b) show max projections of a 500 μm × 500 μm × 300 μm volume of 2 μm-size fluorescence beads embedded in agarose, with no phase mask and with the objective cubic phase mask, respectively. Beads were epi-illuminated at 488 nm. Figure 6(c) shows XY and XZ slices of a bead on the native object plane, along with the corresponding one-dimensional cross sections of these slices. The blue curve shows the lateral profile and the red curve shows the axial profile of the bead. Using the phase mask, both lateral and axial resolution is significantly improved, as can be noted from the full width half maximum measurements. In Fig. 6(c) we show the same plots for a bead 70 μm above the native object plane. In this case, the phase mask improves the lateral resolution (blue curves) slightly, but the axial resolution remains very similar.
To illustrate an application that benefits from the improved resolution around the native object plane we also applied our new technique toward in vivo calcium imaging in larval zebrafish, a task which requires volumetric imaging at video frame rates and ability to resolve neurons about 5–10 μm in size across all z depths. We used our 20× 0.5NA objective to image a volume of the same size as before at 3 Hz (the limiting frame rate of our camera). Epi-illumination at 488 nm was used to excite 535 nm fluorescence emission from active neurons in 8 dpf zebrafish expressing calcium indicators pan-neuronally, Tg (elavl3:GCaMP5G) in nacre background.
Two time series were acquired, one with and without the previously described cubic phase mask, and min-subtracted reconstructed volumes from a single time point in each time series are shown in Fig. 7. The improvement in resolution at the native plane, shown in Figs. 7(a) and 7(b), makes it possible to better resolve fluorescent sources located there, which are indistinguishable without the phase mask. This improvement with the cubic phase mask comes without any loss in z-resolution, as illustrated by the XZ maximum projections in Figs. 7(c) and 7(d).
4.4. Simulation results of additional configurations
The experimental results with the cubic phase mask raise an interesting question: whether designing a different objective phase mask, or using a combination of an objective phase mask and microlenses phase masks may result in better overall performance of the wavefront coded LFM. A question that might follow is whether there is an optimal configuration for the wavefront coded LFM that is preferable to all others. Answering these two questions in full is not straightforward, but we take a step in this direction by simulating and analyzing two additional configurations of phase masks: a configuration with a spiral mask at the back focal plane of the objective and a configuration the cubic phase mask at the back focal plane of the objective, coupled with an additional array of cubic phase masks located at the microlenses apertures.
Our spiral phase mask is the continuous version of the phase function proposed in . Instead of dividing the aperture into several annular Fresnel zones with decreasing width, we define the mask in polar coordinates to be
The second configuration we propose includes a cubic masks at the back focal plane of the objective as well as cubic masks at the aperture of the microlenses. Adding phase masks to the microlenses offers additional degrees of freedom and control over the shape of the light field PSF. With a cubic phase mask at its aperture the point spread function of each microlens becomes a cubic PSF. We chose a value of α = 5 for the free parameter of the mask. As we will see, the addition of microlenses phase mask on top of an objective mask can significantly improve the resolution of the LFM around the native object plane. We note that unlike the objective phase mask, which requires adding at least one additional optical element to the optical setup (and may require more if there is a need to create a conjugate plane as is the case in our setup), the microlenses phase masks can be implemented as part of the microlens array by altering the microlenses sag.
Our simulations were conducted in the following way: we created an empty volume and placed the ground truth image of a USAF 1951 target in it, at a certain z-depth. We then applied the forward optical model on the volume g according to Eq. (1) to create a light field image f from it. This synthetic light field image was deconvolved and processed in the same way as the experimental light fields shown in section 4.2. We repeated this procedure, placing the USAF 1951 image at different z depths in our simulated volume for a range of 200 μm around the native object plane of the microscope.
Figure 8 shows the simulation results for the two configurations we propose (see Figs. 8(c) and 8(d)) as well as the standard LFM configuration with no phase masks and our experimental objective cubic phase mask for comparison (see Figs. 8(a) and 8(b) respectively). The simulated results in Figs. 8(a) and 8(b) are in agreement with our experimental results for both these cases, although the experimental results in Fig. 5 reach a lower limiting resolution and exhibit stronger reconstruction artifacts. In Figs. 8(c) and 8(d) we show the reconstructed USAF 1951 target for the combination of cubic masks and for the spiral mask, respectively and in Fig. 8(e) we show the 10% contrast resolution limit, calculated from the reconstructed USAF 1951 stacks for each z-depth. A comparison to the Fisher information performance metric is given in Fig. 8(f).
The performance of the spiral mask (Fig. 8(d)) is the most uniform over the depth range - the resulting limiting resolution is slightly above 250 lp/mm, more than threefold improvement over the native resolution of the LFM (80 lp/mm in our configuration). Even at a distance of ±100 μm away from the native plane the resolution hardly degrades. This uniform performance is expected since the spiral PSF keeps its size and general shape and only rotates around a center point with depth. However, peak resolution is sacrificed even more compared with the cubic phase mask. Even with the lower peak resolution, the spiral phase mask is a good candidate for applications that require uniform resolution performance over a large range of depths.
In contrast, the combination of cubic masks at the objective back focal plane and the microlenses apertures shown in Fig. 8(c) results in a non-uniform resolution profile across depth, with high resolution at the vicinity of the native object plane that decreases farther away from it more rapidly compared with the other configurations. To understand why the addition of these phase mask trades the resolution of the LFM near the native object plane and farther away from it in this manner, we analyze the PSF under a single microlens as a result of point source in the volume located at two different z depths. Figures 9(a) and 9(b) show a single microlens diffraction spot for an objective cubic phase mask and for a combination of objective and microlenses cubic masks respectively, both for a point source 25 μm away from the native object plane. When using only an objective cubic phase mask, the diffraction pattern under one microlens is a large spot and consequently its 1D cross-section MTF (the absolute value of the Fourier transform of the central horizontal row of the 2D diffraction spot), shown in red in Fig. 9(c), cannot support high frequencies. Adding the cubic phase mask at the microlens aperture shapes this spot into a cubic PSF that has an improved frequency response (blue curve in Fig. 9(c)) and can therefore support higher spatial frequencies, resulting in higher spatial resolution in the reconstruction. Farther away from the native object plane at z = 80 μm the opposite is true: the resulting spot when using standard microlenses (Fig. 9(d)) is more compact than the cubic PSF (Fig. 9(e)) and can therefore support higher spatial frequencies, as shown in Fig. 9(f).
4.5. Ray-space analysis of the wavefront coded light field microscope
Further understanding of the effects of using different phase masks can be gained by observing the Cartesian 2D ray-space diagrams for two wavefront coded LFM configurations: (1) an objective cubic phase mask, and (2) microlenses cubic phase masks. Ray-space diagrams show how the detector pixels integrate light rays from different spatial positions on the native object plane, denoted the x plane, and different angles over the objective’s back focal plane, denoted the u plane (note that for the sake of clarity, the diagrams are two dimensional, having only one spatial coordinate and one angular coordinate). The size of the x plane is limited by the objective’s field of view and the size of the u plane is defined as the size of the telecentric stop of the microscope. Although they are based on ray optics and do not consider diffraction effects, we found that these diagrams do provide insight into the benefit of using phase masks to improve the low resolution at the LFM’s native object plane. Figure 10 shows one-dimensional ray-space diagrams along with backward ray-tracing (from the detector plane to the volume) illustrations for an LFM with no phase masks and for the two phase mask configurations mentioned above. The ray tracing illustrations are based on tracing rays through a paraxial model of the LFM in optical design software Zemax.
When no phase masks are used (Fig. 10(a)), rays back-traced from a single pixel (blue rectangle on the detector) are spread to parallel ray bundles by the microlens which are then focused by the tube lens on the u plane. They are expanded again by the objective to parallel bundles that intersect with the x plane. The ray bundles from one pixel cover an area on the u plane equal to the objective’s back aperture size dobj divided by the number of pixels under a microlens; and on the x plane the rays spread over an area of size dμlens/Mobj. Hence, in u − x space (Fig. 10(e)) every detector pixel samples a rectangular area. The ray-space diagram Fig. 10(e) illustrates the low resolution problem at the native object plane of the LFM: a thin isotropically-emitting object on the x plane, with width smaller than dμlens/Mobj as illustrated by the vertical yellow stripe in Fig. 10(e), is only sampled by pixels under a single microlens (same-color rectangles in Fig. 10(e)). In this case, the resulting light field PSF looks like a full disk as shown in Fig. 4(c). Since all the pixels collect light from the same area on the x plane (but from different angles of view), the measurements are redundant and it is not possible to reconstruct the object with higher spatial resolution than the width of that area (i.e. the native sampling period of the LFM), without using prior knowledge about the object. This redundancy is also illustrated in Fig. 3 in  which illustrates the spatial sampling of the volume by rays back-traced from the detector - at z = 0 μm (the native object plane) the ray intersections are sparse, and the plane is therefore sampled only at intervals corresponding to dμlens/Mobj (6.25 μm in this case).
By adding microlens cubic phase masks, we see that rays traced-back from each point on the detector (Fig. 10(b)) are spread by the phase mask and intersect the u plane at positions that form a parabolic profile. The rays intersect the x plane over an area with the same size as in Fig. 10(a). In u − x space (Fig. 10(f)) each pixel covers the same area dμlens/Mobj on the x plane as in Fig. 10(e), but has a parabolic profile in u (this profile is the sum of many shifted parabolic profiles, each produced by a different point inside the pixel’s integration area on the detector). The parabolic curvature depends on the power of the cubic phase masks. It is evident that the pixels that sample the object on the x plane (vertical yellow stripe) still contain redundant information about its position. Therefore, the microlenses phase masks do not solve the native object plane resolution problem.
Placing a cubic phase mask in the objective’s back focal plane (with no microlenses phase masks) however, changes the intersection of rays with the x plane. Backward-traced rays from a every point on the detector (Fig. 10(c)) intersect the u plane at a single point, and are spread on the x plane over the same area size, as is the case in Fig. 10(a). But since groups of rays coming from different points within a pixel area intersect the u plane at different positions, they are refracted differently by the phase mask and are therefore spaced according to a parabolic profile on the x plane. Summing the contributions from all points within a pixel, results in a parabolic profile on the x plane as shown in Fig. 10(g). Our ray-space analysis for this case matches results presented in  and  for a camera system with a cubic phase mask in its main aperture. When using an objective cubic mask, the resulting light field PSF spreads over several microlenses as shown in Fig. 4(b). An object on the native object plane is therefore sampled by pixels under several microlenses, which collect light from different, partially overlapping spatial positions on x. The additional spatial information can be used to reconstruct the object’s position with higher spatial accuracy than the native resolution of the LFM. When using objective and microlenses masks together (Fig. 10(d)), rays are refracted by both masks and result in parabolic profiles on both u and x planes. The resulting sampling pattern of the detector pixels (Fig. 10(h)) shows that even more pixels sample the object on the native object plane compared with just using an objective mask. The greater diversity of positions and angles of rays in the measurement of the native object plane, and the fact that with the cubic phase mask each microlens PSF can support higher spatial frequencies for that plane result in even higher spatial resolution in the reconstruction, as we show in section 4.4.
In this paper, we have presented an extension to the light field microscope that adds phase masks in the back focal plane of its objective and in the microlenses apertures. We have shown how these phase masks can control the shape of the light field PSF and change the microscope’s resolution profile to be much more uniform across depth. We analyzed the resolution of several configurations of our wavefront coded LFM theoretically, experimentally, and in simulations. We have also proposed a method to characterize and optimize the LFM’s performance based on Fisher information that is independent of the deconvolution algorithm used.
Our proposed extension to the LFM allows more control over its resolution profile and suggest that the profile can be tailored to the desired application. In particular, we note that the spiral objective phase mask is especially well-suited for volumetric imaging, where one can obtain very uniform resolution performance over the entire volume while still enjoying the benefits of light field microscopy (namely the ability to capture high speed three-dimensional data with sub-second time dynamics).
There are several future research directions that are worth mentioning. First, we plan to validate our simulated results experimentally, either by fabricating additional objective phase masks and microlens arrays with phase masks or by using a phase spatial light modulator (SLM). The SLM would allow us to implement and test various mask designs quickly. Second, we plan to incorporate the Fisher information metric in a general optimization scheme to search for phase masks that give even better resolution than those we proposed herein. We believe however, that such an optimization scheme should be constrained and directed in order for it to converge to useful mask designs. The comparisons in Fig. 8 and Fig. 9 and the ray-space analysis we presented suggest two constraints for such an optimization scheme in order to obtain high resolution over a wide range of z depths. First, for the entire range of z depths, the extent of the PSF at the native image plane of the LFM has to be large enough so that it sampled by several microlenses, but not too large so that it remains sensitive to the exact position of the point source in the volume. In addition, the PSF should be sufficiently different in shape for different object z depths to be able to support volumetric reconstructions well (as an example, the spiral mask we presented fulfills these requirements). Second, the diffraction pattern created by the microlenses and phase masks should support high spatial frequencies over the entire depth range. It seems that the design principle for optimizing our wavefront coded light field microscope should address the first requirement by optimizing the objective phase mask, since only this mask can affect the extent and shape of the PSF at the native image plane where the microlens array is located. The second requirement can be satisfied by optimizing the microlenses phase masks to control the diffraction pattern at different depths.
Finally, we would also like to use our extended optical model and Fisher information to explore and compare the performance of more general extensions to the microlens array. These might include using different types of phase masks at neighboring microlenses, or replacing the microlens array with an alternative sensing element, such as the coded attenuation mask proposed in  for capturing compressive light fields in a camera setup.
The authors wish to thank the anonymous reviewers for their comments and suggestions to improve the quality of the paper. This work has been supported by NIH Grants #8-RMH099647A and #1160862-101-EHATOV, NSF Grant #IIS-0964218 and Google. S.Y. is supported by NSF Graduate Research fellowship and National Defense Science and Engineering Graduate fellowship, A.A by the Helen Hay Whitney Foundation, M.B by a National Defense Science & Engineering Graduate fellowship and L.G by NSF Integrative Graduate Education and Research Traineeship (IGERT) fellowship.
References and links
1. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in Proceedings of ACM SIGGRAPH (2006), 924–934. [CrossRef]
3. M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K. Deisseroth, and M. Levoy, “Wave optics theory and 3-D deconvolution for the light field microscope,” Opt. Express 21(21), 25418–25439 (2013). [CrossRef] [PubMed]
4. L. Grosenick, T. Anderson, and S. J. Smith, “Elastic source selection for in vivo imaging of neuronal ensembles,” in IEEE Symposium on Biomedical Imaging: From Nano to Macro (ISBI ’09) (2009), pp. 1263–1266.
5. R. Prevedel, Y. G. Yoon, M. Hoffmann, N. Pak, G. Wetzstein, S. Kato, T. Schrödel, R. Raskar, M. Zimmer, E. S. Boyden, and A. Vaziri, “Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy,” Nat. Methods 11(7), 727–730 (2014). [CrossRef] [PubMed]
6. E. R. Dowski Jr. and W. T. Cathey, “Extended depth of field through wave-front coding,” App. Opt. 34(11), 1859–1866 (1995). [CrossRef]
7. S. Abrahamsson, S. Usawab, and M. Gustafssona, “A new approach to extended focus for high-speed, high-resolution biological microscopy,” Proc. SPIE 6090, 60900N (2006). [CrossRef]
8. M. R. Arnison, C. J. Cogswell, C. J. R. Sheppard, and P. Török, “Wavefront coding fluorescence microscopy using high aperture lenses,” in Optical Imaging and Microscopy: Techniques and Advanced Systems, P. Török and F.-J. Kao, eds. (Springer, 2003), pp. 143–165. [CrossRef]
9. A. Doblas, S. V. King, N. Patwary, G. Saavedra, M. Martínez-Corral, and C. Preza, “Investigation of the SQUBIC phase mask design for depth-invariant widefield microscopy point-spread function engineering,” Proc. SPIE 8949, 894914 (2014). [CrossRef]
11. Y. Shuai and C. Preza, “Computational optical sectioning microscopy using an engineered PSF with reduced depth variability - Proof of concept,” in Proc. of the 9th IEEE International Symposium on Biomedical Imaging (2012), pp. 1739–1742.
13. C. H. Lu, S. Muenzel, and J. Fleischer, “High-resolution light-field microscopy,” in Computational Optical Sensing and Imaging, Microscopy and Tomography I (2013), CTh3B.
14. P. Favaro, “A split-sensor light field camera for extended depth of field and superresolution,” Proc. SPIE 8436, 843602 (2012). [CrossRef]
15. M. Gu, Advanced Optical Imaging Theory (Springer, 1999).
16. J. Goodman, Introduction to Fourier Optics, 2nd ed. (MaGraw-Hill, 1996)
17. M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999). [CrossRef]
18. Q. Yang, L Liu, and J Sun, “Optimized phase pupil masks for extended depth of field,” Opt. Commun. 272(1), 56–66 (2007). [CrossRef]
23. A. Levin, S. W. Hasinoff, P. Green, F. Durand, and W. T. Freeman, “4D frequency analysis of computational cameras for depth of field extension,” in Proceedings of ACM SIGGRAPH (2009), 97.
24. Z. Zhengyun and M. Levoy, “Wigner distributions and how they relate to the light field,” in Proc. Int. Conf. Comput. Photography (Apr. 2009), pp. 1–10.
25. K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph. 32(4), 46 (2013). [CrossRef]