1. Introduction

The space-bandwidth product (SBP) metric characterizes information content transmitted through an optical system; it can be thought of as the number of resolvable points in an image (i.e. the system’s field-of-view (FOV) divided by the size of its point spread function (PSF) [1, 2]). Typical microscopes collect images with SBPs of <20 megapixels, a practical limit set by the systems’ optical design and camera pixel count. For large-scale biological studies in systems biology and drug discovery, fast high-SBP imaging is desired [3–10]. The traditional solution for increasing SBP is to use an automated translation stage to scan the sample laterally, then stitch together high-content images. However, such capabilities are costly, have long acquisition times and require careful auto-focusing, due to small depth-of-field (DOF) and axial drift of the sample over large scan ranges [11].

Instead of using high-resolution optics and mechanically scanning the FOV, new approaches for high-content imaging use a low-NA objective (with a large FOV) and build up higher resolution by computationally combining a sequence of low-resolution measurements [12–25]. Such approaches typically illuminate the sample with customized patterns that encode high-resolution sample information into low-resolution features, which can then be measured. These methods reconstruct features smaller than the diffraction limit of the objective, using concepts from synthetic aperture [26–28] and super-resolution (SR) [29–34]. Though the original intent was to maximize resolution, it is important to note that by increasing resolution, SR techniques also increase SBP, and therefore have application in high-content microscopy. Eliminating the requirement for long-distance mechanical scanning means that acquisition is faster and less expensive, while focus requirements are also relaxed by the larger DOF of low-NA objectives.

Existing high-content methods generally use either an incoherent imaging model to reconstruct fluorescence [18–25], or a coherent model to reconstruct absorption and quantitative phase (QP) [12–17]. Both have achieved gigapixel-scale SBP (milli-/centi- meter scale FOV with sub-micron resolution). However, none have demonstrated cross-compatibility with both coherent (phase) and incoherent (fluorescence) imaging. Here, we demonstrate multi-modal high-content imaging via a computational imaging framework that allows super-resolution fluorescence and QP. Our method is based on structured illumination microscopy (SIM), which is compatible with both incoherent [26, 32, 33, 36] and coherent [37–42] sources of contrast [35, 43–45].

Though most SIM implementations have focused on super-resolution, some previous works have recognized its suitability for high-content imaging [18–24]. However, these predominantly relied on fluorescence imaging with calibrated illumination patterns, which are difficult to realize in practice because lens-based illumination has finite SBP. Here, we use random speckle illumination, generated by scattering through Scotch tape, in order to achieve both high-NA and large FOV illumination. Our method is related to blind SIM [46]; however, instead of using many random speckle patterns (which restricts resolution gain to ∼1.8×), we translate the speckle laterally, enabling resolution gains beyond that of previous methods [46–52] (see Appendix D). Previous works also use high-cost spatial-light-modulators (SLM) [53] or galvonemeter/MEMs mirrors [41, 54] for precise illumination, as well as expensive objective lenses for aberration correction. We eliminate both of these requirements by performing computational self-calibration, solving for the translation trajectory and the field-dependent aberrations of the system.

Our proposed framework enables three key advantages over existing methods:

In our experimental setup, the Scotch tape is placed just before the sample and mounted on a translation stage (Fig. 1). This generates disordered speckles at the sample that are much smaller than the PSF of the imaging optics, encoding SR information. Nonlinear optimization methods are then used to jointly reconstruct multiple calibration quantities: the unknown speckle illumination pattern, the translation trajectory of the pattern, and the field-dependent system aberrations (on a patch-by-patch basis). These are subsequently used to decode the SR information of both fluorescence and phase. Compared to traditional SIM systems that use high-NA objective lenses, our system utilizes a low-NA low-cost lens to ensure large FOV. The Scotch tape generated speckle illumination is not resolution-bound by any imaging lens; this is what allows us to achieve 4× resolution gains. The result is high-content imaging at sub-micron resolutions across millimeter scale regions. Various previous works have achieved cost-effectiveness, high-content (large SBP), or multiple modalities, but we believe this to be the first to simultaneously encompass all three.

 

Fig. 1 Structured illumination microscopy (SIM) with laterally-translated Scotch tape as the patterning element, achieving 4× resolution gain. Our imaging system has both an incoherent arm, where Sensor-F captures raw fluorescence images (at the emission wavelength, ${\lambda }_{\text{em}}=605$ nm) for fluorescence super-resolution, and a coherent arm, where Sensor-C1 and Sensor-C2 capture images with different defocus (at the laser illumination wavelength, ${\lambda }_{\text{ex}}=532$ nm) for both super-resolution phase reconstruction and speckle trajectory calibration. OBJ: objective, AP: adjustable iris-aperture, DM: dichroic mirror, SF: spectral filter, ND-F: neutral-density filter.

Download Full Size | PDF

2. Theory

SIM generally achieves super-resolution by illuminating the sample with a high spatial-frequency pattern that mixes with the sample’s information content to form low-resolution "beat" patterns (i.e. moire fringes). Measurements of these "beat" patterns allow elucidation of sample features beyond the diffraction-limited resolution of the imaging system. Maximum achievable resolution in SIM is set by the sum of the numerical apertures (NAs) of the illumination pattern, ${\text{NA}}_{\text{illum}}$, and the imaging system, ${\text{NA}}_{\text{sys}}$. Thus, SIM enables a resolution gain factor (over the system’s native resolution) of $\left({\text{NA}}_{\text{illum}}+{\text{NA}}_{\text{sys}}\right)/{\text{NA}}_{\text{sys}}$ [33]. Theminimum resolvable feature size is inversely related to this bound, $d\propto 1/\left({\text{NA}}_{\text{illum}}+{\text{NA}}_{\text{sys}}\right)$.

Linear SIM typically maximizes resolution by using either: 1) a high-NA objective in epi-illumination configuration, or 2) two identical high-NA objectives in transmission geometry [33, 35]. Both result in a maximum of $2\times$resolution gain because ${\text{NA}}_{\text{illum}}={\text{NA}}_{\text{sys}}$, which corresponds to an SBP increase by a factor of $4\times$. Given the relatively low native SBP of high-NA imaging lenses, such increases are not sufficient to qualify as high-content imaging. Though nonlinear SIM techniques can enable higher resolution gains [34], they require either fluorophore photo-switching or saturation capabilities, which can associate with photobleaching and low SNR, and are not compatible with coherent QP techniques.

In this work, we aim for $>2\times$ resolution gain; hence, we need the illumination NA to be larger than the detection NA, without using a high-resolution illumination lens (that would restrict the illumination FOV). To achieve this, we use a wide-area high-angle scattering element - layered Scotch tape - on the illumination side of the sample (Fig. 1). Multiple scattering within the tape creates a speckle pattern with finer features than the PSF of the imaging system, i.e. ${\text{NA}}_{\text{illum}}>{\text{NA}}_{\text{sys}}$. This means that spatial frequencies beyond 2× the objective’s cutoff are mixed into the measurements, which gives a chance to achieve resolution gains greater than two.

The following sections outline the algorithm that we use to reconstruct large SBP fluorescence and QP images from low-resolution acquisitions of a sample illuminated by a laterally-translating speckle pattern. Unlike conventional SIM reconstruction methods that use analytic linear inversion, our strategy relies instead on joint-variable iterative optimization, where both the sample and illumination speckle (which is unknown) are reconstructed [25, 55, 56].

2.1. Super-resolution fluorescence imaging

Fluorescence imaging requires an incoherent imaging model. The intensity at the sensor is a low-resolution image of the sample’s fluorescent distribution, obeying the system’s incoherent resolution limit, $d_{\text{sys}}={\lambda }_{\text{em}}/2{\text{NA}}_{\text{sys}}$, where ${\lambda }_{\text{em}}$ is the emission wavelength. The speckle pattern generated through the Scotch tape excites the fluorescent sample with features of minimum size $d_{\text{illum}}={\lambda }_{\text{ex}}/2{\text{NA}}_{\text{illum}}$, where ${\lambda }_{\text{ex}}$ is the excitation wavelength and ${\text{NA}}_{\text{illum}}$ is set by the scattering angles exiting the Scotch tape. Approximating the excitation and emission wavelengths as similar ($\lambda ={\lambda }_{\text{ex}}\approx {\lambda }_{\text{em}}$), the resolution limit of the SIM reconstruction is $d_{\text{SIM}}\approx \lambda /2\left({\text{NA}}_{\text{sys}}+{\text{NA}}_{\text{illum}}\right)$, with a resolution gain factor of $d_{\text{sys}}/d_{\text{SIM}}$. This factor is mathematically unbounded; however, it will be practically limited by the illumination NA and SNR (see Appendix D).

2.1.1. Incoherent forward model for fluorescence imaging

Plane-wave illumination of the Scotch tape, positioned at the $\mathcal{l}$-th scan-point, ${\mathbf{r}}_{\mathcal{l}}$, creates a speckle illumination pattern, $p_f\left(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}}\right)$, at the plane of the fluorescent sample, $o_f\left(\mathbf{r}\right)$, where subscript f identifies variables in the fluorescence channel. The fluorescent signal is imaged through the system to give an intensity image at the camera plane:

In this formulation, $o_f\left(\mathbf{r}\right)$, $h_f\left(\mathbf{r}\right)$, and $I_{f,\mathcal{l}}\left(\mathbf{r}\right)$ are 2D $M\times M$-pixel distributions. To accurately model different regions of the pattern translating into the object’s $M\times M$ FOV with incrementing ${\mathbf{r}}_{\mathcal{l}}$, we initialize $p_f\left(\mathbf{r}\right)$ as a $N\times N$ pixel 2D distribution, with $N>M$, and introduce a cropping operator $\mathcal{C}$ to select the $M\times M$ region of the scanning pattern that illuminates the sample.

2.1.2. Inverse problem for fluorescence imaging

We next formulate a joint-variable optimization problem to extract SR estimates of the sample, $o_f\left(\mathbf{r}\right)$, and illumination distributions, $p_f\left(\mathbf{r}\right)$, from the raw fluorescence measurements, $I_{f,\mathcal{l}}\left(\mathbf{r}\right)$, as well as refine the estimate of the system’s PSF [25] (aberrations) and speckle translation trajectory, ${\mathbf{r}}_{\mathcal{l}}$. We start with a crude initialization from raw observations of the speckle made using the coherent imaging arm (more details in Sec. 2.3). Defining $f_f\left(o_f,p_f,h_f,{\mathbf{r}}_1,\dots ,{\mathbf{r}}_{N_{\text{img}}}\right)$ as a joint-variable cost function that measures the difference between the raw intensity acquisitions and the expected intensities from estimated variables via the forward model, we have:

To solve, a sequential gradient descent [57, 58] algorithm is used, where the gradient is updated once for each measurement. The sample, speckle pattern, system’s PSF and scanning positions are updated by sequentially running through N_img measurements within one iteration. After the sequential update, an extra Nesterov’s accelerated update [59] is included for both the sample and pattern estimate, to speed up convergence. Appendix A contains a detailed derivation of the gradient with respect to the sample, structured pattern, system’s PSF and the scanning position based on the linear algebra vectorial notation. The algorithm is described in Appendix B.

2.2. Super-resolution quantitative-phase imaging

In this section, we present our coherent model for SR quantitative-phase (QP) imaging. A key difference between the QP and fluorescence imaging processes is that the detected intensity at the image plane for coherent imaging is nonlinearly related to the sample’s QP [1, 38]. Thus, solving for a sample’s QP from a single intensity measurement is a nonlinear and ill-posed problem. To circumvent this, we use intensity meaurements from two planes, one in-focus and one out-of-focus, to introduce a complex-valued operator that couples QP variations into measurable intensity fluctuations, making the reconstruction well-posed [60, 61]. The defocused measurements are denoted by a new subscript variable z. Figure 1 shows our implementation, where two defocused sensors are positioned at z₀ and z₁ in the coherent imaging arm.

Generally, the resolution for coherent imaging is roughly half that of its incoherent counterpart [1]. For our QP reconstruction, the resolution limit is $d_{\text{SIM}}={\lambda }_{\text{ex}}/({\text{NA}}_{\text{sys}}+{\text{NA}}_{\text{illum}}$), where the coherent resolution of the native system and the speckle are d_sys = λ_ex/NA_sys and d_illum = λ_ex/NA_illum, respectively.

2.2.1. Coherent forward model for phase imaging

Assuming an object with 2D complex transmittance function $o_c\left(\mathbf{r}\right)$ is illuminated by a speckle field, $p_c\left(\mathbf{r}\right)$, where subscript c refers to the coherent imaging channel, positioned at the $\mathcal{l}$-th scanning position ${\mathbf{r}}_{\mathcal{l}}$, we can represent the intensity image formed via coherent diffraction as:

2.2.2. Inverse problem for phase imaging

We now take the raw coherent intensity measurements, $I_{c,\mathcal{l}z}\left(\mathbf{r}\right)$, and the registered trajectory, ${\mathbf{r}}_{\mathcal{l}z}$, from both of the defocused coherent sensors (more details in Sec. 2.3) as input to jointly estimate the sample’s SR complex-transmittance function, $o_c\left(\mathbf{r}\right)$, and illumination complex-field, $p_c\left(\mathbf{r}\right)$, as well as the aberrations inherent in the system’s PSF, $h_c\left(\mathbf{r}\right)$. The optimization also further refines the scanning trajectory, ${\mathbf{r}}_{\mathcal{l}z}$. Based on the forward model, we formulate the joint inverse problem:

Here, we adopt an amplitude-based cost function, f_c, which robustly minimizes the distance between the estimated and measured amplitudes in the presence of noise [57, 61, 62]. We optimize the pattern trajectories, ${\mathbf{r}}_{\mathcal{l},z_0}$ and ${\mathbf{r}}_{\mathcal{l},z_1}$, separately for each coherent sensor, in order to account for any residual misalignment or timing-mismatch (see Sec. 2.3). As in the fluorescence case, sequential gradient descent [57, 58] was used to solve this inverse problem.

2.3. Registration of coherent images

Knowledge of the Scotch tape scanning position, ${\mathbf{r}}_{\mathcal{l}}$, reduces the complexity of the joint sample and pattern estimation problem and is necessary to achieve SR reconstructions with greater than $2\times$ resolution gain. Because our fluorescent sample is mostly transparent, the main scattering component in the acquired raw data originates from the Scotch tape. Thus, using a sub-pixel registration algorithm [63] between successive coherent-camera acquisitions, which are dominated by the scattered speckle signal, is sufficient to initialize the scanning trajectory of the Scotch tape,

3. Experimental results

Figure 1 shows our experimental setup. A green laser beam (BeamQ, 532 nm, 200 mW) is collimated through a single lens. The resulting plane wave illuminates the layered Scotch tape (4 layers of 3M 810 Scotch Tape, S-9783), creating a speckle pattern at the sample. The Scotch tape is mounted on a 3-axis piezo-stage (Thorlabs, MAX311D) to enable lateral speckle scanning. The transmitted light from the sample then travels through a 4f system formed by the objective lens (OBJ) and a single lens. In order to control the NA of our detection system (necessary for our verification experiment), an extra 4f system with an adjustable iris-aperture (AP) in the Fourier space is added. Then, the coherent and fluorescent light are optically separated bya dichroic mirror (DM, Thorlabs, DMLP550R), since they have different wavelengths. The fluorescence is further spectrally filtered (SF) before imaging onto Sensor-F (PCO.edge 5.5). The (much brighter) coherent light is ND-filtered and then split by another beam-splitter before falling on the two defocused coherent sensors, Sensor-C1 and Sensor-C2 (FLIR, BFS-U3-200S6M-C). Sensor-C1 is focused on the sample, while Sensor-C2 is defocused by 0.8 mm.

For our initial verification experiments, we use a 40× objective (Nikon, CFI Achro 40×) with $\text{NA}=0.65$ as our system’s microscope objective (OBJ). Later high-content experimental demonstrations switch to a 4× objective (Nikon, CFI Plan Achro 4×) with $\text{NA}=0.1$.

3.1. Super-resolution verification

3.1.1. Fluorescence super-resolution verification

We start with a proof-of-concept experiment to verify that our method accurately reconstructs a fluorescent sample at resolutions greater than twice the imaging system’s diffraction-limit. To do so, we use the higher-resolution objective (40×, NA 0.65) and a tunable Fourier-space iris-aperture (AP) that allows us to artificially reduce the system’s NA (${\text{NA}}_{\text{sys}}$), and therefore, resolution. With the aperture mostly closed (to ${\text{NA}}_{\text{sys}}=0.1$), we acquire a low-resolution SIM dataset, which is then used to computationally reconstruct a super-resolved image of the sample with resolution corresponding to an effective NA = 0.4. This reconstruction is then compared to the widefield image of the sample acquired with the aperture open to ${\text{NA}}_{\text{sys}}=0.4$, for validation.

 

Fig. 2 Verification of fluorescence super-resolution with 4× resolution gain. Widefield images, for comparison, were acquired at (a) 0.1 NA and (e) 0.4 NA by adjusting the aperture size. (b) The Scotch tape speckle pattern creates much higher spatial frequencies (∼0.35 NA) than the 0.1 NA detection system can measure. (c) Using the 0.1 NA aperture, we acquire low-resolution fluorescence images for different lateral positions of the Scotch tape. (d) The reconstructed SIM image contains spatial frequencies up to ∼0.4 NA and is in agreement with (e) the deconvolved widefield image with the system operating at 0.4 NA.

Download Full Size | PDF

Results are shown in Fig. 2, comparing our method against widefield fluorescence images at NAs of 0.1 and 0.4, with no Scotch tape in place. The sample is a monolayer of 1 μm diameter microspheres, with center emission wavelength ${\lambda }_{\text{em}}=605$ nm. At 0.1 NA, the expected resolution is ${\lambda }_{\text{em}}/2$NA $\approx$ 3.0 μm and the microspheres are completely unresolvable. At 0.4 NA, the expected resolution is ${\lambda }_{\text{em}}/2$NA $\approx$ 0.76 μm and the microspheres are well-resolved. With Scotch tape and 0.1 NA, we acquire a set of measurements as we translate the speckle pattern in 267 nm increments on a 26 × 26 rectangular grid - $N_{\text{img}}=676$ acquisitions total (details in Sec. 4).

Figure 2(d) shows the final SR reconstruction of the fluorescent sample in real space, along with the amplitude of its Fourier spectrum. Individual microspheres can be clearly resolved, and results match well with the 0.4 NA deconvolved widefield image (Fig. 2(e)). Fourier-space analysis confirms our resolution improvement factor to be 4×, which suggests that the Scotch tape produces ${\text{NA}}_{\text{illum}}\approx 0.3$. To verify, we fully open the aperture and observe that the speckle pattern contains spatial frequencies up to ${\text{NA}}_{\text{illum}}\approx 0.35$ (Fig. 2(b)).

3.1.2. Coherent super-resolution verification

To quantify super-resolution in the coherent imaging channel, we use the low-resolution objective (4×, NA 0.1) to image a USAF1951 resolution chart (Benchmark Technologies). This phase target provides different feature sizes with known phase values, so is a suitable calibration target to quantify both the coherent resolution and the phase sensitivity of our technique.

Results are shown in Fig. 3. The coherent intensity image (Fig. 3(a)) acquired with 0.1 NA (no tape) has low resolution ($\sim 5.32\mu$m), so hardly any features can be resolved. In Fig. 3(b), we show the “ground truth” QP distribution at 0.4 NA, as provided by the manufacturer.

 

Fig. 3 Verification of coherent quantitative phase (QP) super-resolution with 4× resolution gain. (a) Low-resolution intensity image and (b) “ground truth” phase at NA=0.4, for comparison. (c) Raw acquisitions of the speckle-illuminated sample intensity from two focus planes, collected with 0.1 NA. (d) Reconstructed SR amplitude and QP, demonstrating 4× resolution gain.

Download Full Size | PDF

After inserting the Scotch tape, it was translated in 400 nm increments on a 36 × 36 rectangular grid, giving $N_{\text{img}}=1296$ total acquisitions (details in Sec. 4) at each of the two defocused coherent sensors (Fig. 3(c)). Figure 3(d,e) shows the SR reconstruction for the amplitude and phase of this sample, resolving features up to group 9 element 5 (1.23 μm separation). Thus, our coherent reconstruction has a $\sim 4\times$ resolution gain compared to the brightfield intensity image.

3.2. High-content multi-modal microscopy

Of course, artificially reducing resolution in order to validate our method required using a moderate-NA objective, which precluded imaging over the large FOVs allowed by low-NA objectives. In this section, we demonstrate high-content fluorescence imaging with the low-resolution, large FOV objective (4×, NA 0.1) to visualize a 2.7×3.3 mm${ }^2$ FOV (see Fig. 4(a)). We note that this FOV is more than 100× larger than that allowed by the 40× objective used in the verification experiments, so is suitable for large SBP imaging.

Within the imaged FOV for our 1 μm diameter microsphere monolayer sample, we zoom in to four regions-of-interest (ROI), labeled ①, ②, ③, and ④. Widefield fluorescence imaging cannot resolve individual microspheres, as expected. Using our method, however, gives a factor 4× resolution gain across the whole FOV and enables resolution of individual microspheres. Thus, the SBP of the system, natively ∼5.3 mega-pixels of content, was increased to ∼85 mega-pixels, a factor of $4^2=16\times$. Though this is still not in the Gigapixel range, this technique is scalable and could reach that range with a higher-SBP objective and sensors.

We next include the QP imaging channel to demonstrate high-content multimodal imaging, as shown in Fig. 5. The multimodal FOV is smaller (2×2.7 mm${ }^2$ FOV) than that presented in Fig. 4 because our coherent detection sensors have a lower pixel-count than our fluorescence detection sensor. Figure 5 includes zoom-ins of three ROIs to visualize the multimodal SR.

As expected, the widefield fluorescence image and the on-axis coherent intensity image do not allow resolution of individual 2 μm microspheres, since the theoretical resolution for fluorescence imaging is ${\lambda }_{\text{em}}/2{\text{NA}}_{\text{sys}}\approx 3\mu$m and for QP imaging is ${\lambda }_{\text{ex}}/{\text{NA}}_{\text{sys}}\approx 5\mu$m. However, our SIM reconstruction with $4\times$ resolution gain enables clear separation of the microspheres in both channels. Our fluorescence and QP reconstructions match well, which is expected since the fluorescent and QP signal originate from identical physical structures in this particular sample.

The full-FOV reconstructions (Fig. 4 and 5) are obtained by dividing the FOV into small patches, reconstructing each patch, then stitching together the high-content images. Patch-wise reconstruction is computationally favorable because of its low-memory requirement, but also allows us to correct field-dependent aberrations. Since we process each patch separately using our self-calibration algorithm, we solve for each patch’s PSF independently and correct the local aberrations digitally. The reconstruction takes approximately 15 minutes for each channel on a high-end GPU (NVIDIA, TITAN Xp) for a patch with FOV of $110\times 110\mu$ m${ }^2$.

 

Fig. 4 Reconstructed super-resolution fluorescence with 4× resolution gain across the full FOV (See Visualization 1). Four zoom-ins of regions-of-interest (ROIs) are compared to their widefield counterparts.

Download Full Size | PDF

 

Fig. 5 Reconstructed multimodal (fluorescence and quantitative phase) high-content imaging (See Visualization 2 and Visualization 3). Zoom-ins for three ROIs compare the widefield, super-resolved fluorescence, coherent intensity, and super-resolved phase reconstructions.

Download Full Size | PDF

4. Discussion

Unlike many existing high-content imaging techniques, one benefit of our method is its easy compatibility for simultaneous QP and fluorescence imaging. This arises from SIM’s unique ability to multiplex both coherent and incoherent signals into the system aperture [35]. Furthermore, existing high-content fluorescence imaging techniques that use micro-lens arrays [18–23] are resolution-limited by the physical size of the lenslets, which typically have ${\text{NA}}_{\text{illum}}<0.3$. Recent work [24] has introduced a framework in which gratings with sub-diffraction slits allow sub-micron resolution across large FOVs - however, this work is heavily limited by SNR, due to the primarily opaque grating, as well as tight required axial alignment. Though the Scotch tape used in our proof-of-concept prototype also induced illumination angles within a similar range as micro-lens arrays (${\text{NA}}_{\text{illum}}\approx 0.35$), we could in future use a stronger scattering media to achieve ${\text{NA}}_{\text{illum}}\approx 1.0$, enabling further SR and thus larger SBP.

The main drawback of our technique is that we use around $\sim 1200$ translations of the Scotch tape for each reconstruction, which results in long acquisition times (∼ 180 seconds for shifting, pausing, and capturing) and higher photon requirements. Heuristically, for both fluorescence and QP imaging, we found that a sufficiently large scanning range (larger than $\sim 2$ low-NA diffraction limited spot sizes) and finer scan steps (smaller than the targeted resolution) can reduce distortions in the reconstruction. Tuning such parameters to minimize the number of acquisitions without degrading reconstruction quality is thus an important subject for future endeavors.

5. Conclusion

We have presented a large-FOV multimodal SIM fluorescence and QP imaging technique. We use Scotch tape to efficiently generate high-resolution features over a large FOV, which can then be measured with both fluorescent and coherent contrast using a low-NA objective. A computational optimization-based self-calibration algorithm corrected for experimental uncertainties (scanning-position, aberrations, and random speckle pattern) and enabled super-resolution fluorescence and quantitative phase reconstruction with factor $4\times$ resolution gain.

Appendix A: Gradient derivation

A.1. Vectorial notation

A.1.1. Fluorescence imaging vectorial model

In order to solve the multivariate optimization problem in Eq. (2) and (4) and derive the gradient of the cost function, it is more convenient to consider a linear algebra vectorial notation of the forward models. The fluorescence SIM forward model in Eq. (1) can be alternatively expressed as

(6)$${\mathbf{I}}_{f,\mathcal{l}}={\mathbf{H}}_f\text{diag}\left(\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f\right){\mathbf{o}}_f,$$

where ${\mathbf{I}}_{f,\mathcal{l}}$, ${\mathbf{H}}_f$, $\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right)$, ${\mathbf{p}}_f$, and ${\mathbf{o}}_f$ designate the raw fluorescent intensity vector, diffraction-limit low-pass filtering operation, pattern translation/cropping operation, $N^2\times 1$ speckle pattern vector, and $M^2\times 1$ sample’s fluorescent distribution vector, respectively. The 2D-array variables described in (1) are all reshaped into column vectors here. ${\mathbf{H}}_f$ and $\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right)$ can be further broken down into their individual vectorial components:

(7)$$\begin{array}{l}{\mathbf{H}}_f={\mathbf{F}}_M^{-1}\text{diag}\left({\tilde{\mathbf{h}}}_f\right){\mathbf{F}}_M,\\ \mathbf{S}({\mathbf{r}}_{\mathcal{l}})=\mathbf{Q}{\mathbf{F}}_N^{-1}\text{diag}(\mathbf{e}({\mathbf{r}}_{\mathcal{l}})){\mathbf{F}}_N,\end{array}$$

where ${\tilde{\mathbf{h}}}_f$ is the OTF vector and $\mathbf{e}\left({\mathbf{r}}_{\mathcal{l}}\right)$ is the vectorization of the $\exp (-j2\pi \mathbf{u}\cdot {\mathbf{r}}_{\mathcal{l}})$ function, where u is spatial frequency. The notation $\text{diag}\left(\mathbf{a}\right)$ turns a $n\times 1$ vector, a, into an $n\times n$ diagonal matrix with diagonal entries from the vector entries. ${\mathbf{F}}_N$ and ${\mathbf{F}}_M$ denote the $N\times N$-point and $M\times M$-point 2D discrete Fourier transform matrix, respectively, and Q is the $M^2\times N^2$ cropping matrix.

With this vectorial notation, the cost function for a single fluorescence measurement is

(8)$$f_{f,\mathcal{l}}({\mathbf{o}}_f,{\mathbf{p}}_f,{\tilde{\mathbf{h}}}_f,{\mathbf{r}}_{\mathcal{l}})={\mathbf{f}}_{f,\mathcal{l}}^T{\mathbf{f}}_{f,\mathcal{l}}={\Vert {\mathbf{I}}_{f,\mathcal{l}}-{\mathbf{H}}_f\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}}){\mathbf{p}}_f\right){\mathbf{o}}_f\Vert }_2^2,$$

where ${\mathbf{f}}_{f,\mathcal{l}}={\mathbf{I}}_{f,\mathcal{l}}-{\mathbf{H}}_f\text{diag}\left(\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f\right){\mathbf{o}}_f$ is the cost vector and T denotes the transpose operation.

A.1.2. Coherent imaging vectorial model

As with the fluorescence vectorial model, we can rewrite Eq. (3) using vectorial notation:

(9)$${\mathbf{I}}_{c,\mathcal{l}z}={\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}^2,$$

where

(10)$$\begin{array}{l}{\mathbf{g}}_{c,\mathcal{l}z}={\mathbf{H}}_{c,z}\text{diag}(\mathbf{S}({\mathbf{r}}_{\mathcal{l}z}){\mathbf{p}}_c){\mathbf{o}}_c\\ {\mathbf{H}}_{c,z}={\mathbf{F}}_M^{-1}\text{diag}({\tilde{\mathbf{h}}}_c)\text{diag}({\tilde{\mathbf{h}}}_z){\mathbf{F}}_M.\end{array}$$

${\mathbf{o}}_c$ and ${\mathbf{p}}_c$ are the $M^2\times 1$ sample transmittance function vector and $N^2\times 1$ structured field vector, respectively. ${\tilde{\mathbf{h}}}_c$ and ${\tilde{\mathbf{h}}}_z$ are the system pupil function and the deliberate defocus pupil function, respectively. With this vectorial notation, we can then express the cost function for a single coherent intensity measurement as

(11)$$f_{c,\mathcal{l}z}\left({\mathbf{o}}_c,{\mathbf{p}}_c,{\tilde{\mathbf{h}}}_c,{\mathbf{r}}_{\mathcal{l}z}\right)={\mathbf{f}}_{c,\mathcal{l}z}^T{\mathbf{f}}_{c,\mathcal{l}z}=\sqrt{{\mathbf{I}}_{c,\mathcal{l}z}}-{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}_2^2,$$

where ${\mathbf{f}}_{c,\mathcal{l}z}=\sqrt{{\mathbf{I}}_{c,\mathcal{l}z}}-\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|$ is the cost vector for the coherent intensity measurement.

A.2. Gradient derivation

A.2.1. Gradient derivation for fluorescence imaging

To optimize Eq. (2) for the variables ${\mathbf{o}}_f$, ${\mathbf{p}}_f$, ${\tilde{\mathbf{h}}}_f$ and ${\mathbf{r}}_{\mathcal{l}}$, we first derive the necessary gradients of the fluorescence cost function. Consider taking the gradient of $f_{f,l}$ with respect to ${\mathbf{o}}_f$, we can represent the $1\times M^2$ gradient row vector as

(12)$$\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{o}}_f}=\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{f}}_{f,\mathcal{l}}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{f,\mathcal{l}}}{\partial {\mathbf{o}}_f}\right)=\left(2{\mathbf{f}}_{f,\mathcal{l}}^T\right)\cdot \le ft(-{\mathbf{H}}_f\text{diag}\left(\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f\right).$$

Turning the row gradient vector into a $M^2\times 1$ column vector in order to update the object vector in the right dimension, we the final gradient becomes

(13)$${\nabla }_{{\mathbf{o}}_f}{f}_{f,\mathcal{l}}={\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{o}}_f}\right)}^T=-2\text{diag}\left(\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f\right){\mathbf{H}}_f^T{\mathbf{f}}_{f,\mathcal{l}}.$$

To compute the gradient of ${\mathbf{p}}_f$, we first rewrite the cost vector ${\mathbf{f}}_{f,\mathcal{l}}$ as

(14)$${\mathbf{f}}_{f,\mathcal{l}}={\mathbf{I}}_{f,\mathcal{l}}-{\mathbf{H}}_f\text{diag}\left(\mathbf{o}\right)\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f.$$

Now, we can write the gradient of the cost function with respect to the pattern vector in row and column vector form as

(15)$$\begin{array}{l}\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{p}}_f}=\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{f}}_{f,\mathcal{l}}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{f,\mathcal{l}}}{\partial {\mathbf{p}}_f}\right)=\left(2{\mathbf{f}}_{f,\mathcal{l}}^T\right)\cdot \left(-{\mathbf{H}}_f\text{diag}\left({\mathbf{o}}_f\right)\mathbf{S}({\mathbf{r}}_{\mathcal{l}})\right)\\ {\nabla }_{{\mathbf{p}}_f}{f}_{f,\mathcal{l}}={\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{p}}_f}\right)}^T=-2\mathbf{S}{({\mathbf{r}}_{\mathcal{l}})}^T\text{diag}\left({\mathbf{o}}_f\right){\mathbf{H}}_f^T{\mathbf{f}}_{f,\mathcal{l}}.\end{array}$$

Similar to the derivation of the pattern function gradient, it is easier to work with the rewritten form of the cost vector expressed as

(16)$${\mathbf{f}}_{f,\mathcal{l}}={\mathbf{I}}_{f,\mathcal{l}}-{\mathbf{F}}_M^{-1}\text{diag}\left({\mathbf{F}}_M\text{diag}\left(\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}}\right){\mathbf{p}}_f{\mathbf{o}}_f\right)\right){\tilde{\mathbf{h}}}_f.$$

The gradient of the cost function with respect to the OTF vector in the row and column vector form are expressed, respectively, as

(17)$$\begin{array}{l}\frac{\partial f_{f,\mathcal{l}}}{\partial {\tilde{\mathbf{h}}}_f}=\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{f}}_{f,\mathcal{l}}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{f,\mathcal{l}}}{\partial {\tilde{\mathbf{h}}}_f}\right)=\left(2{\mathbf{f}}_{f,\mathcal{l}}^T\right)\cdot \left(-{\mathbf{F}}_M^{-1}\text{diag}\left({\mathbf{F}}_M\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}}){\mathbf{p}}_f{\mathbf{o}}_f\right)\right)\right)\\ {\nabla }_{{\tilde{\mathbf{h}}}_f}{f}_{f,\mathcal{l}}={\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\tilde{\mathbf{h}}}_f}\right)}^{†}=-2\text{diag}\left(\overline{{\mathbf{F}}_M\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}}){\mathbf{p}}_f{\mathbf{o}}_f\right)}\right){\mathbf{F}}_M{\mathbf{f}}_{f,\mathcal{l}},\end{array}$$

where $\overline{\mathbf{a}}$ denotes entry-wise complex conjugate operation on any general vector a. One difference between this gradient and the previous one is that the variable to solve, ${\tilde{\mathbf{h}}}_f$, is now a complex vector. When turning the gradient row vector of a complex vector into a column vector, we have to take a Hermitian operation, $†$, on the row vector following the conventions in [64]. We will have more examples of complex variables in the coherent model gradient derivation.

For taking the gradient of the scanning position, we again rewrite the cost vector ${\mathbf{f}}_{f,\mathcal{l}}$:

(18)$${\mathbf{f}}_{f,\mathcal{l}}={\mathbf{I}}_{\mathcal{l}}-{\mathbf{H}}_f\text{diag}\left({\mathbf{o}}_f\right)\mathbf{Q}{\mathbf{F}}_N^{-1}\text{diag}\left({\mathbf{F}}_N{\mathbf{p}}_f\right)\mathbf{e}\left({\mathbf{r}}_{\mathcal{l}}\right).$$

We can then write the gradient of the cost function with respect to the scanning position as

(19)$$\begin{array}{c}\frac{\partial f_{f,\mathcal{l}}}{\partial q_{\mathcal{l}}}=\left(\frac{\partial f_{f,\mathcal{l}}}{\partial {\mathbf{f}}_{f,\mathcal{l}}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{f,\mathcal{l}}}{\partial \mathbf{e}({\mathbf{r}}_{\mathcal{l}})}\right)\cdot \left(\frac{\partial \mathbf{e}({\mathbf{r}}_{\mathcal{l}})}{\partial q_{\mathcal{l}}}\right)\\ =\left(2{\mathbf{f}}_{f,\mathcal{l}}^T\right)\cdot \left(-{\mathbf{H}}_f\text{diag}\left({\mathbf{o}}_f\right)\mathbf{Q}{\mathbf{F}}_N^{-1}\text{diag}\left({\mathbf{F}}_N{\mathbf{p}}_f\right)\right)\cdot \left(\text{diag}\left(-j2\pi {\mathbf{u}}_q\right)\mathbf{e}({\mathbf{r}}_{\mathcal{l}})\right),\end{array}$$

where q is either the x or y spatial coordinate component of ${\mathbf{r}}_{\mathcal{l}}$. ${\mathbf{u}}_q$ is the $N^2\times 1$ vectorial notation of the spatial frequency function in the q direction.

To numerically evaluate these gradients, we represent them in the functional form as:

(20)$$\begin{array}{l}{\nabla }_{o_f}{f}_{f,\mathcal{l}}(o_f,p_f,h_f,{\mathbf{r}}_{\mathcal{l}})=-2p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\cdot \left[h_f^{*}(-\mathbf{r})\otimes \left(I_{f,\mathcal{l}}(\mathbf{r})-\left[o_f(\mathbf{r})\cdot \mathcal{C}\{p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\}\right]\otimes h_f(\mathbf{r})\right)\right],\\ {\nabla }_{p_f}{f}_{f,\mathcal{l}}(o_f,p_f,h_f,{\mathbf{r}}_{\mathcal{l}})=-2\mathrm{\delta }(\mathbf{r}+{\mathbf{r}}_{\mathcal{l}})\otimes \mathcal{P}\left\{o_f(\mathbf{r})\cdot \left[h_f^{*}(-\mathbf{r})\otimes \left(I_{f,\mathcal{l}}(\mathbf{r})-\left[o_f(\mathbf{r})\cdot \mathcal{C}\{p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\}\right]\otimes h_f(\mathbf{r})\right)\right]\right\},\\ {\nabla }_{{\tilde{h}}_f}{f}_{f,\mathcal{l}}(o_f,p_f,h_f,{\mathbf{r}}_{\mathcal{l}})=-2{\left(\mathcal{F}\left\{o_f(\mathbf{r})\cdot \mathcal{C}\left\{p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\right\}\right\}\right)}^{*}\cdot \mathcal{F}\left\{I_{f,\mathcal{l}}(\mathbf{r})-\left[o_f(\mathbf{r})\cdot \mathcal{C}\left\{p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\right\}\right]\otimes h_f(\mathbf{r})\right\},\\ {\nabla }_{q_{\mathcal{l}}}{f}_{f,\mathcal{l}}(o_f,p_f,h_f,{\mathbf{r}}_{\mathcal{l}})=-2\{{\displaystyle \sum _{\mathbf{r}}\left(I_{f,\mathcal{l}}(\mathbf{r})-\left[o_f(\mathbf{r})\cdot \mathcal{C}\{p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\}\right]\otimes h_f(\mathbf{r})\right)}\cdot h_f(\mathbf{r})\otimes \left[o_f(\mathbf{r})\cdot \mathcal{C}\left\{\frac{\partial p_f(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})}{\partial q_{\mathcal{l}}}\right\}\right]\},\end{array}$$

where $a^{*}$ stands for complex conjugate of any general function, a, $\mathcal{F}$ is the Fourier transform operator, and $\mathcal{P}$ is a zero-padding operator that pads an $M\times M$ image to size $N\times N$ pixels. In this form, $I_{f,\mathcal{l}}($r)$,o_f\left(\mathbf{r}\right)$, and $h_f\left(\mathbf{r}\right)$ are 2D $M\times M$ images, while $p_f\left(\mathbf{r}\right)$ is a $N\times N$ image. The gradients for the sample and the structured pattern are of the same size as $o_f\left(\mathbf{r}\right)$ and $p_f\left(\mathbf{r}\right)$, respectively. Ideally, the gradient of the the scanning position in each direction is a real number. However, due to imperfect implementation of the discrete differentiation in each direction, the gradient will have small imaginary value that will be dropped in the update of the scanning position.

A.2.2. Gradient derivation for coherent imaging

For the coherent imaging case, we will derive the gradients of the cost function in Eq. (11) with respect to the sample transmittance function ${\mathbf{o}}_c$, speckle field ${\mathbf{p}}_c$, pupil function ${\tilde{\mathbf{h}}}_c$, and the scanning position ${\mathbf{r}}_{\mathcal{l}z}$. First, we take the gradient of $f_{c,\mathcal{l}z}$ with respect to ${\mathbf{o}}_c$, we then have the gradient in the row and column vector forms as

(21)$$\begin{array}{c}\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{o}}_c}=\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{f}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial {\mathbf{o}}_c}\right)\\ =\left(2{\mathbf{f}}_{c,\mathcal{l}z}^T\right)\cdot \left(-\frac{1}{2}\text{diag}\left(\frac{\overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right)\right)\cdot \left({\mathbf{H}}_{c,z}\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}z})\right){\mathbf{p}}_c\right)\\ {\nabla }_{{\mathbf{o}}_c}{f}_{c,\mathcal{l}z}={\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{o}}_c}\right)}^{†}=-\text{diag}(\overline{\mathbf{S}({\mathbf{r}}_{\mathcal{l}z}){\mathbf{p}}_c}){\mathbf{H}}_{c,z}^{†}\text{diag}\left(\frac{{\mathbf{g}}_{c,\mathcal{l}z}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right){\mathbf{f}}_{c,\mathcal{l}z},\end{array}$$

where the $\frac{{\mathbf{g}}_{c,\mathcal{l}z}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}$ operation denotes entry-wise division between the two vectors, ${\mathbf{g}}_{c,\mathcal{l}z}$ and $\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|$. In addition, the detailed calculation of $\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}$ can be found in the Appendix of [57].

Next, we take the gradient with respect to the pattern field vector, ${\mathbf{p}}_c$, and write down the corresponding row and column vectors as

(22)$$\begin{array}{c}\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{p}}_c}=\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{f}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial {\mathbf{p}}_c}\right)\\ =\left(2{\mathbf{f}}_{c,\mathcal{l}z}^T\right)\cdot \left(-\frac{1}{2}\text{diag}\left(\frac{\overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right)\right)\cdot \left({\mathbf{H}}_{c,z}\text{diag}\left({\mathbf{o}}_c\right)\mathbf{S}({\mathbf{r}}_{\mathcal{l}z})\right)\\ {\nabla }_{{\mathbf{p}}_c}{f}_{c,\mathcal{l}z}={\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{p}}_c}\right)}^{†}=-\mathbf{S}{({\mathbf{r}}_{\mathcal{l}z})}^{†}\text{diag}\left(\overline{{\mathbf{o}}_c}\right){\mathbf{H}}_{c,z}^{†}\text{diag}\left(\frac{{\mathbf{g}}_{c,\mathcal{l}z}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right){\mathbf{f}}_{c,\mathcal{l}z}.\end{array}$$

In order to calculate $\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial {\mathbf{p}}_c}$, we need to reorder the dot multiplication of ${\mathbf{o}}_c$ and $\mathbf{S}\left({\mathbf{r}}_{\mathcal{l}z}\right){\mathbf{p}}_c$ as we did in deriving the gradient of the pattern for fluorescence imaging.

In order to do aberration correction, we will need to estimate the system pupil function, ${\tilde{\mathbf{h}}}_c$. The gradient with respect to the pupil function can be derived as,

(23)$$\begin{array}{c}\frac{\partial f_{c,\mathcal{l}z}}{\partial {\tilde{\mathbf{h}}}_c}=\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{f}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial {\tilde{\mathbf{h}}}_c}\right)\\ =\left(2{\mathbf{f}}_{c,\mathcal{l}z}^T\right)\cdot \left(-\frac{1}{2}\text{diag}\left(\frac{\overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right)\right)\cdot \left({\mathbf{F}}_M^{-1}\text{diag}\left[{\mathbf{F}}_M\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}z}){\mathbf{p}}_c\right){\mathbf{o}}_c\right]\text{diag}({\tilde{\mathbf{h}}}_z)\right)\\ {\nabla }_{{\tilde{\mathbf{h}}}_c}{f}_{c,\mathcal{l}z}={\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\tilde{\mathbf{h}}}_c}\right)}^{†}=-\text{diag}(\overline{{\tilde{\mathbf{h}}}_z})\text{diag}\left[\overline{{\mathbf{F}}_M\text{diag}\left(\mathbf{S}({\mathbf{r}}_{\mathcal{l}z}){\mathbf{p}}_c\right){\mathbf{o}}_c}\right]{\mathbf{F}}_M\text{diag}\left(\frac{{\mathbf{g}}_{c,\mathcal{l}z}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right){\mathbf{f}}_{c,\mathcal{l}z}.\end{array}$$

In the end, the gradient of the scanning position for refinement can be derived as

(24)$$\begin{array}{l}\frac{\partial f_{c,\mathcal{l}z}}{\partial q_{\mathcal{l}z}}=\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{f}}_{c,\mathcal{l}z}}\right)\cdot \left[\left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial \mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}\right)\cdot \left(\frac{\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}{\partial q_{\mathcal{l}}}\right)+\left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial \overline{{\mathbf{g}}_{c,\mathcal{l}z}}}\right)\cdot \left(\frac{\partial \overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\partial \overline{\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}}\right)\cdot \left(\frac{\overline{\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}}{\partial q_{\mathcal{l}}}\right)\right]\\ =2\left(\frac{\partial f_{c,\mathcal{l}z}}{\partial {\mathbf{f}}_{c,\mathcal{l}z}}\right)\cdot \text{Re}\left\{\left(\frac{\partial {\mathbf{f}}_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{c,\mathcal{l}z}}\right)\cdot \left(\frac{\partial {\mathbf{g}}_{c,\mathcal{l}z}}{\partial \mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}\right)\cdot \left(\frac{\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})}{\partial q_{\mathcal{l}}}\right)\right\}\\ =2\left(2{\mathbf{f}}_{c,\mathcal{l}z}^T\right)\cdot \text{Re}\left\{\left(-\frac{1}{2}\text{diag}\left(\frac{\overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right)\right)\cdot \left({\mathbf{H}}_{c,z}\text{diag}\left({\mathbf{o}}_c\right)\mathbf{Q}{\mathbf{F}}_N^{-1}\text{diag}\left({\mathbf{F}}_N{\mathbf{p}}_c\right)\right)\cdot \left(\text{diag}\left(-j2\mathrm{\pi }{\mathbf{u}}_q\right)\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})\right)\right\}\\ =-2\text{Re}\left\{{\mathbf{f}}_{c,\mathcal{l}z}^T\text{diag}\left(\frac{\overline{{\mathbf{g}}_{c,\mathcal{l}z}}}{\left|{\mathbf{g}}_{c,\mathcal{l}z}\right|}\right){\mathbf{H}}_{c,z}\text{diag}({\mathbf{o}}_c)\mathbf{Q}{\mathbf{F}}_N^{-1}\text{diag}({\mathbf{F}}_N{\mathbf{p}}_c)\text{diag}(-j2\mathrm{\pi }{\mathbf{u}}_q)\mathbf{e}({\mathbf{r}}_{\mathcal{l}z})\right\},\end{array}$$

where q is either the x or y spatial coordinate component of ${\mathbf{r}}_{\mathcal{l}z}$.

In order to numerically evaluate these gradients, we represent them, as we did for the gradients of the fluorescence model, into functional forms:

(25)$$\begin{array}{c}{\nabla }_{o_c}{f}_{c,\mathcal{l}z}(o_c,p_c,h_c,{\mathbf{r}}_{\mathcal{l}z})=-p_c^{*}(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}z})\cdot \left[h_{c,z}^{*}(-\mathbf{r})\otimes \left(\left(\frac{\sqrt{I_{c,\mathcal{l}z}(\mathbf{r})}}{|g_{c,\mathcal{l}z}(\mathbf{r})|}-1\right)\cdot g_{c,\mathcal{l}z}(\mathbf{r})\right)\right]\\ {\nabla }_{p_c}{f}_{c,\mathcal{l}z}(o_c,p_c,h_c,{\mathbf{r}}_{\mathcal{l}z})=-\mathrm{\delta }(\mathbf{r}+{\mathbf{r}}_{\mathcal{l}z})\otimes \mathcal{P}\left\{o_c^{*}(\mathbf{r})\cdot \left[h_{c,z}^{*}(-\mathbf{r})\otimes \left(\left(\frac{\sqrt{I_{c,\mathcal{l}z}(\mathbf{r})}}{|g_{c,\mathcal{l}z}(\mathbf{r})|}-1\right)\cdot g_{c,\mathcal{l}z}(\mathbf{r})\right)\right]\right\}\\ {\nabla }_{{\tilde{h}}_c}{f}_{c,\mathcal{l}z}(o_c,p_c,h_c,{\mathbf{r}}_{\mathcal{l}z})=-{\tilde{h}}_z^{*}(\mathbf{u})\cdot \mathcal{F}{\left\{p_c(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}z})\cdot o_c(\mathbf{r}))\right\}}^{*}\mathcal{F}\left\{\left(\frac{\sqrt{I_{c,\mathcal{l}z}(\mathbf{r})}}{|g_{c,\mathcal{l}z}(\mathbf{r})|}-1\right)\cdot g_{c,\mathcal{l}z}(\mathbf{r})\right\}\\ {\nabla }_{q_{\mathcal{l}z}}{f}_{c,\mathcal{l}z}(o_c,p_c,h_c,{\mathbf{r}}_{\mathcal{l}z})=-2\text{Re}\{{\displaystyle \sum _{\mathbf{r}}\left[\left(\frac{\sqrt{I_{c,\mathcal{l}z}(\mathbf{r})}}{|g_{c,\mathcal{l}z}(\mathbf{r})|}-1\right)\cdot g_{c,\mathcal{l}z}^{*}(\mathbf{r})\right]}\cdot \\ \left[h_{c,z}(\mathbf{r})\otimes \left(o_c(\mathbf{r})\cdot \mathcal{C}\left\{\frac{\partial p_c(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}z})}{\partial q_{\mathcal{l}z}}\right\}\right)\right]\}.\end{array}$$

Appendix B: Reconstruction algorithm

With the derivation of the gradients in Appendix A, we summarize here the reconstruction algorithm for fluorescence imaging and coherent imaging.

B.1. Algorithm for fluorescence imaging

First, we initialize the sample, $o_f\left(\mathbf{r}\right)$, with the mean image of all the structure illuminated images, $I_{f,\mathcal{l}}\left(\mathbf{r}\right)$, which is approximately a widefield diffraction-limited image. As for the structured pattern, $p_f\left(\mathbf{r}\right)$, we initialize it with a all-one image. The initial OTF, ${\tilde{h}}_f\left(\mathbf{u}\right)$, is set as a non-aberrated incoherent OTF. Initial scanning positions are from the registration of the in-focus coherent speckle images, $I_{c,\mathcal{l}z}\left(\mathbf{r}\right)$ (z = 0).

In the algorithm, K_f is the total number of iterations (K_f = 100 is generally enough for convergence). At every iteration, we sequentially update the sample, structured pattern, system’s OTF and the scanning position using each single frame from $\mathcal{l}=1$ to $\mathcal{l}=N_{\text{img}}$. A Nesterov acceleration step is applied on the sample and the structured pattern at the end of each iteration. The detailed algorithm is summarized in Algorithm 1.

Algorithm 1. Fluorescence imaging reconstruction

View Table | View all tables in this article

B.2. Algorithm for coherent imaging

For coherent imaging, we initialize $o_c\left(\mathbf{r}\right)$ with all ones. The pattern, $p_c\left(\mathbf{r}\right)$, is initialized with the mean of the square root of registered coherent in-focus intensity stack. The pupil function is initialized with a circ function (2D function filled with ones within the defined radius) with the radius defined by the objective NA. In the end, we initialize the scanning position, ${\mathbf{r}}_{\mathcal{l}z}$, from the registration of the intensity stacks, $I_{c,\mathcal{l}z}$, for respective focal planes.

For the coherent imaging reconstruction, we use a total number of $K_c\approx 30$ iterations to converge. We sequentially update $o_c\left(\mathbf{r}\right)$, $p_c\left(\mathbf{r}\right)$, $h_c\left(\mathbf{r}\right)$, and ${\mathbf{r}}_{\mathcal{l}},\left(\mathcal{l}=1,\dots ,N_{\text{img}}\right)$ for each defocused plane (total number of defocused planes is N_z) per iteration. Unlike for our fluorescence reconstructions, we do not use the extra Nesterov’s acceleration step in the QP reconstruction.

Algorithm 2. Coherent imaging reconstruction

View Table | View all tables in this article

Appendix C: Sample preparation

Results presented in this work targeted super-resolution of 1 μm and 2 μm diameter polystyrene microspheres (Thermofischer) that were fluorescently tagged to emit at a center wavelength of ${\lambda }_{\text{em}}=605$ nm. Monolayer samples of these microspheres were prepared by placing microsphere dilutions (60 uL stock-solution/500 uL isopropyl alcohol) onto #1.5 coverslips and then allowing to air-dry. High-index oil ($n_m\left(\lambda \right)=1.52$ at λ = 532 nm) was subsequently placed on the coverslip to index-match the microspheres. An adhesive spacer followed by another #1.5 coverslip was placed on top of the original coverslip to assure a uniform sample layer for imaging.

Appendix D: Posedness of the problem

In this paper, we illuminate the sample with an unknown speckle pattern to encode both large-FOV and high-resolution information into our measurement. To decode the high-resolution information, we need to jointly estimate the speckle pattern and the sample. This framework shares similar characteristics with the work on blind SIM first introduced by [46], where completely random speckle patterns were sequentially illuminated onto the sample. Unfortunately, the reconstruction formulation proposed in that work is especially ill-posed due to randomness between the illumination patterns, i.e., if N_imgraw images are taken, there would be $N_{\text{img}}+1$ unknown variables to solve for (N_img illumination patterns and 1 sample distribution). To better condition this problem, priors based on speckle statistics [46–52] and sample sparsity [48, 51] can be introduced, pushing blind SIM to 2s resolution gain. However, to implement high-content microscopy using SIM, we desire a resolution gain of $>2\times$. Even with priors, we found that this degree of resolution gain was not experimentally achievable with uncorrelated and random speckle illuminations, due to the reconstruction formulation being so ill-posed.

In this work, we improve the posedness of the problem by illuminating with a translating speckle pattern, as opposed to randomly changing speckle patterns. Because each individual illumination pattern at the sample is a laterally shifted version of every other illumination pattern, the posedness of the reconstruction framework dramatically increases. Previous works [25, 55, 56] have also demonstrated this concept to effectively achieve beyond $2\times$ resolution gain.

Appendix E: Self-calibration analysis

In Sec. 2.1.2 and 2.2.2, we presented the inverse problem formulation for super-resolution fluorescence and QP. We note that those formulations also included terms to self-calibrate for unknowns in the system’s experimental OTF and the illumination pattern’s scan-position. Here we demonstrate how these calibrations are important for our reconstruction quality.

 

Fig. 6 Algorithmic self-calibration significantly improves fluorescence super-resolution reconstructions. Here, we compare the resconstructed fluorescence image, speckle intensity, and OTF with no correction, OTF correction, and both OTF correction and scanning position correction. The right panel shows the overlay of the uncorrected and corrected scanning position trajectories.

Download Full Size | PDF

To demonstrate the improvement in our fluorescence imaging reconstruction due to the self-calibration algorithm, we select a region of interest from the dataset presented in Fig. 4. Figure 6 shows the comparison of the SR reconstruction with and without self-calibration. The SR reconstruction with no self-calibration contains severe artifacts in reconstructions of both the speckle illumination pattern and the sample’s fluorescent distribution. With OTF correction, dramatic improvements in the fluorescence SR image are evident. OTF correction is especially important when imaging across a large FOV (Fig. 4 and 5) due to space-varying aberrations.

Further self-calibration to correct for errors in the initial estimate of the illumination pattern’s trajectory enables further refinement of the SR reconstruction. We see that this illumination trajectory demonstrates greater smoothness after undergoing self-calibration. We fully expect that this calibration step to have important ramifications in cases where the physical translation stage is of lower stability or more inaccurate incremental translation.

 

Fig. 7 Algorithmic self-calibration significantly improves coherent super-resolution reconstructions. We show a comparison of reconstructed amplitude, phase, speckle amplitude, and phase of the pupil function with no correction, pupil correction, and both pupil correction and scanning position correction. The right panel shows the overlay of scannning position trajectory for the in-focus and defocused cameras before and after correction.

Download Full Size | PDF

We also test how the self-calibration affects our phase reconstruction, using the same dataset as in Fig. 3. Similar to the conclusion from the fluorescence self-calibration demonstration, pupil correction (coherent OTF) plays an important role in reducing SR reconstruction artifacts as shown in Fig. 7. The reconstructed pupil phase suggests that our system aberration is mainly caused by astigmatism. Further refinement of the trajectory of the illumination pattern improves the SR resolution by resolving one more element (group 9 element 6) of the USAF chart. Paying more attention to the uncorrected and corrected illumination trajectory, we find that the self-calibrated trajectory of the illumination pattern tends to align the trajectories from the two coherent cameras. We also notice that the trajectory from the quantitative-phase channels seems to jitter more compared to the fluorescence channel. We hypothesize that this is due to longer exposure time for each fluorescence acquisition, which would average out the jitter.

Funding

Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative (GBMF4562); Ruth L. Kirschstein National Research Service Award (F32GM129966).

Disclosures

The authors declare that there are no conflicts of interest related to this article.

References

1. J. Goodman, Introduction to Fourier optics (Roberts & Co., 2005).

2. A. W. Lohmann, R. G. Dorsch, D. Mendlovic, Z. Zalevsky, and C. Ferreira, “Space-bandwidth product of optical signals and systems,” J. Opt. Soc. Am. A 13, 470–473 (1996). [CrossRef]  

3. B. Mccullough, X. Ying, T. Monticello, and M. Bonnefoi, “Digital microscopy imaging and new approaches in toxicologic pathology,” Toxicol Pathol. 32 (suppl 2), 49–58 (2004). [CrossRef]   [PubMed]  

4. M. H. Kim, Y. Park, D. Seo, Y. J. Lim, D.-I. Kim, C. W. Kim, and W. H. Kim, “Virtual microscopy as a practical alternative to conventional microscopyin pathology education,” Basic Appl. Pathol. 1, 46–48 (2008). [CrossRef]  

5. F. R. Dee, “Virtual microscopy in pathology education,” Human Pathol 40, 1112–1121 (2009). [CrossRef]  

6. R. Pepperkok and J. Ellenberg, “High-throughput fluorescence microscopy for systems biology,” Nat. Rev. Mol. Cell Biol. 7, 690–696 (2006). [CrossRef]   [PubMed]  

7. J. C. Yarrow, G. Totsukawa, G. T. Charras, and T. J. Mitchison, “Screening for cell migration inhibitors via automated microscopy reveals a Rho-kinase Inhibitor,” Chem. Biol. 12, 385–395 (2005). [CrossRef]   [PubMed]  

8. V. Laketa, J. C. Simpson, S. Bechtel, S. Wiemann, and R. Pepperkok, “High-content microscopy identifies new neurite outgrowth regulators,” Mol. Biol. Cell 18, 242–252 (2007). [CrossRef]  

9. A. Trounson, “The production and directed differentiation of human embryonic stem cells,” Endocr. Rev. 27(2), 208–219 (2006). [CrossRef]   [PubMed]  

10. U. S. Eggert, A. A. Kiger, C. Richter, Z. E. Perlman, N. Perrimon, T. J. Mitchison, and C. M. Field, “Parallel chemical genetic and genome-wide RNAi screens identify cytokinesis inhibitors and targets,” PLoS Biol. 2, e379 (2004). [CrossRef]   [PubMed]  

11. V. Starkuviene and R. Pepperkok, “The potential of high-content high-throughput microscopy in drug discovery,” Br. J. Pharmacol 152, 62–71 (2007). [PubMed]  

12. W. Xu, M. H. Jericho, I. A. Meinertzhagen, and H. J. Kreuzer, “Digital in-line holography for biological applications,” PNAS 98, 11301–11305 (2001). [CrossRef]   [PubMed]  

13. W. Bishara, T.-W. Su, A. F. Coskun, and A. Ozcan, “Lensfree on-chip microscopy over a wide field-of-view using pixel super-resolution,” Opt. Express 18, 11181–11191 (2010). [CrossRef]   [PubMed]  

14. A. Greenbaum, W. Luo, B. Khademhosseinieh, T.-W. Su, A. F. Coskun, and A. Ozcan, “Increased space-bandwidth product in pixel super-resolved lensfree on-chip microscopy,” Scientific reports 3: 1717 (2013). [CrossRef]  

15. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photon. 7, 739–745 (2013). [CrossRef]  

16. L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for Fourier ptychography with an LED array microscope,” Biomed. Opt. Express 5, 2376–2389 (2014). [CrossRef]   [PubMed]  

17. L. Tian, Z. Liu, L. Yeh, M. Chen, J. Zhong, and L. Waller, “Computational illumination for high-speed in vitro Fourier ptychographic microscopy,” Optica 2, 904–911 (2015). [CrossRef]  

18. S. Pang, C. Han, M. Kato, P. W. Sternberg, and C. Yang, “Wide and scalable field-of-view Talbot-grid-based fluorescence microscopy,” Opt. Lett. 37, 5018–5020 (2012). [CrossRef]   [PubMed]  

19. A. Orth and K. Crozier, “Microscopy with microlens arrays: high throughput, high resolution and light-field imaging,” Opt. Express 20, 13522–13531 (2012). [CrossRef]   [PubMed]  

20. A. Orth and K. Crozier, “Gigapixel fluorescence microscopy with a water immersion microlens array,” Opt. Express 21, 2361–2368 (2013). [CrossRef]   [PubMed]  

21. S. Pang, C. Han, J. Erath, A. Rodriguez, and C. Yang, “Wide field-of-view Talbot grid-based microscopy for multicolor fluorescence imaging,” Opt. Express 21, 14555–14565 (2013). [CrossRef]   [PubMed]  

22. A. Orth and K. B. Crozier, “High throughput multichannel fluorescence microscopy with microlens arrays,” Opt. Express 22, 18101–18112 (2014). [CrossRef]   [PubMed]  

23. A. Orth, M. J. Tomaszewski, R. N. Ghosh, and E. Schonbrun, “Gigapixel multispectral microscopy,” Optica 2, 654–662 (2015). [CrossRef]  

24. S. Chowdhury, J. Chen, and J. Izatt, “Structured illumination fluorescence microscopy using Talbot self-imaging effect for high-throughput visualization,” arXiv: 1801.03540 (2018).

25. K. Guo, Z. Zhang, S. Jiang, J. Liao, J. Zhong, Y. C. Eldar, and G. Zheng, “13-fold resolution gain through turbid layer via translated unknown speckle illumination,” Biomed. Opt. Express 9, 260–274 (2018). [CrossRef]   [PubMed]  

26. W. Lukosz, “Optical systems with resolving powers exceeding the classical limit. II,” J. Opt. Soc. Am. 57, 932–941 (1967). [CrossRef]  

27. C. J. Schwarz, Y. Kuznetsova, and S. R. J. Brueck, “Imaging interferometric microscopy,” Opt. Lett. 28, 1424–1426 (2003). [CrossRef]   [PubMed]  

28. M. Kim, Y. Choi, C. Fang-Yen, . Y. Sung Ramachandra, R. Dasari, Michael S. Feld, and W. Choi, “High-speed synthetic aperture microscopy for live cell imaging,” Opt. Lett. 36, 148–150 (2011). [CrossRef]   [PubMed]  

29. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19, 780–782 (1994). [CrossRef]   [PubMed]  

30. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313, 1642–1645 (2006). [CrossRef]   [PubMed]  

31. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nature Methods 3, 793–795 (2006). [CrossRef]   [PubMed]  

32. R. Heintzmann and C. Cremer, “Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating,” Proc. SPIE 3568, 185–196 (1999). [CrossRef]  

33. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” Journal of Microscopy 198, 82–87 (2000). [CrossRef]   [PubMed]  

34. M. G. L. Gustafson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” PNAS 102, 13081–13086 (2005). [CrossRef]  

35. S. Chowdhury, W. J. Eldridge, A. Wax, and J. A. Izatt, “Structured illumination multimodal 3D-resolved quantitative phase and fluorescence sub-diffraction microscopy,” Biomed. Opt. Express 8, 2496–2518 (2017). [CrossRef]   [PubMed]  

36. D. Li, L. Shao, B.-C. Chen, X. Zhang, M. Zhang, B. Moses, D. E. Milkie, J. R. Beach, J. A. Hammer, M. Pasham, T. Kirchhausen, M. A. Baird, M. W. Davidson, P. Xu, and E. Betzig, “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics,” Science 349, aab3500 (2015). [CrossRef]  

37. P. von Olshausen and A. Rohrbach, “Coherent total internal reflection dark-field microscopy: label-free imaging beyond the diffraction limit,” Opt. Lett. 38, 4066–4069 (2013). [CrossRef]   [PubMed]  

38. S. Chowdhury, A.-H. Dhalla, and J. Izatt, “Structured oblique illumination microscopy for enhanced resolution imaging of non-fluorescent, coherently scattering samples,” Biomed. Opt. Express 3, 1841–1854 (2012). [CrossRef]   [PubMed]  

39. S. Chowdhury and J. A. Izatt, “Structured illumination quantitative phase microscopy for enhanced resolution amplitude and phase imaging,” Biomed. Opt. Express 4, 1795–1805 (2013). [CrossRef]   [PubMed]  

40. P. Gao, G. Pedrini, and W. Osten, “Structured illumination for resolution enhancement and autofocusing in digital holographic microscopy,” Opt. Lett. 38, 1328–1330 (2013). [CrossRef]   [PubMed]  

41. K. Lee, K. Kim, G. Kim, S. Shin, and Y. Park, “Time-multiplexed structured illumination using a DMD for optical diffraction tomography,” Opt. Lett. 42, 999–1002 (2017). [CrossRef]   [PubMed]  

42. S. Chowdhury, W. J. Eldridge, A. Wax, and J. Izatt, “Refractive index tomography with structured illumination,” Optica 4, 537–545 (2017). [CrossRef]  

43. S. Chowdhury, W. J. Eldridge, A. Wax, and J. A. Izatt, “Structured illumination microscopy for dualmodality 3D sub-diffraction resolution fluorescence and refractive-index reconstruction,” Biomed. Opt. Express 8, 5776–5793 (2017). [CrossRef]  

44. M. Schürmann, G. Cojoc, S. Girardo, E. Ulbricht, J. Guck, and P. Müller, “Three-dimensional correlative single-cell imaging utilizing fluorescence and refractive index tomography,” J. Biophoton. p. e201700145 (2017).

45. S. Shin, D. Kim, K. Kim, and Y. Park, “Super-resolution three-dimensional fluorescence and optical diffraction tomography of live cells using structured illumination generated by a digital micromirror device,” arXiv p. 1801.00854 (2018).

46. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. L. Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photon. 6, 312–315 (2012). [CrossRef]  

47. R. Ayuk, H. Giovannini, A. Jost, E. Mudry, J. Girard, T. Mangeat, N. Sandeau, R. Heintzmann, K. Wicker, K. Belkebir, and A. Sentenac, “Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIM algorithm,” Opt. Lett. 38, 4723–4726 (2013). [CrossRef]   [PubMed]  

48. J. Min, J. Jang, D. Keum, S.-W. Ryu, C. Choi, K.-H. Jeong, and J. C. Ye, “Fluorescent microscopy beyond diffraction limits using speckle illumination and joint support recovery,” Scientific Reports 3, 2075: 1–6 (2013). [CrossRef]  

49. A. Jost, E. Tolstik, P. Feldmann, K. Wicker, A. Sentenac, and R. Heintzmann, “Optical sectioning and high resolution in single-slice structured illumination microscopy by thick slice blind-SIM reconstruction,” PLoS ONE 10, e0132174 (2015). [CrossRef]   [PubMed]  

50. A. Negash, S. Labouesse, N. Sandeau, M. Allain, H. Giovannini, J. Idier, R. Heintzmann, P. C. Chaumet, K. Belkebir, and A. Sentenac, “Improving the axial and lateral resolution of three-dimensional fluorescence microscopy using random speckle illuminations,” J. Opt. Soc. Am. A 33, 1089–1094 (2016). [CrossRef]  

51. S. Labouesse, M. Allain, J. Idier, S. Bourguignon, A. Negash, P. Liu, and A. Sentenac, “Joint reconstruction strategy for structured illumination microscopy with unknown illuminations,” ArXiv: 1607.01980 (2016).

52. L.-H. Yeh, L. Tian, and L. Waller, “Structured illumination microscopy with unknown patterns and a statistical prior,” Biomed. Opt. Express 8, 695–711 (2017). [CrossRef]   [PubMed]  

53. R. Förster, H.-W. Lu-Walther, A. Jost, M. Kielhorn, K. Wicker, and R. Heintzmann, “Simple structured illumination microscope setup with high acquisition speed by using a spatial light modulator,” Opt. Express 22, 20663–20677(2014). [CrossRef]   [PubMed]  

54. D. Dan, M. Lei, B. Yao, W. Wang, M. Winterhalder, A. Zumbusch, Y. Qi, L. Xia, S. Yan, Y. Yang, P. Gao, T. Ye, and W. Zhao, “DMD-based LED-illumination Super-resolution and optical sectioning microscopy,” Scientific Reports 3, 1116 (2013). [CrossRef]   [PubMed]  

55. S. Dong, P. Nanda, R. Shiradkar, K. Guo, and G. Zheng, “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express 22, 20856–20870 (2014). [CrossRef]   [PubMed]  

56. H. Yilmaz, E. G. V. Putten, J. Bertolotti, A. Lagendijk, W. L. Vos, and A. P. Mosk, “Speckle correlation resolution enhancement of wide-field fluorescence imaging,” Optica 2, 424–429 (2015). [CrossRef]  

57. L.-H. Yeh, J. Dong, J. Zhong, L. Tian, M. Chen, G. Tang, M. Soltanolkotabi, and L. Waller, “Experimental robustness of Fourier ptychography phase retrieval algorithms,” Opt. Express 23, 33213–33238 (2015). [CrossRef]  

58. L. Bottou, “Large-scale machine learning with stochastic gradient descent,” International Conference on Computational Statistics pp. 177–187 (2010).

59. Y. Nesterov, “A method for solving the convex programming problem with convergence rate $O\left(1/k^2\right)$,” Dokl. Akad. Nauk SSSR 269, 543–547 (1983).

60. N. Streibl, “Phase imaging by the transport equation of intensity,” Opt. Commun. 49, 6–10 (1984). [CrossRef]  

61. J. R. Fienup, “Phase retrieval algorithms: a comparison,” Appl. Opt. 21, 2758–2769 (1982). [CrossRef]   [PubMed]  

62. R. W. Gerchberg and W. O. Saxton, “Phase determination for image and diffraction plane pictures in the electron microscope,” Optik 34, 275–284 (1971).

63. M. Guizar-Sicairos, S. T. Thurman, and J. R. Fienup, “Efficient subpixel image registration algorithms,” Opt. Lett. 33, 156–158 (2008). [CrossRef]   [PubMed]  

64. K. Kreutz-Delgado, “The Complex Gradient Operator and the CR-Calculus,” arXiv:0906.4835v1 (2009).