Superresolution via saturated virtual modulation microscopy

Ruizhi Cao; Cuifang Kuang; Yong Liu; Xu Liu

doi:10.1364/OE.25.032364

1. Introduction

Over the last few decades, techniques have been developed to break the well-known Abbe diffraction limit [1] in conventional optical microscopies. Stimulated emission depletion microscopy [2, 3], photoactivated localization microscopy [4], and stochastic optical reconstruction microscopy [5] can reach sub-100-nm resolutions and can thus be employed for imaging sub-cellular structures. However, these three methods require special fluorescent proteins such as photoswitchable proteins. Structured illumination microscopy (SIM) [6, 7] can improve the resolution by a factor of two and stand out for its wide-field nature and universality for all kinds of fluorescent proteins. Three-dimensional SIM [8, 9] is introduced to improve the axial resolution in SIM, and blind-SIM [10, 11] has been proposed in order to eliminate the problems that arise from the distortion of the sinusoidal illumination pattern that is required in conventional SIM. Another approach, image scanning microscopy [12], is based on the same idea as SIM but applies a focused spot to illuminate the sample. Multi-focal SIM [13, 14] provides a much higher imaging speed, which enables us to see live cells and tissue in real-time. Apart from these techniques, there are some other methods that enable the same resolution improvement with virtual modulation [15–19]. These virtual modulation methods differ from others as the modulation is processed digitally after the data are acquired, whereas other methods modulate the illumination pattern at the same time when the sample is being imaged. One of the virtual modulation techniques, called virtual k-space modulation optical microscopy (VIKMOM) [15], exhibits a high resistance to noise and aberrations induced by the lens itself and experimental errors using the algorithm inspired by Fourier ptychography microscopy [20, 21].

As the resolution improvement in conventional SIM can only theoretically reach a factor of two, methods have been proposed to achieve a better resolution enhancement such as saturated SIM (SSIM) [22], in which the non-linear response of the fluorescence is essential to achieve better resolution enhancement. However, SSIM requires a much greater laser source intensity than conventional SIM and may cause photodamage to the sample. In order to reduce the power required and obtain the same resolution enhancement at the same time, some other techniques were proposed, such as non-linear SIM implemented with photoswitchable proteins [23, 24] and the saturated focused spot [19]. Both the simulation and experimental results of the saturated focused spot illuminated scanning microscopy have been demonstrated in [19] through virtual temporal modulation (VTM). However, the SIM-based algorithm may be sensitive to aberrations and the saturation level, which may lead to problems in the experiments.

In this paper, we demonstrate that recovering information through virtual spatial modulation (VSM) is also feasible for virtual modulation microscopy under certain conditions. Another multiplex saturated pattern illuminated virtual modulation method is demonstrated in this paper, through which the imaging speed of the virtual modulation microscopy can be improved dramatically. And the proposed saturated virtual modulation microscopy also exhibits a high resistance to aberrations and saturation levels, and could thus be widely employed in biomedical imaging.

2. Forward model

The imaging process in laser scanning microscopy (LSM) can be expressed as follows:

I (x, y, t_{x}, t_{y}) = \iint h_{i l l} (u - t_{x}, v - t_{y}) S (u, v) h_{\det} (x - u, y - v) d u d v,

where

I (x, y, t_{x}, t_{y})

denotes the signal at the receptive plane with the coordinate

(x, y)

when the center of the illumination spot

h_{i l l}

moves to the time variant point

(t_{x}, t_{y})

,

S

denotes the fluorescence distribution of the sample, and

h_{\det}

is the PSF of our detecting system. We can utilize the modulation via digital implementation either in the time domain or spatial domain.

If we utilize the modulation in the time domain, the image after the modulation can be expressed as

V_{t} (x, y) = \iint I (x, y, t_{x}, t_{y}) M (t_{x}, t_{y}) d t_{x} d t_{y} = [(h_{i l l} \otimes M) \cdot S] \otimes h_{\det},

where

V_{t}

denotes the virtual modulated image, where the modulation takes place in the time domain,

\otimes

represents the convolution operation, and

M

denotes the virtual modulation, which can be a sinusoidal pattern, multi-spot pattern, or speckles.

If the illumination spot, such as an airy disk or doughnut-shaped spot, is an even function, the image acquired by the LSM can be expressed as

\begin{array}{l} I (x, y, t_{x}, t_{y}) = \iint h_{i l l} (u - t_{x}, v - t_{y}) S (u, v) h_{\det} (x - u, y - v) d u d v \\ = \iint h_{i l l} (t_{x} - u, t_{y} - v) S (u, v) h_{\det} (u - x, v - y) d u d v . \end{array}

It is assumed that the detection PSF is an even function in the above equation because a circular aperture is usually used in the detecting system and is also basic for VSM.

If the modulation in the spatial domain is utilized, the image after the modulation can be written as

V_{s} (t_{x}, t_{y}) = \iint I (x, y, t_{x}, t_{y}) M (x, y) d t_{x} d t_{y} = [(h_{\det} \otimes M) \cdot S] \otimes h_{i l l},

where

V_{s}

denotes the virtual modulated image and the subscript “s” denotes the modulation taking place in the spatial domain.

And if the illumination photon density is sufficiently large, there will be saturation effect on the response of the fluorescent proteins, which brings high-spatial-frequency component to the efficient illumination patterns. And such high-spatial-frequency component could not be accessed without the saturation effect. According to the Fourier theorem, the high-spatial-frequency component can down modulate the high-spatial-frequency information of the object and makes these information be accessible by the detection system. In a two-state fluorescent system, the response of the fluorescence intensity regarding the excitation intensity is [22, 25]

I_{e m} = \frac{η ψ_{e x c}}{\frac{1}{σ τ} + ψ_{e x c}},

where

ψ_{e x c}

denotes the incident excitation photon flux,

σ

indicates the absorption cross-section,

τ

is the lifetime of the fluorophore, and

η

is the constant related to the emission rate. Hence, the signal obtained when a high-power laser interacts with the fluorescence is

I (x, y, t_{x}, t_{y}) = (\frac{η f_{s} h_{i l l}}{1 + f_{s} h_{i l l}} \cdot S) \otimes h_{\det} .

For simplification, $f_{s} = σ τ$ is herein called the saturation level, which is a property of the material. We can substitute a new illumination function $h_{s} = \frac{η f_{s} h_{i l l}}{1 + f_{s} h_{i l l}}$ for $h_{i l l}$ in Eqs. (2) and (4), which gives

\begin{array}{l} V_{t} (x, y) = [(h_{s} \otimes M) \cdot S] \otimes h_{\det} \\ V_{s} (t_{x}, t_{y}) = [(h_{\det} \otimes M) \cdot S] \otimes h_{s} . \end{array}

If the sample is illuminated by a certain pattern that is shifted in both the x and y-axes, the image acquired under this condition is

I (x, y, t_{x}, t_{y}) = \iint p (u - t_{x}, v - t_{y}) S (u, v) h_{\det} (x - u, y - v) d u d v,

where

p

denotes the illumination pattern applied in the imaging process.

If a cyclical multi-spot pattern is applied and the separation between two adjacent spots is sufficiently large that they appear to not interfere with each other, the data acquired in this condition are approximately equivalent to those acquired in the LSM. Hence we can use detectors such as a charge coupled device (CCD) to improve the imaging speed of the virtual modulation microscopy. As shown in Fig. 1, the blue arrows indicate the scanning route of the illumination pattern of the foci in Fig. 1(a) and the multi-spot pattern in Fig. 1(b). We can see that the object represented by a green star in Fig. 1 is illuminated by every part of the spot and the data acquired in Fig. 1(b) will equal those in Fig. 1(a) after a mathematical transform.

Fig. 1 (a) Scanning process of LSM and (b) multi-spot complex detection system. The green star in (a) and (b) denotes the position of the object, and the blue arrows denote the scanning route of the pattern. The gray grids represent the conjugated sensor plane with each square denoting a single pixel. The illumination pattern is represented by the yellow and red dot.

Download Full Size | PDF

Under the forward model demonstrated in Eq. (7), the method proposed in [15] can be applied to reconstruct the data set in the LSM with virtual modulation either in the time domain or spatial domain. To process the data acquired under the multi-spot illumination system, it is necessary to know the initial position and the scanning steps of the multi-spot pattern. Further, a rearrangement algorithm can thus be applied to the data for further data processing, as shown in Fig. 2(a). As the emission intensity of a fluorophore will reach a maximum when its excitation intensity reaches a maximum, we can use this prior knowledge to estimate the initial position of the illumination multi-spot pattern.

Fig. 2 (a) Procedure before VIKMOM algorithm is applied when illuminated by the multi-spot pattern. The stacked images denote the data acquired by the detector and the Siemens star is the object in the simulation. After the sub-data set along the t_x and t_y dimensions is extracted (process ①), the data should be rearranged to their true position in the same way LSM acquire data along the t_x and t_y dimensions (process ②) using knowledge of the estimated initial position and the scanning step of the multi-spot pattern. (b) Crosstalk between pixel A and B when estimating the initial position of the multi-spot pattern. The excitation photon flux is represented by a dashed line and the fluorescent flux is represented by a solid line. Two different imaging processes are distinguished by different colors; the conjugated detection pixels are represented by gray parallelograms and the green star denotes the position of the fluorophore.

Download Full Size | PDF

Because of the resolution limits of the imaging system shown in Fig. 2(b), the signal (indicated by the solid line) that pixel A receives will reach a maximum when the center of the illumination spot moves to the position of pixel B where the fluorophore exists. To reduce the localization noise introduced by the crosstalk demonstrated in Fig. 2(b), a weighted position function is employed to estimate the true position of our illumination pattern. We also make the minimum separation between two illumination spots larger than the sum of the diameters of the excitation and detection PSF to obtain data with less interference. The separation distance between two adjacent illumination spot equals $\frac{1.5 (λ_{e x} + λ_{e m})}{N A}$ (NA: numerical aperture) approximately in all of our simulations. Apart from the fact that the position can be determined from the relative distance from the center of the nearest illumination spot to the point $p_{0} = (x_{0}, y_{0})$ in the x–y coordinates, whose direction is shown in Fig. 1(a), the initial position of the multi-spot pattern can also be represented by the relative distance $Δ = (Δ_{t x}, Δ_{t y})$ at $p_{0}$ from the time when the center of the spot is located at $p_{0}$ to a given position $t_{0} = (t_{x 0}, t_{y 0})$ in $t_{x} - t_{y}$ coordinates shown in Fig. 2(a). Therefore, the problem of determining the initial position of the multi-spot pattern can be expressed as

\begin{matrix} minimize \\ Δ \end{matrix} f (Δ) = \sum_{p} {| m (r) - m [Δ - a \cdot (p - p_{0})] | \cdot w (p)}

where the constant

a

is the ratio of the scanning step to the length of the side of the conjugated pixel in the sample plane, w is the deconvolved widefield result, which can be obtained via adding the image stack up and then applying the deconvolution, and introduced as a weighted function,

r

is the distance from the maximum of the sub-data set to

t_{0}

, and

m (\cdot)

denotes the modulo operation that is defined as

m (v) = m (v_{1}, v_{2}) = (v_{1} \mod L_{x}, v_{2} \mod L_{y})

where

L_{x}

and

L_{y}

are the distances between two adjacent spots along the x and y-axes, respectively. As the illumination spot is centrosymmetric, we let

t_{0}

be the center of the sub-data set so the relative position after rearranging the sub-data set by

m [Δ - a \cdot (p - p_{0})]

is the same as that of the LSM.

3. Reconstruction algorithms

In order to reduce the influence of the noise that occurs, a low-pass filter is applied to the raw data before the reconstruction algorithm is applied:

\tilde{I} = \frac{c o n j (O T F_{\det}) \cdot \tilde{I}}{| O T F_{\det} | + α}

where

c o n j (\cdot)

denotes the complex conjugate, the ~over I denotes their corresponding Fourier transform, OTF is the object transfer function, and

α

is a positive regularization constant related to noise. Using the forward model shown in Eq. (7), we can reconstruct the sample by the following steps [15, 26].

As we digitally modulate the sample, we can determine the pattern that is applied in the modulation step and, hence, the effective modulation will be

\begin{array}{l} M t_{i - e f f} = h_{s} \otimes M_{i}; \\ M s_{i - e f f} = h_{\det} \otimes M_{i} . \end{array}

The subscript $t$ denotes the temporal modulation and $s$ denotes the spatial modulation. A sinusoidal pattern is used in our simulations and experiments. The modulation depth of the sinusoidal pattern is determined in the way that the modulation depth of the effective modulation pattern is around 1 in this study. An efficient widefield result is achieved when the whole image stack is added together. Then, the widefield result is deconvolved using the Lucy–Richardson deconvolution method to obtain a better result of the sample, which functions as the initial guess. The updated equation when the ith modulation is applied can be expressed as

\begin{array}{l} S^{u p d a t e} = S + \frac{M t_{i - e f f}}{\max {(M t_{i - e f f})}^{2}} [V_{t} - (M t_{i - e f f} \cdot S) \otimes h_{\det}] \otimes h_{\det}; \\ S^{u p d a t e} = S + \frac{M s_{i - e f f}}{\max {(M s_{i - e f f})}^{2}} [V_{s} - (M s_{i - e f f} \cdot S) \otimes h_{s}] \otimes h_{s} . \end{array}

The $h_{\det}$ in Eq. (13) can be updated by a digital correction function:

O T F_{\det}^{u p d a t e} = O T F_{\det} + \frac{n | \tilde{S} \otimes \tilde{M} t_{i - e f f} | \cdot c o n j (\tilde{S} \otimes \tilde{M} t_{i - e f f}) \cdot [{\tilde{V}}_{t} - O T F_{\det} \cdot (\tilde{S} \otimes \tilde{M} t_{i - e f f})]}{\max (| \tilde{S} \otimes \tilde{M} t_{i - e f f} |) [{| \tilde{S} \otimes \tilde{M} t_{i - e f f} |}^{2} + ε]};

where

n

is a normalization factor related to the amplitude of

{\tilde{V}}_{t} - O T F_{\det} \cdot (\tilde{S} \otimes \tilde{M} t_{i - e f f})

and

ε

is a positive regularization constant introduced to remove the zero denominator and is also associated with noise.

4. Procedure of virtual modulation processing in spatial and time domains in a laser scanning microscopy system

In order to apply the virtual modulation method, a detector array is necessary in a LSM system, and CCD/CMOS is required for the implementation of the multiplex virtual modulation as this method based on the widefield detection. As shown in Fig. 3, the key difference in the modulation step is the time the modulation is applied and the direction the image or the modulation pattern is shifted. The image remains still and the modulation pattern is shifted along the $k_{i}$ vector, which indicates how the image shifts relative to that which the central detector captures, before we modulate the captured image in the VSM process. While we modulate the image and then shift the modulated result along the $- k_{i}$ vector in the VTM process (see Visualization 1 for supporting information).

Fig. 3 Procedure for the virtual modulation microscopy. The image captured by the ith detector can be digitally modulated in two ways. The center of the detector array is labeled as “1,” which corresponds to the center of the focusing spot. The blue images denote the modulation patterns that are applied in the virtual modulation step. And the orange and purple squares, which is plotted as the reference of the shifting directions, denote the final image size which depends on the original image size and the maximum value of $| k_{i} |$ . The vector $k_{i}$ describes how the image shifts relative to the image captured by the central detector and all the captured images should be placed at the same position referring to the square, which is termed the “initial position”. The value of $| k_{i} |$ is the distance between the center of the projecting detectors on the sample plane, which depends on the magnification power and the distance between the detector i and the central detector.

Download Full Size | PDF

5. Simulation results

Simulations of the virtual modulation microscopy in an ideal noise-free system

We have simulated the imaging process, and the reconstruction results are presented in Fig. 4. A resolution enhancement of 2.92 × is achieved by VTM in Figs. 4(b) and 4(d), while 2.30 × is achieved for the VSM in Fig. 4(c) compared with the theoretical resolution of the widefield result. The excitation wavelength and the emission wavelength are both equal to 515 nm and the pixel size of the detector is 13 nm in all the simulations. The maximum intensity of the excitation PSF is set to be one in the following simulations except for Fig. 7.

Fig. 4 Simulation of saturated virtual modulation microscopy. (a) 2 × bandwidth-filtered image with NA = 1.49 × 2. (b) Reconstruction image under VTM and (c) VSM. (d) Multiplex detection method with NA = 1.49 and $f_{s} = 6$ when the maximum intensity of PSF equals unity. The diameter of the widefield result is normalized to unity.

Download Full Size | PDF

The reason temporal modulation performs better than spatial modulation can be seen simply via Eq. (7); if a sinusoidal pattern is applied as the modulation function, the image can be written as

\begin{array}{l} {\tilde{V}}_{t} = (\tilde{S} (k) \otimes {O T F_{s} \cdot [1 + a δ (k - k_{0}) + \bar{a} δ (k + k_{0})]}) \cdot O T F_{\det}, \\ {\tilde{V}}_{s} = (\tilde{S} (k) \otimes {O T F_{\det} \cdot [1 + a δ (k - k_{0}) + \bar{a} δ (k + k_{0})]}) \cdot O T F_{s} . \end{array}

where

\bar{a}

denotes the complex conjugate of

a

. The highest-efficiency modulation frequency

k_{0}

should satisfy the following constraint:

k_{0} < support of O T F_{s} for {\tilde{V}}_{t}; k_{0} < support of O T F_{\det} for {\tilde{V}}_{s}

When the VTM frequency $k_{0} > support of O T F_{\det}$ , information beyond the typical 2 × bandwidth can be extracted. However, such information is mixed with lower-spatial-frequency information when VSM is applied.

Multiplex virtual temporal modulation (MVTM) method result against noise, saturated level, and aberrations

Because of the experimental setups and the detector itself, noise and aberrations will appear in the experimental data. The performance of the MVTM method when 5, 10, and 20% Gaussian noise are added to the raw data is presented in Fig. 5.

Fig. 5 Simulation of reconstruction results of MVTM against noise. (a) Noise-free, (b) 5%, (c) 10%, and (d) 20% Gaussian noise added to the raw data. The diameter of the wide-field result, which is not shown here, is normalized to unity.

Download Full Size | PDF

The MTVM achieves resolution enhancements of 2.53 × , 2.40 × , and 2.26 × compared with the widefield result when 5, 10, and 20% Gaussian-type white noise are introduced into the raw data, respectively. Although the MVTM performance decreases when high-intensity noise is added, it still achieves a resolution enhancement beyond 2 × when 20% Gaussian noise is introduced into the raw data.

It should be noted that if we modulate the image with more patterns that have different spatial periodicities, the result will improve. In Fig. 6, 10% Gaussian noise is added to the raw data, and we modulate the image with different sinusoidal patterns whose periodicity varies from 0.9 to 2.3 × of the cutoff frequency $f_{c}$ of the typical supporting of the detection OTF. Relatively more modulation is helpful to overcome the noise appearing in the experiment, as the results of Fig. 6 indicate. However, it will take longer to reconstruct the final image with more digital modulations, and the results of Figs. 6(c) and 6(d) seem to be quite similar. Sinusoidal patterns with about six different modulation frequencies are applied in our experiment in order to obtain a better result without significantly improving the time taken for the virtual modulation step.

Fig. 6 Reconstruction result with different numbers of modulation patterns. (a) Result employing patterns with two different modulation frequencies (1.5 × and 2.3 × $f_{c}$ ), (b) four frequencies (1.1 × , 1.5 × , 1.9 × , and 2.3 × $f_{c}$ ), (c) six frequencies (0.9 × , 1.1 × , 1.5 × , 1.7 × , 2.1 × , and 2.3 × $f_{c}$ ), and (d) eight frequencies (0.9 × , 1.1 × , 1.3 × , 1.5 × , 1.7 × , 1.9 × , 2.1 × , and 2.3 × $f_{c}$ ). 10% Gaussian-type noise is added to the raw data. (e) VTM frequency and its equivalent OTF under saturation level of 6.

Download Full Size | PDF

In order to provide a visual impression about how the resolution enhancement varies with saturation, we simulated and reconstructed the results when the maximum excitation intensity changes. When the same parameters are applied, a better resolution enhancement comes along with a higher excitation intensity, as shown in Fig. 7. The resolution improvement of MVTM limits to a factor of two if there is no saturation effect in our simulation, as shown in Fig. 7(a). And the resolution enhancement of MVTM gradually increases when the power of the excitation laser increases. As the intensity goes up, the saturation of the effective PSF becomes stronger. And the sharp edge of the effective PSF consisting of high-spatial-frequency component will bring high frequency information of the sample into the support range of the detection OTF, according to the Fourier Theorem.

Fig. 7 Simulation results of MVTM under different excitation intensity. (a) Reconstructed result when the fluorescence signal is proportional to the excitation laser intensity. Results when the maximum intensity of the excitation spot equal (b) 0.1, (c) 0.5 and (d) 1 demonstrate resolution enhancement beyond a factor of two. The intensity profile of the corresponding effective PSFs are shown in (e). The saturation level is assumed to be 6 in this simulation.

Download Full Size | PDF

Although it is desired to have a priori knowledge about the saturation level and the intensity of the excitation spot. Sometimes it may be difficult for one to know the saturation level of the fluorescent protein. In that case, the saturated level represented by $f_{s} = σ τ$ [Eq. (6)] can be estimated via measurement. But we might not know the precise value of $f_{s}$ because of noise and photobleaching. However, we can see from Fig. 8 that it does not reduce the result by much when there is about a 33% estimated error relative to its true value.

Fig. 8 Simulation results of MVTM when there are estimation errors in the saturated level. (a)–(e) Reconstruction results using different saturation factors of 2, 4, 6, 8, and 10, respectively. The true saturation factor is set to 6. (f) Normalized effective excitation point-spread-function (PSF) profile under different saturation conditions and a normal PSF (blue line) drawn for reference. (g) Corresponding effective OTF curve under different saturation levels.

Download Full Size | PDF

The aberration is a problem in imaging biospecimen as the optical inhomogeneity is almost unavoidable. Hence, we also analyze how the aberrations affect the outcomes of our reconstruction method, and the simulation results are presented in Fig. 9.

Fig. 9 Simulations of aberrations. The color bar at the upper right shows the phase shift of the pupil that will induce aberrations. The corresponding detection PSFs are presented in the second row. From the results shown in the last row, the MVTM exhibits a strong resistance to aberrations.

Download Full Size | PDF

As the digital correction is introduced in the proposed method, the results when different kinds of aberrations are introduced to the imaging system are quite similar and do not worsen by much compare with those imaged in an ideal system. It can be seen that the contrast and resolution of the confocal result is reduced when aberrations exist as the diameter of PSF become larger than that in the ideal case. However, the corresponding MVTM results give us a high contrast and high resolution improvement.

Influence of the sample size applied to estimate the initial position of the multi-spot pattern

As the MVTM method requires the initial position of the illuminated multi-spot pattern, the current estimation method is less precise when the sample is sparser.

There are only three beads within the orange area in Fig. 10(a), which can be regarded as a sparser sample, and hence cause more estimation errors than the other two results, thus demonstrating the poorest reconstruction result. The results of Figs. 10(d) and 10(e) shift slightly from their original location and artifacts arise at the same time because of the incorrectly estimated initial position. However, we can see that apart from the artifact shown in Figs. 10(d) and 10(e), the resolution enhancement exhibits a high consistency to the more precise one that shown in Fig. 10(f).

Fig. 10 Estimation of initial position. (a) Ground truth image. (b) Widefield result and (c) 2 × bandwidth filtered result, which can be achieved by methods such as structured illumination microscopy [6, 13, 27] (SIM). (d)–(f) Results of using different sampling sizes, which are marked by different colors in (a), to estimate the initial position. (d) uses the data covered by the orange area, (e) uses the blue and orange area, and (f) uses the full sampling area.

Download Full Size | PDF

Artifacts induced by the shift errors

Equations (9) and (10) indicate that the MVTM method requires the scanning steps of the multi-spot illumination pattern, but there may be shift errors in the experiment. To determine how shift errors influence the outcome result, we add uniformly distributed shift errors with a mean value equaling the scanning step designed in the simulations.

As shown in Fig. 11, artifacts appear as the shift error is introduced, and it is clear that the more iterations we apply, the more artifacts appear in the final result. However, it should be noted that when the iteration number in the reconstruction algorithm is less than 5, the reconstruction image is robust to a 20% shift error relative to the designed step. This demonstrates that when a larger shift error appears, fewer iterations can be applied to remove the artifacts induced by the errors and maintain an acceptable image quality.

Fig. 11 Simulations of shift error. The maximum probable relative shift error is shown in the left column. As the iteration time that applied in the reconstruction algorithm goes, the shift error gradually reduces the image quality.

Download Full Size | PDF

6. Experimental results

In order to verify the virtual modulation method, we conducted our experiments using the commercial system Zeiss LSM 800 with an object lens with high NA (Plan-Apochromat 63 × /1.40 Oil DIC M27, Zeiss) to acquire the raw data under the saturated condition, where the intensity of the fluorescence respond non-linearly to the excitation intensity. We also obtained a normal airyscan [28] result, which is an advanced method to improve the signal-to-noise ratio (SNR) and resolution in confocal with a detector array using a pixel-reassignment algorithm [29]. The microtubule is labeled using the Abberior STAR 635P (excitation wavelength: 638 nm/emission wavelength: 651 nm) and the bead is a dark red (excitation wavelength: 660 nm/emission wavelength: 680 nm) fluorescent bead (F8807, Life Technologies). The results in which the virtual modulation method is applied, as shown in Figs. 12(c) and 12(g), provide a much better resolution enhancement compared with those utilizing the airyscan method. And the resolution enhancement of result of the microtubule reconstructed by VTM exhibits a 2.4 × resolution enhancement compared with the theoretical resolution of the widefield result under Rayleigh criterion.

Fig. 12 Experimental results of (a)–(c) microtubule (An effective PSF under saturation condition can be achieved when fs = 6.2 and maximum intensity of the actual excitation PSF equals unity) and (e)–(g) beads (fs = 9.1, maximum excitation intensity equals one under saturation condition). (a) Confocal results with 1 airy unit (AU) detector and (e) 0.2AU detector, (b)(f) airyscan results under unsaturated condition, and (c)(g) VTM reconstruction images under saturated condition. (d) Normalized intensity profiles of the cutline indicated by the arrows in (a)–(c) and (h) the profiles of the cutline in (e)–(g). The scale bar is 500 nm for (e)–(g) and the nominal bead diameter is 200 nm.

Download Full Size | PDF

We also imaged 100-nm carboxylate-modified yellow-green (excitation wavelength: 505 nm/emission wavelength: 515 nm) fluorescent beads (F8803, Life Technologies). From Fig. 13(f), we can see that the VTM improves the resolution compared with that achieved by the airyscan. However, there are also relatively more artifacts shown in the result reconstructed using VTM. The results of Figs. 13(d) and 13(e) show a high consistency with our simulations that show the VTM achieving a better resolution enhancement than the VSM.

Fig. 13 Experiment results of 100-nm beads. (a) Confocal result with 0.2AU detector and (b) 1AU detector, (c) airyscan result, (d) VSM result, and (e) VTM result (fs = 3.5, maximum excitation intensity is unity for the data of VSM and VTM). (f) Intensity profiles of the cut-lines shown in the lower right of (a)–(e).

Download Full Size | PDF

Figure 14 presents the results of the nuclear pore complex (NPC) labeled using Abberior STAR 635P. As the background is strong, the signal-to-noise ratio in the reconstruction result of the VTM appears to be poorer than that of the airyscan. If a strong background is present in the raw data, it is essential to subtract the background first.

Fig. 14 Experimental results of nuclear pore complex. (a) Confocal result with 0.2AU detector and (b) 1 AU detector, (c) airyscan result, and (d) VTM result (fs = 5.7, maximum excitation intensity is unity).

Download Full Size | PDF

7. Conclusions and discussions

From both the simulation results and experimental results, the virtual modulation microscopy presents a considerable resolution enhancement. And the proposed method exhibited a high resistance to aberrations and saturation levels as shown in our simulations, which is useful for the biomedical imaging as the aberrations induced by the optical inhomogeneity of living environment of the cell are almost unavoidable. The basic saturated virtual modulation method could be easily integrated in the existing LSM system as the method only requires for a detector array, and it provides a resolution enhancement above 2 × . In addition, the data acquiring speed could be improved significantly when applying multiplex virtual modulation techniques. However, due to the digital modulation procedures, the virtual modulation microscopy required much more time to reconstruct the final image than the airyscan and the maximum resolution improvement decreased to a much smaller number than 2.92 × when 20% Gaussian noise was introduced (Fig. 5). The image quality of reconstruct result of the NPC is much poorer compared with the fluorescent beads and microtubules due to noises. It may be worthwhile to point out there are still many efforts can be made to improve the proposed method, such as investigating the possibility for the axial resolution improvement. And the MVTM also needs a further research on it to find out the proper parameter of the multi-spot patterns, such as the distance between the adjacent spots, which is a key parameter for improving the image speed. As a relatively higher laser is required for the saturation effect, photobleaching remains a problem in the proposed scheme. Implementations with photoswitchable fluorescent proteins may have the potential to reduce the power required in the proposed method.

Funding.

National Basic Research Program of China (973Program) (2015CB352003); National Key Research and Development Program of China (2017YFC0110303, 2016YFF0101400); National Natural Science Foundation of China (NSFC) (61335003, 61377013, 61378051, 61427818); Natural Science Foundation of Zhejiang province (LR16F050001); and the Fundamental Research Funds for the Central Universities (2017FZA5004).

References and links

1. E. Abbe, “Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung,” Archiv für mikroskopische Anatomie 9, 413– 418 (1873).

2. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef] [PubMed]

3. T. A. Klar, S. Jakobs, M. Dyba, A. Egner, and S. W. Hell, “Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission,” Proc. Natl. Acad. Sci. U.S.A. 97(15), 8206–8210 (2000). [CrossRef] [PubMed]

4. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging Intracellular Fluorescent Proteins at Nanometer Resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef] [PubMed]

5. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–796 (2006). [CrossRef] [PubMed]

6. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef] [PubMed]

7. R. Heintzmann and C. G. Cremer, “Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating,” in 1999), 185–196. [CrossRef]

8. M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-Dimensional Resolution Doubling in Wide-Field Fluorescence Microscopy by Structured Illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef] [PubMed]

9. L. Schermelleh, P. M. Carlton, S. Haase, L. Shao, L. Winoto, P. Kner, B. Burke, M. C. Cardoso, D. A. Agard, M. G. L. Gustafsson, H. Leonhardt, and J. W. Sedat, “Subdiffraction Multicolor Imaging of the Nuclear Periphery with 3D Structured Illumination Microscopy,” Science 320(5881), 1332–1336 (2008). [CrossRef] [PubMed]

10. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics 6(5), 312–315 (2012). [CrossRef]

11. L.-H. Yeh, L. Tian, and L. Waller, “Structured illumination microscopy with unknown patterns and a statistical prior,” Biomed. Opt. Express 8(2), 695–711 (2017). [CrossRef] [PubMed]

12. C. B. Müller and J. Enderlein, “Image scanning microscopy,” Phys. Rev. Lett. 104(19), 198101 (2010). [CrossRef] [PubMed]

13. A. G. York, S. H. Parekh, D. Dalle Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef] [PubMed]

14. A. G. York, P. Chandris, D. D. Nogare, J. Head, P. Wawrzusin, R. S. Fischer, A. Chitnis, and H. Shroff, “Instant super-resolution imaging in live cells and embryos via analog image processing,” Nat. Methods 10(11), 1122–1126 (2013). [CrossRef] [PubMed]

15. C. Kuang, Y. Ma, R. Zhou, G. Zheng, Y. Fang, Y. Xu, X. Liu, and P. T. So, “Virtual k-Space Modulation Optical Microscopy,” Phys. Rev. Lett. 117(2), 028102 (2016). [CrossRef] [PubMed]

16. J. Lu, W. Min, J.-A. Conchello, X. S. Xie, and J. W. Lichtman, “Super-Resolution Laser Scanning Microscopy through Spatiotemporal Modulation,” Nano Lett. 9(11), 3883–3889 (2009). [CrossRef] [PubMed]

17. R.-W. Lu, B.-Q. Wang, Q.-X. Zhang, and X.-C. Yao, “Super-resolution scanning laser microscopy through virtually structured detection,” Biomed. Opt. Express 4(9), 1673–1682 (2013). [CrossRef] [PubMed]

18. Y. Zhi, R. Lu, B. Wang, Q. Zhang, and X. Yao, “Rapid super-resolution line-scanning microscopy through virtually structured detection,” Opt. Lett. 40(8), 1683–1686 (2015). [CrossRef] [PubMed]

19. G. P. J. Laporte, N. Stasio, C. J. R. Sheppard, and D. Psaltis, “Resolution enhancement in nonlinear scanning microscopy through post-detection digital computation,” Optica 1(6), 455–460 (2014). [CrossRef]

20. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photonics 7(9), 739–745 (2013). [CrossRef] [PubMed]

21. X. Ou, G. Zheng, and C. Yang, “Embedded pupil function recovery for Fourier ptychographic microscopy,” Opt. Express 22(5), 4960–4972 (2014). [CrossRef] [PubMed]

22. M. G. Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U.S.A. 102(37), 13081–13086 (2005). [CrossRef] [PubMed]

23. E. H. Rego, L. Shao, J. J. Macklin, L. Winoto, G. A. Johansson, N. Kamps-Hughes, M. W. Davidson, and M. G. Gustafsson, “Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution,” Proc. Natl. Acad. Sci. U.S.A. 109(3), E135–E143 (2012). [CrossRef] [PubMed]

24. D. Li, L. Shao, B.-C. Chen, X. Zhang, M. Zhang, B. Moses, D. E. Milkie, J. R. Beach, J. A. Hammer 3rd, M. Pasham, T. Kirchhausen, M. A. Baird, M. W. Davidson, P. Xu, and E. Betzig, “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics,” Science 349(6251), aab3500 (2015). [CrossRef] [PubMed]

25. R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy--a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19(8), 1599–1609 (2002). [CrossRef] [PubMed]

26. S. Dong, P. Nanda, R. Shiradkar, K. Guo, and G. Zheng, “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express 22(17), 20856–20870 (2014). [CrossRef] [PubMed]

27. F. Ströhl and C. F. Kaminski, “Frontiers in structured illumination microscopy,” Optica 3(6), 667–677 (2016). [CrossRef]

28. J. Huff, “The Airyscan detector from ZEISS: confocal imaging with improved signal-to-noise ratio and super-resolution,” Nat. Methods 12, 388 (2015).

29. C. J. R. Sheppard, “Super-resolution in confocal imaging,” Optik (Stuttg.) 80, 53–54 (1988).

Superresolution via saturated virtual modulation microscopy

Abstract

1. Introduction

2. Forward model

3. Reconstruction algorithms

4. Procedure of virtual modulation processing in spatial and time domains in a laser scanning microscopy system

5. Simulation results

Simulations of the virtual modulation microscopy in an ideal noise-free system

Multiplex virtual temporal modulation (MVTM) method result against noise, saturated level, and aberrations

Influence of the sample size applied to estimate the initial position of the multi-spot pattern

Artifacts induced by the shift errors

6. Experimental results

7. Conclusions and discussions

Funding.

References and links

Supplementary Material (1)

Cited By

Figures (14)

Equations (16)

Optics Express