Open Access
Add to BibSonomy
Abstract
Structured illumination microscopy (SIM) has been widely applied to investigating fine structures of biological samples by breaking the optical diffraction limitation. So far, video-rate imaging has been obtained in SIM, but the imaging speed was still limited due to the reconstruction of a super-solution image through multi-sampling, which hindered the applications in high-speed biomedical imaging. To overcome this limitation, here we develop compressive imaging-based structured illumination microscopy (CISIM) by synergizing SIM and compressive sensing (CS). Compared with conventional SIM, CISIM can greatly improve the super-resolution imaging speed by extracting multiple super-resolution images from one compressed image. Based on CISIM, we successfully reconstruct the super-resolution images in biological dynamics, and analyze the effect factors of image reconstruction quality, which verify the feasibility of CISIM. CISIM paves a way for high-speed super-resolution imaging, which may bring technological breakthroughs and significant applications in biomedical imaging.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Optical microscopy, as an indispensable tool, has promoted the developments of biology by turning human insight into the microscopic scale to observe subcellular structures and biochemical functions. However, suffering from the optical diffraction limitation, the spatial resolution of optical microscopy can only reach about 200 nm, which makes it difficult to acquire the fine structures within organelles. Driven by the urgent demand for visualizing the structural details of sub-organelles, various super-resolution microscopy techniques have been developed to break the optical diffraction limitation. For example, stimulated emission depletion microscopy (STED) [1,2] incorporates a stimulated emission depletion of fluorophores into the scanned confocal microscopy, and minimizes the size of a scanned focal point by depleting the fluorescence in some specific regions of the sample while leaving a center focal spot active to emit the fluorescence, which has achieved the spatial resolution of ∼20 nm in biological samples [3] and ∼6 nm in nitrogen-vacancy centers [4]. However, the scanning-based data acquisition slows down the imaging speed of STED. Besides, the high-power laser is used to induce stimulated emission depletion, which could also lead to phototoxicity and photo-bleaching. Single-molecule localization microscopy (SMLM), which includes stochastic optical reconstruction microscopy (STORM) [5,6] and photoactivated localization microscopy (PALM) [7,8], extracts the positions of fluorophores by single-molecule localization instead of the fluorescence distribution to circumvent the optical diffraction limitation. Here, the photoactive fluorescence probes are necessary for molecular identification in SMLM, which separates the fluorescent emission of neighboring molecules in time. Thanks to high accuracy of single-molecule localization, SMLM has high lateral resolution of about 10 nm [9,10]. However, thousands of sparsely distributed fluorescence images need to be acquired to extract a single super-resolution image in SMLM, thus the imaging speed is quite low and makes it difficult to observe the biological dynamics. Structured illumination microscopy (SIM) [11,12] utilizes a series of structured excitation patterns to illuminate biological samples, which encodes the normally inaccessible high-resolution information into the observed images by moiré effect, thus a super-resolution image can be reconstructed from the recorded images by component separation and reconcentration. So far, SIM has reached 40 nm lateral resolution with saturated structured illumination [13,14].
Compared with STED and SMLM, SIM has several advantages. For example, SIM has no special requirement for fluorescence probes, unlike special energy levels for stimulated emission depletion in STED and photoactivation character in SMLM. The illumination laser power in SIM is also quite low, which greatly suppresses phototoxicity and photo-bleaching. In addition, the imaging speed of SIM is quite high with tens of frames per second, because only nine wide-field images are enough to reconstruct a super-resolution image, while STED needs point scanning, and SMLM needs repetitive switching of fluorophore for thousands of times, both of which are time-consuming. All these advantages make SIM an outstanding tool of dynamic super-resolution microscopy. To improve the imaging speed of SIM, many methods have been developed, such as accelerating the switching of structured illumination patterns by employing the high-speed modulators in hardware (e.g., ferroelectric liquid crystal spatial light modulator in hardware [15] or digital micromirror device [16]) or reducing the number of images required for reconstruction by exploiting the information redundancy of raw images in algorithm [17–20]. Moreover, some high-efficiency reconstruction algorithms in the spatial domain [21,22] or assisted by deep learning [23] were also proposed to improve the computational speed of image reconstruction. Using these strategies, video-rate SIM is available. For example, Kner et al. achieved video imaging of tubulin and kinesin dynamics in living Drosophila melanogaster S2 cells under a total internal reflection mode [15]. Li et al. imaged the dynamics near the plasma membrane of spatially resolved assemblies of clathrin and caveolin, as well as mitochondria, actin, and Golgi apparatus dynamics [24]. Huang et al. observed the long-term structural dynamics of actin filaments and mitochondrial cristae in live cells [25]. Obviously, great success has been achieved in super-resolution dynamic imaging with SIM. However, high-speed dynamics of fine structures in the biological samples are still difficult to be observed, which is limited by multiple measurements with different structured patterns in SIM. Thus, the imaging speed bottleneck of SIM lays in the frame rate of the deployed camera. Under existing technology, it is difficult to further improve the frame rate of a camera with a sufficiently large bit depth and a high pixel resolution.
Inspired by coded aperture compressive temporal imaging (CACTI) [26,27], which uniquely encodes and decodes the high-speed dynamic scene by the image sensors with limited bandwidth, we develop a novel compressive imaging-based structured illumination microscopy (CISIM) to overcome the imaging speed barrier of SIM. Here, a compressive imaging (CI) technique based on compressive sensing (CS) theory is integrated into SIM. CS theory has been introduced into SIM in previous works. For example, Meiniel et al. proposed a CS algorithm to reconstruct super-resolved fluorescence microscopy images using only 25% of camera pixels for each acquisition [28]. This strategy can reduce data acquirement, but it has no benefit for the imaging speed. Ozgurun et al. put forward a CS-based framework to illuminate the sample with scanned down-sampling and structured patterns and collect the fluorescence signal with a photomultiplier tube [29]. Due to the employment of the point detector and the switching modulator, the imaging speed of this CS-based framework is even slower than that of conventional SIM in a large field of view. In contrast, CISIM can obtain multiple frames of images in each exposure, and each frame will be simultaneously modulated by a structured pattern in SIM and random codes in CI. By CS-based image reconstruction algorithm, sequential super-resolution images can be recovered from each recorded image. That is, CISIM can extract multiple super-resolution images from a recorded image, which is different from conventional SIM that recovers a super-resolution image from multiple recorded images. In this way, the imaging speed of SIM can be significantly improved, surpassing the physical imaging speed of a camera. According to our simulations, CISIM can improve the imaging speed by two orders of magnitude compared with conventional SIM.
2. Results
To verify the feasibility of CISIM and seek the optimal parameters of system design, we simulated the optical sampling and image reconstruction of CISIM. These simulations were performed in Matlab (2019b) environment using a computer with a CPU of Intel Core i9-10920X @3.5 GHz and 256GB RAM. In the simulations, the fluorescence wavelength was set as 580 nm, the magnification of the objective lens was 60× and its numerical aperture was 0.8, and the pixel size of the camera was set as 6.5 µm. We also evaluated the image reconstruction quality of CISIM with block matching and 3D filtering denoiser (CISIM-BM3D) under the noiseless and noisy conditions, respectively. Moreover, we analyzed the effect of image compression ratio (ICR) on the image reconstruction quality of CISIM, here ICR is defined as the number of super-resolution images that can be obtained in one exposure.
The image reconstruction results by CISIM and conventional SIM are first compared, here a test chart is used as the ground truth, which contains the “East China Normal University” logo picture and surrounds striped patterns with different orientations and spatial frequencies. The ground truth image, wide-field measured image, OpenSIM [30] reconstructed image and CISIM reconstructed image are shown in Fig. 1, together with the corresponding distribution in the frequency domain by Fourier transform and the details of some striped patterns labeled with red square. Here, the wide-field measured image is calculated from the convolution of the ground truth image with the PSF of the optical system. The OpenSIM reconstructed image is obtained from nine structured illumination images, while the CISIM reconstructed image is obtained from one compressed image that contains nine structured illumination and spatial encoding images. As shown in Fig. 1, both OpenSIM and CISIM can recover the detail of high-frequency information that is missed in the wild-field measured image. Interestingly, the image reconstruction quality by CISIM seems even higher than that by OpenSIM, which is easy to be seen in the spatial frequency spectrum, where the frequency components beyond the SIM-extended bound are also recovered. This phenomenon is mainly due to the sparse prior utilized in CISIM but not in OpenSIM. The pattern of the test chart is simple and monotonous, and thus it can be compressed sparsely in some transform domains, such as the 3D transform domain of BM3D. Although the high-frequency components beyond the SIM-extended bound are not physically collected, the sparsity of the test chart in some transform domains makes it possible to recover more high-frequency information from the compressed sampling result via sparse prior constraints. Actually, by adding the sparse prior constraint to the merging operation of nine images in conventional SIM, the super-resolution image with the high-frequency components beyond SIM-extended bound can also be achieved, such as TV-SIM [31]. However, it should be emphasized that CISIM can acquire multiple super-resolution images from one recorded image, while conventional SIM only can obtain one super-resolution image from nine recorded images.
Fig. 1. Image reconstruction results of test chart. The ground truth image (A1), wide-field measured image (B1), and OpenSIM (C1) and CISIM (D1) reconstructed images, together with the corresponding distribution in the frequency domain by Fourier transform (A2-D2) and the details of the selected striped patterns labeled with red square (A3-D3).
A very important application of CISIM is biomedical imaging. To show the super-resolution results of CISIM in biological samples, two dynamics, which are the meiosis I in a living insect spermatocyte [32] and the movement of mitochondria in a COS-7 cell [25], were measured based on the data acquisition of CISIM, and then were recovered by the image reconstruction of CISIM. Here, the ICR was set as 1, that is, one super-resolution image was obtained from one recorded image. The ground truth image, wide-field measured image and CISIM reconstructed image are shown in Fig. 2 for the two biological dynamics, together with the intensity profile of the selected region labeled with red line. As can be seen in Fig. 2(A4) and Fig. 2(B4), the dense spindle fibers, which are difficult to be distinguished in wide-field imaging, can be clearly recognized in CISIM, here the intensity distribution of the CISIM reconstructed image is almost the same as that of the ground truth image. To quantitatively analyze the image reconstruction performance of CISIM, the peak signal-noise ratio (PSNR) and structural similarity (SSIM) were also calculated. The PSNR values were 35.77 and 42.81 dB for the spermatocyte and mitochondria, and the SSIM values were 0.895 and 0.994, respectively. Compared with the wild-field measured image with the PNSR values of 29.46 and 27.01 dB and the SSIM values of 0.778 and 0.918 for the spermatocyte and mitochondria, both the PSNR and SSIM values of the CISIM reconstructed image were greatly improved. Obviously, CISIM can get very high image reconstruction quality of biological samples, which will have a very important application prospect in biological imaging.
Fig. 2. Image reconstruction results of biological samples. The ground truth image (A1, B1), wide-field measured image (A2, B2) and CISIM reconstructed image (A3, B3) for the meiosis I in a living insect spermatocyte (top row) and the movement of mitochondria in the COS-7 cell (bottom row), together with the intensity profile of the selected region labeled with red line (A4, B4).
Because of the multi-frame super-resolution reconstruction ability of CISIM from one recorded image, we further studied the relationship of ICR and the image reconstruction quality of CISIM. In this simulation, a clip with 16 images for the meiosis I in a living insect spermatocyte was used as the target dynamic scene. To simplify, one representative image by CISIM reconstruction under different ICRs is shown in Fig. 3(A), together with the intensity profile of the selected region labeled with red line. Here, only the central area is enlarged to display for more intuitive observation, and the intensity profiles of the corresponding selected regions in both the ground truth image and wild-field measured image are also provided for comparison. It is obvious that the image reconstruction quality decreases with the increase of ICR. Similarly, the PSNR and SSIM values for the CISIM reconstructed images were also calculated and given in Fig. 3(B) and Fig. 3(C), together with those of the wild-field measured images. Both the PSNR and SSIM values monotonously decrease when ICR increases. The decrease of image reconstruction quality can be understood based on statistical signal processing and convex optimization theory. On the one hand, considering that the bit depth of the compressed image is limited, the weight of each super-resolution image in the compressed image decreases as ICR increases. In other words, higher ICR results in more information loss in the compressive sampling of CISIM. On the other hand, more super-resolution images need to be recovered from one compressed image for higher ICR, and thus the condition number for the inverse problem of CISIM is larger, which makes it more ill-conditioned and leads to more calculation error in the image reconstruction. That is to say, higher ICR will decrease the accuracy of information decoding and image reconstruction. Here, only when both the PSNR and SSIM values of the CISIM reconstructed image exceed those of the wild-field measured image, the super-resolution imaging is considered to be obtained. As shown in Fig. 3(B) and Fig. 3(C), when ICR is increased to 13, the PSNR and SSIM values of the CISIM reconstructed image are, respectively, 27.17 dB and 0.747, both the two values are larger than those of the wild-field measured image with 26.27 dB and 0.741, which still can demonstrate the ability of super-resolution image reconstruction. Compared with conventional SIM that needs nine images to recover a super-resolution image, CISIM can greatly improve the imaging speed by two orders of magnitude. Considering the imaging speed of SIM with tens of frames per second, CISIM can obtain an imaging speed of thousands of frames per second. Of course, the critical threshold of ICR depends on the complexity of the structure and dynamics of biological samples. Usually, simpler structure and dynamics of biological samples can obtain higher critical threshold of ICR and so get higher super-resolution imaging speed.
Fig. 3. Effect of image compression ratio on image reconstruction quality. The CISIM reconstructed images for the meiosis I in a living insect spermatocyte under different image compression ratios ICRs, together with the intensity profile of the selected region labeled with red line (A), and the PSNR (B) and SSIM (C) values with the increase of ICR of the CISIM reconstructed images (dots) and wild-field measured images (squares).
Fig. 4. Effect of physical noise on image reconstruction quality. The CISIM reconstructed images for the meiosis I in a living insect spermatocyte with different additive white Gaussian noises, together with the intensity profile of the selected region labeled with red line (A-F), and the corresponding PSNR (G) and SSIM (H) values of the CISIM reconstructed images (dots) and wild-field measured images (squares).
Considering the physical noise in the actual CISIM system, we further exploited the effect of additive white Gaussian noise (AWGN) on the image reconstruction quality of CISIM. Similarly, the dynamics of the meiosis I in a living insect spermatocyte are compressed to generate one image based on the data acquisition of CISIM, here the ICR is also set to 1. The CISIM reconstructed images under different noise levels are shown in Figs. 4(A-F), together with the intensity profile of the selected region labeled with red line, and the PSNR and SSIM values for the CISIM reconstructed images and wild-field measured images are also calculated and given in Fig. 4(G) and Fig. 4(H). The noise level is indicated by a noise standard derivation σ. To intuitively compare, the intensity profiles of the corresponding selected regions in the ground truth image and wild-field measured image are also given in Figs. 4(A-F). As can be seen, with the increase of σ, the CISIM reconstructed image shows blur and artifacts due to the noise effect, and both the PSNR and SSIM values decrease. However, compared with the wild-field measured image, the CISIM reconstructed image can get higher PSNR and SSIM values and clearer intensity profile of the selected region. That is, a small amount of noise does not affect the ability of CISIM to improve the image reconstruction quality. However, the larger the noise, the worse the super-resolution image reconstruction effect. Moreover, the denoiser module has been utilized in the image reconstruction of CISIM, but the noise still cannot be eliminated, which greatly depresses the image reconstruction accuracy of CISIM. In the experiment, the optical system of CISIM should be carefully calibrated to maximize the signal-noise ratio of the compressed image, and thus the super-resolution images can be well reconstructed.
3. Discussion
As shown above, CISIM provides a novel strategy to greatly improve the super-resolution imaging speed by combining SIM and CS, and the computational algorithm plays an important role, which determines the final image reconstruction quality. Here, we utilized CISIM-BM3D to recover the original dynamics. As a widely used denoiser, BM3D has high performance and good robustness. To obtain a higher image reconstruction quality and more resistance to noise, the denoisers with sparser constrain are easy to be embedded into the PnP framework in the CISIM reconstruction algorithm. For example, the weighted nuclear norm minimization (WNNM) [33,34], and even some deep-learning-based image denoisers, such as FFDNet [35] and CBDNet [36]. Besides, the temporal continuity can also be utilized to further improve the image reconstruction quality, such as VBM3D [37] and FastDVDnet [38]. Moreover, the framework of CISIM reconstruction is based on the spatial domain operation, which has the feature of high computation speed, but it is vulnerable to complex physical noise, and therefore such a framework puts forward strict requirements for the calibration of the optical sampling system. Thus, the frequency domain-based framework can be further exploited to increase the image reconstruction robustness for CISIM. In addition, deep-learning-based end-to-end algorithms can also be developed to improve the image reconstruction quality and computational efficiency of CISIM. In the future research, these computational imaging algorithms will be further explored. It should be pointed out that, compared with traditional SIM, CISIM can acquire higher super-resolution imaging speed, but it is at the cost of lower super-resolution ability due to the lower photon efficiency.
In actual experiment, the CISIM strategy is also feasible, and the experimental scheme is designed in Fig. 5. A continuous-wave laser is expanded with a beam expander (BE), and then hits a high-speed digital micromirror device (DMD1), which is used to generate strip patterns of different directions and phase-shift steps. A spatial filter (SF) is used to select the ±1st order diffraction beams for the generation of cosine fringes in three orientations via interference. A nearby vortex half-wave plate (VHP) is utilized to passively adjust the polarization of beams in different directions for the high modulation contrast. After being relayed by a 4f system that consists of a lens pair (L2 and L3), the beams are focused with an objective lens to interfere with the sample to form the structured illumination. The fluorescence signal generated by the structured light illumination is collected with the objective lens, reflected by a dichroic mirror (DM), relayed by a tube lens (TL), and then projected onto another digital micromirror device (DMD2) for spatially encoding by a 4f system consisting of a lens pair (L4 and L5). Finally, the encoded fluorescence signal is collected by the same 4f system, reflected by a beam splitter (BS) and detected by a camera. Here, DMD1 and DMD2 are synchronously triggered, and the camera is also synchronized with DMD1 and DMD2, but the working frequency of the camera is much lower than that of DMD1 and DMD2 to obtain the compressed image. This designed scheme in Fig. 5 can provide a reference for future CISIM experiment.
Fig. 5. Experimental design of CISIM. Here, BE: beam expander; DMD: digital micromirror device; L: lens; P: polarizer; SF: spatial filter; VHP: vortex half-wave plate; DM: dichromatic mirror; OL: objective lens; TL: tube lens; BS: beam splitter; CM: camera.
However, some points need to be considered in the experiment of CISIM. Firstly, the micromirror array of DMD has the effect of a reflection blazed grating, and thus the angle between the incident light field and DMD should be calibrated carefully to maximize the diffraction efficiency. Secondly, since the encoding plane is not perpendicular to the propagation direction of the light field, the final camera should also be placed with a tilt angle by the Scheimpflug principle [39] to collect a clear image, which can avoid the blur caused by the limited depth of the light field. Thirdly, the pixel size of spatial codes should be enough small to avoid the resolution loss due to the compressive sampling. In a word, the optical components of CISIM should be precisely adjusted to guarantee the compressed image acquirement with high quality, and thus the dynamics can be successfully recovered with the super-resolution images. Another limitation of CISIM is that the collected fluorescence photon efficiency will be reduced due to the additional optical elements, such as beam splitter and DMD, which will greatly affect the super-resolution image reconstruction quality. Additionally, a higher excitation power is needed compared with conventional SIM, which may induce phototoxicity and photo-bleaching. A feasible solution is to utilize complementary encoding strategy to reduce the loss of fluorescence photons [40]. These key technologies that need to be noticed in CISIM experiment are very helpful to guide the subsequent experimental studies.
In summary, we have developed a novel CISIM to achieve super-resolution imaging for high-speed dynamics. By synergizing SIM and CI, CISIM could acquire multiple super-resolution images from a single exposure. We have verified the feasibility of CISIM by reconstructing the super-resolution images of the meiosis I in a living insect spermatocyte and the mitochondrial movement in a COS-7 cell and analyzing the effects of image compression ratio and physical noise on the image reconstruction quality. Compared with conventional SIM, CISIM could improve the super-resolution imaging speed by two orders of magnitude, which has important applications in the super-resolution imaging of various high-speed biological dynamics, such as neuron firing [41], organelle dynamics during mitosis [42] and laser disruption of subcellular organelles [43]. Additionally, the CI strategy can also be extended to three-dimensional (3D) SIM [44], the super-resolution imaging speed of 3D-CISIM can be greatly improved by capturing the super-resolution cross-section in each exposure. Given the high-speed and super-resolution imaging ability of CISIM, it is expected to provide a new technical method for the study of nanoscale microsecond dynamics in physics, chemistry, and biology.
4. Methods
In conventional SIM, nine recorded images with different illumination patterns are usually required to extract a super-resolution image. Therein, three images with phase-shifted illumination patterns along a single orientation are used to uncouple spatial frequency components, and three different directions are used to provide isotropic resolution enhancement. However, because of multiple sampling, there is some information redundancy in the acquired images of SIM, which is reflected in the repetition of zero-order low-frequency information and the overlap of one-order high-frequency information among the three directions in the spatial frequency spectrum. When a microscopic dynamic process is continuously imaged using SIM, the information redundancy will further increase due to the temporal correlation and continuity of the dynamic scene. Given the abundant information redundancy, CS can be employed in SIM to improve its imaging speed. In this work, CISIM is using CS theory to break the imaging speed barrier of SIM. The flow chart of CISIM is shown in Fig. 6, which is composed of two parts: data acquisition and image reconstruction. The data acquisition mainly involves structured illumination, spatial encoding, and compressed sampling. The image reconstruction mainly employs an iterative search algorithm based on CS theory.
Fig. 6. Flow chart of CISIM. The data acquisition including structured illumination, spatial encoding and compressed sampling (top row), and the image reconstruction by iterative search algorithm (bottom row).
In data acquisition, as shown in top row of Fig. 6, the illumination light with nine structured patterns is periodically projected onto the fluorescently-labeled sample with a high-speed dynamics d(x, y, t). Nine structured patterns are assigned in three directions and three phase-shift steps p(x, y, n). The number of images included in the dynamics is denoted by T. In this way, a series of structured fluorescent images with the number of 9 T is generated, which contain the super-resolution information in different directions. After passing through an objective lens, the acquired stripped images s(x, y, t, n) can be considered as a convolution of the structured fluorescent images with the point spread function (PSF) h(x, y) of the objective lens according to the optical diffraction, which can be written as
In image reconstruction, it is worth noting that a simple concatenation of the reconstruction algorithms for CACTI and SIM does not work for CISIM, although it seems that the optical system of CISIM is just a combination of SIM and CACTI. The basic reason is that some high-frequency information will be treated as noise and abandoned in CACTI reconstruction, which actually contains the necessary super-resolution details. In the image reconstruction, it is difficult to recover a super-resolution dynamic scene from a CISIM measurement due to the difference in data processing between the reconstruction algorithms for SIM and CACTI. To solve the inverse problem for decoding the objective information of the dynamic scene from the compressed image, an integrated image reconstruction algorithm is specially designed for CISIM. This image reconstruction algorithm is an iterative searching algorithm based on a Plug-and-Play (PnP) framework [45,46], and its flow chart is shown in bottom row of Fig. 6. A random objective dynamic scene is set as the initialized iterative variable in the searching algorithm. Firstly, the current expectation of the compressed image for the iterative variable is estimated based on the optical transfer process of CISIM, which can be measured in the system calibration. Secondly, the loss between the current expectation and the measured value of CISIM is calculated. Thirdly, the iterative variable is automatically optimized by the generalized projection method based on above loss value and loss function. Fourthly, the result from the gradient descent searching is filtered with a specified image denoiser that satisfies some kind of sparse constraints. Finally, the discriminator will judge the convergence status of the algorithm. If the residual between the last iterative variable and the new one is below the threshold value, the iterative algorithm will stop searching and output the reconstruction result. Otherwise, the algorithm will continue the next iterative searching. Benefiting from the modularization of the PnP framework, image/video denoisers with different priors can be flexibly switched in this image reconstruction algorithm for CISIM, such as the classical image denoiser block-matching 3D filtering (BM3D) [47,48].
Based on CS theory, the inverse problem of the image reconstruction in CISIM can be described mathematically as
Considering the maximum likelihood estimation (MLE), the above-mentioned optimization problem can be solved iteratively using alternative steps with alternating direction method of multipliers (ADMM) [49] and is given as
Funding
National Natural Science Foundation of China (11727810, 11774094, 11804097, 12034008, 12074121, 62105101, 62175066, 91850202); Science and Technology Commission of Shanghai Municipality (19560710300, 20ZR1417100, 21XD1400900).
Acknowledgments
J. Liang acknowledges support from Ministère des Relations internationales et de la Francophonie du Québec.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Supplemental document
See Supplement 1 for supporting content.
References
1. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef]
2. H. Blom and J. Widengren, “Stimulated Emission Depletion Microscopy,” Chem. Rev. 117(11), 7377–7427 (2017). [CrossRef]
3. D. Wildanger, R. Medda, L. Kastrup, and S. W. Hell, “A compact STED microscope providing 3D nanoscale resolution,” J. Microsc. 236(1), 35–43 (2009). [CrossRef]
4. E. Rittweger, K. Y. Han, S. E. Irvine, C. Eggeling, and S. W. Hell, “STED microscopy reveals crystal colour centres with nanometric resolution,” Nat. Photonics 3(3), 144–147 (2009). [CrossRef]
5. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–796 (2006). [CrossRef]
6. B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science 319(5864), 810–813 (2008). [CrossRef]
7. S. Manley, J. M. Gillette, G. H. Patterson, H. Shroff, H. F. Hess, E. Betzig, and J. Lippincott-Schwartz, “High-density mapping of single-molecule trajectories with photoactivated localization microscopy,” Nat. Methods 5(2), 155–157 (2008). [CrossRef]
8. H. Shroff, C. G. Galbraith, J. A. Galbraith, and E. Betzig, “Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics,” Nat. Methods 5(5), 417–423 (2008). [CrossRef]
9. K. Xu, H. P. Babcock, and X. Zhuang, “Dual-objective STORM reveals three-dimensional filament organization in the actin cytoskeleton,” Nat. Methods 9(2), 185–188 (2012). [CrossRef]
10. J. Griffié, L. Shlomovich, D. J. Williamson, M. Shannon, J. Aaron, S. Khuon, G. L. Burn, L. Boelen, R. Peters, A. P. Cope, E. A. K. Cohen, P. Rubin-Delanchy, and D. M. Owen, “3D Bayesian cluster analysis of super-resolution data reveals LAT recruitment to the T cell synapse,” Sci. Rep. 7(1), 4077 (2017). [CrossRef]
11. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef]
12. L. Schermelleh, P. M. Carlton, S. Haase, L. Shao, L. Winoto, P. Kner, B. Burke, M. C. Cardoso, D. A. Agard, M. G. L. Gustafsson, H. Leonhardt, and J. W. Sedat, “Subdiffraction multicolor imaging of the nuclear periphery with 3D structured illumination microscopy,” Science 320(5881), 1332–1336 (2008). [CrossRef]
13. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A. 102(37), 13081–13086 (2005). [CrossRef]
14. E. H. Rego, L. Shao, J. J. Macklin, L. Winoto, G. A. Johansson, N. Kamps-Hughes, M. W. Davidson, and M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution,” Proc. Natl. Acad. Sci. U. S. A. 109(3), E135–E143 (2012). [CrossRef]
15. P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, and M. G. L. Gustafsson, “Super-resolution video microscopy of live cells by structured illumination,” Nat. Methods 6(5), 339–342 (2009). [CrossRef]
16. M. Li, Y. Li, W. Liu, A. Lal, S. Jiang, D. Jin, H. Yang, S. Wang, K. Zhanghao, and P. Xi, “Structured illumination microscopy using digital micro-mirror device and coherent light source,” Appl. Phys. Lett. 116(23), 233702 (2020). [CrossRef]
17. S. Dong, J. Liao, K. Guo, L. Bian, J. Suo, and G. Zheng, “Resolution doubling with a reduced number of image acquisitions,” Biomed. Opt. Express 6(8), 2946–2952 (2015). [CrossRef]
18. A. Lal, C. Shan, K. Zhao, W. Liu, X. Huang, W. Zong, L. Chen, and P. Xi, “A frequency domain SIM reconstruction algorithm using reduced number of images,” IEEE Trans. Image Process. 27(9), 4555–4570 (2018). [CrossRef]
19. F. Ströhl and C. F. Kaminski, “Speed limits of structured illumination microscopy,” Opt. Lett. 42(13), 2511–2514 (2017). [CrossRef]
20. C. Ling, C. Zhang, M. Wang, F. Meng, L. Du, and X. Yuan, “Fast structured illumination microscopy via deep learning,” Photonics Res. 8(8), 1350–1359 (2020). [CrossRef]
21. D. Dan, Z. Wang, X. Zhou, M. Lei, T. Zhao, J. Qian, X. Yu, S. Yan, J. Min, P. R. Bianco, and B. Yao, “Rapid Image Reconstruction of Structured Illumination Microscopy Directly in the Spatial Domain,” IEEE Photonics J. 13(1), 1–11 (2021). [CrossRef]
22. S. Tu, Q. Liu, X. Liu, W. Liu, Z. Zhang, T. Luo, C. Kuang, X. Liu, and X. Hao, “Fast reconstruction algorithm for structured illumination microscopy,” Opt. Lett. 45(6), 1567–1570 (2020). [CrossRef]
23. L. Jin, B. Liu, F. Zhao, S. Hahn, B. Dong, R. Song, T. C. Elston, Y. Xu, and K. M. Hahn, “Deep learning enables structured illumination microscopy with low light levels and enhanced speed,” Nat. Commun. 11(1), 1934 (2020). [CrossRef]
24. D. Li, L. Shao, B. Chen, X. Zhang, M. Zhang, B. Moses, D. E. Milkie, J. R. Beach, J. A. Hammer, M. Pasham, T. Kirchhausen, M. A. Baird, M. W. Davidson, P. Xu, and E. Betzig, “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics,” Science 349(6251), aab3500 (2015). [CrossRef]
25. X. Huang, J. Fan, L. Li, H. Liu, R. Wu, Y. Wu, L. Wei, H. Mao, A. Lal, P. Xi, L. Tang, Y. Zhang, Y. Liu, S. Tan, and L. Chen, “Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy,” Nat. Biotechnol. 36(5), 451–459 (2018). [CrossRef]
26. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526 (2013). [CrossRef]
27. X. Yuan, D. J. Brady, and A. K. Katsaggelos, “Snapshot Compressive Imaging: Theory, Algorithms, and Applications,” IEEE Signal Proc. Mag. 38(2), 65–88 (2021). [CrossRef]
28. W. Meiniel, P. Spinicelli, E. D. Angelini, A. Fragola, V. Loriette, F. Orieux, E. Sepulveda, and J. Olivo-Marin, “Reducing data acquisition for fast structured illumination microscopy using compressed sensing,” in Proceedings of the IEEE International Symposium on Biomedical Imaging (IEEE, 2017), pp. 32–35.
29. B. Ozgurun and M. Cetin, “Compressed sensing structured illumination microscopy,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine & Biology Society (IEEE, 2020), pp. 1828–1831.
30. A. Lal, C. Shan, and P. Xi, “Structured Illumination Microscopy Image Reconstruction Algorithm,” IEEE J. Sel. Top. Quant. Electron. 22(4), 50–63 (2016). [CrossRef]
31. K. Chu, P. J. McMillan, Z. J. Smith, J. Yin, J. Atkins, P. Goodwin, S. Wachsmann-Hogiu, and S. Lane, “Image reconstruction for structured-illumination microscopy with low signal level,” Opt. Express 22(7), 8687–8702 (2014). [CrossRef]
32. R. Oldenbourg and J. R. LaFountain, “CIL:9066, Nephrotoma suturalis, spermatocyte,” CIL. Dataset, (2010), [CrossRef] .
33. S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2014), pp. 2862–2869.
34. Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, “Rank Minimization for Snapshot Compressive Imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2990–3006 (2019). [CrossRef]
35. K. Zhang, W. Zuo, and L. Zhang, “FFDNet: toward a fast and flexible solution for CNN-based image denoising,” IEEE Trans. Image Process. 27(9), 4608–4622 (2018). [CrossRef]
36. S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang, “Toward convolutional blind denoising of real photographs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE/CVF, 2019), pp. 1712–1722.
37. G. Chen, T. D. Bui, A. Krzyzak, and S. Coulombe, “Adaptive video denoising using block matching 3-D filtering,” in Proceedings of the IEEE Canadian Conference on Electrical and Computer Engineering (IEEE, 2012), pp. 1–4.
38. M. Tassano, J. Delon, and T. Veit, “Fastdvdnet: Towards real-time deep video denoising without flow estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE/CVF, 2020), pp.1354–1363.
39. C. Steger, “A Comprehensive and Versatile Camera Model for Cameras with Tilt Lenses,” Int. J. Comput. Vision 123(2), 121–159 (2017). [CrossRef]
40. J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017). [CrossRef]
41. E. Yaksi and R. W. Friedrich, “Reconstruction of firing rate changes across neuronal populations by temporally deconvolved Ca2+ imaging,” Nat. Methods 3(5), 377–383 (2006). [CrossRef]
42. M. L. M. Jongsma, I. Berlin, and J. Neefjes, “On the move: organelle dynamics during mitosis,” Trends Cell Biol. 25(3), 112–124 (2015). [CrossRef]
43. W. Watanabe, N. Arakawa, S. Matsunaga, T. Higashi, K. Fukui, K. Isobe, and K. Itoh, “Femtosecond laser disruption of subcellular organelles in a living cell,” Opt. Express 12(18), 4203–4213 (2004). [CrossRef]
44. M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-Dimensional Resolution Doubling in Wide-Field Fluorescence Microscopy by Structured Illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef]
45. X. Yuan, Y. Liu, J. Suo, and Q. Dai, “Plug-and-play algorithms for large-scale snapshot compressive imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE/CVF, 2020), pp. 1447–1457.
46. K. Zhang, W. Zuo, and L. Zhang, “Deep plug-and-play super-resolution for arbitrary blur kernels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE/CVF, 2019), pp. 1671–1681.
47. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process. 16(8), 2080–2095 (2007). [CrossRef]
48. M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising using separable 4D nonlocal spatiotemporal transforms,” Proc. SPIE 7870, 787003 (2011). [CrossRef]
49. S. Boyd, N. Parikh, and E. Chu, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010). [CrossRef]
