Super-resolution localization microscopy (SRLM) enables insightful understanding of biological structure and function with the advanced ability to observe the structure, distribution, and interaction of macromolecules, organelles, and cells with a resolution well beyond the diffraction limit [1]. In SRLM, the target structure is densely labeled with molecules that are controlled to fluorescent sparsely in space and time. In this way, only a small subset of molecules are presented in a single raw image, facilitating a sparse distribution, detection, and then localization of individual molecules with nanoscale precision. After repeated acquisition of raw image frames (typically over several thousand times), precise positions of molecules (or called localizations) can be accumulated and used to reconstruct a super-resolution image [1].

Although SRLM brings incredible details of the biological world, it also faces serious challenges in data transfer, storage, and analysis [2]. Currently, researchers tend to acquire a huge amount of raw image frames (typically ranging from 10,000 to 40,000 frames [3]) in an SRLM experiment to increase detected molecules and hope to maximize the quality of a final super-resolution image. With a popular scientific complementary metal-oxide-semiconductor (sCMOS) camera, Hamamatsu Flash4.0 V3, as the detector, a final super-resolution image would require a raw data volume of up to 312.5 GB (corresponding to 40,000 raw image frames). Therefore, the computer can be easily filled up after several rounds of SRLM experiments, leading to huge pressure on further data transfer, storage, and analysis.

It is worth noting that the ultimate goal of SRLM is to resolve biological structures with a final super-resolution image rather than obtain a full location list of fluorescent molecules used to label the structures [4]. And, the key of SRLM is to isolate adjacent molecules within a diffraction-limited area into different image frames, and then determine their positions with a localization precision value much smaller than the diffraction limit. However, even if adjacent molecules are separated into different frames, they can also be too close (e.g., 5 nm) to be resolved with a certain localization precision (e.g., 15 nm). Hence, practically in SRLM, the resolving power of a final super-resolution image is usually limited by localization precision, and excess acquisition of adjacent molecules will present redundant localizations, which bring negligible resolving power improvement. In other words, it may be possible to find an ending point of a certain SRLM experiment to minimize excess acquisition of such redundant localizations, without sacrificing the resolving power. Furthermore, automatic ending would be especially helpful for applying SRLM in high-content screening [5] or high-throughput imaging [6], where even a small amount of excess acquisition in each experiment would result in significant waste of labor and resources since hundreds of repeated experiments are involved.

Here we present structure-resolving index (SRI) as an efficient criterion for automatically determining the ending point of an SRLM experiment. SRI explores the temporal relationship among localizations and can theoretically reduce the excess data volume brought by redundant localizations from the same molecules with multiblinking property [7] or adjacent molecules that are too close to be resolved. Using simulated and experimental data, we verify that SRI ensures the resolving power of a super-resolution image with minimized data volume.

We assume that a local structure has been resolved if the spatial distribution of a group of molecules well defines the structure with localizations. Here the local resolved structure (called r-Spot in this study) is determined by the spatial distribution of a group of adjacent molecules and corresponding localization precision. Since localizations are used to present molecule distribution in SRLM imaging, we use localizations to estimate r-Spot. At any given time point, r-Spots are calculated from the localizations obtained from previous image frames (hereafter called previous localizations). We propose that, if there is no new r-Spot recognized from the localizations in new frames (hereafter called new localizations), the image acquisition can be ended with negligible quality loss in the final super-resolution image. The relations among previous localizations, new localization, and r-Spot are shown in Fig. 1(a).

 

Fig. 1. Definition and calculation of SRI. (a) The relation between localizations and r-Spots. Previous (red cross) and new localizations (blue cross) are both Gaussian-distributed around their nearest r-Spot (black dot). (b) The normalized possibility density function of the distance between new localizations and their nearest r-Spot when $\sigma =10\mathrm{nm}$. (c) Scheme of using SRI to end image acquisition.

Download Full Size | PDF

Mathematically, an SRLM experiment can be ended if the distances between all new localizations and their nearest r-Spots obey Gaussian distribution

Figure 1(c) describes the scheme of using SRI as an ending criterion of image acquisition. We group previous localizations into r-Spots based on their relative positions, after considering localization precision. Given that the number of localizations from thousands of molecules could be up to several millions, classic cluster analysis methods ($K$-means, DBSCAN, etc.) will take hours to group these localizations into r-Spots. To realize fast SRI calculation, we developed a simplified cluster analysis method consisting of the following steps:

 

Fig. 2. Relation between SRI and image acquisition time on simulated circle. (a), (d) Scattered images of molecules. (b), (e) Rendered images from the localizations obtained before SRI reaches 0.5. (c), (f) Time-dependent SRI (cyan squares) and rMole (blue dots). The red lines represent the SRI threshold (0.5). Data points are from 10 repeated measurements. Molecules are uniformly distributed in a circle with an interval distance of (a)–(c) 60 nm and (d)–(f) 12 nm, corresponding to visually nonoverlapping and overlapping molecule density, respectively. Localization precision, 10 nm. Scale bars, 150 nm.

Download Full Size | PDF

 

Fig. 3. Image quality comparison with different ending criteria on simulated NPCs and microtubules. (a), (b) The relation among ending time, ending criteria, and localization precision. (a) and (b) are from NPCs and microtubules, respectively. Three criteria are compared here: FRC (red dots), SRI (green triangles), and rMole (blue squares). Data points are from 10 repeated measurements. (c)–(f) Rendered RGB (R, G, B channels are based on FRC, SRI, and rMole, respectively) and zoom-in single-channel images from accumulated localizations before each ending criterion is reached. Here (c), (d) NPCs and (e), (f) microtubules are simulated under different localization precision: 5 nm for (c), (e); 20 nm for (d), (f). Estimated complex radiuses of NPC across the full-field image are 41.3 nm (FRC), 44.0 nm (SRI), 44.3 nm (rMole) in (c), and 46.8 nm (FRC), 46.1 nm (SRI), 46.8 nm (rMole) in (d). (g), (h) Intensity profiles along the lines drawn in (e) and (f). FWHM in (g): 30.8 nm (FRC), 25.9 nm (SRI), 24.2 nm (rMole); FWHM in (h): 48.4 nm (FRC), 48.7 nm (SRI), 46.6 nm (rMole). Scale bars: 1 μm [(c)–(d), left panels], 90 nm [(c)–(d), right panels]; 2 μm [(e)–(f), top panels], 200 nm [(e)–(f), bottom panels].

Download Full Size | PDF

 

Fig. 4. Image quality comparison with different ending criteria on experimental NPCs and microtubules. (a), (c) Rendered RGB (R, FRC; G, SRI; B, Whole) and zoom-in single-channel images from accumulated localizations before each ending criterion is reached [NPCs (a) and microtubules (c)]. Whole is rendered with localizations obtained from all raw images (20,000 frames for NPCs, and 10,000 frames for microtubules). (b), (d) Time-dependent SRI (green lines) and FRC (red dashes) values on NPCs (b) and microtubules (d). The arrows show the ending times based on SRI and FRC, respectively. (e) Intensity profiles along the line drawn in (c). FWHM: 66.9 nm (FRC), 68.2 nm (SRI), 65.3 nm (Whole). Scale bars: 3 μm [(a), left panel], 200 nm [(a), right panel]; 4 μm [(c), left panel], 500 nm [(c), right panel].

Download Full Size | PDF

After Steps 1–3, a r-Spot list is obtained. Then, SRI is calculated with each new localization assigned to its nearest r-Spot. If SRI is less than 0.5, a certain number of new raw image frames are further acquired. For accurate SRI calculation, the number of new frames should be set to ensure that the number of new localizations is larger than the number of r-Spots. Repeat image acquisition and SRI calculation until SRI reaches 0.5.

We compared the performance of our simplified cluster analysis method with two well-known clustering methods used in SRLM, including DBSCAN and K-means [10] on simulated data, where 300 isolated clusters are randomly distributed in a $2\mathrm{\mu m}\times 2\mathrm{\mu m}$ area and each cluster contains 10 localizations with 10 nm localization precision. To estimate the capability of different clustering algorithms in grouping localizations into r-Spots, we defined accuracy as the average distance between the r-Spot and the cluster. Using 10 repeated measurements, we verified that our clustering algorithm ($7.2\pm 0.1\mathrm{nm}$) exhibits better accuracy than DBSCAN ($8.8\pm 0.1\mathrm{nm}$) and $K$-means ($8.8\pm 0.1\mathrm{nm}$).

We further show the behavior of SRI as an ending criterion with a different relation of molecule distance and localization precision. In the case where the distances between each molecule and its nearest molecule are all greatly larger than r, molecules are “visually nonoverlapping” in a final super-resolution image [Figs. 2(a) and 2(b)]. One molecule determines one r-Spot. Any undetected molecules could cause quality loss in the super-resolution image. When all molecules are detected, SRI reaches 0.5 [Fig. 2(c)], which is consistent with our previous theoretical prediction. SRI not only ensures that each molecule in the structure is represented in the final super-resolution image [Fig. 2(b)], but also reduces the need of using extra image frames to capture redundant localizations from the same molecule. Note that a molecule may be switched on/off (also called blinking) and detected for more than 100 times [7], thus producing a large number of redundant localizations.

In the case where the distances between each molecule and its nearest molecule are all close to or less than r, molecules are “visually overlapping” in a final super-resolution image [Figs. 2(d) and 2(e)]. If a group of adjacent molecules detected in previous image frames already determines one r-Spot, subsequent repeated acquisition of adjacent molecules could not enhance the resolving power in the super-resolution image. Before all molecules are detected, SRI reaches 0.5 [Fig. 2(f)], confirming our previous theoretical prediction. SRI ensures the maximum resolving power in the final super-resolution image [Fig. 2(e)], while reducing the need of using extra image frames to capture redundant localizations from adjacent molecules. Note that these extra frames produce excess data volume, but bring negligible improvement in the resolving power of the super-resolution image.

In this study, we compared the performance of our proposed ending criterion, SRI, with the Fourier ring correlation (FRC) [11] resolution, a well-known quantitative method in SRLM. Although FRC resolution has not been reported as an ending criterion, it has good potential to be used for this purpose when FRC resolution gets stable ($<1\mathrm{nm}$ in the standard deviation), since further acquisition brings negligible resolution improvement. For simulated data, we used an additional ending criterion called rMole, which is the ratio between the number of acquired molecules and the total number of molecules in the whole dataset. When rMole reaches 1, the acquisition can be ended, since all molecules are already presented in the final image.

To evaluate the performance difference among the three ending criteria, we generated simulated datasets using two representative structures: nuclear pore complexes [NPCs, for randomly distributed structures, Figs. 3(c) and 3(d)], and microtubules [for line structures, Figs. 3(e) and 3(f)]. The dataset for NPCs consists of randomly distributed complexes, and eight protein clusters distribute in each NPC with high symmetry. The radius of each NPC is set to be 50 nm, and the total antibody length is set to be 15 nm [3]. The width of the lines in simulated microtubules is set to be 40 nm [2]. These datasets simulate scenarios with different localization precision (from 5 to 20 nm) with random molecule activation.

From the simulated data in Fig. 3, we found that (1) for both structures, the ending time based on FRC increases with the value of localization precision, while the ending time based on SRI exhibits an opposite trend [Figs. 3(a) and 3(b)]; as expected, the ending time based on rMole is unrelated to localization precision; (2) for the same localization precision, SRI provides more stable ending times than FRC, especially on randomly distributed structure [Fig. 3(a)]; (3) for high localization precision (5 nm), FRC ends acquisition earlier than SRI. However, this earlier ending may cause structure missing in the final super-resolution image [Figs. 3(c) and 3(e)]. Further analysis of the estimated complex radiuses [Fig. 3(c)] and the FWHM values [Fig. 3(g)] verified that the SRI-based image is closer to the ground truth image (rMole) than the FRC-based image; (4) for low localization precision (20 nm), FRC ends acquisition later than SRI. However, the acquisition of more image frames brings negligible improvement to the image quality [Figs. 3(d) and 3(f)], as confirmed by the similar estimated complex radiuses [Fig. 3(d)] and FWHM values [Fig. 3(h)]. From the findings above, we verified that SRI is a better ending criterion than FRC.

We further testified the performance of SRI on experimental images. In our SRLM imaging, NPCs [Fig. 4(a)] or microtubules [Fig. 4(c)] in U-2 OS cells were labeled with primary antibody (NPCs: rabbit anti-NUP133 antibody, PA5-63774, Invitrogen; microtubules: mouse monoclonal anti–$\alpha$-tubulin antibody, T5168, Sigma) and secondary antibody (NPCs: CF680 labeled donkey anti-rabbit IgG, 20820, Biotium; Microtubules: Alexa Fluor 647 labeled Goat anti-Mouse IgG, A-21235, Invitrogen), according to the protocols described in [12]. The cells were soaked in standard buffer [12], then imaged on a home-built SRLM system based on an Olympus IX71 microscope, a 100X/NA1.4 Olympus objective, a 640 nm laser at the intensity of $\sim 5\mathrm{kW}{\mathrm{cm}}^{-2}$, and a Hamamatsu Flash4.0 V3 camera with the exposure time of 10 ms. Image analysis was performed with MaLiang [13] plug-in on ImageJ.

From Fig. 4, we observed that for both structures, SRI ends SRLM experiments earlier than FRC [Figs. 4(b) and 4(d)], while presenting comparable image quality [Figs. 4(a), 4(c), and 4(e)]. Specifically, for the NPC sample, the ending times are 10,000 frames for FRC and 7,400 frames for SRI, respectively. This earlier ending from SRI provides 26.0% reduction in data volume, but brings negligible loss in the resolving power, as shown by the similar structures in Fig. 4(a). We observed an even greater data reduction (from 7,100 frames to 3,200 frames, 54.9%) in the microtubules sample [Fig. 4(d)]. Line profiles of microtubules further confirm the negligible quality loss [see the similar FWHM results in Fig. 4(e)]. Interestingly, we observed less artifacts in the SRI-based image than FRI and Whole [Fig. 4(c)], probably because more free fluorescent molecules are detected in extra frames.

Finally, we evaluated the computation speed of the SRI criterion. We noticed that the time cost in SRI calculation is proportional to $N^2$ and the number of new localizations. Here $N$ is the size of the scatter image (see Step 1). For a scatter image of $200\mathrm{\mu m}\times 200\mathrm{\mu m}$ (for typical sCMOS cameras), with 20 nm pixel size and 20,000 new localizations, the SRI calculation costs about 5 s with MATLAB on an Intel i5-7400 at 3.00 GHz personal computer. Hence, if we perform molecule localization online (which is achievable according to [14]) and calculate SRI every 500 frames with 10 ms exposure time, we can end the experiment with only a constant delay of 5 s. The delay is possible to be further reduced with GPU parallel computation or another acceleration method. Note that 500 frames corresponds to a data volume of 3.9 GB for a typical sCMOS camera.

In summary, after exploring the data redundancy in localizations, we present an efficient criterion, called SRI, for automatically ending image acquisition in SRLM. Through theoretical analysis, we have shown that SRLM experiments can be ended with satisfying resolving power, when SRI reaches its theoretical threshold (0.5). In this way, the excess data volume brought by redundant localizations from multiblinking or adjacent molecules can be reduced. Using simulated and experimental datasets, we have verified the superior performance of SRI as an ending criterion. Compared to the potential FRC criterion, SRI reduces data volume of 26.0% for NPCs and 54.9% for microtubules, while ensuring comparable image quality. SRI can also be combined with localization-based approaches for faster SRLM [15] to further reduce data volume. Furthermore, SRI automatically ends experiments with only a small and constant delay (5 s in this study), which could be helpful for enabling long-term and automatic operation in high-content or high-throughput SRLM [5,6].