Super-resolution optical fluctuation imaging (SOFI) delivers an enhanced spatial resolution in fluorescence imaging by analyzing spontaneous fluctuations in the fluorophore emission. SOFI images are usually obtained by calculating cross-cumulants between adjacent pixels on the detector, which provides the increased pixel densities required to sample the improved spatial resolution, but can result in pixelation artifacts. In this contribution, we describe a simple, model-free, and computationally efficient algorithm to correct such artifacts by matching the means and variances of the different cross-cumulants. We show that this strategy not only results in pixels that are essentially free of artifacts, but can also correct for detector imperfections such as the spurious correlations present in electron-multiplied CCD cameras.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Super-resolution optical fluctuation imaging (SOFI) is an established approach for sub-diffraction fluorescence imaging. It delivers a theoretically-unlimited spatial resolution by sampling the stochastic emission fluctuations or ‘blinking’ events present in many types of fluorophores . The technique has been applied to a wide variety of samples, with fluorophores including fluorescent proteins [2,3], organic dyes , genetically-encoded biosensors [5,6], and quantum dots . Compared to other sub-diffraction approaches, SOFI imaging can work over a broad range of conditions including low emitter brightness, high emitter density, and/or high background, with a temporal resolution of seconds and typically an up to two- or three-fold improvement in spatial resolution [7–9]. Especially, when quantitative comparisons between samples need to be performed, much can be gained from the robust nature of the analysis when compared to localization microscopy . The technique can be applied standalone or can even be combined with other modalities to deliver an enhanced imaging performance [11–13].
The full theoretical description of the image formation shows that the SOFI imaging process is reliable if the sample is stationary and the emitters show independent fluorescence fluctuations . This analysis process can also be extended to separate emitters based on their blinking dynamics , or to correct for photodestruction or diffusion of the emitters [16,17]. Furthermore, the accuracy of the resulting images can be readily estimated using statistical resampling, which also allows the estimation of the sample quality , and, when combined with other metrics, can lead to quantitative and objective tools to evaluate the quality of SOFI images .
In practice, SOFI operates on stacks of wide-field images acquired from the same position on the sample, where a single stack containing hundreds or more fluorescence images is analyzed to obtain a single sub-diffraction SOFI image. The analysis calculates the cumulants of the intensity distributions seen in every detector pixel, where the order of the cumulants used is known as the order of the SOFI calculation. The values of these cumulants then become the pixel values in the resulting SOFI image. To avoid shot noise, one usually does not calculate autocumulants, but rather calculates cross-cumulants between different detector pixels (analogous to the distinction between variance and co-variance). These cross-cumulants then result in the creation of ‘virtual pixels’ , which makes it possible to exploit the increase in spatial resolution by increasing the sampling density of the calculated SOFI image. Alternatively, one can also use Fourier interpolation to increase the sampling density , though to our knowledge the calculation of virtual pixels remains the most common methodology.
A typical SOFI image thus consists of multiple different types of virtual pixels, where each type of virtual pixel results from a distinct combination of detector pixels at different distances from each other (Fig. 1(a)). However, since the signal of a blinking molecule correlates only with itself, the SOFI signal arising from a cross-cumulant will scale with the distances between the detector pixels relative to the size of the point spread function (PSF). This means that the SOFI image will in general show pixelation artifacts due to the different scale factors associated with each virtual pixel type, which must be corrected.
Previous work suggested the calculation of appropriate correction factors based on knowledge of the PSF . Alternatively, if exact information on the PSF is unavailable, the authors suggested an iterative search for suitable correction factors by optimizing a Gaussian model for the unknown PSF. In practice, however, we found that this approach does not always lead to pixelation-free SOFI images on real datasets, even when the PSF can be considered to be Gaussian-like and is known, hinting to the presence of additional contributions that are not well captured by this method.
While the theoretical description of SOFI imaging assumes that signal arises only from the blinking of the fluorophores, in practice the SOFI signal may arise from a range of other sources, including photodestruction, probe diffusion, but also imperfections in the used detector. We used SOFI to judge the relative performance of industrial CMOS, scientific CMOS (sCMOS), and electron-multiplied CCD (EM-CCD) cameras , and observed that the industrial CMOS camera showed imperfections that contributed spurious SOFI signals. Such distortions not only affect the raw values of both, the conventional and SOFI images, but can also interfere with the required pixelation correction. We therefore wondered whether it is possible to identify alternative strategies that provide a more robust correction.
In this contribution, we describe a model-free and computationally efficient strategy for the correction of cross-cumulant pixelation in SOFI imaging. We show how this strategy is straightforward to implement and can also results in increased robustness against detector imperfections.
The simulation data were obtained following the simulation procedure described in Ref. , consisting of randomly distributed lines of 50,000 emitters with a brightness of 45000 photons per second within 100 by 100 pixel images with a simulated optical pixel size of 100 nm. The emitters switch independently and randomly between two states. The on-time ratio is $14$ % with an on-time of $ 10 $ ms. The simulated movie consisted of 2000 fluorescence images with as simulated exposure time of 10 ms. All calculations were performed using Igor Pro 8 (WaveMetrics, Inc.) and the Localizer package .
2.2 Live cell data
The sample preparation and imaging of the live cells was detailed in Ref. . A commercial cellTIRF microscope (Olympus) and an ImageEM camera were used for the imaging of HeLa cells labeled with Lyn-tagged rsGreen0.8, excited at $488 $ nm. The camera exposure time was set to $ 31 $ ms, and the EM gain was adjusted for an optimal dynamic range. $2,000$ frames were acquired for the dataset shown here.
A detailed description of the sample preparation and acquisition of live cell data with a Hamamatsu Fusion sCMOS camera was given in . In brief, a Nikon Eclipse TI2 motorized microscope was used in combination with an Oxxius L6Cc Laser combiner box which provided $488 $ nm excitation light. A dataset of 500 frames of microtubules of Cos-7 cells was acquired with the sCMOS camera. The microtubules were stained with ffDronpa linked to microtubule-associated protein 4 (MAP4).
2.3 EM-CCD dark images
The EM-CCD dark current data without any incident light was acquired using the Andor iXon Ultra 897 with a temperature setting of $20$ °C. The pixel readout rate was set to $17 $ MHz and $1 $ MHz, depending on the measurement. The electron multiplier gain and the exposure time were set to 100 and 1 s, respectively. 50,000 frames were acquired for each dataset. We used an Andor camera for this acquisition instead of the Hamamatsu camera simply because the cooling can be switched off in this system, supported by our observation that all of the EM-CCD cameras that we have tested over our years of performing SOFI imaging show very similar distortions.
We excluded $ 0.005 $ % of the sensor pixels from the average and variance estimation since these appeared to have been affected by background radiation (cosmic rays or similar) at some point during the 13 hour measurements.
3. Results and discussion
Figure 1(a) illustrates how the virtual pixels in a SOFI image arise from the calculation of cross-cumulants between different detector pixels. Using this approach, a second-order SOFI image has effectively four times more pixels compared to the raw fluorescence images from which it was calculated.
We first examined a simulated dataset of randomly distributed lines of blinking emitters (Fig. 1(b)), which clearly showed extensive pixelation artifacts that must be corrected to result in accurate SOFI images (Fig. 1(c)). As was outlined in the introduction, Dertinger and colleagues  suggested the direct calculation of compensating scale factors from a Gaussian PSF model, either based on direct knowledge of the PSF, or by iteratively optimizing such a model to minimize the pixelation artifacts. This strategy indeed works well for simulated data (Fig. 1(d)), though it requires that the PSF can be well approximated by a Gaussian function and introduces the additional computational overhead and uncertainty inherent in using iterative nonlinear solvers.
Surprisingly, we observed that this strategy did not appear to work well on actual measured SOFI images, even in situations where the PSF was found to be well-approximated by a Gaussian function. Figure 2(a) and 2(b) reveal that pixelation still occurs despite the correction procedure, as is also confirmed by the presence of high spatial frequency components in the Fourier transforms of the uncorrected and corrected SOFI images. The iterative correction approach does succeed in reducing the high frequency components associated with the pixelation artifacts, though their effects remain apparent.
This finding motivated us to devise a correction strategy that is both easier to calculate and imposes fewer assumptions on the system. Our key observation was that the pixelation of the detector is independent from the structure of the sample. For a sufficiently large imaged region, it is therefore reasonable to assume that the average signals of the different virtual pixels must be equal. Our initial correction approach was to calculate the average value of each type of SOFI virtual pixel, and then to apply a scale factor to all but one of these pixels such that they all have the same average signal. In principle, it does not matter which virtual pixel is chosen as the reference value since the linearity of the imaging with respect to the fluorophore concentration will still be preserved, and different SOFI images can still be compared directly as long as one consistently picks the same virtual pixel type as the reference.
We first tested whether this strategy worked successfully on simulated data, observing that the resulting images indeed did not show any pixelation artifacts (Fig. 1(e)). However, we did observe residual pixelation artifacts when applying this strategy to actual data recorded using an EM-CCD camera, similar to what we observed using the iterative correction approach (Fig. 2(c)).
While the origin of the discrepancy between the simulated and measured data remained unclear, we reasoned that the intensity distributions of the different virtual pixels should not only show the same average value, but also the same variance. We therefore extended our correction process by retaining both a scaling factor but also adding an offset to each virtual pixel value, such that both the mean and variance are matched for all of the virtual pixels. For each pixel with a value $x$, a corrected value $y = a_i x + b_i$ can be determined with the slope $a_i$ and y-intercept $b_i$ associated to the pixel’s virtual pixel type:
The index $i=2, 3, \ldots$ refers to the virtual pixel types, using the virtual pixel type denoted with $i=1$ as a reference which is not adjusted during the pixelation correction. The variance $\sigma _i^2$ and mean $\mu _i$ are calculated from all uncorrected pixel values of the pixel type $i$. Figure 2(d) shows an application of this correction algorithm, which indeed reveals a very good suppression of the high-frequency signals in the associated Fourier transform. The pixelation correction for this dataset increases the runtime by less than 500 ms on a commercially available laptop, corresponding to approximately $ 1.3 $ % of the SOFI calculation runtime. It therefore has a very small impact on the overall calculation time. Importantly, this correction algorithm is easy and fast to calculate, essentially eliminates all pixelation artifacts, and does not require any knowledge of the particulars of the imaging.
Though we have not described this approach before, the algorithm has been integrated in our Localizer software package . For several years, it has been successfully and routinely applied on the many datasets that we have since acquired on a variety of different cameras, including also higher-order SOFI images. A similar strategy was independently reported in Ref. , though this included only the estimation of scale factors by equalizing the variances, entirely analogous to our original approach of equalizing the averages. Though it has been very useful, our expansion of the strategy to include both mean and variance equalization was largely empirical in nature, since the SOFI image formation model predicts that scaling the intensities to match the average values (or variances) should be sufficient to correct all pixelation. Indeed that is also confirmed by the simulations. So why did we need to also correct for the variance? We reasoned that the deviations observed in real datasets must originate from additional correlations in the measurement process, most likely from limitations in the camera used to record the images. We examined other datasets acquired using different kinds of EM-CCD cameras, and indeed found that the inclusion of both average and variance matching was required to achieve an adequate pixelation correction for all of these.
To better understand this observation, we examined the raw values for four different virtual pixels, calculated in the horizontal, vertical, and diagonal direction (Fig. 1(a)), on actual measured data. The diagonal and the vertical combinations have similar pixel values, but to our surprise the horizontal virtual pixel differed markedly (Fig. 3(a)), even though its intensity distribution should in principle be identical to those of the vertical pixel combination. This deviation was found in all inspected experimental datasets, and was consistently found associated with the horizontal pixel combination. This asymmetry was reminiscent of the readout process of an EM-CCD camera, where each of the pixel values is read out by a single amplifier. Pixels in a single line are consecutively shifted and read out at a rapid rate by this amplifier (‘serial’ direction), while the different lines are shifted towards the readout line in a much slower process (‘parallel’ direction). The deviating SOFI values would then indicate the presence of correlations in the camera readout process, where the readout value of a given pixel depends not only on the charge contained in that pixel, but also on the charge of the pixel(s) that preceded it in the readout process.
While this explanation aligned with our observations, a clear analysis of this phenomenon was complicated because the observed signals included contributions both from the sample and from possible other factors such as detector imperfections. We therefore sought a more robust way of investigating these distortions, independent from the peculiarities of the sample. Homogeneous illumination of the detector would provide a solution to this issue, but can be surprisingly difficult to realize in practice. Instead, we reasoned that the dark current signal from spontaneously induced electrons could be used as a sample- and instrument-independent data source.
Figure 3(b) and 3(c) show an average and SOFI image of 50,000 dark images acquired on an EM-CCD camera for which the sensor cooling had been set to $20 $ °C. We do not fully understand why the average image displays a clear pattern, which was identified as an ‘annealing pattern’ by the camera manufacturer. Such patterns arise through the back-thinning manufacturing process, and should be unobservable in the absence of illumination. We assume that very small amounts of lights are generated within the system that become observable over the 13 hour integration. Repetitions of the dark current measurement under different conditions reinforced this assumption. The experiment was executed under different lab lighting conditions, during which the camera sensor was always completely shielded by a protective cover, and, hence, should not have been influenced by the outside conditions. The resulting average images showed the same signal strength and overall pattern. As before, the virtual pixel histograms reveal an aberrant distribution for the horizontal virtual pixel oriented along the serial direction (Fig. 3(d)), entirely similar to what was observed in the real measurement data. This distortion was reduced by lowering the readout rate from 17 MHz to 1 MHz, further supporting our assignment of this deviation to correlations in the readout process.
The observed offset between the pixel combinations readily explains why a simple scaling factor was inadequate to fully correct the pixelation artifacts. Matching both the mean and variance of the virtual pixels makes it possible to not only scale the SOFI pixel values, but also to shift these by an offset. The result of applying such a correction is shown in Fig. 3(e), demonstrating that readout-induced distortions inherent to the EM-CCD imaging can be adequately compensated using our correction scheme. While this correction does make the images more appealing, more fundamentally it also restores the quantitative nature of SOFI imaging with respect to concentration, which was disturbed by the additional offset signal.
We next examined to what extent the introduced pixelation correction approach can also be useful for sCMOS-based detectors. Figure 4 shows a dataset of ffDronpa-stained microtubules of Cos-7 cells acquired using a sCMOS camera. As expected, the uncorrected SOFI image showed clear pixelation artifacts (Fig. 4(a)). Applying the PSF optimization (Fig. 4(b)) and the average and variance matching (Fig. 4((c)) led to a similar reduction in pixelation artifacts, also confirmed by the Fourier transforms. These Fourier transforms show a non-symmetric behaviour around the center which stems from the sample structure and should, hence, not be removed by any pixelation correction. The larger numbers of pixels associated with sCMOS cameras also highlights the computational efficiency of our approach, requiring around two seconds to process these images in comparison to over an hour for the PSF optimization approach.
In this contribution, we have examined the correction of pixelation artifacts in cross-cumulant based SOFI imaging. We initially proposed a straightforward and model-free correction approach in which the values of the different pixel combinations are scaled such that all pixel combinations show the same average value. This approach is consistent with the theoretical model of SOFI imaging and works well on simulated data.
However, we found that residual pixelation remained observable in real data even when the average-matching correction strategy is applied. Detailed examination of the images and of thermal noise data showed that this residual distortion was due to the presence of imperfections in the camera readout process. These imperfections could be adequately suppressed by introducing an additional additive correction term such that the averages and variances of the different pixel combinations are matched.
Overall, our strategy provides a straightforward and robust method for the correction of cross-cumulant induced pixelation, which does not require any knowledge on the particulars of the imaging system or sample. Our results also confirm that cross-cumulant or cross-correlation based image assessment is a straightforward approach to characterize the non-idealities of imaging systems.
European Research Council (714688); Fonds Wetenschappelijk Onderzoek (G090819N).
We thank Sam Duwé (Hasselt University), Robin Van den Eynde (KU Leuven) and Marcel Müller (Bielefeld University) for acquiring the data shown in Fig. 2 and Fig. 4. V.G. thanks the Research-Foundation Flanders (FWO) for a doctoral fellowship. This work was supported by the FWO via grant G090819N and the European Research Council via grant 714688 NanoCellActivity.
The authors declare that there are no conflicts of interest related to this article.
1. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proc. Natl. Acad. Sci. 106(52), 22287–22292 (2009). [CrossRef]
2. P. Dedecker, G. C. H. Mo, T. Dertinger, and J. Zhang, “Widely accessible method for superresolution fluorescence imaging of living systems,” Proc. Natl. Acad. Sci. U. S. A. 109(27), 10909–10914 (2012). [CrossRef]
3. A. G. Tebo, B. Moeyaert, M. Thauvin, I. Carlon-Andres, D. Böken, M. Volovitch, S. Padilla-Parra, P. Dedecker, S. Vriz, and A. Gautier, “Orthogonal fluorescent chemogenetic reporters for multicolor imaging,” Nat Chem Biol (2020).
4. K. Grußmayer, T. Lukeš, T. Lasser, and A. Radenovic, “Self-Blinking Dyes Unlock High-Order and Multiplane Super-Resolution Optical Fluctuation Imaging,” ACS Nano 14(7), 9156–9165 (2020). [CrossRef]
5. G. C. H. Mo, B. Ross, F. Hertel, P. Manna, X. Yang, E. Greenwald, C. Booth, A. M. Plummer, B. Tenner, Z. Chen, Y. Wang, E. J. Kennedy, P. A. Cole, K. G. Fleming, A. Palmer, R. Jimenez, J. Xiao, P. Dedecker, and J. Zhang, “Genetically encoded biosensors for visualizing live-cell biochemical activity at super-resolution,” Nat. Methods 14(4), 427–434 (2017). [CrossRef]
6. F. Hertel, G. C. H. Mo, S. Duwé, P. Dedecker, and J. Zhang, “RefSOFI for Mapping Nanoscale Organization of Protein-Protein Interactions in Living Cells,” Cell Rep. 14(2), 390–400 (2016). [CrossRef]
7. S. Geissbuehler, C. Dellagiacoma, and T. Lasser, “Comparison between sofi and storm,” Biomed. Opt. Express 2(3), 408–420 (2011). [CrossRef]
8. H. Deschout, T. Lukes, A. Sharipov, D. Szlag, L. Feletti, W. Vandenberg, P. Dedecker, J. Hofkens, M. Leutenegger, T. Lasser, and A. Radenovic, “Complementarity of PALM and SOFI for super-resolution live-cell imaging of focal adhesions,” Nat. Commun. 7(1), 13693 (2016). [CrossRef]
9. A. Girsault, T. Lukes, A. Sharipov, S. Geissbuehler, M. Leutenegger, W. Vandenberg, P. Dedecker, J. Hofkens, and T. Lasser, “SOFI Simulation Tool: A Software Package for Simulating and Testing Super-Resolution Optical Fluctuation Imaging,” PLoS One 11(9), e0161602 (2016). [CrossRef]
10. T. Lukeš, D. Glatzová, Z. Kvíčalová, F. Levet, A. Benda, S. Letschert, M. Sauer, T. Brdička, T. Lasser, and M. Cebecauer, “Quantifying protein densities on cell membranes using super-resolution optical fluctuation imaging,” Nat. Commun. 8(1), 1731 (2017). [CrossRef]
11. A. Descloux, K. S. Grußmayer, A. Bostan, A. Sharipov, S. Geissbuehler, A.-L. Mahul-Mellier, H. A. Lashuel, M. Leutenegger, and T. Lasser, “Combined multi-plane phase retrieval and super-resolution optical fluctuation imaging for 4D cell microscopy,” Nat. Photonics 12, 165–172 (2018). [CrossRef]
12. A. Purohit, W. Vandenberg, T. Dertinger, D. Wöll, P. Dedecker, and J. Enderlein, “Spatio-temporal correlation super-resolution optical fluctuation imaging,” Europhys. Lett. 125(2), 20005 (2019). [CrossRef]
13. A. Sroda, A. Makowski, R. Tenne, U. Rossman, G. Lubin, D. Oron, and R. Lapkiewicz, “SOFISM: Super-resolution optical fluctuation image scanning microscopy,” Optica 7(10), 1308–1316 (2020). [CrossRef]
14. W. Vandenberg, M. Leutenegger, S. Duwé, and P. Dedecker, “An extended quantitative model for super-resolution optical fluctuation imaging (SOFI),” Opt. Express 27(18), 25749–25766 (2019). [CrossRef]
15. S. Duwé, W. Vandenberg, and P. Dedecker, “Live-cell monochromatic dual-label sub-diffraction microscopy by mt-pcSOFI,” Chem. Commun. 53(53), 7242–7245 (2017). [CrossRef]
16. Y. Peeters, W. Vandenberg, S. Duwé, A. Bouwens, T. Lukeš, C. Ruckebusch, T. Lasser, and P. Dedecker, “Correcting for photodestruction in super-resolution optical fluctuation imaging,” Sci. Rep. 7(1), 10470 (2017). [CrossRef]
17. W. Vandenberg and P. Dedecker, “Effect of probe diffusion on the SOFI imaging accuracy,” Sci. Rep. 7(1), 44665 (2017). [CrossRef]
18. W. Vandenberg, S. Duwé, M. Leutenegger, B. Moeyaert, B. Krajnik, T. Lasser, and P. Dedecker, “Model-free uncertainty estimation in stochastical optical fluctuation imaging (SOFI) leads to a doubled temporal resolution,” Biomed. Opt. Express 7(2), 467–480 (2016). [CrossRef]
19. B. Moeyaert, W. Vandenberg, and P. Dedecker, “SOFIevaluator: a strategy for the quantitative quality assessment of SOFI data,” Biomed. Opt. Express 11(2), 636–648 (2020). [CrossRef]
20. T. Dertinger, R. Colyer, R. Vogel, J. Enderlein, and S. Weiss, “Achieving increased resolution and more pixels with Superresolution Optical Fluctuation Imaging (SOFI),” Opt. Express 18(18), 18875–18885 (2010). [CrossRef]
21. S. C. Stein, A. Huss, D. Hähnel, I. Gregor, and J. Enderlein, “Fourier interpolation stochastic optical fluctuation imaging,” Opt. Express 23(12), 16154–16163 (2015). [CrossRef]
22. R. Van den Eynde, A. Sandmeyer, W. Vandenberg, S. Duwé, W. Hübner, T. Huser, P. Dedecker, and M. Müller, “Quantitative comparison of camera technologies for cost-effective super-resolution optical fluctuation imaging (SOFI),” JPhys Photonics 1(4), 044001 (2019). [CrossRef]
23. P. Dedecker, S. Duwé, R. K. Neely, and J. Zhang, “Localizer: fast, accurate, open-source, and modular software package for superresolution microscopy,” J. Biomed. Opt. 17(12), 126008 (2012). [CrossRef]
24. S. Duwé, E. De Zitter, V. Gielen, B. Moeyaert, W. Vandenberg, T. Grotjohann, K. Clays, S. Jakobs, L. Van Meervelt, and P. Dedecker, “Expression-enhanced fluorescent proteins based on enhanced green fluorescent protein for super-resolution microscopy,” ACS Nano 9(10), 9528–9541 (2015). [CrossRef]
25. S. Geissbuehler, A. Sharipov, A. Godinat, N. L. Bocchio, P. A. Sandoz, A. Huss, N. A. Jensen, S. Jakobs, J. Enderlein, F. Gisou van der Goot, E. A. Dubikovskaya, T. Lasser, and M. Leutenegger, “Live-cell multiplane three-dimensional super-resolution optical fluctuation imaging,” Nat. Commun. 5(1), 5830 (2014). [CrossRef]