uperresolution Optical Fluctuation Imaging (SOFI) as initially demonstrated allows for a resolution enhancement in imaging by a factor of square-root of two. Here, we demonstrate how to increase the resolution of SOFI images by re-weighting the Optical Transfer Function (OTF). Furthermore, we demonstrate how cross-cumulants can be exploited to obtain a fair approximation of the underlying Point-Spread Function. We show a two-fold increase of resolution (over the diffraction limit) of near-infrared quantum dot labeled tubulin-network of 3T3 fibroblasts
©2010 Optical Society of America
Far-field optical super-resolution microscopy overcomes the diffraction limit as formulated by Ernst Abbe over a century ago . Since its first inception  and demonstration, the area of (far-field) super-resolution microscopy has experienced a tremendous growth and has already been thoroughly reviewed [3, 4]. Superresolution Optical Fluctuation Imaging (SOFI)  is a recent addition to the toolset of previously introduced super-resolution methods. In comparison to previously established superresolution techniques, SOFI offers a convenient alternative. It is conceptually different to e.g. localization based methods such as PALM , STORM  and their variants [8, 9], or to STED . Even though it still relies on the ‘on’ and ‘off’ switching properties of the probe, the blinking/photoswitching requirements are greatly relaxed in comparison to STED or PALM (e.g. one could picture a SOFI embodiment that is based on stochastic reorientations of non-blinking probes analyzed by polarization optics). Unique SOFI attributes include compatibility with all microscopic platforms (no need for hardware modifications), inherent 3D super-resolved z-sectioning, and inherent background elimination.
SOFI is based on the spatio-temporal evaluation of the optical signal which stems from independently fluctuating emitters. This is done by taking a movie of the sample and subsequently calculating the temporal n th order cumulant for each pixel. Plotting the value of the n th order cumulant for almost any given set of time lags yields an image featuring a resolution enhancement of a factor n over the conventional image.
We previously demonstrated SOFI resolution enhancement of a factor of 2 in imaging applications. Here we report on an enhancement to the previously published SOFI algorithm, resulting in a linear scaling of the resolution enhancement with the cumulant order, i.e. the second-order cumulant results in a two-fold resolution enhancement. Furthermore, we show how spatio-temporal cross-cumulants can be advantageously used to find a good approximation of the Point Spread Function (PSF). The use of spatio-temporal cross-cumulants also leads to an up-sampled SOFI image with an effectively reduced pixel size. This in turn means that the systems magnification (nm / pixel) on array detectors (such as CCD-cameras) is not limiting the achievable resolution in SOFI as will be shown in simulations and experiments. In the following we will briefly summarize the underlying theory of SOFI. Then, we will develop the formalism which enables us to increase the apparent resolution of SOFI. Furthermore, we will demonstrate the advantages of spatio-temporal cumulants.
Mathematically, image formation using an optical system can be written as:Eq. (1) describes a convolution of the systems PSF U(r) with the time-dependent emitter fluctuation functions fi(t), where the index i refers to the i th emitter. The summation index is extending over all N emitters. Note that fi(t) is not accounting for an emitter's movement but only for the time-dependent fluctuations of its emission, such as fluorescence intermittency. We assume that the emitters are immobile during image (movie) acquisition. Any motion (such as drift) during SOFI acquisition would lead to a blurred image in SOFI and might even result in negative image values.
Then, fi(t) can be written as
In SOFI, the n th-order (auto-) cumulant of the detected signal fluctuations δF(r,t) = F(r,t) − 〈F(r,t)〉t is calculated for each pixel and subsequently the amplitude of the cumulant is assigned as the SOFI pixel’s value. The resulting image is called AC- resp. XC-SOFI image, depending whether it is derived from the auto- or cross-cumulant. The latter will be discussed later in this paper. For example, the second-order (auto-) cumulant AC 2 is given by:
The second line in Eq. (3) is only true if the emitter fluctuations are independent from each other, which is a key ingredient to SOFI.
For the n th-order (auto-) cumulant ACn, the following expression holds:
Where is a function composed of single-emitter correlation functions of orders up to n. The n th-order cumulant will give raise to an SOFI image which is featuring the PSF of the original imaging system taken to the n th power. Note the importance of an immobile sample. Assuming a Gaussian shaped PSF this translates into an apparent resolution enhancement of a factor n 1/2. However, using a simple reweighting scheme for the Optical Transfer function OTF and knowing that the support (supp[Ũ(k)]) of a real OTF is finite, the resolution enhancement can be further improved to a factor n as shown in the following scheme:
Here, represents the OTF (Fourier-transformed PSF) of the imaging system, is the OTF underlying the SOFI image. is the Fourier-reweighted OTF which gives raise to the n-fold resolution enhancement. is the reweighting function, and α is a damping factor to prevent division by close-to-zero numbers. This reweighting is possible because the support of and is identical (this can be deduced by using the Titchmarsh convolution theorem ). The specific shape of the reweighting function is chosen in that way that the resulting PSF has the same functional form as the PSF underlying the original intensity image. Note that even though this approach resembles a deconvolution, it is not inferring spatial frequencies which are higher than physically possible. The sole purpose of this reweighting is to transform the SOFI image in a way that the underlying SOFI PSF has the same functional form as the original PSF, but with smaller width. However if the Fourier reweighting scheme from Eq. (5) is applied to conventional laser scanning confocal microscopy it will not yield any resolution enhancement information (assuming imaging with a perfect PSF; ‘perfect’ refers to a PSF whose OTF carries the maximum achievable spatial frequencies and the PSF is described by a diffraction limited airy disk. In this case the full scope of resolution is already displayed).
In essence, Eq. (5) states that the resolution of SOFI images (or any images which are based on an n-fold product of the PSF with itself, as occurring for example in 2-photon excitation laser scanning microscopy or tri-exciton imaging ) could be increased up to a factor of n-times. Taking into account that novel deconvolution algorithms for confocal and 2P microscopy are still being developed we expect that such advances could be applied beneficially to SOFI images as well due to the mathematical similarity of the problem (For a general review of deconvolution in microsopy we suggest the excellent review of Sibarita ).Using the Fourier reweighting scheme, the SOFI image can be transformed into a high-resolution SOFI image :
We first evaluated the Fourier-reweighting scheme on simulated data set. is calculated once using (i) the exact theoretical PSF (given by a Bessel function of the first kind divided by r = |r|) and (ii) an approximated Gaussian-shaped PSF which we estimated using cross-cumulants (as discussed later).Fig. 1 shows the results for various orders of AC-SOFI images. As can be seen, the images show better resolution when Fourier-reweighting is applied. The analysis of the relative FWHM of the PSF shows that the theoretically expected and actually achieved values are in excellent agreement. The deviation from the theoretical values for increasing order is due to the diminished amplitudes of higher spatial frequencies of the OTF O(k). In order to recover the full resolution at higher-orders, one would need to reduce the damping factor α, risking an unfavorable increase of the noise level and “ringing” artifacts. The quality of the Fourier-reweighting depends only weakly on the type of PSF (exact vs. Gaussian) which is used for calculating the weight function . Even though the Gaussian PSF approximation (as shown in Fig. 1) yields slightly lower resolution values, the resulting images exhibit slightly less ringing as compared to the exact PSF (data not shown). It should be noted that the shown simulations do not contain any kind of noise or background.
When applying the same strategy to noise-afflicted data sets, it turns out that the Fourier-reweighing is fairly robust (see Fig. 2 ). Obviously the Fourier-reweighting scheme breaks down first for the higher-order SOFI images if noise becomes too strong ord the signal to background ratio is too low. The second-order SOFI images however can be recovered conveniently with full resolution even at relatively low signal to noise levels and relatively short simulation times (1000 frames).
The Fourier reweighting scheme described above therefore allows one to obtain SOFI superresolution that scales linearly with the cumulant order n (rather than square-root scaling).
Cross-Cumulants: PSF estimation and higher image sampling-frequency
When an image sequence (movie) is acquired by a CCD-camera (or any other array detector), circumjacent pixels carry information which can be exploited for SOFI using a temporal cross-cumulant approach. For the sake of simplicity, let us first consider a second-order SOFI image. By temporally cross-correlating two pixels and assuming a Gaussian-shaped PSF the following expression can be found [analogous to Eq. (3)]:Eq. (7).
- 1. The location of the resulting “cross-correlation pixel” lies in the geometric center of and :
- 2. This cross-correlation pixel is weighted by a PSF-shaped weight factor, which depends on the distance of and : , and therefore in the following is referred to as distance factor.
The distance factor can be understood in the following sense. The signal contribution of an emitter to adjacent pixels at and persists with a characteristic length of the PSF’s FWHM (as magnified onto the camera). The distance factor states that the magnitude of the cross-correlation is decaying along this length, i.e. the further the cross-correlated pixels are apart from each other the smaller the distance factor. Thus, for the cross-correlation to yield significant values, a slight oversampling of the PSF has to be maintained. Note that the special case of an auto-correlation can be recovered by setting .
Applying the cross-correlation approach to directly neighboring pixels, virtual “cross-correlation” pixels which interpolate between neighboring pixels on the CCD can be generated (see (1.)). Thus, a fourfold higher density of pixels (2 × 2) compared to the pure auto-correlation approach can be achieved by cross-correlating vertically, horizontally and diagonally neighboring pixels of the movie sequence (see Fig. 3 ). These cross-correlation pixels carry true information (in contrast to pixels generated by a simple interpolation of the auto-correlation (AC-) SOFI image). Note that not only directly neighboring pixels can be used to generate interleaving pixels, but also next-to-neighbor pixels and so on, as long as the size of the PSF is not exceeded. Note further, that it is even possible to generate an up-sampled (XC-) SOFI image while completely omitting auto-correlations (e.g. by cross-correlating pixels which are one pixel apart from each other. See Fig. 3). The latter can be useful if one wants to generate SOFI images for the zero-time lag, which would be affected by shot-noise and camera read-out noise if one uses an autocorrelation approach.
One feature of the pixels which are generated by cross-correlation is that they exhibit an intensity which is weighted by the above mentioned distance factor. This leads to systematic intensity differences in the XC-SOFI image (see Fig. 4 ). In order to compensate these intensity differences one has to counter-balance the distance factor for the cross-correlation pixels by multiplying these pixels with the inverse of the distance factor. In the following we propose a way to estimate the distance factor using the already acquired data set.In an XC-SOFI image we know that certain pixels carry the same distance factor. For example, every second pixel along x will be a cross-correlation pixel which has been generated by using e.g. the left and right neighboring pixels, whose distances are always the same throughout the image. The same applies to every second pixel along y, along the diagonal and to the autocorrelation pixels (where the distance is zero and thus the distance factor is one). Thus, we know that there are only four different values for the distance factor within the second-order XC-SOFI image and these values have to be applied to particular pixels only. By varying the intensity of these particular pixels with the same inverse distance factor we look for the XC-SOFI image which appears most “smooth”. This is done by minimizing the relative variance of the resulting image by varying a parameterized description of the Gaussian PSF [i.e. the distance factor, see (2.)] (see Fig. 4). Since this fit involves only three fit parameters (width ωx, ωy and orientation ϕ of the 2D Gauss) which are determined from a pool of Mx × My pixels (where M is the number of pixels along each axis of the up-sampled image) it can be considered very restrictive and accurate.
The described approach holds true also for higher-order XC-SOFI images (Fig. 3). The general formula is then given by:
For each cumulant order more pixels can be generated. The order of the cumulant determines how many pixels can be correlated. The n th-order will theoretically yield an n × n-fold up-sampled XC-SOFI image, i.e. an n-fold higher magnification per pixel.
We demonstrate this on a simulation up to the fourth-order cross-cumulant. The pixels of a simulated data set (movie) were pooled in bigger pixels (1×1, 2×2, 3×3, and 4×4 pixels). The n th-order XC-SOFI image was created from the n × n-times binned movie using the cross-cumulant approach as described above (i.e. the second-order XC-SOFI image was derived from the 2 × 2 binned data set, which yields 2 × 2 times more pixels). The resulting XC-SOFI image was than compared to the AC-SOFI image calculated from the original (not-binned) data set (1 × 1). Both SOFI images have therefore the same number of pixels, feature the same magnification (nm / pixel) and are almost identical. Differences arise from the Gaussian approximation of the PSF.
As can be seen in Fig. 5 , the differences between the normalized AC-SOFI images and the XC-images are not exceeding 0.4% for the second-order. Similar results were obtained for the third- and fourth-order SOFI images, though the mismatch was slightly larger (1.4% resp. 9%). The recovered FWHM of the approximated PSFs deviated less than 2% from the original PSFs’ FWHM for all cumulant-orders.
The decreased accuracy of the reconstruction can be assigned to the decreased oversampling of the PSF, but mainly to the order of the cumulant to be reconstructed. The latter refers to the fact that the statistics for higher-order cumulants have to stem from an increasingly longer data set in order to fully recover a SOFI image. When binning pixels, information gets pooled and therefore the data set has to provide longer time traces in order to recover this information. This can be seen from Fig. 5 where the fourth-order cumulant differences are biggest in regions of high emitter density and therefore the fluctuations of many emitters are pooled in a single pixel. This leads to an incomplete resolved fourth-order image (due to limited measurement time). Therefore the assumption for the cross-cumulant approach that the underlying SOFI image is fully resolved is violated and ill-compensated in these regions.
In the following simulation we explicitly show that the cross-correlation pixels indeed do carry information on the resolution. For this reason a different data set was analyzed, where the resolution enhancement could not be realized since the magnification / spatial oversampling was intentionally chosen to be too low (the effective pixel size of 160 nm was exceeding the obtainable resolution ~220 nm (second-order AC-SOFI)). Using the cross-correlation approach the effective pixel size can be reduced (80 nm/pixel) so it becomes sufficiently small to resolve all emitters. Figure 6 shows the result for the XC-approach versus the AC-approach. The emitter positions are marked to demonstrate that indeed the recovered pixels are carrying the higher resolution information.
To show that the above mentioned approaches (Fourier-reweighting and cross-cumulant calculation) work well for a real imaging application, we performed an experiment on fixed 3T3 fibroblasts whose tubulin network was labeled with infrared emitting quantum dots (QD800, life technologies, USA). The camera magnification was chosen to be 167 nm which on the one hand allows large field-of-view imaging, and on the other hand requires the cross-cumulant approach to realize the full resolution gain of a factor two by applying the Fourier-reweighting approach.
From the data set we estimated the shape of the PSF using the above described cross-cumulant approach, showing a slight ellipticity of 1.06 and a mean FWHM of 462 nm. This PSF was used in the subsequent Fourier-reweighting of the XC-SOFI image. The result can be seen in Fig. 7 . Cross-sections (taken as indicated by the white lines in the original image) are shown on the lower panel. It can be seen that the resolution of the Fourier-reweighted XC-SOFI image is increased as compared to the original image and also as compared to the conventional (unweighted) XC-SOFI image. Using infrared emission we could increase the resolution from ~400 nm as defined by the Rayleigh limit down to at least 220 nm.
We demonstrated a resolution enhancement of SOFI using a Fourier-reweighting scheme with a linear dependence of resolution vs. cumulant order. We also demonstrated that the use of spatio-temporal cross-cumulants holds many advantages over the auto- cumulant approach. The most obvious is the accurate description of the underlying PSF of the imaging system. The use of cross-cumulants enables the straight forward calculation of zero-time lag XC-SOFI images, eliminating shot-noise and camera read-out noise. Since most fluorescence intermittency phenomena occur on short time-scales and decay rapidly, the use of zero-time lag XC-SOFI images might prove beneficial in terms of higher signal to noise ratio.
The increased pixel number and information content afforded by XC-SOFI eliminates the need to change magnification (as long as slight oversampling of the PSF is guaranteed in the original image). No additional magnification in turns means that the fluorescent signal of a single emitter is not “diluted” on too many pixels, yielding a better signal-to-noise ratio. The magnification of commercial high NA microscopes is often designed to be in the 150-170 nm / pixel range. This magnification is sufficient to generate SOFI images with the resolution obtainable by any cumulant order while preserving the same field of view. Furthermore, relatively short acquisition times in SOFI (as compared to localization-based superresolution techniques) relax platform dependent issues such as sample drift during the measurement.
We would like to thank Dr. Gopal Iyer for helping with the preparation of cells. This work was supported by the National Institutes of Health (NIH) grant# 5R01EB000312 and NIH grant# 1R01GM086197. Thomas Dertinger is supported by the German Science Foundation (DFG, fellowship # DE 1591/1 1). Jörg Enderlein acknowledges financial support by the Human Frontier Science Program (RGP46/2006) and by the German Federal Ministry of Education and Research (FKZ 13N9236). Correspondence and requests for materials should be addressed to Thomas Dertinger. (email@example.com) or Jörg Enderlein (firstname.lastname@example.org) or Shimon Weiss (email@example.com).
References and links
1. E. Abbe, “Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung,” Archiv für Mikroskopische Anatomie 9(1), 413–418 (1873). [CrossRef]
2. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef] [PubMed]
4. R. Heintzmann and M. G. L. Gustafsson, “Subdiffraction resolution in continuous samples,” Nat. Photonics 3(7), 362–364 (2009). [CrossRef]
5. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” (2009), pp. 22287–22292.
8. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef] [PubMed]
9. M. Heilemann, S. van de Linde, M. Schüttpelz, R. Kasper, B. Seefeldt, A. Mukherjee, P. Tinnefeld, and M. Sauer, “Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes,” Angew. Chem. Int. Ed. Engl. 47(33), 6172–6176 (2008). [CrossRef] [PubMed]
10. E. C. Titchmarsh, “The Zeros of Certain Integral Functions,” (1926), pp. 283–302.
12. J. B. Sibarita, “Deconvolution microscopy,” Adv. Biochem. Eng. Biotechnol. 95, 201–243 (2005). [PubMed]