Algorithmic corrections for localization microscopy with sCMOS cameras - characterisation of a computationally efficient localization approach

Ruisheng Lin; Alexander H. Clowsley; Isuru D. Jayasinghe; David Baddeley; Christian Soeller

doi:10.1364/OE.25.011701

1. Introduction

Super-resolution (SR) microscopy has advanced biomedical research by revealing molecular detail at the nanometer scale. Among the SR techniques, single-molecule localization microscopy (SMLM), known under acronyms such as PALM [1], fPALM [2] and STORM [3] or dSTORM [4], breaks the diffraction limit by measuring the position of a large number of marker molecules and is capable of routinely achieving lateral and axial resolutions ~10-fold better than that of a confocal microscope [5]. When combined with the recent DNA-PAINT technique, the lateral localization precision can be further improved to better than ~5 nm [6]. SMLM is attractive when compared with other major SR techniques, such as STED [7] and SIM [8], due to its high resolution and comparably lower instrument complexity, but suffers from considerably lower temporal resolution, limiting its application in high-throughput applications such as live-cell SR imaging.

The time to obtain a single SMLM image is determined by the time required to accumulate a sufficient number of events, as the spatial resolution is limited by the sampling of the marker densities on the structure of interest [9]. To accumulate a sufficient density of localization events tens of thousands of camera frames must typically be acquired. One way to speed up SMLM is to reduce the integration time which requires both fast photo-switching rate and high-speed image recording but is often limited by camera speed. This makes new camera technologies increasingly critical in enabling robust and fast SR imaging.

Due to their high sensitivity and low effective readout noise, electron multiplying charge-coupled devices (EMCCDs) have been the de-facto choice for SMLM imaging. EMCCDs are hampered by comparatively slow readout and small sensor areas and are slowly being supplanted by sCMOS cameras which now offer much faster readout, better quantum efficiency, and lower cost. They also have a larger sensor size which is particularly useful for ratiometric multi-color SR imaging where each color channel can only use half of the sensor area [10]. The use of sCMOS cameras in SMLM was demonstrated in 2011 by Z. Huang et al. in [11] and Saurabh in [12], although at that time the localization precision was worse than with EMCCDs, presumably due to the lower quantum efficiency of first generation sCMOS cameras and uncorrected algorithms that were originally developed for EMCCDs.

A potential drawback of sCMOS cameras is non-uniform pixel characteristics due to process variation in the on-pixel amplifier circuitry and column based readout structures. These result in a subset of pixels having significantly higher noise or offset values, which may lead to imprecise localizations near these pixels when using standard SMLM algorithms which assume uniform pixels properties. By developing a sCMOS-specific maximum likelihood estimation (MLE) algorithm where pixel dependent offset, read noise, and gain variation are pre-determined during a camera calibration step and taken into account in the fitting process F. Huang et al. were able to mitigate these effects and obtain unbiased localization performance [13]. There remain, however, shortcomings with the existing sCMOS specific MLE implementation, including a lack of an open-source implementation and the requirement for custom GPU code to obtain acceptable performance. In addition, error-estimates are computed by only taking the PSF model and photon count into account but not how well the best-fit model agrees with the data, implying that they will not accurately reflect factors such as signal distortions by nearby molecules. A more severe shortcoming is the complexity involved in extending the fitting algorithm to new model functions, especially those without an analytical derivative. This makes implementing PSF model based 3D fitting [10] extremely difficult, and also poses a significant barrier to the extension of current MLE algorithms to other scenarios such as multi-emitter fitting.

Reducing localization error and bias is one critical aspect of improving the resolution of SMLM approaches. The resolution of SMLM is useful in revealing near-molecular detail in biological preparations [1–4, 9, 10] and if this can be achieved at high frame rates it opens the door to new types of high-throughput studies. Therefore, any measures to improve the localization of emitters directly lend themselves to advancing biological understanding.

Here we revisit the impact of sCMOS non-uniform pixel properties on localization of single fluorescent emitters (hereafter often referred to as “single emitter” or simply “emitter”). Whereas previous work [13] had focused on weak signals and on low backgrounds, our simulations and corresponding experimental results show that in many practically relevant imaging situations the impact of sCMOS pixel non-uniformities is small and at least some of the corrections can be reasonably neglected, simplifying practical sCMOS super-resolution imaging. Our study provides specific criteria for users to decide when camera individual corrections should be considered. For scenarios when correction is required to improve localization precision and avoid bias, we present a new sCMOS localization algorithm based on weighted least squares (WLS) estimation that is both fast and essentially independent of the specific fitting model. Like previous MLE based work, our algorithm fully recovers the performance that would be obtained with a spatially uniform camera with equal median properties as the sCMOS sensor. For all experimentally typical photon counts our WLS localization routine attains the precision predicted by the Cramer-Rao lower bound (CRLB). In addition, we demonstrate that the algorithm performs well for 3D localization. Extension to other fitting models, e.g. multi-emitter, is straight-forward and has been implemented in a publically available software package.

2. sCMOS pixel maps

In addition to the pixel-dependent offset and sensitivity which together constitute “fixed pattern noise” [12], sCMOS cameras have significant variability in read noise between pixels. To account for these variations within our fitting routines we must estimate ‘maps’ characterizing all three components of pixel non-uniformity. Following the approach used in [13], offset and read noise maps were measured by recording a series of full-region dark frames and calculating the series mean and variance on a pixel-by-pixel basis. In principle, the standard deviation of the dark frames represents the total temporal noise which consists of the read noise and dark current. In practice, the distinction between these noise sources is generally not required, given that with typical short integration times for SMLM, the dark current is very low (median < 0.3e^-/pixel/s according to the manufacturer for the camera model used here) and the read noise is dominant. In the camera we used, only a very small portion (< 0.01% with >150 A/D units at 50 ms integration time) of pixels have a high dark current (“hot pixels”). Below, we therefore use the term “read noise” synonymous with the total temporal noise of a pixel, unless stated otherwise.

The variation of the camera sensitivity, also known as photo response non-uniformity, that contributes to the fixed pattern noise, was measured by using a local flat-fielding method. A smooth illumination field is used as the input signal (achieved by strong defocusing of a luminescent sample) and the measured camera response, after pixel-based averaging and offset subtraction, is compared to a spatially filtered version of the response. Given the low spatial frequency input light field, high frequency components reflect the local pixel-to-pixel non-uniformities of the camera.

The procedure is implemented as follows, starting with the offset corrected mean image:

I_{r a w} = \frac{1}{N} \sum_{1}^{N} I_{i} - o

where

I_{r a w}

is the averaged and offset subtracted image,

N

is the number of frames,

I_{i}

is the

i^{t h}

frame in the series (in A/D units or ADUs), and

o

is the offset map. The raw image is then processed by a Gaussian smoothing filter with a sigma of 9 pixels:

I_{f i l t e r e d} = g a u s s i a n_f i l t e r (I_{r a w})

A local flat-fielding map f by which raw images are multiplied to correct for variations in sensitivity is calculated by ratioing the filtered and unfiltered images $f = I_{f i l t e r e d} / I_{r a w}$ .

We note that f is a local flat fielding map because photo sensitivity non-uniformities that slowly vary across the whole chip will be ignored. To localize events in small regions of interest, however, local flat-fielding is the only critical aspect. We quantify the non-uniformity of the measured flat-fielding map as the standard deviation around its mean-value of one and refer to this number as the sensitivity variation of the camera. Alternative methods, such as determination of the photon transfer curve (PTC), have been previously used to determine the response of a sCMOS camera [11]. With PTC, the measured response is the gain of each pixel in the sensor. The flat-fielding approach used here should be more applicable for the purposes of localization since it takes throughput and absorption from any optics in front of the sensor (e.g. microlenses) into account.

Maps of offset, read-noise and flat-fielding recorded with an Andor Zyla 4.2 sCMOS camera are shown in Fig. 1. The offset and read noise maps were determined from a dark series of 5k frames, i.e. with a read noise precision of better than 2% [14]. The sensitivity variation determined by local flat-fielding had a standard deviation of 0.4% around the mean of 1.0. Camera real time filters (termed “spurious noise filter” and “static blemish correction” by the manufacturer) were disabled prior to camera calibration to avoid signal distortions.

Fig. 1 Maps and associated histograms of the (a) offset, (b) temporal variance and (c) flat-fielding values (sensitivity variation) of an Andor Zyla 4.2 sCMOS camera. The camera has a 2048x2048 pixel active area with a physical pixel size of 6.5 µm. The offset and variance were measured with a frame integration time of 50 ms. The offset ranges from 92 ADUs (Analog to Digital Units) to 879 ADUs with a mean of 103 ADUs and a standard deviation of 1.76 ADUs. The variance ranges from 1 to 867 e⁻² (i.e. read noise 1 to ~30 e^-) with a mean of 4 e⁻². The flat-fielding value distribution has a mean of 1.0 and a standard deviation of ~0.4%.

Download Full Size | PDF

Repeated camera calibrations over several months showed that maps remained essentially unaltered (tested up to 10 months) implying that camera calibration is essentially a one-off procedure and camera maps can be stored for all future SMLM analysis.

3. sCMOS localization algorithm

The camera maps were determined as described and stored as 2D floating point TIFF files. To determine event localizations, the raw frame data from the camera is corrected for local pixel offsets, sensitivity variations and converted to photo-electrons:

d_{i} = g_{a v} f_{i} (d_{r a w, i} - o_{i})

where

d_{i}

is the corrected data for pixel

i

in photo-electrons (e^-),

g_{a v}

is the average gain of the camera, obtained from the manufacturer’s performance sheet or by performing a PTC calibration [15],

f_{i}

is the flat field correction for pixel

i

,

d_{r a w, i}

is the raw image data in ADUs, and

o_{i}

is the offset for pixel

i

.

We use a weighted least-squares (WLS) approach for event fitting. The noise of each pixel value $d_{i}$ is estimated as a combination of read noise and photon shot noise:

ε_{i}^{2} = ε_{R N, i}^{2} + s_{i}^{2}

where

ε_{i}^{2}

is the total noise variance of pixel

i

and

ε_{R N, i}^{2}

is the read noise variance which we obtain from the pre-recorded read noise map. Due to its Poisson nature, the shot noise variance

s_{i}^{2}

is equal to the population mean

λ

of the mean photon count of the pixel. As we do not know

λ

we estimate it based on the measured photon count

d_{i}

and use Bayes’ Theorem to obtain an improved estimate over the “naïve” estimate

d_{i}

. Given an observed sample mean

d_{i}

, the expectation value

E (λ)

of

λ

is:

E (λ) = \int_{0}^{\infty} λ \cdot p (λ | d_{i}) d λ

where the conditional probability

p (λ | d_{i})

of a mean

λ

given the measured

d_{i}

is according to Bayes’ theorem:

P (λ | d {}_{i}) = \frac{p (d_{i} | λ) p (λ)}{p (d_{i})}

and assuming uniform priors on

λ

and

d_{i}

, i.e.

p (λ | d_{i}) \approx p (d_{i} | λ)

, we get

E (λ) = \int_{0}^{\infty} λ \cdot p (λ | d_{i}) d λ \approx \int_{0}^{\infty} λ \cdot p (d_{i} | λ) d λ = \int_{0}^{\infty} \frac{λ^{d_{i} + 1} e^{- λ}}{d_{i}!} d λ

Using the gamma function

E (λ)

can be written as:

E (λ) = \frac{Γ (d_{i} + 2)}{d_{i}!} = \frac{(d_{i} + 1)!}{d_{i}!} = d_{i} + 1

This gives the following total noise variance estimate of a pixel, where we also enforce positivity of

d_{i}

, which can become negative due to noise during offset removal [Eq. (3)]:

ε_{i}^{2} = ε_{R N, i}^{2} + \max (d_{i}, 0) + 1

Note that the use of $s_{i}^{2} = E (λ) = d_{i} + 1$ avoids the singularity (excess weighting of zero valued data) that can be encountered when performing WLS under the more naïve assumption of $s_{i}^{2} = d_{i}$ , effectively stabilizing the fit. The weight of each pixel for WLS is then given by $w_{i} = ε_{i}^{- 2}$ . Finally, the Levenberg-Marquardt algorithm, or an alternative solver is used to find a parameter set that minimizes the weighted least squares difference between $d_{i}$ and model intensities $E_{\mod e l, i}$ :

χ^{2} = \sum_{i} w_{i} {(d_{i} - E_{\mod e l, i} (x_{c}, y_{c}, z_{c}, A, σ, b^{2}))}^{2}

In the simplest case the model is a 2D Gaussian with background; our implementations also include PSF-based 3D models as well as multi-emitter and ratiometric multi-color models [10]. Fit parameters include center locations, amplitude, width and background $x_{c}, y_{c}, z_{c}, A, σ, b^{2}$ with additional parameters for more complex models ( $z_{c}$ is only determined for 3D fitting). The WLS estimator is implemented using a standard Levenberg-Marquardt solver available in the Python scipy library. In contrast to model specific solvers required to implement sCMOS-MLE, the scipy Levenberg-Marquardt solver has built-in support for both analytic and finite-difference gradients allowing models such as interpolated PSFs without analytical gradient. Molecule detection uses difference of Gaussian filtering followed by thresholding at a constant signal-to-noise ratio (SNR) based on our per-pixel sCMOS noise estimates. By deliberately setting the noise variance of a pixel to an extremely high value (e.g. 10⁸ e⁻²) we can implement a blemish pixel map which eliminates the impact of “bad pixels” by giving zero weighting in the fit.

The performance of our WLS estimator was assessed by comparing to the CRLB which sets a lower bound on the variance of any unbiased estimator [16]. Simulated images (1000 frames) of a single emitter were processed by our WLS implementation to obtain errors of the fit parameters. As least-squares estimators are known to be optimal for normally distributed noise, we considered the most challenging pure-Poisson case (i.e. only photon shot noise in the noise model) with low backgrounds (0, 1, 5 and 20 e^-). A comparison of the localization error using our WLS implementation and the CRLB for the error in the x coordinate is shown in Fig. 2. With reasonable event photon counts (> 300 photons) WLS achieves the CRLB, indicating that our simplified noise model does not negatively impact localization precision. Our CPU based WLS code achieves ~1600 Localizations/s on a single CPU, including molecule detection and all I/O. This is surprisingly similar to the performance of the GPU based sCMOS-MLE code by Huang et al [13], which in our tests achieved ~3000 localizations/s (nVidia M4000) without any overheads for detection or I/O, suggesting that the Levenberg-Marquadt algorithm we use is more efficient than the custom Newton-Raphson solver used for sCMOS-MLE.

Fig. 2 Comparison of the error (measured as standard deviation of simulation errors) of our WLS algorithm and the CRLB for the error in the x coordinate as a function of event photon count at a range of background levels. CRLB = Cramer-Rao lower bound; WLS = weighted least-squares; bg = background level (e^-).

Download Full Size | PDF

The WLS algorithm, data acquisition and analysis are all implemented in our Python-based Python Microscopy Environment (PYME) software package which can be obtained at http://python-microscopy.org. For sCMOS corrected fitting, the analysis software looks for the camera maps in a default directory and uses the maps to implement the WLS as described above. If the maps are not present it falls back to a “uncorrected localization” which uses a chip-uniform offset, read-noise that match the median camera properties and no flat field correction. The use of the maps incurs no measurable slow-down of the fitting process compared to the original EMCCD targeted implementation that used scalar, i.e. uniform, values for offset and read noise. This property makes the routine use of corrections practical and straight-forward once the maps are recorded and placed in the appropriate directory. In practice, maps are generated with a script in the PYME distribution that calculates and places variance and offset maps into the default calibration directories in a one-step procedure.

4. sCMOS simulations

We developed an sCMOS simulation package, included with PYME, to investigate sCMOS localization in SMLM. The simulation uses an sCMOS noise model which combines photon shot noise and a Gaussian read noise model which is imposed on simulated images of a single emitter generated with a selectable PSF model. The WLS analysis described above is then used to estimate emitter localizations. Several simulation parameters are configurable, including event photon counts, simulated camera properties (offset, read noise, flat-fielding), as well as background and spatial sampling rate. One can choose between two algorithms, either the standard fitting algorithm (which we refer to as ‘uncorrected localization’ or ‘standard algorithm’) that assumes uniform pixel properties, or the map-based sCMOS algorithm (Section 3, termed ‘sCMOS algorithm’ or ‘corrected localization’). We focus on each pixel property in turn, i.e. read noise, sensitivity variation or offset, to explore their effects on localization. Spatial sampling was 70 nm per pixel unless otherwise stated.

4.1 Effects of read noise and the role of background

In simulations to study read-noise a pixel with configurable read noise is located close to the left of the center of a 1000 photons/event single emitter [Fig. 3(a)]. Other pixel properties in the simulation are as obtained in a region of the actual map measurements of our camera. For each read noise and background combination, 1000 camera frames were simulated.

Fig. 3 Simulation of read noise effects on emitter localization. (a) A pixel with configurable read noise is located ~2 pixels to the left of the center of a 1K photons/event single emitter. (b) With low background, a read noise of 20 e^- introduces a broadening effect using ‘uncorrected localization’. Using ‘corrected localization’, the localization precision is recovered. (c) A schematic comparison of localization with map correction (blue), the uncorrected algorithm (green) and for comparison simulated uniform sensor data localized with the standard algorithm (dashed). Centers of ellipses represent the mean positions of the localization clouds (localization bias), and the major/minor axes show the localization precision. (d) Effect of a noisy pixel on localization precision as a function of background level. (e) Map of localization precision versus read noise and background, event photon count = 1K photons. (f) Map of localization precision versus read noise and event photon count at a fixed background level of 50 photons. (g) Map of localization bias versus read noise and background, 1K photon events. (e)-(g) were calculated using the uncorrected algorithm. loc err = localization error.

Download Full Size | PDF

In Fig. 3(b), each dot represents a single localization and dots from all frames form a localization cloud. When performing ‘uncorrected localizations’, a clear broadening of the localization cloud is observed in the x direction, indicating a decrease of the localization precision caused by the presence of the high-read noise pixel. When using the sCMOS algorithm with matching camera maps, i.e. ‘corrected localizations’, the effect is eliminated and the localization precision is recovered, as schematically summarized in Fig. 3(c). In SMLM significant background often arises from out-of-focus fluorescence or sample auto-fluorescence and is known to decrease the localization precision [17]. In situations when the background levels are high read noise should ultimately become small compared to the photon noise and local pixel properties would have less of an effect. To characterize the non-uniform broadening of localization precision, we plot the ratio of the localization precisions in x and y directions. When read noise dominates, the localization cloud has an elliptic shape and the ratio is larger than one [Fig. 3(d)]. With increasing background this ratio approaches unity. In other words, with elevated background, the effect of read noise distortions is effectively overcome and applying algorithmic corrections becomes largely unnecessary.

We typically perform our dSTORM imaging in larger cells such as myocytes where backgrounds often exceed 200 e^- or in 20-30 µm thick tissue slices which can have backgrounds > 300 e^-. As shown in Fig. 3(e), the correction for read noise becomes largely unnecessary at these background values, even in the presence of pixels with read-noise approaching 30 e^-, which is larger than the worst read noise in our camera [see Fig. 1(b)]. Towards the edge of thin adherent cells, it is possible to reduce background to < 30 e^- and correction for pixel non-uniformities becomes advantageous.

To investigate how the impact of high read noise pixels depends on event brightness we set the background to a low level (30 e^-) and varied the photon count from 1000 e^- to 8000 e^-, as shown in Fig. 3(f). Severe distortions are observed with high read noise and less bright events (photon count < 2000 e^-), as expected from Fig. 3(e). When the signal is sufficiently high (~6k e^- for more common read noise amplitudes ≦20 e^-) effects of read noise become negligible making algorithmic corrections unnecessary.

In addition to the localization broadening effect, high read noise pixels may also introduce bias into the localization. Similar to the findings on localization precision, the localization bias only becomes significant for uncorrected localization when the background is sufficiently low. The actual background values where bias becomes essentially negligible depend on the level of read-noise present [Fig. 3(g)].

4.2 Flat fielding and sensitivity variation

Figure 4 shows simulations of single-emitter localization with several levels of camera sensitivity variation, i.e. 0% (camera image is flat, no sensitivity variation), 0.4%, 2%, 5% and 10%, quantified as the standard deviation of the sensitivity distribution. Different amplitudes were generated by scaling the amplitude of the measured flat-fielding map that we determined for our camera [Fig. 1(c)]. For each amplitude, 1000 camera frames were simulated with different photon counts (2000 to 10000 e^-) and background levels (0, 20, 50, 100 and 200 e^-). In Fig. 4(a), a clear vertical stripe pattern was observed when the simulated sensitivity variation is ≥5%. In practice, we do not see such a pattern in raw camera images consistent with the small amplitude of the sensitivity variation that we measured, 0.4%.

Fig. 4 Simulation of the effect of uncorrected sensitivity variation on the localization of a single emitter. (a) Simulated frames for different values of sensitivity variation obtained by scaling of our measured flatfield map. The emitter has a photon count of 2000 with a background of 200 e^-. A clear vertical stripe pattern is observed when the simulated sensitivity variation is ≥5%. (b) Localization results with different sensitivity variation levels. The centers of ellipses represent the mean positions of the localization clouds, i.e. localization bias, and the major/minor axes represent the localization precisions. (c) 3D plot of the ratio of bias and precision for different sensitivity variation levels, event photon counts and background levels. Abbreviation: loc err = localization error; sv = sensitivity variation.

Download Full Size | PDF

The localization results with different values of the sensitivity variation in Fig. 4(a) are shown in Figs. 4(b) and 4(c). In Fig. 4(b), simulating events with 2000 photon counts/event and 200 e^- background, the grey cloud is the localization distribution using a uniform sensor. The major effect of a high sensitivity variation is a bias which leads to an incorrect estimate of emitter position. Compared with the flat case (blue), a variation of 0.4% (green) introduces a bias of only 1.7 nm, well below the 11.5 nm localization precision of the uniform sensor.

A comparison for different photon counts and background is shown in a 3D plot in Fig. 4(c) which measures the bias in x direction normalized by localization precision. The plot reflects several underlying tendencies: (1) with a given photon count and background, higher sensitivity variation introduces more bias but has little impact on localization precision, and thus increases the ratio. (2) For a given background level and sensitivity variation >2% increasing the event photon count considerably reduces the localization precision as expected, resulting in a relative increase of the importance of bias. For small sensitivity variation (0 and 0.4%), the normalized bias remains small (<0.22), indicating that the effects of sensitivity variation of this camera type can be ignored for most applications. (3) For a given photon count, higher background increases both the bias and the localization error, but bias dominates and the ratio is increased.

4.3 Offset effects

The offset of sCMOS cameras is usually preset to a positive value to avoid clipping of pixel values at zero. As shown in Fig. 1(a), the camera we used has offsets of ~100 ADUs but there are some pixels whose offsets are considerably larger than the pre-set value (>150 ADUs when using 50 ms integration time). These “hot” pixels are generally caused by leakage currents (or “dark currents”) in the electronics of the camera sensor and offset values increase with integration time. Hot pixels in our camera exhibit a noise variance approximately equal to the dark signal, quantified as the excess offset (i.e. the pixel offset minus the mean offset) converted to photon-electrons [Fig. 5(a)]. This was replicated in the simulations.

Fig. 5 Simulation of hot pixel effects on emitter localization. (a) Plot of temporal variance vs dark signal of pixels with offset >150 ADUs. The dashed line is a line of equal variance to dark signal. (b) A simulated frame of a emitter (1k photons) with a background of 100 e^- and a hot pixel of 400 ADUs. (c) A bias of ~18 nm is introduced into the localization of the emitter in (b). (d) The bias in (c) is eliminated by using either blemish correction (blue) or standard map correction (cyan), both achieve localization equivalent to a uniform sensor (dashed red). The crosses represent the mean localization of the emitter and the axes represent localization precision. (e) Bias from hot pixel presence and backgrounds, event photon count is 1000 photons. (f) Bias from hot pixel presence and event photon counts. Background = 200 e^-.

Download Full Size | PDF

A configurable offset pixel was added ~2 pixels away from the center of an emitter [Fig. 5(b), 1000 photon counts, background of 100 e^-]. An offset value of 400 ADUs introduced a bias of ~18 nm, compared to the true emitter position [Fig. 5(c)]. The hot pixel slightly affects the shape of the localization cloud with relatively little change to the localization precision.

The bias can be eliminated by applying a correction [Fig. 5(d)], either by using the measured maps, or, by setting the hot pixel in the variance map to a very high value (“blemish correction”). The latter method effectively discards data from this pixel as its weight is close to zero. Both methods remove the bias and restore localization precision [Fig. 5(d)].

The localization bias of a 1000 photon event due to a hot pixel depends weakly on background level [Fig. 5(e)]. Not unexpectedly, increasing the event photon count considerably de-emphasizes the effect of a hot pixel and reduces the bias [Fig. 5(f)].

4.4 Spatial sampling rate

In SMLM, the pixel size should be generally chosen to comply with the Nyquist–Shannon sampling theorem, i.e. high enough to avoid undersampling and provide sufficient information about the emitter profile. On the other hand, excessive oversampling significantly reduces the number of photons per pixel, which will emphasize the effect of read noise, as previously demonstrated [16]. In addition, oversampling also reduces the effective field size.

We simulated the effect of pixel size by sampling an emitter (1000 photon counts, background 50 e^-/10⁴nm²) at pixel sizes from 25 nm to 250 nm using a high NA scalar PSF model (NA of 1.47 at 700 nm) with a FWHM of 230 nm. A comparison of localization results is shown in Fig. 6 with a uniform sensor model as a reference. With the corrected algorithm the sCMOS data (with a high read noise pixel of 22 e^- ~2 pixels from the center) can be localized essentially as well as when using a uniform sensor, with precision deteriorating for very small and very large pixel sizes as expected. By contrast, uncorrected localizations exhibit an increased sensitivity to oversampling. This is compatible with the idea that increasing the photon counts per pixel by using coarser sampling helps mitigate the effect of isolated high read noise pixels in sCMOS cameras. This observation can be used to choose sampling that maximizes the range of conditions compatible with simpler uncorrected localization approaches. Nevertheless, overly coarse sampling is penalized just as observed with uniform sensors. In addition, coarse sampling may affect the ability to determine axial emitter position from lateral PSF shape. With these criteria a pixel size in the range of ~100-120 nm is the suggested choice for the conditions simulated here.

Fig. 6 Simulation of localization with different spatial sampling rates. (a) Localization precision versus pixel sampling for 1000 photon emitter events and 50 e^- background/10⁴nm². (b) Corresponding localization bias versus pixel sampling. Uniform sensor data is shown as reference versus an sCMOS model with a high read noise pixel (22 e^-) ~2 pixels from the center, using the WLS algorithm with and without map correction. The PSF model had a FWHM diameter of 230 nm. scmos uncorr/corr = scmos localization without/with correction.

Download Full Size | PDF

4.5 When is sCMOS correction required?

The measured sensitivity variation of ~0.4% (see Section 4.2) causes a bias of < ~20% of localization precision (< 10% for more common parameter sets), even with large event counts and background. A level of 0.4% sensitivity variation incurs effectively no loss of localization precision. As a result, flat-fielding can generally be neglected although with high brightness events (e.g. DNA-PAINT) one may have to consider how much bias can be tolerated.

Variations in read noise are more important, but can be neglected if background or event photon counts are high. Even around pixels with the highest read noise, once background exceeds 75 photons/pixel, or alternatively events have > ~6k photons, localization precision is essentially unaffected and bias is small compared to the localization precision. However, if the data is analyzed in ways that strongly depend on bias even small bias should be avoided.

High dark current pixels are more problematic. We therefore suggest to always add a preprocessing stage to the analysis pipeline where the offset map is subtracted from the raw data. This can be added to any localization implementation and is a low complexity operation.

High read-noise and offset pixels are sparse across the chip so that the regions of interest in which fits of event candidates are performed (in our implementation typically 9x9 pixels) contain generally at most one ‘bad’ pixel, i.e. matching the cases simulated above.

In summary, when imaging bright events (e.g. DNA-PAINT) or when background photon levels are relatively high, e.g. when performing dSTORM of more complex biological samples with 3D structure (e.g. large cell types such as muscle or tissue sections), correction may often be neglected. In addition, to maximize the range of conditions requiring no correction, sampling should be chosen around ~120 nm as shown in section 4.4 – assuming a system with a high NA objective and a comparable PSF size (~230 nm). In our own work the conditions described above cover a sizable fraction of applications. Nevertheless, there are a broad range of conditions where correction is highly advisable, e.g. imaging with low event counts, or generally when imaging in thin samples with low background. Since our WLS implementation recovers performance of a uniform sensor at negligible computational cost (as compared to uncorrected WLS) we recommend to adopt the safe option to always use offset and read noise map corrections.

4.6 3D localization

Our sCMOS algorithm can be directly applied to 3D localization by substituting a suitable 3D model without the need to define an analytical gradient function. A simulation of astigmatism based 3D localization [18] using an interpolated PSF model with an sCMOS camera is shown in Fig. 7. The PSF underlying the simulation is shown in Fig. 7(a). Figures 7(b)-7(e) depict the localization results (5000 frames) of a 1000-photon emitter using the standard and sCMOS algorithms, respectively. With the standard algorithm, a clear spread of data points is observed, especially in the region far from the focal plane. Localization is improved using the sCMOS algorithm, with an improvement of axial localization precision from 60.0 nm to 27.7 nm and axial bias from 17.8 nm to 0.3 nm. Due to the relatively low event photon count, read noise becomes evident [simulated frames are shown in Fig. 7(f)] and considerably affects the localization quality when using the standard algorithm. With the correction sCMOS data is localized in 3D as well as it would be with an equivalent uniform sensor [Fig. 7(g)].

Fig. 7 Simulation of 3D localization in sCMOS data. (a) PSF at z-steps of 50nm, photon event count = 1000, background = 10 e^-. (b) and (c) Estimation of the axial position and error without algorithmic correction. The z-position of the emitter is varied over a range of 800 nm around the focal plane. (d) and (e) Estimation of the axial position and error with map based correction. An improvement of axial localization precision from 60.0nm to 27.7nm and bias from 17.8nm to 0.3nm is achieved. (f) Simulated frames at different emitter positions. (g) A schematic comparison of the sCMOS and standard algorithm. Blue: sCMOS algorithm; green: standard algorithm; dashed: localizations from an equivalent uniform sensor.

Download Full Size | PDF

5. Experimental results

5.1 Microscope setup

Data were obtained using a custom-built super-resolution microscope system constructed around a Nikon Eclipse Ti-E body with 60X oil-immersion 1.49 NA objective (Nikon Apo TIRF 60X Oil). Beams from several laser modules (405, 561, 642, 671 nm) are coupled and attenuated by a neutral density filter wheel (Thorlabs FW102CNEB). Widefield illumination is provided by a LED light source (CoolLED pE-4000). Fluorescence images are recorded using either an EMCCD (Andor Ixon Ultra DV97) or a sCMOS camera (Andor Zyla 4.2 10tap) mounted on opposite side-ports of the microscope. The EMCCD is used in frame transfer mode, the Zyla 4.2 sCMOS offers only a rolling shutter mode. This is compatible with high precision SMLM as the temporal shifts between adjacent lines covering individual fluorescence events are negligible. During data acquisition, the sample is simultaneously illuminated through the condenser and transmitted light images are recorded by a tracking camera (Thorlabs DCC3240N), similar to the method described in [19]. The tracking images are used to implement a focus lock as well as 2D drift tracking.

5.2 Validation with sub-resolution beads

Figure 8 shows an experimental validation of the simulation findings using localizations of 200 nm dark-red fluorescent beads. Illumination is provided by the LED light source at 635 nm. The intensity was adjusted to obtain ~1800 photons per bead in a single frame, broadly similar to the photon yield of bright single fluorochrome molecules. In Fig. 8(a), the temporal variance image of a 3000-frame image series reveals the presence of a noisy pixel (~15 e^-) close to a bead (see inset) where we exploit that the beads themselves are revealed due to the Poisson photon noise variance. When using the bead sample [Fig. 8(b)] the background was as low as ~5 e^-. The localization results from both the standard, uncorrected algorithm versus the sCMOS algorithm are shown in Figs. 8(c)-8(h). A broadening is introduced by the noisy pixel when no correction is applied, with a bias of ~2.5 nm in the x direction in Fig. 8(e). With correction the localization precision is restored from 10.8 nm to 8.1 nm, i.e. an improvement of ~25%, and the bias is eliminated.

Fig. 8 Application of the sCMOS algorithm and effect of read noise and background. (a) Temporal variance image of a 3K-frame image series reveals a noisy pixel (~15 e^-) close to a bead (brightness ~1800 photons). (b) A single frame of the image series with low background (5 e^-). (c), (e) and (g) Localizations of the bead in (b) using the standard algorithm. (d), (f) and (h) Localizations of the bead in (b) using the sCMOS algorithm. (i) A single frame of the image series with high background (90 e^-). (j), (l) and (n) Localizations of the bead in (i) using the standard algorithm. (k), (m) and (o) Localizations of the bead in (i) using the sCMOS algorithm.

Download Full Size | PDF

By adjusting the condenser illumination the background level was increased to ~90 e^- per pixel [Fig. 8(i)] which should mask the effect of read noise. The localization results using standard and sCMOS algorithms are shown in Figs. 8(j)-8(o). A residual bias of <1 nm is observed, consistent with the data in Fig. 3(e). The localization precisions in x and y directions are degraded by the background (17.6 nm vs 17.2 nm) and no non-uniform broadening is apparent, suggesting, as expected, that the uncorrected localization algorithm is adequate.

5.3 sCMOS localization tests using a DNA-PAINT nanoruler sample

We further tested sCMOS localization by using a DNA origami nanoruler sample (GATTA-PAINT HiRes 40R). The sample utilizes the recent DNA-PAINT technique [6] where stochastic switching is implemented by reversible binding between fluorescently labelled imager strands diffusing in solution and immobilized docking sites positioned in a linear arrangement with 40 nm distance between 3 adjacent sites. It also contains fiducial markers (with a spherical appearance) intended for drift correction. This sample has the advantage that a known structure can be imaged in a “steady-state” where event rates are stationary for thousands of frames. Imager strands are labelled with ATTO 655 dye excited by a 642 nm laser. A filter cube containing a Cy5 dichroic (Semrock FF660-Di02) and a Cy5 emission filter (Semrock FF01-692/40) were used to match the spectrum. The laser power and integration times were adjusted to obtain a photon count of ~2000 e^- per event and frame, background intensity was ~30-50 e^- in TIRF mode.

The resulting super-resolution images were rendered by a jittered triangulation method as described in [20]. As shown in Fig. 9, the 40 nm distant spots were well resolved. For comparison, we applied both uncorrected standard and corrected sCMOS localization algorithms to the data and did not observe any evident difference in the reconstructed super-resolution image [Figs. 9(a) and 9(b)]. In the sample region shown (2.3 × 1.6 µm²), corresponding to 32 × 23 pixels, the maximal sCMOS read noise was 9.9 e^-. We expect that with a background of >30 e^- the effect of read noise is negligible (section 4). The maximal offset in this region was 109 ADUs which should not introduce detectable bias, consistent with the findings.

Fig. 9 Imaging of a nanoruler sample with corrected and uncorrected localization algorithms. (a) Image reconstructed using the sCMOS corrected WLS algorithm, and (b) using the same raw frame data but processed with the uncorrected algorithm. The area in the green box is compared in detail in (c), (d) and also related to an image of the same structure obtained with a more uniform sensor, an Andor Ixon EMCCD (e). (f) and (g). Comparison of intensity profiles through the structure as shown in (c) and (e). (h). Corresponding localization errors estimated from the co-variance of the WLS penalty function at convergence. (i) Distribution of localizations along the direction orthogonal to the origami axis in (c) and (e).

Download Full Size | PDF

Due to the stationary nature of the sample it was possible to record data from the same sample area using a camera with a more uniform sensor, an Andor Ixon Ultra DV97, similar to a method suggested in a recent technical note [21]. This enabled comparison of the structures recorded with the more uniform EMCCD sensor versus the sCMOS sensor. With both cameras, more than 120K raw images of the same region were recorded to ensure sufficient sampling of the nanoruler structures. When structures were investigated in detail [Figs. 9(c)-9(g)] there was little difference between the images. This establishes experimentally that sCMOS cameras, under suitable conditions, can match the localization performance of a uniform sensor even with uncorrected algorithms.

For each lobe, 1-D Gaussian fitting was employed to estimate the peak widths and the sCMOS data showed a slight advantage [see profiles in Figs. 9(f) and 9(g)]. A statistical comparison of localization error in normalized histograms [Fig. 9(h)] shows an improved overall precision (~9 nm vs ~10 nm). The distribution of the localizations along the orthogonal direction to the origami axis in Figs. 9(c) and 9(e), as a direct measure of localization errors, shows a ~15% improvement (σ of 9.4 nm vs 11.0 nm from Gaussian fits) using the sCMOS camera [Fig. 9(i)]. While the quantum efficiency (QE) of the EMCCD camera is higher than that of our sCMOS camera (90% vs 60%), the additional noise introduced in EMCCDs during electron amplification reduces the signal to noise ratio equivalent to halving the quantum efficiency [22], predicting a ratio of localization precisions as the square root of the effective QEs, i.e. a ~13% advantage for the sCMOS, similar to the observations. In other words an sCMOS camera can perform as well (if not better) in terms of photon budget as an EMCCD camera for SMLM which is likely going to be enhanced by ongoing CMOS sensor developments.

6. Conclusion

We introduce an efficient algorithm applicable to a wide range of SMLM modalities to compensate for sCMOS pixel non-uniformities. The algorithm is integrated into a public software suite for routine use and incurs little speed penalty over the ‘uncorrected’ algorithm that was originally developed for uniform sensors. With these approaches, sCMOS cameras essentially perform as well for SMLM as cameras with more uniform sensor properties (e.g. EMCCDs). They provide large chip sizes and high frame rates while matching or exceeding effective quantum efficiencies of competing technologies. In addition, we show that for some practically relevant conditions, the pixel non-uniformities of current sCMOS cameras have little impact on localizations performed with algorithms assuming uniform pixel properties.

Funding

Human Frontier Science Program (RGP0027/2013) and EPSRC (EP/N008235/1).

Acknowledgments

We would like to thank Joerg Bewersdorf & Fang Huang for providing us with a copy of their sCMOS-MLE GPU code.

References and links

1. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef] [PubMed]

2. S. T. Hess, T. P. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophys. J. 91(11), 4258–4272 (2006). [CrossRef] [PubMed]

3. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–795 (2006). [CrossRef] [PubMed]

4. S. van de Linde, A. Löschberger, T. Klein, M. Heidbreder, S. Wolter, M. Heilemann, and M. Sauer, “Direct stochastic optical reconstruction microscopy with standard fluorescent probes,” Nat. Protoc. 6(7), 991–1009 (2011). [CrossRef] [PubMed]

5. T. J. Gould, S. T. Hess, and J. Bewersdorf, “Optical nanoscopy: from acquisition to analysis,” Annu. Rev. Biomed. Eng. 14(1), 231–254 (2012). [CrossRef] [PubMed]

6. M. Dai, R. Jungmann, and P. Yin, “Optical imaging of individual biomolecules in densely packed clusters,” Nat. Nanotechnol. 11(9), 798–807 (2016). [CrossRef] [PubMed]

7. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef] [PubMed]

8. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef] [PubMed]

9. H. Shroff, C. G. Galbraith, J. A. Galbraith, and E. Betzig, “Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics,” Nat. Methods 5(5), 417–423 (2008). [CrossRef] [PubMed]

10. D. Baddeley, D. Crossman, S. Rossberger, J. E. Cheyne, J. M. Montgomery, I. D. Jayasinghe, C. Cremer, M. B. Cannell, and C. Soeller, “4D super-resolution microscopy with conventional fluorophores and single wavelength excitation in optically thick cells and tissues,” PLoS One 6(5), e20645 (2011). [CrossRef] [PubMed]

11. Z. L. Huang, H. Zhu, F. Long, H. Ma, L. Qin, Y. Liu, J. Ding, Z. Zhang, Q. Luo, and S. Zeng, “Localization-based super-resolution microscopy with an sCMOS camera,” Opt. Express 19(20), 19156–19168 (2011). [CrossRef] [PubMed]

12. S. Saurabh, S. Maji, and M. P. Bruchez, “Evaluation of sCMOS cameras for detection and localization of single Cy5 molecules,” Opt. Express 20(7), 7338–7349 (2012). [CrossRef] [PubMed]

13. F. Huang, T. M. Hartwich, F. E. Rivera-Molina, Y. Lin, W. C. Duim, J. J. Long, P. D. Uchil, J. R. Myers, M. A. Baird, W. Mothes, M. W. Davidson, D. Toomre, and J. Bewersdorf, “Video-rate nanoscopy using sCMOS camera-specific single-molecule localization algorithms,” Nat. Methods 10(7), 653–658 (2013). [CrossRef] [PubMed]

14. S. Ahn and J. A. Fessler, “Standard errors of mean, variance, and standard deviation estimators,” (2003).

15. L. Li, M. Li, Z. Zhang, and Z.-L. Huang, “Assessing low-light cameras with photon transfer curve method,” J. Innov. Opt. Health Sci. 9(3), 1630008 (2016). [CrossRef]

16. R. J. Ober, S. Ram, and E. S. Ward, “Localization accuracy in single-molecule microscopy,” Biophys. J. 86(2), 1185–1200 (2004). [CrossRef] [PubMed]

17. H. Deschout, F. C. Zanacchi, M. Mlodzianoski, A. Diaspro, J. Bewersdorf, S. T. Hess, and K. Braeckmans, “Precisely and accurately localizing single emitters in fluorescence microscopy,” Nat. Methods 11(3), 253–266 (2014). [CrossRef] [PubMed]

18. B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science 319(5864), 810–813 (2008). [CrossRef] [PubMed]

19. R. McGorty, D. Kamiyama, and B. Huang, “Active microscope stabilization in three dimensions using image correlation,” Opt. Nanoscopy 2(1), 3 (2013). [CrossRef] [PubMed]

20. D. Baddeley, M. B. Cannell, and C. Soeller, “Visualization of localization microscopy data,” Microsc. Microanal. 16(1), 64–72 (2010). [CrossRef] [PubMed]

21. Hamamatsu, “Using Super-Resolution Nanorulers to study the Capabilities of EM-CCD and sCMOS Cameras beyond the Diffraction Limit,” (Hamamatsu Application Note).

22. K. I. Mortensen, L. S. Churchman, J. A. Spudich, and H. Flyvbjerg, “Optimized localization analysis for single-molecule tracking and super-resolution microscopy,” Nat. Methods 7(5), 377–381 (2010). [CrossRef] [PubMed]

Algorithmic corrections for localization microscopy with sCMOS cameras - characterisation of a computationally efficient localization approach

Abstract

1. Introduction

2. sCMOS pixel maps

3. sCMOS localization algorithm

4. sCMOS simulations

4.1 Effects of read noise and the role of background

4.2 Flat fielding and sensitivity variation

4.3 Offset effects

4.4 Spatial sampling rate

4.5 When is sCMOS correction required?

4.6 3D localization

5. Experimental results

5.1 Microscope setup

5.2 Validation with sub-resolution beads

5.3 sCMOS localization tests using a DNA-PAINT nanoruler sample

6. Conclusion

Funding

Acknowledgments

References and links

Cited By

Figures (9)

Equations (10)

Optics Express