Open Access
Add to BibSonomy
Abstract
Accurate image alignment is critical in multicolor single-molecule fluorescence microscopy. Global alignment using affine transformations leaves residual errors due to the nonlinearity of the distortions, which decreases the effective field of view. Subsequent local refinement demands either large amounts of reference data and processing time or specialized imaging techniques like active stabilization. Here, we present a global alignment method, S/T polynomial decomposition, that uses sums of Zernike polynomial gradients to decompose the distortion between two images, correcting both linear and nonlinear distortions simultaneously. With minimal reference data, we gain diagnostic information about the distortion and achieve a colocalization accuracy comparable to local registration methods across the entire field of view.
© 2021 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Modern biology has advanced through an ever-deeper understanding of the workings of cellular processes. To obtain how the dynamics of such cellular processes are governed at the molecular scale, the focus of study moves to smaller (sub-)systems, including individual biological molecules. Frequently, such individual molecules are tagged with fluorescent markers, making it possible to e.g. visualise their localization and/or dynamics. Similarly, labelling different types of biological molecules in distinct colors makes it possible to monitor the interactions between them at the single-molecule level.
Two types of widefield, multicolor single-molecule fluorescence microscopy are frequently applied to examine molecular interactions. Multi-colour [1–3] Förster Resonance Energy Transfer (FRET) microscopy exploits the nonradiative dipole-dipole coupling energy transfer between two fluorescent markers to report on distances between them of 1 – 10 nm [4]. CoSMoS (Colocalization Single-Molecule Spectroscopy) and SHREC (Single-molecule High-Resolution Colocalization) rely on the coincidence of components labeled in distinct colors to deduce the composition and accompanying kinetics of macromolecular complexes [5–8]. Both approaches rely on accurate registration of images in multiple wavelengths (Fig. 1(a)). FRET measurements require that fluorescent spots in multiple colors be linked with high confidence to calculate a meaningful energy transfer efficiency. In CoSMoS, an accuracy in the 10s of nm is necessary to reliably distinguish true complexes from mere binding at proximal sites.
Fig. 1. Connection of Zernike polynomials to CoSMoS image analysis. (a) Schematic of the detection pathway in a multicolor CoSMoS microscope. Fluorescence from multiple fluorophores is split into multiple channels, which are imaged on different sections of a camera chip. In our setup, the two halves of an EMCCD chip are used, but the schematic is general. In principle, the two images are identical and can be overlaid to extract precise localization and colocalization information. However, in practice, chromatic distortions and differential optical imperfections between the color channels lead to distortions between the images that prevent precise colocalization and result in information loss. (b) The first 21 Zernike polynomials, which are scalar functions on the unit disk. Rows of the pyramid correspond to the maximum power of the radial coordinate, $r$. The inset illustrates the relationship between the Zernike polynomials and the vectorial S/T polynomials using $\mathbf {S}_{6}$. $\mathbf {S}_{6}=\frac {1}{\sqrt {12}}\nabla Z_{6}$, where $Z_{6}$ is indicated in red. Since gradients of Zernike polynomials are sums of other Zernike polynomials, $\mathbf {S}_{6}$ can also be expressed as $\mathbf {S}_{6}=\frac {1}{\sqrt {2}}(Z_{2}\hat {x}-Z_{3}\hat {y})$, where $Z_{2}$ and $Z_{3}$ are circled in blue.
FRET and CoSMoS have especially extreme colocalization requirements, but the need to accurately register images in multiple color channels is common to many more microscopy techniques. A practical challenge to all of these approaches is that widefield images in multiple color channels are typically distorted relative to one another due to slight differences in the imaging optics. This introduces differences in translation, rotation, and scale, as well as additional optical distortions. Even if multicolor imaging is performed using only a single imaging channel (e.g. through alternating excitation), the wavelengths still experience different chromatic distortions [9], and the final images will be slightly different.
The twin issues of localization and colocalization—the precision with which the locations of individual PSFs can be defined and the accuracy with which PSFs in different images can be located relative to each other, respectively—must be addressed separately. Super-resolution techniques like PALM and STORM typically localize individual fluorophores with nm precision [10–13]. Subnanometer precisions are possible, even in three dimensions [14], through careful theoretical treatment of point spread functions with high photon counts [15–17]; thorough calibration of each element of the microscope; and fitting of the remaining variations [14,18,19]. One downside of the techniques with the highest localization precisions is that reference fluorophores must be imaged simultaneously with the sample [14], which is common in cell and nanomaterial imaging, but less so in CoSMoS and FRET microscopy.
However, high localization precision does not translate automatically into accurate colocalization [19]. Chromatic and other optical distortions will lead to differences between images in multiple color channels, whether they are collected through the same optics or not.
Relative distortions between images can be corrected either globally or on a PSF-by-PSF basis. The simplest global distortions between two images, so-called "affine" distortions, are relative translation (shift), rotation, and scaling (magnification), sometimes with the inclusion of shear. The affine shift between any two images can be calculated by interpolating between a selection of reference points (for example, fluorescent calibration beads or an etched reference grid) and then applying the inverse transformation to align the two (or more) images [5,20].
The affine correction does not take into account the higher-order distortions that may be present in an image. In particular, if the image is optically distorted at its edges, affine transformation cannot improve the image quality. For this reason, higher-order corrections are often employed. For example, the Traces algorithm [21] removes residual distortions by globally fitting the difference between the affine-corrected image and the reference image with a fourth-order polynomial in the radial coordinate r.
Even higher alignment accuracy is possible if one employs active feedback controls [22] or performs PSF correction locally rather than globally [8,17,23]. Friedman et al. [23] used a local correction method in which the exact local offset between channels was calculated for each reference fluorophore in a sample of several hundred spanning the field of view. Colocalizations with a residual mapping error as low as 20 nm could then be achieved by interpolating the correction from the table of local correction values. In a related technique called "piecewise affine correction," a local affine transformation was defined by the reference fluorophores in each discrete region of the field of view and applied to any experimental probes therein [24].
The disadvantage of local affine approaches is that a significant amount of calibration data is required to robustly estimate the large number of parameters (usually multiple images of a standard calibration bead slide). Moreover, the number of reference spots required and the number of localizations to be done increase proportionally to the area of the field of view, as does, in consequence, the computing time required.
Given the cumbersome implementation of local correction, a global correction method that provides similar accuracy would be ideal. In this paper, we propose such a solution. We begin by observing that the distortions we see in microscopy images can be both predicted and modeled. The well-known optical distortions in play—spherical aberration, astigmatism, and so on—can be mathematically described by the Zernike polynomials [18] (see Fig. 1(b)).
Sums of Zernike polynomials are already used in some applications to improve the localization precision of fluorescence imaging [14,19]. However, the Zernike polynomials are scalar functions. To refine the colocalization of two images, a vector quantity that can describe the distortion between the images is necessary. Recently, Zhao and Burge proposed a basis set of vector polynomials on the unit disk—the so-called S and T polynomials—that use sums of Zernike polynomial gradients to describe vector fields subject to these distortions [25,26]. The S polynomials describe distortions related to curl-less distortions such as astigmatism, while the T polynomials describe the rotational versions of these distortions associated with non-zero curl. One S polynomial, $\mathbf {S}_6$, and its relationship to the Zernike polynomials are shown in the inset to Fig. 1(b).
S/T polynomial decomposition has several advantages compared to the alternative methods. First, the decomposition is physically meaningful compared to a generic polynomial fit because distortions that play a greater role in the field will be more significant in the sum. In addition, four low-order S and T polynomials describe x and y translation, scale, and rotation, allowing one to directly integrate the simple affine corrections with the higher-order corrections. Compared to local affine approaches, S/T decomposition has a much smaller number of free parameters, allowing it to be performed with far less calibration data. Finally, it specifically addresses the issue of image degradation at the edges of a field of view.
In this paper, we use S/T polynomial decomposition (STPD) to model the distortions between a pair of reference images taken on a commercial TIRF microscope using an EMCCD camera with two color channels. Our localization precision approaches the Cramér-Rao lower bound—the theoretical lower limit defined by the illumination intensity, camera noise threshold, magnification, and other parameters [27]. By subtracting the modeled field from the distortion field, we reduce the mean residual colocalization error in a DNA-PAINT nanoruler image from 62 nm to 17 nm. The residual error remains less than 20 nm more than 30 $\mathrm{\mu}$m from the center of the image, whereas with affine correction only, the accuracy degrades past this benchmark 11 $\mathrm{\mu}$m from the center. STPD is easy to implement in combination with open-source analysis software such as Picasso [13].
2. Methods
2.1 S/T polynomial decomposition
The S/T polynomials ($\mathbf {S}_j$, $\mathbf {T}_j$) are a set of complete, orthonormal vector fields over the unit circle, given in two papers by Zhao and Burge [25,26] They are defined as linear combinations of derivatives of the Zernike polynomials $Z_j$; these derivatives are given by recurrence relations [28].
Our distortion map $\mathbf {D}$ is defined at a finite set of $N$ coordinates $\mathbf {r}_k$ (Supplement 1, Fig. S1). In this case, we can calculate decomposition coefficients using:
The distortion field $\mathbf {D}$ can be approximated by a correction field:
It is important to note that the S/T polynomials are not strictly orthonormal on the finite grid defined by $\mathbf {r}_k$, which means that the initial correction field $\tilde {\mathbf {D}}$ is not optimal. In other words, the residual field $\mathbf {R} = \mathbf {D} - \tilde {\mathbf {D}}$ would have nonzero decomposition coefficients if we were to apply the decomposition procedure again. Hence, we propose an iterative procedure, where the decomposition is calculated for the residual field until the result converges.
The procedure looks as follows:
- 2. Calculate the residual distortion map $\mathbf {R}^{(1)} = \mathbf {D} - \tilde {\mathbf {D}}^{(1)}$.
- 3. Calculate $\tilde {\mathbf {D}}^{(i+1)}$, given by $a_{S,j}^{(i+1)}, a_{T,j}^{(i+1)}$, from the residual distortion map $\mathbf {R}^{(i)} = \mathbf {D} - \sum _{l=1}^i \tilde {\mathbf {D}}^{(l)}$ iteratively until convergence.
- 4. The final correction field $\tilde {\mathbf {D}}$ is given by the coefficients $a_{S,j} = \sum _l a_{S,j}^{(l)}$, $a_{T,j} = \sum _l a_{T,j}^{(l)}$.
Convergence is reached once the mean absolute residual error ($MARE = \frac {1}{N} \sum _{k=1}^N |\mathbf {R}^{(i)}(\mathbf {r}_k)|$) remains approximately constant ($\Delta MARE < 10^{-6}$) for increasing $i$. In general, the iterative procedure converges more quickly for denser populations of reference points. Figure 2(a) and (b) show a vector field with randomly-generated S/T coefficients ($j_{Smax}=45$, $j_{Tmax}=45$) on a sparse circular grid (76 points) and the same field up-sampled to 2892 points. Figure 2(c) shows that the decomposition of the dense field in (b) converges to a $MARE$ of $2.7 \times 10^{-9}$ in 68 iterations (purple dots), while the sparse field in (a) converges to $MARE = 10^{-8}$ in 180 iterations (orange dots). Grids with sampling densities between the two also converge, while for an undersampled grid (21 points; blue dots) the procedure can diverge.
Fig. 2. Iterative determination of an optimal S/T polynomial decomposition set. (a) A randomly-generated, sparse distortion field (76 points). (b) The same distortion field as in (a), up-sampled to 2892 points. (c) Mean residual error versus iteration number for distortion fields ranging from 21 points to 2892 points. The field in (a) is shown in orange dots, and the field in (b) is shown in purple. The mean absolute residual error ($MARE$) from iteratively decomposing the dense grid in (b) reaches $2.7 \times 10^{-9}$ in 68 iterations, while the $MARE$ of the sparse grid in (a) converges to $10^{-8}$ in 180 iterations. Grids with 180 or 2892 points (green or purple dots, respectively) converge in less than 100 iterations, while a very sparse grid (blue dots) diverges.
The S and T polynomial series are infinite in length, but in practice, the maximum orders used in the decomposition, $j_{Smax}$ and $j_{Tmax}$, are limited by the resolution of the reference grid (or more generally, the density of reference probe fluorophores). To determine $j_{Smax}$ and $j_{Tmax}$, the optimization procedure above is performed for a range of values of $j_{Smax}$ and $j_{Tmax}$. As illustrated in Fig. 1(b), the Zernike and S/T polynomials depend on powers of the radial coordinate $r$ that systematically increase with $j_{Smax}$ and $j_{Tmax}$. We performed the decomposition by adding S/T polynomials in blocks that correspond to powers of $r$. The $j$ values that result in the lowest residual error are used.
The residual error of an iterative polynomial decomposition also depends on the density of reference probes. As discussed in the context of Fig. 3(d) and in Supplement 1, Fig. S1(d), higher sampling density leads to convergence of higher-order polynomials, but not necessarily improved residual error. For a very dense grid, the optimal $j_{Smax}$, $j_{Tmax}$ values could tend to infinity.
2.2 Microscopy
Data were collected on a TIRF microscope assembled from commercial parts at the Kavli Nanolab Imaging Center at Delft University of Technology. The microscope comprises a Ti2-E inverted microscope base (Nikon), a Nikon Apo TIRF 100x objective (1.49 NA), a commercial illumination module (iLAS 2, Gataca Systems), and an iXON Ultra EMCCD camera (Andor). Excitation at 561 and 642 nm was interleaved with 100 ms exposure time.
2.3 Samples
Reference data were collected using a PSFcheck slide [29] (PSFcheck, Exeter, UK), which contains several laser-written patterns that can be used for 2D and 3D microscope calibration. Briefly, high-intensity infrared laser pulses are used to create microscopic voids in three-dimensional patterns in a glass substrate. The dense layer of material around the void regions exhibits increased autofluorescence [29]. We imaged the ‘3D point array,’ a grid of 350 nm diameter features with 10 $\mathrm{\mu}$m spacing under 561 nm and 642 nm illumination for 100 frames, then translated the grid by 5 $\pm$ 0.5 $\mathrm{\mu}$m and repeated the measurement. By translating the grid twice more and concatenating the four image stacks, we generated a grid with effective, though approximate, 5 $\pm$ 0.5 $\mathrm{\mu}$m spacing and four times greater point density.
Validation data were collected using a Gatta-PAINT 80RY DNA-PAINT calibration sample obtained from GATTAquant DNA Nanotechnologies (Gräfelfing, Germany). Gatta-PAINT "nanorulers" are DNA origami structures with precisely-spaced binding sites for one or more fluorophores, sold by GATTAquant pre-mounted on a microscope slide. 80RY nanorulers have three binding sites arranged linearly and spaced by $80 \pm 5$ nm. Each site can transiently bind either ATTO 655 (red) or ATTO 565 (green/yellow).
2.4 Data analysis
Initial localization of reference grid points and nanorulers was performed using Picasso Localize [13] (EM gain 1, baseline 135, sensitivity 0.06 (calculated using [30]), and quantum efficiency 1). Images of the DNA-PAINT nanorulers were generated in Picasso Render [13] using pixel size 125 nm, and selection of localizations for quantification was performed in the same software. Throughout the manuscript, the brighter, green channel has been rendered in red, while the red channel is shown in green. The rendering software did not permit separate intensity scaling of the two channels, and we found that inverting the false color made the PSFs easier to see in print format, particularly in Fig. 4.
All polynomial decomposition and image correction were performed using a custom Python library, Naclib, written in-house and available with documentation at [31].
3. Results
The calibration workflow for STPD is shown in Fig. 3(a), which includes references to subsequent figure panels. In short, the vector distortion field, $\mathbf {D}$, between reference images in two color channels is calculated by determining the offset between the fitted centroid positions of corresponding fluorescent spots. Since the reference sample has a finite number of points, there will be an optimal number of S/T polynomials to include in the polynomial decomposition of the distortion field. We determine the optimal number, calculate the correction field, $\tilde {\mathbf {D}}$, and export the decomposition coefficients, which can then be used to correct experimental images taken on the same microscope. These coefficients characterize the relative distortion between the red and green channels of the imaging pathway (optics and EMCCD camera).
The reference images that we use to illustrate STPD are shown in Fig. 3(b). The upper panel depicts the PSFcheck reference grid, an array of etched fluorescent features (350 nm, grid spacing $10 \times 10\;\mathrm{\mu}$m [29]). To increase the feature density, we translated the grid three times by $\sim$5 $\mathrm{\mu}$m and projected the resulting image stacks. The final reference image in the green channel is shown in the center panel, and the red channel in the lower panel. Some large-scale affine differences, especially differences of translation and scale, are visible when comparing the green and red channels by eye.
Figure 3(c) shows the distortion field (displacement vectors) between the green PSF positions and the red ones. We designate (arbitrarily) the red channel as the reference and apply all corrections to the green channel. In Fig. 3(c), the arrow color indicates the magnitude of the distortion at each grid point, while the size is scaled for visual clarity. The distortion magnitude is smaller at the center of the image, and the divergence of the field indicates that scaling differences play a large role in the distortion.
Fig. 3. S/T polynomial decomposition of a reference distortion field. (a) Workflow for calibration of image distortions using STPD. (b) The calibration sample, a PSFcheck slide with laser-written features [29]. The point array is in the upper left of the top panel. (Image reproduced from https://www.psfcheck.com/psfcheck-slides.) The calibration images in the green (middle) and red (bottom) channels are composed of four superimposed images of the grid, translated by approx. 5 $\mathrm{\mu}$m each time. (c) The measured vectorial distortion field between the green and red reference images. (d) Selection of the optimal polynomial decomposition set. The S/T polynomial set truncating at fifth order in S ($\mathbf {S}_{28}$) and third order in T ($\mathbf {T}_{15}$) minimizes the residual field, as measured by the mean absolute residual error ($MARE$) of the distortion vectors. The optimal $MARE$ is $0.085$ px. (e) The correction field, with coefficients optimized to model the input distortion field. (Compare panels c and e.) (f) The residual field following subtraction of (e) from (c). Panels c, e, and f have the same color scale, although the arrow lengths are scaled differently for visual clarity. (g) The coefficients of the terms in the S/T polynomial sum, with S polynomial coefficients in red and T polynomial coefficients in navy. Vector fields are shown as insets for the polynomials with the largest weights: $\mathbf {S}_{4}$, $\mathbf {T}_{7}$, $\mathbf {S}_{11}$, and $\mathbf {S}_{12}$.
We computed the S/T polynomial decomposition of this distortion field for $\mathbf {S}_j$, $\mathbf {T}_j$ combinations up to $j_{max}$ = 45, which corresponds to seventh-order distortions in the radial coordinate $r$ (Fig. 3(d)). We imposed that $j_{Tmax} \le j_{Smax}$, since in our experience the maximum required order of T was consistently less than that of S.
Figure 3(d) shows the mean absolute residual error ($MARE$) of the residual field computed for each polynomial tested, beginning with polynomials that are first order in $r$ ($j = 2, 3$). A decomposition that includes polynomials up to $\mathbf {S}_{28}$ and $\mathbf {T}_{15}$ minimizes the residual ($MARE = 0.085$ px), corresponding to S polynomials up to fifth order in $r$ and T polynomials up to third order in $r$. The $MARE$ diverges for most higher-order polynomials (grey cells). We selected the polynomial ending at $\mathbf {S}_{28}$, $\mathbf {T}_{15}$ to define our correction field. The affine correction alone is not shown in Fig. 3(d) because it ends at $j = 4$, which does not correspond to a complete order of $r$. (The first order in $r$ includes the astigmatism terms, $j = 5, 6$.) However, its mean residual, $0.186$ px, is comparable to that shown for the polynomial ending at $j = 3$.
The optimal correction field is shown in Fig. 3(e), with the residual distortions that remain after correction shown in Fig. 3(f). The residual distortions are far smaller than the ones in the original field (Fig. 3(c)). Before correction, the maximum displacement vector magnitude was $0.94$ px ($118$ nm), and the $MARE$ was $0.50$ px ($63$ nm). and After STPD correction (Fig. 3(f)), the maximum displacement vector magnitude was only $0.21$ px ($27$ nm), compared to a $MARE$ of $0.085$ px, or $10.6$ nm.
The polynomial coefficients in the optimal decomposition field are shown in Fig. 3(g). Not every T polynomial with $j \le 15$ has a coefficient because many T polynomials are identical to the corresponding S polynomials. These T polynomials are excluded from our analyses. The primary contributions come from $\mathbf {S}_{4}$ (the affine scale term), $\mathbf {S}_{8}$ (coma), $\mathbf {S}_{11}$ (spherical), $\mathbf {S}_{12}$ (second-order astigmatism) and $\mathbf {S}_{14}$ (tetrafoil). The strongest T-polynomial contribution is $\mathbf {T}_{7}$, coma with rotation.
The scaling difference between the red and green channels was apparent from visual inspection of the distortion field (Fig. 3(c)), while the higher-order components reveal the most significant optical distortions in the system. It is important to recall that these are relative distortions between the reference channel and the other channel or channels. The choice of reference is arbitrary in principle, but if one channel is known to be nearly distortion-free, then the polynomial coefficients meaningfully identify the real distortions present in the optical system and can allow informed inference as to their possible causes. Below, we will discuss ways to achieve absolute distortion diagnosis even if there is no such "ideal" reference image.
The polynomial coefficients of the optimal decomposition field contain all the relevant information for experimental image correction, and they can be saved in a small csv file (< 1 KB). To implement STPD correction, the polynomial coefficients are used to shift the $(x, y)$ positions of the superresolved fluorescent features in an experimental image. The full (vector) value of the correction field at each location is subtracted from the $(x, y)$ coordinates, and the corrected spot positions are saved (Fig. 4(a)).
Fig. 4. Correction of CoSMoS images using STPD. (a) Workflow for the correction of relative distortions between images with STPD. (b) The reference probes are DNA PAINT nanorulers (GATTAquant) with three binding sites spaced by $80 \pm 5$ nm, which can be bound by either ATTO 655 or ATTO 565. The two channels are colored red and green, and pixels where the PSFs overlap are shown in yellow. (c) Insets from (i) the top left, (ii) the center, and (iii) the bottom right of a 64$\times$32 $\mathrm{\mu}$m field of view. (iv) Histogram of residual colocalization distances between red and green localizations that correspond to the same binding site with mean $\pm$ standard deviation (s.d.) $61.6 \pm 26.7$ nm. (d) Nanorulers (i-iii) and colocalization distances (iv) after affine correction. The mean $\pm$ s.d. is $22.7 \pm 13.3$ nm. (e) Nanorulers (i-iii) and colocalization distances (iv) after STPD correction. The mean $\pm$ s.d. is $17.3 \pm 9.5$ nm. (f) The colocalization distance as a function of distance from the center of the image. Data points and clouds are the mean and s.d. within each 20-px bin. For the uncorrected images (red), the slope (m) of the linear fit is 0.327 nm/px with intercept (b) 17.0 nm. After affine correction (green), m = 0.050 nm/px, and b = 15.6 nm. After STPD correction (blue), m = 0.024 nm/px, and b = 13.9 nm. The dashed black line is 20 nm. (g) Colocalization distance distribution following image registration with a shear-affine transformation. The mean $\pm$ s.d. is $18.4 \pm 10.0$ nm. (h) Following shear-affine registration, a fourth-degree polynomial transformation was applied. The mean $\pm$ s.d. of the colocalization distance distribution is $18.6 \pm 10.5$ nm. (i) The colocalization distance as a function of distance from the center of the image. The affine-corrected and STPD-corrected data in panel (f) are reproduced for comparison. After shear-affine correction (orange), the slope m = 0.032 nm/px, with intercept b = 13.85 nm. After fourth-degree polynomial refinement (purple), m = 0.027 nm/px, and b = 14.7 nm. The dashed black line is 20 nm.
We used STPD correction to align two-color images of DNA-PAINT nanorulers (GattaPAINT 80RY, GATTAquant). These DNA nanostructures have three binding sites for the dyes ATTO 655 and ATTO 565, arranged linearly and separated by $80 \pm 5$ nm. Either dye can bind transiently at each site, so a well-aligned image will show colocalization at each site and clear separation of the sites on each nanoruler.
We illuminated the nanoruler sample with green and red excitation light (561 and 642 nm, respectively) and collected a 5000-frame image stack (Fig. 4(b) (left panel) and Supplement 1, Figs. S2(a) and (b)). We used Picasso Localize [13] to fit each PSF, then determined their positions with a localization precision of $1.8 \pm 0.5$ nm in the green channel and $4.6 \pm 1.2$ nm in the red channel (Supplement 1, Fig. S2(c) and (d)). The difference was due to the lower photon counts and larger diffraction-limited PSF in the red channel, both of which decrease the maximum achievable localization precision.
To highlight the variation in the distortion across the field of view and the effect of correction, we show close views of clusters of nanorulers in three different parts of the $512 \times 256$ px ($64 \times 32\;\mathrm{\mu}$m) field of view in Fig. 4(c)-(e): the top left (first column), middle (second column), and bottom right (third column). The first row (c) shows the localizations identified in the raw image, the second (d) shows the localizations after the green image has been affine-corrected relative to the red image, and the third (e) shows the STPD-corrected localizations, all generated in Picasso Render.
It is clear from the middle column of Fig. 4(c)-(e) (the insets from the center of the FOV) that image correction has little impact at the center of the image. Since distortions tend to be worse at larger $r$, this is to be expected. However, at opposite corners of the image, approximately 72 $\mathrm{\mu}$m apart, the uncorrected channels deviate from one another in different directions due to the nonlinear distortions between the images (compare c-i and c-iii), while affine and STPD correction bring the red and green nanoruler images into progressively closer alignment.
The overall distribution of colocalization distances after each type of correction is shown in Fig. 4(c)-(e), panels iv. Panel c-iv shows the raw, uncorrected distribution of residual distances between colocalized binding site PSFs. The uncorrected colocalization distances [32] have mean $\pm$ standard deviation $61.6 \pm 26.7$ nm.
After affine correction (Fig. 4(d)-(iv)), the mean of the distribution decreases to $22.7 \pm 13.3$ nm. STPD correction (Fig. 4(e)-(iv)) decreases the mean residual separation further by $\sim$24% to $\mathrm{\mu} = 17.3 \pm 9.5$ nm. The mean accuracy is substantially improved by STPD, although the positive skew toward larger separations (see Fig. 4(e)-(iv)) indicates that some image degradation still occurs at the image periphery.
That said, Fig. 4(f) clearly shows that STPD correction largely compensates for this degradation compared to the uncorrected or affine-corrected images. In Fig. 4(f), the colocalization distances in 4(c,d,e)-(iv) are plotted as a function of distance from the center of the image with a bin size of 20 px or 2.5 $\mathrm{\mu}$m (markers are means, and clouds represent the standard deviation) and linearly fitted.
Although all three methods perform similarly in the inner $\sim$40 px (5 $\mathrm{\mu}$m) of the FOV, the uncorrected colocalizations (red) degrade at a rate of 0.327 nm/px, leading to an average distance of 20 nm between nominally colocalized spots at 10 pixels, and 100 nm at the image edge (256 px or 32 $\mathrm{\mu}$m away). Since 20 nm is the mean colocalization error achieved across the entire FOV using local affine correction [23], it is a useful metric to define the size of the effective FOV achieved by our correction method. Before any correction, the effective FOV using this metric is only 4.2 $\mathrm{\mu}$m$^{2}$.
Affine correction decreases the rate of degradation to only 0.050 nm/px, expanding the field of view in square pixels by almost a factor of 100, to 375 $\mathrm{\mu}$m$^{2}$. STPD correction reduces the rate of degradation by more than a factor of two, to 0.024 nm/px. As the mean colocalization distance remains below 20 nm out to 254 px, this expands the effective area by an additional factor of 8, allowing it to encompass almost the entire field of view (more than 3000 $\mathrm{\mu}$m$^{2}$). Importantly, this enables colocalizations on the periphery of the image to be assessed with essentially the same accuracy as those in the center.
Finally, we compared the performance of our S/T polynomial decomposition to two other global approaches to channel registration: an affine transformation that includes shear based on the CoSMoS Pipeline from Ref. [20], and a fourth-order polynomial decomposition based on the Traces algorithm from Ref. [21]. Image shear does not correspond to an S or T polynomial, so its inclusion is expected to provide better image correction than the STPD-based affine correction that lacks it, while refining that transformation with a fourth-order polynomial interpolation is an alternative way of capturing nonlinear distortions. We implemented both steps in MATLAB and applied both approaches to the same data set shown in Fig. 4(c)-(e).
In Fig. 4(g), registration using a shear-affine transformation results in a residual colocalization distance of $18.4 \pm 10.0$ nm . This is better than the affine correction implemented without shear using S and T polynomials (22.7 nm), but not as accurate as STPD correction with higher-order terms (17.3 nm). Figure 4(h) shows that following the shear-affine correction with a fourth-order polynomial fit does not improve the fit (colocalization distance $18.6 \pm 10.5$ nm). The STPD decomposition outperforms the general fourth-order polynomial correction.
Figure 4(i) compares the performance of these four channel registration approaches as a function of position in the field of view. The shear-affine correction flattens the field of residual colocalization distances to a degradation rate of 0.032 nm/px, again outperforming the affine correction without shear at the image edges. Although fourth-order polynomial correction does not improve the mean colocalization accuracy compared to shear-affine correction, it does flatten the field of view to almost the same extent as STPD (slope = 0.027 nm/px, compared to 0.024 nm/px for STPD). However, the polynomial-corrected data crosses the 20 nm threshold 198 px from the image center, resulting in a FOV 60% the size of the one resulting from STPD.
4. Discussion
We have shown that a global image registration method can produce colocalization accuracies similar to those reported for high-resolution local alignment procedures like those used by Friedman and colleagues [23]. By decomposing the distortion between two images as a sum of S/T polynomials, we can determine the highest-order optical distortion that contributes significantly to the field, then apply the modeled field to experimental data to correct distortions that degrade colocalization measurements. This global correction method outperforms three other common approaches: affine correction with and without shear, and correction with a generalized fourth-order polynomial.
Like the other global correction methods, STPD correction benefits from a dense field of reference points, in particular because it can then accurately detect the contributions of higher-order polynomials (Fig. 2(b)). However, the number of calculations required does not scale with spot count or field of view size, remaining tractable over large chips or dense data sets. We found that even with a comparatively sparse grid of 78 reference points, STPD correction increased the accuracy of colocalization measurements on our instrument to $\sim$17 nm residual separation, which exceeds the accuracy of local affine correction despite using an order of magnitude less reference data. On an instrument with larger optical distortions, the improvement would be more significant.
The primary advantage of local affine correction, aside from its high channel registration accuracy, is the uniformity of the correction across a large FOV. Since every PSF registration is calculated using only the nearest reference probes, the registration accuracy is essentially independent of probe location. The quality of global corrections, especially affine corrections, tends to vary across the FOV, reaching its best accuracy in the center of the FOV and degrading toward the edges.
Although STPD is also a global correction technique, it decreases the rate of degradation across the field of view by an order of magnitude compared to the uncorrected images, enabling large-FOV, multiplexed measurements to be conducted with high accuracy without sacrificing data at the image periphery or greatly increasing the computation time needed to perform the correction, as with local affine methods. In addition, the underlying distribution of residual offsets in the x and y directions is more uniform following STPD correction than an affine correction, assisting data interpretation and distance measurements.
These features of STPD will gain value as sCMOS cameras with large detection chips become more common in TIRF microscopy. The large chip allows a much larger field of view, and hence more data, to be captured, improving experimental throughput. However, field distortions and inhomogeneity in both detection sensitivity and noise can degrade image quality at the periphery so much that only the middle portion of the chip can effectively be used, negating much of the advantage [19]. STPD provides an efficient way to register microscopy data uniformly over extra-large fields of view.
STPD operates by shifting the fitted (x,y) locations of the PSFs in one channel relative to another, or relative to an absolute reference, as will be discussed below. Accordingly, it is applicable to any microscopy technique in which PSFs can be localized in this way with high precision. In particular, it can compensate for the larger fields of view in SIM and other superresolution microscopies. In principle, STPD could also be applied to brightfield images by shifting every pixel in a raw image using the correction field, then generating a corrected image using pixel interpolation [33]. However, this is outside the scope of this work.
In addition, although the S/T polynomials have no extension to 3D, we envision that 2D polynomial decomposition could be applied to each slice in an optically-sectioned volume, decomposing and correcting the more severe distortions encountered deeper into the sample separately to the layers above. Care would have to be taken in volumetric imaging to ensure that the decomposition varies approximately continuously from slice to slice.
The distortions we have been discussing are relative: one imaging channel compared to another. However, even if two channels have been well aligned to one another, they could still be distorted relative to a mathematically ideal plane. Copeland et al. have vividly illustrated that systematic errors can lead to drastically incorrect distance measurements despite high localization precision [19]. Although STPD allows high-accuracy registration of one channel to another, which is itself sufficient for FRET or CoSMoS measurements, the reference channel may still have significant distortions that would render distance measurements unreliable.
STPD allows one to correct these systematic errors, even in single-channel images, with the addition of one element. If the images can be compared to a ground truth that can be accurately modeled in silico—for example, a dense, evenly-spaced grid of points, or a single point being translated across the FOV in uniform steps by a nanopositioning stage—then the measured (x,y) positions in the image can be corrected relative to their simulated positions to determine the absolute optical distortions in the image. The corrections in our Python library, Naclib, proceed identically whether the reference points are experimental or simulated. Absolute correction in this way would also permit accurate, distortion-corrected measurements of absolute distances in microscopy images, provided that the nanopositioning stage or sample is itself well-characterized [14].
Finally, STPD is information-rich, unlike the coefficients produced by a general high-order polynomial correction. The coefficient magnitudes indicate the relative weight of specific field distortions, which may provide useful information for optical system design and diagnostics. We anticipate that S/T polynomial decomposition of microscopy images will provide a wealth of insights into optical systems and increased flexibility for microscopists.
5. Conclusion
In this paper, we have shown that S/T Polynomial Decomposition (STPD) can be used to register fluorescence microscopy images and correct relative distortions with comparable accuracy to local registration, despite being a global registration technique. In STPD, we identify and correct the nonlinear distortions in an image instead of fitting with generalized polynomials, which decreases the number of fit variables and reference data points required. We needed fewer than 100 reference points to decrease the residual separation between colocalized nanoruler images to 17 nm. Because STPD addresses the real distortions affecting an entire image, this high degree of colocalization accuracy extends across the whole 64 x 32 $\mathrm{\mu}$m field of view, largely avoiding degradation and information loss at the edges. We expect that STPD will be useful in single-molecule and superresolution microscopy to maximize the field of view and increase experimental throughput.
Funding
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (16761, VENI; 16PR1047, FOM Projectruimte; 740.018.015, START-UP); European Research Council (789267, Advanced Grant).
Acknowledgments
The authors thank Dr. Serge Vincent (TU Delft, Delft, Netherlands) for assistance with data processing and Dr. Jérémie Capoulade (Kavli Nanolab Imaging Centre, TU Delft, Delft, Netherlands) for use of the microscopy facility.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are available in Ref. [34]. The Naclib library and further example data are available as part of the GitLab repository at Ref. [31].
Supplemental document
See Supplement 1 for supporting content.
References
1. S. Hohng, C. Joo, and T. Ha, “Single-molecule three-color fret,” Biophys. J. 87(2), 1328–1337 (2004). [CrossRef]
2. A. A. Deniz and J.-P. Clamme, “Three-color single-molecule fluorescence resonance energy transfer,” ChemPhysChem 6(1), 74–77 (2005). [CrossRef]
3. N. K. Lee, A. N. Kapanidis, H. R. Koh, Y. Korlann, S. O. Ho, Y. Kim, N. Gassman, S. K. Kim, and S. Weiss, “Three-color alternating-laser excitation of single molecules: Monitoring multiple interactions and distances,” Biophys. J. 92(1), 303–312 (2007). [CrossRef]
4. R. B. Sekar and A. Periasamy, “Fluorescence resonance energy transfer (fret) microscopy imaging of live cell protein localizations,” J. Cell Biol. 160(5), 629–633 (2003). [CrossRef]
5. L. J. Friedman, J. Chung, and J. Gelles, “Viewing dynamic assembly of molecular complexes by multi-wavelength single-molecule fluorescence,” Biophys. J. 91(3), 1023–1031 (2006). [CrossRef]
6. A. A. Hoskins, L. J. Friedman, S. S. Gallagher, D. J. Crawford, E. G. Anderson, R. Wombacher, N. Ramirez, V. W. Cornish, J. Gelles, and M. J. Moore, “Ordered and dynamic assembly of single spliceosomes,” Science 331(6022), 1289–1295 (2011). [CrossRef]
7. D. J. Crawford, A. A. Hoskins, L. J. Friedman, J. Gelles, and M. J. Moore, “Single-molecule colocalization fret evidence that spliceosome activation precedes stable approach of 5’ splice site and branch site,” Proc. Natl. Acad. Sci. U. S. A. 110(17), 6783–6788 (2013). [CrossRef]
8. L. S. Churchman, Z. Ökten, R. S. Rock, J. F. Dawson, and J. A. Spudich, “Single molecule high-resolution colocalization of cy3 and cy5 attached to macromolecules measures intramolecular distances through time,” Proc. Natl. Acad. Sci. U. S. A. 102(5), 1419–1423 (2005). [CrossRef]
9. A. Matsuda, T. Koujin, L. Schermelleh, T. Haraguchi, and Y. Hiraoka, “High-accuracy correction of 3d chromatic shifts in the age of super-resolution biological imaging using chromagnon,” J. Visualized Exp. 160, e60800 (2020). [CrossRef]
10. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef]
11. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm),” Nat. Methods 3(10), 793–796 (2006). [CrossRef]
12. M. Bates, B. Huang, G. T. Dempsey, and X. Zhuang, “Multicolor super-resolution imaging with photo-switchable fluorescent probes,” Science 317(5845), 1749–1753 (2007). [CrossRef]
13. J. Schnitzbaur, M. T. Strass, T. Schlichthaerle, F. Schueder, and R. Jungmann, “Super-resolution microscopy with dna-paint,” Nat. Protoc. 12(6), 1198–1228 (2017). [CrossRef]
14. C. R. Copeland, C. D. McGray, B. R. Ilic, J. Geist, and S. M. Stavis, “Accurate localization microscopy by intrinsic aberration calibration,” Nat. Commun. 12(1), 3925 (2021). [CrossRef]
15. L. S. Churchman, H. Flyvbjerg, and J. A. Spudich, “A non-gaussian distribution quantifies distances measured with fluorescence localization techniques,” Biophys. J. 90(2), 668–671 (2006). [CrossRef]
16. S. Niekamp, J. Sung, W. Huynh, G. Bhhabha, R. D. Vale, and N. Stuurman, “Nanometer-accuracy distance measurements between fluorophores at the single-molecule level,” Proc. Natl. Acad. Sci. U. S. A. 116(10), 4275–4284 (2019). [CrossRef]
17. K. I. Mortensen, J. Sung, H. Flyvbjerg, and J. A. Spudich, “Optimized measurements of separations and angles between intra-molecular fluorescent markers,” Nat. Commun. 6(1), 8621 (2015). [CrossRef]
18. F. Zernike, “Beugungstheorie des schneidenverfahrens und seiner verbesserten form, der phasenkontrastmethode,” Physica 1(7-12), 689–704 (1934). [CrossRef]
19. C. R. Copeland, J. Geist, C. D. McGray, V. A. Aksyuk, J. A. Liddle, B. R. Ilic, and S. M. Stavis, “Subnanometer localization accuracy in widefield optical microscopy,” Light: Sci. Appl. 7(1), 31 (2018). [CrossRef]
20. C. S. Smith, K. Jouravleva, M. Huisman, S. M. Jolly, P. D. Zamore, and D. Grunwald, “An automated bayesian pipeline for rapid analysis of single-molecule binding data,” Nat. Commun. 10(1), 272 (2019). [CrossRef]
21. S. L. Johnson, M. J. Johnson, and G. J. Narlikar, “Traces: single molecule fret analysis code,” GitHub repository (2018), https://github.com/stephlj/Traces.
22. A. Pertsinidis, Y. Zhang, and S. Chu, “Subnanometre single-molecule localization, registration and distance measurements,” Nature 466(7306), 647–651 (2010). [CrossRef]
23. L. J. Friedman and J. Gelles, “Multi-wavelength single-molecule fluorescence analysis of transcription mechanisms,” Methods 86, 27–36 (2015). [CrossRef]
24. A. Pitiot, G. Malandain, E. Bardinet, and P. M. Thompson, Piecewise Affine Registration of Biological Images (Springer-Verlag, Berlin, 2003), vol. 2717, pp. 91–101.
25. C. Zhao and J. H. Burge, “Orthonormal vector polynomials in a unit circle, part i: basis set derived from gradients of zernike polynomials,” Opt. Express 15(26), 18014–18024 (2007). [CrossRef]
26. C. Zhao and J. H. Burge, “Orthonormal vector polynomials in a unit circle, part ii: completing the basis set,” Opt. Express 16(9), 6586–6591 (2008). [CrossRef]
27. C. S. Smith, N. Joseph, B. Rieger, and K. A. Lidke, “Fast, single-molecule localization that achieves theoretically minimum uncertainty,” Nat. Methods 7(5), 373–375 (2010). [CrossRef]
28. T. B. Andersen, “Efficient and robust recurrence relations for the zernike circle polynomials and their derivatives in cartesian coordinates,” Opt. Express 26(15), 18878–18896 (2018). [CrossRef]
29. A. D. Corbett, M. Shaw, A. Yacoot, A. Jefferson, L. Schermelleh, T. Wilson, M. Booth, and P. S. Salter, “Microscope calibration using laser written fluorescence,” Opt. Express 26(17), 21887–21899 (2018). [CrossRef]
30. R. Heintzmann, P. K. Relich, R. P. J. Nieuwenhuizen, K. A. Lidke, and B. Rieger, “Calibrating photon counts from a single image,” (2018). ArXiv: physics.ins-det/1611.05654.
31. K. A. McCluskey, E. N. van Veen, J. P. Cnossen, W. J. Wesselink, F. M. Asscher, C. S. Smith, and N. H. Dekker, “Code for Global correction of optical distortions in multicolor single-molecule microscopy using Zernike polynomial gradients,” GitLab (2021), https://gitlab.tudelft.nl/nynke-dekker-lab/public/naclib/.
32. Because the distributions of separate x and y displacements are normally distributed with mean ≈ 0 and distinct variances (Supplement 1, Fig. S3), the Hoyt/Nakagami distribution is appropriate to accurately describe the measured interfluorophore distances. The only exception to this is the raw, uncorrected data (Supplement 1, Fig. S3(a)), which we described with a normal distribution because the underlying x and y displacements did not satisfy the requirements of the Hoyt/Nakagami distribution. The shape and scale parameters of the Hoyt/Nakagami fits are given in Supplement 1, Table S1.
33. C. S. Smith, S. Preibisch, A. Joseph, S. Abrahamsson, B. Rieger, E. Myers, R. H. Singer, and D. Grunwald, “Nuclear accessibility of β-actin mRNA is measured by 3D single-molecule real-time tracking,” J. Cell Biol. 209(4), 609–619 (2015). [CrossRef]
34. K. A. McCluskey, E. N. van Veen, J. P. Cnossen, W. J. Wesselink, F. M. Asscher, C. S. Smith, and N. H. Dekker, “Supporting data for Global correction of optical distortions in multicolor single-molecule microscopy using Zernike polynomial gradients,” 4TU.ResearchData (2021), https://www.doi.org/10.4121/17048552.
