We investigated the penalized maximum likelihood estimation of lifetime and amplitude images for fluorescence lifetime imaging microscopy. The proposed method penalizes large variations in the lifetimes and amplitudes in the spatial domain to reduces noise in the images, which is a serious problem in the conventional maximum likelihood estimation method. For an effective optimization of the objective function, we applied an optimization transfer method that is based on a separable surrogate function. Simulations show that the proposed method outperforms the conventional MLE method in terms of the estimation accuracy, and the proposed method yielded less noisy images in real experiments.
© 2013 OSA
Fluorescence lifetime imaging microscopy (FLIM) is based on the excited-state decay properties of fluorophores and is useful for studying molecular environments in living cells and human tissues [1, 2] because the lifetime of a fluorophore is dependent on the molecular environment . FLIM has been used to study, for example, the intracellular pH distribution from an estimated lifetime map [3,4] and fluorescence resonance energy transfer in a cell . Since the lifetime is an indicator of the molecular environment, an accurate and precise estimation of the lifetime is crucial for applying FLIM to biological studies. The fluorescence lifetime can be estimated in the time domain using a time-correlated single photon counting (TCSPC) technique that fits an exponential decay model to measured photon counts by maximizing a similarity measure such as χ2 or the Poisson likelihood function . It was reported that maximum likelihood estimation (MLE) based on maximizing the Poisson likelihood function outperforms other methods for TCSPC-based FLIM [6–8]. Alternatively, the lifetime can be estimated in the frequency domain using intensity-modulated light, the associated phase shifts and signal demodulation . For both methods, however, the precision of the estimated lifetime map is limited by the signal-to-noise ratio (SNR) of the measured photon counts , which suffer from Poisson noise. Note that the precision of unbiased lifetime estimation becomes worse as SNR decreases since the variance of the estimation is bounded by the Cramer-Rao bound, which becomes higher as SNR becomes lower [9–12]. As the SNR is proportional to the total number of photon counts, increasing the exposure time of the fluorophores to the laser source may improve the SNR  but since the exposure time is usually limited due to photobleaching and imaging speed, such an attempt may not be successful .
To improve the performance of the lifetime estimation without increasing the exposure time, global analysis-based methods have been studied [5, 13]. A central assumption is that the lifetimes of all the fluorophores in an image are the same [5, 13]. Based on this assumption, data sets from different locations are analyzed simultaneously to estimate the lifetime, while the fluorophore concentration at each pixel location is allowed to be different [5, 13, 14]. The method is successful in improving the accuracy of the lifetime estimation provided the assumption is valid, but this is not always true; for example, when the lifetime is an indicator of pH, the lifetime at each pixel in an image will not necessarily be the same [3, 15].
A different approach is based on image restoration techniques such as denoising and deconvolution [2, 16, 17]. Since the variance in the lifetime estimation becomes larger as the SNR of the fluorescence decreases, an estimated lifetime map from a low-SNR signal is usually noisy. Therefore, denoising of the estimated lifetime image may decrease the variance in the lifetime estimation. Previous investigations have applied the iterative constrained Tikhonov–Miller algorithm to restore the Fourier coefficients in a frequency-domain lifetime estimation  and total variation (TV)-based denoising methods for time-gated FLIM [2,17]. However, the former method was applied to frequency domain FLIM, while the latter method was investigated for time-gated FLIM. These methods are also unable to handle data that should be excluded from the lifetime estimation, i.e., the background signal and dark pixels, where the total number of photon counts is very small . To the best of our knowledge, no investigations applying image restoration techniques to TCSPC-based lifetime estimation have been published, even though such investigations are essential as TCSPC-based lifetime estimation performs better than any other method, especially for low photon counts (i.e., low SNR cases) [3, 19, 20].
In this paper, we propose a penalized maximum likelihood estimation (PMLE) of the lifetimes and amplitudes for TCSPC-based FLIM. To overcome the noise problem of independent MLE of the lifetimes and amplitudes at each pixel, the proposed method estimates the lifetimes and amplitudes by maximizing an objective function that is the weighted sum of a negative Poisson likelihood function and a penalty function that penalizes large variations in the lifetimes and amplitudes in the spatial domain. A similar PMLE method has been shown to be effective for reducing noise in medical image reconstructions [21–24]. We calculate the variations in the lifetimes and amplitudes after excluding pixels for which the number of photon counts is below a certain threshold. We also introduce a novel parameterization to determine the lifetimes uniquely. Since the proposed objective function is not an additively separable function of parameters at each pixel, the optimization of the proposed objective function requires simultaneous optimization of a significantly large size parameter vector (i.e., lifetimes and amplitudes from every pixel), which is often intractable. To overcome this, we apply the principle of optimization transfer, which is based on a series of optimizations of a separable surrogate function [21–23] for the penalty function. It has been shown that the series of optimizations of the surrogate functions eventually converges to the minimizer of the original objective function [21–23]. We demonstrate here the effectiveness of the proposed method in comparison with the MLE through simulations and experiments.
2.1. Problem formulation
We model the number of photon counts detected at a location (xi, yj) at a time instance tk by the Poisson random variable, g(tk, xi, yj), as follows :Eq. (4), the p-th decay time constant can be represented by
2.2. Proposed method
To reduce the noise in the estimated lifetime and amplitude images using the MLE method, we propose a method that incorporates a priori knowledge: the amplitudes and lifetimes do not change rapidly in the spatial domain. The objective function of the proposed method Φ(Θ;g) consists of the weighted sum of the Poisson likelihood function and a penalty function R(Θ) that discourages large variations in the spatial domain:23]. We instead adopt an edge preserving penalty function such as the TV penalty function, which is defined as follows : Eq. (13) and (14) are not additively separable functions of θi,j, minimization of the objective function defined in Eq. (11) requires the simultaneous optimization of the 2P × 𝒞 -size parameter vector Θ, which is often intractable. To overcome this difficulty, we apply an optimization transfer approach to determine the minimizer of the objective function. For the minimization of an objective function Φ(θ), the iterative minimization of a surrogate function Φs(θ; θk) (θk denotes the estimated value of θ at the k-th iteration) monotonically converges to the minimizer of the objective function Φ(θ) if the surrogate function satisfies the following two conditions : Eq. (13) and (14) so that we can estimate the parameters at each pixel independently. To do this, we first apply Huber’s method  to design the quadratic surrogate functions of the penalty functions and then apply De Pierre’s approach  to construct the quadratic surrogate functions separately as follows [21–23]: 23]. The curvature c that guarantees the first inequality of a quadratic or TV penalty function can be determined using Huber’s method . The last inequality in Eq. (19) is due to the convex inequality . Therefore, for the p-th component, the following inequality holds: Eq. (20), the separable quadratic surrogate function of the penalty function, Rs(Θ), satisfies the following inequality: Eq. (23) can be solved numerically. The minimizer sequence defined in Eq. (23) monotonically converges to the minimizer of the objective function defined in Eq. (11).
To evaluate the performance of the proposed PMLE method and compare it with the conventional MLE method, we conducted simulation studies and real experiments. We implemented the MLE and the proposed PMLE method using the MATLAB (Mathworks, USA). In addition, we solve the minimization problems in Eq. (10) for the MLE and in Eq. (23) for the PMLE method using the fmincon function in the Optimization toolbox of the MATLAB. We run the methods on a workstation equipped with two Intel Xeon X5650 processors (2.67 GHz) and 96 GB memory.
We conducted simulation studies using a 64×64 synthetic image. The synthetic image has an elliptical structure that contains a square and a circle. We synthesized a double exponentially decaying fluorescence signal at each pixel location that has different decay constants (τ1 and τ2) and amplitudes (A1 and A2) in different regions of the image and convolved these synthesized signals with an IRF function of [0.014, 0.042, 0.121, 0.273, 0.357, 0.194]. We then generated 261 Poisson random variables at each pixel, whose means were uniformly sampled values of the convolved decaying fluorescence signal. The true values of the lifetime, amplitude and the average photon counts in each region are summarized in Table 1.
For estimating the lifetime and amplitude images from the synthesized noisy fluorescence signals (i.e., the Poisson random variables), we applied the MLE, the PMLE with a quadratic penalty function (PMLEQ), or the PMLE with a TV penalty function (PMLETV). Regularization parameters for (τ1, τ2, A1, A2) were (250, 100, 0.5, 1.5) in PMLEQ method and (8, 6, 0.4, 0.5) in PMLETV method. We also applied the MLE method with binning of the measured photon counts (i.e., summing the photon counts of neighboring pixels) from 3 × 3 neighboring pixels, which is used to reduce the effect of noise in the commercial software SPCImage (Becker & Hickl GmbH). The parameters Fig. 1 shows the estimated lifetime and amplitude images using the MLE method, which as expected yielded noisy lifetime and amplitude images due to Poisson noise. To reduce the effect of Poisson noise, we applied the PMLEQ and PMLETV methods. Figure 2 shows the estimated images obtained from the PMLEQ method, and as can be seen, the PMLEQ method was able to reduce the effect of noise. However, the quadratic penalty function blurred the edges in the estimated images. The blurring can be reduced by applying the PMLETV method, as shown in Fig. 3. Figure 4 shows estimated images using the MLE method with binning. As shown in the figure, although the binning strategy was able to reduce the effect of noise, it yielded images with artifacts, especially in the region where lifetimes in neighboring pixels are different. This is due to that photon counts data from different lifetimes are summed together.
We repeated the estimation of the lifetime and amplitude images using the four methods for 10 times with different noise realizations and computed the bias and standard deviation (STD) of the lifetimes and amplitudes for each region using the pixel values in all the estimated images. For the MLE method with binning, we normalized the bias and the standard deviation by the number of pixels in the binning window. We summarize the biases and standard deviations of the lifetimes and amplitudes in each region in Table 2. The biases of the PMLETV method were quite similar to those of the MLE method (slightly larger in estimating amplitudes), whereas the standard deviations were significantly lower than the MLE method, indicating that the effect of random noise is greatly reduced by the PMLETV method. While the PMLEQ method also yielded lower standard deviations than the MLE method, the standard deviations of the PMLEQ method were larger than those of the PMLETV method. The standard deviations of the MLE method with binning were larger than those of the PMLE methods. Note that as the effect of penalty function increases, noise can be reduced at the price of resolution. The regularization parameter controls trade-off between noise and resolution . Therefore, one should design the regularization parameter based on desired resolution and noise property. Automatic determination of the parameter is a challenging task, and we defer it to future study.
We also evaluated the similarity between the estimated and true images by computing the correlation coefficients of the four estimated images (τ1, τ2, A1, A2). Table 3 summarizes the results and reveals that the PMLE methods yield images that are more similar to the true images than those images obtained with the MLE method and the MLE method with binning. Among the two PMLE methods, the PMLETV method yielded better results than the PMLEQ method.
The disadvantage of the proposed method is longer computation time. We summarize the average computation time for estimating the lifetime and amplitude maps using the MLE, PM-LEQ and PMLETV methods in Table 4. Since we initialized lifetime and amplitude maps using the ones estimated by the MLE method, one should think that total computation times for the PMLE methods are approximately the same as the sums of the computation time for the MLE and for the PMLE methods. We expect that computation time for the PMLE methods can be reduced by using graphics processing unit and parallel computing technology as well as by the improvement of computer technology. Investigations on faster PMLE methods are deferred to future study.
3.2. Real data
3.2.1. polymer data
To prove the effectiveness of the proposed method for real data, we conducted experiments using a polymer sample which contains two different dyes that have different lifetimes. The sample was prepared as follows. Coumarin 6 (C6) and 4-(4-methoxybenzylamino)-7-nitrobenzofurazan (MBD), tetrahydrofuran (THF), poly(methyl methacrylate) (PMMA, MW 120,000) and polyvinyl chloride (PVC, MW 97,000) were purchased from Sigma-Aldrich. C6 and PVC were dissolved in THF and MBD and PMMA were dissolved in THF. Each solution was dropped at contact on a cover glass and this sample was dried in an oven at 80°C for 24 hours. The light sources were two picosecond diode lasers operating at the 442 nm wavelength with the repetition rate of 20 MHz (Picoquant). A platform for sample excitation and fluorescence detection was an inverted confocal microscope (Nikon, TE2000-S) with an oil immersion objective lens (NA 1.4, ×60). A long pass cut-off filter was placed in front of the detector to block any remaining excitation beam. The total fluorescence was detected by a micro channel plate photomultiplier tube (Hamamatsu R3809U-51). The signal was amplified by a GHz preamplifier (EG&G 9306) and the signal was processed by a fast time-correlated single photon counting (TCSPC) board (Becker-Hickl, SPC-830). The x–y images consist of arrays of pixels (256 × 256), each of which contains the fluorescence decay curve. We obtained low and high photon counts by controlling the excitation laser intensities. Using the fluorescence decay curves from 80×80 sub-image region, we attempted to estimate lifetime and amplitude images. Figure 5(a) shows estimated lifetime image from high photon counts data (average number of photons per each pixel is 2686) using the MLE method while Fig. 5(b) does from low photon counts data (average number of photons per each pixel is 335), which is noisier than Fig. 5(a). The short and long lifetime components correspond to C6 and MBD, respectively. To acquire less noisy lifetime image, we applied the PMLEQ and the PMLETV method to the low photon counts data. The results are shown in Fig. 5(c) and Fig. 5(d). As shown in the figures, the estimated lifetime images using the PMLE methods look less noisier than the image using the MLE method. The correlation coefficients between the image from the high photon counts data (which serves as a ground truth image) and the three images from the low photon counts data are 0.959 (MLE), 0.983 (PMLEQ) and 0.977 (PMLETV), respectively. As shown in the results, the estimated lifetime images using the PMLE methods are more similar to the lifetime image from high photon counts data.
3.2.2. cell data
We also attempted to estimate the lifetime and amplitude images from a HeLa cell. The HeLa cell was stained with a fluorescein dye that was covalently attached to amyloid11–25 (Fl-amyloid11–25). The fluorescence lifetime of fluorescein is known to be pH-sensitive, and we know that the amyloid11–25 is localized to specific positions in the cells. By incubating the HeLa cells with Fl-amyloid11–25, it is thought that information about the probe distributions in the cells, which is related to the pH conditions of the localized states, will be obtained from FLIM images. The fluorescence decay curves were acquired using the same system as in the polymer experiment except for that a 467-nm-wavelength laser was used for this case. The average number of photon counts from pixels where the maximum number of photon count was greater than a given threshold (for this experiment 10) was 308. We applied the MLE, PMLEQ, and PMLETV methods to estimate the lifetime and amplitude maps.
Figure 6 shows the two estimated lifetimes (τ1 and τ2) and associated amplitudes (A1 and A2) obtained using the MLE method. The MLE method yielded very noisy τ1 and τ2 maps, thereby making it difficult to identify meaningful pH states. Figure 7 shows the estimated lifetime and amplitudes obtained using the PMLEQ method, and those obtained using the PMLETV method are shown in Fig. 8. We can see that the effects of noise have been reduced in the estimated images obtained using these two methods. Compared with the images obtained with the PMLEQ method, the lifetime maps estimated using the PMLETV method show more distinct features. Figure 8(a) and Fig. 8(b) show areas where the lifetimes are larger than the surrounding area, and we suspect that the pH states in this area are different based on the lifetime images. The variation of the fluorescence lifetime of the fluorocein as a function of pH is well known phenomenon [27, 28]. It is associated with the proton transfer of the dye in the electronically excited state. The existence of two lifetimes stems from a chemical origin in which the normal and proton-transferred species are in dynamic equilibrium. Although it is not possible to directly show that the estimated lifetime maps obtained using the PMLE methods are more similar to true lifetime maps since we do not know the true lifetimes in the HeLa cell, it is evident that the estimated images using the proposed method are less noisy than those obtained using the MLE method. Moreover, we believe that the PMLE methods provide more accurate results based on the simulated results and the assumption that the true pH states do not change abruptly in the spatial domain.
We have proposed a PMLE method for estimating the lifetime and amplitude images from measured fluorescence signals. Because the objective function in the proposed method considered the spatial correlation in the lifetime and amplitude images, the method was able to reduce the effects of noise more efficiently than the conventional MLE method. The objective function of the PMLE method, however, is not a separable function, and so we applied the optimization transfer principle to design a separable surrogate function to update the lifetimes and amplitudes at each pixel independently. The simulation results showed that the proposed PMLE method outperforms the conventional MLE method in terms of accuracy, and the proposed method yielded less noisy image in real experiments. We believe that the proposed method will be useful in estimating lifetime and amplitude images for FLIM, especially for cases in which the true lifetimes and amplitudes do not change rapidly in the spatial domain.
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology ( NRF-2012R1A1A2009370) and Ewha Global Top 5 Grant 2011 of Ewha Womans University.
References and links
1. J. R. Lakowicz, Principles of Fluorescence Spectroscopy (Kluwer Academic/Plenum, 1999). [CrossRef]
6. Z. Bajzer, T. M. Therneau, J. C. Sharp, and F. G. Prendergast, “Maximum likelihood method for the analysis of time-resolved fluorescence decay curves,” Eur. Biophys. J. 20, 247–262 (1991). [CrossRef]
7. M. Kollner and J. Wolfrum, “How many photons are necessary for fluorescence-lifetime measurements?” Chem. Phys. Lett. 200, 199 –204 (1992). [CrossRef]
9. H. Cramer, Mathematical Methods of Statistics (PMS-9), Princeton Landmarks in Mathematics and Physics (Princeton University Press, 1999).
10. H. C. Gerritsen, M. A. H. Asselbergs, A. V. Agronskaia, and W. G. J. H. M. Van Sark, “Fluorescence lifetime imaging in scanning microscopes: acquisition speed, photon economy and lifetime resolution,” J. Microsc. 206, 218–224 (2002). [CrossRef]
12. J. Philip and K. Carlsson, “Theoretical investigation of the signal-to-noise ratio in fluorescence lifetime imaging,” J. Opt. Soc. Am. A 20, 368–379 (2003). [CrossRef]
14. S. Pelet, M. J. R. Previte, L. H. Laiho, and P. T. C. So, “A fast global fitting algorithm for fluorescence lifetime imaging microscopy based on image segmentation.” Biophys. J. 87, 2807–2817 (2004). [CrossRef]
15. K. M. Hanson, M. J. Behne, N. P. Barry, T. M. Mauro, E. Gratton, and R. M. Clegg, “Two-photon fluorescence lifetime imaging of the skin stratum corneum ph gradient,” Biophys. J. 83, 1682–1690 (2002). [CrossRef]
17. D. Sud and M.-A. Mycek, “Image restoration for fluorescence lifetime imaging microscopy (FLIM),” Opt. Express 16, 19192–19200 (2008). [CrossRef]
18. M. Heilemann, D. P. Herten, R. Heintzmann, C. Cremer, C. Mller, P. Tinnefeld, K. D. Weston, J. Wolfrum, and M. Sauer, “High-resolution colocalization of single dye molecules by fluorescence lifetime imaging microscopy,” Anal. Chem. 74, 3511–3517 (2002). [CrossRef]
20. E. Gratton, S. Breusegem, J. Sutin, Q. Ruan, and N. Barry, “Fluorescence lifetime imaging for the two-photon microscope: time-domain and frequency-domain methods,” J. Biomed. Opt. 8, 381–390 (2003). [CrossRef]
21. J. Fessler and A. Hero, “Penalized maximum-likelihood image reconstruction using space-alternating generalized em algorithms,” IEEE Trans. Image Process. , 4, 1417–1429 (1995). [CrossRef]
22. J.-H. Chang, J. Anderson, and J. Votaw, “Regularized image reconstruction algorithms for positron emission tomography,” IEEE Trans. Med. Imag. , 23, 1165 – 1175 (2004). [CrossRef]
23. J. Fessler, “Image reconstruction: Algorithms and analysis,” Online preprint of book in preparation.
24. A. De Pierro, “A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography,” IEEE Trans. Med. Imag. , 14, 132–137 (1995). [CrossRef]
25. P. Huber, Robust Statistics (Wiley, 1974).
26. J. Fessler and W. Rogers, “Spatial resolution properties of penalized-likelihood image reconstruction: space-invariant tomographs,” IEEE Trans. Image Process. , 5, 1346–1358 (1996). [CrossRef]
27. H.-J. Lin, P. Herman, and J. R. Lakowicz, “Fluorescence lifetime-resolved ph imaging of living cells,” Cytometry Part A 52A, 77–89 (2003). [CrossRef]
28. C. Hille, M. Berg, L. Bressel, D. Munzke, P. Primus, H.-G. Lahmannsraben, and C. Dosche, “Time-domain fluorescence lifetime imaging for intracellular ph sensing in living tissues,” Anal. Bioanal. Chem. 391, 1871–1879 (2008). [CrossRef]