Compressive fluorescence microscopy has been proposed as a promising approach for fast acquisitions at sub-Nyquist sampling rates. Given that signal-to-noise ratio (SNR) is very important in the design of fluorescence microscopy systems, a new saliency-guided sparse reconstruction ensemble fusion system has been proposed for improving SNR in compressive fluorescence microscopy. This system produces an ensemble of sparse reconstructions using adaptively optimized probability density functions derived based on underlying saliency rather than the common uniform random sampling approach. The ensemble of sparse reconstructions are then fused together via ensemble expectation merging. Experimental results using real fluorescence microscopy data sets show that significantly improved SNR can be achieved when compared to existing compressive fluorescence microscopy approaches, with SNR increases of 16-9 dB within the noise range of 1.5%–10% standard deviation at the same compression rate.
© 2012 Optical Society of America
Fluorescence is broadly used in many biological microscopy applications such as single molecular studies of individual proteins and living cells, in-vivo tracking of targets using fluorescent labeling, and molecular associations in live cells . One key advantage of fluorescence microscopy is its inherently greater optical sensitivity and dynamic range in comparison to other methods based on optical density changes or chemiluminescent emission . Fluorescence microscopy approaches are usually categorized by the way in which excitation light is delivered to the sample: i) wide-field and structured illumination, ii) surface illumination, and iii) scanning illumination. One of common most approaches in fluorescence microscopy is scanning confocal microscopy, where the sample is scanned by a laser in two dimensions to reconstruct the image . One drawback to confocal scan microscopy is its inherent slow acquisition speed since the image is been acquired pixel-by-pixel. Furthermore, the quantum efficiencies of common confocal microscopy CCD detectors is lower , resulting in lower signal to noise ratio (SNR). Achieving high SNR is of particular interest in biological fluorescence microscopy since the reflective index of cells is similar to water which forms the background. Therefore, the intensity difference between the regions of interest and the background is low and sensitivity to noise is high.
To deal with the issue of SNR, many physical noise reduction methods have been developed for fluorescence microscopy. Such methods include: reducing unwanted room background light, reducing unwanted fluorescence emission through a diaphragm that filters reflections or scattering light path [3,4], time-gated imaging [5–7] using short laser pulses for sample illumination, as well as minimizing illumination time. Another way to improve SNR is maximizing signal cleanliness through specialized lenses .
Another issue that can affect SNR is the chemical destruction phenomenon known as photobleaching, where higher illumination levels beyond the fluorescence saturation point do not contribute to signal amplitude but continue to increase background scattering leading to increased noise levels and SNR degradation [1, 9, 10]. Photobleaching can occur at timescales of milliseconds to minutes. Common strategies to decrease photobleaching effects can be by reducing illumination levels, applying specialized filters and lenses, and using less sensitive fluorophore [1, 11, 12]. The combination of very weak fluorescence signal (10−6 of the level of excitation light that produced it ) and photobleaching effects that limits the excitation light levels provides low SNR. Those limitations can lead to laser confocal microscopy that needs to measure 10–20 photons from brightest pixels in the image and as low as zero or one photon from the background . Beyond physical methods, computational methods have been attempted reduce noise levels through image filters such as anisotropic diffusion and wavelet thresholding [13–15].
One promising approach for improving acquisition speed while maintaining low system complexity and high reconstruction quality for fluorescence microscopy is the concept of compressive fluorescence microscopy (CFM) [16–19]. CFM makes use of compressive sensing (CS) theory, which allows for greatly reduced fluorescence microscopy acquisition times through the use of sparse measurements (samples). The theory of compressive sensing (CS) provides an unambiguous proof that discrete signals, which represent a sparse representation in the domain of a suitable linear transform, can be accurately recovered from their sub-Nyquist measurements [20–24]. Such reconstructions can be performed through solution of a convex optimization problem, which maximizes the sparseness of the signal representation coefficients subject to a set of measurement constraints. In such cases, the measurement constraints are derived from a signal/image acquisition model. This model normally describes the measurements as a (noise-contaminated) sequence of inner products between the quantity of interest and the elements of a sampling basis. Wu et al.  demonstrated the applicability of CFM in fast optical-sectioning imaging. Studer et al.  demonstrated the use of CFM for fast hyperspectral imaging. In addition to tackling the issue of acquisition speed, Marim et al. [16,17] recently proposed a CFM system for improving SNR by fusing multiple CFM reconstructions.
While the literature in CFM systems (or CS in general) has focused primarily on the hardware design as well as image reconstruction step, the design of the sampling procedure has not been well investigated for practical applications such as fluorescence microscopy where the regions of interest have structured characteristics. Recently, a novel saliency-guided sparse measurement model [25,26] was developed that significantly improves CS performance for situations where regions of interest have structured characteristics. Given the benefits of such a model, one is motivated to extend upon this concept for improving CFM.
The main contribution of this paper is the introduction of a compressive fluorescent microscopy system based on a new concept called saliency-guided sparse reconstruction ensemble fusion (SSREF), which is designed to improve SNR while using only a subset of measurement locations. The proposed SSREF method adaptively optimizes the sampling probability density function based on the regions of interest rather than the usual sampling of the entire area uniformly. Furthermore, a number of saliency-guided sparse reconstructions are fused together into the final reconstruction. Through these two processes, the sampling patterns are optimized in an automatic fashion in order to reduce the number of samples and improve SNR. The SS-REF for fluorescence microscopy system requires considerably fewer measurement locations for accurate fluorescence microscopy reconstruction in comparison to the traditional Nyquist measurements requirement, as well as significantly improves SNR when compared to existing CFM approaches.
2. Saliency-guided sparse reconstruction ensemble fusion (SSREF) model
In traditional CS systems, the sampling patterns used are random sequences (e.g., Gaussian, Bernoulli, etc.) which sample the entire scene uniformly [20, 27]. However, in practical fluorescence microscopy imaging scenarios, the regions of interest are typically characterized by structured characteristics with high saliency. Since we are primarily interested in preserving such important salient characteristics when aiming to achieve high reconstruction performance, the use of conventional pseudo-random sequences are limited in their suitability for such scenarios. Furthermore, many fluorescence microscopy imaging scenarios are characterized by high noise levels, which motivates the integration of improved noise compensation strategies.
To address these important issues, we propose to extend an existing saliency-guided sparse measurements model [25, 26] into a new noise robust model called the saliency-guided sparse reconstruction ensemble fusion (SSREF) model, which can be described as follows. Consider the scene being measured using the fluorescence microscopy system to contain M × N sampling locations organized in a finite, separable, rectangular lattice ΩM×N, with the measured value at each sampling location representing intensity level at that location. Such a lattice can be described as:Eq. (3)). Subset ΩD is defined based on an operator Γ(m, n) that returns a quantitative measure of saliency at sampling location (m, n):
Subsequently, we partition ΩM×N into three complementary sets ΩD, ΩS and such thatEq. (5)).
The fluorescence microscopy measurements are intended to provide discrete intensity quantity of interest, which we assume to be bounded. Then, given a collection of K ≤ NM sampling functions , the linear measurements of f can be generally described as
In the case of the proposed SSREF model, when φk, k = 1, 2,...,K is sparse, having Q non zero elements, the measurement model (Eq. (4)) needs to be properly adjusted. In particular, we need to account for the fact that there are no observations corresponding to the locations in as well as that the distributions of ΩD and ΩS are different (Eq. (7), Eq. (8)). Therefore we modify the sampling basis according toEq. (4)) are been formed in two steps: In the first step, which can be referred as the acquisition step, sampling locations (ΩD and ΩS) as well as non-sampled locations ( ) are determined. This selection decision is a sequence of Bernoulli random variables taking on values 1 and 0, which represent sampled locations and non-sampled locations, respectively. As such, we end up with a binary pattern which can be generated using a Digital Micro-mirror Device (DMD) in a similar manner as [18, 19]. In the second step, which can be referred as post-acquisition step, the sampled values (obtained via a binary pattern) are multiplied by a random variable whose probability is Gaussian distributed. Consequently, the SSREF model can now be modeled as Eq. (7)) and (Eq. (8)):
The probability density function (pdf) of is normal distributed with average μ and variance .
The acquisition and post-acquisition steps are integrated in one probability model pS (Eq. (8)) referred as Gauss-Bernoulli. As such, the proposed sampling function can be practically implemented in a real imaging system via this two-step process.Eq. (8)) is a Gauss-Bernoulli meaning pS(x) = 0 with probability π and pS(x) is Gaussian distributed with probability (1 − π). The scene is measured T times by the measurement function (Eq. (6)). Based on this SSREF model, an ensemble of saliency-guided reconstructions can be computed based on sparse measurement locations ΩD and ΩS. Finally, the ensemble of reconstructions are then fused to obtain the final reconstructed image f̃:
3. Practical realization of the SSREF model
Given the SSREF model, we present an example practical realization of the model for compressive fluorescence microscopy. Assume in this practical realization that saliency for the purpose of CFM applications can be quantified based on large spatial range variations. Therefore, define a saliency function S(m, n) (Eq. (11)), which is high at location (m, n) for situations characterized by large spatial range variations, based on a frequency-tuned saliency map strategy :28]).
In this practical realization of the SSREF model, the probability density functions pD (Eq. (7)) and pS (Eq. (8)) are selected as follows. The probability density function of , denoted by pD, is a Gaussian distribution with zero mean and unit variance (Eq. (7)) :
The probability density of , denoted by pS, is defined as a Gauss-Bernoulli distribution, where pS(x) = 0 with probability 0.9 and pS(x) is Gaussian distributed with probability 0.1.Eq. (8)), initial experiments were performed. (Fig. 1). This initial experiment for determining high compression rate where CS can still produce reasonably reconstructed data that will be considered for the learning phase. From this initial experiment (Fig. 1) it can be seen, that for very high compression rate (99%) which means that only 1% of the data is used, the learning phase is not sufficient therefore the SSREF reconstruction performance is poor and is even lower then CFM. The reason that SSREF reconstruction performance is lower than CFM is that it is based on insufficient learning data. When 5% of sampling locations are been used, there is a significant improvement to the SSREF reconstruction in comparison to the 1% case. At high compression rates (above 85%) learning using 5% of the samples improves performance compared to learning using 10% of the samples. The reason is that in the 5% case, guided sampling starts at higher compression rate. From the other end, performance of the 10% case is better at lower compression rate because the learning stage relays on more accurate and reliable data therefore is more efficient to identify regions of interest. To achieve a high compression rate in the final phase of SSREF reconstruction while including the sampling locations used for learning to determine compression rate, not considering more than 10% samples for learning is preferable. According to this initial experiment, probability π (Eq. (8)) was selected to be 0.9, representing 90% compression rate or 10% of sampling locations.
Based on the aforementioned saliency function and probability density distributions, the SS-REF model can be realized in three phases. In the first phase, which can be considered as the learning phase, a standard CS procedure is used where f is sampled by sparse with the pdf pS (Eq. (13)) using only 10% of the locations in the scene. The saliency function is then used to obtain a rough saliency map of the scene based on the reconstructed fluorescence microscopy image using this first phase. In other words, we determine subset ΩD using operator Γ (Eq. (10)).
In the second phase, f is sampled multiple times by with pdf pD(Eq. (12)) in order to measure regions of high saliency with higher accuracy. The samples from the two subsets ΩD and ΩS are then combined for creating the sampling basis φT,k,t, with the acquired samples used to reconstruct the fluorescence microscopy image at a higher accuracy than that obtained in the first phase. To reconstruct the image, a ℓ1-based total variation minimization approach was employed. The noise-free range image f can be approximated via the following ℓ1-based total variation minimization formulation:29]:
The minimization problem can be solved using the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) [29, 30]. This stage is repeated n times (based on empirical testing, n = 10 was found to produce strong results) through T acquisitions at ΩD and ΩS measurement locations to create an ensemble of T saliency-guided reconstructions . The third phase is the ensemble expectation merging phase. At this stage, the final reconstruction f̃ is computed via ensemble expectation merging  (Eq. (9))
It is noted that while compressive fluorescence microscopy signal acquisition is faster in comparison to standard confocal microscope systems (since only a small subset of a scene is sampled), it involves additional digital signal processing that may result in an overall slower reconstruction process. Nevertheless, fast acquisition for fluorescence microscopy is important for many applications, especially for situations where one wishes to image moving objects. Interesting approaches that can be employed include reconstructing the ensemble images in parallel using GPUs as well as implementing the algorithm on an FPGA, which can shorten the entire reconstruction process.
4. Experimental results and discussion
To evaluate the performance of the proposed SSREF model for use in CFM systems, a number of experiments were conducted. For comparison purposes, the state-of-the-art CFM approach proposed by Marim et al.  (which we will denote as (FCFM) in which multiple CFM reconstructions are fused was also evaluated as a baseline reference.
In the first experiment, reconstruction performance was evaluated in the situation where the measurements made were contaminated by different levels of synthetic noise. In the second experiment, reconstruction performance was evaluated at different compression rates. In the third experiment, reconstruction performance of real noisy fluorescence microscopy data was evaluated.
4.1. Experimental setup
The experiments were performed with existing fluorescence microscopy data sets provided by . Five data sets where used for the synthetic noise experiments: YRC PIR ID: 5, 12, 27, 62 and 64. The data sets are largely noiseless, and have the following imaging settings: pixel size: 0.12758 μm × 0.12758 μm, objective: 100× and image size 512×512. In addition, three noisy data sets were used for the real noise experiments: YRC PIR ID: 3499, 5352 and 8565. The data sets contain time series of at least 10 images, with the following imaging settings: pixel size: 0.12758 μm × 0.12758 μm, objective: 100× and image size 512×512.
Examples of fully sampled data (where each sampling location is measured), along with corresponding noise contaminated versions used for the first experiment, are shown in Fig. 2 and Fig. 3. Fully sampled real noisy images are shown in Fig. 4.
4.2. Experiment 1 - noise sensitivity tests
In the first experiment, a comparison of the proposed SSREF model’s reconstruction performance vs. noise levels (additive white Gaussian noise) with different ensembles is shown in Fig. 5. Up to 4% noise standard deviation there are insignificant gains from increasing ensemble size. At higher noise level, larger ensemble sizes lead to improved SNR reconstruction, with a 4dB difference between ensemble sizes of 4 and 7, and 3dB between ensemble sizes of 7 to 10.
At this point, an ensemble size of 10 is used for further experiments. A comparison between the FCFM approach and the proposed SSREF model’s reconstruction performance at different noise levels is shown in Fig. 6. The compression rate is set to 75%. It can be observed that the proposed model outperforms FCFM significantly and consistently through the entire selected noise range of 1.5–10%. For example, a 16.2 dB difference is observed at 5% noise level and a 9 dB difference is observed at 10% noise.
4.3. Experiment 2 - compression rate sensitivity tests
In the second experiment, the reconstruction performance was evaluated via a parametric analysis by computing the signal-to-noise ratio (SNR) of the reconstructed fluorescence microscopy image for a wide range of compression rates, where the compression rate ρ is defined as one minus the ratio between the number of sampling locations measured and the total number of sampling locations. Therefore, the higher the compression rate achieved, the fewer the number of measurements made. For illustrative purposes, the SNR was measured for fluorescence microscopy image, contaminated with 3% standard deviation noise and reconstructed via ensembles of 10 reconstructions across the range of 0% – 80% compression rates.
From the SNR vs. compression rate plots shown in Fig. 7 (Gaussian noise with standard deviation of 3%), the FCFM approach achieves SNR that is significantly lower than that achieved using the proposed SSREF model. For example, at the same compression rate (60% for example), the SSREF model outperforms the FCFM approach by 10.7 dB (32.5 dB vs. 21.9 dB) for Gaussian noise with a standard deviation of 3%.
To visualize the reconstruction performance of the proposed SSREF model, Fig. 8 and Fig. 9 demonstrate reconstruction of FCFM as well as the proposed SSREF model at different levels of synthetic noise. The fluorescence microscopy images produced using the proposed SSREF model contain significantly more important detail than the FCFM approach. This is most evident in the weak structure definition in the fluorescence microscopy images constructed using FCFM, which is well captured in the fluorescence microscopy images produced using the SSREF model. In addition, the proposed model provides improved noise suppression in the background regions more efficiently than FCFM.
4.4. Experiment 3 - noisy fluorescence microscopy reconstruction tests
4.5. Reconstruction examples
In the third set of tests, real noisy fluorescence microscopy images were used to compare reconstruction performance of FCFM vs. the proposed SSREF model Fig. 10 – Fig. 12. The proposed SSREF model outperforms the FCFM approach for real noisy images in terms of image quality namely better structure definition and reduced noise.
4.6. Summary of testing
Based on the three set of extensive experiments performed using fluorescence microscopy data sets, it was found that:
- The SSREF method produces significantly higher SNR under different synthetic noise scenarios when compared to existing CFM systems (9 to 16 dB within the entire tested noise range).
- The reconstruction performance of the SSREF method increases as the ensemble size increases.
- The SSREF method produces significantly higher SNR under different compression rates (up to 11 dB) when compared to existing CFM systems.
- The SSREF method produces fluorescence microscopy images from real noisy measurements with noticeably better image detail when compared to existing systems.
5. Conclusions and future work
In this paper, we propose a robust saliency-guided sparse reconstruction ensemble fusion (SSREF) design paradigm for improving the SNR of compressive fluorescence microscopy systems. The SSREF model adaptively optimizes the sampling probability to the regions of interest rather the common uniform sampling used in other approaches and also employs fusion to improve SNR. The performance of the proposed model was demonstrated for fluorescence microscopy data acquisition with synthetic noise levels as well as real noisy data. Sparse sampling might decrease optical sectioning capability therefore can affect sample thickness [18,19]. Several strategies are proposed in literature mitigating degradation of florescence microscopy image reconstruction from sparse sampling . It would be very interesting to research in the future the effects of sparse sampling and optical sectioning capabilities when the proposed method will be implemented in hardware This proposed work offers a framework that set the foundation for future research as well as core technology for improved and robust acquisition design. The investigation of different probability density functions and their effect on reconstruction performance would be an interesting future research. In addition it would be interesting to research different operators for detecting regions of high saliency.
This research has been sponsored by the Natural Sciences and Engineering Research Council of Canada (NSERC) and Geomatics for Informed Decisions (GEOIDE). The authors would also like to thank Yeast Resource Center - University of Washington in Seattle for the test fluorescence microscopy data.
References and links
2. J. B. Pawley and B. R. Masters, “Handbook of biological confocal microscopy, third edition,” J. Bio-Med. Opt. 13(029902), (2008).
3. S. Inoue and K. R. Spring, Video MicroscopyNew York: Plenum Press13, (1997). [CrossRef]
4. J. Zakrzewski, “Integrating a spectrometer with an optical microscope presents challenges,” SPIE Magazine pp. 29 (2003).
5. R. Connally, D. Veal, and J. Piper, “High resolution detection of fluorescently labeled microorganisms in environmental samples using time-resolved fluorescence microscopy,” FEMS Microbiol Ecol 41, 239–245 (2002). [CrossRef] [PubMed]
6. R. Connally, D. Veal, and J. Piper, “Novel flashlamp based timeresolved fluorescence microscope reduces autofluorescence for 30-fold contrast enhancement in environmental samples,” Proc. SPIE 4964, 14–23 (2003). [CrossRef]
7. R. Connally, D. Veal, and J. Piper, “Flash lamp-excited time-resolved fluorescence microscope suppresses autofluorescence in water concentrates to deliver an 11-fold increase in signal-to-noise ratio,” J. Biomed. Opt. 9, 725–734 (2004). [CrossRef] [PubMed]
11. N. Panchuk-Voloshina, R. P. Haugland, J. Bishop-Stewart, M. K. Bhalgat, P. J. Millard, F. Mao, W. Leung, and R. P. Haugland, “Alexa dyes, a series of new fluorescent dyes that yield exceptionally bright, photostable conjugates,” J. Histochem. Cytochem. 47, 1179–1188 (1999). [CrossRef] [PubMed]
12. B. R. Renikuntla, H. C. Rose, J. Eldol, A. S. Waggoner, and B. A. Armitage, “Improved photostability and fluorescence properties through polyfluorination of a cyanine dye,” Organic. Lett. 6(6), 909–912 (2004). [CrossRef]
13. W. C. Moss, S. Haase, J. M. Lyle, D. A. Agard, and J. W. Sedat, “A novel 3d wavelet-based filter for visualizing features in noisy biological data,” J. Microscopy 219, 43–49 (2005). [CrossRef]
14. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,” IEEE Trans. Pattern. Anal. Mach. Intell. 12, 629–639 (1990). [CrossRef]
15. S. Sabri, F. Richelme, A. Pierres, A. Benoliel, and P. Bongrand, “Interest of image processing in cell biology and immunology,” J. Immunol. Methods. 208, 1–27 (1997). [CrossRef]
16. M. Marim and E. Angelini, “Denoising in fluorescence microscopy using compressed sensing with multiple reconstructions and non-local merging,” Eng. Med. Biol. (EMBC), 2010 Annual International Conference of the IEEE 3394(7), 3394–3397 (2010). [CrossRef]
17. M. Marim, E. Angelini, and J. C. Olivo-Marin, “Compressed sensing in biological microscopy,” in Proc. SPIE Wavelets XIII 7446, 3394–3397 (2009).
18. V. Studer, J. Bobin, M. Chahid, H. Moussavi, E. J. Candes, and M. Dahan, “Compressive fluorescence microscopy for biological and hyperspectral imaging,” Proceedings of the National Academy of Sciences of the United States of America pp. 10 (2011).
19. Y. Wu, P. Ye, I. O. Mirza, G. R. Arce, and D. W. Prather, “Experimental demonstration of an optical-sectioning compressive sensing microscope (csm),” Opt. Express 18, 24565–24578 (2010). [CrossRef] [PubMed]
20. R. Baraniuk, M. Davenport, R. DeVore, and M. Wakin, “A simple proof of the restricted isometry property for random matrices,” Const. App. 28(3), 253–263 (2008). [CrossRef]
21. E. Candes and J. Romberg, “Quantitative robust uncertainty principles and optimally sparse decompositions,” Found. Comput. Math. 6(2), 227–254 (2006). [CrossRef]
22. E. Candes, J. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Commun. Pure Appl. Math. 59(8), 1207–1221 (2006). [CrossRef]
23. D. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory. 52(4), 1289–1306 (2006). [CrossRef]
24. J. Romberg, E. Candes, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inform. Theory. 52(2), 489–509 (2006). [CrossRef]
25. S. Schwartz, A. Wong, and D. A. Clausi, “Saliency-guided compressive fluorescence microscopy” In 34th Int. Conf. Eng. Med. Biol. (EMBC 2012) (to be published).
26. S. Schwartz, A. Wong, and D. A. Clausi, “Saliency-guided compressive sensing approach to efficient laser range measurement,” J. Visual Commun. Image Represent. (DOI:http://dx..org/10.1016/j.jvcir.2012.02.002), (2012). [CrossRef]
27. E. J. Candes, “Restricted isometry property and its implications for compressed sensing,” Comptes rendus - Mathematique 346(9–10), 589–592 (2008).
28. R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned salient region detection,” IEEE International Conference on Computer Vision and Pattern Recognitio pp. 1597–1604 (2009). [CrossRef]
29. A. Beck and M. Teboulle, “Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems,” IEEE Trans. Imag. Proc. 18(11), 2419–2434 (2009). [CrossRef]
30. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imag. Sci. 1, 183–202 (2009). [CrossRef]
31. M. Riffle and T. N. Davis, “The Yeast Resource Center Public Image Repository: A large database of fluorescence microscopy images,” http://images.yeastrc.org/imagerepo/searchImageRepoInit.do.