Abstract

Studying dynamic biological processes, such as heart development and function in zebrafish embryos, often relies on multi-channel fluorescence labeling to distinguish multiple anatomical features, yet also demands high frame rates to capture rapid cell motions. Although a recently proposed method for imaging dynamic samples in transmission or reflection allows to conveniently switch between color imaging or boosting the frame rate by use of spectrally-encoded, temporally-modulated illumination sequences and a hue-encoded shutter (hue-encode shutter method, HESM), the technique is not applicable directly in fluorescence microscopy, where the emitted light spectrum is mostly independent of the excitation wavelength. In this paper, we extend HESM by using samples labeled with multiple fluorophores, whose emission signal can either be used to distinguish multiple anatomical features when imaged in multi-channel mode or, if the fluorophores are co-localized in a dynamic tissue, to increase the frame rate via HESM. We detail the necessary steps to implement this method in a two-color light-sheet microscope to image the beating heart of a zebrafish embryo. Specifically, we propose an adapted laser modulation scheme for illumination, we identify caveats in choosing a suitable multi-color fluorophore labeling strategy, and derive an ℓ1-regularized reconstruction technique that is sufficiently robust to handle the low signal-to-noise ratio and labeling inhomogeneities in the fluorescence images at hand. Using the case of a beating heart in a zebrafish embryo, we experimentally show an increase in the frame rate by a factor two while preserving the ability to image static features labeled in distinct channels, thereby demonstrating the applicability of HESM to fluorescence. With a suitable illumination setup and fluorescent labeling, the method could generalize to other applications where flexibility between multiple channel and high-speed fluorescence imaging is desirable. For fluorophores that are not co-localized, the imaging system is similar to a conventional light sheet microscope.

© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

1. Introduction

Fluorescence microscopy is a tool used in many biological studies. It is possible to have fluorescence light emitted by a sample from specific biological features such as nuclei, membranes, or selected molecules [1]. In addition to the need for distinctly labeling anatomical, cellular, or subcellular features, many live biological processes are highly dynamic and a high temporal resolution in imaging is required to study them [2]. Although a recently proposed method for imaging dynamic sample in transmission or reflection allows to conveniently switch between color imaging or boosting the frame rate by use of a spectrally encoded illumination and a hue-encoded shutter method (HESM), the technique is not applicable directly in fluorescence microscopy, where the emitted light spectrum is mostly independent of the excitation wavelength.

In this paper, we propose to extend the hue-encoded shutter method presented in [3] to fluorescence microscopy and to apply it to light-sheet microscopy [4]. The method in [3] increases the flexibility of microscopes by allowing to rapidly switch between color imaging and fast imaging, or to do both simultaneously, within regions of interest. Briefly, that method uses a multi-color LED illumination source that is synchronized to a color camera. The scene (which is assumed to be monochromatic) is illuminated with a temporally-varying hue at a sub-frame rate frequency. The individual color frames are then processed to convert the color channels into multiple (monochrome) time frames. The monochrome requirement of [3] cannot be translated directly to fluorescence microscopy, as the spectral signature of the illumination is lost in the excitation-emission process at the heart of fluorescence. To overcome this limitation, we propose to use multiple co-localized fluorophore species with different emission spectra. The overview of the proposed method is presented in Fig. 1.

 figure: Fig. 1.

Fig. 1. Hue-encoded shutter for fluorescence light-sheet microscopy. (a) The excitation objective (EO) transmits consecutive pulses of different wavelengths within the exposure time $E$ of one frame. The sample ($\boldsymbol{x}$) contains co-localized fluorophores (e.g. green and red fluorescent proteins) that respond to the excitation by emitting fluorescent light. The sample may also contain static regions that can be labeled with a single fluorophore type. The color sensor of the camera (CAM) captures the light emitted over the entire exposure time through the detection objective (DO). (b) Acquired (raw) multi-channel images (c) Our proposed method converts the color frames to multiple monochrome frames at a higher frame rate (here, by a factor 2) in a region of interest that contains dynamic features. The color information can be preserved in static regions outside of the region of interest.

Download Full Size | PPT Slide | PDF

Improving temporal or spatial resolution of microscopes has been tackled by many different approaches. We briefly review some that are of particular interest or use related concepts.

Several temporal super-resolution methods directly capitalize on strong prior knowledge of the signal itself, for example that sample motion is repetitive [5,6] or that the signal is sparse in some basis [79]. Other approaches do not require any prior knowledge on the signal. Bub et al. [10] used a modified camera with individually addressable pixels to stagger exposure times. This method fits well for fluorescence microscopy, but requires a modified camera, which may not be widely available. Wagnert et al. [11] presented a method to instantaneously acquire full volumes of isotropic resolution thanks to two light-field imagers. While limited to small fields of view and having a lower spatial resolution than light-sheet microscopy, their method shows great potential as whole volumes can be acquired at camera frame rate, despite requiring a dedicated setup.

Our method takes advantage of temporally patterned illumination. With illumination long known to be an essential factor to a microscope’s performance (as shown by Köhler [12]), the field of structured illumination microscopy has demonstrated the possibility of improving spatial resolution beyond the limits of the acquisition device [1318]. These methods mostly target spatial resolution improvements, often at the cost of temporal resolution, since they require multiple modulated images be merged into a higher spatial resolution image. Methods that specifically seek to improve temporal resolution by taking advantage of fast modulable illumination sources include the coded illumination method by Gorthi et al. [19], who perform motion deblurring for fluorescence microscopy using the fluttered shutter principle [20]. This approach is limited to linear motion deblurring. Yet other approaches, such as that of Dillavou et al. in [21] assume that the imaged sample is binary (either black or white) and use the dynamical range of a camera to drastically increase its frame rate. This strong assumption on the signal might, however, limit applicability to fluorescence microscopy.

Our method allows us to do either color imaging or fast imaging in fluorescence microscopy without the limitations of the aforementioned methods as it does not require multiple or other custom modified cameras, neither do we require the signal to be sparse, repetitive, binary, or following any particular (linear) motion.

The main contributions of the present paper are (i) the adaptation of the method in [3] to fluorescence microscopy; (ii) an improved temporal super-resolution reconstruction algorithm to compensate for slight model mismatches; (iii) demonstration of the applicability to light-sheet microscopy for biological applications.

The paper is organized as follows. In Section 2, we present the signal and imaging models and detail the assumptions on the image acquisition and the signal and derive the algorithm to perform temporal super-resolution. In Section 3, we demonstrate our method in practice by performing frame-rate doubling in light-sheet microscopy of the beating heart of a double-labeled, 2 days post-fertilization old zebrafish embryo. We finally discuss our experiments and conclude in Section 4.

2. Methods

Before we delve into the details of our proposed method, we briefly recall assumptions of the HESM method developed in [3] and the limitations that prevent it from being directly applicable to fluorescence microscopy. In [3], the light captured by the camera (Fig. 2(A, B)) was assumed to be either reflected by the imaged scene or transmitted through a sample. This implied that the light spectrum measured by the camera was that of the illumination light source modulated by the spectral sample response. To follow a monochromatic object with higher temporal resolution, temporal variations could be encoded spectrally in the illumination pattern by using light sources of different hues. In the case of fluorescence microscopy with a single type of fluorophore (Fig. 2(C, D)), while the emitted light (which results from the fluorophores’ relaxation following their excitation by the incoming light) has an overall intensity that depends on the excitation light via the molecules’ spectral absorption properties, its spectrum only depends on the emission properties of the fluorophore but not on the spectrum of the excitation. Hence, the spectrum captured by the camera does no longer depend on the light source used, which precludes the possibility of using HESM in a way similar to the brightfield case. Our proposed way to extend HESM to fluorescence involves the use of multiple co-localized fluorophores (Fig. 2(E, F)). Although each fluorescent species’ individual emission spectrum is independent of the illumination spectrum, the fluorophores’ combined emission spectrum depends on the spectral overlap of the illumination with the absorption curve of each fluorophore. An added challenge for the fluorescence case is that fluorophores typically have broad and overlapping emission spectra, which makes it more difficult to isolate their contributions in distinct RGB channels as could be done with narrow-band illuminations in the case of transmission HESM. This overlap, therefore, requires an unmixing step.

 figure: Fig. 2.

Fig. 2. Fluo-HESM requires co-localized multi-color fluorescent labeling. (Top row) HESM [3] assumes a bright field microscopy setup, where the illumination light is transmitted through (or reflected by) a sample then captured by an RGB camera. The transmitted (or reflected) spectrum directly depends on the combination of the illumination spectrum and the sample’s spectral transmittance. HESM relies on using a temporally-variable spectrum in the illumination, which can be decoded following capture with an RGB camera. (Middle row) If using a single fluorophore and two different illumination sources (C and D), the emission from the sample only changes in overall intensity, but the shape of the emission spectrum remains unchanged. Hence, temporal modulation of the illumination spectrum would not univocally modulate the spectral shape of the emission and HESM could therefore not be used as-is to improve the frame-rate. (Bottom row) In this paper, we propose to use two light sources E and F and a sample labeled with two co-localized fluorophore species such that the combined emitted light spectrum is directly dependent on temporal modulation patterns of the illumination spectrum, making the use of HESM to improve the frame rate possible.

Download Full Size | PPT Slide | PDF

2.1 Imaging model

The fluorescence imaging setup we consider is made of a color camera with $C$ channels and $L$ lasers of different wavelengths (Fig. 1). We further consider a dynamic sample $\boldsymbol{x}(t)$ labeled with $F$ different, co-localized fluorophore species. We write the intensity measured in the $c^{\textrm {th}}$ channel of a single pixel, during the $k^{\textrm {th}}$ acquisition at time $t_k$, as:

$$y_{c,k} = \int_{t_k}^{t_k + E} \sum_{\ell =1}^{L} \sum_{f=1}^{F} \gamma_{c,f}b_{f,\ell} s_\ell (t) x(t) \textrm{d}{t} + d_c,$$
where $E$ is the exposure time of the camera, $s_\ell (t)$ the time-varying intensity function of the $\ell ^{\textrm {th}}$ laser, $b_{f,\ell }$ is the absorption cross section of the $f^{\textrm {th}}$ fluorophore to the $\ell ^{\textrm {th}}$ laser, $\gamma _{c, f}$ is the sensitivity of the $c^{\textrm {th}}$ color channel to the $f^{\textrm {th}}$ fluorophore emission spectrum and $d_c$ is an offset from the sensor’s electronics. Similarly to the model in [3], we assume that the temporal signal of interest lies in the space spanned by shifted B-splines of degree 0 , i.e.:
$$x(t) = \sum_{i\in\mathbb{Z}} x[i]\beta^{0}\big((t-i)Q\big),$$
where $Q$ is the super-resolution factor (i.e. $x(t)$ takes $Q$ different values over the time of an exposure time $E$) and $\beta ^{0}(t)$ is the indicator function over the interval $[-\frac {1}{2}, \frac {1}{2}]$. We can rewrite Eq. (1) as:
$$\begin{aligned} y_{c,k} &= d_c + \sum_{\ell =1}^{L} \sum_{f=1}^{F} \gamma_{c,f} b_{f,\ell} \int_{t_k}^{t_k + E} s_\ell (t) \sum_{i=0}^{Q-1} x[t_{k,i}]\beta^{0}\left(\left(t- \frac{E}{Q}i\right)Q\right) \textrm{d}{t}\\ &= d_c + \sum_{\ell =1}^{L} \sum_{f=1}^{F} \gamma_{c,f} b_{f,\ell} \sum_{i=0}^{Q-1} x[t_{k,i} ] \underbrace{\int_{t_k + \frac{E}{Q}i}^{t_k + \frac{E}{Q}(i+1)} s_\ell (t) \textrm{d}{t}}_{s_\ell [t_{k,i}]}, \end{aligned}$$
where $t_{k,i}$ is the index for the values of $x[\cdot ]$ and $s_\ell [\cdot ]$ at time $t_k + \frac {E}{Q} i$. We adopt a matrix notation to write Eq. (3) for the full system:
$$\boldsymbol{y}_k = \boldsymbol{S \Gamma x}_k + \boldsymbol{d}_0,$$
where $\boldsymbol{\Gamma } \in \mathbb {R}^{LCQ \times Q}$ expresses both the response of the fluorophores to the excitation light ($b_{f,\ell }$) and the sensitivity of the camera’s sensor to the emitted fluorescent light ($\gamma _{c,f}$), $\boldsymbol{S} \in \mathbb {R}^{C \times LCQ}$ is the matrix built by computing the integral in Eq. (3) and by shifted zero-padded vectors of time-functions for each laser, $\boldsymbol{x}_k = \left [x[t_{k,0}], x[t_{k,1}],\ldots , x[t_{k,Q-1}] \right ]^{\top }$ is the unknown vector of $Q$ values of $x(t)$ during $E$, and $\boldsymbol{d}_0 \in \mathbb {R}^{C \times 1}$ is the offset vector [22].

As a concrete example (which is the case we will consider in our experiments) we set $C=3$, $L=2$, and $Q=2$. We choose to set the illumination temporal functions of the $L=2$ lights as $s_1 = [1, 0]$ and $s_2 = [0, 1]$. That is, each light source will be switched on for half of the exposure time, consecutively. In this setting, the matrix $\boldsymbol{S}$ is written as $\boldsymbol{S} = \begin {pmatrix} \boldsymbol{S}^{1} & \boldsymbol{S}^{2} \end {pmatrix}$ and (see [3] for further details):

osac-3-8-2195-e001
is the matrix expressing the first light source’s temporal activation pattern. Similarly,
osac-3-8-2195-e002
is the matrix expressing the second light source’s temporal activation pattern. We recall that $\boldsymbol{S}_1, \boldsymbol{S}_2 \in \mathbb {R}^{C \times C Q = 3 \times 6}$ and $\boldsymbol{S} \in \mathbb {R}^{C\times C Q L = 3\times 12}$. Our example further leads to $\boldsymbol{\Gamma } = \begin {pmatrix} \boldsymbol{(\Gamma }^{1})^{\top } & (\boldsymbol{\Gamma }^{2})^{\top } \end {pmatrix}^{\top }$ with $\boldsymbol{\Gamma }^{\ell }$, $\ell =1,2$, the spectral mixing matrix for the $\ell ^{\textrm {th}}$ light source:
$$\boldsymbol{\Gamma}^{\ell} = \begin{pmatrix} (\gamma_{1,1}b_{1,\ell} + \gamma_{1,2} b_{2, \ell}) \boldsymbol{I}_{2} \\ (\gamma_{2,1}b_{1,\ell} + \gamma_{2,2} b_{2, \ell}) \boldsymbol{I}_{2} \\ (\gamma_{3,1}b_{1,\ell} + \gamma_{3,2} b_{2, \ell}) \boldsymbol{I}_{2} \end{pmatrix},$$
where $\boldsymbol{I}_2$ is the identity matrix of size $2\times 2$. We recall that $\gamma _{c,f}$ is the spectral impact of the $f^{\textrm {th}}$ fluorophore on the $c^{\textrm {th}}$ channel, and that $b_{f,\ell }$ is the response of the $f^{\textrm {th}}$ fluorophore to the $\ell ^{\textrm {th}}$ light. Thus, $(\gamma _{2,1}b_{1,1} + \gamma _{2,2} b_{2, 1})$ is the compound contributions from both fluorophores reacting to the first illumination source as captured by the camera’s second channel. Figure 3 further illustrates these quantities and displays the values corresponding to the experimental setting in Section 3.

 figure: Fig. 3.

Fig. 3. Forward matrix $\boldsymbol{S\Gamma }$ combines temporal illuminations patterns of two lasers and the combined absorption and emission responses of the two fluorophores, together with the spectral response of the camera. Notice that the left part of the matrix $\boldsymbol{S}$, boxed in green, contains repetitions of the temporal function of the green laser: $s_1[i]$ and the right part of $\boldsymbol{S}$ contains the same information for the blue laser. We also framed in green, respectively blue, the part that corresponds to the green, respectively blue, laser in the $\boldsymbol{\Gamma }$ matrix which expresses the efficiency of each fluorophore to the lasers combined with the sensitivity of the camera to the fluorescence emission spectra.

Download Full Size | PPT Slide | PDF

2.2 Temporal super-resolution from hue encoded signal

In [3] we performed temporal super-resolution by solving

$$\boldsymbol{x}_k^{\star} = {\mathop{\hbox{arg min}}_{\boldsymbol{x}_k}} \left\lVert{ \boldsymbol{y}_k - \boldsymbol{d}_0 - \boldsymbol{S\Gamma x}_k}\right\rVert_2^{2}$$
for every acquired frame, where $\boldsymbol{d}_0$ and $\boldsymbol{\Gamma }$ were obtained via calibration. We solved Eq. (8) for each pixel and each acquired frame independently, without any dependency between consecutive frames. While this approach performs well as long as the hue is homogeneous over the imaged scene, we observed that fluorophore labeling can be inhomogeneous, as illustrated in Fig. 5.

To make the method in [3] more robust to model mismatches, we propose to add a regularization term over time which introduces a dependency over consecutive frames (temporal dependency). Given the model presented in Eq. (4), the vector $\boldsymbol{d}_0$ and matrix $\boldsymbol{\Gamma }$ obtained through calibration [3] and a sequence of $M$ acquired color pixels $\boldsymbol{y} = \left [ \boldsymbol{y}_0^{\top }, \boldsymbol{y}_1^{\top }, \cdots , \boldsymbol{y}_{M-1}^{\top } \right ]^{\top } \in \mathbb {R}^{M C\times 1}$, we want to reconstruct the temporal high-resolution signal $\boldsymbol{x} = \left [ \boldsymbol{x}_0^{\top }, \boldsymbol{x}_1^{\top }, \cdots , \boldsymbol{x}_{M-1}^{\top } \right ]^{\top } \in \mathbb {R}^{M Q\times 1}$ with a temporal super-resolution factor of $Q$. We propose to solve

$$\boldsymbol{x^{\star}} = {\mathop{\hbox{arg min}}_{x}} \frac{1}{2}\left\lVert{ \boldsymbol{y} - \boldsymbol{d} - \boldsymbol{A}\boldsymbol{x}}\right\rVert_2^{2} + \lambda \left\lVert{\boldsymbol{Tx}}\right\rVert_1,$$
where $\boldsymbol{T} \in \mathbb {R}^{MQ\times MQ}$ is a first-order derivative matrix, $\lambda \in \mathbb {R}_+$ is a free parameter and $\boldsymbol{d} \in \mathbb {R}^{MC\times 1}$ is $\boldsymbol{d}_0$ stacked $M$ times. The matrix $\boldsymbol{A} \in \mathbb {R}^{MC \times MQ}$ is a block-diagonal matrix built by repeating $\boldsymbol{S\Gamma }$ along the diagonal $M$ times. The regularization is known as Total Variation [24] and promotes piecewise-constant solutions, which is natural in time series of images as it preserves sharp edges. Equation 9 has the generalized LASSO form [25] and can be solved with the Alternating Direction Method of Multipliers (ADMM) [23,26], an efficient choice among other possible methods [25,27,28]. The ADMM considers a problem equivalent to that in Eq. (9),
$$\begin{array}{cc} \textrm{minimize } & \frac{1}{2}\left\lVert{ \boldsymbol{y} - \boldsymbol{d} - \boldsymbol{A}\boldsymbol{x}}\right\rVert_2^{2} + \lambda \left\lVert{\boldsymbol{u}}\right\rVert_1 \\ \textrm{such that } & \boldsymbol{Tx} = \boldsymbol{u} \end{array},$$
and builds the augmented Lagrangian defined as
$$\begin{aligned} \mathcal{L}(\boldsymbol{x}, \boldsymbol{u}, \boldsymbol{y}) = &\left\lVert{ \boldsymbol{y} - \boldsymbol{Ax} - \boldsymbol{d}}\right\rVert^{2}_2 + \lambda \left\lVert{\boldsymbol{u}}\right\rVert_1\\ &+ \boldsymbol{z}^{\top} (\boldsymbol{Tx} - \boldsymbol{u}) + (\rho/2) \left\lVert{\boldsymbol{Tx} -\boldsymbol{u}}\right\rVert^{2}_2, \end{aligned}$$
where $\rho > 0$ is a penalty parameter, $\boldsymbol{u}$ is a primal variable (as well as $\boldsymbol{x}$) and $\boldsymbol{z}$ is a dual variable, also called Lagrangian multiplier. The ADMM algorithm consists of iterative steps in which the gradient of the augmented Lagrangian is set to zero for the primal variables [23]. At each iteration $k$, we alternate between solving a least-squares problem with fixed regularization weights and updating the regularization parameters (Lagrange multipliers). The global regularization weight $\lambda$, which balances the relative importance of the data and regularisation terms, is fixed during the entire process (we set its value empirically but it could be set using an L-curve method [29,30]). The iterations stop when a maximal number of iterations $k_{\textrm {max}}$ has been reached or when the difference between consecutive solutions for $\boldsymbol{x}^{k}$ is lower than a given value $\epsilon \in \mathbb {R}_{+}$. Algorithm 1 provides the corresponding pseudo-code in our case.

3. Experiments

We used our implementation of a light sheet microscope (based on OpenSPIM [31]), with two lasers (Stradus, Vortran Laser Technology) of wavelengths 488nm and 561nm to generate the excitation illumination light sheet. A four-dimensional stage (USB 4D-STAGE, Picard) held the sample. The detection axis consisted of a 20$\times$/0.5 water immersion microscopy lens (UMPLFLN 20$\times$W, Olympus) coupled to a 180 mm tube-lens (U-TLU-1-2, Olympus), terminated by a 1.2 mega pixels CMOS color camera equipped with an RGGB Bayer filter (DCC3240C, Thorlabs Instruments). The lasers and camera were electronically synchronized using a micro-controller (Arduino Due, Arduino, Italy). The micro-controller monitored the flash output of the camera and controlled the lasers using their ON/OFF electrical input, similarly to the electronics scheme provided in [32].

We conducted experiments on more than one fish sample, and there were differences in fluorescent proteins expression between different samples. In order to have acceptable SNR in the acquired images, we adjusted the exposure time of the camera, hence different frame-rates in the following experiments.

3.1 Temporal super-resolution light-sheet imaging of the beating heart of a zebrafish

To illustrate the potential of our method, we tested whether it was applicable to imaging the beating heart of a zebrafish embryo. To that end, we chose zebrafish that co-express ubiquitous cytoplasmic green fluorescent protein, EGFP, and red fluorescent protein, mCherry [33] (two co-localized fluorophores, $F=2$). We embedded the embryos on a glass bottom dish in 1% low melting agarose and acquired 30 frames ($M=30$) with our RGB camera ($C=3$); setting the super-resolution factor to $Q=2$, the two lasers ($L=2$) were controlled so that they were each switched on for half of the exposure time (consecutively), so that we had $s_1[i] = [1, 0]$ and $s_2[i] = [0, 1]$, $i \in \{0, 1\}$. Figure 4 shows two consecutively acquired frames (Fig. 4(a),(e)), all color channels of the acquisitions (Fig. 4(b)-(d), (g)-(h)), and the corresponding four reconstructed frames (Fig. 4(i)-(l)). Note that, as discussed in Fig. 3, the blue layer (Fig. 4(d),(h)) contains very little energy. Our method was therefore capable of doubling the frame rate. Here, we had $F=2$, which limited the frame rate increase to $Q=2$, but with additional fluorophore species and acquisition channels, higher $Q$ could be considered.

 figure: Fig. 4.

Fig. 4. Fluo-HESM allows simultaneous fast imaging of the beating heart and color imaging of its surroundings. (a),(e) two consecutive acquired frames. (b)-(d), respectively (f)-(h) are the red, green and blue layers of the image in (a), respectively (e). (i)-(l) reconstructed frames using Algorithm 1 with $\lambda =0.001$. The super-resolution factor is 2, hence for two acquired frames, we observe 4 reconstructed frames. Note that there are 60 milliseconds between the two acquired frames (a) and (e) and 30 milliseconds between the super-resolved region in frames (i)-(l), framed in a white rectangle. Outside of this region, our method allows keeping the color information. On (j), we marked two regions with drastically different colors, green (osac-3-8-2195-i001) and orange ($\clubsuit$). Standard imaging, in the presence of a single fluorophore species, would only produce frames (c) and (g), respectively (b) and (f) using samples labeled with only green fluorescent proteins, respectively only red fluorescent proteins. With Fluo-HESM, there are twice the number of these frames. This is also visible in Visualization 4, which shows a comparison between standard light-sheet data and reconstructions from our method in 3D+time. To highlight the link between this data and the schematic of Fig. 1, (a) and (e) on this figure correspond to Fig. 1(b) and (i)-(l) to Fig. 1(c). See Visualization 1 and Visualization 2 for the full movie. Fish orientation indicated: anterior (A), posterior (P), right (R) and left (L). Scale bar: 100 $\mu$m.

Download Full Size | PPT Slide | PDF

The matrix $\boldsymbol{\Gamma }$ and the vector $\boldsymbol{d}$ were obtained with the calibration procedure in [3], using the code available online [32]. The area used for calibration is shown in Fig. 5(d). Figure 3 illustrates how the matrix $\boldsymbol{A} = \boldsymbol{S\Gamma }$ is built. We observe that the green laser has a high impact (first value, 0.7) on the red channel of the camera, which is due to the green laser mostly exciting the red fluorophores and the ensuing red fluorescent light will mostly be captured on the red channel of the camera. Similarly, the blue laser has a high impact on the green channel of the camera (value of 0.8), as it mostly excites the green fluorophores. Finally, note that none of the lasers really impact the blue channel, since there are no blue fluorophores. The values displayed on Fig. 3 are the ones obtained after calibration and were used for the following experiments.

 figure: Fig. 5.

Fig. 5. A static region is required to calibrate our method. The region must also have a similar red over green (R/G) ratio than in the heart. (a) Hue-encoded light-sheet image of a 2 days post-fertilization zebrafish heart. The ventricle (V), the atrium (At) and the pericardium (p) are outlined. (b) The ratio R/G of each pixel on the image in (a). Notice that the ratio R/G is not homogeneous over the whole image. In order to calibrate our method, we need a static region with the same ratio R/G as in the heart; it is shown in (d). (d’) shows the histogram of values within (d). Similarly, (c’) and (e’) show histograms of their corresponding regions (c) and (e). Notice that the calibration region (d) has a histogram similar to that of the heart (c), with a maximum centered around 1.5. Since Histogram (e’) is different from Histograms (c’) and (d’), our method would require an additional calibration to be usable in Area (e). Fish orientation indicated: anterior (A), posterior (P), right (R) and left (L). Scale bar: 100$\mu$m.

Download Full Size | PPT Slide | PDF

3.2 Homogeneity of the fluorescent co-expression ratio

Our method requires that the imaged sample shows consistent fluorescent co-expression. We set out to make sure that the (static) calibration area and the (dynamic) heart area have indeed homogeneous and matching green and red fluorescent protein expression levels. To that end, we computed the red over green ratio (R/G) for each pixel in the images we acquired to compute the calibration, displayed in Fig. 5(b). We then computed the R/G histograms over various areas to compare them (Fig. 5(c’), (d’) and (e’)).

Figure 5 shows an acquired color image where the lasers were temporally controlled as explained in Fig. 3. The ventricle (V), atrium (At) and pericardium (p) are outlined. In the pericardium in Fig. 5(a), some patches appear green, while the heart appears orange. This is better seen in Fig. 5(b), comparing (c), (d) and (e) and their corresponding histograms on (c’), (d’) and (e’).

Inhomogeneities in the co-expression of red and green fluorescent proteins are visible over the sample (for example Fig. 5(e) and (e’) compared to Fig. 5(d) and (d’)). Given this expression variability (which is common from individual to individual or within the sample), it is thus important to choose a motionless calibration area (Fig. 5(d) and (d’)) that exhibits similar R/G ratio than that of the dynamic phenomenon of interest, like the beating heart (Fig. 5(c) and (c’)). Even with a careful choice, slight differences in the histogram remain (compare (c’) and (d’)). This justifies the use of an algorithm that is sufficiently robust to mismatches than a plain least-squares method without regularization. We further investigated this aspect in Section 3.4.

3.3 Combined fast and color imaging on the beating heart of a zebrafish

We applied Algorithm 1 only in the region comprising the beating heart. In this way, we could maintain the color information in static areas and do fast imaging where needed. Since we require the imaged sample (in this case, the heart) to be homogeneously labeled for temporal super-resolution, after Algorithm 1 we can virtually assign the (single) hue back to the reconstructed images. Figure 6 shows two frames where we have temporal super-resolution reconstructed frames only in a region of interest (Fig. 6(c)) and where the acquired hue is preserved in static areas (Fig. 6(a) and (b)).

 figure: Fig. 6.

Fig. 6. Fluo-HESM is robust to fluorescence labeling inhomogeneities thanks to the temporal $\ell$1 regularization. (a)-(b) Composite color images and temporal super-resolution reconstructions. The dynamic area of the scene (c), with the beating heart, has been manually selected and inside of (c), the frame rate is twice that of outside. This illustrates the ability to choose between fast single hue imaging and slow color imaging. (d) Reconstructions at the location shown on (c) with a heart symbol (osac-3-8-2195-i001) using least-squares (without regularization, LSTSQ) or ADMM. The TV regularization we used in Eq. (9) favors piecewise constant reconstructions and preserves sharp edges. (e) Reconstructions at the location shown on (c) with a club symbol ($\clubsuit$) using LSTSQ or ADMM. At this location, there is little motion of the sample, and the LSTSQ reconstructions oscillate a lot, while the ADMM reconstructions show little motion, justifying the use of ADMM over LSTSQ. Within (c), we assigned a hue to the reconstructed images, that we measured at calibration. Scale bar: 100$\mu$m. See Visualization 2 and additional Visualization 3 on a different sample.

Download Full Size | PPT Slide | PDF

3.4 ADMM compared to least-squares without regularization

As observed in Section 3.2 and Fig. 5, there can be inhomogeneities in the co-expression ratios of the fluorescent proteins over the sample. We investigated whether the algorithm we introduced in Section 2 (ADMM) is more robust to these variations than the least-squares method (LSTSQ) without regularization of [3]. To that end, we compared reconstructions obtained using either ADMM or LSTSQ with the same input data. Figure 6(d) shows a comparison of both reconstructions at the same location, where the heart beats. The cardiac period is visible in both plots yet the ADMM reconstruction is smoother with sharp transitions maintained, unlike the LSTSQ reconstruction. This behavior is to be expected with the Total-Variation regularization. In Fig. 6(e), we compared ADMM and LSTSQ reconstruction at a location where the imaged sample is mostly static. There the reconstructions is expected to be nearly constant, which is indeed the case for the ADMM reconstruction, while the LSTSQ reconstructions oscillates, most likely because of a slight image model mismatch.

For the ADMM reconstruction, the processing time for 45 time-points on a single pixel (i.e. 90 reconstructed time-points) was $\approx 1.7$ milliseconds on a MacBook Pro 2018 (2.9 GHz 6-Core Intel Core i9, 32 Gb RAM). The code is in standard Python 3. We believe that with GPU programming, separating the processing of each individual pixel, and some tweaking, the code could run in real-time.

3.5 Impact on the reconstruction of 3D volumes of the beating heart

Light-sheet microscopy is a well-established tool for developmental studies thanks to its high spatio-temporal resolution and its low level of induced optical damage to the sample [34]. Furthermore, it has good sectioning capabilities, which we took advantage of to produce a dynamic 3D+time volume series (XYZ + time). We acquired movies of optical sections at different depths within the sample, applied our temporal super-resolution method to each individual movie, then temporally registered consecutive (in z) movies for the heartbeat to be synchronous in all slices. For the temporal registration, we used a dynamic programming method [35]. The method finds the best warping function of a temporal series onto another one by maximizing the sum of normalized mutual information between all frames of both time-series, which has already been shown to be well-suited for cardiac images [3638]. Visualization 4 shows a 3D+time rendering that highlights the raw data where motion is encoded within the color channels and the doubled frame rate reconstruction.

3.6 Roadmap to higher super-resolution factors

In this paper, we presented a method to double the frame-rate of imaging in fluorescence microscopy. According to our model, the temporal super-resolution factor is determined by the rank and conditioning number of matrix $\boldsymbol{A}$ in Eq. (9). The rank is upper-bounded by both $C$, $F$, and $L$ (respectively the number of color channels, the number of co-localized fluorophores and the number of lasers at our disposal), so to have a super-resolution factor of, say, $Q =3$, we would need at least three color channels, three co-localized fluorophores and three lasers.

Also, the conditioning number of the matrix $\boldsymbol{A}$ provides a means to predict the quality of the reconstructions, as we have illustrated in [3]. If the chosen fluorophores have too close light emission spectra, this conditioning number will increase, hence lowering the quality of the reconstruction. In [1], Shaner et al. provide insights on how to chose fluorescent proteins and optical filters for maximal spectral separation, which improves the conditioning of our system. A recommended setup of [1] would have so little spectral cross-talk that we could have $Q=3$ with close to a perfect conditioning number, hence good quality reconstructions at three times the frame-rate. This setup recommends using three bandpass detection filters centered at 480 nm (bandpass width 40 nm), 575 nm (bandpass width 25 nm) and 675 nm (bandpass width 130 nm) and the following three co-localized fluorophores: Cerulean (center wavelength of emission 475 nm), mOrange (center wavelength of emission 562 nm) and mPlum (center wavelength of emission 649 nm).

4. Discussion and conclusion

In this paper, we investigated whether a hue-encoded shutter method (HESM) to improve the temporal resolution [3] could be applied to fluorescence microscopy. We showed that despite the physical differences between fluorescence (where the illumination spectrum cannot be inferred from the emission spectrum) and light absorption or reflection properties (which directly modulates the illumination spectrum to produce the transmitted or reflected light), HESM can still be used in fluorescence microscopy, provided one can use temporally modulated illumination, a color camera, and samples labeled with co-localized fluorophores in the dynamic regions where a frame rate boost is desired. We showed imaging of the beating heart of a zebrafish labeled with co-localized red and green fluorescent proteins at twice the acquisition frame rate on a light-sheet microscope, with the ability to preserve the hue-information in static areas. Although we experimentally showed an increase of the frame rate by a factor of 2, our method is general enough that it should allow for higher factors if additional co-localized fluorescent species, lasers, and color channels are available as discussed in Section 3.6.

Our use of a low SNR RGB camera leads to the performance of the current implementation of our method to be below that of the highest frame-rates achievable with dedicated high-sensitivity cameras (100–200 fps) built into some light-sheet microscope implementations, we foresee that the use of alternative imaging components (light splitting and multiple high-speed cameras) could lead to even higher frame rates. Also, alternating between the excitation of multiple populations of fluorophores might be beneficial as it allows for additional recovery time between excitations or lower excitation intensities at each wavelength. However, since this effect will likely depend on the specificities of each use case (in vivo imaging, labeling procedure, choice and proximity of fluorophores) further studies will be required to fully characterize the benefits of using multiple populations of fluorophores over simply increasing the concentration of a single one (when such a concentration increase is biologically feasible).

A key element that we introduced in this paper for the HESM approach to be sufficiently robust in practice, is the temporal regularization, which can cope with sample labeling inhomogeneities and low signal quality inherent to fluorescence imaging, as shown in Fig. 6(d). We further observed that two aspects are particularly important when selecting a fluorescent sample for HESM: (1) the co-localized fluorescent labels should be as homogeneous as possible in the dynamic regions and (2) a static area with a labeling ratio sufficiently similar to the dynamic region is required to calibrate the system. This means that a sample where only moving cells are labeled would not be suitable for our method, as the calibration could not be performed.

In Section 3.3, we illustrated the versatility afforded by our method (either slow color imaging or fast monochrome imaging), by applying it to a region of interest surrounding the beating heart of a zebrafish, while preserving the acquired color images outside of the region. Furthermore, the added possibility of using fluorescence opens the use of HESM to sectioning methods that rely on fluorescence (such as light-sheet microscopy). We illustrated the benefits of a doubled frame rate in the reconstruction of the beating heart in four dimensions (3D + t).

Our method is generic enough that it could potentially speed-up other fluorescence microscopy methods, provided samples with co-localized fluorophores can be generated and multiple lasers can be modulated in time at high-speed, in synchrony with the color camera.

Funding

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (200020_179217, 200021_159227, 206021_164022, 310030E-164245).

Disclosures

The method [3] used in this paper is the subject of a European patent application (EP19154253) which also covers the method presented in this paper.

References

1. N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005). [CrossRef]  

2. J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008). [CrossRef]  

3. C. Jaques, E. Pignat, S. Calinon, and M. Liebling, “Temporal Super-Resolution Microscopy Using a Hue-Encoded Shutter,” Biomed. Opt. Express 10(9), 4727–4741 (2019). [CrossRef]  

4. J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004). [CrossRef]  

5. K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016). [CrossRef]  

6. A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011). [CrossRef]  

7. T. H. Tsai, P. Llull, X. Yuan, L. Carin, and D. J. Brady, “Spectral-temporal compressive imaging,” Opt. Lett. 40(17), 4054–4057 (2015). [CrossRef]  

8. R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. Cossairt, G. Schuster, and A. K. Katsaggelos, “High spatio-temporal resolution video with compressed sensing,” Opt. Express 23(12), 15992 (2015). [CrossRef]  

9. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526 (2013). [CrossRef]  

10. G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010). [CrossRef]  

11. N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019). [CrossRef]  

12. A. Koehler, “Ein neues Beleuchtungsverfahren für mikrophotographische Zwecke,” Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik 10, 433–440 (1893).

13. W. Lukosz, “Optical systems with resolving powers exceeding the classical limit,” J. Opt. Soc. Am. 56(11), 1463–1471 (1966). [CrossRef]  

14. W. Lukosz, “Optical systems with resolving powers exceeding the classical limit. II,” J. Opt. Soc. Am. 57(7), 932–941 (1967). [CrossRef]  

15. P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998). [CrossRef]  

16. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef]  

17. R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy—a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19(8), 1599–1609 (2002). [CrossRef]  

18. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. 102(37), 13081–13086 (2005). [CrossRef]  

19. S. S. Gorthi, D. Schaak, and E. Schonbrun, “Fluorescence imaging of flowing cells using a temporally coded excitation,” Opt. Express 21(4), 5164–5170 (2013). [CrossRef]  

20. R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006). [CrossRef]  

21. S. Dillavou, S. M. Rubinstein, and J. M. Kolinski, “The virtual frame technique: ultrafast imaging with any camera,” Opt. Express 27(6), 8112–8120 (2019). [CrossRef]  

22. J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

23. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010). [CrossRef]  

24. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992). [CrossRef]  

25. R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996). [CrossRef]  

26. D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods (Academic Press, 1982).

27. S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge Press University, 2004).

28. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009). [CrossRef]  

29. P. C. Hansen, “Analysis of discrete ill-posed problems by means of the l-curve,” SIAM Rev. 34(4), 561–580 (1992). [CrossRef]  

30. P. C. Hansen, Discrete inverse problems: insight and algorithms, vol. 7 (Siam, 2010).

31. P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013). [CrossRef]  

32. C. Jaques, “Hue-encoded shutter method code,” https://github.com/idiap/hesm_distrib (2019).

33. Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013). [CrossRef]  

34. Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171. [CrossRef]  

35. M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

36. M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009). [CrossRef]  

37. J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011). [CrossRef]  

38. C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

References

  • View by:
  • |
  • |
  • |

  1. N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
    [Crossref]
  2. J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
    [Crossref]
  3. C. Jaques, E. Pignat, S. Calinon, and M. Liebling, “Temporal Super-Resolution Microscopy Using a Hue-Encoded Shutter,” Biomed. Opt. Express 10(9), 4727–4741 (2019).
    [Crossref]
  4. J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
    [Crossref]
  5. K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
    [Crossref]
  6. A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
    [Crossref]
  7. T. H. Tsai, P. Llull, X. Yuan, L. Carin, and D. J. Brady, “Spectral-temporal compressive imaging,” Opt. Lett. 40(17), 4054–4057 (2015).
    [Crossref]
  8. R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. Cossairt, G. Schuster, and A. K. Katsaggelos, “High spatio-temporal resolution video with compressed sensing,” Opt. Express 23(12), 15992 (2015).
    [Crossref]
  9. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526 (2013).
    [Crossref]
  10. G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
    [Crossref]
  11. N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
    [Crossref]
  12. A. Koehler, “Ein neues Beleuchtungsverfahren für mikrophotographische Zwecke,” Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik 10, 433–440 (1893).
  13. W. Lukosz, “Optical systems with resolving powers exceeding the classical limit,” J. Opt. Soc. Am. 56(11), 1463–1471 (1966).
    [Crossref]
  14. W. Lukosz, “Optical systems with resolving powers exceeding the classical limit. II,” J. Opt. Soc. Am. 57(7), 932–941 (1967).
    [Crossref]
  15. P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
    [Crossref]
  16. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000).
    [Crossref]
  17. R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy—a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19(8), 1599–1609 (2002).
    [Crossref]
  18. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. 102(37), 13081–13086 (2005).
    [Crossref]
  19. S. S. Gorthi, D. Schaak, and E. Schonbrun, “Fluorescence imaging of flowing cells using a temporally coded excitation,” Opt. Express 21(4), 5164–5170 (2013).
    [Crossref]
  20. R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
    [Crossref]
  21. S. Dillavou, S. M. Rubinstein, and J. M. Kolinski, “The virtual frame technique: ultrafast imaging with any camera,” Opt. Express 27(6), 8112–8120 (2019).
    [Crossref]
  22. J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).
  23. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
    [Crossref]
  24. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
    [Crossref]
  25. R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996).
    [Crossref]
  26. D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods (Academic Press, 1982).
  27. S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge Press University, 2004).
  28. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009).
    [Crossref]
  29. P. C. Hansen, “Analysis of discrete ill-posed problems by means of the l-curve,” SIAM Rev. 34(4), 561–580 (1992).
    [Crossref]
  30. P. C. Hansen, Discrete inverse problems: insight and algorithms, vol. 7 (Siam, 2010).
  31. P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
    [Crossref]
  32. C. Jaques, “Hue-encoded shutter method code,” https://github.com/idiap/hesm_distrib (2019).
  33. Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
    [Crossref]
  34. Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
    [Crossref]
  35. M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).
  36. M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009).
    [Crossref]
  37. J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
    [Crossref]
  38. C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

2019 (4)

C. Jaques, E. Pignat, S. Calinon, and M. Liebling, “Temporal Super-Resolution Microscopy Using a Hue-Encoded Shutter,” Biomed. Opt. Express 10(9), 4727–4741 (2019).
[Crossref]

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

S. Dillavou, S. M. Rubinstein, and J. M. Kolinski, “The virtual frame technique: ultrafast imaging with any camera,” Opt. Express 27(6), 8112–8120 (2019).
[Crossref]

Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
[Crossref]

2016 (1)

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

2015 (2)

2013 (4)

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526 (2013).
[Crossref]

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

S. S. Gorthi, D. Schaak, and E. Schonbrun, “Fluorescence imaging of flowing cells using a temporally coded excitation,” Opt. Express 21(4), 5164–5170 (2013).
[Crossref]

2011 (2)

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
[Crossref]

2010 (2)

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

2009 (2)

A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009).
[Crossref]

M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009).
[Crossref]

2008 (1)

J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
[Crossref]

2006 (1)

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

2005 (2)

M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. 102(37), 13081–13086 (2005).
[Crossref]

N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
[Crossref]

2004 (1)

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

2002 (1)

2000 (1)

M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000).
[Crossref]

1998 (1)

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

1996 (1)

R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996).
[Crossref]

1992 (2)

P. C. Hansen, “Analysis of discrete ill-posed problems by means of the l-curve,” SIAM Rev. 34(4), 561–580 (1992).
[Crossref]

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
[Crossref]

1967 (1)

1966 (1)

1893 (1)

A. Koehler, “Ein neues Beleuchtungsverfahren für mikrophotographische Zwecke,” Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik 10, 433–440 (1893).

Agrawal, A.

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Baker, H.

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

Balázs, B.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Bapst-Wicht, L.

C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

Beck, A.

A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009).
[Crossref]

Bertsekas, D. P.

D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods (Academic Press, 1982).

Bhatti, N.

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

Boyd, S.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge Press University, 2004).

Brady, D. J.

Bub, G.

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

Calinon, S.

Carin, L.

Chan, K. G.

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

Chu, E.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

Ciruna, B.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Cossairt, O.

Cremer, C.

de Medeiros, G.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Del Bene, F.

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

Dickinson, M. E.

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Dillavou, S.

Eckstein, J.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

Eliceiri, K. W.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Fatemi, E.

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
[Crossref]

Forouhar, A. S.

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Fraser, S. E.

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
[Crossref]

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Freundlich, T.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Gharib, M.

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Gierten, J.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Gorthi, S. S.

Gustafsson, M. G. L.

M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. 102(37), 13081–13086 (2005).
[Crossref]

M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000).
[Crossref]

Hanley, Q. S.

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

Hansen, P. C.

P. C. Hansen, “Analysis of discrete ill-posed problems by means of the l-curve,” SIAM Rev. 34(4), 561–580 (1992).
[Crossref]

P. C. Hansen, Discrete inverse problems: insight and algorithms, vol. 7 (Siam, 2010).

Harville, M.

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

Heintzmann, R.

Helmes, M.

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

Hufnagel, L.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Huisken, J.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

Jaques, C.

C. Jaques, E. Pignat, S. Calinon, and M. Liebling, “Temporal Super-Resolution Microscopy Using a Hue-Encoded Shutter,” Biomed. Opt. Express 10(9), 4727–4741 (2019).
[Crossref]

C. Jaques, “Hue-encoded shutter method code,” https://github.com/idiap/hesm_distrib (2019).

C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

Jovin, T. M.

R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy—a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19(8), 1599–1609 (2002).
[Crossref]

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

Katsaggelos, A. K.

Keller, P. J.

Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
[Crossref]

Kittle, D.

Koehler, A.

A. Koehler, “Ein neues Beleuchtungsverfahren für mikrophotographische Zwecke,” Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik 10, 433–440 (1893).

Kohl, P.

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

Kolinski, J. M.

Koller, R.

Lansford, R.

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

Lee, P.

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

Liao, X.

Lichtman, J. W.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Liebling, M.

C. Jaques, E. Pignat, S. Calinon, and M. Liebling, “Temporal Super-Resolution Microscopy Using a Hue-Encoded Shutter,” Biomed. Opt. Express 10(9), 4727–4741 (2019).
[Crossref]

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009).
[Crossref]

J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
[Crossref]

C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Llull, P.

Lukosz, W.

Marguier, J.

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

Matsuda, N.

McDole, K.

Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
[Crossref]

Niederberger, T.

Norlin, N.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Ohn, J.

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

Osher, S.

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
[Crossref]

Pan, Y. A.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Parikh, N.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

Peleato, B.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

Pignat, E.

Pitrone, P. G.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Preibisch, S.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Prevedel, R.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Ranganathan, H.

M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009).
[Crossref]

Raskar, R.

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
[Crossref]

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Reddy, D.

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
[Crossref]

Rubinstein, S. M.

Rudin, L. I.

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
[Crossref]

Sanes, J. R.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Sapiro, G.

Schaak, D.

Schier, A. F.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Schindelin, J.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Schmid, L.

Schonbrun, E.

Schoppik, D.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Schorderet, D. F.

C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

Schuster, G.

Shaner, N. C.

N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
[Crossref]

Spinoulas, L.

Steinbach, P. A.

N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
[Crossref]

Stelzer, E. H. K.

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

Streichan, S. J.

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

Stuyvenberg, L.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Süsstrunk, S.

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

Swoger, J.

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

Teboulle, M.

A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009).
[Crossref]

Tecza, M.

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

Tibshirani, R.

R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996).
[Crossref]

Tomancak, P.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Trinh, L. A.

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

Tsai, T. H.

Tsien, R. Y.

N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
[Crossref]

Tumblin, J.

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Van Vliet, L. J.

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

Vandenberghe, L.

S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge Press University, 2004).

Veeraraghavan, A.

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
[Crossref]

Verbeek, P. W.

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

Vermot, J.

J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
[Crossref]

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Verveer, P. J.

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

Wagner, N.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

Wan, Y.

Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
[Crossref]

Wang, X. C.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Weber, M.

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

Weissman, T. A.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Wittbrodt, J.

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

Yang, J.

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526 (2013).
[Crossref]

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

Yuan, X.

Zimmerman, S.

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

ACM Trans. Graph. (1)

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: Motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Annu. Rev. Cell Dev. Biol. (1)

Y. Wan, K. McDole, and P. J. Keller, “Light-sheet microscopy and its potential for understanding developmental processes,” Annu. Rev. Cell Dev. Biol. 35(1), 655–681 (2019). PMID: 31299171.
[Crossref]

Biomed. Opt. Express (1)

Development (1)

Y. A. Pan, T. Freundlich, T. A. Weissman, D. Schoppik, X. C. Wang, S. Zimmerman, B. Ciruna, J. R. Sanes, J. W. Lichtman, and A. F. Schier, “Zebrabow: multispectral cell labeling for cell tracing and lineage analysis in zebrafish,” Development 140(13), 2835–2846 (2013).
[Crossref]

Found. Trends Mach. Learn. (1)

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010).
[Crossref]

Genesis (1)

J. Ohn, J. Yang, S. E. Fraser, R. Lansford, and M. Liebling, “High-speed multicolor microscopy of repeating dynamic processes,” Genesis 49(7), 514–521 (2011).
[Crossref]

HFSP J. (1)

J. Vermot, S. E. Fraser, and M. Liebling, “Fast fluorescence microscopy for imaging the dynamics of embryonic development,” HFSP J. 2(3), 143–155 (2008).
[Crossref]

IEEE Trans. Comput. Imaging (1)

K. G. Chan, S. J. Streichan, L. A. Trinh, and M. Liebling, “Simultaneous temporal superresolution and denoising for cardiac fluorescence microscopy,” IEEE Trans. Comput. Imaging 2(3), 348–358 (2016).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded strobing photography: Compressive sensing of high speed periodic videos,” IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011).
[Crossref]

J. Microsc. (2)

P. J. Verveer, Q. S. Hanley, P. W. Verbeek, L. J. Van Vliet, and T. M. Jovin, “Theory of confocal fluorescence imaging in the programmable array microscope (pam),” J. Microsc. 189(3), 192–198 (1998).
[Crossref]

M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000).
[Crossref]

J. Opt. Soc. Am. (2)

J. Opt. Soc. Am. A (1)

J. Royal Stat. Soc. Ser. B (1)

R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996).
[Crossref]

Nat. Methods (4)

P. G. Pitrone, J. Schindelin, L. Stuyvenberg, S. Preibisch, M. Weber, K. W. Eliceiri, J. Huisken, and P. Tomancak, “OpenSPIM: an open-access light-sheet microscopy platform,” Nat. Methods 10(7), 598–599 (2013).
[Crossref]

G. Bub, M. Tecza, M. Helmes, P. Lee, and P. Kohl, “Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging.,” Nat. Methods 7(3), 209–211 (2010).
[Crossref]

N. Wagner, N. Norlin, J. Gierten, G. de Medeiros, B. Balázs, J. Wittbrodt, L. Hufnagel, and R. Prevedel, “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods 16(6), 497–500 (2019).
[Crossref]

N. C. Shaner, P. A. Steinbach, and R. Y. Tsien, “A guide to choosing fluorescent proteins,” Nat. Methods 2(12), 905–909 (2005).
[Crossref]

Opt. Express (4)

Opt. Lett. (1)

Phys. D (1)

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992).
[Crossref]

Proc. Natl. Acad. Sci. (1)

M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. 102(37), 13081–13086 (2005).
[Crossref]

Proc. SPIE (1)

M. Liebling and H. Ranganathan, “Wavelet domain mutual information synchronization of multimodal cardiac microscopy image sequences,” Proc. SPIE 7446, 744602 (2009).
[Crossref]

Science (1)

J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science 305(5686), 1007–1009 (2004).
[Crossref]

SIAM J. Imaging Sci. (1)

A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009).
[Crossref]

SIAM Rev. (1)

P. C. Hansen, “Analysis of discrete ill-posed problems by means of the l-curve,” SIAM Rev. 34(4), 561–580 (1992).
[Crossref]

Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik (1)

A. Koehler, “Ein neues Beleuchtungsverfahren für mikrophotographische Zwecke,” Zeitschrift für wissenschaftliche Mikroskopie und für Mikroskopische Technik 10, 433–440 (1893).

Other (7)

P. C. Hansen, Discrete inverse problems: insight and algorithms, vol. 7 (Siam, 2010).

D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods (Academic Press, 1982).

S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge Press University, 2004).

J. Marguier, N. Bhatti, H. Baker, M. Harville, and S. Süsstrunk, “Color correction of uncalibrated images for the classification of human skin color,” Proceedings of the 15th IS&T/SID Color Imaging Conference pp. 331–335 (2007).

C. Jaques, “Hue-encoded shutter method code,” https://github.com/idiap/hesm_distrib (2019).

C. Jaques, L. Bapst-Wicht, D. F. Schorderet, and M. Liebling, “Multi-spectral widefield microscopy of the beating heart through post-acquisition synchronization and unmixing,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), pp. 1382–1385.

M. Liebling, J. Vermot, A. S. Forouhar, M. Gharib, M. E. Dickinson, and S. E. Fraser, “Nonuniform temporal alignment of slice sequences for four-dimensional imaging of cyclically deforming embryonic structures,” Proc. IEEE Int. Symp. Biomed. Imag. pp. 1156–1159 (2006).

Supplementary Material (4)

NameDescription
» Visualization 1       This video shows hue-encoded data acquired (left) with the corresponding reconstructions next to them (right). The reconstructions have a double frame rate compared to the acquisitions, and are monochrome.
» Visualization 2       This video corresponds to Figure 4 in the paper. It shows hue-encoded acquisitions with the corresponding reconstructions applied in a region of interest (ROI).
» Visualization 3       This video corresponds to Figure 6 in the paper. It shows a comparison between unregularized least-squares (LSTSQ) and alternating direction method of multiplier (ADMM) reconstructions.
» Visualization 4       This video shows a 3D+time reconstruction of the beating heart of a zebrafish embryo. We acquired a z-stack of videos, using the method presented in the paper at each z-position to boost the imaging frame rate.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1.
Fig. 1. Hue-encoded shutter for fluorescence light-sheet microscopy. (a) The excitation objective (EO) transmits consecutive pulses of different wavelengths within the exposure time $E$ of one frame. The sample ( $\boldsymbol{x}$ ) contains co-localized fluorophores (e.g. green and red fluorescent proteins) that respond to the excitation by emitting fluorescent light. The sample may also contain static regions that can be labeled with a single fluorophore type. The color sensor of the camera (CAM) captures the light emitted over the entire exposure time through the detection objective (DO). (b) Acquired (raw) multi-channel images (c) Our proposed method converts the color frames to multiple monochrome frames at a higher frame rate (here, by a factor 2) in a region of interest that contains dynamic features. The color information can be preserved in static regions outside of the region of interest.
Fig. 2.
Fig. 2. Fluo-HESM requires co-localized multi-color fluorescent labeling. (Top row) HESM [3] assumes a bright field microscopy setup, where the illumination light is transmitted through (or reflected by) a sample then captured by an RGB camera. The transmitted (or reflected) spectrum directly depends on the combination of the illumination spectrum and the sample’s spectral transmittance. HESM relies on using a temporally-variable spectrum in the illumination, which can be decoded following capture with an RGB camera. (Middle row) If using a single fluorophore and two different illumination sources (C and D), the emission from the sample only changes in overall intensity, but the shape of the emission spectrum remains unchanged. Hence, temporal modulation of the illumination spectrum would not univocally modulate the spectral shape of the emission and HESM could therefore not be used as-is to improve the frame-rate. (Bottom row) In this paper, we propose to use two light sources E and F and a sample labeled with two co-localized fluorophore species such that the combined emitted light spectrum is directly dependent on temporal modulation patterns of the illumination spectrum, making the use of HESM to improve the frame rate possible.
Fig. 3.
Fig. 3. Forward matrix $\boldsymbol{S\Gamma }$ combines temporal illuminations patterns of two lasers and the combined absorption and emission responses of the two fluorophores, together with the spectral response of the camera. Notice that the left part of the matrix $\boldsymbol{S}$ , boxed in green, contains repetitions of the temporal function of the green laser: $s_1[i]$ and the right part of $\boldsymbol{S}$ contains the same information for the blue laser. We also framed in green, respectively blue, the part that corresponds to the green, respectively blue, laser in the $\boldsymbol{\Gamma }$ matrix which expresses the efficiency of each fluorophore to the lasers combined with the sensitivity of the camera to the fluorescence emission spectra.
Fig. 4.
Fig. 4. Fluo-HESM allows simultaneous fast imaging of the beating heart and color imaging of its surroundings. (a),(e) two consecutive acquired frames. (b)-(d), respectively (f)-(h) are the red, green and blue layers of the image in (a), respectively (e). (i)-(l) reconstructed frames using Algorithm 1 with $\lambda =0.001$ . The super-resolution factor is 2, hence for two acquired frames, we observe 4 reconstructed frames. Note that there are 60 milliseconds between the two acquired frames (a) and (e) and 30 milliseconds between the super-resolved region in frames (i)-(l), framed in a white rectangle. Outside of this region, our method allows keeping the color information. On (j), we marked two regions with drastically different colors, green (osac-3-8-2195-i001) and orange ( $\clubsuit$ ). Standard imaging, in the presence of a single fluorophore species, would only produce frames (c) and (g), respectively (b) and (f) using samples labeled with only green fluorescent proteins, respectively only red fluorescent proteins. With Fluo-HESM, there are twice the number of these frames. This is also visible in Visualization 4, which shows a comparison between standard light-sheet data and reconstructions from our method in 3D+time. To highlight the link between this data and the schematic of Fig. 1, (a) and (e) on this figure correspond to Fig. 1(b) and (i)-(l) to Fig. 1(c). See Visualization 1 and Visualization 2 for the full movie. Fish orientation indicated: anterior (A), posterior (P), right (R) and left (L). Scale bar: 100 $\mu$ m.
Fig. 5.
Fig. 5. A static region is required to calibrate our method. The region must also have a similar red over green (R/G) ratio than in the heart. (a) Hue-encoded light-sheet image of a 2 days post-fertilization zebrafish heart. The ventricle (V), the atrium (At) and the pericardium (p) are outlined. (b) The ratio R/G of each pixel on the image in (a). Notice that the ratio R/G is not homogeneous over the whole image. In order to calibrate our method, we need a static region with the same ratio R/G as in the heart; it is shown in (d). (d’) shows the histogram of values within (d). Similarly, (c’) and (e’) show histograms of their corresponding regions (c) and (e). Notice that the calibration region (d) has a histogram similar to that of the heart (c), with a maximum centered around 1.5. Since Histogram (e’) is different from Histograms (c’) and (d’), our method would require an additional calibration to be usable in Area (e). Fish orientation indicated: anterior (A), posterior (P), right (R) and left (L). Scale bar: 100 $\mu$ m.
Fig. 6.
Fig. 6. Fluo-HESM is robust to fluorescence labeling inhomogeneities thanks to the temporal $\ell$ 1 regularization. (a)-(b) Composite color images and temporal super-resolution reconstructions. The dynamic area of the scene (c), with the beating heart, has been manually selected and inside of (c), the frame rate is twice that of outside. This illustrates the ability to choose between fast single hue imaging and slow color imaging. (d) Reconstructions at the location shown on (c) with a heart symbol (osac-3-8-2195-i001) using least-squares (without regularization, LSTSQ) or ADMM. The TV regularization we used in Eq. (9) favors piecewise constant reconstructions and preserves sharp edges. (e) Reconstructions at the location shown on (c) with a club symbol ( $\clubsuit$ ) using LSTSQ or ADMM. At this location, there is little motion of the sample, and the LSTSQ reconstructions oscillate a lot, while the ADMM reconstructions show little motion, justifying the use of ADMM over LSTSQ. Within (c), we assigned a hue to the reconstructed images, that we measured at calibration. Scale bar: 100 $\mu$ m. See Visualization 2 and additional Visualization 3 on a different sample.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

y c , k = t k t k + E = 1 L f = 1 F γ c , f b f , s ( t ) x ( t ) d t + d c ,
x ( t ) = i Z x [ i ] β 0 ( ( t i ) Q ) ,
y c , k = d c + = 1 L f = 1 F γ c , f b f , t k t k + E s ( t ) i = 0 Q 1 x [ t k , i ] β 0 ( ( t E Q i ) Q ) d t = d c + = 1 L f = 1 F γ c , f b f , i = 0 Q 1 x [ t k , i ] t k + E Q i t k + E Q ( i + 1 ) s ( t ) d t s [ t k , i ] ,
y k = S Γ x k + d 0 ,
Γ = ( ( γ 1 , 1 b 1 , + γ 1 , 2 b 2 , ) I 2 ( γ 2 , 1 b 1 , + γ 2 , 2 b 2 , ) I 2 ( γ 3 , 1 b 1 , + γ 3 , 2 b 2 , ) I 2 ) ,
x k = arg min x k y k d 0 S Γ x k 2 2
x = arg min x 1 2 y d A x 2 2 + λ T x 1 ,
minimize  1 2 y d A x 2 2 + λ u 1 such that  T x = u ,
L ( x , u , y ) = y A x d 2 2 + λ u 1 + z ( T x u ) + ( ρ / 2 ) T x u 2 2 ,

Metrics