Flexible and accurate total variation and cascaded denoisers-based image reconstruction algorithm for hyperspectrally compressed ultrafast photography

Zihan Guo; Jiali Yao; Dalong Qi; Dalong Qi; Pengpeng Ding; Chengzhi Jin; Yilin He; Ning Xu; Zhiling Zhang; Yunhua Yao; Lianzhong Deng; Zhiyong Wang; Zhiyong Wang; Zhenrong Sun; Shian Zhang; Shian Zhang; Shian Zhang; Shian Zhang

doi:10.1364/OE.506723

1. Introduction

Single-shot ultrafast optical imaging (UOI) is an advanced research field aiming at visualizing non-repeatable or difficult-to-repeat transient events in a single exposure with the requirements of an imaging speed of more than billions of frames per second (fps) and multiple frames [1,2]. As one of the cutting-edge passive imaging technologies achieving the incredibly high frame rate of tens of trillions of fps and a sequence depth of several hundred, the compressed sensing-based UOI (i.e., CS-UOI) technology has developed into a powerful tool in the field of single-shot UOI in the last decade. To date, various CS-UOI techniques have successfully realized the capture of flying photons [3,4], the measurement of ultrafast optical fields [5–7], the ultrafast phase-sensitive imaging of transparent objects [8,9], and the real-time observation of optical chaos [10] and optical rogue waves [11].

CS-UOI is a combination of CS and streak imaging, in which a spatially undersampled dynamic scene is firstly compressed into a two-dimensional (2D) image by streak imaging, and then an image reconstruction algorithm is applied to recover the original high-dimensional information from the acquired 2D image. At present, CS-UOI is developing from temporal imaging to multidimensional imaging. For example, compressed ultrafast photography (CUP) [3] and compressed ultrafast spectral-temporal photography (CUST) [12] can acquire spatiotemporal x-y-t three-dimensional (3D) intensity information, and compressed optical field topography (COFT) [13] can obtain spatiotemporal x-y-t intensity and phase information. Hyperspectrally compressed ultrafast photography (HCUP) [14] and compressed ultrafast spectral photography (CUSP) [15,16] can acquire spatial-temporal-spectral x-y-t-λ four-dimensional (4D) intensity information, and ultrafast light field tomography (LIFT) [17] can resolve volumetric-temporal x-y-z-t 4D intensity information. Moreover, volumetric-spectral CUP (SV-CUP) can acquire volumetric-temporal-spectral x-y-z-t-λ five-dimensional (5D) intensity information [18], and stereo-polarimetric CUP (SP-CUP) can capture volumetric-temporal-polarimetric x-y-z-t-φ 5D intensity information [19]. These techniques can be classified into two categories based on the way of streak imaging. One is to form the streak only along a single direction, for instance, CUP, SP-CUP and LIFT perform the temporal shearing of a dynamic scene only along the vertical direction by using streak cameras, and CUST and COFT perform the spectral (temporal) shearing of a dynamic scene illuminated by a temporally chirped pulse only along the horizontal direction by using gratings. The other is to form streaks along two orthogonal directions, including HCUP and CUSP, which perform the temporal shearing and spectral dispersion independently and orthogonally. For convenience, CUP represents the former, and HCUP represents the latter. Obviously, HCUP can maximally access to the information carried by different photon tags benefitting from its two independent shearing directions. Currently, HCUP has been successfully used for the detections of various dynamic scenes, such as the measurement of a chirped picosecond laser pulse [14], spectrally resolved fluorescence lifetime imaging microscopy [15], and laser-induced filament observation [16].

The image reconstruction algorithm is as important as the data acquisition hardware in computational imaging, CS-UOI techniques are no exception. To improve the image reconstruction quality of CUP, a series of advanced algorithms have been developed as the alternatives of the primary two-step iterative shrinkage/thresholding (TwIST) [20] algorithm in recent years. For example, Yang et al. developed a hybrid augmented Lagrangian (AL)- and deep learning-based algorithm to optimize the sparse domain and relevant iteration parameters, which apparently improves the efficiency and accuracy of the image reconstruction [21]. Ma et al. developed an end-to-end deep learning algorithm, which is capable of recovering the dynamic scene with sharper boundaries, higher feature contrast, and fewer artifacts [22]. Yao et al. developed a total variation (TV) combined 3D block-matched filtering algorithm that can simultaneously utilize gradient sparsity and non-local similarity for the image reconstruction, which not only improves the reconstruction quality, but also strengthens the noise immunity of CUP [23]. Jin et al. developed a multi-channel coupled and multi-scale weighted denoising algorithm based on a plug-and-play (PnP) framework, which effectively improves the accuracy and quality of reconstructed images by removing non-Gaussian distributed noises based on a weighted multi-scale denoising strategy [24]. Compared with the imaging modality of CUP, the increase of an additional shearing direction makes the data compression ratio of HCUP show about an order of magnitude increasement, which leads to the fact that the two mainstream iterative algorithms currently used in HCUP, i.e., the TwIST and AL [25] algorithms, cannot achieve the desired reconstruction quality. The two algorithms are based on the TV regularization. However, this regularization only takes into account the local features of the images, which is insufficient to express the priori information. It is due to this reason that the TV-regularized algorithms usually produce the staircasing phenomenon in the smoothing region, and finally result in the loss of the image details [26]. Therefore, the quality of the reconstructed image will rapidly deteriorate in the face of the increase of the data compression ratio. Unfortunately, no advanced image reconstruction algorithm has been applied to HCUP so far.

To solve this problem, we develop a new image reconstruction algorithm based on TV and cascaded denoisers (CD) for HCUP, referred to as the TV-CD algorithm. The TV-CD algorithm adapts a PnP-based alternating direction method of multipliers (PnP-ADMM) as an iterative framework and jointly uses the TV denoising model along with three advanced deep learning-based denoising models, involving the fast and flexible denoising convolutional neural network (FFDNet), the dilated-residual U-Net denoising neural network (DRUNet) and the fast deep video denoising network (FastDVDNet) in the framework, which not only preserves the image smoothness with TV, but also obtains more priori with CD. Therefore, it solves the common sparsity representation problem in local similarity and motion compensation. By comparing the TV-CD algorithm with the traditional TwIST and AL algorithms in both numerical simulation and experimental measurement results, the superiority of the proposed algorithm in HCUP image reconstruction is proved. It can be prospected that the TV-CD algorithm can vastly promote the practical applications of HCUP in capturing high-dimensional information of ultrafast dynamic scenes.

2. Basic principles

2.1 Forward data acquisition

As mentioned above, HCUP can be divided into the data acquisition and image reconstruction. The forward data acquisition process is shown in Fig. 1(a). Firstly, a 4D scene I(x, y, t, λ) with spatial-temporal-spectral information is spatially undersampled by a pseudo-randomly distributed binary matrix, which is typically a mask generated by a spatial light modulator. Subsequently, different spectral components of the encoded scene are dispersed by a prism or grating, and different temporal frames of the coded scene are deflected orthogonally to the direction of spectral dispersion, typically by an ultrafast streak camera or electro-optical deflector [27]. Finally, the encoded, dispersed and deflected 4D scene is integrated and recorded as a 2D image E(x’, y’) by the spatial-temporal-spectral integration using a planar array detector. Overall, this data acquisition process can be mathematically formulated as [14]

(1)$$E({{x^,},{y^,}} )= {\mathbf{MTSC}}I(x,y,t,\lambda ) = \mathbf{A}I(x,y,t,\lambda ),$$

where C is the spatial encoding operator, S is the spectral shearing operator, T is the temporal shearing operator, M is the spatial-temporal-spectral integration operator, and A = MTSC is a combined operator.

Fig. 1. (a) Schematic diagram of the data acquisition for HCUP, where t: time; λ: spectrum; x, y: spatial coordinates of the dynamical scene; x′, y′: spatial coordinates at the detector. (b) Data flowchart of the TV-CD algorithm for HCUP reconstruction, where a 5-step cascaded [TV, FFDNet, DRUNet, FastDVDNet, TV] denoising is used and each step is composed of the projections G and D representing the operations in Eqs. (8) and (9), respectively.

Download Full Size | PDF

2.2 Inverse image reconstruction

In the inverse image reconstruction process, the original scene is recovered based on the observed 2D image E(x’, y’) and the shifted 4D sampling cube, i.e., solving the inverse problem of Eq. (1). However, the inverse problem is ill-conditioned, and traditional optimization algorithms based on CS usually use a regularization term Φ(I) to confine the solution of this inverse problem to the desired signal space, where the optimal estimation I* of I can be found by minimizing an objective function f(I), and is given by as follows [28–30]

(2)$${I^\ast } = \mathop {\arg \min }\limits_I f(I )= \mathop {\arg \min }\limits_I \frac{1}{2}||{E - {\mathbf A}I} ||_2^2 + b\Phi (I ),$$

where ||•||₂ is the l₂ norm, $||{E - {\mathbf A}I} ||_2^2$ is the fidelity term presenting that the reconstructed scene needs to conform to the sampling equation, Φ(I) is the regularization term representing that the reconstructed scene needs to satisfy the prior, and b is the regularization parameter to balance the two terms. Here, a TV-CD algorithm based on the PnP-ADMM framework is proposed to solve problem (2). In the PnP framework [31,32], by introducing an auxiliary variable v, problem (2) can be rewritten as

(3)$$\mathop {\arg \min }\limits_{I,v} \left\{ {\frac{1}{2}\|{E - {\mathbf A}I} \|_2^2 + b\Phi (v )} \right\}\textrm{ }s.t.\textrm{ }v = I.$$

According to the AL multiplier method, the optimal solution of problem (3) can be obtained by minimizing an AL function, and is expressed as follows

(4)$$\mathop {\arg \min }\limits_{I,v,u} \frac{1}{2}\|{E - {\mathbf A}I} \|_2^2 + b\Phi (v )+ {u^T}({v - I} )\textrm{ + }\frac{\rho }{2}\|{v - I} \|_2^2,$$

where u is another auxiliary variable, and ρ is the penalty parameter. Furthermore, problem (4) can be solved by the ADMM algorithm, which can be decomposed into the following sequence of subproblems [33]

(5)$${I^{k + 1}} = \mathop {\arg \min }\limits_I \frac{1}{2}\|{E - {\mathbf A}I} \|_2^2 + \frac{\rho }{2}\left\|{I - \left( {{v^k} - \frac{1}{\rho }{u^k}} \right)} \right\|_2^2,$$

(6)$${v^{k + 1}} = \mathop {\arg \min }\limits_v b\Phi (v )+ \frac{\rho }{2}\left\|{v - \left( {{I^k} + \frac{1}{\rho }{u^k}} \right)} \right\|_2^2,$$

(7)$${u^{k + 1}} = {u^k} + \rho ({{I^{k + 1}} - {v^{k + 1}}} ).$$

Here, the superscript k represents the k-th iteration. Subproblem (5) is an equation of quadratic form that has a closed-form solution shown as [34]

(8)$${I^{k + 1}} = {[{{{\mathbf A}^T}{\mathbf A} + \rho I} ]^{ - 1}}\left[ {{{\mathbf A}^T}E + {v^k} - \frac{{{u^k}}}{\rho }} \right],$$

and subproblem (6) is a denoising problem based on the Φ(v) regularization, where $\left( {{I^k} + \frac{1}{\rho }{u^k}} \right)$ is a noisy image and v is a denoised image. The problem can be solved by using off-the-shelf denoising algorithms, and can be formulated as [32]

(9)$${v^{k + 1}} = {D_\sigma }\left( {{I^k} + \frac{1}{\rho }{u^k}} \right),$$

where D is the denoising algorithm being used and σ is the noise standard deviation. The TwIST and AL algorithms commonly applied to HCUP so far are both based on the TV regularization, and therefore the TV denoising algorithm is also considered for this subproblem. However, the TV denoisers based on local smoothness prior tend to introduce staircasing artifacts in the image reconstruction, which greatly limits the image reconstruction quality. Inspired by recent advances in deep learning-based denoisers [7,24] and combinations of multiple conventional denoisers [23,35] within the PnP framework to achieve image quality improvement in CUP, we combine the traditional TV denoiser with three advanced deep learning-based denoisers in series to solve subproblem (6), including FFDNet [36], DRUNet [37] and FastDVDNet [38]. Here, FFDNet and DRUNet remove the noise in a single frame out of the dynamic scene by taking the noise level maps as inputs. Complementarily, FastDVDNet effectively utilizes the information in the temporal neighborhood and strengthens the temporal correlation of the remaining noise in the output frames to further improve the overall denoising effect of the dynamic scene. Theoretically, different denoisers represent the utilization of different priors, and these deep learning networks driven by a large amount of data can acquire richer priors to better recover the intrinsic features of the structures in the image or scene. The flowchart of the image reconstruction for HCUP via the TV-CD algorithm is shown in Fig. 1(b). Here, the 2D observed image E(x’, y’) and the combined linear operator A are input into the PnP-ADMM iterative framework, where the projection G represents the operation in Eq. (8), and the projection D represents the operation in Eq. (9). A cascaded denoising configuration with [TV, FFDNet, DRUNet, FastDVDNet, TV] is utilized and individual numbers of iterations to each denoiser are assigned. When the number of iterations for one denoiser is completed, other denoisers are replaced in the subsequent iterations, and the reconstructed 4D scene is output until all of the iterations are finished. In each iteration, the noise standard deviation $\sigma$ is adaptively updated by considering the relative residue, and it is monotonously decreased [39].

3. Theoretical simulations

3.1 Simulation configuration

In order to validate the performance of the TV-CD algorithm, three different types of scenes were compressed and reconstructed for simulation. In the first two types of scenarios, several multispectral and video scenes were selected, in which the images differ from each other in only one dimension. Moreover, the images in the third type of scenario are variable in both the temporal and spectral dimensions. According to the HCUP image acquisition process, all scenes were first encoded with a {0, 1} pseudo-random spatial encoding for each image, with the pixel size of the mask being the same as that of the detector. Then, the encoded images with different spectra in each scene were sequentially shifted along the horizontal direction (x-direction), with each image shifted to the right by one pixel relative to the previous one to simulate the spectral dispersion. Similarly, the images at different time instants were sequentially shifted along the vertical direction (y-direction), and each frame was shifted downward by one pixel with respect to the previous one to simulate the temporal deflection. Finally, all the images were superimposed to obtain a 2D observation image. In each simulation, the proposed TV-CD algorithm, as well as the mainstream TwIST and AL algorithms, were employed to recover the original 4D scenes from the 2D observed images, respectively. For each algorithm, the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values of all the reconstructed images were calculated based on the ground truths as quantitative image quality assessments (IQAs). All of the image reconstruction tasks were performed with the hardware configuration of Intel i9-10920X (12 cores) and GPU RTX3090-24 G (Dual Card 3090), and the computation time were recorded for comparison.

3.2 Simulation on 3D multispectral and video scenes

Firstly, five multispectral scenes including Clay, Flowers, Painting, Pompoms and Spools from the CAVE dataset [40] were selected to examine the performance of the TV-CD algorithm for spectral scenes, and each of them contains a data size of 256 × 256 × 8 × 8 with 8 temporal and 8 spectral channels. It is worth noting that each scene varies in the spectral dimension but keeps unchanged in the temporal dimension. Each scene was forward compressed based on the image acquisition settings and then inversely reconstructed with the TwIST, AL and TV-CD algorithms, respectively. The IQA values of the reconstruction results by the three algorithms are displayed on the left side of Table 1, including the PSNR and SSIM. As can be seen, the reconstruction result by the TV-CD algorithm yields much higher IQA values than those of the TwIST and AL algorithms for each scene. In addition, the IQA values for all the five scenes are averaged to obtain average PSNR (SSIM) improvements of 3.83 dB (0.223) and 3.76 dB (0.165), and average computation time reduction of 68.32% and 72.57%, for the TV-CD algorithm over the TwIST and AL algorithms, respectively, which are shown in bold font. For a more visual comparison, the ground truth data cube and reconstructed results of Flowers are shown in Fig. 2(a). For simplicity, only the second, fifth and eighth frames in both the temporal and spectral channels are displayed, and the values below each sub-image are the corresponding PSNR (dB) and SSIM. In addition, specific regions are selected for zoomed-in display. It can be obviously seen that, although the TwIST algorithm is able to recover the spectral evolution of the scene, the reconstructed images are too blurred with serious loss of detailed information. What’s worse, the AL algorithm fails to recover the spectral evolution information. In contrast, the TV-CD algorithm not only correctly reconstructs the spectral evolution, but also obtains clearer structural boundaries and more image details.

Fig. 2. (a) and (b) Representative reconstructed frames of Flowers and Detonators by the algorithms, respectively, together with the ground truths for comparison. The sub-image in the bottom left corner of each frame is the enlarged scene in the corresponding red box.

Download Full Size | PDF

Table 1. The averaged results of PSNR (dB) and SSIM by different algorithms on multispectral and video scenes

View Table

Similarly, five video scenes including Welding, Detonators, Snowboard, Tennis and Filament captured by ultrahigh-speed cameras were also selected to examine the performance of the TV-CD algorithm for dynamic scenes, in which the data size of each scene is the same to that of a multispectral scene. Nevertheless, each scene varies in the temporal dimension but keeps unchanged in the spectral dimension. The IQA values of the five video scenes reconstructed by the three algorithms are also listed on the right side of Table 1, in which the TV-CD algorithm still shows superiority over the other two algorithms. Moreover, the averaged PSNR (SSIM) value of the TV-CD algorithm for all the video scenes shown in bold font is 3.32 dB (0.175) and 3.24 dB (0.102) higher, and the averaged computation time is 52.23% and 28.55% faster, than those of the TwIST and AL algorithms, respectively. As a visual comparison, Fig. 2(b) shows representative frames of the ground truth and reconstruction results of the Detonators with the same temporal and spectral indices as those in Fig. 2(a). It can still be observed that the reconstructed images of both the TwIST and AL algorithms show serious artifacts, which can hardly recover the actual dynamics of the gradually expanding shockwave generated by the detonator explosion. On the contrary, the TV-CD algorithm can recover the characteristics of the shockwave propagation in rather high fidelity, which can be clearly distinguished by the information in the magnified box.

3.3 Simulation on a 4D spectral-temporal-varying scene

As a reconstruction algorithm for HCUP, the performance of dealing with simultaneous temporal-spectral-varying scenes is very important. Therefore, a 4D spectral-temporal-varying scene containing 10 temporal and 10 spectral channels with each frame size of 358 × 358 was further generated for simulation. The scene consists of a triangle, a circle and a rectangle positioned with fixed relative distances between each other. Moreover, the shapes not only show different intensities in different spectral channels, but also rotate globally with a constant speed within the temporal frames. Ten representative frames out of the overall 100 frames of the scene with the same indices in the temporal and spectral dimensions are shown in Fig. 3(a). As can be seen, these three shapes in each image are rotated counterclockwisely by 18^° with respect to the previous one in the temporal dimension. In the spectral dimension, the intensities of the three shapes individually evolve according to a self-defined law. The intensity evolution settings for each shape in different spectral channels are shown in Fig. 3(b), where the intensity within any shape is uniform. With the intensity normalization in consideration, 1 denotes the normalized maximum intensity and 0.1 denotes an intensity that is one-tenth of the maximum intensity. In the spectral channels, the intensity of the triangle monotonously weakens from 1 to 0.1 with a fixed intensity difference of 0.1. For the circle, the intensity gets stronger and then weaker, where the strongest value is only half of that of the triangle at 0.5. For the rectangle, its intensity evolution is just the opposite of that of the triangle. Representative five frames (indicated by Ti/Si (i = 2, 4, 6, 8, 10)) of the reconstructed results obtained by the TwIST, AL and TV-CD algorithms are shown in Fig. 3(c), respectively. It can be seen that the TwIST and AL algorithms can only roughly recover the shape features with high intensity, such as the triangle in T2/S2 and the rectangle in T10/S10. However, they fail to recover the shape features with low intensity, such as the circle and rectangle in T2/S2, and the circle and triangle in T10/S10. In contrast, the TV-CD algorithm not only clearly recovers all the shape features of the scene, but also correctly reconstructs the spectral-temporal intensity evolution of the scene.

Fig. 3. (a) Ten representative ground truth frames of the 4D spatial-temporal-spectral scene. (b) Intensity settings of each shape in the spectral dimension. (c) Representative reconstructed frames of the algorithms. (d) Average PSNR and SSIM values of the reconstructed results. (e) and (f) Comparisons of the centroid positions of the three shapes in reconstruction to the ground truth in the X and Y directions, respectively. T: triangle, R: rectangle, C: circle.

Download Full Size | PDF

For quantitative comparison, the average PSNR and SSIM values of all the reconstructed frames by the three algorithms are further calculated, and the results are shown in Fig. 3(d). As can be seen, the average PSNR value by the TV-CD algorithm can be up to 34.98 dB, which is far superior to the other two, and the average SSIM value of 0.867 is also the best among the three algorithms. In addition, the centroid X-Y coordinates of the shapes with distinguishable contours in the reconstruction results were also extracted for comparison, and the results are shown in Figs. 3(e) and 3(f), in which the ten representative frames are indexed sequentially. It can be clearly seen that the temporal evolutions of the centroid coordinates of the three shapes reconstructed by the TV-CD algorithm are the most consistent with those of the ground truth, while anomalous coordinate points that significantly deviate from the ground truth are appeared in the results reconstructed by the TwIST and AL algorithms. It is worth noting that, restricted by the abilities of dealing with the scenes with low SNR for the TwIST and AL algorithms, several shapes are failed to be reconstructed, such as the circle in T1/S1 and T2/S2 reconstructed by the AL algorithm, as well as the triangle in T8/S8 and T9/S9 reconstructed by the TwIST algorithm. In addition, the image reconstruction performance comparisons in various conditions are also explored, and the results are shown in Supplement 1 Note 1. All of these results indicate that the TV-CD algorithm has an excellent performance in terms of the image quality and measurement accuracy.

4. Experimental results

Furthermore, the TV-CD algorithm was employed to reconstruct two sets of ultrafast scenes captured by a home built HCUP system to visualize the spatial-temporal-spectral information, including an ultrafast laser field and photo-induced fluorescence dynamics. As reported previously, the temporal and spectral frame intervals of the HCUP system are 2 ps and 1.72 nm, respectively [14]. Moreover, a spatial resolution enhancement of over 20% is brought by the TV-CD algorithm for the HCUP system compared with the other two algorithms, and the details are shown in Supplement 1 Note 2.

4.1 Single-shot measurement of an ultrafast laser field

4.1.1 Experimental configuration

In experiment, a temporally chirped picosecond laser pulse was firstly measured by the HCUP system, and the configuration of the experiment is shown in Fig. 4(a). A mode-locked Ti:Sapphire laser amplifier is used to generate femtosecond laser pulses with a central wavelength of 800 nm. An output laser pulse is first frequency chirped by a pulse stretching device (PSD), then an E-shaped mask is used to spatially modulate the chirped pulse, and the chirped E-shape pulse is then acquired by the HCUP system. In the HCUP system, the chirped E-shaped laser pulse firstly passes through a camera lens (CL) and a lens L1 to form an intermediate scene in the plane of a transmissive spatial encoder with a pseudo-random binary pattern, and the spatially encoded scene is then transferred by an optical 4f system consisting of lenses L2 and L3 to a streak camera with a fully opened entrance slit. A transmissive grating (G) is placed in front of the streak camera to disperse the scene in spectrum horizontally. In the streak camera, photons in the laser pulse are first converted to electrons by a photocathode, and the accelerated electron pulse is subsequently operated by a vertical sweeping voltage for temporal deflection. Finally, the electrons are multiplied by a microchannel plate and bombard on a phosphor screen to be converted back into photons, and a 2D measurement image is captured by the internal CCD.

Fig. 4. (a) The experimental configuration for measuring a temporally chirped picosecond laser pulse using an HCUP system. (b) Representative frames of the results reconstructed by the algorithms. (c) The BRISQUE and RMSE values of the results reconstructed by the algorithms. (d) Normalized intensities along the white dotted lines in the reconstructed images at 80 ps/801.06 nm in (b) and the reference. (e) and (f) Extracted temporal and spectral intensity evolutions from the reconstruction results, respectively, together with the reference curves.

Download Full Size | PDF

4.1.2 Reconstructed laser field evolution and analysis

After being acquired by the HCUP system with a suitable time window of the streak camera, the spatial-temporal-spectral 4D information of this temporally chirped E-shaped laser pulse was reconstructed by the TwIST, AL and TV-CD algorithms, respectively. After each image reconstruction, a data cube size of 54 × 54 × 23 × 23 for I(x, y, t, λ) with a data compression ratio of up to 529:1 was yielded, and representative frames of the reconstruction results are selected for display in Fig. 4(b). Considering the pulse duration and central wavelength of the laser pulse, the frames at the time instants of 60, 80, 100, 120 and 140 ps and the spectral channels of 794.18, 797.62 and 801.06 nm are selected. It can be seen that all the three algorithms are capable of recovering the positive frequency chirp characteristics of the laser pulse, with the longer wavelength components appearing earlier compared to the shorter ones. However, with such a high data compression ratio, the TwIST and AL algorithms solely based on the TV priori are unable to recover the complete E-shaped spatial structure at the frames with low intensities. What’s worse, the spatial intensity distributions of the recovered structures are extremely heterogeneous, e.g., in the images at 80 ps/801.06 nm reconstructed by the TwIST and AL algorithms. In contrast, the TV-CD algorithm obtains better reconstruction effect in terms of spatial structure and intensity distribution.

To quantitatively evaluate the image reconstruction quality, a blind/referenceless image spatial quality evaluator (BRISQUE) [41] is introduced. The BRISQUE does not compute distortion-specific features, but instead uses scene statistics of mean subtracted contrast normalized coefficients to quantify the loss of image “naturalness” in the image due to the presence of distortions, which leads to an overall measurement of quality. Meanwhile, considering the repeatability of the laser pulse, the spectral-temporal unsheared view acquired by the streak camera in the static mode is used as the reference image, and all the reconstructed spectral-temporal resolved frames are integrated to calculate the root-mean-square error (RMSE) between the integrated image and the reference image. The values of the BRISQUE and RMSE are shown in Fig. 4(c), and it should be noted that smaller values represent better image qualities in both the indicators. It can be obviously seen that the results reconstructed by the TV-CD algorithm show the smallest BRISQUE and RMSE values of 43.49 and 0.264, which are reduced by 11.43% and 27.41% to those by the TwIST algorithm, as well as 5.49% and 17.49% to those by the AL algorithm, respectively. Moreover, the normalized spatial intensity distributions along the white dashed lines in the reconstructed images at 80 ps/801.06 nm by the three algorithms in Fig. 4(b) are also extracted and compared with that of the reference image, and the results are shown in Fig. 4(d). Obviously, the intensity distribution reconstructed by the TV-CD algorithm is the closest to that of the reference, while the other two algorithms result in significant intensity degradations in the second and third peaks of the E-shaped structure with low SNR. In addition, in order to further verify the reconstruction accuracy of the three algorithms in both the temporal and spectral dimensions, their intensity evolution curves from all the reconstruction results are extracted for comparison, and the results are shown in Figs. 4(e) and 4(f), respectively. Here, the temporal intensity evolution measured by the streak camera in one-dimensional (1D) mode [42] and the spectral intensity distribution obtained with a spectrometer are used as reference curves. It can be found that the reconstruction results of the three algorithms are in good agreement with the reference curve for the temporal intensity evolution, achieving a consistent pulse duration in full width at half-maximum (FWHM) of about 100 ps. For the spectral intensity distribution, the result reconstructed by the TV-CD algorithm shows a high fidelity to the reference curve, achieving a spectral width in FWHM of 16 nm, while the spectral widths by the other two algorithms show varying degrees of expansion and deviation.

4.2 Single-shot lifetime imaging of fluorescence dynamics

4.2.1 Experimental configuration

Fluorescence lifetime imaging (FLI) is playing an irreplaceable role in photobiological and biomedical applications, and HCUP have shown its promising capability of realizing spectral-resolved FLI in a single shot in recent works [15,18]. Therefore, the second ultrafast scene in experiment is the photo-induced FLI observation of a mannequin model coated with CdSe quantum dots. As shown in Fig. 5(a), a femtosecond laser pulse with a central wavelength of 800 nm and pulse width of 50 fs is output from the Ti:Sapphire laser amplifier for excitation. The central wavelength is frequency doubled to 400 nm with a barium borate (BBO) crystal, which is within the optical absorption wavelength range of CdSe quantum dots [43,44], and the pulse energy is 300 μJ. Subsequently, the pulse is diffused by an engineered scatterer (ED) and reflected by a 425 nm long-pass dichroic mirror (DM) to illuminate a mannequin model coated with CdSe quantum dots, which has a beam diameter of 15 cm at the sample position, corresponding to a fluence of 5.3 µJ/cm². The quantum dots were purchased from Mesolight Inc., which have a quantum yield of 80% at the concentration of 300-360 µmol/g. After excitation, the emitted fluorescence scene then passes through a bandpass filter (BF), and it is further split into two paths by a beam splitter (BS) for detection. The reflected scene is directly imaged by an external CCD camera to form a spectral-temporal unsheared integral reference image, and the transmitted scene is captured by the HCUP system. Similarly, the TwIST, AL and TV-CD algorithms were respectively used to reconstruct the spatial-temporal-spectral 4D information of the fluorescence dynamics for comparison. In addition, the mannequin structure captured by the external camera was used as a spatial constraint to limit the boundaries of the reconstructed images.

Fig. 5. (a) Experimental configuration for observing photoinduced fluorescence dynamics using the HCUP system. (b) Representative frames of the results reconstructed by the algorithms. (c) The BRISQUE and RMSE values of the results reconstructed by the algorithms. (d) and (e) Extracted temporal and spectral intensity evolutions from the reconstruction results, respectively, together with the reference curves. (f) Extracted fluorescence lifetimes of representative spectral components of the algorithms, together with the measured result by the streak camera in 1D mode.

Download Full Size | PDF

4.2.2 Reconstructed fluorescence lifetime images and analysis

Here, the reconstructed data cube size is 67 × 105 × 42 × 33 for I(x, y, t, λ) with the data compression ratio as high as 1386:1. For simplicity, the frames at the time instants of 9.6, 12.0, 14.4, 19.2 and 48.0 ns and the spectral channels of 526.84, 532.00 and 537.16 nm are selected for display in Fig. 5(b). It can be seen that the fluorescence spectra reconstructed by the three algorithms have the same central wavelength of about 532.00 nm in spectrum. In the temporal dimension, the fluorescence intensity rises to a peak at the time instant of 12.0 ns, and then gradually decreases. It is not until 48 ns that the fluorescence intensity reconstructed by the TV-CD algorithm disappears, while the TwIST and AL algorithms still have significant fluorescence intensity. As a quantitative comparison, the BRISQUE and RMSE values of the reconstruction results compared to those of the reference image recorded by the external camera were calculated, and the results are shown in Fig. 5(c). As expected, the TV-CD algorithm brings the smallest BRISQUE and RMSE values of 53.6736 and 0.2043, which are reduced by 21.29% and 30.86% to those of the TwIST algorithm, as well as 7.68% and 22.02% to those of the AL algorithm, respectively. These values indicate the best reconstruction quality. The intensity evolution curves in the temporal and spectral dimensions are also extracted from the reconstruction results, and they are compared with the measured results by the streak camera in 1D mode and the spectrometer to confirm the reconstruction accuracy of the three algorithms. The results of the comparisons are shown in Figs. 5(d) and 5(e), respectively. It is clear that both the temporal and spectral intensity evolutions of the fluorescence dynamics reconstructed by the TV-CD algorithm are in the best agreement with the reference curve, while the temporal and spectral intensity evolutions by the TwIST and AL algorithms are significantly noisy and broadened. In addition, the fluorescence lifetimes in different spectral channels are also calculated, and the lifetimes of some representative spectral components are shown in Fig. 5(f). It can be seen that the lifetimes of different fluorescence spectral components are similar, which indicates that all the fluorescence spectral components come from the relaxation of the same excited state in CdSe quantum dots. Compared to the average fluorescence lifetime of 19.28 ns reconstructed by the TwIST algorithm and 12.61 ns reconstructed by the AL algorithm, the average lifetime reconstructed by the TV-CD algorithm is 8.68 ns, which is in excellent consistent with the lifetime of 8.66 ns measured by the streak camera in 1D mode. It is worth noting that the fluctuations between the lifetimes of different spectral components are minimized in the TV-CD algorithm, demonstrating that this algorithm has a superior reconstruction accuracy.

5. Discussions

Considering the single-shot imaging capability of HCUP-class techniques, as well as their unparalleled advantages in imaging speed, imaging dimension, and sequence depth, the recovery of high-quality scenes from compressed images will undoubtedly continue to gain widespread attention in the future. With the improvement of computing power and the development of new computational frameworks, the emergence of various iterative optimization denoising algorithms and end-to-end deep denoising networks have brought higher reconstruction accuracies and speeds to PnP-based algorithms.

First, based on the flexibility of the PnP framework, the type of denoisers can be selected according to the dimensional characteristics of the ultrafast dynamic scene. For example, spectral denoisers [45,46] can be selected for the spectral dimension, and video denoisers [47] can be selected for the temporal dimension. Exploring the optimal combination of multiple denoisers is also one of the critical problems to concern. For instance, in addition to cascaded denoising, a parallel denoising strategy can also be adopted in each iteration, and different denoising weights can be set for multiple denoisers to jointly improve the image reconstruction quality of HCUP [48].

Second, further enhancement of the robustness of the PnP framework is necessary. At present, pre-trained denoising networks are plugged into the PnP framework as the priori, and the image reconstruction quality is degenerated when the experimental task differs significantly from the training data of the networks. In perspective, an adaptive PnP framework can be further developed to automatically update parameters in the deep denoising network according to specific dynamic scenes and imaging models to solve the problem of mismatch between pre-trained networks and specific desired scenes [49]. In addition, the combination of the PnP framework and new neural network models will also effectively relieve the pressure of information reconstruction caused by the extremely high data compression ratio for HCUP. For example, the new transformer model relies on a self-attention mechanism to avoid the defect of limited receptive field in convolutional neural networks, and can utilize remote long-range non-local similarities to significantly improve the quality of image reconstruction [50,51].

Besides, compared with other imaging techniques based on CS, such as coded aperture compressive temporal imaging [52] with much slower imaging speeds, and coded aperture compressive spectral imaging [53] with spectrum mapping only, HCUP suffers from the low light throughput, resulting in signal drowning in noise when recording ultrafast dynamic scenes with ultrashort durations in a passive imaging mode. Therefore, it is of great significance to develop a deep denoising network for scenes in an extremely low light environment [54]. In addition, advanced deep denoising networks for other photon tags such as spectrum, polarization and phase are also indispensable.

6. Conclusion

In summary, we have developed a flexible and accurate image reconstruction algorithm based on TV and CDs for HCUP. The TV-CD algorithm combines the TV and three advanced deep denoising models in an iterative framework of PnP-ADMM, which can robustly recover the inherent features of the structures in ultrafast scenes by utilizing a deep learning network driven by a large amount of data to obtain a richer prior when preserving the smoothness of the image. Through the reconstructions of various types of simulation and experimental scenes, it is proved that the TV-CD algorithm can effectively improve the quality and accuracy of the reconstruction results compared to the widely used TwIST and AL algorithms, including reducing the spatial artifacts and improving the imaging details. In addition, it is foreseen that the continuous development of the more advanced PnP framework and deep denoising networks will enable PnP-based algorithms to further improve the image reconstruction quality of HCUP in future, which will greatly promote its applications in exploring complex high-dimensional ultrafast scenes.

Funding

National Natural Science Foundation of China (12074121, 12274129, 12274139, 12325408, 62105101, 62175066, 92150301); Science and Technology Commission of Shanghai Municipality (21JM0010700, 21XD1400900).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 for supporting content.

References

1. J. Liang and L. V. Wang, “Single-shot ultrafast optical imaging,” Optica 5(9), 1113–1127 (2018). [CrossRef]

2. X. Zeng, X. Lu, C. Wang, et al., “Review and prospect of single-shot ultrafast optical imaging by active detection,” Ultrafast Sci. 3, 0020 (2023). [CrossRef]

3. J. Liang, C. Ma, L. Zhu, et al., “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017). [CrossRef]

4. L. Gao, J. Liang, C. Li, et al., “Single-shot compressed ultrafast photography at one hundred billion frames per second,” Nature 516(7529), 74–77 (2014). [CrossRef]

5. F. Cao, C. Yang, D. Qi, et al., “Single-shot spatiotemporal intensity measurement of picosecond laser pulses with compressed ultrafast photography,” Opt. Laser Eng. 116, 89–93 (2019). [CrossRef]

6. J. Liang, L. Zhu, and L. V. Wang, “Single-shot real-time femtosecond imaging of temporal focusing,” Light Sci. Appl. 7(1), 42 (2018). [CrossRef]

7. C. Jin, D. Qi, Y. Yao, et al., “Single-shot real-time imaging of ultrafast light springs,” Sci. China Phys. Mech. 64(12), 124212 (2021). [CrossRef]

8. T. Kim, J. Liang, L. Zhu, et al., “Picosecond-resolution phase-sensitive imaging of transparent objects in a single shot,” Sci. Adv. 6(3), eaay6200 (2020). [CrossRef]

9. Y. Zhang, B. Shen, T. Wu, et al., “Ultrafast and hypersensitive phase imaging of propagating internodal current flows in myelinated axons and electromagnetic pulses in dielectrics,” Nat. Commun. 13(1), 5247 (2022). [CrossRef]

10. L. Fan, X. Yan, H. Wang, et al., “Real-time observation and control of optical chaos,” Sci. Adv. 7(3), eabc8448 (2021). [CrossRef]

11. U. Teğin, P. Wang, and L. V. Wang, “Real-time observation of optical rogue waves in spatiotemporally mode-locked fiber lasers,” Commun. Phys. 6(1), 60 (2023). [CrossRef]

12. Y. Lu, T. Wong, F. Chen, et al., “Compressed ultrafast spectral-temporal photography,” Phys. Rev. Lett. 122(19), 193904 (2019). [CrossRef]

13. H. Tang, T. Men, X. Liu, et al., “Single-shot compressed optical field topography,” Light: Sci. Appl. 11(1), 244 (2022). [CrossRef]

14. C. Yang, F. Cao, D. Qi, et al., “Hyperspectrally compressed ultrafast photography,” Phys. Rev. Lett. 124(2), 023902 (2020). [CrossRef]

15. P. Wang, J. Liang, and L. V. Wang, “Single-shot ultrafast imaging attaining 70 trillion frames per second,” Nat. Commun. 11(1), 2091 (2020). [CrossRef]

16. P. Wang and L. V. Wang, “Single-shot reconfigurable femtosecond imaging of ultrafast optical dynamics,” Adv. Sci. 10(13), 2207222 (2023). [CrossRef]

17. X. Feng and L. Gao, “Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging,” Nature Commun. 12(1), 2179 (2021). [CrossRef]

18. P. Ding, Y. Yao, D. Qi, et al., “Single-shot spectral-volumetric compressed ultrafast photography,” Adv. Photon. 3(04), 045001 (2021). [CrossRef]

19. J. Liang, P. Wang, L. Zhu, et al., “Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution,” Nat. Commun. 11(1), 5252 (2020). [CrossRef]

20. J. Bioucas-Dias and M. Figueiredo, “A new TwIST: Two step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. on Image Process. 16(12), 2992–3004 (2007). [CrossRef]

21. C. Yang, Y. Yao, C. Jin, et al., “High-fidelity image reconstruction for compressed ultrafast photography via an augmented Lagrangian and deep-learning hybrid algorithm,” Photon. Res. 9(2), B30–B37 (2021). [CrossRef]

22. Y. Ma, X. Feng, and L. Gao, “Deep-learning-based image reconstruction for compressed ultrafast photography,” Opt. Lett. 45(16), 4400–4403 (2020). [CrossRef]

23. J. Yao, D. Qi, Y. Yao, et al., “Total variation and block-matching 3D filtering-based image reconstruction for single-shot compressed ultrafast photography,” Opt. Laser Eng. 139, 106475 (2021). [CrossRef]

24. C. Jin, D. Qi, J. Yao, et al., “Weighted multi-scale denoising via adaptive multi-channel fusion for compressed ultrafast photography,” Opt. Express 30(17), 31157–31170 (2022). [CrossRef]

25. C. Yang, D. Qi, F. Cao, et al., “Improving the image reconstruction quality of compressed ultrafast photography via an augmented Lagrangian algorithm,” J. Opt. 21(3), 035703 (2019). [CrossRef]

26. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992). [CrossRef]

27. C. Yang, D. Qi, F. Cao, et al., “Single-shot receive-only ultrafast electro-optical deflection imaging,” Phys. Rev. Applied 13(2), 024001 (2020). [CrossRef]

28. M. A. Figueiredo, R. D. Nowak, and S. J. Wright, “Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems,” IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007). [CrossRef]

29. M. Elad, B. Matalon, and M. Zibulevsky, “Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization,” Appl. Comput. Harmon. Anal. 23(3), 346–367 (2007). [CrossRef]

30. S. J. Wright, R. D. Nowak, and M. A. Figueiredo, “Sparse reconstruction by separable approximation,” IEEE Trans. Signal Process. 57(7), 2479–2493 (2009). [CrossRef]

31. S. Venkatakrishnan, C. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in Global Conference on Signal and Information Processing (IEEE, 2013), pp. 945–948.

32. S. Chan, X. Wang, and O. Elgendy, “Plug-and-play ADMM for image restoration: Fixed-point convergence and applications,” IEEE Trans. Comput. Imaging 3(1), 84–98 (2017). [CrossRef]

33. S. Boyd, N. Parikh, E. Chu, et al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010). [CrossRef]

34. Y. Liu, X. Yuan, J. L. Suo, et al., “Rank minimization for snapshot compressive imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2990–3006 (2019). [CrossRef]

35. Y. Lai, Y. Xue, C.-Y. Côté, et al., “Single-shot ultraviolet compressed ultrafast photography,” Laser Photonics Rev. 14(10), 2000122 (2020). [CrossRef]

36. K. Zhang, W. M. Zuo, and L. Zhang, “FFDNet: Toward a fast and flexible solution for CNN-based image denoising,” IEEE Trans. on Image Process. 27(9), 4608–4622 (2018). [CrossRef]

37. K. Zhang, Y. Li, W. Zuo, et al., “Plug-and-play image restoration with deep denoiser prior,” IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6360–6376 (2022). [CrossRef]

38. M. Tassano, J. Delon, and T. Veit, “DVDNet: A fast network for deep video denoising,” in International Conference on Image Processing (IEEE, 2019), pp. 1805–1809.

39. X. Yuan, Y. Liu, J. Suo, et al., “Plug-and-play algorithms for large-scale snapshot compressive imaging,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 1444–1454.

40. F. Yasuma, T. Mitsunaga, D. Iso, et al., “Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum,” IEEE Trans. on Image Process. 19(9), 2241–2253 (2010). [CrossRef]

41. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. on Image Process. 21(12), 4695–4708 (2012). [CrossRef]

42. Guide to streak cameras (https://www.hamamatsu.com/content/dam/hamamatsu-photonics/sites/documents/99_SALES_LIBRARY/sys/SHSS0006E_STREAK.pdf) (Accessed: 22 August 2023).

43. K. Gong, J. E. Martin, L. E. Shea-Rohwer, et al., “Radiative lifetimes of zincblende Cdse/Cds quantum dots,” J. Phys. Chem. C 119(4), 2231–2238 (2015). [CrossRef]

44. Z. Pan, H. Zhang, K. Cheng, et al., “Highly efficient inverted Type-I Cds/Cdse core/shell structure Qd-sensitized solar cells,” ACS Nano 6(5), 3982–3991 (2012). [CrossRef]

45. Z. Lai, K. Wei, and Y. Fu, “Deep plug-and-play prior for hyperspectral image restoration,” Neurocomputing 481, 281–293 (2022). [CrossRef]

46. K. Wei, Y. Fu, and H. Huang, “3-D Quasi-recurrent neural network for hyperspectral image denoising,” IEEE Trans. Neural Netw. Learning Syst. 32(1), 363–375 (2021). [CrossRef]

47. M. Maggioni, Y. Huang, C. Li, et al., “Efficient multi-stage video denoising with recurrent spatio-temporal fusion,”, in IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 3465–3474.

48. S. Gu, R. Timofte, and L. Van Gool, “Integrating local and non-local denoiser priors for image restoration,” in 24th International Conference on Pattern Recognition (2018), pp. 2923–2928.

49. Z. Wu, C. Yang, X. Su, et al., “Adaptive deep PnP algorithm for video snapshot compressive imaging,” Int. J. Comput. Vision 131(7), 1662–1679 (2023). [CrossRef]

50. C. Zheng, Y. Zhang, J. Gu, et al., “Cross aggregation transformer for image restoration,” arXiv, arXiv:2211.13654v2 (2023). [CrossRef]

51. L. Wang, M. Cao, Y. Zhong, et al., “Spatial-temporal transformer for video snapshot compressive imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 45, 1–18 (2022). [CrossRef]

52. P. Llull, X. Liao, X. Yuan, et al., “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013). [CrossRef]

53. A. Wagadarikar, R. John, R. Willett, et al., “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. 47(10), B44–B51 (2008). [CrossRef]

54. K. Monakhova, S. R. Richter, L. Waller, et al., “Dancing under the stars: video denoising in starlight Kristina,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 16220–16230.

Data	IQAs		Methods		Data	IQAs		Methods
Multispectral Scenes		TwIST	AL	TV-CD	Video Scenes		TwIST	AL	TV-CD
Clay	$\bar{P S N R}$	20.64	21.00	27.73	Welding	$\bar{P S N R}$	18.81	19.39	22.46
	$\bar{S S I M}$	0.7011	0.7704	0.9141		$\bar{S S I M}$	0.5733	0.6830	0.8174
	Time	22.7000	13.6510	8.1171		Time	25.7833	11.6586	6.4758
Flowers	$\bar{P S N R}$	26.18	25.76	27.32	Detonators	$\bar{P S N R}$	18.94	19.71	24.19
	$\bar{S S I M}$	0.6281	0.6701	0.8255		$\bar{S S I M}$	0.6663	0.7572	0.8788
	Time	16.4800	13.7164	6.0889		Time	25.4333	10.9276	8.7600
Painting	$\bar{P S N R}$	25.44	24.01	28.28	Snowboard	$\bar{P S N R}$	20.95	21.47	24.08
	$\bar{S S I M}$	0.5416	0.6077	0.8198		$\bar{S S I M}$	0.7372	0.7792	0.8601
	Time	20.6500	12.9904	5.9875		Time	22.7167	8.4229	6.6755
Pompoms	$\bar{P S N R}$	24.47	25.35	29.53	Tennis	$\bar{P S N R}$	20.36	19.16	23.76
	$\bar{S S I M}$	0.6878	0.7566	0.9156		$\bar{S S I M}$	0.5915	0.6631	0.8008
	Time	24.3667	15.2823	7.2520		Time	27.1167	8.1857	7.5225
Spools	$\bar{P S N R}$	23.48	24.45	26.50	Filament	$\bar{P S N R}$	23.10	22.80	24.26
	$\bar{S S I M}$	0.6856	0.7293	0.8833		$\bar{S S I M}$	0.8470	0.8985	0.9360
	Time	23.3000	15.6458	6.6071		Time	25.0833	9.2219	5.1609
Average	$\bar{P S N R}$	24.04	24.11	27.87	Average	$\bar{P S N R}$	20.43	20.51	23.75
	$\bar{S S I M}$	0.6488	0.7068	0.8717		$\bar{S S I M}$	0.6831	0.7562	0.8586
	Time	21.4993	14.2572	6.8105		Time	25.2267	9.6833	6.9189

Flexible and accurate total variation and cascaded denoisers-based image reconstruction algorithm for hyperspectrally compressed ultrafast photography

Abstract

1. Introduction

2. Basic principles

2.1 Forward data acquisition

2.2 Inverse image reconstruction

3. Theoretical simulations

3.1 Simulation configuration

3.2 Simulation on 3D multispectral and video scenes

3.3 Simulation on a 4D spectral-temporal-varying scene

4. Experimental results

4.1 Single-shot measurement of an ultrafast laser field

4.1.1 Experimental configuration

4.1.2 Reconstructed laser field evolution and analysis

4.2 Single-shot lifetime imaging of fluorescence dynamics

4.2.1 Experimental configuration

4.2.2 Reconstructed fluorescence lifetime images and analysis

5. Discussions

6. Conclusion

Funding

Disclosures

Data availability

Supplemental document

References

Supplementary Material (1)

Data availability

Cited By

Figures (5)

Tables (1)

Equations (9)

Optics Express