The compressed ultrafast photography (CUP) technique, providing the fastest receive-only camera so far, has shown to be a well-established tool to capture the ultrafast dynamical scene. This technique is based on random codes to encode and decode the ultrafast dynamical scene by a compressed sensing algorithm. The choice of random codes significantly affects the image reconstruction quality. Therefore, it is important to optimize the encoding codes. Here, we develop a new scheme to obtain the optimized codes by combining a genetic algorithm (GA) into the CUP technique. First, we measure the dynamical scene by the CUP system with random codes and obtain the dynamical scene image at each moment. Second, we use these reconstructed dynamical scene images as the optimization target and optimize the encoding codes based on the GA. Finally, we utilize the optimized codes to recapture the dynamical scene and improve the image reconstruction quality. We validate our optimization scheme by the numerical simulation of a moving double-semielliptical spot and the experimental demonstration of a time- and space-evolving pulsed laser spot.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
The streak camera has been considered as one of the most important ultrafast photo-detection tools . Its temporal resolution can reach the order of hundreds of femtoseconds. However, it can only be used as a one-dimensional imaging device. Recently, by combining a compressed sensing (CS) algorithm [2,3], the compressed ultrafast photography (CUP) technique can transform the streak camera into a two-dimensional (2D) imaging device without employing any mechanical or optical scanning mechanisms . Therefore, the CUP technique allows the direct measurement of an ultrafast dynamical scene. Similar to traditional photography, the CUP technique is receive-only and does not need specialized active illumination, which is different from some previous ultrafast imaging techniques, such as sequentially timed all-optical mapping photography  or serial time-encoded amplified imaging . By now, the CUP technique has been successfully applied to measure some fundamental ultrafast optical phenomena [4,7], such as laser pulse reflection and refraction, photon racing in two media, and photonic Mach cone. In addition, the CUP technique has also been further extended and was successfully used for dynamic volumetric imaging by leveraging the time-of-fight information  and color ultrafast imaging by using a dichroic mirror to separate the signals into two color channels .
In the CUP experiment, the ultrafast dynamical scene is obtained by three-dimensional (3D) image reconstruction based on the CS algorithm with random codes. However, with random codes, the image reconstruction quality is usually poor. In previous studies, it was shown that optimizing the encoding codes in the CS algorithm can improve image reconstruction accuracy [9–12]. However, the CUP technique requires a sparse basis in the gradient domain, rather than the gradient domain. Therefore, we cannot optimize the encoding codes in the CUP system using the conventional CS method, such as minimizing the mutual coherence [11,12]. Herein, we propose an alternative scheme to optimize the encoding codes for the CUP system by combining a genetic algorithm (GA). Here, the GA is a conventional optimization algorithm, which can self-adaptively find the direction in searching space and obtain the global solution. In our optimization process, the whole dynamic scene is considered for optimization. We first obtain the dynamical scene image at each moment by encoding and decoding based on the random codes. We then optimize the encoding codes by using these reconstructed images as the optimization target. We finally improve the image reconstruction quality with the optimized codes. We numerically simulate the movement of a double-semielliptical spot and also experimentally demonstrate the time and space evolution of the pulsed laser spot. Both the theoretical and experimental results confirm the feasibility of our proposed scheme.
The whole flow chart for optimizing the encoding codes in the CUP system by our scheme is shown in Fig. 1. In Step I, we encode the dynamical scene with the random codes and obtain the dynamical scene image at each moment using a two-step iterative shrinkage/thresholding (TwIST) algorithm . In Step II, we use these reconstructed dynamical scene images in Step I as the simulated dynamical scene and optimize the codes by encoding and decoding the simulated dynamical scene based on the GA. Here, the reconstructed dynamical scene images in Step I are used as the optimization target. In Step III, we encode and decode the dynamical scene by using the optimal codes in Step II and improve the image reconstruction quality. In our optimization scheme, Steps I and III are the experimental measurement, while Step II is the theoretical simulation. Because of the image similarity between the reconstructed images in Step I and the original images, the two sets of images will have almost the same sparse basis. Thus, the optimized codes in Step II should also be optimal for the original dynamical scene. In principle, the optimized codes can effectively improve the image reconstruction accuracy.
In Step II of Fig. 1, we use the GA to optimize the encoding codes. The flow chart for the GA is shown in Fig. 2(a). First, the random codes are generated by the computer, which are used as the first generation. Second, the simulated dynamical scene is reconstructed based on the random codes, and the CC value is calculated. Third, according to the CC value, the new encoding codes are generated by selection, crossover, and mutation operation, which are used as the next generation. Finally, the above steps are repetitively performed until the CC value approaches convergence. In the optimization process, we will stitch all the reconstructed images in the space to form a new image in each generation. The image stitching method is shown in Fig. 2(b), where each image is shifted by a one image size in the space along the vertical direction relative to the previous image. To determine the image reconstruction quality, we will compare the two stitching images based on the reconstructed images with the random codes in Step I of Fig. 1 and optimizing codes in Step II. In mathematics, the normalized correlation coefficient (CC) value can be used to describe the image similarity, and it can be written as 2(c). The population size is 150, the crossover fraction is 0.8, and the mutation fraction is 0.2. Moreover, the population type is set as bit string, and thus it can satisfy 0 or 1 distribution, which is in accordance with digital micromirror device (DMD) settings.
We simulate the optimization of a double-semielliptical spot moving from left to right. Here, the two semielliptical spots are separated in the space for a certain distance and are used as the base image. In our simulation, the dynamical scene contains nine base images, as shown in Fig. 3(a). Similar to the CUP measurement, these base images are superimposed in sequence, where each base image is shifted by a pixel in the space along the vertical direction relative to the previous image. Finally, all the base images are projected on one plane and form a 2D image. We first encode and decode the dynamical scene with the random codes, and the reconstructed images are shown in Fig. 3(b). As can be seen, the reconstructed images are almost the same as the original images. However, the reconstructed images have very large background noise, especially the position in the gap between the two semielliptical spots. Moreover, the spatial contour is relatively fuzzy. We then reconstruct the simulated dynamical scene based on the above reconstructed images by optimizing the encoding codes. In the optimization process, the reconstructed images in Fig. 3(b) are used as the optimization target. By our optimization, the CC value increases to 99.34% from 98.62%, as shown in Fig. 3(d). The reconstructed images with the optimized codes are shown in Fig. 3(c). It is obvious that the optimized codes can improve the image reconstruction quality. The background noise in the reconstructed images is almost eliminated. Furthermore, there is a sharp spatial contour. To prove that the image reconstruction quality can be greatly improved by the optimized codes, we reconstruct the dynamical scene 100 times with the random codes. The corresponding CC values are also calculated, and the calculated results are shown in Fig. 3(e), together with that with the optimized codes. Here, the CC value is given by comparing the reconstructed images with the base images, which is different from the optimization process in Fig. 3(d). Compared with the CC values with the random codes, the CC value with the optimized codes is always much higher. This observation further confirms that our optimization scheme is effective for improving the image reconstruction quality.
As shown in Figs. 1 and 3, it needs to experimentally measure the ultrafast dynamical scene two times. That is, the first time is to obtain the optimized codes by the GA, and the second time is to improve the image reconstruction quality with the optimized codes. However, in many cases, the ultrafast dynamical scene can be theoretically simulated or predicted in advance, even for some nonrepetitive transient events. In particular, with the development of computational physics, chemistry, and biology, many ultrafast dynamical scenes can be accurately calculated. In this case, the first experimental measurement can be removed, i.e., Step I in Fig. 1. That is, we can theoretically simulate the dynamical scene that is to be experimentally measured and obtain the optimized codes for the dynamical scene. Thus, our optimization scheme can be performed with a single shot, i.e., Step III in Fig. 1, which maintains the major advantage of the CUP system.
To validate the feasibility of our optimization scheme, we measure the time- and space-evolving pulsed laser spot by the CUP system, and the experimental arrangement is shown in Fig. 4. A nanosecond laser pulse (Q-smart 450, Quantel) is used as our measured object with a pulse width of 3 ns, a central wavelength of 532 nm, and a repetition rate of 10 Hz. Here, the laser spot in the space has a Gaussian intensity distribution. To obtain the pulsed laser spot with complex spatial structure, we use a thin wire to divide the laser spot into two components in the space. The spatially modulated nanosecond laser is projected onto thin white paper, and a small fraction of photons can pass through the white paper. Thus, the time- and space-evolving laser spot on the white paper can be imaged with the CUP system. The CUP system has been described in a previous study , and therefore only a brief description is given here. First, the dynamical scene is imaged via a camera lens and a 4f imaging system. Then, the image is encoded by a DMD (Texas Instruments). Finally, the encoded image is sent to the streak camera (Model 2200, XIOPM) for measurement via the same 4f imaging system. In our CUP system, the temporal resolution of the streak camera is 180 ps, and the spatial resolution is .
In the theory, the time and space evolution process of the pulsed laser spot can be well simulated. Therefore, we use the single shot strategy. We first obtain the optimized codes by numerical simulation of the dynamical scene, and then we use the optimized codes to encode and decode the experimental dynamical scene. In the experiment, the spatial structure of the pulsed laser spot will not change in the evolution process. Therefore, we utilize an external CCD to measure the pulsed laser spot shape, and the measured result is shown in Fig. 5(a). Here, the measured image is used as the base image in the image reconstruction. In our optimization process, the CC value increases to 99.17% from 97.55%, as shown in Fig. 5(b). We encode and decode the experimental dynamical scene with the optimized codes, and the reconstructed images are shown in Fig. 5(c). One can see that the pulsed laser spot can be well reconstructed in both the space structure and time evolution. To demonstrate the improvement of image reconstruction quality with the optimized codes, we also give the reconstructed images with the random codes, as shown in Fig. 5(d). By comparing, it is easy to see that the spatial resolution of the reconstructed images with the optimized codes is significantly improved, especially the center part being blocked. To illustrate that the optimized codes by our optimization scheme are optimal, we perform the same experiment with the random codes 50 times. Similarly, their corresponding CC values are also calculated, and the calculated results are shown in Fig. 5(e), together with that with the optimized codes. As expected, the CC value with the optimized codes is consistently larger than those with the random codes. The experimental results can verify the feasibility of our optimization scheme. By combining the GA into the CUP system, the encoding codes can be optimized to improve the image reconstruction quality.
In the image reconstruction based on the CS algorithm, optimizing the encoding codes can improve the image reconstruction quality. In the traditional method, the optimal codes are based on the smallest mutual coherence between the representing and measurement matrices [9–12], which should be a normal 2D matrix. However, in some applications, such as in a snapshot spectral imager  or in CUP [4,7], the measurement matrix is pseudo. Therefore, it cannot be expressed as a normal 2D matrix. In this work, our optimization scheme in the CUP system shows that the GA can be used as a tool to optimize the encoding codes in the CS algorithm for the pseudo matrix. In the practical application, two measurement methods can be employed. One is the single measurement, and the other is two measurements. Typically, if the ultrafast dynamical scene can be theoretically simulated or predicted in advance, we propose to employ the single measurement strategy, where the optimized codes are obtained by the numerical simulation. An important advantage for the single measurement is that the CUP system can measure the nonrepetitive transient events.
In summary, we have developed a new method to optimize the encoding codes for the CUP system by combining the GA. We numerically simulated the moving double-semielliptical spot and experimentally demonstrated the time- and space-evolving pulsed laser spot, and these optimization results showed that the image reconstruction quality can be significantly improved with the optimized codes by our proposed scheme. This work solved a key problem of the CUP system that the choice of random codes will significantly affect the image reconstruction quality, and this is very helpful for obtaining the complex image information with the higher spatial resolution in future studies. Recently, an optimized image reconstruction method, i.e., space- and intensity-constrained reconstruction, has shown to be a powerful technique in improving the image reconstruction quality . By combining the optimized codes with the improved image reconstruction algorithm, the image reconstruction quality can be further improved. In addition, our optimization scheme can provide a paradigm to optimize the encoding codes for applications based on the CS algorithm, where the traditional optimization methods falter due to the pseudo matrix.
National Natural Science Foundation of China (NSFC) (11474096, 11727810, 11774094, 61720106009); Science and Technology Commission of Shanghai Municipality (STCSM) (16520721200, 17ZR146900).
2. E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Commun. Pure Appl. Math. 59, 1207–1223 (2006). [CrossRef]
3. Y. C. Eldar and G. Kutyniok, “Compressed sensing: theory and applications,” in Compressed Sensing: Theory and Applications (Cambridge University, 2012).
4. L. Gao, J. Liang, C. Li, and L. V. Wang, “Single-shot compressed ultrafast photography at one hundred billion frames per second,” Nature 516, 74–77 (2014). [CrossRef]
5. K. Nakagawa, A. Lwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, “Sequentially timed all-optical mapping photography (STAMP),” Nat. Photonics 8, 695–700 (2014). [CrossRef]
6. K. Goda, K. K. Tsia, and B. Jalali, “Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena,” Nature 458, 1145–1149 (2009). [CrossRef]
7. J. Liang, C. Ma, L. Zhu, Y. Chen, L. Gao, and L. V. Wang, “Single-shot realtime video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3, e1601814 (2017). [CrossRef]
8. J. Liang, L. Gao, P. Hai, C. Li, and L. V. Wang, “Encrypted threedimensional dynamic imaging using snapshot time-of-flight compressed ultrafast photography,” Sci. Rep. 5, 15504 (2015). [CrossRef]
9. M. Elad, “Optimal projections for compressed sensing,” IEEE Trans. Signal Process. 55, 5695–5702 (2007). [CrossRef]
10. J. M. Duarte-Carvajalino and G. Sapiro, “Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization,” IEEE Trans. Image Process. 18, 1395–1408 (2009). [CrossRef]
11. V. Abolghasemi, S. Ferdowsi, B. Makkiabadi, and S. Sanei, “On optimization of the measurement matrix for compressive sensing,” in European Signal Processing Conference (2010), Vol. 51, p. 427.
12. V. Abolghasemi, S. Ferdowsi, and S. Sanei, “A gradient-based alternating minimization approach for optimization of the measurement matrix in compressive sensing,” Signal Process. 92, 999–1009 (2012). [CrossRef]
13. J. M. Bioucas-Dias and M. A. T. Figueiredo, “A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16, 2992–3004 (2007).
14. Y. Yue, J. Yu, Y. Wei, X. Liu, and T. Cui, “A improved CoSaMP algorithm based on correlation coefficient for compressed sensing image reconstruction,” J. Comput. Inf. Syst. 9, 7325 (2013).
15. D. Kittle, K. Choi, A. Wagadarikar, and D. J. Brady, “Multiframe image estimation for coded aperture snapshot spectral imagers,” Appl. Opt. 49, 6824–6833 (2010). [CrossRef]
16. L. Zhu, Y. Chen, J. Liang, Q. Xu, L. Gao, C. Ma, and L. V. Wang, “Space- and intensity-constrained reconstruction for compressed ultrafast photography,” Optica 37, 694–697 (2016). [CrossRef]