AI-enabled real-time dual-comb molecular fingerprint imaging

Hyperspectral imaging extends traditional imaging approaches by providing detailed spectral information for each pixel in an image [1]. As a general method, it has been employed with impressive success across scientific disciplines, including Earth remote sensing [2] and medical sciences [3]. Hyperspectral imaging instruments often rely on dispersive or filtering optics or scanning high-resolution spectroscopy. In such instruments, it can be challenging to achieve simultaneously fast real-time imaging and high spectral resolution that is capable of resolving the narrow absorption lines of gases. Having the capability to rapidly image specific molecular species in real-time would unlock new opportunities for label-free bio-medical imaging, environmental monitoring, or industrial applications, including leak detection, chemical process optimization, and identification of hazardous substances. One attractive approach towards real-time and high-resolution imaging is combining dual-frequency comb spectroscopy [4,5] with an imaging detector array [6,7], where spatial and spectral multiplexing provides opportunities for rapid acquisition of high-resolution hyperspectral data without any moving mechanical parts.

Here, we demonstrate real-time dual-comb hyperspectral imaging of a narrow gas absorption feature with a high frame rate uncooled lead-selenide (PbSe) 1–5 µm infrared detector array. The high 1 kHz frame rate of the detector permits heterodyne detection above the detector’s technical flicker noise band and enables fast image acquisition. Hyperspectral images containing approximately 30 spectral channels in all 16,384 pixels are acquired at a rate of 10 Hz. Importantly, we show that the recorded data can be processed, and gas concentration images can be derived in real-time by an artificial intelligence (AI) convolutional neural network (CNN) [8–10] on a personal computer. The AI approach reduces the processing time by 4 orders of magnitude when compared to a conventional analysis and opens opportunities for real-time dual-comb hyperspectral imaging.

In dual-frequency comb spectroscopy [4,5,11–13], two optical frequency combs (1 and 2) are used. Each comb represents a well defined set of laser lines spaced by their respective repetition rate $f_{{\rm rep}}^{(1)}$ and $f_{{\rm rep}}^{(2)} = f_{{\rm rep}}^{(1)} + \Delta \!{f_{{\rm rep}}}$ ($\Delta \!{f_{{\rm rep}}} \ll f_{{\rm rep}}^{(1,2)}$) with a relative central frequency offset ${f_{\rm c}} \ll f_{{\rm rep}}^{(1,2)}$ between both combs. Simultaneous photo-detection of both combs by a photo-detector results in a multi-heterodyne signal composed of interferograms, each with a duration of $\Delta\! f_{{\rm rep}}^{- 1}$. Fourier transforming (at least one of) the interferograms yields the multi-heterodyne spectrum comprising beatnotes at frequencies of ${f_{ c}} + n \cdot \Delta \!{f_{{\rm rep}}}$ ($n = 0, \pm 1, \pm 2,\ldots $). Effectively, the optical spectrum is compressed by a factor of $(f_{{\rm rep}}^{(1)} + f_{{\rm rep}}^{(2)})/(2\Delta \!{f_{{\rm rep}}})$ and down-converted from the optical domain to multi-heterodyne frequencies around ${f_{ c}}$. Dual-comb hyperspectral imaging is achieved when a dual-comb light source illuminates a sample and then is imaged on a two-dimensional detector array where each pixel performs a multi-heterodyne detection [6]. In this way, massively multiplexed spatial and spectral information of the sample is simultaneously acquired at a high rate.

In our setup, a dual-comb source with a central wavelength of 1536.7 nm illuminates a PbSe photo-detector array (Fig. 1). The detector array is sensitive to light over the entire 1–5 µm wavelength range, which covers the characteristic spectral fingerprint domain of a large number of gas molecules. The array consists of $128 \times 128 \;{\rm pixels}$ and can be read out with a maximal frame rate of 4 kHz. Between the dual-comb source and the detector, we arrange a series of small nozzles through which acetylene gas (${{\rm C}_2}{{\rm H}_2}$) can be released, resulting in a jet of gas that is probed by the large diameter dual-comb beam. In this way, the spatial structure of the gas flow is projected onto the detector array, where the large number of pixels provides high spatial resolution and massively parallel acquisition.

 

Fig. 1. Dual-comb hyperspectral imaging. Dual combs are generated by electro-optic modulation (EOM) of a single continuous (CW) wave laser. The dual-comb light is sent through a sample, here a flow of absorbing acetylene (${{\rm C}_2}{{\rm H}_2}$) gas, and then detected by a high frame rate near-to-mid-infrared detector array. Each pixel of the $128 \times 128$ detector array simultaneously digitizes the dual-comb multi-heterodyne interferograms, which contain spectral information about the sample.

Download Full Size | PPT Slide | PDF

Dense heterodyne encoding of spectral information in a given frequency interval requires dual combs of high mutual coherence (heterodyne beatnotes must be narrower than their frequency spacing $\Delta \!{f_{{\rm rep}}}$). Such high mutual coherence dual combs have been demonstrated in the near- and mid-infrared based on mode-locked lasers, electro-optic modulation, and optical parametric oscillators [6,7,14–20]. In this demonstration, two near-infrared frequency combs are generated via electro-optic modulation [21] of a single continuous wave laser (see Supplement 1). Each comb consists of more than 30 comb lines spaced by $f_{{\rm rep}}^{(1)} = 1\;{\rm GHz} $ and $f_{{\rm rep}}^{(2)} = f_{{\rm rep}}^{(1)} + 10\;{\rm Hz} $, respectively, so that narrow molecular absorption features (typical spectral width ${\sim} 10\;{\rm GHz}$) are well resolved, and short acquisition duration down to $\Delta\! f_{{\rm rep}}^{- 1} = 0.1\; {\rm s}$ is possible. The relative central frequency offset $\Delta \!{f_{c}}$ between both combs is chosen to be 250 Hz, so that all beatnotes are above 100 Hz. This choice is made possible by the detector’s high frame rate and avoids the frequency band below 100 Hz where uncooled PbSe detectors exhibit significant technical flicker noise (see Supplement 1). As all heterodyne beatnote frequencies are below 400 Hz, we operate the detector at a frame rate of 1 kHz, which could be increased to accommodate more spectral sampling points (i.e., more heterodyne beatnotes), without decreasing the acquisition rate.

An example of several interferograms recorded by a single pixel of the detector array is shown in Fig. 2(a) (after removal of low frequency components). Fourier transformation of the raw interferogram trace yields the multi-heterodyne spectra, as shown in Fig. 2(b), for different acquisition durations of 0.1 s, 1 s, and 10 s (i.e., 1, 10, and 100 interferograms). The spectral envelope of the heterodyne beatnotes reflects the unabsorbed spectral envelope of the dual combs. For the shortest acquisition time of 0.1 s (corresponding to 100 frames or a single interferogram), the spectral resolution of the heterodyne spectrum corresponds to the frequency spacing of the heterodyne beatnotes. Longer acquisition durations provide higher spectral resolution in the heterodyne spectrum, and the heterodyne beatnotes show as narrow spectral peaks. As a higher spectral heterodyne resolution effectively rejects the incoherent white noise contribution, the signal-to-noise ratio (SNR) of the heterodyne beatnotes grows proportionally with the square root of the acquisition duration, which can, however, only be increased if the sample is evolving slowly. Importantly, already single interferogram acquisition provides a useful SNR of approximately 10. Note that the high mutual coherence of the combs would in principle permit longer acquisition durations [22], and phase correction [23–29] can extend this well beyond 1000 s.

 

Fig. 2. Raw interferograms, spectrum, single-pixel transmittance spectrum, comparison with HITRAN, and integrated concentration image. (a) Raw and unabsorbed dual-comb multi-heterodyne interferograms as recorded by a single pixel. (b) Multi-heterodyne spectra obtained by Fourier transforming the raw interferograms for different acquisition durations (10 s in orange, 1 s in blue, 0.1 s in black). (c) Single pixel dual-comb absorption spectrum of acetylene (${{\rm C}_2}{{\rm H}_2}$) retrieved from a 10 s acquisition (blue dots) compared to a HITRAN model (black line). The standard deviation of the absorption spectrum for acquisition times of 0.1 and 1 s is shown in light blue and orange bands (centered around the 10 s based data points). (d) Reconstructed integrated concentration image of an acetylene gas flow obtained by fitting a HITRAN model to the recorded transmittance for each pixel. Transverse absorption profiles are shown for three different positions along the gas flow (white curves).

Download Full Size | PPT Slide | PDF

For a first test of hyperspectral imaging, we probe the jet of ${{\rm C}_2}{{\rm H}_2}$ gas across the field-of-view. The dual-comb heterodyne signal is recorded for each pixel and Fourier transformed to yield the heterodyne spectra. For each pixel, the transmittance of the dual-comb light on its specific path is obtained by normalizing the heterodyne spectrum by an unabsorbed reference heterodyne spectrum. In our case, the reference heterodyne spectrum is derived from nine pixels in the top left corner of the detector (where essentially unabsorbed comb light is detected). Alternatively, a pre-recorded reference spectrum may be used. An example of the transmittance signature recorded by a single pixel is shown in [Fig. 2(c)] for a 10 s long acquisition, showing very good agreement with the HITRAN database [30,31] (residuals below 3%). Note that the instrument function is well approximated by a zero-width delta function (width of a comb line and comb line separation at least 6 orders of magnitude narrower than the absorption feature). In addition, standard-error bands for shorter acquisition duration (0.1 s and 1 s) are shown, indicating that absorption on the few-percent level can already be detected based on single interferograms.

A natural measure for the number of absorbing molecules is their integrated concentration along the light path (cf. Supplement 1), which can be retrieved from a fit to a HITRAN model. It is computed for each pixel based on the 10 s recording (100 interferograms) and shown in Fig. 2(d). The gas jet is imaged based on its infrared absorption signature, allowing retrieval of integrated concentrations as small as $0.015\;{\rm mol}/{{\rm m}^2}$. Profiles of the integrated concentration for different heights of the gas flow show a transverse expansion of the gas jet along its flow direction. While in our case only one gas species is imaged, the analysis can readily be generalized to multiple gas species.

Figure 2(d) demonstrates that hyperspectral imaging with the near-to-mid-infrared detector array is possible. However, the high data rate of approximately 35 MB/s and the necessity of performing the analysis on each pixel requires large memory size and results in long computation times, in our case 20 to 30 min on a desktop computer for one frame. Overcoming this data processing bottleneck is crucial in applications demanding fast feedback, such as leak detection, chemical process monitoring, or bio-medical imaging.

To reduce the data processing time, we explore an AI-based approach that bypasses the conventional analysis: a CNN is used that directly processes the temporal interferograms. It is implemented using the Keras library running on a Tensorflow backend [32]. The response of the CNN is invariant against translation of the input data along the time axis; a dedicated trigger or temporal alignment of the input interferogram is therefore not required (any input of a duration of $\Delta\! f_{{\rm rep}}^{- 1}$ is suitable). This property of CNN greatly simplifies the data analysis and results in better versatility (i.e., applicability to unknown data) compared to densely connected networks with a comparable number of parameters. The detailed architecture of the CNN is described in Supplement 1. While the capability of neural networks for fast data processing is widely recognized [33–35], the difficulty of obtaining a reliable labelled training data set (containing training input data as well as the correct analysis outcome) is often prohibitive to their use. Here, a training data set is rapidly built, owing to the large amount of data generated by the setup. The details of this data set and its use in training the neural network are described in Supplement 1. Statistically, the trained CNN-based analysis differs by less than 3% of the maximum integrated concentration value from the results obtained through conventional analysis. As one can intuitively expect, a detailed analysis of the CNN’s first layer weights shows that the CNN learns to focus its “attention” on the heterodyne frequency components that encode the absorbed dual-comb frequencies (see Supplement 1).

To test the CNN’s performance, we observe the dynamics when the gas flow is turned on. The multi-heterodyne data of each pixel are processed for each 100 ms time window (single interferogram), so that good temporal resolution is achieved. From the series of reconstructed gas images, three snapshot frames, separated in time by 1 s, are shown in Fig. 3. In frame 0, no ${{\rm C}_2}{{\rm H}_2}$ gas was released, the gas jet is emerging from one out of several nozzles in frame 10, and the gas jet from several nozzles is fully developed in frame 20. The results in Fig. 3 show that the CNN can reliably work on single interferograms (0.1 s acquisitions), permitting the observation of dynamic processes (also see Visualization 1). Importantly, the trained CNN can process the data at a rate that exceeds the raw data recording rate, therefore enabling real-time molecule specific imaging with a frame rate of $\Delta\! f_{{\rm rep}}^{- 1} = 10\;{\rm Hz} $. The CNN also alleviates the need for large memory storage by reducing the heterodyne raw data frame rate from 1000 down to 10 frames per second for the gas images. If desired, the neural network could be trained to output other parameters, e.g., gas temperature (based on line shapes), or be extended to multi-species imaging by adding outputs on the last layer and adjusting the training accordingly.

 

Fig. 3. AI-based imaging. Three selected frames from a movie (cf. Visualization 1) that has been reconstructed in real-time by the neural network (frame rate 10 Hz). The frames show the dynamics of the ${{\rm C}_2}{{\rm H}_2}$ gas jet and are separated by 1 s each. The first frame is recorded before the gas jet is turned on, the second frame depicts the onset of gas emission, and the last frame shows the established gas jet.

Download Full Size | PPT Slide | PDF

In summary, we have shown that real-time dual-comb hyperspectral imaging can be performed with a near-to-mid-infrared photo-detector array enabling imaging of a gas with molecular specificity based on its fingerprint absorption signature. Hyperspectral data has been simultaneously recorded in 16,384 pixels with 30 spectral channels, and short acquisition times of 100 ms enabled observation of dynamic phenomena. Key to this demonstration is the high frame rate of the detector array as well as the high mutual coherence of the dual-comb illumination, which permits recording of the heterodyne signal in a frequency band (here above 100 Hz) that does not suffer from technical low-frequency flicker noise. Hundreds of spectral channels could be implemented without reducing the acquisition rate by utilizing a higher detector frame rate. Importantly, we have also shown that the high data rate resulting from the massively parallelized hyperspectral data acquisition can be processed in real-time by a CNN, providing gas concentration images at 10 Hz rate. Built on direct processing of time-domain interferograms, this method adds to the AI toolbox for spectroscopy [36–40] and can readily be applied to other dual-comb spectroscopy schemes. As the detector array is sensitive across the entire 1–5 µm wavelength range, our demonstration can be extended to cover the characteristic molecular absorption fingerprints of a wide range of molecular species. Possible extensions of our demonstration include the use of high-repetition-rate mid-infrared quantum cascade [41,42], mid-infrared microresonator [43,44], or mid-infrared electro-optic combs [45] for broadband spectral imaging of transparent condensed phase samples.