LED-based compressive spectral-temporal imaging

Xiao Ma; Xin Yuan; Chen Fu; Gonzalo R. Arce

doi:10.1364/OE.419888

Optics Express
Vol. 29,
Issue 7,
pp. 10698-10715
(2021)
•https://doi.org/10.1364/OE.419888

LED-based compressive spectral-temporal imaging

Xiao Ma, Xin Yuan, Chen Fu, and Gonzalo R. Arce

Open Access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Xiao Ma, Xin Yuan, Chen Fu, and Gonzalo R. Arce, "LED-based compressive spectral-temporal imaging," Opt. Express 29, 10698-10715 (2021)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Check for updates

Related Topics
Table of Contents Category
- Imaging Systems, Microscopy, and Displays
Optics & Photonics Topics
?

The topics in this list come from the Optics and Photonics Topics applied to this article.

About this Article
History
- Original Manuscript: January 22, 2021
- Revised Manuscript: March 3, 2021
- Manuscript Accepted: March 3, 2021
- Published: March 22, 2021

Abstract

A compressive spectral-temporal imaging system is reported. A multi-spectral light-emitting diode array is used for target illumination and spectral modulation, while a digital micro-mirror device (DMD) encodes the spatial and temporal frames. Several encoded video frames are captured in a snapshot of an integrating focal plane array (FPA). A high-frame-rate spectral video is reconstructed from the sequence of compressed measurements captured by the grayscale low-frame-rate camera. The imaging system is optimized through the design of the DMD patterns based on the forward model. Laboratory implementation is conducted to validate the performance of the proposed imaging system. We experimentally demonstrate the video acquisition with eight spectral bands and six temporal frames per FPA snapshot, and thus a 256 × 256 × 8 × 6 4D cube is reconstructed from a single 2D measurement.

Full Article | PDF Article

More Like This

Snapshot compressive imaging based digital image correlation: temporally super-resolved full-resolution deformation measurement

Wenwu Chen, Bo Zhang, Liuning Gu, Haibo Liu, Jinli Suo, and Xinxing Shao
Opt. Express 30(19) 33554-33573 (2022)

Compressive temporal imaging using a rolling shutter camera array

Felipe Guzmán, Pablo Meza, and Esteban Vera
Opt. Express 29(9) 12787-12800 (2021)

Coded aperture compression temporal imaging based on a dual-mask and deep denoiser

YouRan Ge, GangRong Qu, YuHao Huang, and Duo Liu
J. Opt. Soc. Am. A 40(7) 1468-1477 (2023)

Previous Article Next Article

Supplementary Material (17)

Name	Description
Visualization 1	Video of the simulation input 4D data-cube with size 256 by 256 by 8 by 32.
Visualization 2	Video of the simulation reconstruction result with 12.5% transmittance random coding.
Visualization 3	Video of the simulation reconstruction result with 50% transmittance random coding.
Visualization 4	Video of the simulation reconstruction result with 12.5% transmittance blue noise coding.
Visualization 5	Video of the simulation reconstruction result with 50% transmittance blue noise coding.
Visualization 6	Video of the simulation reconstruction result with 6 temporal frames compression.
Visualization 7	Video of the simulation reconstruction result with 10 temporal frames compression.
Visualization 8	Video of the simulation reconstruction result with GPSR4D algorithm.
Visualization 9	Video of the simulation reconstruction result with GAPTV4D algorithm.
Visualization 10	Video of the simulation reconstruction result with DeSCI4D algorithm.
Visualization 11	Video of the expeirmental reconstruction result of natural plants with gaptv4D algorithm.
Visualization 12	Video of the expeirmental reconstruction result of color checker with 4 temporal frames compression and DeSCI4D algorithm.
Visualization 13	Video of the expeirmental reconstruction result of color checker with 6 temporal frames compression.
Visualization 14	Video of the expeirmental reconstruction result of Super Mario with 4 temporal frames compression.
Visualization 15	Video of the expeirmental reconstruction result of Super Mario with 6 temporal frames compression.
Visualization 16	Video of the expeirmental reconstruction result of natural plants with 4 temporal frames compression.
Visualization 17	Video of the expeirmental reconstruction result of natural plants with 6 temporal frames compression.

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.

Fig. 1. LeSTI testbed. Spectral LED illumination is used for spectral scene modulation. An imaging lens focuses the spectrally modulated scene onto the DMD imaging plane, where spatial modulations are imposed. A second objective lens focuses the spatial modulated scene onto the imaging plane, where a grayscale detector captures multiple temporally modulated video frames.

Download Full Size | PDF

Fig. 2. (a) LeSTI system architecture. Multi-spectral LEDs response for illumination and spectral modulations; a DMD is used for imposing spatial binary patterns. A grayscale sensor is utilized for image capture. (b) Eight types of LED’s spectra shown in the 400-700nm wavelength range. (c) The designed LED illuminator with multi-spectral LEDs used in our system. Eight different types of LEDs are used and evenly placed on a circuit board with three elements of each type.

Download Full Size | PDF

Fig. 3. (a) Timing sequence in the LeSTI sensing. (b) LeSTI system sensing process (bottom row). A pixel on the scene shows the spectral modulation in the system (middle row).

Download Full Size | PDF

Fig. 4. PSNR and SSIM plots with different system configurations. Notice that the GAP-TV reconstruction algorithm is applied for reconstructions. Here ‘white’ refers to random binary random patterns used in the DMD and ‘Blue’ means the blue noise patterns are used. The following percentage numbers denote the pattern transmittance.

Download Full Size | PDF

Fig. 5. Original image cube (row 1, Visualization 1) and simulation reconstruction results of

$N_t = 2$

with random (white noise) coding transmittance 12.5% (row 2, Visualization 2), random coding 50% (row 3, Visualization 3), blue noise coding 12.5% (row 4, Visualization 4), and blue noise coding 50% (row 5, Visualization 5). In the first row, original data is presented with spectral images, reproduced RGB images and zoomed information. Reconstruction results with different configurations (rows 2-5) are shown with the same layout. The compressed measurements are shown at the bottom left (last row). Spectral plots of selected regions are shown in the bottom right.

Download Full Size | PDF

Fig. 6. Simulation reconstruction results of 25% transmittance blue noise with different temporal compression, i.e.

$N_t = 6$

(row 2, Visualization 6),

$10$

(row 3, Visualization 7). Notice that the P3 interest point of reconstruction

$N_t = 6$

is on frame 4, and for reconstruction

$N_t = 10$

it is on frame 6. The coordinates of P3 on these two frames move with LEGO.

Download Full Size | PDF

Fig. 7. Simulation reconstruction with moving LEGO and static background shown in reproduced RGB images. Simulation measurement and the reconstruction of 4 frames from a snapshot when the objects have (a) small motion, (b) fast motion, and (c) very fast motion with respect to the LED scanning. Notice that the GAP-TV algorithm is applied to generate the reconstruction results.

Download Full Size | PDF

Fig. 8. Simulation reconstruction results with different reconstruction algorithms, i.e. GPSR4D (row 2, Visualization 8), GAP-TV4D (row 3, Visualization 9), DeSCI4D (row 4, Visualization 10). The results are based on 25% transmittance blue noise with

$N_t = 8$

frames. Four reproduced RGB images are selected to be shown.

Download Full Size | PDF

Fig. 9. Three imaging scenes: (a) color checker, (b) Super Mario and background, and (c) natural plants.

Download Full Size | PDF

Fig. 10. Experimental reconstruction spectral video excerpts with

$N_t = 4$

using GAPTV4D (top-right, Visualization 11) and DeSCI4D (bottom-right, Visualization 12) algorithm.

Download Full Size | PDF

Fig. 11. Real data reconstruction spectral video excerpts with different temporal compression ratios, i.e. Color checker with

$N_t = 4$

(Visualization 12) and

$N_t = 6$

(Visualization 13), Super Mario with

$N_t = 4$

(Visualization 14) and

$N_t = 6$

(Visualization 15), Natural plants with

$N_t = 4$

(Visualization 16) and

$N_t = 6$

(Visualization 17).

Download Full Size | PDF

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

g (x^{'}, y^{'}) = \iint f (x, y, λ, t) m (x, y, λ, t) d λ d t,

f_{n_{x}, n_{y}, n_{t}}^{*} = S f_{n_{x}, n_{y}, n_{t}} .

G = \sum_{n_{l} = 1}^{N_{l}} \sum_{n_{t} = 1}^{N_{t}} C_{n_{l}, n_{t}} ⊙ F_{n_{l}, n_{t}}^{*} + E,

M_{n_{λ}, n_{t}} = \sum_{n_{l} = 1}^{N_{l}} s_{n_{l}, n_{λ}} \cdot C_{n_{l}, n_{t}} .

G = \sum_{n_{λ} = 1}^{N_{λ}} \sum_{n_{t} = 1}^{N_{t}} M_{n_{λ}, n_{t}} ⊙ F_{n_{λ}, n_{t}} + E .

H = [d i a g (M_{1, 1}), d i a g (M_{2, 1}), \dots, d i a g (M_{N_{λ}, 1}), d i a g (M_{1, 2}), \dots, d i a g (M_{N_{λ}, N_{t}})],

g = H f + e .

\hat{f} = \arg max_{f} \exp {- \frac{1}{2 σ_{e}^{2}} ‖ g - H f ‖_{2}^{2} + \log p (f)} = \arg min_{f} \frac{1}{2} ‖ g - H f ‖_{2}^{2} - σ_{e}^{2} \log p (f) .

\hat{f} = \arg {min}_{f} \frac{1}{2} ‖ g - H f ‖_{2}^{2} + λ R (f) .

f^{k + 1} = \arg {min}_{f} \frac{1}{2} ‖ g - H f ‖_{2}^{2} + \frac{ρ}{2} ‖ f - (z^{k} - u^{k}) ‖_{2}^{2},

z^{k + 1} = \arg {min}_{f} λ R (z) + \frac{ρ}{2} ‖ z - (f^{k + 1} + u^{k}) ‖_{2}^{2},

u^{k + 1} = u^{k} + (f^{k + 1} - z^{k + 1}),

f^{k + 1} = (z^{k} - u^{k}) + H^{⊤} (g - H (z^{k} - u^{k})) ⊘ (Diag (H H^{⊤}) + ρ),

z^{k + 1} = D_{{\hat{σ}}_{k}} (f^{k + 1} + u^{k}),

u^{k + 1} = u^{k} + (f^{k + 1} - z^{k + 1}),

Abstract

Supplementary Material (17)

Cited By

Figures (11)

Equations (15)

Optics Express