Computer-generated holography (CGH) holds transformative potential for a wide range of applications, including near-eye displays used in virtual and augmented reality systems and automotive heads up displays (HUDs) [1]. In these applications, holographic displays offer unprecedented image brightness and dynamic range with ultra low power budgets and enable 3D display modes optimizing perceptual realism and user comfort. However, a compact device form factor is crucial for such displays to be adopted in the aforementioned applications.

CGH algorithms have recently made significant progress in optimizing image quality and algorithm runtime [2–7]. Yet, the physical optical image formation of a holographic display includes high diffraction orders (HDOs), which must be optically filtered in all of these approaches. Such filters typically use 4f systems or related opto-mechanical components between the phase-only spatial light modulator (SLM) generating the hologram and the viewer or eyepiece. This requires opto-mechanical bulk, often preventing the small form factors required for near-eye display or HUD applications.

To overcome this challenge, a few recent works have attempted to compactly handle these HDOs. Bang et al. [8] demonstrated a volume grating that could compactly filter out the high-order copies, but it was designed for a single wavelength. Park et al. [9] and Kim et al. [10] also compactly addressed the high-order copies with non-periodic pinhole filters, but this technique greatly reduces light efficiency and has been demonstrated only on very simple scenes with tens of scene points. Li et al. [11] utilized HDOs by optimizing different phase patterns for each of the orders and using time multiplexing and controllable shutters to show one optimized order at a time. Unfortunately, this approach still uses a bulky 4f filtering system with the additional system complexity of a controllable filter and requires fast SLMs for time multiplexing.

 

Fig. 1. Diagram of unfiltered holography setup. L1, collimation lens; P, polarizer; BS, beam splitter; L2-L3, 4f system; F, filter. The phase pattern, $\phi$, displayed on the SLM produces a wavefront that propagates in free space to form ${u_{\rm{unfilt}}}(\phi)$, including high diffraction orders, at the target plane. For the filtering needed by prior works, a 4f system with a filter in the Fourier plane is placed.

Download Full Size | PDF

 

Fig. 2. Diagram illustrating computation of HOGD. Each wavefront is depicted with its amplitude (black) and phase (green). The top-right values indicate the range in the spatial or frequency domain for each dimension, where $N$ is the number of SLM pixels along the dimension, and $p$ is the pixel pitch. The Fourier transform of the SLM wavefront, ${e^{{i\phi}}}$, is repeated in the frequency domain to account for the effect of high diffraction orders. This is multiplied with a propagation kernel with sinc attenuation, $A({f_x},{f_y})$ to account for the effect of the finite SLM pitch. The resulting product is converted back to the primal domain to get the target plane wavefront with HDOs, ${u_{\rm{unfilt}}}(\phi)$. Then the L2 loss is calculated between the target image ${a_{\rm{target}}}$ and $|{u_{\rm{unfilt}}}(\phi)|$, and the error is backpropagated to update the phase pattern via gradient descent.

Download Full Size | PDF

Here, we introduce a computational optics approach that removes the need for optical filtering altogether using optimized algorithms. For this purpose, we propose the high-order gradient descent (HOGD) algorithm, which uniquely optimizes phase patterns accounting for all diffraction orders simultaneously. We demonstrate that the HOGD algorithm yields superior image quality without the need for optical filtering, thereby unlocking the potential for high image quality with compact holographic displays (Fig. 1). We prototype such an unfiltered holographic display and demonstrate the benefits of HOGD.

We use an iterative phase-retrieval-type approach to optimize phase-only SLM patterns for a target intensity. We first introduce this approach for systems using optical filtering. Specifically, we combine variants of the stochastic gradient descent (SGD) algorithm with the angular spectrum method (ASM) to model free-space wave propagation from the SLM plane to the target plane [3]. For an SLM pattern, $\phi$, the SLM wavefront is modeled as ${e^{{i\phi}}}$. The ASM then uses a Fourier transform, ${\cal F}$, to propagate this wavefront over a distance, $z$, to the filtered wavefront at the target plane, ${u_{\rm{filt}}}$, as

For the HOGD algorithm, we modify this ASM propagation to model the HDOs produced by the sampling resolution of the SLM and the attenuation due to the finite pixel pitch of the SLM. The spatial domain sampling resolution of the SLM produces shifted copies of ${\cal F}({e^{{i\phi}}})$ in the frequency domain. Additionally, the finite square pixel pitch produces an attenuation of these frequency domain copies with a 2D Sinc function. For an SLM with pixel pitch $p$, this results in the modified wave propagation model, where $\alpha$ is the number of orders considered:

This mathematical formulation is illustrated visually through the pipeline in Fig. 2. As shown in the diagram, the HDOs are integrated into the propagation pipeline by tiling the frequency domain representation of the SLM wavefront. This tiled representation is then multiplied in the frequency domain by the 2D Sinc amplitude to account for the pixel pitch and the ASM kernel phase extended to propagate the high-order frequencies.

With this HDO wave propagation model, all of the orders can be optimized together to produce high image quality without optical filtering. This is performed by minimizing the squared L2 loss in Eq. (3) with gradient descent [3], where $s$ is a global scale factor and ${a_{\rm{target}}}$ is the target amplitude. This HOGD algorithm leverages the flexibility of the gradient descent CGH algorithm from Peng et al. [3] to optimize phase patterns through our modified propagation model:

As discussed by Chen et al. [12], the amplitude of $|{u_{\rm{unfilt}}}(\phi)|$ is computed after padding in the frequency domain to avoid aliasing. This objective is minimized purely using the propagation model from Eq. (2) for the HOGD algorithm. We also adapt the recently proposed camera-in-the-loop (CITL) optimization strategy for the unfiltered holography setup to overcome the mismatch between ideal wave propagation and the physical optics [3,5]. To adjust the CITL algorithm, the gradients of our HDO propagation model are used instead of the gradients of the ASM propagation model while minimizing the difference between the camera captured amplitude and the target amplitude.

The performance of our HOGD algorithm is compared against prior CGH algorithms on a $1{,}080 \times 1{,}920$ 8-bit phase-only SLM with a pixel pitch of 6.4 µm with illumination wavelengths of 636.4 nm, 517.7 nm, and 440.8 nm for red, green, and blue, respectively. The optimizations for the three color channels are processed sequentially. During the optimizations, the central $3 \times 3$ orders ($\alpha = 3$) are simulated because due to the frequency domain attenuation, orders higher than this are negligible.

 

Fig. 3. Comparison of simulated PSNR of reconstructed amplitudes produced by different CGH methods with and without optical filtering at varying distances from the SLM. The full set of images used to compute these PSNR values can be seen in Supplement 1.

Download Full Size | PDF

 

Fig. 4. Simulated unfiltered results with numbers indicating PSNR with respect to the target image. (Top) Natural scene exemplifying how the HOGD algorithm can effectively optimize phase patterns to conceal the energy and interference produced by the HDOs. (Bottom) Sparse resolution chart illustrating the challenge with hiding high-order copies in sparse scenes with high frequency content.

Download Full Size | PDF

Figure 3 evaluates the simulated performance, averaged over a small set of target images, of prior CGH algorithms with and without ideal filtering at different distances from the SLM. The double phase method (DPM) produces high frequency amplitude copies within the first order that need to be filtered to enable amplitude and phase modulation on a phase-only SLM [2,6]. This additional filtering enables images to be created directly at the filtered SLM plane, which is relayed through the 4f filtering system but produces a large degradation in image quality when the filtering is not present. The SGD algorithm can be flexibly used with different levels of filtering. As a result, with filtering, it can also produce high-quality images at the filtered SLM plane. Additionally since the SGD algorithm requires only optical filtering of high-order copies, it can work with less filtering than the DPM. Consequently, it can produce better image quality than the DPM when the filtering is removed.

Compared to these prior methods, the HOGD algorithm produces the highest image quality at all distances from the SLM when filtering is not present. In the unfiltered scenario, the amplitude of the illumination wavefront is unchanged at the SLM plane, so some distance from the SLM is needed to produce high image quality as shown in Fig. 3. For the remainder of the results, we chose an image distance of 35 mm from the SLM to have a compact holographic display with good image quality.

Figure 4 illustrates two simulated comparisons between the algorithms in unfiltered setups for a monastery scene and a sparse scene showing a resolution chart. Here we can see the significant reduction in image quality due to the unwanted amplitude copies when using the DPM without filtering. We can also see the unwanted interference noise produced by the high-order copies with the SGD algorithm. When looking at the monastery scene, we can see how the HOGD algorithm can very effectively “hide” the high-order copies using constructive and destructive interference to produce a desired natural scene.

The sparse resolution chart scene in the second row of Fig. 4 illustrates a challenging case for unfiltered holography. In this sparse scenario, the high-order copies cannot simply be hidden within the rest of the scene. As a result, even with the HOGD algorithm, the high-order copies are visible. However, the HOGD algorithm does reduce the amplitude of the HDOs by primarily using frequencies that send less energy to the high-order copies, as is discussed further in Supplement 1.

For our unfiltered holography display prototype, we use a Holoeye LETO SLM, a FISBA RGBeam fiber-coupled module with three optically aligned laser diodes, and a Nikon 50 mm lens for the eyepiece that is placed after the SLM to relay the target images to virtual distances away from the viewer. These components match the simulated parameters discussed prior. The camera used to capture results is a FLIR Grasshopper3 2.3 MP color USB3 vision sensor with a Canon EF 50 mm lens. Additional hardware details are included in Supplement 1.

Figure 5 shows results generated with each of the CGH algorithms captured on the unfiltered holography prototype. These captured results exhibit the same trends discussed for the simulated results. The amplitude copies and high-order interference noise can be seen degrading the results for the DPM and SGD methods, respectively. Compared to these methods, the HOGD result qualitatively appears to have much less noise and also quantitatively has better performance, as measured by the peak signal-to-noise ratio (PSNR). However, due to the mismatch between ideal wave propagation and physical optics, the captured performance does not quite match the simulated performance. The CITL approach reduces this gap using a camera in the loop, and the comparison between the SGD-CITL and HOGD-CITL results, using the ASM and HDO propagation models, respectively, highlights the performance gains due to the more accurate gradients of our HDO propagation model.

 

Fig. 5. Experimentally captured results on unfiltered holography setup with numbers indicating PSNR with respect to the target image. The HOGD algorithm shows about a 1 dB improvement over the SGD algorithm, and the HOGD-CITL algorithm performs more than 1 dB better than the SGD-CITL algorithm in both examples. This improved performance due to the reduction in unwanted interference with the novel algorithms is clearly exhibited in the zoomed crops under each example. Additional captured results, which are viewable in Supplement 1, exhibit similar patterns.

Download Full Size | PDF

In summary, these results illustrate that holographic displays without filtering can produce high image quality with properly modeled HDOs. Our HOGD and HOGD-CITL algorithms demonstrate marked performance improvements over prior algorithms in the absence of optical filtering. As discussed, our method does have some challenges with sparse scenes but performs well for natural images. The unfiltered holography setup requires a small propagation distance between the SLM and target images. However, as SLM pixel pitches shrink, this distance will be significantly reduced. As detailed in Supplement 1, with smaller commercially available pixel pitches, 30 db simulated performance is possible with 20 mm of propagation.

Our unfiltered holography algorithms enable simpler, more compact holographic displays by removing the need for optical filtering. Future work could integrate this HDO propagation model into learned propagation models [3,4,7] to enable high unfiltered image quality without a camera in the loop. This could be especially helpful with smaller SLM pixels that produce further non-ideal behavior such as increased pixel cross talk. Additionally, while we did not focus on fast hologram synthesis, prior work has shown that propagation models could be used to train real-time hologram synthesis networks [3,6]. Extension of the proposed techniques for 3D holograms is an important step for future research, which is beyond the scope of this Letter. Finally, this algorithm could be used with recently proposed wearable form factor holographic displays [2,13] to improve the image quality provided by these displays.