Speckle-free compact holographic near-eye display using camera-in-the-loop optimization with phase constraint

Lizhi Chen; Runze Zhu; Hao Zhang

doi:10.1364/OE.475066

1. Introduction

Holographic display is a promising technique to provide compact near-eye reconstructions with comfortable viewing experience for augmented and virtual reality [1–3]. Recently, numerous researches have been conducted to illustrate the potential of computer holography to optimize some intrinsic challenges in near-eye display, such as focus cues, form factor, aberration correction and vision correction [4–6]. Computer holography allows presenting dynamic three-dimensional scenes by reconstructing the object wavefront through the spatial light modulators (SLMs). As the commercial SLMs cannot modulate the amplitude and phase components of incident light independently, the object wavefront is necessarily required to be encoded into amplitude-only or phase-only computer-generated holograms (CGHs). The phase-only CGHs have been widely used in holographic near-eye display due to the merits of high optical efficiency, flexible configuration and free of conjugate image [7–9]. Whereas, the phase-only CGH synthesis is an ill-posed inverse problem owing to the loss of amplitude component. Optimizing the phase-only CGHs is an important issue in holographic near-eye display for achieving high image fidelity.

During past decades, various methods have been proposed to optimize the phase-only CGHs, which can be mainly categorized into non-iterative [10–14] and iterative methods [15–24]. Non-iterative methods can synthesize the phase-only CGHs at a fast speed; whereas, the generality of these methods would be easily degraded due to the lack of effective constraint. Iterative methods are more effective for improving the image quality by iteratively optimizing the phase-only CGHs according to the specific target images. Early iterative methods include the error reduction algorithm, which was introduced by Gerchberg and Saxton to solve the phase retrieval problem [15]. The error reduction algorithm was widely used to optimize the phase-only CGHs due to its simplicity. Extensions of such algorithm include the input-output algorithm [16], and others with various relaxations [17–19]. These algorithms try to enhance the convergence of the error reduction algorithm by broadening the optimization space. Recently, the gradient descent algorithm has been introduced to optimize the phase-only CGHs [20–24]. In the gradient descent algorithm, the loss function of the reconstructed field is calculated in each iteration and back-propagated to update the phase-only CGHs based on gradient descent strategy. The loss function can be customized in the gradient descent algorithm, which provides more flexibility in the optimization process.

The mismatch between the optical wave propagation of holographic reconstruction system and its corresponding numerical propagation model would degrade the image quality, such as wavefront distortion and non-uniform illumination. In order to narrow the gap between the numerical simulation and optical reconstruction, the camera-in-the-loop (CITL) optimization has been proposed to improve the image quality by using a hybrid physical-digital wave propagation model [25–28]. In the CITL method, the error between the optically captured results and numerical target images is calculated and back-propagated to update the phase-only CGHs on the SLM. Recently, the off-axis CITL method with noise reduction strategy is introduced to solve the direct current noise problem of the SLM [29]. The Gaussian blur is applied to the optically captured results as well as the numerical target images to suppress the high-frequency noise and enhance the convergence. Whereas, in these CITL methods, the phase randomness of reconstructed field was not discussed in-depth. Random phase distribution usually induces extensive phase singularities, also known as optical vortices, which would cause severe speckle noise in the reconstructed field [30]. The phase singularities cannot be self-annihilated in the first-order optimization algorithm and would bring the iterations into the vortex stagnation [31]. Even with the CITL optimization, the speckles cannot be effectively eliminated if the phase distribution of reconstructed field was not handled properly.

In this study, we present a compact holographic near-eye display system with high-quality speckle-free optical reconstructions by using the CITL optimization with phase constraint strategy. The reconstruction properties of the compact holographic near-eye display system are comprehensively analyzed. In the compact holographic near-eye display system, the band-limited angular-spectrum method (ASM) is used to model the inverse diffraction reconstruction from the SLM to the reconstructed images. The CITL method with phase constraint strategy is proposed to iteratively optimize the phase-only CGHs for the compact holographic near-eye display system based on the in-system optical feedback. The phase constraint is embedded into the whole iterative process from the CGH initialization to the CGH optimization, which helps to solve the vortex stagnation problem by annihilating the phase singularities in the reconstructed field. The proposed CITL method with phase constraint can provide effective controls of the reconstructed intensity field and suppress the speckle noise caused by the phase singularities. Numerical and optical results confirm that our proposed method can be used to synthesize the phase-only CGHs with excellent image fidelity for the holographic near-eye display system in a compact setup.

2. Compact holographic near-eye display with CITL optimization

Figure 1 shows the schematic of the CITL method for optimizing phase-only CGHs in the compact holographic near-eye display system. Here, the single-mode (SM) fiber laser beam is collimated by the lens to illuminate the phase-only SLM. The polarization direction of the SM- fiber laser meets the requirement of the SLM. The phase-only CGH is uploaded onto the SLM. It deserves to be noted that the lens acts as a collimator as well as an eye-piece, which helps to achieve a compact optical configuration. By the lens, the image reconstructed from the SLM is imaged to a specific distance from the observer. The filter is used here to eliminate the high-order diffraction light from the SLM. During the CGH optimization, the loss function is constructed from the error between the optically captured images and numerical target images. The loss function is back-propagated to update the phase-only CGHs on the SLM by the utilization of stochastic gradient descent (SGD) algorithm.

Fig. 1. Schematic of the CITL method for optimizing phase-only CGHs in the compact holographic near-eye display system. SM-Fiber, single-mode fiber; BS, beam splitter; SLM, spatial light modulator.

Download Full Size | PDF

Figure 2(a) shows the unfolded optical path of the proposed compact holographic near-eye display system. In the unfolded optical path diagram, the single lens in the proposed compact near-eye display system shown in Fig. 1 is illustrated as a separated collimator and an eye-piece on both sides of the SLM, and the beam splitter is omitted. In Fig. 2(a), the coherent light from the point light source is collimated to illuminate the SLM. The image reconstructed from the SLM is observed by the human eye from the eye-piece. The imaging relation of the proposed compact holographic near-eye display system is shown in Fig. 2(b). The eyebox is located at the back focal plane of the eye-piece. The image reconstructed from the SLM is located at the plane P. The distance between the plane P and the eye-piece is less than the focal length of the eye-piece. The plane P is imaged to the plane P’ by the eye-piece. The enlarged observed image is located at the plane P’. According to the imaging relation, the distance between the plane P’ and the eye-piece is calculated as

(1)$$l^{\prime} = \frac{{lf}}{{f - l}}, $$

where l represents the distance between the plane P and the eye-piece, f represents the focal length of the eye-piece. The size of the enlarged observed image is calculated as

(2)$${L_x}^{\prime} = {L_x}\frac{{l^{\prime}}}{l}, $$

where L_x represents the size of the image reconstructed from the SLM.

Fig. 2. Optical principle of the compact holographic near-eye display system: (a) unfolded optical path diagram, (b) imaging geometry. l, the distance between the image reconstructed from the SLM and the eye-piece; z, the distance between the reconstructed image and the SLM; f, the focal length of the eye-piece; l’, the distance between the observed image and the eye-piece; L_x, the size of image reconstructed from the SLM; L'_x, the size of observed image.

Download Full Size | PDF

The plane P lies behind the SLM along the light path. Here, the inverse diffraction calculation requires to be used to model the ideal wave propagation from the SLM to the plane P. In the ASM [32], the inverse diffraction calculation is easy to be implemented by changing the sign of the propagation distance. Herein, we use the ASM to model the ideal inverse diffraction reconstruction from the SLM to the plane P as

(3)$$\begin{array}{l} \textrm{ }g(\varphi )= {\mathrm{{\cal F}}^{ - 1}}\{{\mathrm{{\cal F}}\{{{a_{\textrm{src}}}{e^{i({\varphi + {\phi_{\textrm{src}}}} )}}} \}\cdot \mathrm{{\cal H}}({{f_x},{f_y}, - z} )} \},\\ \mathrm{{\cal H}}({{f_x},{f_y}, - z} )= \left\{ {\begin{array}{ll} {{e^{ - ikz\sqrt {1 - {{({\lambda {f_x}} )}^2} - {{({\lambda {f_y}} )}^2}} }},}&{if\sqrt {f_x^2 + f_y^2} \le \frac{1}{\lambda },}\\ {0,\textrm{ }}&{\textrm{otherwise}\textrm{. }} \end{array}} \right. \end{array}$$

where φ is the phase distribution of phase-only CGH on the SLM, a_src and ϕ_src are the amplitude and phase distribution of illumination light at the SLM plane, respectively, g(φ) is the complex amplitude distribution at the plane P, ${\mathrm{\cal F}}$ and ${\mathrm{\cal F}}$^-1 represent the Fourier transform and inverse Fourier transform, respectively, Η(f_x, f_y, -z) is the angular spectrum transfer function, (f_x, f_y) are spatial frequencies, k represents the wavenumber equal to 2π/λ, λ represents the wavelength, and z is the propagation distance between the SLM and the plane P.

In order to improve the image quality for the presented compact holographic near-eye display system, the CITL method is used here to optimize the phase-only CGHs, shown in Fig. 1. The optimization of phase-only CGHs is performed by solving the problem

(4)$$\mathop {\textrm{minimize}}\limits_\varphi \textrm{ }\mathrm{{\cal L}}({s|{\hat{g}(\varphi )} |,{a_{\textrm{target}}}} ), $$

where φ is the phase distribution of phase-only CGHs, Λ is a loss function which defines the error between the optically captured image and the numerical target image, $|{\mathrm{\hat{g}(\varphi )}} |$ is the amplitude distribution of the optically captured image, a_target is the amplitude distribution of the numerical target image, and s is a learnable scale factor that accounts for the fact that the amplitude values of optically captured images and numerical target images might distribute in a different range. The above problem can be solved by the SGD algorithm, and the update rules are given as

(5)$${\varphi ^{({k + 1} )}} = {\varphi ^{(k )}} - \alpha \frac{{\partial \mathrm{{\cal L}}}}{{\partial \varphi }}, $$

where α is an adaptive step size in the SGD algorithm. In the CITL method, the gradient of wave propagation in the actual holographic reconstruction system is not accessible. Hence, the gradient $\frac{{\partial \mathrm{{\cal L}}}}{{\partial \varphi }}$ is approximated as $\frac{{\partial \mathrm{{\cal L}}}}{{\partial \mathrm{\hat{g}}}}\cdot \frac{{\partial \textrm{g}}}{{\partial \varphi }}$. Here, $\textrm{g}$ is the ideal wave propagation model in Eq. (3).

3. Phase constraint strategy for speckle-free CGH optimization

In our previous papers [33,34], the bandwidth properties of Fresnel diffraction field have been analyzed comprehensively. It reveals that the global bandwidth of the intensity distribution of Fresnel diffraction field is twice the maximum local bandwidth of the complex amplitude distribution of Fresnel diffraction field. Here, in the proposed compact holographic near-eye display system, the maximum local bandwidth of the complex amplitude distribution of reconstructed diffraction field is equal to which of the modulated wave field at the SLM plane, which is decided by the pixel pitch of SLM. Hence, the global bandwidth of the reconstructed intensity field is twice the maximum local bandwidth of the complex amplitude distribution of modulated wave field at the SLM plane. In the conventional ASM, the numerical sampling interval remains the same during the diffraction calculation, which cannot correctly sample the reconstructed intensity field according to the Nyquist sampling theorem. To ensure the correct sampling of reconstructed intensity field and bridge the gap between the numerical simulation and optical reconstruction, the band-limited ASM [35] is used here, and the frequency components larger than half of maximum frequency of diffraction light are filtered out in the frequency domain. Whereas, the frequency filtering imposes the bandwidth constraint to the reconstructed complex amplitude field, which would always bring the CGH optimization into the vortex stagnation if the phase distribution of reconstructed diffraction field was not handled properly [31].

Figure 3 shows the numerical simulation results. The number of iterations was set as 500 in the SGD algorithm, and a uniform random phase from a range of [-2.0, 2.0] was set as the initial guess of phase-only CGHs to start the CGH optimization. It shows that severe intensity fluctuations with dark spots occur in the reconstructed field, which would cause severe speckle noise in the optical reconstruction. The appearance of dark spots had been investigated and were found that they are associated with the phase singularities extensively existing in the reconstructed field, which are also known as the optical vortices [31,36]. The optical vortex is characterized by the wavefront having helical phase structure [37]. The phase singularities cannot be self-annihilated in the first-order optimization algorithm, and would cause the vortex stagnation problem. Even with more iterations, the speckles caused by the phase singularities would still occur in the reconstructed field. The phase singularities can be eliminated by globally judging and constructing an annihilating vortex field [31,36], which would help to solve the vortex stagnation problem. Whereas, the global judgement and annihilation of the phase singularities is time-consuming.

Fig. 3. Numerical simulation results: (a) intensity distribution and (b) phase distribution of the reconstructed field.

Download Full Size | PDF

From Fig. 3(b), it can be seen that the higher frequency components of reconstructed phase field are always located around the phase singularities. In order to solve the vortex stagnation problem caused by the phase singularities, a possible means is to smooth the phase distribution of reconstructed field. By smoothing the phase distribution of reconstructed field, the phase singularities could be annihilated. It would be helpful to smooth the phase distribution of reconstructed field by using a constant phase as the initial guess of phase-only CGHs to start the CGH optimization. While, the random initialization is vital to enhance the convergence and search a better solution in the gradient descent algorithm [38].

Here, the phase constraint strategy is proposed to annihilate the phase singularities in the reconstructed field for speckle-free CGH optimization. The phase constraint strategy includes two different operations. One is to use a band-limited pseudo-random phase as the initial guess of phase-only CGHs to start the CGH optimization. Another is to introduce the phase regularization of reconstructed field during the CGH optimization. Specifically, the bandwidth constraint iterative Fourier transform algorithm (BC-IFTA) [19] is used here to obtain an optimized band-limited pseudo-random phase, which is similar with the method proposed in our previous paper [11]. A random phase uniformly distributed in a range of [-1.0, 1.0] is utilized as the initial phase to iteratively optimize a band-limited pseudo-random phase by the BC-IFTA. The optimized band-limited pseudo-random phase can be used as the initial guess of phase-only CGHs for any target images. Although the phase initialization for the CGH optimization has been discussed recently [39], the purpose of phase initialization is different in our proposed method. Here, the optimized band-limited pseudo-random phase is used as the initial guess of phase-only CGHs to ensure the bandwidth constraint property of reconstructed complex amplitude field and reduce the sum of phase singularities in the reconstructed field. Whereas, the pseudo-random initial phase cannot control the generated phase singularities during the CGH optimization.

In the SGD algorithm, the loss function can be customized, which provides more flexibility in the optimization process. Herein, the phase regularization is introduced into the loss function to constrain the phase distribution of reconstructed field, which could effectively annihilate the phase singularities in the reconstructed field during the CGH optimization. The phase regularization has been proposed recently to improve the image quality in the holographic display [27]. Here, the phase regularization is used to annihilate the phase singularities in the reconstructed field, which helps to solve the vortex stagnation problem. The phase regularization of reconstructed field is calculated as $\Delta \mathrm{\Phi \{\ g(\varphi )\}\ }$, where g(φ) is the complex amplitude distribution of reconstructed field, $\mathrm{\Phi \{\ }\cdot \textrm{\} }$ is the phase extraction operation of the complex amplitude field, and $\Delta $ denotes the Laplace operator used to calculate the second derivative of the phase values. By the phase constraint strategy, the phase singularities can be effectively annihilated in the reconstructed field, which helps to solve the vortex stagnation problem and achieve speckle-free CGH optimization.

The CITL method with phase constraint strategy is proposed to optimize the image quality and achieve high-quality speckle-free optical reconstructions for the compact holographic near-eye display system. The complete loss function is constructed from both the amplitude and phase constraints of the reconstructed field and given by

(6)$$\mathrm{{\cal L}} = {\mathrm{{\cal L}}_a} + {\mathrm{{\cal L}}_\phi } = {\left\|{({s \cdot |{\hat{g}(\varphi )} |- {a_{\textrm{target}}}} )\circ m} \right\|_2} + \tau {\left\|{\Delta \Phi \{{g(\varphi )} \}\circ m} \right\|_1}, $$

where $|{\mathrm{\hat{g}(\varphi )}} |$ is the amplitude distribution of optically captured images, a_target is the amplitude distribution of numerical target images, g(φ) is the complex amplitude distribution of reconstructed field in the numerical diffraction calculation, $\Delta \mathrm{\Phi \{\ }\cdot \textrm{\} }$ denotes the phase regularization of reconstructed field, s is an adaptive scale factor that accounts for the difference between the energy of optically captured results and numerical target images, ${\circ} $ denotes element-wise multiplication, m is a binary mask which defines the signal region and non-signal region of the reconstructed field, ||·||₂ denotes L2-norm, ||·||₁ denotes L1-norm, and $\tau $ is a user-defined regularization weight which balances the two loss terms. The SGD algorithm is used to optimize the phase-only CGHs according to the given loss function. In order to broden the optimization space, the weighted constraint strategy is used here [18]. During the CGH optimization, only the signal region of the reconstructed field would be constrained, which could further improve the image quality in the signal region by introducing the amplitude freedom of non-signal region. To reduce the influence of the flickers of illumination light in the optical experiments, the adaptive scale factor s is used here rather than a learnable scale factor in the original CITL method, which is set as

(7)$$s = \frac{{\sum {|{\hat{g}(\varphi )} |\circ {a_{\textrm{target}}}} }}{{\sum {{{({{a_{\textrm{target}}}} )}^2}} }}. $$

Here, the adaptive scale factor s is designed to speed up the convergence of SGD algorithm by reducing the difference between the optically captured image and numerical target image, which would not affect the visual quality of optical results.

4. Results

To validate the feasibility of our proposed method, numerical simulations and optical experiments have been conducted. Several evaluation functions are used here to evaluate the image quality quantitatively. The first one is peak signal-to-noise ratio (PSNR) between the target images and the reconstructed images. The PSNR is calculated as

(8)$$\begin{array}{c} \textrm{RMSE} = \sqrt {\frac{1}{{MN}}\sum\limits_{m = 1}^M {\sum\limits_{n = 1}^N {{{[{{I_{\textrm{rec}}}({m,n} )- {I_{\textrm{target}}}({m,n} )} ]}^2}} } } \\ \textrm{ PSNR} = 20 \times {\log _{10}}\left( {\frac{{{2^b} - 1}}{{\textrm{RMSE}}}} \right) \end{array}, $$

where I_rec(m,n) is the intensity distribution of reconstructed image, and I_target(m,n) is the intensity distribution of target image, M and N are the horizontal and vertical resolution of images, and b is the bit depth of images. The PSNR indicates how well the reconstructed image agrees with the target image. The higher the value of PSNR is, the better the reconstruction quality will be.

Another evaluation function is the speckle contrast (SC) [40] and it is given by

(9)$$\textrm{SC} = \frac{\sigma }{\mu }, $$

where μ is the mean value of intensity distribution of the region of interest (ROI), and σ is the standard deviation of intensity distribution of the ROI.

In the numerical simulations, the resolution and sampling interval of the phase-only CGHs is set as 1920 × 1152 and 9.2 µm, respectively. The resolution of target images is set as 1200 × 800. The target images are zero-padded to 1920 × 1152 during the CGH optimization. The resolution of signal region is set as 1200 × 800. The band-limited ASM is used to model the numerical inverse diffraction reconstruction from the SLM plane to the reconstructed image plane. The inverse diffraction distance is set as 5 cm. The SGD algorithm is used to optimize the phase-only CGHs. The Adam optimizer is implemented for the SGD algorithm [41]. Python 3.8.0 and PyTorch 1.10.0 are used to achieve the CGH optimization. The central processing unit (CPU) and graphics processing unit (GPU) used here are Intel Core i7-10700F processor and NVidia GeForce RTX 2060 Super with CUDA version 11.3.1, respectively. The number of iterations is set as 500 for the numerical simulations. In the proposed method with phase constraint, the regularization weight $\tau $ is set as 0.1 in the loss function.

Figure 4 shows the numerical simulation results of the SGD algorithm with different initial guess of phase-only CGHs and the phase regularization. The PSNR is calculated for the signal region, and the SC is calculated inside the orange box. In addition, the phase singularities are detected in the reconstructed field to further validate the effectiveness of phase constraint strategy according to the vortex search algorithm [36]. The vortex search algorithm evaluates the line integral $\mathrm{\oint }\nabla \mathrm{\Phi }\cdot \textrm{d}s = m2\mathrm{\pi } \ne 0$, where m is an integer determining the topological charge of the dislocation, $\nabla \mathrm{\Phi }$ represents the phase gradient of reconstructed field, and $\textrm{d}s$ represents the line element of closed integral path on the phase distribution. It can be seen that severe intensity fluctuations with dark spots occur in the reconstructed images of the SGD algorithm with an initial random guess from a range of [-2.0, 2.0] and without phase regularization, which degrades the image quality and would cause the speckle noise in the optical reconstruction. The phase distribution of the reconstructed field contains lots of high-frequency components, which induces a large number of phase singularities. The appearance of dark spots is related to the phase singularities extensively existing in the reconstructed field. By using a band-limited pseudo-random phase as the initial guess of phase-only CGHs in the SGD algorithm, the reconstructed phase field becomes smooth, and the most phase singularities can be eliminated, which helps to solve the vortex stagnation problem and improve the image quality. Whereas, there are still some high-frequency components in the phase distribution of reconstructed field, which would induce the speckle noise in the out-of-focus images. It shows that the SGD algorithm with phase regularization can achieve high-quality reconstructions with high PSNR and low SC. By the utilization of phase regularization, the reconstructed phase field becomes smoother, and the phase singularities can be effectively annihilated, which helps to solve the vortex stagnation problem and eliminate the severe speckles in the reconstructed images. Although the PSNR of the reconstructed image would slightly decrease by the phase regularization when using a band-limited pseudo-random phase as the initial guess, the difference is negligible in the visual perception.

Fig. 4. Numerical simulation results: reconstructed images of the SGD algorithm with (a) initial random guess from [-2.0, 2.0], (b) initial random guess from [-2.0, 2.0] and phase regularization, (c) initial band-limited pseudo-random guess, and (d) initial band-limited pseudo-random guess and phase regularization; reconstructed phase distributions of the SGD algorithm with (e) initial random guess from [-2.0, 2.0], (f) initial random guess from [-2.0, 2.0] and phase regularization, (g) initial band-limited pseudo-random guess, and (h) initial band-limited pseudo-random guess and phase regularization.

Download Full Size | PDF

Figure 5 shows the numerical results at different reconstructed depths. During the CGH optimization, the inverse diffraction distance from the SLM to the reconstructed image plane is set as 5 cm. The optimized CGHs are used to calculate the reconstructed intensity field at different reconstructed depths by the inverse diffraction calculation. The reconstructed depths are set as 5 cm, 5.5 cm and 6 cm, respectively. It shows that when the reconstructed depth is set as 5 cm, the reconstructed images are of highest visual quality. When the reconstructed plane is away from the target image plane, the reconstructed images become blurred, which is known as out-of-focus behavior. In the SGD algorithm without phase regularization, the severe speckle noise would occur in the out-of-focus images due to the high-frequency phase components of reconstructed field. It can be seen that only the initial band-limited pseudo-random phase cannot effectively suppress the speckle noise in the out-of-focus images. By the phase regularization, the speckle noise in the out-of-focus images can be effectively suppressed by smoothing the phase distribution at the target image plane.

Fig. 5. Numerical simulation results at different reconstructed depths: reconstructed images of the SGD algorithm with initial random guess from [-2.0, 2.0] (first row), initial random guess from [-2.0, 2.0] and phase regularization (second row), initial band-limited pseudo-random guess (third row), initial band-limited pseudo-random guess and phase regularization (last row).

Download Full Size | PDF

We implement a prototype of the compact holographic near-eye display system for the optical experiments, shown in Fig. 6. We use a Meadowlark Optics 256 gray-scale level reflective phase-only SLM. The pixel pitch of the SLM is 9.2 µm, and the pixel resolution is 1920 × 1152. The laser is a FISBA READYBeam fiber-coupled module with three laser diodes with wavelengths of 450 nm, 520 nm and 638 nm. The lens 1 is an achromatic diverging lens with the focus length of -50 mm to enlarge the diverging angle of out-coupling light. The lens 2 is an achromatic converging lens with the focus length of 80 mm and used as a collimator as well as an eye-piece. The diverging light from the lens 1 is collimated by the lens 2 to illuminate the SLM. The amplitude and phase distribution of the incident light on the SLM can be measured and used in the numerical wave propagation model to bridge the gap between the numerical simulation and optical reconstruction, which would help to enhance the convergence of the CITL method. The image reconstructed from the SLM is imaged by lens 2 to the distance of 1.0 diopter (D) or 1.0 m from the camera. The filter is used to filter out the frequency components larger than half of maximum frequency of first-order diffraction. We use a FLIR Blackfly S USB3 51S5C color vision sensor with a Canon EF 50mm f/1.8 STM lens to capture the reconstructed images. An Arduino micro-controller is used to drive the Canon lens to enable the programmable varifocal acquisition. The gamma correction problem in holographic display has been discussed recently [25,40]. Here, we use the gamma-corrected intensity as the input target amplitude for CGH optimization. The experimentally captured images are calibrated by the planar homography method [25] for calculating the PSNR, SC, and the loss function in the CITL method. We compare the experimentally captured results of the numerical SGD algorithm, the CITL method without phase constraint and our proposed method with phase constraint. The number of iterations is set as 300 in all the methods.

Fig. 6. Prototype of the compact holographic near-eye display system. ND, neutral density filter; BS, beam splitter; SLM, spatial light modulator; CCD, charge-coupled device; Lens 1, achromatic diverging lens with focus length of -50 mm; Lens 2, achromatic converging lens with focus length of 80 mm; lens 3, Canon EF 50mm f/1.8 STM lens.

Download Full Size | PDF

Figure 7 shows the experimentally captured results of the numerical SGD algorithm, the CITL method without phase constraint and the proposed method with phase constraint. It shows that the captured images of the numerical SGD algorithm are degraded by severe speckle noise. In the CITL method without phase constraint, the image quality can be improved by considering the mismatch between the numerical simulation and optical reconstruction. Whereas, the speckle noise cannot be eliminated if the phase singularities are not annihilated in the reconstructed field. In our proposed method with phase constraint, the captured images are of highest visual quality. By smoothing the phase distribution of reconstructed field, our proposed method can effectively suppress the speckle noise by annihilating the phase singularities in the reconstructed field and achieve high-quality speckle-free optical reconstructions.

Fig. 7. Experimentally captured results of the numerical SGD algorithm(left), the CITL method without phase constraint (middle), and the proposed method with phase constraint (right).

Download Full Size | PDF

Figure 8 shows the evolution of experimentally captured results of the CITL method without phase constraint and the proposed method with phase constraint. The captured images of the CITL method without phase constraint are shown in the lower right triangle, and the captured images of our proposed method with phase constraint is shown in the upper left triangle. It can be seen that during the CGH optimization, the captured images gradually approach the target one in both the methods. In our proposed method with phase constraint, the speckle noise can be effectively suppressed by smoothing the phase distribution of reconstructed field, which helps to further improve the image fidelity. The PSNRs are plotted in Fig. 8(i), and the SCs are plotted in Fig. 8(j). It can be seen that the proposed method with phase constraint can achieve high-quality optical reconstructions with higher PSNR and lower SC. From Fig. 8(i) and (j), it shows that at the beginning of CGH optimization, the value of PSNR is higher and the value of SC is lower in our proposed method, which is thanks to the usage of band-limited pseudo-random initial guess of CGHs. During the CGH optimization, the convergence of our proposed method is faster than the CITL method without phase constraint strategy, which benefits from the phase regularization, qualitatively.

Fig. 8. Evolution of experimentally captured results of the CITL method without phase constraint ((a)-(h), lower right triangle) and the proposed method with phase constraint ((a)-(h), upper left triangle) at different iteration epochs. (i) Comparisons of the PSNRs, and (j) comparisons of the SCs.

Download Full Size | PDF

Figure 9 shows the experimentally captured results at different focus depth. The target image plane is located at a distance of 1.0 D from the camera. It shows that when the camera focuses at the distance of 1.0 D, the captured images are of highest visual quality. When the camera focuses at the planes away from the target image plane, the captured images become blurred, which is known as the out-of-focus behavior. Whereas, in the numerical SGD algorithm and the CITL method without phase constraint, the severe speckle noise would occur in the out-of-focus images, which is due to the high-frequency phase components of reconstructed field. In our proposed method with phase constraint, the speckle noise in the captured out-of-focus images can be effectively suppressed by smoothing the phase distribution at the target image plane.

Fig. 9. Experimentally captured results of the numerical SGD algorithm (top), the CITL method without phase constraint (middle) and the proposed method with phase constraint (bottom) at different focus depth.

Download Full Size | PDF

Figure 10 shows the full-color target images and experimentally captured results of the numerical SGD algorithm, the CITL method without phase constraint and our proposed method with phase constraint. The full-color images are captured as separate exposures for each wavelength and then combined in post-processing. It shows that the reconstructed images of our proposed method are of highest PSNR, which would help to enhance the visual quality and improve the color saturation for the true full-color holographic near-eye display.

Fig. 10. Full-color target images (first column) and captured results of the numerical SGD algorithm (second column), the CITL method without phase constraint (third column) and the proposed method with phase constraint (last column).

Download Full Size | PDF

We also performed the optical experiments for dual-plane holographic near-eye display. In the proposed method with phase constraint for the dual-plane holographic near-eye display, the loss function is constructed as

(10)$$\mathrm{{\cal L}} = {\mathrm{{\cal L}}_a} + {\mathrm{{\cal L}}_\phi } = \sum\limits_{k = 1,2} {{{\left\|{({s \cdot |{\hat{g}(\varphi )} |- {a_{\textrm{target}}}} )\circ {m^{(k)}}} \right\|}_2}} + \tau \sum\limits_{k = 1,2} {{{\left\|{\Delta \Phi \{{g(\varphi )} \}\circ {m^{(k)}}} \right\|}_1}}, $$

where m^(k) is a binary amplitude mask which defines the signal region at the kth plane. In the optical experiments, the Arduino micro-controller is used to drive the Canon EF 50mm lens to achieve the programmable multi-plane acquisition. Figure 11 shows the target images at two planes and experimentally captured results for the dual-plane holographic near-eye display. It can be seen that our proposed method with phase constraint can effectively suppress the speckle noise in both the in-focus and out-of-focus images by smoothing the phase distribution at the target image plane, which helps to improve the visual quality for the dual-plane holographic near-eye display. Compared with the numerical SGD algorithm and the CITL without the phase constraint strategy, the amount of defocus blur is limited in our proposed method due to the smooth phase profile of reconstructed field. The time-multiplexing holography with focal stack supervision [28] and the diffraction-engineered holography [42] can be combined with our proposed method to achieve real-world-like defocus blur in the future.

Fig. 11. Target images at two planes (first row) and experimentally captured results of the dual-plane holographic near-eye display by the numerical SGD algorithm (second row), the CITL method without phase constraint (third row) and the proposed method with phase constraint (last row).

Download Full Size | PDF

5. Discussion

In the numerical simulations and optical experiments, a random phase from a range of [-2.0, 2.0] rather than [-π, π] was chosen as the initial guess of phase-only CGHs in the numerical SGD algorithm and the CITL method without phase constraint as the comparisons. Generally, a random phase from a range of [-π, π] would be set as the initial guess of phase-only CGHs during the CGH optimization. Whereas, in the proposed compact holographic near-eye display system, the phase singularities widely existing in the random phase would bring the CGH optimization into the vortex stagnation, which would cause severe speckle noise in the optical reconstruction. The phase singularities exist in the form of the optical vortices. According to the property of phase singularities, a random phase from a larger range would induce more phase singularities in the reconstructed field, statistically. When a random phase from a range of [-π, π] is set as the initial guess of phase-only CGHs, the reconstructed phase field would be totally random due to the random interference of numerous vortex fields. In this case, the worst vortex stagnation problem would occur.

In order to illustrate the above problem, the numerical simulations of the SGD algorithm with random initial guess of phase-only CGHs from different range have been performed, and the numerical results are shown in Fig. 12. It shows that in the SGD algorithm with random initial guess from a range of [-π, π], the reconstructed image is of the lowest PSNR, and the reconstructed phase field contains the most phase singularities. Due to the random interference of numerous vortex fields, the reconstructed phase field is totally random, which would cause severe speckle noise in the reconstructed image. While, in the SGD algorithm with random initial guess from a range of [-2.0, 2.0], the reconstructed image contains less speckles, and the reconstructed phase field contains less phase singularities, which makes it easy to observe the relation of speckles and phase singularities. In addition, we found that a random phase from a range of [-π, π] could not effectively ensure the stable convergence of the CITL method in the optical experiments. Hence, we use a random phase from a range of [-2.0, 2.0] as the initial guess of phase-only CGHs in the numerical SGD algorithm and the CITL method without phase constraint as the comparisons.

Fig. 12. Numerical simulation results: reconstructed images of the SGD algorithm with initial random guess of phase-only CGHs from a range of (a) [-π, π], (b) [-2.5, 2.5], (c) [-2.0, 2.0], and (d) [-1.5, 1.5]; reconstructed phase distributions of the SGD algorithm with initial random guess of phase-only CGHs from a range of (e) [-π, π], (f) [-2.5, 2.5], (g) [-2.0, 2.0], and (h) [-1.5, 1.5].

Download Full Size | PDF

In the proposed method, the phase constraint strategy is used to annihilate the phase singularities in the reconstructed field and solve the vortex stagnation problem, which helps to suppress the speckle noise and enhance the image quality in the optical reconstructions. Whereas, a smooth reconstructed phase field will be generated by the phase constraint strategy, shown in Fig. 4(h). The smooth reconstructed phase distribution implies the limited angular spectrum range of the reconstructed field [39], which would sacrifice the effective eyebox in the proposed holographic near-eye display system. In order to enlarge the effective eyebox, a scanning-based eyebox expansion method can be utilized in the future [43].

6. Conclusion

In conclusion, we present a compact holographic near-eye display system with high-quality speckle-free optical reconstructions by the utilization of the CITL optimization with phase constraint strategy. A compact optical setup is presented and analyzed for holographic near-eye display. In the compact holographic near-eye display system, the band-limited ASM is used to effectively model the inverse diffraction reconstruction from the SLM to the reconstructed images. The CITL method with phase constraint strategy is proposed to optimize the phase-only CGHs to achieve speckle-free compact holographic near-eye reconstructions with high image fidelity. The phase constraint embedded into the whole CGH optimization process can effectively annihilate the phase singularities in the reconstructed field, which helps to solve the vortex stagnation problem. The proposed method with phase constraint can provide effective controls of reconstructed intensity field and achieve high-quality optical reconstructions with suppressed speckle noise. Numerical and optical experiments have been performed to validate that our proposed method can be used to achieve excellent image fidelity for the single-plane and dual-plane holographic near-eye display. By using the advanced depth camera to obtain the focal stack data of reconstructed images, the proposed method can be further extended to achieve high-quality multi-plane and continuous-depth holographic near-eye display in the near future.

Funding

National Key Research and Development Program of China (2021YFB2802100); National Natural Science Foundation of China (61875105, 62035003).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. J. H. Park and B. H. Lee, “Holographic techniques for augmented reality and virtual reality near-eye displays,” Light: Adv. Manuf. 3(1), 137–150 (2022). [CrossRef]

2. D. Pi, J. Liu, and Y. Wang, “Review of computer-generated hologram algorithms for color dynamic holographic three-dimensional display,” Light: Sci. Appl. 11(1), 231 (2022). [CrossRef]

3. C. Chang, K. Bang, G. Wetzstein, B. Lee, and L. Gao, “Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective,” Optica 7(11), 1563–1578 (2020). [CrossRef]

4. A. Mainmone, A. Georgiou, and J. Kollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. Graph. 36(4), 85 (2017). [CrossRef]

5. S. W. Nam, S. Moon, B. Lee, D. Kim, S. Lee, C. K. Lee, and B. Lee, “Aberration-corrected full-color holographic augmented reality near-eye display using a Pancharatnam-Berry phase lens,” Opt. Express 28(21), 30836–30850 (2020). [CrossRef]

6. D. Kim, S. W. Nam, K. Bang, B. Lee, S. Lee, Y. Jeong, J. M. Seo, and B. Lee, “Vision-correcting holographic display: evaluation of aberration correcting hologram,” Biomed. Opt. Express 12(8), 5179–5195 (2021). [CrossRef]

7. P. Sun, S. Chang, S. Liu, X. Tao, C. Wang, and Z. Zheng, “Holographic near-eye display system based on double-convergence light Gerchberg-Saxton algorithm,” Opt. Express 26(8), 30368–30378 (2018). [CrossRef]

8. S. Kazempourradi, E. Ulusoy, and H. Urey, “Full-color computational holographic near-eye display,” J. Inf. Disp. 20(2), 45–59 (2019). [CrossRef]

9. L. Shi, B. Li, C. Kim, P. Kellnhofer, and W. Matusik, “Towards real-time photorealistic 3D holography with deep neural networks,” Nature 591(7849), 234–239 (2021). [CrossRef]

10. P. W. M. Tsang and T. C. Poon, “Novel method for converting digital Fresnel hologram to phase-only hologram based on bidirectional error diffusion,” Opt. Express 21(20), 23680–23686 (2013). [CrossRef]

11. L. Chen, H. Zhang, L. Cao, and G. Jin, “Non-iterative phase hologram generation with optimized phase modulation,” Opt. Express 28(8), 11380–11392 (2020). [CrossRef]

12. D. Pi, J. Liu, and S. Yu, “Speckleless color dynamic three-dimensional holographic display based on complex amplitude modulation,” Appl. Opt. 60(25), 7844–7848 (2021). [CrossRef]

13. X. Sui, Z. He, G. Jin, and L. Cao, “Spectral-envelope modulated double-phase method for computer-generated holography,” Opt. Express 30(17), 30552–30563 (2022). [CrossRef]

14. D. Pi, J. Wang, J. Liu, J. Li, Y. Sun, Y. Yang, W. Zhao, and Y. Wang, “Color dynamic holographic display based on complex amplitude modulation with bandwidth constraint strategy,” Opt. Lett. 47(17), 4379–4382 (2022). [CrossRef]

15. R. W. Gerchberg and W. O. Saxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).

16. J. R. Fienup, “Iterative method applied to image reconstruction and to computer-generated holograms,” Opt. Eng. 19(3), 193297 (1980). [CrossRef]

17. A. Georgiou, J. Christmas, N. Collings, J. Moore, and W. A. Crossland, “Aspects of hologram calculation for video frames,” J. Opt. A: Pure Appl. Opt. 10(3), 035302 (2008). [CrossRef]

18. L. Chen, H. Zhang, Z. He, X. Wang, L. Cao, and G. Jin, “Weighted constraint iterative algorithm for phase hologram generation,” Appl. Sci. 10(10), 3652 (2020). [CrossRef]

19. L. Chen, S. Tian, H. Zhang, L. Cao, and G. Jin, “Phase hologram optimization with bandwidth constraint strategy for speckle-free optical reconstruction,” Opt. Express 29(8), 11645–11663 (2021). [CrossRef]

20. J. Zhang, N. Pégard, J. Zhong, H. Adesnik, and L. Waller, “3D computer-generated holography by non-convex optimization,” Optica 4(10), 1306–1313 (2017). [CrossRef]

21. P. Chakravarthula, Y. Peng, J. Kollin, H. Fuchs, and F. Heide, “Wirtinger holography for near-eye displays,” ACM Trans. Graph. 38(6), 1–13 (2019). [CrossRef]

22. S. Liu and Y. Takaki, “Optimization of phase-only computer-generated holograms based on the gradient descent method,” Appl. Sci. 10(12), 4283 (2020). [CrossRef]

23. C. Chen, B. Lee, N. N. Li, M. Chae, D. Wang, Q. H. Wang, and B. Lee, “Multi-depth hologram generation using stochastic gradient descent algorithm with complex loss function,” Opt. Express 29(10), 15089–15103 (2021). [CrossRef]

24. S. Liu and Y. Takaki, “Gradient descent based algorithm of generating phase-only holograms of 3D images,” Opt. Express 30(10), 17416–17436 (2022). [CrossRef]

25. Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein, “Neural holography with camera-in-the-loop training,” ACM Trans. Graph. 39(6), 1–14 (2020). [CrossRef]

26. S. Choi, J. Kim, Y. Peng, and G. Wetzstein, “Optimizing image quality for holographic near-eye displays with Michelson holography,” Optica 8(2), 143–146 (2021). [CrossRef]

27. S. Choi, M. Gopakumar, Y. Peng, J. Kim, and G. Wetzstein, “Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays,” ACM Trans. Graph. 40(6), 1–12 (2021). [CrossRef]

28. S. Choi, M. Gopakumar, Y. Peng, J. Kim, M. O’Toole, and G. Wetzstein, “Time-multiplexed neural holography: a flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators,” ACM Trans. Graph. 32, 1–9 (2022). [CrossRef]

29. C. Chen, D. Kim, D. Yoo, B. Lee, and B. Lee, “Off-axis camera-in-the-loop optimization with noise reduction strategy for high-quality hologram generation,” Opt. Lett. 47(4), 790–793 (2022). [CrossRef]

30. H. Aagedal, M. Schmid, T. Beth, S. Teiwes, and F. Wyrowski, “Theory of speckles in diffractive optics and its application to beam shaping,” J. Mod. Opt. 43(7), 1409–1421 (1996). [CrossRef]

31. P. Senthilkumaran, F. Wyrowski, and H. Schimmel, “Vortex stagnation problem in iterative Fourier transform algorithms,” Opt. Lasers Eng. 43(1), 43–56 (2005). [CrossRef]

32. Y. Zhao, L. Cao, H. Zhang, D. Kong, and G. Jin, “Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method,” Opt. Express 23(20), 25440–25449 (2015). [CrossRef]

33. S. Tian, L. Chen, and H. Zhang, “Optimized Fresnel phase hologram for ringing artifacts removal in lensless holographic projection,” Appl. Opt. 61(5), B17–B24 (2022). [CrossRef]

34. J. Xiao, W. Zhang, and H. Zhang, “Sampling analysis for Fresnel diffraction fields based on phase space representation,” J. Opt. Soc. Am. A 39(2), A15–A28 (2022). [CrossRef]

35. K. Matsushima and T. Shimobaba, “Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields,” Opt. Express 17(22), 19662–19673 (2009). [CrossRef]

36. P. Senthilkumaran and F. Wyrowski, “Phase synthesis in wave-optical engineering: mapping- and diffuser-type approaches,” J. Mod. Opt. 49(11), 1831–1850 (2002). [CrossRef]

37. J. E. Curtis and D. G. Grier, “Structure of optical vortices,” Phys. Rev. Lett. 90(13), 133901 (2003). [CrossRef]

38. Y. Chen, Y. Chi, J. Fan, and C. Ma, “Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval,” Math. Program. 176(1-2), 5–37 (2019). [CrossRef]

39. D. Yoo, Y. Jo, S. W. Nam, C. Chen, and B. Lee, “Optimization of computer-generated holograms featuring phase randomness control,” Opt. Lett. 46(19), 4769–4772 (2021). [CrossRef]

40. B. Lee, D. Kim, S. Lee, C. Chen, and B. Lee, “High-contrast, speckle-free, true 3D holography via binary CGH optimization,” Sci. Rep. 12(1), 2811 (2022). [CrossRef]

41. D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” arXiv, arXiv:1412.6980 (2014). [CrossRef]

42. D. Yang, W. Seo, H. Yu, S. Kim, B. Shin, C. Lee, S. Moon, J. An, J. Y. Hong, G. Sung, and H. S. Lee, “Diffraction-engineered holography: beyond the depth representation limit of holographic displays,” Nat. Commun. 13(1), 6012 (2022). [CrossRef]

43. C. Jang, K. Bang, G. Li, and B. Lee, “Holographic near-eye display with expanded eye-box,” ACM Trans. Graph. 37(6), 1–14 (2018). [CrossRef]

Speckle-free compact holographic near-eye display using camera-in-the-loop optimization with phase constraint

Abstract

1. Introduction

2. Compact holographic near-eye display with CITL optimization

3. Phase constraint strategy for speckle-free CGH optimization

4. Results

5. Discussion

6. Conclusion

Funding

Disclosures

Data availability

References

Data availability

Cited By

Figures (12)

Equations (10)

Optics Express