Open Access
Add to BibSonomy
Abstract
We present a deep neural network for generating a multi-depth hologram and its training strategy. The proposed network takes multiple images of different depths as inputs and calculates the complex hologram as an output, which reconstructs each input image at the corresponding depth. We design a structure of the proposed network and develop the dataset compositing method to train the network effectively. The dataset consists of multiple input intensity profiles and their propagated holograms. Rather than simply training random speckle images and their propagated holograms, we generate the training dataset by adjusting the density of the random dots or combining basic shapes to the dataset such as a circle. The proposed dataset composition method improves the quality of reconstructed images by the holograms generated by the network, called deep learning holograms (DLHs). To verify the proposed method, we numerically and optically reconstruct the DLHs. The results confirmed that the DLHs can reconstruct clear images at multiple depths similar to conventional multi-depth computer-generated holograms. To evaluate the performance of the DLH quantitatively, we compute the peak signal-to-noise ratio of the reconstructed images and analyze the reconstructed intensity patterns with various methods.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
The holographic display is considered as a promising future display technology since it can accurately restore the wavefront information of a three-dimensional object. It has the advantage of being able to express a full parallax by modulating the wavefront [1,2]. The holographic display using a spatial light modulator (SLM) usually displays the digitally generated interference pattern called computer-generated hologram (CGH) to address the wavefront of the coherent light source electronically. There are various methods to generate CGHs, including the angular spectrum method (ASM) [3,4], point cloud methods [5–7], polygon methods [8–11], and depth map methods [12,13]. However, for the practical use, there are some obstacles to generating CGH, such as a considerable amount of calculation load. To reduce the computational weight of the CGH, several techniques have been proposed, such as lookup table method (LUT) [14,15], wavefront recording plane method [16,17], foveated hologram [18,19], and using graphics processing units (GPUs) computing [20,21].
Recently, deep learning has been actively studied in various fields of optics, such as microscopy [22–25], digital holography [26–29], computational imaging [30–32], super-resolution [33–35], image segmentation [36], light-field generation [37,38] and speckle reduction [39]. In particular, studies of generating holograms using deep learning methods have been introduced [40–43], and they showed the possibility that deep learning techniques could accelerate the calculation speed of holograms. For example, Kang et al. used a generative adversarial network (GAN) to create point cloud CGH [40]. They replaced the LUT of the point light sources with the GAN and tried to reduce the computational load. However, since the network can calculate a Fresnel zone plate of a single point light source at once, it needs repetitive calculation to generate a hologram for a complicated image.
Also, Horisaki et al. developed a method for generating phase-only CGH using a deep neural network based on residual network architecture [41]. They used the random phase patterns as inputs and their propagated speckle intensity patterns as targets of the training dataset. They claimed that the CGH made by the network showed better quality in the reconstructed image than that of the Gerchberg-Saxton algorithm [44]. Nevertheless, since the network is trained for a single input depth, it is hard to make CGH with multiple depths, and the result is only presented for low resolution images such as simple handwriting numbers.
In this paper, we propose a deep-learning-based hologram generation method for multi-depth representation. We design a deep neural network that can generate multi-depth holograms and propose a dataset compositing strategy that can effectively train the hologram generation network. The proposed multi-depth hologram generation network (MDHGN) receives multiple images of different depths as inputs and produces a complex hologram that can reconstruct input images at their originated depths. Also, instead of using only random speckles for the training dataset, the quality of generated deep learning holograms (DLHs) is dramatically improved by varying density of the input data or mixing specific shapes to the dataset. This dataset composition lets the network generate multi-depth holograms of various types of images without learning specific objects.
The detailed network structure and the configuration of the dataset are introduced in the following section. In section 3, to verify the proposed method, we numerically and optically restored the multi-depth DLHs and compared them with the reconstructed results of the conventional ASM-based multi-depth CGHs. In section 4, further analysis of MDHGN is presented. Reconstruction of a point light source is analyzed and diffraction patterns calculated by MDHGN is compared to that of ASM. Also, the calculation time of MDHGN is measured according to the number of depth planes.
2. Proposed network and its training strategy
2.1 Network architecture
Figure 1 describes the architecture of the proposed network, MDHGN. Five intensity images for five different depth planes are concatenated in channel axis and provided as input data to the MDHGN. It calculates complex holograms, which can reconstruct each input image at corresponding depth. The network is designed to output the complex hologram as being divided into a real part and an imaginary part.
Fig. 1. Overall schematic diagram of the MDHGN. Multiple intensity profiles are concatenated in channel axis and provided as input data. The MDHGN generates a multi-depth DLH from the inputs. The real and imaginary parts of the complex hologram are calculated respectively.
The MDHGN is based on the residual network architecture [45]. It consists of a convolutional layer, down-sampling blocks, residual blocks, and up-sampling blocks, which are presented in Fig. 2(a). At first, the down-sampling blocks decrease the resolution of concatenated input images and increase the dimensions of the channel. After the down-sampling blocks, the processed data pass through several residual blocks. Then, the original resolution of the image is restored by the up-sampling blocks. Finally, the real and imaginary parts of the complex hologram are calculated through two output branches at the end. The dimensions of data after each operation are denoted between the blocks in the parenthesis, which mean (height, width, channel) of the data matrix. We set the resolution of the input data as 512 $\times$ 512 pixels and the number of the convolution filters in the first layer as 128, indicated by $K$ in Fig. 2(a).
Fig. 2. (a) Architecture of the MDHGN. It is composed of a convolutional layer marked as “Conv”, (b) down-sampling blocks, (c) residual blocks, (d) up-sampling blocks, and (e) output branches. The dimensions of data are denoted as (height, width, channel) between the operation blocks. The pair of numbers in the convolutional layer, for example Conv(7,1), indicates the size of kernel and the stride in order.
A down-sampling block is composed of a convolutional layer, a batch normalization [46], and a rectified linear unit (ReLU) layer, as depicted in Fig. 2(b). The size of kernel and the stride is indicated by a pair of numbers in the convolutional layer box. The residual block consists of two convolutional layers, two batch normalization layers, and one ReLU layer as shown in Fig. 2(c). In the residual blocks, there are adding shortcuts over the blocks to keep the information of the original data while the network goes deeper [45]. The up-sampling block illustrated in Fig. 2(d) has the similar structure with the down-sampling block adding the interpolation layer to up-scale the resolution. The output branch consists of a convolutional layer and a hyperbolic tangent as an activation function so that the hologram has a pixel value in a range from −1 to 1. Each part of the hologram is encoded to 8-bit grayscale to be saved as an image file.
2.2 Training strategy of the MDHGN
In order to train a deep learning network effectively, it is important to construct a well-organized training set as well as to design a structure of a network. In this section, we explain how to construct a training set which can effectively train a hologram generation network. To compose a training set, the input data of the network and the target data, which are compared with the output data of the network, are required. In the case of the MDHGN, the image (intensity profile) at each depth is required for the input data, and the wavefront information which is propagated from the input images and superimposed at the hologram plane is required for the target data.
Figure 3 explains how to make the corresponding input and target data for training the MDHGN. Five different images of randomly positioned dots are generated for the input data. The images are backpropagated to the hologram plane respectively according to each depth $d_1$ to $d_5$, and superimposed as a complex hologram. The first plane $d_1$ is nearest to the hologram plane and 0.5 cm away. The last plane $d_5$ is 2.5 cm away from the hologram plane and the spacing of all planes is equal to 0.5 cm. For the numerical backpropagation of the wavefront, ASM [3,4] is used. We set the pixel pitch of the hologram plane and the wavelength of the light source as 8 $\mu$m and 532 nm, respectively. The calculated complex hologram is divided to a real hologram and an imaginary hologram, and stored as target data. The resolution of input images and target images is 512 $\times$ 512 pixels. In summary, in a single training data group, there are five input images with random dots and two target images with real and imaginary holograms. The total training set is composed of 10,000 data groups. For the loss function, mean-square error loss and L1 loss are combined with ratio of 10:1 empirically. The loss function is shown in Eq. (1), where $M$ is the number of data in a batch, $Y_{true}$ is target hologram and $Y_{pred}$ is output of MDHGN.
Fig. 3. Generation method of the training set using numerical propagation. Five input planes are located at $d_i$ ($i$=1,2,3,4,5) from the hologram plane. Multiple input data are individually backpropagated to the hologram plane, according to each depth, and superposed to generate complex holograms to be used as target data. ASM is used for the freespace propagation method.
First, we train the network for 10 epochs using a dataset consisting only of random dots, similar to the training method of Ref. [41], and the trained result is presented in Fig. 4(a). When the network is trained with the dataset with sparse random dots, the trained result also appears as an image with sparse dots, which makes the quality of the image very low. To improve the quality of the output image, secondly, we add the train data with dense random dots as well as train data with sparse random dots. The trained result is presented in Fig. 4(b). By mixing the dense dots in the training set, it is shown that the high-frequency region of the output hologram is well restored compared to the previous result. The number of images in the training set is the same as before, but the ratio of the sparse dot data and dense dot data is set to 1:1. In order to restore the low-frequency region of the image, circle-shaped faces are used as input to the dataset and the results are presented in Fig. 4(c). However, in that case, the network has difficulty to express the high-frequency region. To reconstruct the high-frequency and low-frequency region of the image simultaneously, random dots and circle-shaped faces are mixed for inputs to the dataset. The ratio of the three datasets is set to 1:1:1. As shown in Fig. 4(d), the low-frequency region of the output hologram is reconstructed well as the high-frequency region.
Fig. 4. The result of the generated holograms according to the dataset composition: (a) sparse random dots only, (b) sparse and dense random dots, (c) random circles, and (d) random dots (sparse+dense) and random circles.
To train the network, we used Nvidia’s Quadro P5000 GPU, and the network was trained for about 24 hours with ten epochs. In our method, the training dataset is easily generated and amount of obtained data is sufficient to converge the network in ten epochs. The training error is shown in Fig. 5.
Fig. 5. Loss history graph of MDHGN during training. Blue line is training loss and orange line is validation loss. Since the amount of training data is 10,000 samples, the MDHGN converges in spite of only ten epochs.
In summary, unlike previous studies [40,41] that trained a network using a single kind of training set (random speckle or point light sources), we present a way to construct a dataset composed of various types of the image that can produce a higher quality result image. In order to restore the high-frequency region of the hologram image, it was empirically confirmed that high-density dots are necessary for the training dataset. Also, to reconstruct the low-frequency region of the hologram image, a dataset composed of filled shapes, such as circles, is required.
3. Results
The most anticipated question for this study is whether it is possible to reconstruct multiple images at different depths using DLHs as much as using conventional CGHs. To answer the question, we reconstructed DLHs through numerical simulations and optical experiments.
3.1 Numerical reconstruction of the DLH
For the image reconstruction experiment, a DLH is generated by inserting alphabet images of “A”, “B”, “C”, “D” and “E” as the inputs of the trained MDHGN as shown in Fig. 6(a). They are arranged in order from “A” to “E” so that “A” is nearest to the hologram plane and “E” is farthest. Figures 6(b) and 6(c) show the real and imaginary parts of the generated DLH with the alphabet images. In addition, to compare with the DLH, a multi-depth CGH with 5 alphabets is generated by the conventional ASM in a similar way of generating a training dataset. Figures 6(d) and 6(e) show the real and imaginary parts of the CGH by the ASM.
Fig. 6. (a) Generation of holograms for the simple alphabet images using the trained MDHGN. The alphabets from “A” to “E” are floated at corresponding depths. The (b) real part and (c) imaginary part of DLH, and the (d) real part and (e) imaginary part of the CGH by the conventional ASM.
The DLH and the CGH by the ASM are numerically propagated to five reconstruction planes from $d_1$ (0.5 cm) to $d_5$ (2.5 cm). For the numerical reconstruction simulation, the ASM is used for calculating free space propagation with 8 $\mu$m pixel pitch and 532 nm wavelength of the light source. The reconstructed images of the DLH and the CGH by the ASM are shown in Fig. 7. Figures 7(a) and 7(b) show the entire holograms of reconstructed results. To make sure that each hologram is focused on the correct depth plane, the alphabets of the corresponding depths are enlarged in Figs. 7(c) and 7(d). By comparing these results, it can be seen that the DLH can reconstruct clear images at the same depths as the CGH by the ASM does. Also, by comparing Figs. 7(e) with 7(f), and 7(g) with 7(h), respectively, it can be seen that the reconstructed intensity patterns of the DLH are blurred in similar ways to the intensity patterns reproduced from the CGH by the ASM.
Fig. 7. The simple alphabet image reconstructed numerically from (a) the CGH by ASM and (b) the DLH. The enlarged images of the reconstructed holograms: (c), (d) enlarged alphabets of each focused plane, (e), (f) the alphabet “A”, and (g), (h) the alphabet “E”. In each enlarged pair, the left side is the result of the CGH by the ASM and the right side is the result of the DLH by the MDHGN.
From the above results, it is confirmed that DLHs can reconstruct inputs of various depths, but we conducted additional numerical simulations to investigate how accurately the input image of a specific depth can be restored by DLHs. Figure 8 shows the reconstructed results of the DLH calculated from the more complicated image. When calculating the hologram, the network receives a grayscale image of a cameraman for one input plane, and dummy black images for the other input planes. Each column in Fig. 8 is divided according to the depth plane in which the grayscale image is placed. For example, Fig. 8(a) is the reconstructed result of a DLH in which the grayscale image is located in the first input plane, $d_1$. Each DLH is propagated to the five reconstruction planes, from $d_1$ to $d_5$, and the peak signal-to-noise ratio (PSNR) is calculated at the focused plane. Also, the cameraman’s face in the focused depth plane denoted as a red box is enlarged in the last row of Fig. 8. It demonstrates that DLH can reconstruct the complicated grayscale image clearly at the intended depth as well as the simple alphabet image.
Fig. 8. Reconstructed holograms generated by the MDHGN for complicated images. The image floats at one selected input plane and the other planes are filled with dummy black images. Each column shows the selected input plane: (a) $d_1$ (nearest to the hologram plane), (b) $d_2$, (c) $d_3$, (d) $d_4$, and (e) $d_5$ (furthest to the hologram plane). The values of PSNR at focused depth plane are indicated below the corresponding images. The face of cameraman in each focused depth plane is enlarged.
3.2 Optical reconstruction of the DLH
In order to confirm that the DLH can be reconstructed well not only numerically but also optically, we built up a holographic display experiment setup as presented in Fig. 9. To display a complex hologram, we used an amplitude SLM with sideband filtering technique and a 4-f system with anamorphic Fourier transform [47,48]. We used a 4K (3840 $\times$ 2160 pixels, 3.74 $\mu$m pixel pitch) amplitude SLM which was taken apart from SONY SXRD projector. Also, an additional 4-f system was used in front of the setup to match the pixel pitch of the DLH precisely by adjusting the distance between the lenses. As a result, the final hologram plane can reproduce 1024 $\times$ 1024 complex pixels with 8 $\mu$m pixel pitch, which has the same pixel pitch of the training dataset.
Fig. 9. The experimental setup for optical reconstruction: (a) schematic diagram and (b) implemented setup. The lenses L2, L3, L4 are cylindrical lenses. L2 and L4 lie on vertical direction, and L3 lies on horizontal direction. The (c) real and (d) imaginary parts of the tiled CGH and DLH which are displayed on the final hologram plane of the experimental setup (1024 $\times$ 1024 pixels, 8 $\mu$m pixel pitch).
We optically reconstruct the DLH and the CGH by the ASM in Fig. 6 using the holographic display setup in Fig. 9. To make it easier to compare the results, the DLH and the CGH by the ASM are tiled horizontally and reconstructed simultaneously. To capture the reconstructed hologram according to each depth plane, from $d_1$ to $d_5$, a CCD camera (FLIR, Grasshopper 3) with an additional eyepiece lens is implemented. The experimental results are presented in Fig. 10 and the arrangement of the figures is similar with the numerical results. By comparing Figs. 10(b) and 10(c), it reveals that the DLH can reconstruct clear images at correct depth plane as well as the CGH by the ASM. In addition, when comparing Figs. 10(d) with 10(e), and 10(f) with 10(g), it is confirmed that the patterns of focused holograms and blurred holograms are similar in both the DLH and the CGH by ASM in optical experiment.
Fig. 10. The optical reconstruction of the CGH by the ASM and the DLH according to the depths from $d_1$ to $d_5$. (a) The CGH (left) and DLH (right) are tiled horizontally and reconstructed simultaneously. The enlarged images of the reconstructed holograms: (b), (c) corresponding alphabets of each focused plane, (d), (e) the alphabet “A”, and (f), (g) the alphabet “E”. In each enlarged pair, the left side is the result of the CGH by the ASM and the right side is the result of the DLH by the MDHGN.
Furthermore, we also compared optically reconstructed results of the more complicated grayscale image. When generating the hologram, the grayscale image is located at $d_5$ and the other planes are filled with the black dummy images as the same way to Fig. 8(e). We conducted similar steps with the cameraman image and the optically reconstructed result are presented in Fig. 11. By comparing the two results (focusing on the ear, the collar, and thickness of the leg of the tripod), the patterns of focused holograms and blurred holograms are similar in both holograms for the more complicated image too. Therefore, we confirmed by the experiments that the DLH can optically reconstruct images as the similar way of the CGH by the ASM.
Fig. 11. The optical reconstruction of the CGH by the ASM and the DLH according to the depths from $d_1$ to $d_5$ with more complicated grayscale image (cameraman). (a) The CGH (left) and DLH (right) are tiled horizontally and reconstructed simultaneously. The enlarged images of the reconstructed holograms: (b), (c) the face of the cameraman and (d), (e) the leg of the tripod. In each enlarged pair, the left side is the results of the CGH by the ASM and the right side is the results of the DLH by the MDHGN.
4. Further analysis of the MDHGN
4.1 MTF analysis of the DLH and reconstruction of point light sources
In order to quantitatively analyze how sharply the DLH is focused at the intended depth plane, DLHs of a point light source on each depth plane are generated and modulation transfer functions (MTFs) of the DLHs according to the reconstruction depths are measured. A DLH is generated by inserting a single point light source to one selected input depth plane and black dummy images to the other input depth planes. We repeat this process for every input depth plane. As a result, five DLHs of a point light source with different planes are generated. The DLHs are reconstructed numerically along the five depth planes. The MTFs are calculated by Fourier transforming the cropped intensity profiles of the propagated DLH according to the reconstructed depths. The results of the calculated MTFs are presented in Fig. 12.
Fig. 12. MTF curves of the DLH. A single point light source is placed in one of the input planes selected from (a) $d_1$ to (e) $d_5$ and other planes are filled with black dummy images to generate the DLHs. The curve of the depth selected for the point light source has the highest contrast in all spatial frequency bands.
By analyzing the MTF curves, it can be seen that the MTF of the hologram reconstructed at the target depth, which is the input depth of the point light source, has the highest contrast in all spatial frequency bands. In addition, the MTF curve falls steeper if the reconstruction plane is farther from the input plane of the point light source. This means that the DLHs can reconstruct the sharpest point spread function at the correct depth. It supports the numerical and optical reconstruction results quantitatively.
During the analysis of the MTF curves, it is found that the contrast of the highest MTF curves on Fig. 12 is gradually decreased as the input plane of the point light source becomes farther from the hologram plane. For example, the highest MTF curve of Fig. 12(e) is located lower than that of Fig. 12(a). This phenomenon can occur because the area of the diffraction pattern generated by the point light source is larger when the input point light source is located farther from the hologram plane. In general, to interpret the larger information, the bigger size of the network is required to learn it. For the same size of the MDHGN, therefore, it can be said that as the input plane is closer to the hologram plane, the reconstruction result has better quality.
In addition to a single point light source, we tested multiple point light sources with point cloud data of Utah teapot. Depth information of the point cloud is quantized in five depth planes as shown in Fig. 13(a). DLH and CGH by ASM are calculated from the quantized point cloud data and reconstructed images are presented respectively in Fig. 13(b). The upper row is reconstructed results from CGH by ASM and the lower row is from DLH. Each column represents depth plane, and the focused points are cropped. Looking at the results, focused points and blurred points at each depth are identical. The handle and spout of teapot is focused at $d_3$ and blurred at $d_1$ and $d_5$ since the teapot is symmetric along $z$-axis (depth). In summary, we analyzed the performance of MDHGN about a single point light source and verified that MDHGN can be used for multiple point light sources.
Fig. 13. (a) Point cloud data. The depth information ($z$-axis) is quantized in five planes. (b) Reconstructed images of holograms generated from quantized point cloud data. The upper row is results from CGH by ASM, and the lower row is from DLH.
4.2 Comparison of diffraction patterns with ASM
The main purpose of this study is to make MDHGN learn the wavefront pattern of CGH generated by ASM algorithm for 5 different depth planes. There are two ways to verify that the network has learned ASM well. One is whether the propagated pattern (numerically or optically) of the DLH is similar to that of the CGH generated by ASM and this is described enough in section 3.
The other is to compare whether the diffraction patterns produced by MDHGN and ASM are similar at the hologram plane. We put profile of a square beam in each of the five depth planes where MDHGN can generate holograms, and investigated the generated diffraction patterns (intensity profiles) in the hologram plane and the backpropagated diffraction patterns at the corresponding depth by ASM.
Figure 14 shows the obtained diffraction patterns and their transverse profile plot. Each column represents the distance in which the square beam is away from the hologram plane, and it can be seen that the shape of the diffraction pattern calculated by the ASM and the network at each depth is similar. The last row of Fig. 14 plots the normalized intensity distribution along the $x$-axis based on the center pixel of the diffracted image and Table 1 shows full-width at half maximum (FWHM) of them. From the results of comparing the intensity distribution and FWHM, it was confirmed that MDHGN effectively learned the wave propagation by ASM and reproduced the diffraction pattern well.
Fig. 14. Diffraction pattern of DLH and CGH by ASM. Each column represents the position of square beam from the hologram plane. The last row is profile plot of diffraction field in $x$-axis.
Table 1. Full width at half maximum (in mm) of profile plot in Fig. 14
4.3 Calculation time of the MDHGN
In this section, the calculation time of MDHGN is measured and analyzed according to the number of depth planes. MDHGN is implemented using Tensorflow 2.1, and the computational time is measured on GPU of Nvidia’s TITAN XP. Figure 15(a) presents a graph of the calculation time using MDHGN while increasing the number of depth planes from 1 to 5. Each depth plane is spaced 0.5 cm apart, starting at 0.5 cm away from the hologram plane. Instead of simply measuring the calculation time of MDHGNs by increasing the number of depth planes, we measured the time by setting a standard for the quality of the reconstructed images from DLHs. The networks are optimized by controlling the number of residual blocks to satisfy that the average PSNR of the reconstructed images of each depth plane is more than 19 dB. To satisfy the condition, we trained the network by increasing the number of residual blocks in sequence. As a result, the minimum number of residual blocks and parameters to reproduce image quality more than 19 dB PSNR is increased, as the number of the depth planes is increased as shown in Fig. 15(b). When the depth plane is increased to five, the memory of the GPU we used for training, 16 GB VRAM of Nvidia’s Quadro P5000, was fully occupied. Looking at the calculation time graph, there is a tendency that calculation time per depth decreases as the number of depths increases. From the result, we expected that the network can learn the rules of calculating wavefront and can generate holograms of various depths more efficiently than simple iterative calculation. If network is implemented big enough to learn more depth planes, such as a continuous object, it is expected that the proposed network may have more advantages in the future.
Fig. 15. (a) Calculation time of the MDHGN. The calculation time is measured while increasing the number of depth planes from 1 to 5. (b) The number of the parameters in the network according to the number of depth planes.
5. Conclusion
We proposed a method for generating a multi-depth hologram using a deep neural network called MDHGN. The structure of the MDHGN consists of down-sampling blocks, residual blocks and up-sampling blocks. We also established the strategy to train the network effectively. Using not only a single type of dataset but combining various types of dataset, the DLH can express both the high-frequency region and the low-frequency region of the reconstructed image. Furthermore, a large amount of training data can be easily obtained since the dataset is composed of the basic element of an image (such as dots and circles). By numerical simulations and optical experiments, it is demonstrated that DLHs can reconstruct the complex grayscale images as well as simple alphabet images at the intended depth plane. The performance of the MDHGN is evaluated quantitatively by the analysis of the MTFs and reconstruction of point light sources. Also it is confirmed that MDHGN learns to reproduce the wavefront pattern generated by ASM through comparison of diffraction patterns. In addition, the calculation time by the number of depth planes are measured. So far, the MDHGN has not considered additional problems of the holograms, such as occlusion or speckle reduction, but it is expected that these problems can be solved in the future as the structure and the training strategy of the network is developed.
Funding
Institute of Information and Communications Technology Planning and Evaluation (IITP) Grant funded by the Korean Government (MSIT) (2020-0-00548).
Acknowledgment
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (No.2020-0-00548, (Sub3) Development of technology for deep learning-based real-time acquisition and pre-processing of hologram for 5G service).
Disclosures
The authors declare no conflicts of interest.
References
1. A. Symeonidou, D. Blinder, A. Munteanu, and P. Schelkens, “Computer-generated holograms by multiple wavefront recording plane method with occlusion culling,” Opt. Express 23(17), 22149–22161 (2015). [CrossRef]
2. B. Lee, D. Yoo, J. Jeong, S. Lee, D. Lee, and B. Lee, “Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing,” Opt. Lett. 45(8), 2148–2151 (2020). [CrossRef]
3. K. Matsushima and T. Shimobaba, “Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields,” Opt. Express 17(22), 19662–19673 (2009). [CrossRef]
4. K. Matsushima, H. Schimmel, and F. Wyrowski, “Fast calculation method for optical diffraction on tilted planes by use of the angular spectrum of plane waves,” J. Opt. Soc. Am. 20(9), 1755–1762 (2003). [CrossRef]
5. R. H.-Y. Chen and T. D. Wilkinson, “Computer generated hologram from point cloud using graphics processor,” Appl. Opt. 48(36), 6841–6850 (2009). [CrossRef]
6. P. Su, W. Cao, J. Ma, B. Cheng, X. Liang, L. Cao, and G. Jin, “Fast computer-generated hologram generation method for three-dimensional point cloud model,” J. Disp. Technol. 12(12), 1688–1694 (2016). [CrossRef]
7. S.-C. Kim and E.-S. Kim, “Effective generation of digital holograms of three-dimensional objects using a novel look-up table method,” Appl. Opt. 47(19), D55–D62 (2008). [CrossRef]
8. Y. Pan, Y. Wang, J. Liu, X. Li, and J. Jia, “Fast polygon-based method for calculating computer-generated holograms in three-dimensional display,” Appl. Opt. 52(1), A290–A299 (2013). [CrossRef]
9. K. Matsushima and S. Nakahara, “Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method,” Appl. Opt. 48(34), H54–H63 (2009). [CrossRef]
10. H. Nishi, K. Matsushima, and S. Nakahara, “Rendering of specular surfaces in polygon-based computer-generated holograms,” Appl. Opt. 50(34), H245–H252 (2011). [CrossRef]
11. H. Kim, J. Hahn, and B. Lee, “Mathematical modeling of triangle-mesh-modeled three-dimensional surface objects for digital holography,” Appl. Opt. 47(19), D117–D127 (2008). [CrossRef]
12. K. Matsushima, M. Nakamura, and S. Nakahara, “Silhouette method for hidden surface removal in computer holography and its acceleration using the switch-back technique,” Opt. Express 22(20), 24450–24465 (2014). [CrossRef]
13. N. Okada, T. Shimobaba, Y. Ichihashi, R. Oi, K. Yamamoto, M. Oikawa, T. Kakue, N. Masuda, and T. Ito, “Band-limited double-step Fresnel diffraction and its application to computer-generated holograms,” Opt. Express 21(7), 9192–9197 (2013). [CrossRef]
14. H. Wei, G. Gong, and N. Li, “Improved look-up table method of computer-generated holograms,” Appl. Opt. 55(32), 9255–9264 (2016). [CrossRef]
15. M. E. Lucente, “Interactive computation of holograms using a look-up table,” J. Electron. Imaging 2(1), 28–35 (1993). [CrossRef]
16. D. Arai, T. Shimobaba, T. Nishitsuji, T. Kakue, N. Masuda, and T. Ito, “An accelerated hologram calculation using the wavefront recording plane method and wavelet transform,” Opt. Commun. 393, 107–112 (2017). [CrossRef]
17. T. Shimobaba, N. Masuda, and T. Ito, “Simple and fast calculation algorithm for computer-generated hologram with wavefront recording plane,” Opt. Lett. 34(20), 3133–3135 (2009). [CrossRef]
18. Y.-G. Ju and J.-H. Park, “Foveated computer-generated hologram and its progressive update using triangular mesh scene model for near-eye displays,” Opt. Express 27(17), 23725–23738 (2019). [CrossRef]
19. L. Wei and Y. Sakamoto, “Fast calculation method with foveated rendering for computer-generated holograms using an angle-changeable ray-tracing method,” Appl. Opt. 58(5), A258–A266 (2019). [CrossRef]
20. D. Blinder and P. Schelkens, “Accelerated computer generated holography using sparse bases in the STFT domain,” Opt. Express 26(2), 1461–1473 (2018). [CrossRef]
21. R. H.-Y. Chen and T. D. Wilkinson, “Computer generated hologram with geometric occlusion using GPU-accelerated depth buffer rasterization for three-dimensional display,” Appl. Opt. 48(21), 4246–4255 (2009). [CrossRef]
22. Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang, and A. Ozcan, “Deep learning microscopy,” Optica 4(11), 1437–1443 (2017). [CrossRef]
23. E. Nehme, L. E. Weiss, T. Michaeli, and Y. Shechtman, “Deep-storm: super-resolution single-molecule microscopy by deep learning,” Optica 5(4), 458–464 (2018). [CrossRef]
24. W. Ouyang, A. Aristov, M. Lelek, X. Hao, and C. Zimmer, “Deep learning massively accelerates super-resolution localization microscopy,” Nat. Biotechnol. 36(5), 460–468 (2018). [CrossRef]
25. T. Pitkäaho, A. Manninen, and T. J. Naughton, “Focus prediction in digital holographic microscopy using deep convolutional neural networks,” Appl. Opt. 58(5), A202–A208 (2019). [CrossRef]
26. Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light: Sci. Appl. 7(2), 17141 (2018). [CrossRef]
27. Z. Ren, Z. Xu, and E. Y. Lam, “Learning-based nonparametric autofocusing for digital holography,” Optica 5(4), 337–344 (2018). [CrossRef]
28. T. Nguyen, V. Bui, V. Lam, C. B. Raub, L.-C. Chang, and G. Nehmetallah, “Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection,” Opt. Express 25(13), 15043–15057 (2017). [CrossRef]
29. Z. Ren, Z. Xu, and E. Y. Lam, “Autofocusing in digital holography using deep learning,” Proc. SPIE 10499, 104991V (2018). [CrossRef]
30. A. Sinha, J. Lee, S. Li, and G. Barbastathis, “Lensless computational imaging through deep learning,” Optica 4(9), 1117–1125 (2017). [CrossRef]
31. G. Barbastathis, A. Ozcan, and G. Situ, “On the use of deep learning for computational imaging,” Optica 6(8), 921–943 (2019). [CrossRef]
32. M. Lyu, H. Wang, G. Li, S. Zheng, and G. Situ, “Learning-based lensless imaging through optically thick scattering media,” Adv. Photonics 1(03), 1 (2019). [CrossRef]
33. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). [CrossRef]
34. J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE Conf. Comp. Vis. Patt. Recog., (2016), pp. 1646–1654.
35. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE Conf. Comp. Vis. Patt. Recog., (2017), pp. 136–144.
36. Z. Xu, S. Zuo, E. Y. Lam, B. Lee, and N. Chen, “AutoSegNet: An automated neural network for image segmentation,” IEEE Access 8, 92452–92461 (2020). [CrossRef]
37. L. Xiao, A. Kaplanyan, A. Fix, M. Chapman, and D. Lanman, “DeepFocus: Learned image synthesis for computational displays,” ACM Trans. Graph. 37(6), 1–13 (2019). [CrossRef]
38. M. Gupta, A. Jauhari, K. Kulkarni, S. Jayasuriya, A. Molnar, and P. Turaga, “Compressive light field reconstructions using deep learning,” in Proceedings of the IEEE Conf. Comp. Vis. Patt. Recog., (2017), pp. 11–20.
39. D.-Y. Park and J.-H. Park, “Hologram conversion for speckle free reconstruction using light field extraction and deep learning,” Opt. Express 28(4), 5393–5409 (2020). [CrossRef]
40. J.-W. Kang, J.-E. Lee, Y.-H. Lee, D.-W. Kim, and Y.-H. Seo, “Interference pattern generation by using deep learning based on GAN,” in 34th Int. Tech. Conf. Circuits/Systems, Comp. and Comm.(ITC-CSCC), (IEEE, 2019), pp. 1–2.
41. R. Horisaki, R. Takagi, and J. Tanida, “Deep-learning-generated holography,” Appl. Opt. 57(14), 3859–3863 (2018). [CrossRef]
42. B. Lee, J. Jeong, D. You, and J. Lee, “Learning-based synthesis of computer-generated hologram,” in Digital Holography and Three-Dimensional Imaging, Imaging and Applied Optics Congress 2020, (Optical Society of America, 2020), HTu4B.1.
43. J. Lee, J. Jeong, J. Cho, D. Yoo, and B. Lee, “Complex hologram generation of multi-depth images using deep neural network,” in Digital Holography and Three-Dimensional Imaging, Imaging and Applied Optics Congress 2020, (Optical Society of America, 2020), JTh2A.12.
44. R. W. Gerchberg, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik 35, 237–246 (1972).
45. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conf. Comp. Vis. Patt. Recog., (2016), pp. 770–778.
46. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Int. Conf. Machine Learning, (2015), pp. 448–456.
47. H. Kim, C.-Y. Hwang, K.-S. Kim, J. Roh, W. Moon, S. Kim, B.-R. Lee, S. Oh, and J. Hahn, “Anamorphic optical transformation of an amplitude spatial light modulator to a complex spatial light modulator with square pixels,” Appl. Opt. 53(27), G139–G146 (2014). [CrossRef]
48. J. Jeong, J. Lee, C. Yoo, S. Moon, B. Lee, and B. Lee, “Holographically customized optical combiner for eye-box extended near-eye display,” Opt. Express 27(26), 38006–38018 (2019). [CrossRef]
