Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

HoloPhaseNet: fully automated deep-learning-based hologram reconstruction using a conditional generative adversarial model

Open Access Open Access

Abstract

Quantitative phase imaging with off-axis digital holography in a microscopic configuration provides insight into the cells’ intracellular content and morphology. This imaging is conventionally achieved by numerical reconstruction of the recorded hologram, which requires the precise setting of the reconstruction parameters, including reconstruction distance, a proper phase unwrapping algorithm, and component of wave vectors. This paper shows that deep learning can perform the complex light propagation task independent of the reconstruction parameters. We also show that the super-imposed twin-image elimination technique is not required to retrieve the quantitative phase image. The hologram at the single-cell level is fed into a trained image generator (part of a conditional generative adversarial network model), which produces the phase image. Also, the model’s generalization is demonstrated by training it with holograms of size 512×512 pixels, and the resulting quantitative analysis is shown.

© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

1. Introduction

The first suggested holography scheme was in-line or Gabor holography [1]. It provides a powerful biological samples imaging tool with minimal sample preparation, i.e., without the need for staining, fixation, or labeling. After the invention of digital cameras and the increase in computing power available, Gabor holography became useful, in conjunction with a reconstruction algorithm, for particle image analysis, 3-D tracking, or swimming cells in a liquid flow [25]. The main drawback of the Gabor configuration is that the reconstructed image inevitably suffers from the presence of the zero-order image (or the intensity of reference wave and object wave) and the conjugate image, which overlaps with the reconstructed image and causes blurring. It is because all the light fields propagate in the same direction in the reconstruction stage, and they are observed simultaneously. To resolve this problem, several instrumental and numerical solutions have been proposed. Instrumental approaches include phase shifting by adding wave plates, a piezo-electric transducer mirror, or an electro-optic modulator [613]. In this case, multiple holograms (usually two or three holograms) corresponding to various phase differences between the object and the reference light are recorded. Other phase recovery implementations require recording additional intensity information, such as two or three holograms at different distances [1416]. The extra intensity data can help in retrieving the missing phase information. However, a limitation arises in the imaging area in that objects are required to stay immobile, or else a slight vibration in the optical setup will disturb the results. This problem is particularly important for biological sample studies, especially in real-time flow cytometry applications. Iterative phase recovery was also suggested for Gabor holography to remove unfocused twin-image noise [1720]. The main drawback of iterative phase recovery is that it requires several back-and-forth light propagations to recover the actual phase value. The iterative phase recovery method also requires a convergence criterion or the number of iterations to be defined, which is generally unknown. This convergence criterion determination is particularly difficult for real-time biological sample analyses.

The off-axis configuration was instrumental in different studies of biological samples [21,22]. A small tilt (a few degrees) is inserted between the object wave and reference wave and allows for separating the twin-image and real image by spatial filtering in the spectrum domain (see Fig. 1). The off-axis configuration for studying biological samples is also called digital holographic microscopy (DHM). DHM enables non-destructive investigations of biological samples as well as marker-free and time-resolved analysis of cell biological processes. Traditionally, a numerical reconstruction algorithm must be used to obtain the contrast image (both phase and amplitude images) [23]. The hologram reconstruction algorithm requires knowing the specific parameters of wave vectors and the reconstruction distance, as well as implementing a proper phase unwrapping algorithm. Besides setting all parameters, numerical reconstruction usually requires applying the Fourier transformation which is computationally expensive. The development of the fast Fourier transform (FFT) accelerated numerical reconstruction. Indeed, GPU acceleration can also be utilized to enhance the speed of the image reconstruction. However, specific parameters of numerical reconstruction still need to be adjusted. The wave vector plays a crucial role in numerical reconstruction. During the quantitative phase-image reconstruction, the digital hologram is multiplied by a digital reference wave, which is a replica of the reference wave. The perfect reconstruction can be achieved if the components of the wave vector are adjusted such that the propagation direction of digital reference matches as closely as possible that of the experimental reference wave. This approach is similar to the hologram illumination with the reference wave in classic holography.

 figure: Fig. 1.

Fig. 1. (left side) General scheme of off-axis hologram recording. The camera (here a CMOS camera is connected to the PC for numerical reconstruction). After hologram recording, the phase images are conventionally reconstructed by the numerical light propagation method, but we suggest using a deep-learning model to reconstruct single-cell images (right side). Single-level off-axis hologram without spatial filtering in Fourier domain is fed into the model and phase image is automatically generated.

Download Full Size | PDF

Finding the reconstruction distance is also important. Different algorithms can automatically determine reconstruction distance by applying numerical reconstruction at several possible reconstruction distances and using a function to evaluate the quality of each reconstruction. For example, one method minimizes the spatial standard deviation in the reconstructed intensity image [24]. This method works because the ideal phase objects minimally affect the amplitude of the light thus only the border of the cell is visible, so the spatial standard deviation is at its lowest value for the perfect reconstruction distance. The drawback is that several numerical reconstructions should be performed to find the best reconstruction distance, in addition to the need to know the span of the search and the increment/decrement step for each reconstruction distance.

After numerical reconstruction of the off-axis hologram, a quantitative phase image related to the content and morphology of a biological sample can be obtained [2226]. More specifically, the interpretation of a quantitative phase-contrast image (QPI) with DHM (QPI-DHM) gives access to quantitative measurements of both cellular morphology and the refractive index in only one shot. It is a real-time method, and in the absence of mechanical focus adjustment, it can be used for high-speed time-lapse studies of biological samples.

This paper shows that quantitative phase-contrast images can alternatively be obtained by utilizing a deep learning image-to-image transformation model. Specifically, we use a Conditional Generative Adversarial Network (C-GAN) model to convert a hologram to the corresponding QPI. We only address quantitative phase-contrast images because most of the biological samples are transparent or semi-transparent; thus, they only alter the phase of the transmitted light, and the amplitude of light stays unchanged (unless a chromatic dye is used). After training the model with sufficient cell numbers and correct hyper-parameters, the Generator of the C-GAN model (we call it HoloPhaseNet) is fed an unobserved hologram and the phase image is obtained. We focus on the single-cell level image transformation. One reason is that the single-cell image study is one advantage of the DHM application. It allows the scientist to follow the changes at the single-cell level. We also show that the single-hologram reconstruction can be generalized to bigger hologram sizes. Indeed, the HoloPhaseNet does not require isolation of the real image from the twin image in the hologram domain. Key advantages of a deep-learning-based holographic reconstruction algorithm include alternative fast image reconstruction, performing auto-focusing, and not requiring perfect settings of the numerical reconstruction parameters (such as finding the correct reconstruction distance, digital propagation and wave vectors).

Convolutional neural network (CNN) and deep learning approaches were proposed for several optical applications. Examples include single-cell-based reconstruction distance estimation by a regression CNN model [27], Gabor hologram reconstruction with C-GAN [28], super-resolution fringe patterns by deep learning holography [29], virtual staining of non-stained samples [30], image reconstruction and enhancement in optical microscopy [31], CNN for in-line holography application [32], fast phase retrieval in off-axis holography [33], a one-to-two CNN model for simultaneous amplitude and phase recovery [34], and lens-less computational imaging by deep learning [35]. Deep-learning-based phase recovery by a residual CNN model was also suggested [36], but the application is limited because it needs several recordings to provide the noise-free phase image for the training of deep learning. It also requires numerical reconstruction to convert the hologram to the superimposed phase image before feeding it to deep learning. For biological samples and particularly moving micro-objects (cancer cells and blood cells in flow cytometry applications), the proposed residual CNN of Rivenson et al. [36] has limited application. Another small drawback of their method is that blurriness might occur in the output of the model. The reason is that the MSE is used to measure the difference between the model’s output and ground truth images. A network optimization method seeks to minimize the error between images and is equivalent to the averaging, which causes blurry images at the end.

In this study, the leukemia cancer cells and isolated nuclei of cancer cells are used to show an application of the proposed method. The holograms of the nuclei were previously recorded to study the cell’s dry mass and nuclei section [37]. The setup used an off-axis configuration, and DryMass (an open-source Python library) [38] was used to reconstruct the phase images for the training of the model. During the training, the input to the model is an off-axis hologram, and the output is the corresponding numerically reconstructed phase image. After enough training epochs (a hyperparameter that defines the number of times that the learning algorithm will observe the entire training dataset), the model learns the conversion of the hologram domain to the phase image domain. Choosing the correct number of training epochs is also crucial because a longer training time might result in overfitting. One way to avoid this is to evaluate the model on an unobserved validation data set. In this paper, we employed the average value of the structural similarity index measure (SSIM) and mean-squared error (MSE) of the validation data set to avoid overfitting during the training. The trained model can mimic the numerical light propagation without the requirement of knowing the light properties and reconstruction parameters. The trained model is tested with a set of unobserved cells to validate the C-GAN model and perform quantitative analysis.

2. Proposed deep learning model for the hologram to phase image transformation

We propose using a conditional generative adversarial network (C-GAN) used in several image-to-image translation studies [28,30,3942]. The advantage of this model is that it can be generalized for different image-to-image applications without changing the model’s loss function or its structure. Examples include aerial to map, labels to façade, grayscale image to color image, and edges to photos [39]. A C-GAN learns a mapping from input images to output images. Learning this mapping is achieved by simultaneous training of the generator and the discriminator aiming to maximize the performance of both. The generator is trained to generate output that “trick” the discriminator, while the discriminator is trained to distinguish between real and fake data. Maximizing the generator’s performance means maximizing the loss of the discriminator when given generated labeled data. Ideally, the training results in a generator that outputs convincingly realistic data corresponding to the input labels and a discriminator that has learned strong feature representations characteristic of each label's training data.

The C-GAN consists of a U-net image generator (a form of the fully connected CNN model that consists of a contracting path and an expansive path that resemble a U-shaped architecture) and a PatchGAN classifier or discriminator. The generator (Fig. 2(a) and Fig. 2(c)) is trained to produce images that cannot be distinguished from actual QPI images by an adversarially trained discriminator. The discriminator (Fig. 2(b) and Fig. 2(d)) learns to classify between the generator’s synthesized image (a “fake” image) and the “real” image (the QPI image). The discriminator in a C-GAN is simply a classifier. It tries to distinguish real data from the data created by the generator. A discriminator can use any network architecture corresponding to the type of data it is classifying. Training can be simplified as follows: the generator attempts to produce a phase image with the same statistical features as the QPI image (the reference images), while the discriminator tries to distinguish whether the input image is the phase image or the generator’s output. The training procedure seeks a state of equilibrium in which the generator’s output and QPI image share very similar statistical distributions.

 figure: Fig. 2.

Fig. 2. (a) The U-net generator or so-called HoloPhaseNet. The generator receives an input hologram and outputs a phase image. (b) A discriminator evaluates the generator’s output and provides a “real” or “fake” answer during the training process. (c) shows the detail of generator and (d) shows the architecture of discriminator. Batch-Normalization normalizes activation vectors from hidden layers using the mean and variance of the current batch. Batch-Normalization can make the training stable and faster. Leaky Rectified Linear Unit (Leaky ReLU) is a type of activation function based on a ReLU, but it has a small slope for negative values instead of a flat slope. Leaky ReLU is popular in tasks where it may suffer from sparse gradients, for example training generative adversarial networks. The hyperbolic tangent activation function (Tanh) takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1. The sigmoid activation function’s domain is the set of all real numbers, and its output is between 0 and 1. Hence, if the input to the function is either a very large negative number or a very large positive number, the output is always between 0 and 1.

Download Full Size | PDF

The generator (see Fig. 2©), in this work, consists of seven down-sampling and seven up-sampling units following the general shape of a U-net. Down-sampling extracts the features of the input image for translation, and up-sampling reconstructs the image based on the extracted features. Down-sampling is performed before up-sampling, resulting in a blurry output due to much of the high-frequency information of the original image being lost. Thus, a skip connection is added to share high-frequency information between the input and output. The skip connection reduces the blurry effects of the generated image by connecting the information of the ith layer of the down-sampling process to the information of the n-ith layer of the up-sampling process. The discriminator receives a 2N×N input image, passes through five convolution layers, and derives a 16×16 pixel patch (see Fig. 2(d)). The discriminator learns to distinguish between real and fake patches. The reason for evaluating images with patches is that the model can be trained faster with fewer parameters. The generator and discriminator are composed of convolution-BatchNorm-Leaky ReLU with a 3×3 filter size.

Regarding the sizes of kernels, it is already shown that filters with the 3×3 or 4×4 pixel kernels are the best choices. Smaller filters also facilitate and accelerate the training procedure because most of the common useful features (usually small size features) may be found in more than one place in an image. Also, an added benefit of using a small kernel instead is to benefit from weight sharing and a reduction in computational costs. In other words, the same kernel for a different set of pixels in an image is used; thus, the same weights are shared across these pixel sets. Increasing the depth and number of filters will increase the training time and complexity of the model, which in return results in a higher-performance model for the task in most cases.

3. Experiment details and results

3.1 Sample preparation

The human myelocytic leukemia cell line (HL60) was cultured under standard conditions at 37 °C, 5% CO2. The cell culture media were each supplemented with 10% FBS, 1% penicillin-streptomycin. The nuclear isolation protocol was as follows: Cells (∼2 million/ml) were first centrifuged for 5 min at 130 × g and 4 °C and washed with PBS (4 °C). Cells were then resuspended in a hypotonic buffer containing 10 mM HEPES (pH 7.5), 1 mM DL-Dithiothreitol (Sigma), and protease inhibitor (Roche), and incubated on ice for 10 min. For chemical isolation, IGEPAL solution (a non-ionic detergent, Sigma) and citric acid were added to a final concentration of 0.1% and 1 mM, respectively. The solution was then quickly vortexed for 10 seconds, before being centrifuged at 4 °C for 5 min at 180 × g to obtain a nuclear pellet. The solution was washed again with 4 °C PBS to remove residual IGEPAL and citric acid. For mechanical isolation, following the 10 min incubation in the hypotonic buffer, the cells were transferred to a pre-cooled 7 ml Dounce tissue homogenizer (Sigma) and mechanically broken up with typically 50 strokes using pestle B. The released nuclei were then collected by centrifugation (4 °C, 5 min at 180 × g) [37,38]. For the imaging with DHM, isolated nuclei were slightly attached to a coverslip coated with 0.01% Poly-L-Lysine solution (Sigma) to reduce motion while preserving their spherical shape for measurement.

3.2 Imaging setup details

The imaging setup is shown in Fig. 1. A HeNe-laser light source (λ= 632.8 nm, HNL050L-EC, Thorlabs) with the power of about 0.5mW, was split into an object beam and a reference beam by a 50:50 beam-splitter (BS1). The object beam passes through the sample and subsequently through a microscope objective (MO, 40×/NA 0.75, Zeiss, lateral resolution of 450nm). The object beam is overlapped with the reference beam using a second beam-splitter (BS2) with a small tilt angle to provide off-axis geometry. The small angle between the object- and the reference- beam can provide isolation of twin-image, real image, and zero-order noise by applying spatial filtering in the Fourier domain. The resulting holograms were captured by a gray-scale CMOS camera. Since a laser has a coherent illumination, it is important to use a camera without a glass cover in front of the chip to avoid additional detrimental interference artifacts. The monochrome CMOS camera used here has 1280×1024 pixels with a pixel size of 5.2 µm (MCE-B013-UW, Mightex Systems, Pleasanton, California, USA).

To determine the RI of the surrounding cell’s media an Abbe refractometer (2WAJ, Arcarda GmbH) was used. The DHM has a diffraction-limited spatial resolution of about 850 nm. The phase accuracy of the system was estimated by determining the standard deviation (SD) of the phase in images taken without sample, which results in a phase accuracy of Δφ=0.099 ± 0.004 rad(N = 20) [37]. The numerical reconstruction method including all intermediate steps, such as filtering in Fourier-space, background correction, and the phase unwrapping algorithm is explained in the Supplementary section.

3.3 Quantitative results

The holograms were recorded initially at the size of 1280×1024 pixels. After the numerical reconstruction, each QPI image was thresholded with Otsu’s method to obtain a binary mask for cells. After binarization, the cells were labeled, and single cells that are located at about the center of a 128×128-pixel ROI were extracted. Our primary analysis showed that a 128×128 pixels area (an area around 20 µm×20 µm) could almost contain cells of different sizes. Cells with bigger sizes and cells at the border of the original QPI image were removed from the data set. The same area in the hologram is also selected to match the single-cell area at the QPI image. The selected single-cell level hologram and QPI images construct the training, validation, and test set. A similar model was trained with 512×512 pixel holograms and the corresponding QPI image was used to evaluate the generalization of the proposed concept.

The C-GAN model was trained with 900 single-cell level hologram and the corresponding QPI cancer cell images. To augment the number of samples used during the training, random rotation (90, 180, and 270 degrees) of each training batch was employed too. This can also train the model for conditions where the orientation of the fringes change, and the model will be more generalized. We tried to include cells with different shapes (a gallery of some of the images used for the training is shown in Fig. 3). Also, 50 cells were used for the validation of the model. The accuracy of the model was assessed with 100 unseen images (the test data set). We used the Adam solver (a popular extension to stochastic gradient descent. It uses mini-batch optimization and can make progress faster while seeing less data than the other Neural Network optimization solver) as the optimization process with adaptive momentum and the parameters β1 = 0.5 and β2 = 0.999. The number of epochs is 500, and the learning rate (a tunable parameter in an optimization algorithm determining the step size at each iteration while moving toward a loss function’s minimum value) is 0.0002. We used an NVIDIA GeForce RTX 2080 Ti GPU for training. Batch training with a batch size of 44 was used. We implemented the model on a computer with an Intel Core i9-9900K CPU@3.60 GHz and 64 GB of RAM running Windows 10 Enterprise. The network was implemented using Python version 3.6.8 and TensorFlow framework version 1.14.0. The network testing was performed on the same PC.

 figure: Fig. 3.

Fig. 3. A gallery of sample images used for training the model. The first row shows the off-axis hologram, and the second row shows the corresponding QPI image obtained by the numerical reconstruction algorithm. Single-cell extraction was performed on the QPI image, and the same cell’s location from the hologram image was selected in an automated way. The third row shows the 3D surface representation of each QPI image.

Download Full Size | PDF

During the training, the generator’s output (with a validation dataset, separate from the training data set) was analyzed using some criteria (SSIM and MSE) to determine whether to stop the training. A small gallery of the validation set is shown in Fig. 4(a). Figure 4(b) shows the generator’s output at each specific epoch when the unobserved validation dataset was fed into the generator.

 figure: Fig. 4.

Fig. 4. (a) shows a gallery of the validation dataset. The first row shows the off-axis hologram, and the second row shows the corresponding QPI image obtained by the numerical reconstruction algorithm. Single-cell extraction was performed on the QPI image, and the same cell’s location from the hologram image was selected in an automated way. The third row shows the 3D surface representation of each QPI image. (b) The development of the HoloPhaseNet at different epochs and the QPI results for the validation dataset.

Download Full Size | PDF

Figure 5(a) shows the loss function (a measure showing how far a particular iteration of the model is from the actual values) for both the generator and discriminator at different epochs for the training epochs. The discriminator loss value does not change further around epoch 300, but we still observe that generator loss is dropping. To investigate this and confirm that the model is not overfitting, the HoloPhaseNet was tested with 50 hologram images that were not used during the model training (validation set). We calculated the MSE and SSIM between the QPI image obtained from numerical reconstruction and HoloPhaseNet’s output, and the results were averaged for the plot (See Fig. 5(b)). Looking at the average MSE and average SSIM of validation data, it is apparent that the average SSIM increases, and the average MSE drops for around 400 epochs. After 400 epochs, SSIM and MSE are in the plateau, and the values do not change. Accordingly, 400 epochs can be a protentional point to stop the training.

 figure: Fig. 5.

Fig. 5. (a) HoloPhaseNet’s Generator and Discriminator loss during the training. (b) The average MSE and SSIM for the validation data set.

Download Full Size | PDF

3.3.1 phase reconstruction validation

To validate the phase reconstruction using HoloPhaseNet, the phase values generated for the cancer cell and the isolated nuclei section were compared. It is shown that the phase value of the cell is larger than the isolated nuclei [37]. After training the model with the cells and isolated nuclei, a two-sample t-test (p-value < 0.001; n = 50) determines if the test dataset’s average phase values are different. Figure 6 shows the average phase values for cells and isolated nuclei.

 figure: Fig. 6.

Fig. 6. Average phase values of cells and isolated nuclei for the test dataset. p-value < 0.001; n = 50. The colormap is adjusted for the visualization.

Download Full Size | PDF

3.3.2 Performance evaluation

To assess the performance of the HoloPhaseNet, several quantitative analyses were performed with a test data set. For this purpose, the trained HoloPhaseNet model was saved to the PC, and the test data set were fed into the model. Visualization 1 shows a video of all test holograms and HoloPhaseNet’s output alongside the numerically reconstructed QPI images. In one analysis, we compare the projected surface area (PSA) of the model’s output and the QPI images provided by numerical reconstruction (See Fig. 7(a)). The PSA of HoloPhaseNet for the test data set is compared with exact cell from the QPI numerical reconstruction method where PSA is defined as:

$$PSA = N{p_s}^2,$$
where N is the total number of pixels constituting the cell’s projected area resulting from the image binarization algorithm, and ps denotes the pixel size in the phase image (0.144µm). We also compared the average phase value of the model’s output versus that of the QPI image (See Fig. 7(b)). The average phase value is defined as follows:
$$\phi = \frac{{\sum\nolimits_{(i,j) \in {S_p}} {\varphi ({i,j} )} }}{N},$$
where $\varphi ({i,j} )\; $ is the phase value at pixel (i,j), and the summation is achieved over all the pixels (i,j) of Sp (projected area). In addition to the above, the aspect ratio and circularity of the binarized results for the model’s output and QPI imaging were evaluated and are shown, respectively, in Fig. 7(c) and Fig. 7(d). The aspect ratio is calculated as follows:
$$AR = {{Major} / {Minor}},$$
where Major and Minor are the primary and secondary axis of the best fitting ellipse on the cell’s PSA. Finally, circularity, denoted Cir, is obtained by using the following equation:
$$cir = 4\pi \frac{{PSA}}{{{p_r}^2}},$$

 figure: Fig. 7.

Fig. 7. Quantitative analysis of the proposed method and comparison versus that of numerical reconstruction method(a) The PSA of the cells when QPI is obtained with numerical reconstruction method versus that of the PhaseHoloNet for the test dataset, (b) the average phase value (c) the Aspect Ratio, (d) the circularity and (f) shows the average reconstruction error for 100 test images.

Download Full Size | PDF

Here, pr denotes the length of the outside boundary of the cell. We also measured the dry mass (Fig. 7(e)) of the HoloPhaseNet and compared them with the QPI numerical reconstruction method. The dry mass is determined by integrating the phase over the projected surface area of the cell:

$$DM = \frac{\lambda }{{2\pi \alpha }}\sum\nolimits_{(i,j) \in {S_p}} {\varphi ({i,j} )} ,$$
where $\lambda $ is the wavelength of the laser source, and α is the specific refractive index increment factor (reported for proteins and DNA to be α≈0.2 ml g−1) [43]. Additionally, Fig. 7(f) shows the average reconstruction error between the HoloPhaseNet’s output and the QPI method. The average error is obtained for 100 cells between the QPI method and the model’s output. One can see that the average error around the corner of ROI is the lowest compared to the center, where a sphere has the highest thickness. This small systematic error (π/20) can be neglectable in the applications where the instant reconstruction is important, and a small error does not affect, for instance, the study of flowing cells or micro-fluid studies. This error can be reduced if the model is trained with more samples.

4. Discussion

The proposed deep learning model HoloPhaseNet can directly reconstruct the phase image from the off-axis hologram without performing numerical light propagation and twin image isolation. Looking at the quality of the reconstructed cells (See Fig. 8), one can see that the model well preserves the general structure of the cell. Reconstructed cells seem slightly blurry, but the high-frequency changes (phase changes due to the uneven distribution of materials within the nuclei) are still preserved. While designing a deep learning model for image reconstruction applications, it is crucial to find the best metric that suits the application. Looking at the validation set’s average SSIM and MSE changes versus the epoch number (Fig. 5(b)), one can see that epoch number around 400 is the best value to stop the training even though the Generator’s loss for the training data set (Fig. 5(a)) is still dropping.

 figure: Fig. 8.

Fig. 8. Some test data set sample of the reconstructed phase image by HoloPhaseNet. The high-frequency changes shown by an arrow, due to the uneven distribution of material inside the cell is, are preserved by the model.

Download Full Size | PDF

Additionally, the PSA of the model’s output (Fig. 7(a)) is similar to that of the QPI numerical reconstruction method proving that HoloPhaseNet can completely retrieve the lateral features. The aspect ratio of the HoloPhaseNet model and that of the numerical reconstruction is also similar (Fig. 7(c)), but the interquartile range for the model’s circularity is very packed compared to the QPI method (Fig. 7(d)). Dry mass, which is the most important feature obtained by the DHM method, represents the mass of the cell. We can see in Fig. 7 (e) that the dry mass of the proposed method is very similar to that of the QPI method. The average error map (Fig. 7 (f)) between the HoloPhaseNet’s output and QPI reconstruction shows that the error is mainly happening at the center of the reconstructed area while the borders have less error.

4.1 Architecture generalization of the HoloPhaseNet:

The application of HoloPhaseNet is not limited only to single-cell level reconstruction. It can simply be extended to the larger hologram reconstruction as well. To show this, we trained the model with the input-output size of 512×512 pixels and the results are presented in this section. The architecture of the model is similar to the single-cell level reconstruction model. {900} images of cancer cells and isolated nuclei were used for the training, and 50 holograms were used for testing. We trained the model for 400 epochs and then tested the model with 50 unobserved holograms. The MSE and SSIM for the test dataset, respectively, are π/10 ± π/20, 0.86 ± 0.03 (mean ± sd). Figure 9 shows a few images from the training and testing datasets.

 figure: Fig. 9.

Fig. 9. generalization of the HoloPhaseNet for the bigger hologram reconstruction. The model is trained with 900 images of the size 512×512 pixels. The first row shows three sample holograms used for the training and 2nd row shows the corresponding phase image reconstructed by the numerical method. The third row shows three holograms used for the HoloPhaseNet test. The last row shows the model’s output.

Download Full Size | PDF

One advantage of single-cell reconstruction is that cells can be reconstructed quickly. The HoloPhaseNet’s time to reconstruct 1000 single cells was around 20 seconds. Also, the HoloPhaseNet’s time to reconstruct 900 images with the size of 512×512 was around 24 seconds. A fair comparison with the numerical reconstruction requires considering several aspects. Numerical reconstruction in CPU might be slow due to not taking advantage of GPU technology. Also, running a phase unwrapping algorithm can slower down the reconstruction algorithm because of the computations associated with the phase unwrapping. It is also important to consider the reconstruction distance estimation algorithm. As it is mentioned before, several reconstructions are required to find the best plane. As a result, the running time for numerical reconstruction for the 100 whole holograms, was around 125 seconds. The running time for the numerical reconstruction is obtained by running the code in the CPU in a Python environment when both phase unwrapping and reconstruction distance estimation was activated. But it is possible to get a very fast numerical reconstruction of about 30 holograms per second if someone develops codes that can take advantage of GPU. The commercial DHM machines can reconstruct around 30 images while a phase unwrapping algorithm is running too.

5. Conclusion

This paper shows that deep learning can be considered an alternative way for the complex numerical reconstruction procedure. An image-to-image translation model can achieve this reconstruction, and, specifically, we used conditional generative adversarial networks. For the training of the model, we used the cancer cells holograms, and the model was evaluated with unobserved test cells. We learned that the model can completely reconstruct the projected surface area of the cell and average phase value, but there is a slight loss in the aspect ratio of the cell when the results are compared with the numerical reconstruction method.

Funding

NSERC (RGPIN-2020-06785); Concordia University’s Horizon Postdoctoral Fellowship program.

Acknowledgments

This work was supported by Fevens’ NSERC grant (RGPIN-2020-06785). We also acknowledge support from Concordia University’s Horizon Postdoctoral Fellowship program.

Disclosures

“The authors declare no conflicts of interest.”

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 for supporting content.

References

1. D. Gabor, “A new microscopic principle,” Nature 161(4098), 777–778 (1948). [CrossRef]  

2. I. Moon, M. Daneshpanah, B. Javidi, and A. Stern, “Automated three-dimensional identification and tracking of micro/nano biological organisms by computational holographic microscopy,” Proc. IEEE 97(6), 990–1010 (2009). [CrossRef]  

3. G. D. Caprio, A. Mallahi, P. Ferraro, R. Dale, G. Coppola, B. Dale, G. Coppola, and F. Dubois, “4D tracking of clinical seminal samples for quantitative characterization of motility parameters,” Biomed. Opt. Express 5(3), 690–700 (2014). [CrossRef]  

4. P. Memmolo, L. Miccio, M. Paturzo, G. D. Caprio, G. Coppola, P. A. Netti, and P. Ferraro, “Recent advances in holographic 3D particle tracking,” Adv. Opt. Photonics 7(4), 713–755 (2015). [CrossRef]  

5. M. Paturzo, V. Pagliarulo, V. Bianco, P. Memmolo, L. Miccio, F. Merola, and P. Ferraro, “Digital Holography, a metrological tool for quantitative analysis: Trends and future applications,” Opt. Lasers Eng. 104, 32–47 (2018). [CrossRef]  

6. J. Barton, “Removing multiple scattering and twin images from holographic images,” Phys. Rev. Lett. 67(22), 3106–3109 (1991). [CrossRef]  

7. P. Guo and A. Devaney, “Digital microscopy using phase-shifting digital holography with two reference waves,” Opt. Lett. 29(8), 857–859 (2004). [CrossRef]  

8. L. Cai, Q. Liu, and X. Yang, “Phase-shift extraction and wave-front reconstruction in phase-shifting interferometry with arbitrary phase steps,” Opt. Lett. 28(19), 1808–1810 (2003). [CrossRef]  

9. S. Lai, B. King, and M. Neifeld, “Wavefront reconstruction by means of phase-shifting digital in-line holography,” Opt. Commun. 173(1-6), 155–160 (2000). [CrossRef]  

10. G. Koren, D. Joyeux, and F. Polack, “Twin-image elimination in in-line holography of finite-support complex objects,” Opt. Lett. 16(24), 1979–1981 (1991). [CrossRef]  

11. T. Tahara, K. Ito, T. Kakue, M. Fujii, Y. Shimozato, Y. Awatsuji, K. Nishio, S. Ura, T. Kubota, and O. Matoba, “Parallel phase-shifting digital holographic microscopy,” Biomed. Opt. Express 1(2), 610–616 (2010). [CrossRef]  

12. T. Tahara, Y. Shimozato, Y. Awatsuji, K. Nishio, S. Ura, O. Matoba, and T. Kubota, “Spatial-carrier phase-shifting digital holography utilizing spatial frequency analysis for the correction of the phase-shift error,” Opt. Lett. 37(2), 148–150 (2012). [CrossRef]  

13. J. Liu and T. Poon, “Two-step-only quadrature phase-shifting digital holography,” Opt. Lett. 34(3), 250–252 (2009). [CrossRef]  

14. Y. Zhang, G. Pedrini, W. Osten, and H. Tiziani, “Reconstruction of in-line digital holograms from two intensity measurements,” Opt. Lett. 29(15), 1787–1789 (2004). [CrossRef]  

15. T. Xiao, H. Xu, Y. Zhang, J. Chen, and Z. Xu, “Digital image decoding for in-line X-ray holography using two holograms,” J. Mod. Opt. 45(2), 343–353 (1998). [CrossRef]  

16. Y. Zhang and X. Zhang, “Reconstruction of a complex object from two in-line holograms,” Opt. Express 11(6), 572–578 (2003). [CrossRef]  

17. L. Denis, C. Fournier, T. Fournel, and C. Ducottet, “Numerical suppression of the twin image in in-line holography of a volume of micro-objects,” Meas. Sci. Technol. 19(7), 074004 (2008). [CrossRef]  

18. T. Nakamura, K. Nitta, and O. Matoba, “Iterative algorithm of phase determination in digital holography for real-time recording of real objects,” Appl. Opt. 46(28), 6849–6853 (2007). [CrossRef]  

19. G. Koren, F. Polack, and D. Joyeux, “Iterative algorithms for twin-image elimination in in-line holography using finite-support constraints,” J. Opt. Soc. Am. A 10(3), 423–433 (1993). [CrossRef]  

20. T. Latychevskaia and H. Fink, “Solution to the twin image problem in holography,” Phys. Rev. Lett. 98(23), 233901 (2007). [CrossRef]  

21. E. Cuche, P. Marquet, and C. Depeursinge, “Simultaneous amplitude-contrast and quantitative phase-contrast microscopy by numerical reconstruction of Fresnel off-axis holograms,” Appl. Opt. 38(34), 6994–7001 (1999). [CrossRef]  

22. B. Kemper and G. Bally, “Digital holographic microscopy for live cell applications and technical inspection,” Appl. Opt. 47(4), A52–A61 (2008). [CrossRef]  

23. U. Schnars and W. P. O. Jueptner, “Digital recording and numerical reconstruction of holograms,” Meas. Sci. Technol. 13(9), R85–R101 (2002). [CrossRef]  

24. P. Langehanenberg, B. Kemper, D. Dirksen, and G. Bally, “Autofocusing in digital holographic phase contrast microscopy on pure phase objects for live cell imaging,” Appl. Opt. 47(19), D176–D182 (2008). [CrossRef]  

25. A. Calabuig, M. Mugnano, L. Miccio, S. Grilli, and P. Ferraro, “Investigating fibroblast cells under “safe” and “injurious” blue-light exposure by holographic microscopy,” J. Biophotonics 10(6-7), 919–927 (2017). [CrossRef]  

26. K. Jaferzadeh, B. Rappaz, F. Kuttler, B.K. Kim, I. Moon, Inkyu, P. Marquet, and G. Turcatti, “Marker-Free Automatic Quantification of Drug-Treated Cardiomyocytes with Digital Holographic Imaging,” Acs Photonics 7(1), 105–113 (2020). [CrossRef]  

27. K. Jaferzadeh, S. Hwang, I. Moon, and B. Javidi, “No-search focus prediction at the single cell level in digital holographic imaging with deep convolutional neural network,” Biomed. Opt. Express 10(8), 4276–4289 (2019). [CrossRef]  

28. I. Moon, K. Jaferzadeh, Y. Kim, and B. Javidi, “Noise-free quantitative phase imaging in Gabor holography with conditional generative adversarial network,” Opt. Express 28(18), 26284–26301 (2020). [CrossRef]  

29. Z. Ren, H. So, and E. Lam, “Fringe pattern improvement and super-resolution using deep learning in digital holography,” IEEE Trans. Ind. Inf. 15(11), 6179–6186 (2019). [CrossRef]  

30. Y. Rivenson, T. Liu, Z. Wei, Y. Zhang, K. Haan, and A. Ozcan, “PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning,” Light: Sci. Appl. 8(1), 1–11 (2019). [CrossRef]  

31. K. Haan, Y. Rivenson, Y. Wu, and A. Ozcan, “Deep-learning-based image reconstruction and enhancement in optical microscopy,” Proc. IEEE 108(1), 30–50 (2020). [CrossRef]  

32. H. Wang, M. Lyu, and G. Situ, “eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction,” Opt. Express 26(18), 22603–22614 (2018). [CrossRef]  

33. G. Zhang, T. Guan, Z. Shen, X. Wang, T. Hu, D. Wang, Y. He, and N. Xie, “Fast phase retrieval in off-axis digital holographic microscopy through deep learning,” Opt. Express 26(15), 19388–19405 (2018). [CrossRef]  

34. K. Wang, J. Dou, Q. Kemao, J. Di, and J. Zhao, “Y-Net: a one-to-two deep learning framework for digital holographic reconstruction,” Opt. Lett. 44(19), 4765–4768 (2019). [CrossRef]  

35. A. Sinha, J. Lee, S. Li, and G. Barbastathis, “Lensless computational imaging through deep learning,” Optica 4(9), 1117–1125 (2017). [CrossRef]  

36. Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light: Sci. Appl. 7(2), 17141 (2018). [CrossRef]  

37. M. Schürmann, J. Scholze, P. Müller, J. Guck, and C. J. Chan, “Cell nuclei have lower refractive index and mass density than cytoplasm,” J. Biophoton 9(10), 1068–1076 (2016). [CrossRef]  

38. P. Müller, G. Cojoc, and J. Guck, “DryMass: handling and analyzing quantitative phase microscopy images of spherical, cell-sized objects,” BMC Bioinformatics 21(1), 226 (2020). [CrossRef]  

39. P. Isola, J. Zhu, T. Zhou, and A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition (2017).

40. P. Moeskops, M. Veta, M. Lafarge, K. Eppenhof, and J. Pluim, “Adversarial training and dilated convolutions for brain MRI segmentation,” in international workshop on deep learning in medical image analysis (2017).

41. M. Faisal, B. Daniel, C. Richard, M. Gregory, S. Kevan, B. Alexander, and D. Nicholas., “Deep adversarial training for multi-organ nuclei segmentation in histopathology images,” IEEE Transactions on Medical Imaging, (to be published).

42. Y. Xue, T. Xu, H. Zhang, L. Long, and X. Huang, “Segan: Adversarial network with multi-scale L1 loss for medical image segmentation,” Neuroinform 16(3-4), 383–392 (2018). [CrossRef]  

43. R. Barer, “Determination of dry mass, thickness, solid and water concentration in living cells,” Nature 172(4389), 1097–1098 (1953). [CrossRef]  

Supplementary Material (2)

NameDescription
Supplement 1       supplementary materials
Visualization 1       Visualization. 1 shows the results of HoloPhaseNet for the 100 images in the test data set

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1.
Fig. 1. (left side) General scheme of off-axis hologram recording. The camera (here a CMOS camera is connected to the PC for numerical reconstruction). After hologram recording, the phase images are conventionally reconstructed by the numerical light propagation method, but we suggest using a deep-learning model to reconstruct single-cell images (right side). Single-level off-axis hologram without spatial filtering in Fourier domain is fed into the model and phase image is automatically generated.
Fig. 2.
Fig. 2. (a) The U-net generator or so-called HoloPhaseNet. The generator receives an input hologram and outputs a phase image. (b) A discriminator evaluates the generator’s output and provides a “real” or “fake” answer during the training process. (c) shows the detail of generator and (d) shows the architecture of discriminator. Batch-Normalization normalizes activation vectors from hidden layers using the mean and variance of the current batch. Batch-Normalization can make the training stable and faster. Leaky Rectified Linear Unit (Leaky ReLU) is a type of activation function based on a ReLU, but it has a small slope for negative values instead of a flat slope. Leaky ReLU is popular in tasks where it may suffer from sparse gradients, for example training generative adversarial networks. The hyperbolic tangent activation function (Tanh) takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1. The sigmoid activation function’s domain is the set of all real numbers, and its output is between 0 and 1. Hence, if the input to the function is either a very large negative number or a very large positive number, the output is always between 0 and 1.
Fig. 3.
Fig. 3. A gallery of sample images used for training the model. The first row shows the off-axis hologram, and the second row shows the corresponding QPI image obtained by the numerical reconstruction algorithm. Single-cell extraction was performed on the QPI image, and the same cell’s location from the hologram image was selected in an automated way. The third row shows the 3D surface representation of each QPI image.
Fig. 4.
Fig. 4. (a) shows a gallery of the validation dataset. The first row shows the off-axis hologram, and the second row shows the corresponding QPI image obtained by the numerical reconstruction algorithm. Single-cell extraction was performed on the QPI image, and the same cell’s location from the hologram image was selected in an automated way. The third row shows the 3D surface representation of each QPI image. (b) The development of the HoloPhaseNet at different epochs and the QPI results for the validation dataset.
Fig. 5.
Fig. 5. (a) HoloPhaseNet’s Generator and Discriminator loss during the training. (b) The average MSE and SSIM for the validation data set.
Fig. 6.
Fig. 6. Average phase values of cells and isolated nuclei for the test dataset. p-value < 0.001; n = 50. The colormap is adjusted for the visualization.
Fig. 7.
Fig. 7. Quantitative analysis of the proposed method and comparison versus that of numerical reconstruction method(a) The PSA of the cells when QPI is obtained with numerical reconstruction method versus that of the PhaseHoloNet for the test dataset, (b) the average phase value (c) the Aspect Ratio, (d) the circularity and (f) shows the average reconstruction error for 100 test images.
Fig. 8.
Fig. 8. Some test data set sample of the reconstructed phase image by HoloPhaseNet. The high-frequency changes shown by an arrow, due to the uneven distribution of material inside the cell is, are preserved by the model.
Fig. 9.
Fig. 9. generalization of the HoloPhaseNet for the bigger hologram reconstruction. The model is trained with 900 images of the size 512×512 pixels. The first row shows three sample holograms used for the training and 2nd row shows the corresponding phase image reconstructed by the numerical method. The third row shows three holograms used for the HoloPhaseNet test. The last row shows the model’s output.

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

P S A = N p s 2 ,
ϕ = ( i , j ) S p φ ( i , j ) N ,
A R = M a j o r / M i n o r ,
c i r = 4 π P S A p r 2 ,
D M = λ 2 π α ( i , j ) S p φ ( i , j ) ,
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.