1. Introduction

As a non-destructive and high-resolution imaging technology, optical coherence tomography (OCT) has been developed rapidly since it was proposed in 1991 [1]. There are many physical methods for OCT resolution enhancement, such as micro-OCT [2], synthetic aperture [3], structured illumination for lateral super-resolution [4] and multiframe superresolution [5]. Its axial resolution reaches micron level by using ultra-wide spectrum of Ti: Sapphire laser [6] or supercontinuum (SC) light source [7–8] up to now. However, limited by cost and available technological level, it is difficult to continuously upgrade the resolution immensely only based on physical methods.

Benefiting from the improvement of computing power, some digital signal or image processing methods have been attempted to improve the resolution of OCT images. In 2013, L. Fang et al. [9] introduced the sparsity-based simultaneous denoising and interpolation (SBSDI) which used coupled orthogonal matching pursuit to construct sparse representation dictionaries from previously collected datasets to reconstruct the retinal OCT images. In 2017, they further proposed a segmentation-based framework sparse reconstruction (SSR) method, which used automatically segmented retinal layer information to build layer specific structure dictionaries for retinal OCT image reconstruction [10]. In 2018, A. Abbasi et al. [11] proposed a method based on nonlocal weighted sparse representation, which combined multiple sparse representations of similar noise and denoised patches to better estimate the sparse representation of each patch. However, the sparsity-based methods may smooth the reconstructed HR images, which will lead to the loss of the sharp boundaries.

Deep learning has ultra-strong learning ability and has also been used in the field of image super-resolution. In 2014, C. Dong et al. [12] proposed the image super-resolution method based on convolutional neural network (CNN) for the first time. After that, many super-resolution networks based on deep learning were proposed in succession, such as the very deep convolutional networks (VDSR) [13], the enhanced deep residual networks (EDSR) [14], super-resolution using a generative adversarial network (SRGAN) [15], the blind super-resolution with iterative kernel correction (IKC) [16] and the deep plug-and-play super-resolution for arbitrary blur kernels (DPSR) [17]. Super-resolution methods based on deep learning have been also studied in medical images, such as MRI [18–19] and microscopy [20].

Y. Huang et al. [21] firstly applied the super-resolution method based on deep learning to OCT images, and they denoised and super-resolved the low-resolution (LR) retinal OCT images simultaneously. It showed better performance compared with the sparsity-based methods. They got HR images through registering and averaging several B-scans obtained at the same position. In order to get LR images, they downsampled the noisy patches based on bicubic interpolation in the patch pairs with the scale factor of 2×, 4× and 8×. In 2020, V. Das et al. [22] introduced an unsupervised OCT image super-resolution method, which used two generators and two discriminators and did not require one-to-one alignment between the LR and the HR images during training. Their LR images are also synthetically generated by down-sampling the noisy images with the factor of 2× and 4× of bicubic interpolation.

The above two studies both used public datasets of retinal OCT images and the data downsampling method they used are different from the imaging principle of OCT. In order to get closer to principle of real OCT imaging, we use a degradation method similar to OCT sampling to degrade the OCT axial resolution by multiplying a narrow spectral width of the OCT light source in our study. Then, three super-resolution networks, deep residual network for super-resolution (SRResNet) [15], residual-in-residual dense block network (RRDBNet) [23] and super-resolution using a generative adversarial network (SRGAN) [15] are selected to improve the resolution of OCT images.

2. Method

2.1 Collection of OCT images

The signals of OCT system are obtained by low coherence interference technology, which collects reflected light from samples with different depths. The B-scan images obtained from the OCT system can show the morphological features of the tissue microstructure, whose theoretical axial resolution is mainly determined by the center wavelength and the full width half maximum (FWHM) of its light source. When the output spectrum of light source is Gaussian type, the theoretical formula of axial resolution of OCT system can be calculated as follows [24],

In our study, we obtained HR images and LR images based on a home-made spectral domain OCT (SD-OCT) system [25], whose light source is a super luminescent diode (SLD) (BLM2-D, Superlum) with the central wavelength of 840 nm and the FWHM of 100 nm. Its axial resolution in air and transverse resolution in the focal plane are 3.4 µm and 13 µm, respectively.

After collecting the OCT signal of the sample, the original interference spectrum is multiplied by two Gaussian functions with different widths, to obtain HR and LR signals, whose FWHM are 64 nm and 17 nm, respectively. Then, HR and LR images were obtained by doing inverse Fourier transform, whose axial resolutions are about 5 µm and 18 µm, respectively. HR images will be regarded as the ground-truth images. The advantage of this kind of degradation method is that image pairs of HR and LR images with the same size can be calculated directly, which avoids image pixel registration when they are collected separately.

2.2 Generation of dataset

Skin of zebrafish has rich microstructure details, which is very suitable for the evaluation of image super-resolution effect. Zebrafish, older than 90 days, were scanned to obtain OCT images in our study. As described in Ref. [25], zebrafish were immersed in 0.024% and 0.012% tricaine solutions successively to be anesthetized. Then, they were scanned by the SD-OCT system, and HR and LR OCT images were obtained based on the methods of Section 2.1.

We collected 25 pairs HR and LR OCT images of zebrafish with 1000 × 2048 pixels × pixels (width × height), respectively, and extracted 1000 × 1004 pixels × pixels with effective information in each image. Each training and test dataset consists of 20 and 5 pairs of OCT HR and LR images, respectively.

For comparison, we also did 4× and 8× bicubic interpolation for HR images in axial direction to get bicubic LR images. In super-resolution study, bicubic interpolation is widely used for the up-sampling and down-sampling of images. When we expressed the meaning of bicubic interpolation in this paper, we used the concept of bicubic down-sampling in order to be consistent with the existing description in super-resolution study. In order to reduce the axial resolution of the image by bicubic interpolation, the lateral scale was set to 1, and the axial scale was 1/4 or 1/8. Then, a bicubic interpolation was again carried out by setting the lateral scale to 1 and the axial scale to 4 or 8, and the image was resized back to the original size. Finally, we obtained the OCT image with reduced axial resolution by bicubic interpolation (4× and 8×).

Rotation and mirror flip were carried out to expand the datasets. Then, each big image was cropped into small images with 64 × 64 pixels × pixels. At last, 2566 and 439 pairs of small images were obtained in each training and test dataset, respectively.

2.3 Networks

Three classical super-resolution networks, SRResNet, RRDBNet and SRGAN, were trained to obtain the super-resolution images. Because the size of our LR - HR image pair is the same, the up-sampling module of these networks were removed, and other parameters were set according to their original networks. In training section, we used Adam optimizer for all of these networks. The learning rate was initialized with 10⁻⁴ and it was decayed with the increasing of iterations. The models were trained on batch-size of 16 for 400,000 iterations. All work was implemented by using Pytorch software and performed on an NVIDIA TITAN RTX GPU with 24G graphic memory volume.

3. Results

The images of different degradation methods are shown in Fig. 1. Figure 1(a) is the HR image. Figures 1(b) and (c) are LR images of bicubic interpolation downsampling of 4× and 8×, respectively. Figure 1(d) is the LR image. The zoomed versions of the highlighted regions pointed out by red rectangles in each image are displayed on the lower side of the corresponding image for better visualization. As shown in Fig. 1(b), after 4× bicubic interpolation downsampling for the OCT image, the image becomes slightly blurred, but the thin white lines in the image can still be clearly distinguished. After 8× bicubic interpolation downsampling, these white lines are more blurred in Fig. 1(c). Compared with Figs. 1(b) and (c), although the resolution is degraded from 5 µm to 18 µm, the white lines are severely widened in the axial direction in Fig. 1(d), which makes them difficult to be distinguished.

 

Fig. 1. Image comparison of different degradation methods. (a) HR image. (b) and (c) LR images after 4× and 8× bicubic interpolation, respectively. (d) LR image of actual OCT degradation. Scale in each image is the same.

Download Full Size | PDF

The reconstruction results of super-resolution images based on SRGAN, SRResNet and RRDBNet are shown in Fig. 2. The zoomed versions of the highlighted regions pointed out by yellow and red rectangles at the bottom of the corresponding image for better visualization. Figure 2(a) is the HR image. In order to better compare the differences of these three degradation methods, profiles of A-lines pointed out by red dashed lines in Figs. 2(a), (c1), (d1) and (e1) are shown in Fig. 2(b). As shown in Fig. 2(b), in axial direction, the red curve (4× bicubic) and green curve (8× bicubic) are close to the blue curve (HR image), and the purple curve (the actual OCT degradation) deviates greatly from the blue curve, which explains why the images generated by actual OCT degraded imaging look more different from HR images than those of the bicubic interpolation. Although the axial resolution is degraded from 5 µm to 18 µm, the results of Fig. 2(b) demonstrate that the actual OCT downsampling loses more information than 4× and 8× bicubic interpolations.

 

Fig. 2. Reconstruction results of super-resolution images. (a) HR image. (b) Comparison of A-line profiles from different degradation methods. (c1) - (c4), and (d1) - (d4)) are LR images of 4× and 8× bicubic interpolation and their reconstruction results of SRGAN, SRResNet and RRDBNet, respectively. (e1) - (e4) are the LR image of actual OCT degraded imaging and its reconstruction results of SRGAN, SRResNet and RRDBNet, respectively. The scale of all OCT images is shown in (a).

Download Full Size | PDF

Figures 2(c1) - (c4), and 2(d1) - (d4) are LR images of 4× and 8× bicubic interpolation and their reconstruction results of SRGAN, SRResNet and RRDBNet, respectively. Figures 2(e1) - (e4) are the LR image of actual OCT degraded imaging and its reconstruction results of SRGAN, SRResNet and RRDBNet, respectively. As we can see, the results of SRAGN [Fig. 2(c2)], SRResNet [Fig. 2(c3)] and RRDBNet [Fig. 2(c4)] for 4× bicubic dataset are all pretty good. As indicated by the yellow arrows in Figs. 2(c3) and 2(c4), the track trace and thickness of the reconstructed scales of zebrafish are very close to the HR image [Fig. 2(a)]. This is because the LR image [Fig. 2(c1)] did not lose much detail information compared with the HR image. As shown in Figs. 2(d2), (d3) and (d4), these three networks also achieve good performance for the LR image of 8× bicubic interpolation. The track trace and thickness of scales of zebrafish can be also reconstructed very well. For the LR image generated by actual OCT degradation, the reconstructed image of SRGAN [Fig. 2(e2)] is the best among these three networks, where the micro-texture of scales is almost reconstructed. The images of SRResNet [Fig. 2(e3)] and RRDBNet [Fig. 2(e4)] are better than the LR image [Fig. 2(e1)], but it is not good enough to reconstruct the scales of zebrafish completely.

Three commonly used metrics, peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and mean opinion score (MOS) were used to evaluate the super-resolution performance quantitatively.

PSNR is based on the error between corresponding pixels of two images, which is simply defined by mean squared error (MSE) as follows.

SSIM is used to compare the texture change between two images by calculating the similarity from the aspects of luminance (l), contrast (c), and structure (s), which is defined as follows:

MOS is a visual evaluation parameter, which is obtained by the observer’s subjective evaluation of image quality. We performed the mean-opinion-score test to quantify the visual effect of different methods. Specifically, we invited 20 volunteers to assign a score from 1(bad) to 5(excellent) by comparing the output images of different networks with the HR images. The scores were averaged as the final evaluation results.

The mean values of these parameters for the test dataset are shown in Table 1. As shown in Table 1, for 4× and 8× bicubic datasets, SRResNet achieved the best mean of PSNR and the highest SSIM value. However, compared with their LR images, the improvement of PSNR and SSIM is very small, since their LR images are not much different from the HR image. For the dataset generated using the actual OCT degradation, RRDBNet has the best mean of PSNR value and SRResNet has the best mean of SSIM value. The unit of PSNR in Table 1 is dB.

Table 1. Quantitative evaluation results of different methods

View Table

In each row, SRGAN has the smallest values in PSNR and SSIM among these three networks. Although the PSNR and SSIM of SRResNet and RRDBNet are higher than those of SRGAN for actual OCT degradation, the super-resolution images of SRResNet and RRDBNet are over-smoothed, which are difficult to reconstruct the details, and their visual effect is not good. SRGAN has the best visual effect among these methods.

In each column, we can see that the LR image of the actual OCT degradation has the worst PSNR and SSIM, which means it is the most different from the HR image, and induces these three networks also have the minimum PSNR and SSIM value. However, compared with all of LR images, the PSNR and SSIM of the actual OCT degradation have the most obvious improvement.

From Table 1, we can notice that the PSNR and SSIM of SRGAN method for 4× and 8× bicubic datasets are lower than the results of themselves without any process. We think that the main reason for this phenomenon is that SRGAN was trained based on content loss and adversarial loss, which may reduce the PSNR and SSIM values of the results. This is caused by competition between the MSE-based content loss and the adversarial loss. In addition, we found PSNR and SSIM sometimes cannot reveal the improved visual effect authentically, and high PSNR and SSIM do not mean better visual performance.

The subjective evaluation parameter, MOS, was often alternatively used to evaluate the super-resolution performance quantitatively. As shown in Table 1, we can see SRGAN has the best results for all kinds of super-resolution images.

4. Discussions and conclusion

Based on the actual resolution degradation in OCT imaging, we proposed a more realistic low-resolution OCT image generation approach for training deep neural networks. It is demonstrated that the resolution of LR OCT image can be improved based on deep learning, and the extent of improvement is better than those of 4× and 8× bicubic interpolation.

In order to obtain better super-resolution results, many high-performance network structures have been proposed in the previous super-resolution research, but their downsampling methods are almost predefined as the simple and uniform bicubic interpolation, which is quite different from the specific OCT imaging. According to the above qualitative and quantitative comparison, for the actual OCT degradation, image super-resolution is more challenging than those of simple bicubic interpolation and it has higher training difficulty. In the meantime, we think that the more reasonable degradation model is of great significance for the practical application in OCT imaging, which determines the actual super-resolution effect of OCT images.

In recent two years, some new networks like IKC [16], DPSR [17] and USRNet [26] were proposed in the field of image super-resolution, which achieved excellent performance. In addition, the discriminator of generative adversarial network [27] can help the generator to recover genuine textures. We will compare and select optimum networks in OCT image super-resolution in the future. Besides, we will further use larger and richer datasets to study the super-resolution of OCT images to improve the robustness of the model.

In addition, one important scenario for super-resolution study is in vivo clinical applications like surgery guidance where low-resolution acquisition is preferred to minimize motion artefacts and to save time. We will further pursue the applicability in in vivo OCT images acquired by our handheld probe [28].

Axial and lateral resolutions of OCT system are determined by different factors, which provide us the possibility of super-resolution study separately. Only axial super-resolution is studied in our study. Actually, lateral super-resolution is more important for obtaining high lateral resolution and large depth of focus. Our ultimate goal is to improve the resolution of two directions and make the existing OCT system have higher resolution by digital methods at the lowest cost.