1. Introduction

It can reduce the visibility of the scene behind the glass for reflections when shooting through glass surfaces in the real world. Reflections not only degrade image quality, but also degrade computer vision performance. This situation can be addressed by a single image reflection removal (SIRR) process designed to improve scene visibility of the background. Reflection removal is a challenging problem, and the challenges come from at least two aspects: (1) It is difficult to obtain real reflection datasets in the real world, and many studies will conduct research in a synthetic way [1–3]; (2) reflections have diverse variations, e.g., glass thickness [4], image misalignment [5], etc., that affects the rendering of reflections, so reflection removal needs to be given high priority in computer vision.

To address this problem, many non-learning-based methods [6, 7] employ a two-stage approach that first locates the reflective regions, for instance, is solved by classifying the background and reflective edges [6, 7] and the problem is then solved using [8] a method proposed to recover the background layer based on edge information. However, locating the reflective area itself is a very challenging task, so these methods mainly rely on some heuristic observations [7], or must involve user interaction [8], which are not applicable in many scenarios. In order to solve the reflection removal problem more efficiently, recent methods start to use deep learning techniques to solve this problem, such as CEINet [9] proposed by Fan et al., which is a pioneering technique. It is the first method to use deep learning and propose an end-to-end framework to solve reflection removal problems. Benefiting from deep learning frameworks, these methods exhibit good modeling ability to capture the features of various reflection images.

Nevertheless, gradient inference is still used like many non-learning-based methods [7, 8] and do not fully exploit the multi-scale information of background restoration. Furthermore, these methods [9] are mainly trained with synthetic images. However, synthetic images cannot capture comprehensive information during real-world image formation, so the removal of reflections from real-world images is still limited. In order to solve the problems encountered [9], Wan et al. [10] proposed a network with a feature sharing strategy (Cooperative Reflection Removal Network, CoRRN), by integrating image background information and multi-scale gradient information to solve the problem of reflection removal in a network, in addition to introducing a new loss function, based on the statistical loss of gradient level for the removal of local strong reflections. Wei et al. [5] use a context encoding module to reduce the uncertainty of local strong reflections, and in the loss function part, an alignment-invariant loss function is introduced to target the data set with image misalignment. To sum up, the current methods assume the removal of reflections in specific cases and provides its own dataset for study.

When we decomposed the reflection image into high and low frequency sub-images at different levels through wavelet transform, and observed that the reflections are almost present in high frequency sub-images. More specifically, we randomly selected 30 pairs of reflection and non-reflection images from the SIR² dataset [11], decomposed them into high and low frequency sub-images at different levels through SWT, and calculated the difference of mean square error (MSE). Most of the reflections can be observed at high frequencies, as shown in Fig. 1. In addition, current deep learning methods have not combined wavelet transform to remove reflections, but this method has been widely used in the fields of super-resolution [12, 13], haze removal [14] and object detection [15]. Hence, we proposed to use wavelet transform as prior knowledge to process high-frequency reflection and low-frequency background context separately to obtain the better reflection-removal results in this study.

 

Fig. 1. Percentage of MSE difference between reflected and non-reflected images.

Download Full Size | PDF

In this study, we propose a novel Translation-invariant Context-retentive Wavelet Reflection Removal Network (TiCrWRRNet), which can not only remove reflections but also retain the background context. The reflection image is decomposed into high-frequency and low-frequency sub-images at different levels several times with SWT. The decomposed high-frequency and low-frequency sub-images are then processed with different sub-networks to learn the end-to-end mapping to obtain the reflection maps at different levels, and finally to use to remove the reflections. Detail-enhanced reflection layer Removal Subnetwork (DRSn) and Detail-enhanced Reflection Information Transmission (DRIT) are both proposed to transmit the reflection layer information from DRSn to Context Retention Subnetwork (CRSn) through DRIT, so that CRSn can effectively separate the transmission layer and the reflection layer to further improve the results of reflection removal. To retain the background context and remove a small number of reflections that appear at low frequencies, we propose CRSn and establish a recursive context retention method to gradually remove low-frequency reflections at each level and retain the background context. We reconstructed the nth-level high and low-frequency de-reflection results by inverse wavelet transform (ISWT), performed Context Level Blending (CLB) with the low-frequency sub-images in the (n-1)th-level de-reflection results, and used for ISWT reconstruction with the high-frequency de-reflection results in the (n-1)th-level. It is repeated for the output results at each level with DRSn and CRSn until the final clean image is reconstructed. The contributions of this study are summarized as follows:

2. Related work

2.1. Single image reflection removal

It has been widely investigated for reflection removal in recent years. Current studies can be divided into multi-image reflection removal and single-image reflection removal. The former usually utilizes conditional constraints to alleviate the uncertainty of resolving reflections, e.g., polarization angles [16–18], or reflection differences [19]. These conditional priors are rarely considered in the context of a single image. However, this paper mainly focuses on the latter. At present, the reflection removal method for single image is mainly based on the content of the image, and usually makes assumptions about the image, as shown below:

In the latter case, i.e. single image reflection removal, since deep learning has achieved good performance on both high-level and low-level computer vision problems, its comprehensive modeling capabilities are also beneficial for reflection removal. Fan et al. [21] exploited edge information to better preserve image details. Zhang and Chen [1] utilized perceptual information during training to improve the fidelity of predictions. However, they mainly used on synthetic images, which led them to limitations in removing real reflections. Wan et al. [10] integrated image background information and multi-scale gradient information in a cooperative and unified framework to solve the reflection problem. In addition, in order to remove the strong reflections in some local areas, a statistical loss considering the gradient level statistics between background and reflections is proposed and a reflection image dataset is provided. Ding et al. [22] used the polarization properties of reflections to detect and remove reflections in LWIR images and invented a joint reflection detection method, they showed that this detection method can effectively identify the real reflection areas in the image, so it can effectively remove the reflection in LWIR images. Li et al [23] used the Stokes algebra and Mueller calculus formulae with a novel edge-based technique to separate the reflections of visible light, near infrared, and Long Wave Infrared Wavelengths (LWIR), in addition to exploiting spectral information and patch-wise operation to improve robustness, and results show that it can be applied to optically smooth reflecting and partially transmitting surfaces, such as those caused by glass surfaces. Zheng et al. [4] thought that the absorption effect needed to be considered, so they re-examined the formation model of the reflection image, and theoretically provided a single image reflection removal method. From the above, it can be seen that most of the current methods assume reflection removal under specific conditions. Although these methods have good results, real-world reflections are not specific to a certain hypothesis. Therefore, these methods are still limited.

2.2. Image restoration with wavelet prior

Wavelet transform is often used as image preprocessing or postprocessing [24,25], and down-sampling or up-sampling operations are also used to design deep networks [26,27] in today's deep learning process. However, wavelet transform is widely used in various fields [28–31] except reflection removal. For example, in the field of super resolution, Guo et al. [13] proposed a deep wavelet super resolution (DWSR) technique to obtain good SR results by using wavelet sub-bands as input to recover the missing details. In image restoration, Liu et al. [26] proposed a multi-level wavelet network (MWCNN) for image restoration and that can be effectively applied in image denoising, SISR and removal of artifacts generated by JPEG compression. In the field of rain removal, Yang et al. [14] proposed a recurrent wavelet learning and constructed dilated residual dense networks to alleviate the problem of different sizes of rain streaks in the training and testing phases, thereby effectively removing rain streaks.

3. Proposed method

3.1. Translation-invariant wavelet transform

In Eq. (1), we decompose the reflection image (${R^0}$) with Haar stationary wavelet transform to obtain wavelet bands of different scales. It is expressed as follows:

3.2. Multi-level subnetwork models

In Fig. 2, The reflection image is decomposed into multiple levels of high and low frequency images with SWT and then processed with the Context Retention Subnetwork (CRSn) and Detail-enhanced reflection layer Removal Subnetwork (DRSn) respectively to predict the corresponding images with reflection removal and context retention. In order for the network to learn higher-level complex mappings, lower-level features are made shareable with higher-level ones. For each network structure, local residual learning and whole residual learning are used, which can help when learning highly complex features and gradient backpropagation.

 

Fig. 2. Network architecture of proposed TiCrWRRNet.

Download Full Size | PDF

3.2.1 Detail-enhanced reflection layer removal subnetwork

In Fig. 3, the DRSn is modified from [32] by removing the upscale part. In addition, in order to obtain the high-frequency image $\textrm{I}_{\textrm{Detail}}^\textrm{n}$ of the nth-level SWT wavelet decomposition, we use a convolutional layer with a kernel size of 3*3 to get the shallow feature $C_{H{C_1}}^n$, as follows:

 

Fig. 3. Architecture of proposed CRSn and DRSn.

Download Full Size | PDF

Finally, after putting $C_{H{C_2}}^n$ into the convolution layer output, the high-frequency de-reflection image $G_{Detail}^n$ can be obtained, as shown below:

3.2.2 Context retention subnetwork

In Fig. 1, although most of the background context exists at low frequencies, there are still a few reflections. Therefore, we strengthen the Context Retention Subnetwork to achieve the effect of retaining the background context and removing reflections. As shown in Fig. 3, our CRSn refers to the network structure [33] and modifies it to remove the batch-normalization layers and sub-pixel layers in the network structure respectively to obtain a better recursive effect. First, we extract the shallow features of the low-frequency image ${I_{LL}}$ decomposed by the SWT operation through the convolution layer, as shown below:

3.2.3 Context level blending

Context-level blending (CLB) can effectively remove reflections from low-frequency images and retain background context. To enhance our method's result of retaining low-frequency background context at each level, we use CLB at each level recursively. As shown in Fig. 2, we first connect the high and low frequency images of the n-th level through DRSn and CRSn to do the ISWT inverse wavelet transformation. Then, the result of ISWT of the n-th level high and low frequency image and the (n-1)-th level low frequency context retention result $G_{LL}^{n - 1}$ are used for CLB operation to help strengthen the low frequency reflection removal and retain the background context information. Finally, an ISWT inverse wavelet transformation is performed on the weighted image and the (n-1)-th order high-frequency de-reflection result $G_{Detail}^{n - 1}$, and it is repeated recursively until the last-level de-reflection image is reconstructed, that is, the final de-reflection image is restored. The reflection result, ${\textrm{r}^0}$, looks like this:

3.2.4 Detail-enhanced reflection information transmission

In the low-frequency DRSn and high-frequency CRSn networks, the terms ${\textrm{f}_{RLB}}$ and ${\textrm{f}_{\textrm{RDB}}}$ were mentioned. RLB and RDB are operations for extracting deep features to remove reflections. The enrich ${\textrm{f}_{\textrm{RLB}}}$ and ${\textrm{f}_{\textrm{RDB}}}$ are enriched through Detail-enhanced Reflection Information Transmission (DRIT) to enrich features and increase robustness [34]. In addition, in order for the network to learn how to separate the Transmission layer and the Reflection layer and to get a clear image, our network only learns the residual between the input and output and applies a whole residual connection [35].

The architecture of RLB and RDB is shown in Fig. 4 respectively. RLB contains H residual blocks with the same structure. We introduce the residual block [33], but remove the batch regularization layer and use PReLU [36] as the activation function instead as well as two convolutional layers with kernel size 3*3 and 64 feature maps.

 

Fig. 4. Architecture of RLB and RDB.

Download Full Size | PDF

${C_{D{D_{h - 1}}}}$ is defined as the input of the extracted depth feature of the h-th residual dense block. For the h-th residual dense block, the input is the concatenation of all the outputs of the previous layer and the original input. It can be expressed as $[{{C_{D{D_{h - 1}}}},{C_{D{D_{h,1}}}},{C_{D{D_{h,2}}}}, \ldots ,{C_{D{D_{h,m - 1}}}}} ]$. So, the output looks like this:

3.3. Loss function

There are p pairs of reflective and non-reflective images ${\{{{y_i},{x_i}} \}_{i = 1, \ldots ,p}}$. The nth-level images ${y_{i,n}}$ and ${x_{i,n}}$ are obtained by SWT in the n-th level (n = 1, 2, …, N):

4. Experiments

4.1. Dataset description

It is difficult to obtain the reflection datasets in the real world and the size of datasets is usually too small, and it would lead to the deep learning model over-fitted. Therefore, we perform data augmentation to overcome the issues. In order to train the deep learning model with high accuracy and good generalization, in addition to the structure of the model itself, one of the most important things is how much data we have. The larger the amount of data, the more complete the data, the better. Hence, the SIR² dataset [11] is used by changing the color and flipping the image to augment the data. More specifically, in data augmentation of the experiments, we perform image flipping (including horizontal flipping, vertical flipping, horizontal and vertical flipping) and image enhancement (including adjusting brightness and contrast).

Furthermore, data balancing is used to avoid over-learning the features of a certain category, causing images of other categories to be unrecognizable. Therefore, when there are multiple categories of data sets, it is necessary to ensure that the data of each category does not fall too far to improve the accuracy of the model. The SIR² dataset [11] is divided into three categories with a total of 454 images, namely 199 Postcard images, 200 Object images, and 55 Wild images. However, the number of images in each category has a gap, which may affect the training of the model. Therefore, we balance the number of images for each category. After balance, the numbers of images are 1990 Postcards, 2000 Objects, and 2200 Wilds in the experiments respectively.

4.2. Implementation details and hyper-parameter setting

The proposed TiCrWRRNet is trained on an NVIDIA RTX2080 TI GPU with the Tensorflow framework. For the training set, the size of each mini batch is 16. We use Haar SWT to perform wavelet decomposition on the training set, and the level of wavelet transform is set to 2, and the decomposed high and low frequency sub-images of each level are input into DRSn and CRSn respectively for training. Adam [37] is used as the network optimization, and the learning rate for training CRSn and DRSn is set to 10⁻⁴. The numbers of RLB and RDB in the network are both set to a kernel size of 16.

4.3. Comparisons with the state-of-the-art

To evaluate the performance of our method, we compare our method with six state-of-the-art reflection removal approaches, namely Chang [38], Kim [3], Peng [39], RmNet [2], ERRnet [5] and IBCLN [40], on the SIR² dataset [11], which contains images from three categories: Postcard, Object and Wild. In addition, the Nature [40] dataset is also used to demonstrate that our method is not limited to specific assumptions for reflection removal.

4.3.1 Quantitative evaluation

In the experiments, for a fair comparison, the codes provided by the authors were used and the training parameters were directly used as suggested in their original paper. Table 1 lists the quantitative comparison of the state-of-the-art approaches and ours on four benchmark datasets in terms of SSIM and PSNR. The results indicate that our method outperforms the state-of-the-art approaches on both SSIM and PSNR, and it means that our method recovers images with better visual effect and closer to the reflection-free image.

Table 1. Quantitative comparison of state-of-the-art and our methods on four benchmark datasets, including three sub-datasets of SIR²[11] dataset.^a

View Table | View all tables in this article

4.3.2. Visual comparison

Visual comparisons among the six state-of-the-art approaches and our methods in SIR² dataset [11] are shown in Fig. 5. In first example, most approaches cannot effectively remove the chromatic aberration caused by reflection. However, our method can fade this part and make the color closer to the original image. In the second example, on the background pattern and in the third, on the right half, our results clearly outperform the results of other approaches. In these examples, although there are some areas where reflections from small details could not be removed completely, our method can remove reflections more effectively and restore the details of the background more clearly.

 

Fig. 5. Visual comparison among the state-of-the-art approaches and our method in SIR2 dataset [11].

Download Full Size | PDF

4.4. Ablation studies

4.4.1 Detail-enhanced reflection information transmission

DRIT is proposed to extract reflections at high frequencies, which is passed on to CRSn to help achieve better performance. Experimental analysis is used to find how the proposed DRIT helps to improve the results of reflection removal. To demonstrate its advantages, ablation experiments are conducted by comparing w/o DRIT and DRIT. Both models are evaluated on the SIR² dataset [11]. The results are shown in Table 2. The quantitative results show that the effect of adding DRIT is better in terms of both PSNR and SSIM, thus verifying that the use of DRIT can effectively improve reflection removal. In the visual comparison, as shown in Fig. 6, in the first example, it can be seen that the result of w/o DRIT cannot completely remove the reflection although the reflection is diluted, so there is still an afterimage of residual reflection. However, the result of adding DRIT shows that it can clearly remove reflections relatively cleanly. In the second example, the same result of adding DRIT is better and closer to the ground truth. From the quantitative and visual comparison, it can be observed that DRIT performs better than “w/o DRIT” in both detail and reflection removal.

 

Fig. 6. Visual comparison of w/o DRIT and DRIT.

Download Full Size | PDF

Table 2. Quantitative evaluation without and with DRIT

View Table | View all tables in this article

5. Conclusion

A novel TiCrWRRNet is proposed to effectively achieve the reflection removal. Deep learning sub-networks can converge efficiently and quickly by using high and low frequency sub-images obtained from the decomposition of SWT. In addition, a multi-level wavelet transform is used to further enhance the removal of reflections. The sub-images at different levels are trained with DRSn and CRSn to extract richer image background context and reflections respectively. Since a small number of reflections still exists in low frequencies, a DRIT is proposed to extract the reflection layer features of high-frequency sub-images to aid CRSn to separate the transmission layer and the reflection layer more effectively to achieve the performance of reflection removal. Furthermore, CLB and ISWT are employed on the low-frequency results at each level recursively to recover the final clean image. Finally, comparative and ablation studies are conducted to demonstrate that the proposed method both shows better performance than state-of-the-art ones, and it is validated to generalize effectively to other complex datasets.