Open Access
Add to BibSonomy
Abstract
Underwater images captured by optical cameras can be degraded by light attenuation and scattering, which leads to deteriorated visual image quality. The technique of underwater image enhancement plays an important role in a wide range of subsequent applications such as image segmentation and object detection. To address this issue, we propose an underwater image enhancement framework which consists of an adaptive color restoration module and a haze-line based dehazing module. First, we employ an adaptive color restoration method to compensate the deteriorated color channels and restore the colors. The color restoration module consists of three steps: background light estimation, color recognition, and color compensation. The background light estimation determines the image is blueish or greenish, and the compensation is applied in red-green or red-blue channels. Second, the haze-line technique is employed to remove the haze and enhance the image details. Experimental results show that the proposed method can restore the color and remove the haze at the same time, and it also outperforms several state-of-the-art methods on three publicly available datasets. Moreover, experiments on an underwater object detection dataset show that the proposed underwater image enhancement method is able to improve the accuracy of the subsequent underwater object detection framework.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Underwater vision plays an important role in the missions of marine investigations and underwater robotic explorations. However, the images captured underwater are usually degraded by the absorption and scattering effects which occur during the light propagation [1,2]. Underwater images acquired in this situation frequently suffer from haze, low contrast and color distortion, especially in the marine environment (as shown in Fig. 1). These degraded image quality threatens the underwater object recognition task [3]. Therefore, enhancing the degraded underwater images becomes significant to a variety of vision-based applications in the underwater scenes.
Fig. 1. The underwater imaging model (left) and the results of the proposed underwater image enhancement method (right).
Underwater image enhancement techniques are usually regarded as a pre-processing operation for other underwater tasks, e.g. real-time navigation [4] and deep-sea organisms tracking [5]. Many researchers dedicate to develop underwater image enhancement approaches to restore the images captured in seawater [6–19]. Color distortion and haze removal are two main objectives of underwater image enhancement task. The generation of color distortion and haze in underwater images are interrelated, both of which dependent on the distance light travels in the water and the wavelength of the light [2]. Dark channel prior (DCP) [20] and haze-line [21] have achieved great success in dehazing and image enhancement for images captured in the air, and also provide solutions for underwater image enhancement task. Inherited from [20] and [21], researchers have proposed several methods to enhance underwater images [22–25]. However, it is still a tricky issue to simultaneously restore the color distortion and remove the haze for underwater images.
In this paper, a new underwater image enhancement method will be proposed, which aims to restore severely hazed underwater images of blueish or greenish tones collected in seawater. To address these two concerns (color and haze) and achieve robust recovery performance, we firstly build an adaptive color compensation model to restore the color distortion, which consists of three steps: background light estimation, color recognition, and color compensation. Then we use haze-line technique to further remove the haze and enhance the image details. Compared with several state-of-the-art methods, our proposed method are more robust and stable.
The main contributions of this paper are summarized as follows: (1) We propose a novel underwater image enhancement framework that can robustly restore the color distortion and remove the haze in underwater images; (2) To restore the color distortion, we propose an algorithm that can automatically estimate the background light and recognize the color of the water body; (3) we carry out extensive experiments, qualitative and quantitative results demonstrate that our proposed method is superior to other state-of-the-art methods.
The rest of this paper is outlined as follows: Section 2 summarizes the related works. Section 3 introduces the proposed underwater image enhancement method in detail. Section 4 describes the summary of the restoration algorithm. Section 5 reports and discusses the experimental results. Section 6 concludes the paper.
2. Related works
The underwater image enhancement techniques are closely related to the complex underwater imaging environments, which mainly focus on restoring the color distortion and removing the haze from the underwater images. To address these tricky issues, the mainstream techniques on underwater image enhancement include model-based and model-free restoration methods [3].
2.1 Model-based restoration methods
DCP was firstly proposed by He et al. [20] for enhancing images captured in the air, where there is at least one channel with low intensity for a pixel based on the observation on a haze-free image. However, DCP shows a limited ability in restoring underwater images due to the heavily information lost in the red channel. To address this issue, several works [11,22,23,25,26] have been proposed to promote the development of DCP. Drews et al. [11] propose underwater dark channel prior (UDCP) with a statistical prior based on the properties of the images obtained in outdoor natural scenes, which can restore the color in real and simulated underwater scenes. Codevilla et al. [22] take a further step on DCP to improve the underwater image quality using rough distance map estimation and model simplifications. Later, Galdran et al. [23] propose a variant of the dark channel method, namely the red channel method. The red channel method is potential in restoring the light of long wavelengths. Li et al. [26] propose a method using the combination of DCP, gray-world assumption, and an adaptive exposure map to achieve underwater image quality recovery, but the contrast and brightness of the restored images still need further improvement in the subsequent results. Zhou et al. [14] propose an approach using the inverted red channel to achieve the transmission map estimation for underwater images, but the approach is unstable and some results appear reddish. This is because they increase the value of the red channel to reduce the influence of artificial light sources. Studies have shown that DCP has largely contributed to the development of the underwater image enhancement techniques. However, there is a drawback for DCP-based methods that the accuracy of pixel estimation declines when severe color distortion exists (e.g. severe greenish or bluish underwater images) and brightness variation changes in underwater images. The underlying assumption does not hold in these underwater scenarios, and it is only a rough estimation strategy in the process of color restoration and haze removal for underwater images.
The work of haze-line is first proposed by Berman et al. [21], they assume that colors of a haze-free image are well approximated by many distinct colors, that form tight cluster lines in the RGB space. As a color-cluster-based method, the haze-line model can directly be employed in underwater images and effectively avoid the problem of inaccurate pixel estimation due to bright areas in the images using the DCP-based methods. However, there are usually severe noise and insufficient color varieties in underwater images. Thus, the haze-line technique needs further improvement when enhancing underwater images. Inspired by [21], Liu et al. [27] propose a color space dimensionality reduction method to enhance underwater images. It shows potential results for restoring slight degraded underwater images, but the color shift occurs for heavily degraded underwater images. Later in [6], they modify the method using a robust search method to estimate the background light. In this way, they improve the transmission estimation accuracy, and effectively restore the color distortion and remove the haze. Berman et al. [24] make improvements and refinements to the haze-line technique, and they compensate the transmission for different color attenuation and optimize the transmission results using the martingale distance. This generates clear underwater images but extremely time-consuming, and requires an expensive hardware facility. Haze-line-based methods have an advantage over the DCP-based methods in handling image brightness variation. However, the recovered images based on haze-line model still exhibit a color shift when processing severe color distortion images. This is because the haze-line assumption is limited for the variety attenuation caused by seawater. It creates a certain color shift at the pixel level when performing an underwater enhancement task.
2.2 Model-free restoration methods
In addition to DCP-based and haze-line-based methods, some researchers attempt to restore the underwater images from a model-free perspectives, most of which rely on adjusting the values of the pixels [7,9,10,13,28–31,33–35] or training deep learning-based networks [32,36–40].
Fu et al. [7] combine a color correcting strategy and a contrast improvement method to form a two-step approach for enhancing underwater images, but it produces an unsmooth result. Peng et al. [9] propose a depth estimation with the information of underwater image blurriness and light absorption, but it shows a limited ability in severe degradation situations especially for greenish underwater images. Both Ancuti et al. [10] and Zhao et al. [13] propose an image fusion strategy that aim to eliminate image noise and improve sharpness through different branches. The degraded underwater images can be better restored in the fusion way, but the model parameters need to be set manually, which is not adaptive for restoring greenish or bluish underwater images.
Due to the fast development of hardware performance and convolutional neural networks (CNNs), deep learning-based framework provides a robust solution for underwater image enhancement missions. Li et al. [39] propose an all-in-one dehazing network (ADN) based on a re-formulated atmospheric scattering model. To learn the mapping between hazy images and their corresponding transmission maps, Ren et al. [32] propose a multi-scale deep neural network for single-image dehazing. Both [32] and [39] rely on the synthetic underwater images when training their deep learning networks. Li et al. [38] propose a deep learning model named WaterGAN, which is an unsupervised color correction model based on generative adversarial network (GAN). WaterGAN generates realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. However, the training data generated in [32,38,39] using image rendering techniques or a GAN-based method are quite different from the real-world underwater images. This is because these algorithms neglect the attenuation effect when light is traveling in the water. Therefore, the training data limit the generalization performance of their networks and these methods are not stable for restoring realistic and complex underwater degradation images. To address the problem of synthetic underwater images for deep learning network training, Li et al. [40] construct a real underwater image dataset and propose a basic underwater image enhancement network. They obtain satisfactory results when using the real-world underwater image dataset. Islam et al. [37] propose a fast underwater image enhancement framework based on a conditional GAN, and they improve the real-time visual perception. But there are color shift and artifacts in the results when using their method. Later, Chen et al. [36] propose an underwater image enhancement algorithm based on deep learning and image formation model. They train their network using the dataset in [40], which shows good performance in color restoration, but it does not perform well in removing haze and improving sharpness.
Both the model-based and the model-free restoration methods contribute to better approximation of the underwater image enhancement problem, especially for color reconstruction. They have reported inspiring results. However, the model-based restoration methods are based on rough pixel estimation strategies (or inaccurate attenuation estimations) due to strong assumptions. The fusion-based methods in the model-free restoration methods usually require manual parameter adjustment for underwater images with different color tones. Deep learning-based methods only use image rendering techniques to generate synthetic underwater images for paired training, which frequently show limited generalization performance. They fail to improve the sharpness and to remove the haze with real-world underwater image enhancement benchmarks. Moreover, existing underwater image enhancement methods do not perform well in handling both color and haze in a severe situation, which limit their applicability in the images captured in the real-world ocean scenarios. To the best of our knowledge, there has not been any method that can balance color correction and haze removal in severely degraded underwater images. We attempt to address these issues in this paper.
3. Proposed underwater image enhancement method
The pipeline of our proposed underwater single image enhancement method is illustrated in Fig. 2. The degraded underwater image IT suffers from haze and color distortion. We aim to recover a sharp image $\hat{J}$, which looks like the image as clear as the one captured in the air. The proposed enhancement algorithm consists of four major steps:
- (1) Background light estimation: the severely degraded image is divided into four sub-regions and the corresponding sub-region scores are calculated and compared. The process is carried out in an iterative manner till the sub-region with the best score is less than a predefined threshold, and then the final sub-region patch is reserved;
- (2) Color recognition: by referring to the RGB color cross-reference table, our color recognition model is able to determine whether the image patch is blue or green tone;
- (3) Adaptive color compensation: once the image background hue is determined, adaptive color compensation is applied to the degraded underwater image to remove the blue-green tones;
- (4) Haze removal: a haze-line algorithm is employed to dehaze and enhance the output image of step 3.
3.1 Principles
According to the Jaffe-Mcglamery model [41–43], the total irradiance incident on a pixel x of the imaging plane (as dipicted in Fig. 1) includes three-term components, which can be expressed as
where IT(x) is the total irradiance incident at pixel x, ID(x) is the direct component that contains the main content of the image, IFS(x) is the forward scattering component, and IBS(x) is the backscattering component. Both IFS(x) and IBS(x) lead to the degraded image. Compared to IBS(x), IFS(x) plays a minor role in the image degradation process, which can be addressed using a simple filter. Thus, IFS(x) is usually ignored in the underwater imaging model. ID(x) is the light reflected directly on the imaging plane at the image coordinate x, which is related to the water attenuation coefficient β and the distance d(x) that the light travels. ID(x) is defined as where J(x) is the irradiance on the object, t(x) is the transmission in the water medium that equals to the exponential term. As for IBS(x), it can be expressed as where A(x) is the backscattering light. Thus, IT(x) can be formulated asThe simplified underwater imaging model shares the same model with the light propagation in the atmosphere [44]. However, the color degradation is not discussed in Eq. (4), and the underwater image can be regarded as a superimposition of a degraded color image and a hazed image. On account of this, we first apply color compensation to the underwater image so that the compensated image is similar to the haze images taken in the air. Then the haze will be further removed using the haze-line technique. Combing color compensation and haze-line, severe color degradation and haze can be removed robustly.
3.2 Background light estimation
In the color distorted underwater image, the background light shares the same color as the water body to a certain extent, and pixels in an image patch of the underwater image can accurately characterize the environmental color of the water body. These pixels have the following three characteristics: (1) The region of the image patch where the pixels are located is smooth; (2) These pixels are not the representative of any foreground objects in the image; (3) For the analysis in mathematics, the image patch should have a small variance and a color mean close to the color difference of the water body. To find an appropriate image patch to represent the image tone, we define a sub-region score $S({A_i})$, which includes two components Sd and Sσ. Sd denotes the color difference between the sub-region and the water body, and Sσ represents the sub-region smoothness. Thus, the sub-region score can be expressed as
where Ai is the sub-region and i${\in} ${1, 2, 3, 4}.Jerlov water-type is a specific classification scheme for standard oceanic water body color evaluation [45]. As shown in Fig. 3, it displays different water-type colors at different distances considering both coastal water and oceanic water. We select the 5th meter colors as the standard colors of the water body as the corresponding colors are not particularly bright or dark. Following the color difference calculation method in [46], we define the color difference as
where E is the color difference parameter from [46], w is the color of water body, W is the standard color of Jerlov water-type, ${E_{{A_i} - w}}$ is the color difference between the water body and the sub-region. As for the sub-region smoothness component, we introduce the mean value of the pixel intensity and the standard deviation of the RGB channels. This is because the mean value of the pixel intensity reflects the brightness of the image, the higher the mean value the brighter the image. The standard deviation of RGB channels indicates the dispersion degree of the image pixel values from the mean value, the larger the standard deviation the better the image quality. In this way, the sub-region smoothness component is able to find a bright region and gets rid of dark region estimation in an image. The sub-region smoothness component is defined asFig. 3. Selective absorption effect of water body on color spectra and RGB simulation of the appearance of a perfect white surface viewed in 1-20 m depth field in different water-types.
To find the background sub-region that best represents the water body color, we inherited the previous wisedom in [47] which uses the quad-tree subdivision iteration algorithm. The algorithm core is to divide an image equally into four first-level patches, which is in the order of upper left A1, upper right A2, lower left A3 and lower right A4. Then, the attributes of the patch are compared block by block. A pre-defined condition is set in this algorithm as the termination condition. If the attributes meet the termination condition, the patch will not be further divided, or the patch will be further divided into another four sub-regions in an iterative manner till the sub-region attributes meet the pre-defined condition. In the proposed method, the sub-region score and a preset threshold are the sub-region attribute and the termination condition, respectively. Using the quad-tree subdivision iteration algorithm with the sub-region score, the final sub-region score can be written as
where $\gamma $ is the number of iterations, $\eta $ is a weighted parameter that controls the gradient of Sσ and it is set as 2. Therefore, the background light estimation model uses the priors to perform the best color representative task in an iterative manner, which turns into an optimization problem to find the maximum score of the sub-region:3.3 Color recognition
According to the background light estimation, an image patch can be obtained, which has a color similar to the water body. As our goal is to improve the image quality in the marine environment, a color recognition strategy is employed to determine whether the image is in blueish or greenish tones. This process is to guide the adaptive color compensation accurately. We have made a full reference to the RGB colour cross-reference table [48] and our color recognition strategy consists of four steps: (1) Retrieve the R, G, and B image maps of the image patch separately; (2) Go through the R, G, and B image maps to collect the pixel values of each color; (3) Set up thresholds for each pixel value (R, G, and B); (4) Determine the value of each pixel (R, G, and B) referring to the thresholds and output the final color of each image patch. In detail, we first extract image patch, which reflects the color of the water by means of a background light estimation. The width and height of the image patch are determined from the image patch pixel point coordinates, and it is achieved by performing a difference operation on the four corner coordinates of the image patch. In this paper, we combine the traversed pixels and the threshold together to recognize the color. When the pixel value of the green channel is higher than 200, the pixel value of the blue channel is lower than100 and the pixel value of the red channel is lower than 100, then the image is recognized as a green image; When the pixel value of the blue channel is higher than 200, the pixel value of the green channel is lower than 200 and the pixel value of the red channel is less than 150, then the image is recognized as a blue image.
3.4 Adaptive color compensation
An adaptive color compensation technique is applied to removing the color distortion, which is built on several principles. (1) For observing severe degraded underwater images especially captured in the sea, there is not only water body-based color distortion, but also heavy haze; (2) In greenish or blueish images, the green or blue channel is relatively well preserved, while the color information in the red channel is lost due to the long wavelength; (3) The distribution of blue and green wavelengths has a continuum (blue light 450-480nm, blue-green light 480-500nm, green light 500-560nm). Hence, the color information in the blue channel is well preserved compared to the color information in the red channel in greenish images; (4) Based on (3), the color information in the green channel is well preserved compared to the color information in the red channel in blueish images. Therefore, we aim to compensate the red channel while also compensating the other color channel, depending on the particular hue of the image. Different from [13], we carry out a further step to classify the color distortion images into two categories based on the color recognition strategy. In [13], a separate red channel compensation method is used for underwater image color compensation, which ignores the overall hue of the image rendered in different water bodies. To overcome the above drawback, we propose two schemes, namely green and blue model compensation, to compensate the losses of different colors. As shown in Fig. 3, the red light is lost first regardless of the fact that the light travels in blueish or greenish water body. The blue and the green channels are relatively well-preserved in blueish and greenish water, respectively. Different from violently adding a fraction of the green channel to the red channel, we take full consideration of the priors presented in 3.2 and 3.3, and compensate the color channel in a more reasonable way. For an image with the greenish tone, we use the green mode compensation, and the compensated red channel IRc can be defined as
where IR and IG are the pixel values in the red and green channels of the raw image, and ${\bar{I}_R}$ and ${\bar{I}_G}$ are the mean pixel value of the red and green channels of the raw image, respectively. α is an empirical constant (we set it to 1 in our experiments). Meanwhile, we also compensate the blue channel IBc using a fraction of the green channel to the blue channel, which is defined as where ${I_B}$ and ${\bar{I}_B}$ are the pixel values in the blue channel and the mean pixel value of the blue channel, respectively. As for the blue mode compensation, it shares the same strategy as the green model compensation. In an image with blueish tone, the red and the green channels can be compensated by3.5 Haze removal
After having performed the adaptive color compensation, the color distortion in underwater images has been removed, and we can reach a color balance and transfer the input underwater image into an image observed in the air. However, the image still suffers from haze effects and lacks natural elements that the adaptive color compensation cannot handle. Thus, we apply the non-local dehazing technique to rendering a realistic underwater image.
According to the theory of haze-line [16], the atmospheric image IA(x) is defined as
where $I_T^{out}$ is the output image of the adaptive color compensation algorithm. Following Eq. (4), IA(x) can also be expressed as ${I_A}(x) = (J(x) - A) \cdot t(x)$, and it can be transferred into spherical coordinates where $\theta (x)$ and $\varphi (x)$ are the longitude and latitude, respectively. $r(x)$ is the distance between the pixel and the background light origin, which is expressed asFrom Eq. (16), we can confirm ${r_{\max }}(x) = ||{J(x) - A} ||$ when $t(x) = 1$. Thus, the transmission can be expressed as $t(x) = \frac{{r(x)}}{{{r_{\max }}}}$. There may be haze-free pixels existing on a haze-line H, then $r(x)$ should be ${\hat{r}_{\max }}(x) = \mathop {\max }\limits_{x \in H} \{ r(x)\} $. The transmission can be rewritten as
As the radiance $J > 0$, from Eq. (4), we have the lower bound can be expressed as
Finally, we acquire the restored image $\hat{J}$ using the transmission $\hat{t}(x)$ and A via Eq. (19)
4. Summary of algorithm
5. Experimental results and analysis
The proposed method was implemented on a Windows 10 PC with an Intel i5-7500 CPU, running on Matlab R2019b [49]. The comparison methods in our experiments are Liu et al. [6] Peng et al. [9], Ancuti et al. [10], Fu et al. [7], He et al. [20], Drews et al. [11], Hou et al. [35], Chen et al. [36], and Islam et al. [37]. We evaluated these ten methods (including our proposed method) on two publicly available underwater datasets, namely the ChinaMM dataset [50] and SQUID [24]. The ChinaMM dataset consists of 2,747 images in total (2,071 images in the training set and 676 images in the validation set), which is publicly released for competition in the field of underwater image enhancement and underwater object detection. The images in the ChinaMM dataset are captured near the Bohai Sea and appear greenish or blue-greenish tones over the whole dataset. As for SQUID, the diver captured 114 images using a specific underwater binocular camera in the waters of the Red Sea and the Mediterranean, all the images in SQUID are blueish or blue-greenish. Both the ChinaMM dataset and SQUID are typical representative datasets in underwater image degradation. Additionally, the underwater images in two datasets suffer from serious haze. Moreover, we collect some degraded underwater images from UIEB [40] and Bali Sea to form a generalized dataset for carrying out generalizability experiments. All of these images have varying degrees of color distortion and random color tones, some of which are hazy and some of which suffer from low resolutions. We select 100 images from the validation set of the ChinaMM dataset, 114 images from SQUID, and 100 images from the generalized dataset to verify the proposed algorithm. Several non-reference evaluation metrics, including PIQE [51], BRISQUE [52], UIQM [53], and UCIQE [54], have been selected to compare the experimental results. Given the situation of the underwater image enhancement task without ground-truth images, we applied the experimental results to the object detection application. By comparing the accuracy of the object detection under different underwater image enhancement algorithms, we can indirectly evaluate the performance of the algorithms in an “all-reference” way.
5.1 Background light estimation
We select the image patch with the highest score using Eqs. (6), (7), (8) and (9) in the background light estimation process, and the operation repeated in an iterative manner using the quad-tree subdivision algorithm till the output image patch size is less than the set threshold. Then, we use the color recognition model to perform traversal operations using the three channels’ pixels (R, G, and B) of the image. According to the RGB color cross-reference table, we perform a binary classification task on the image patch. This is a key step before carrying out the color compensation strategy for the degraded images. The results of typical background light estimation are shown in Fig. 4. As shown in Fig. 4, our method is able to avoid searching for objects with a small depth of field in complex situations where the image is rich in content. In addition, it can accurately find the sub-region that represents the color of the water body for different degrees of color degradation. The proposed background light estimation method shows excellent adaptability.
Fig. 4. Background light estimation. (a) The raw underwater images; (b) Background light search with the quad-tree subdivision algorithm; (c) The results of image patch and color recognition.
5.2 Results of the ablation study
The proposed framework mainly contains color compensation and haze-line. The color compensation restores the color distortion with high adaptability, then the haze-line technique removes the haze and enhances the whole image. To verify the effectiveness of the color compensation component and the haze-line component, we carry out the ablation study. The qualitative and quantitative results are demonstrated in Fig. 5 and Table 1, respectively. From the results shown in Fig. 5, it can be confirmed that the color compensation is able to remove the blueish and greenish distortions, and restore the colors of the underwater objects to a certain extent. Meanwhile, the brightness of the images is significantly improved due to the white balance component in the color compensation process. However, the resultant images still suffer from haze since the color compensation algorithm does not process the haze mask. The haze-line technique shows a good dehazing effect, but it makes very limited contributions to color restoration. The proposed framework not only has impressive dehazing capabilities, but also shows potential in color correction. It ranks the first in six of seven metrics on SQUID and the Generalized dataset, and ranks the first in five of seven metrics on the ChinaMM dataset. According to Table 1, there is a small gap between the results of the full framework and the results of color compensation for UICM, which is consistent with the qualitative analysis shown in Fig. 5.
Fig. 5. Qualitative experimental results of the ablation study. (a) The raw underwater images; (b) The results of the color compensation; (c) The results of the haze-line; (d) The results of our proposed method.
5.3 Qualitative analysis
The qualitative comparisons are displayed in Figs. 6 and 7. In this subsection, we evaluate the image quality from a visual perspective especially on color and sharpness. It can be seen that the methods of Liu et al. [6], Peng et al. [9], Ancuti et al. [10], He et al. [20], Drews et al. [11], Hou et al. [35], Chen et al. [36], Islam et al. [37], and our proposed method are all effective in haze removal while the method of Fu et al. [7] shows a very limited ability in dehazing and it even aggravates the haze in the images. There is a severe color shift in the results of Liu’s method, this is because of the inaccurate pixel projection and the transmission estimation. All the methods are potential in restoring generalized data (see the results on the 5th and 6th volumes in Fig. 6) except the results using Liu’s and Fu’s methods. The results of Hou’s method are similar to the results of Ancuti’s method, however, Hou’s method is slightly inferior compared to Ancuti’s method in terms of recovering the color and the contrast. For deep learning-based networks trained on the real-world underwater dataset, both Chen’s method and Islam’s method are able to improve the image quality. The results of Chen’s method are superior to the results of Islam’s method and other traditional comparison methods. But there is still a slight haze in the images processed by Chen’s method and the results are deficient in terms of color saturation. As for the results of Islam’s method, it is potential for restoring blueish underwater images, but has a significant disadvantage with greenish images, and it does not perform well in terms of the sharpness. This is because Islam’s method focuses on real-time application of underwater image enhancement, thus, the network generalization capability is limited. According to the experimental results on the ChinaMM dataset and SQUID, both Drews’ and Peng’s methods improve little on color distortion. He’s method emphasizes on haze removal, therefore, it contributes very little to the color restoration. As for the results of Ancuti’s method and Chen’s method, the visual perspective of these two methods is the closest to the results of our proposed method. However, Ancuti’s method is not stable and also limited in removing the haze (e.g. see the sea cucumbers, the sea urchins, and the scallops in the first two rows of Fig. 7). The qualitative results exhibited by the other methods are consistent with the analysis in Fig. 6. From Figs. 6 and 7, we can confirm that our proposed method achieves the best visual performance in restoring the color and removing the haze. It is superior to the other methods in terms of qualitative analysis.
Fig. 6. Qualitative results on the Generalized dataset. (a) The raw underwater images; (b) The results of Liu et al. [6]; (c) The results of Peng et al. [9]; (d) The results of Ancuti et al. [10]; (e) The results of Fu et al. [7]; (f) The results of He et al. [20]; (g) The results of Drews et al. [11]; (h) The results of Hou et al. [35]; (i) The results of Chen et al. [36]; (j) The results of Islam et al. [37]; (k) The results of our proposed method.
Fig. 7. Qualitative results on the ChinaMM dataset and SQUID. (a) The raw underwater images;(b) The results of Liu et al. [6]; (c) The results of Peng et al. [9]; (d) The results of Ancuti et al. [10]; (e) The results of Fu et al. [7]; (f) The results of He et al. [20]; (g) The results of Drews et al. [11]; (h) The results of Hou et al. [35]; (i) The results of Chen et al. [36]; (j) The results of Islam et al. [37]; (k) The results of our proposed method.
5.4 Quantitative analysis
Quantitative results of using different comparison methods are reported in Table 2, where we aim to evaluate the performance of these methods in a quantitative way on the datasets with different water body tones. It can be seen that the method proposed by Ancuti et al. [10] is comparable to our proposed method on the ChinaMM dataset (greenish images) and the generalized dataset, but the results of our proposed method has better perceptual quality. Moreover, the method proposed by Ancuti et al. [10] does not perform well on SQUID (blueish images) while our proposed method performs on SQUID as it did on the ChinaMM and the generalized datasets. The method proposed by Drews et al. [11] has a potential in restoring the blueish underwater images on SQUID, improving the sharpness and removing the haze. Nevertheless, it contributes little to the color reconstruction. Meanwhile, we can see from the quantitative results on the ChinaMM and the generalized datasets that our proposed method is more stable and robust than the method of Drews et al. [11]. In summary, our proposed method has shown potential performance on three datasets. It is able to restore heavily degraded underwater images in different water bodies and obtain excellent visual perception and quantitative scores. For slightly degraded underwater images, our method shows even better performance. Other comparison methods are only effective for some special evaluation metrics, and our method shows superior performance across different evaluation metrics.
In addition to the non-reference quantitative evaluations, we also report an “all-reference” quantitative evaluation metric. Previous works state that underwater image enhancement techniques are significant for improving the accuracy of underwater object detection. Therefore, we decide to use the underwater object detection model to investigate how much these underwater image enhancement algorithms bring improvement to the detection accuracy. We processed 2,747 images from the ChinaMM dataset using ten image processing methods mentioned in this paper. Then the single shot multibox detector (SSD) [55] is chosen as the standard detection network for training and testing. The SSD experiment is conducted on a server with Intel Xeon CPU @ 2.40GHz and 2 parallel Nvidia Tesla P100 GPUs, running on a Keras platform [56]. The mean Average Precision (mAP) is shown in Table 3 and the typical object detection results of the single shot multibox detector after using different image processing methods are shown in Fig. 8. Our proposed method achieves the best score in the underwater detection task, which further demonstrates the effectiveness in the practical application. Furthermore, we investigate the relationship between the mAP and the image quality evaluation metrics (as shown in Fig. 9). In Fig. 9, the distribution of BRISQUE and UCIQE scores are similar to the trend of the detection accuracy except the scores of Chen’s method and Islam’s method. However, we find discrepancies between the evaluation scores and the recognition accuracy for PIQE and UIQM. For example, the scores of Fu’s, He’s, and Drews’ methods on PIQE decrease sequentially, but the results of their recognition accuracy increase sequentially. A similar situation exists in the relationship between mAP and UIQM. Thus, we conclude that there is no absolute relationship between the underwater image evaluation metrics and the underwater object detection accuracy. The result of our proposed method has the potential on both the professional underwater evaluation metrics and the underwater object recognition task.
Fig. 8. The typical object detection results of the single shot multibox detector after using different image processing methods. (a) Groundtruth; (b) Input; (c) Liu et al. [6]; (d) Peng et al. [9]; (e) Ancuti et al. [10]; (f) Fu et al. [7]; (g) He et al. [20]; (h) Drews et al. [11]; (i) Hou et al. [35]; (j) Chen et al. [36]; (k) Islam et al. [37]; (l) Ours.
Fig. 9. Image quality evaluation metrics and mAP on the ChinaMM dataset. The polyline represents different image quality evaluation metrics and the histogram represents the mAP. Numbers 1 to 10 refer to ten image processing methods ordered according to increasing mAP values, they are Fu et al. [7], He et al. [20], Drews et al. [11], Islam et al. [37], Liu et al. [6], Ancuti et al. [10], Hou et al. [35], Chen et al. [36], Peng et al. [9], and Ours, respectively.
6. Conclusion
In this paper, we proposed a robust underwater image enhancement framework based on color restoration and dehazing, which can automatically restore the color distortion and remove the haze in the images. Firstly, the background light was estimated using the quad-tree subdivision iteration algorithm. Then, the color recognition model examined whether the image patch is greenish or blueish. A specific adaptive color compensation model was proposed, compensating the distorted underwater image based on the output of the color recognition model. During the process of color compensation, severe color distortion can be restored by our proposed green mode compensation or blur mode compensation. Finally, the image quality can be stably and progressively improved with the haze-line technique. We have implemented the proposed method on the public images from the ChinaMM dataset, SQUID, and UIEB. Qualitative and quantitative results demonstrated that our proposed method performed better than the state-of-the-art methods in improving the image quality. Moreover, we applied the images processed by different underwater image enhancement algorithms to training the deep detection network in the underwater object recognition task, and the results of our proposed method are also superior to the other methods.
Funding
National Natural Science Foundation of China (62001443); Natural Science Foundation of Shandong Province (ZR2020QE294); China Scholarship Council (202106330040).
Acknowledgments
The authors would like to thank National Natural Science Foundation of China and Dalian Municipal People’s Government for providing the public underwater object detection dataset (the ChinaMM dataset) for research purpose. They also express their sincere gratitude to the teams from Tel Aviv University of Israel and Tianjin University of China for providing the public datasets SQUID and UIEB for research purpose.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. C. O. Ancuti, C. Ancuti, C. De Vleeschouwer, and P. Bekaert, “Color balance and fusion for underwater image enhancement,” IEEE Trans. on Image Process. 27(1), 379–393 (2018). [CrossRef]
2. C. D. Mobley, “Light and water: Radiative transfer in natural waters,” Cambridge, MA, USA: Academic Press, 1994.
3. J. Zhou, D. Zhang, and W. Zhang, “Classical and state-of-the-art approaches for underwater image defogging: a comprehensive survey,” Front. Inform. Technol. Electron. Eng. 21(12), 1745–1769 (2020). [CrossRef]
4. D. J. Lee, S. Redd, R. Schoenberger, X. Xu, and P. Zhan, “An automated fish species classification and migration monitoring system,” in IECON'03. 29th Annual Conference of the IEEE Industrial Electronics Society (IEEE, 2003), pp. 1080–1085.
5. H. Lu, T. Uemura, D. Wang, J. Zhu, Z. Huang, and H. Kim, “Deep-sea organisms tracking using dehazing and deep learning,” Mobile Netw. Appl. 25(3), 1008–1015 (2020). [CrossRef]
6. Y. Liu, S. Rong, X. Cao, T. Li, and B. He, “Underwater single image dehazing using the color space dimensionality reduction prior,” IEEE Access. 8, 91116–91128 (2020). [CrossRef]
7. X. Fu, Z. Fan, M. Ling, Y. Huang, and X. Ding, “Two-step approach for single underwater image enhancement,” in 2017 International Symposium on Intelligent Signal Processing and Communication Systems (IEEE, 2017), pp. 789–794.
8. J. Zhou, T. Yang, W. Ren, D. Zhang, and W. Zhang, “Underwater image restoration via depth map and illumination estimation based on single image,” Opt. Express. 29(19), 29864–29886 (2021). [CrossRef]
9. Y. T. Peng and P. C. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE Trans. on Image Process. 26(4), 1579–1594 (2017). [CrossRef]
10. C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 81–88.
11. P. Drews, E. Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” in Proceedings of the IEEE International Conference on Computer Vision Workshops (IEEE, 2013), pp. 825–830.
12. J. Zhou, Y. Wang, W. Zhang, and C. Li, “Underwater image restoration via feature priors to estimate background light and optimized transmission map,” Opt. Express. 29(18), 28228–28245 (2021). [CrossRef]
13. W. Zhao, S. Rong, T. Li, X. Cao, Y. Liu, and B. He, “Underwater single image enhancement based on latent low-rank decomposition and image fusion,” in Twelfth International Conference on Graphics and Image Processing (International Society for Optics and Photonics, 2021) pp. 1–10.
14. J. Zhou, Z. Liu, W. Zhang, D. Zhang, and W. Zhang, “Underwater image restoration based on secondary guided transmission map,” Multimed. Tools Appl. 80(5), 7771–7788 (2021). [CrossRef]
15. S. Lin, K. Chi, T. Wei, and Z. Tao, “Underwater image sharpening based on structure restoration and texture enhancement,” Appl. Opt. 60(15), 4443–4454 (2021). [CrossRef]
16. K. Liu and Y. Liang, “Enhancement of underwater optical images based on background light estimation and improved adaptive transmission fusion,” Opt. Express. 29(18), 28307–28328 (2021). [CrossRef]
17. K. Liu and Y. Liang, “Underwater image enhancement method based on adaptive attenuation-curve prior,” Opt. Express. 29(7), 10321–10345 (2021). [CrossRef]
18. Y. Tao, L. Dong, L. Xu, and W. Xu, “Effective solution for underwater image enhancement,” Opt. Express. 29(20), 32412–32438 (2021). [CrossRef]
19. X. Deng, Y. Zhang, H. Wang, and H. Hu, “Robust underwater image enhancement method based on natural light and reflectivity,” JOSA A. 38(2), 181–191 (2021). [CrossRef]
20. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010). [CrossRef]
21. D. Berman and S. Avidan, “Non-local image dehazing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016), pp. 1674–1682.
22. F. M. Codevilla, S. S. D. C. Botelho, P. Drews, N. Duarte Filho, and J. F. D. O. Gaya, “Underwater single image restoration using dark channel prior,” in 2014 Symposium on Automation and Computation for Naval, Offshore and Subsea (IEEE, 2014), pp. 18–21.
23. A. Galdran, D. Pardo, A. Picón, and A. Alvarez-Gila, “Automatic red-channel underwater image restoration,” J. Vis. Commun. Image Represent. 26, 132–145 (2015). [CrossRef]
24. D. Berman, D. Levy, S. Avidan, and T. Treibitz, “Underwater single image color restoration using haze-lines and a new quantitative dataset,” IEEE Trans. Pattern Anal. Mach. Intel. 43(8), 1 (2020). [CrossRef]
25. Y. T. Peng, K. Cao, and P. C. Cosman, “Generalization of the dark channel prior for single image restoration,” IEEE Trans. on Image Process. 27(6), 2856–2868 (2018). [CrossRef]
26. C. Li, J. Quo, Y. Pang, S. Chen, and J. Wang, “Single underwater image restoration by blue-green channels dehazing and red channel correction,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE, 2016), pp. 1731–1735.
27. Y. Liu, S. Rong, X. Cao, T. Li, and B. He, “Underwater image dehazing using the color space dimensionality reduction prior,” in Proceedings of the IEEE International Conference on Image Processing (IEEE, 2020), pp. 1013–1017.
28. C. Li, J. Guo, R. Cong, Y. Pang, and B. Wang, “Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior,” IEEE Trans. on Image Process. 25(12), 5664–5677 (2016). [CrossRef]
29. C. Li, J. Guo, C. Guo, R. Cong, and J. Gong, “A hybrid method for underwater image correction,” Pattern Rec. Lett. 94, 62–67 (2017). [CrossRef]
30. X. Fu, P. Zhuang, Y. Huang, Y. Liao, X. P. Zhang, and X. Ding, “A retinex-based enhancing approach for single underwater image,” in Proceedings of the IEEE International Conference on Image Processing, (IEEE, 2014), pp. 4572–4576.
31. P. L. Drews, E. R. Nascimento, S. S. Botelho, and M. F. M. Campos, “Underwater depth estimation and image restoration based on single images,” IEEE Comput. Graph. Appl. 36(2), 24–35 (2016). [CrossRef]
32. W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M. H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in Proceedings of the IEEE Conference on European Conference on Computer Vision, (IEEE, 2016), pp. 154–169.
33. A. S. A. Ghani and N. A. M. Isa, “Underwater image quality enhancement through integrated color model with Rayleigh distribution,” Appl. Soft Comput. 27, 219–230 (2015). [CrossRef]
34. A. Ghani and N. Isa, “Enhancement of low quality underwater image through integrated global and local contrast correction,” Appl. Soft Comput. 37, 332–344 (2015). [CrossRef]
35. G. Hou, Z. Pan, B. Huang, G. Wang, and X. Luan, “Hue preserving-based approach for underwater color image enhancement,” IET Image Process. 12(2), 292–298 (2018). [CrossRef]
36. X. Chen, P. Zhang, L. Quan, C. Yi, and C. Lu, “Underwater image enhancement based on deep learning and image formation model,” arXiv preprint arXiv:2101.00991 (IEEE, 2021), pp. 1–7.
37. M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement for improved visual perception,” IEEE Robot. and Autom. Lett. 5(2), 3227–3234 (2020). [CrossRef]
38. J. Li, K. A. Skinner, R. M. Eustice, and M. Johnson-Roberson, “WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images,” IEEE Robot. Autom. Lett. 3(1), 1 (2017). [CrossRef]
39. B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “AOD-Net: All-in-one dehazing network,” in Proceedings of the IEEE Conference on International Conference on Computer Vision (IEEE, 2017), pp. 4780–4788.
40. C. Li, C. Guo, W. Ren, R. Cong, J. Hou, S. Kwong, and D. Tao, “An underwater image enhancement benchmark dataset and beyond,” IEEE Trans. on Image Process. 29, 4376–4389 (2020). [CrossRef]
41. J. S. Jaffe, “Computer modeling and the design of optimal underwater imaging systems,” IEEE J. Oceanic Eng. 15(2), 101–111 (1990). [CrossRef]
42. J. S. Jaffe, “Monte Carlo modeling of underwater-image formation: validity of the linear and small-angle approximations,” Appl. Opt. 34(24), 5413–5421 (1995). [CrossRef]
43. B. L. McGlamery, “A computer model for underwater camera systems,” Ocean Optics VI. International Society for Optics and Photonics , 0208, 221–231 (1980). [CrossRef]
44. K. Harald, “Theorie der horizontalen sichtweite: Kontrast und sichtweite,” Keim and Nemnich, Munich 12 (1924).
45. N. G. Jerlov, “Marine optics,” Elsevier, 1976.
46. G. Sharma, W. Wu, and E. N. Dalal, “The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia,” Centre Français de la Couleur , 30(1): 21–30 (2005). [CrossRef]
47. J. H. Kim, W. D. Jang, J. Y. Sim, and C. S. Kim, “Optimized contrast enhancement for real-time image and video dehazing,” J. Vis. Commun. Image Represent. 24(3), 410–425 (2013). [CrossRef]
48. https://tool.oschina.net/commons?type=3
50. R. Liu, X. Fan, M. Zhu, M. Hou, and Z. Luo, “Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light,” IEEE Trans. Circ. Syst. Vid. Technol. 30(12), 4861–4875 (2020). [CrossRef]
51. N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in Proceedings of the IEEE Twenty First National Conference on Communications (IEEE, 2015), pp. 1–6.
52. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. on Image Process. 21(12), 4695–4708 (2012). [CrossRef]
53. K. Panetta, C. Gao, and S. S. Agaian, “Human-visual-systeminspired underwater image quality measures,” IEEE J. Oceanic Eng. 41(3), 541–551 (2016). [CrossRef]
54. M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Trans. on Image Process. 24(12), 6062–6071 (2015). [CrossRef]
55. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Proceedings of the IEEE Conference on European Conference on Computer Vision (IEEE, 2016), pp. 21–37.
