Hierarchical deconvolution dehazing method based on transmission map segmentation

Xiaotian Shi; Feng Huang; Lin Ju; Zhigang Fan; Shuxuan Zhao; Shouqian Chen

doi:10.1364/OE.510100

1. Introduction

When light travels through a scattering medium (such as water, fog, sand, etc.), it is scattered and absorbed by the particles of the medium. Due to the scattering effect of medium particles, the transmission path of light is no longer a straight line, which leads to a change in the spatial intensity distribution of light after passing through the medium. Due to the absorption of medium particles, the intensity and energy of light will be seriously reduced. Therefore, the contrast of the image taken in the scattering medium is seriously decreased, the detail information is lost, and the noise amplitude is increased [1,2].

To solve this problem, many image dehazing methods have been proposed in recent years. It mainly includes the image enhancement method, the image dehazing method based on the model, and the method based on deep learning.

The image enhancement method does not consider the process of image degradation. This method mainly studies the brightness, color, and histogram distribution of images affected by scattering [3]. Therefore, such methods can directly improve the subjective visual effect of the image, but it is very easy to cause excessive enhancement of the image [4]. Moreover, it is difficult to achieve accurate image restoration because this method does not rely on any model of image degradation in the scattering medium.

Other methods that are not based on image degradation and scattering imaging models are deep learning methods. With the rapid development of computer hardware, machine learning, and deep learning have been widely used in the field of image dehazing. After good training, the neural network can obtain an excellent dehazing effect [5–9]. However, the deep learning method requires a large number of images with and without fog for training, which is constrained by training samples. In addition, such methods do not depend on the scattering imaging model, so they provide little contribution to the study of the scattering imaging mechanism.

At present, there are two kinds of methods to deblur according to the model. The first type is the atmospheric scattering model, and the second type is the image degradation model. The atmospheric scattering model considers that the interference of water or fog on imaging mainly includes two parts, one is atmospheric light, and the other is the transmittance of the medium. Atmospheric light interferes with the light, causing the image to shine and whiten as a whole. In the process of image restoration, atmospheric light needs to be removed. However, the transmittance of the medium makes the target light attenuate seriously and the contrast of the target decreases seriously. In the process of image restoration, it is necessary to estimate the transmittance of different areas to restore the contrast and color of the image. This kind of model includes many algorithms, such as the polarization defogging method, color prior method, DCP method, and a series of improved algorithms based on the DCP method [10,11].

By taking advantage of the different polarization characteristics of atmospheric light and target light, the polarization defogging method can distinguish atmospheric light and target light. In this kind of method, atmospheric light and target light are distinguished by their difference in polarization degree and polarization angle [12–17]. However, this kind of method depends on the performance of polarization detection equipment, so its application scope is limited.

The DCP method uses some pixels in the image to estimate the atmospheric light value and then calculates the transmittance of media in different regions of the image by using the dark channel prior law [18]. The dark channel prior law refers to the fact that all pixel values of the dark channel image without fog are close to 0. Where the dark channel image is defined as the minimum value of each pixel value in RGB three channels. For example, for a red object, the pixel values of its green channel and blue channel are almost 0, and so is the pixel value of the dark channel image of the object in the image. Combined with pooling and other methods, all pixels of the dark channel image without fog are close to 0. For the image with fog, the dark channel image is no longer 0 due to the influence of atmospheric light, but close to the value of atmospheric light. According to this rule, the method can estimate the atmospheric light parameters and the transmittance of each pixel accurately. However, this method is difficult to apply to images with sky and white objects. Because such objects no longer conform to the dark channel prior law. Therefore, based on this method, various improved algorithms are proposed. For example, the gamma transform is used to correct the transmittance calculated by the dark channel method to estimate atmospheric light and transmittance more accurately [19]. Similar to the DCP method, the color prior method is also applied in the field of image de-fogging. This method estimates atmospheric light and transmittance, to achieve target image restoration [20].

The image degradation model regards the scattering medium of water mist as a kind of optical system and thinks that the medium will haze the image and introduce noise. These methods estimate the point spread function (PSF) and noise of the medium to achieve image dehazing, such as the Wiener filter method. In this model, the medium is regarded as an optical system with a fixed PSF and noise and does not transform with space. Based on the Wiener filter method, the propagating deconvolution (PD) method is proposed [21]. This method divides the thick medium into multiple thin media. By modeling the imaging process of each layer of thin media, the point diffusion function and noise of thick media are derived. This method is more complex than the Wiener filter, closer to the actual imaging process, and has a better deblurring effect. However, this method assumes that all scenes in the image have the same thickness of scattering medium, that is, the number of inverse convolutional layers in the whole image is the same, which is difficult for images with large depth of field and complex scenes [22]. Therefore, this method is more suitable for imaging scenes with a small depth of field and has a poor dehazing effect for images with rich scene depth information.

In the above algorithm, the propagating deconvolution method can better restore the color of the image and avoid excessive enhancement of the sky region. However, this method uses the same PSF and noise to restore the whole image. Usually, in a real imaging environment, the depth and thickness of the scene vary with the location of the image. Therefore, the PSF of the medium and the introduced noise should also be different. Based on this phenomenon, this paper proposes a subregion-layered image dehazing strategy. Firstly, the media transmittance of each region in the image is estimated. According to the transmission distribution, the image is divided into several sub-images. The degradation function of each sub-image is estimated independently to achieve accurate restoration of each distance sub-image. Finally, each sub-image is spliced and fused to get the dehazing result.

2. Principle

2.1 Image degradation model in haze

The current image defogging is mainly based on the following two models. The first type is an image degradation model, which considers haze as an optical imaging system, and its basic principles are as follows.

When light is transmitted in the scattering medium, the forward scattering will cause the ideal point source to become a spot during imaging. Based on this phenomenon, the scattering medium can be equivalent to an optical system. Its contribution to image blurring is defined as $h({x,y} )$, the PSF of the medium. In addition, the backscattered light of the medium will affect the imaging, which is $n({x,y} )$ in Eq. (1), i.e. the noise. Here, the imaging optical system is regarded as an ideal optical system, that is, the PSF of the system is an ideal Dirac function, and the point light source will not be dispersed. Therefore, it is only the scattering medium that causes the blur and noise of the image. Since the scattering medium PSF and noise are usually unknown, perfect image recovery is extremely difficult. If it can accurately estimate degradation function and noise, this model can realize high-quality image restoration.

(1)$$I({x,y} )= g({x,y} )\ast h({x,y} )+ n({x,y} ),$$

where $g({x,y} )$ is the target light.

The second type is the atmospheric scattering model, which assumes that haze causes a decrease in image contrast, and the degree of decrease is determined by the transmittance of the haze, as shown in Eq. (2). In this model, the image is affected by atmospheric light and haze transmittance, resulting in a non-uniform decrease in contrast across different regions. In Eq. (2), $g({x,y} )$ is the target light, $t({x,y} )$ is the transmittance of haze, and A is atmospheric light.

(2)$$I({x,y} )= g({x,y} )t({x,y} )+ A[{1 - t({x,y} )} ]\textrm{.}$$

The methods based on this model usually use prior methods to estimate the transmittance and atmospheric light of each region in the image. By removing the impact of both on the image, the restoration of fog images is achieved and the image contrast is significantly improved.

The two models described in Eq. (1) and Eq. (2) are the theoretical basis for most image dehazing methods based on physical models at present [2,4]. However, the first model ignores the PSF and noise differences caused by the different transmittance of each region in the image, resulting in identical restoration levels for thick and thin fog regions, making it difficult to achieve accurate restoration of each region in the entire image. The second model only considers the degradation of image contrast, ignoring the blurring effect and noise impact of haze. Within the framework of this model, the restoration results of images heavily rely on the estimation of transmittance. When there is a deviation in the estimation of transmittance, the dehazing effect will significantly deteriorate.

In order to balance the advantages of both models, we propose a segmented and layered image degradation model. In the proposed image degradation model, the PSF and noise of haze are closely related to the transmittance of haze.

2.2 Hierarchical degradation model for images in scattering media

Firstly, we consider the scattering medium as a combination of multi-layer thin scattering media, as shown in Fig. 1.

Fig. 1. Hierarchical degradation model of target light in scattering media

Download Full Size | PDF

When imaging in water or haze, there is a scattering medium between the target and the sensor. According to the image degradation model, the scattering medium is considered an optical system. Furthermore, the layered model decomposes the scattering medium into multi-layer sub-systems. The total thickness of the medium is the distance z. Suppose there are k layers of media between the target and the detector, and the thickness of each layer is $\varDelta$, then $z = k\varDelta$. The optical properties of each layer are the same, that is, the PSF and noise are the same.

In the process of transmitting the target light to the detector, the target light will undergo k degradation processes and finally get the imaging result. Each degradation will result in blurred images and increased noise. Equation (3) describes the image degradation process under this model. Where ${h_k} = {h_{k - 1}} = \cdots = {h_1} = {h_\varDelta }$ refers to the PSF of each layer of media. ${n_d}$ is the noise produced by each layer of the medium.

(3)$$\begin{aligned} I &= {f^{(k )}}\\ \textrm{ }{f^{(k )}} &= {f^{({k - 1} )}} \ast {h_k} + {n_d}^{(k )}\\ {f^{({k - 1} )}} &= {f^{({k - 2} )}} \ast {h_{k - 1}} + {n_d}^{({k - 1} )}\\ &\vdots \\ \textrm{ }{f^{(1 )}} &= f \ast {h_1} + {n_d}^{(1 )}. \end{aligned}$$

The PSF expression of each sub-system is shown in Eq. (4) and ${\theta _\varDelta }$ represents the fuzzy coefficient of the medium.

(4)$${h_\varDelta }({x,y} )= \frac{{\exp \left( { - \frac{{{x^2} + {y^2}}}{{{\theta_\varDelta }}}} \right)}}{{\sqrt {\pi {\theta _\varDelta }} }}.$$

Fourier transforms Eq. (4) into the frequency domain, and the result is shown in Eq. (5).

(5)$${H_\varDelta }({u,v} )= \exp [{ - {\theta_\varDelta }{\pi^2}({{u^2} + {v^2}} )} ].$$

In the layered model, when the thickness of the medium layer approaches 0, the broadening effect also approaches 0. In this case, PSF approaches the ideal Dirac function, as shown in Eq. (6).

(6)$$\left\{ \begin{array}{l} \mathop {\lim }\limits_{\varDelta \to 0} {\theta_\varDelta } = 0\\ \mathop {\lim }\limits_{\varDelta \to 0} {h_\varDelta }({x,y} )= \delta ({x,y} )\end{array} \right..$$

In this method, the iterative Wiener filter is used as the algorithm for image restoration. As shown in Eq. (7). A Wiener filter is used to restore the degradation effect caused by each layer of medium. Where ${P_\textrm{n}}({u,v} )$ and ${P_\textrm{f}}({u,v} )$ are the power spectra of noise and target signal.

(7)$${F_{k - 1}}({u,v} )= \frac{{{H_\varDelta }({u,v} ){F_k}({u,v} )}}{{{{|{{H_\varDelta }({u,v} )} |}^2} + \frac{{{P_\textrm{n}}({u,v} )}}{{{P_\textrm{f}}({u,v} )}}}}.$$

In Eq. (7), ${F_k}({u,v} )$ is the Fourier transform of the hazed image ${f^{(k )}}$, and ${F_{k - 1}}({u,v} )$ is the Fourier transform of the image ${f^{({k - 1} )}}$ after removing the influence of a layer of the scattering medium. By iterating the formula, we can obtain the dehazing image f after removing the influence of a multilayer scattering medium.

In order to reduce the complexity of the model, the number of parameters in Eq. (5) and Eq. (7) is reduced. Let the ratio of the noise power spectrum to signal power spectrum be ${\eta _1} = {{{P_\textrm{n}}({u,v} )} / {{P_\textrm{f}}({u,v} )}}$, and let ${\eta _2} = {\theta _\varDelta }{\pi ^2}$. The parameters are reduced as shown in formula (8). The physical meaning of ${\eta _1}$ is the reciprocal of the image signal-to-noise ratio. Usually, the noise power of an image is much lower than the signal power, so the value of this parameter is set to around 0.01. ${\eta _2}$ represents the broadening effect of haze. Usually, the broadening effect of haze is low, so the value of this parameter is set to around 0.01.

(8)$$\left\{ \begin{array}{l} {F_{k - 1}}({u,v} )= \frac{{{H_\varDelta }({u,v} ){F_k}({u,v} )}}{{{{|{{H_\varDelta }({u,v} )} |}^2} + {\eta_1}}}\\ {H_\varDelta }({u,v} )= \exp [{ - k{\eta_2}({{u^2} + {v^2}} )} ]\end{array} \right..$$

By iterating k times through Eq. (8), the dehazing result of the target at $z = k\varDelta$ can be obtained. Based on this model, we can obtain the defogging results of the target image at a specific distance. However, in the actual scene, the thickness of fog in different areas of the fog image is often different. Therefore, based on the hierarchical model, a hierarchical scattering imaging model based on transmission map segmentation is proposed.

The proposed model mainly includes the following innovative points.

1. According to the transmittance $t({x,y} )$, the image is divided into multiple sub-regions;
2. Each sub-region corresponds to an independent number of degradation layers;
3. The number of degradation layers k in the image is closely related to the transmittance $t({x,y} )$ of haze.

2.3 Hierarchical scattering imaging model based on transmission map segmentation

In real imaging, targets are usually widely distributed over a variety of distances, as shown in Fig. 2. As a result, objects at different distances will degrade to different degrees in the imaging process. Long-range targets degraded the most, followed by targets in medium distances. Close-in targets have the least degradation. The degree of degradation is closely related to the thickness and transmittance of haze. The thickness of haze is the distance from the target to the detector.

Fig. 2. Hierarchical degradation process of imaging targets at different distances

Download Full Size | PDF

According to the atmospheric scattering model, the relationship between the distance of the target $z({x,y} )$ and the transmittance of the medium $t({x,y} )$ is shown in Eq. (9).

(9)$$t({x,y} )= \exp [{ - z\beta ({x,y} )} ].$$

Assuming that the absorption coefficient $\beta$ of the scattering medium is constant, the transmittance will be determined only by the distance. Therefore, we can use the transmittance for distance division.

Taking the hazed image shown in Fig. 3. (a) as an example, the haze image is divided into four sub-regions according to the transmittance, namely ${I_{t1}}({x,y} )$, ${I_{t2}}({x,y} )$, ${I_{t3}}({x,y} )$ and ${I_{t4}}({x,y} )$. The corresponding transmittance of each region is $t1$, $t2$, $t3$ and $t4$ respectively. The specific segmentation criteria can be found in Chapter 3.1.

Fig. 3. Hazed image segmented into multiple regions based on transmittance: (a) Haze image; (b) Transmittance map; (c) Sub-region 1 ${I_{t1}}({x,y} )$; (d) Sub-region 2 ${I_{t2}}({x,y} )$; (e) Sub-region 3 ${I_{t3}}({x,y} )$; (f) Sub-region 4 ${I_{t4}}({x,y} )$.

Download Full Size | PDF

Equation (10) describes the degradation function of the image under the proposed model.

(10)$$\begin{aligned} I({x,y} )&= \sum\limits_{i = 1}^4 {{I_{ti}}({x,y} )} \\ &= {I_{t1}}({x,y} )+ {I_{t2}}({x,y} )+ {I_{t3}}({x,y} )+ {I_{t4}}({x,y} )\\ &= {f^{({k1} )}} + {f^{({k2} )}} + {f^{({k3} )}} + {f^{({k4} )}},ki \propto ({1 - ti} ). \end{aligned}$$

The degradation function $h({x,y} )$ varies for regions with different transmittance. That is to say, $h({x,y} )$ is a function of $t({x,y} )$. The number of degraded layers in the transmittance $t1$∼$t4$ region is set to $k1$∼$k4$ respectively. Obviously, the lower the transmittance, the more dielectric layers. This represents k is positively correlated with $1 - t({x,y} )$.

Combine the two models given in Eq. (1) and Eq. (2), and perform independent dehazing on each sub-region. The dehazing process for each sub-region is shown in Eq. (11).

(11)$$\left\{ \begin{array}{l} {F^{({ki - 1} )}}({u,v} )= \frac{{{H_\varDelta }({u,v} ){F^{({ki} )}}({u,v} )}}{{{{|{{H_\varDelta }({u,v} )} |}^2} + {\eta_1}}}\\ {H_\varDelta }({u,v} )= \exp [{ - {\eta_2}({{u^2} + {v^2}} )} ]\\ {F^{({ki} )}}({u,v} )= {\cal F}\left[ {{f^{({ki} )}}({x,y} )- \frac{{A{\eta_3}}}{{ki}}} \right]\\ {f^{({ki - 1} )}}({x,y} )= \frac{{{{\cal F}^{ - 1}}[{{F^{({ki - 1} )}}({u,v} )} ]}}{{{{\{{{{{\textrm{mean}} [{ti({x,y} )} ]} / {{\textrm{mean}} [{t1({x,y} )} ]}}} \}}^\gamma }}} \end{array} \right.,$$

for the $i$th sub-region ${f^{({ki} )}}({x,y} )$, the Fourier transform is ${F^{({ki} )}}({u,v} )$. The number of inverse convolutions of the sub-region is $ki$, and its transmittance image is $ti({x,y} )$. The recovered sub-region is ${f^{({ki - 1} )}}({x,y} )$ and its Fourier transform is ${F^{({ki - 1} )}}({u,v} )$. A is airlight.

In the proposed strategy, different deconvolution layers are taken for sub-regions with different transmittance. And the influence of airlight A is removed during the iterative process. The transmittance function $t({x,y} )$ is applied to adjust the brightness of the image. In the last equation in Eq. (11), the denominator is the $\gamma$ power of the relative transmittance of each sub-region, which is used to adjust the brightness of each sub-image and reduce the sense of fragmentation between sub-images during fusion. Normally, the initial value of $\gamma$ can be set between 0 and 0.05. The purpose of setting this parameter is to introduce the transmittance term from the atmospheric scattering model into the image degradation model, which is used to improve the image contrast in low transmittance areas. The ${\eta _3}$ in the formula is used to adjust the value of atmospheric light and reduce excessive enhancement caused by inaccurate estimation of airlight. The initial value of ${\eta _3}$ can be set between 0.2 and 1.2.

According to Eq. (11), we can use the iterative Wiener filtering method to calculate the recovery results of each sub-region. Finally, all sub-regions are fused to obtain the final dehazing image.

It should be noted that in the proposed method, each region in the image corresponds to an independent number of degradation layers and is determined by the transmittance of the region. In contrast, the PD method uses the same number of degradation layers in all regions of the whole image, which is independent of the transmittance. Therefore, the PD method cannot balance the recovery effects of low transmittance and high transmittance regions. Moreover, the PD method ignores the influence of atmospheric light on fog images, while the proposed method eliminates the influence of atmospheric light in the process of deconvolution, so it can obtain a better dehazing effect.

2.4 Calculation method of transmission map

In the proposed strategy, we use the DCP method to estimate the transmittance of each target region in the image. DCP method is an image dehazing method based on the atmospheric scattering model. The atmospheric scattering model is shown in Eq. (2).

Let the dark channel image of image $f({x,y} )$ be ${f_{\textrm{dark}}}({x,y} )$, and the calculation method of the dark channel image is shown in Eq. (12).

(12)$${f_{\textrm{dark}}}({x,y} )= \mathop {\min }\limits_{C \in \{{R,G,B} \}} \left\{ {\mathop {\min }\limits_{({x,y} )\in \Omega } [{{f^C}({x,y} )} ]} \right\}.$$

where is the pooled area, which can be set to a $5 \times 5$ square, while RGB is the red, green, and blue channels of the color image respectively. For an image without fog, all pixels in the dark channel image are close to 0. In the presence of a scattering medium such as water or fog, the relationship between dark channel image and transmittance is as follows:

(13)$$t({x,y} )\propto 1 - {f_{\textrm{dark}}}({x,y} ).$$

The above theory is based on the dark channel prior law [18]. This law comes from the researchers’ summary of the dark channel image features of a large number of fog images. The content of the law is as follows. The dark channel image of the hazed-free image approaches 0, in which case the haze transmittance is about 1. For the areas in the image that are severely affected by haze and turn gray to white, the dark channel graph is close to 1, and the corresponding haze transmittance is about 0. By summarizing a large number of images, the researchers obtained the conclusion in Eq. (13).

It is worth noting that for sky areas and white objects in the image, the transmittance calculated using the dark channel method is usually biased. This phenomenon will reduce the dehazing effect of many methods. However, the proposed method uses an iterative approach to find the optimal number of inverse convolution layers in each sub-region. According to the estimated transmittance of each sub-region, the optimal deconvolution layer number corresponding to the transmittance in the laboratory experiment results in Section 3.2 is used as the initial value of the iterative layer number. The specific number of iteration layers is determined according to the image entropy, that is, when the image entropy reaches the maximum, the iteration in this region is stopped. Therefore, the proposed method can effectively avoid the influence of transmittance estimation bias on the recovery results. In any case, a more accurate transmittance estimation method is effective for the improvement of image dehazing algorithms. For the proposed method, more accurate transmittance estimates help to shorten the number of iterations and the calculation time.

3. Hierarchical deconvolution dehazing method based on transmission map segmentation

3.1 Image hierarchical dehazing process based on transmission image segmentation

According to the hierarchical scattering imaging model based on transmission map segmentation, we propose a corresponding image restoration method, as shown in Fig. 4.

Fig. 4. Schematic diagram of hierarchical deconvolution dehazing method based on transmission map segmentation

Download Full Size | PDF

Firstly, the hazed image is input, and the transmittance map of the image is calculated by using the DCP method. Next, a histogram analysis of the transmittance map is carried out. After that, we curve to fit the peaks in the histogram.

For images with large depth of field or complex scenes, the transmittance of each pixel decreases with the increase of distance. In real photos, the distance is usually not continuous. The image contains the sky area, the long-range area, and the close-in area. Therefore, the histogram of its transmittance image usually contains multiple peaks, as shown in the histogram curve in Fig. 4. In general, histogram curves are very uneven. Therefore, the curve can be further fitted only after it is smoothed. Here, the histogram curve is processed using mean filtering, and the window length is 7. Gaussian function fitting is carried out on the filtered curve, and each peak value in the curve is fitted respectively. The regions are divided according to the intersection of the peaks of fitted curves. Different regions in the image are deconvolved with different layers to get the sub-image recovery results. The number of inverse convolution layers increases with decreasing transmittance. Finally, the sub-images of each region are fused to obtain the dehazing image.

In the process of Gaussian fitting to the transmittance image histogram, the peak position of the histogram is taken as the initial value of the Gaussian peak position. After the initial fitting is completed, the first sub-image is obtained. The first fitting result is subtracted from the smoothed histogram curve, and then the second fitting is performed. And so on until the maximum residual is less than 0.1. According to the above principle, the image is divided into multiple subregions, and the number of partitions depends on the number of Gaussian functions obtained by fitting. For large depth-of-field images, the number of partitions is variable, usually between 2-5.

Normally, the sky region will be divided into sub-images with the lowest transmittance. In order to avoid the influence of sky region, we use the sub-image with the second lowest transmittance to calculate atmospheric light. The calculation method of atmospheric light is similar to that in the DCP. Firstly, the dark channel image of the sub-image is calculated, and the brightest pixel in the dark channel image is selected as the value of atmospheric light A. When atmospheric light is introduced into the image degradation model, the overall brightness of the image will decrease greatly.

According to the above process and the inverse convolution equation given in Eq. (11), the dehazing image can be obtained. In the proposed method, the number of inverse convolutional layers corresponding to each sub-region has the greatest impact on the defogging results. Therefore, the optimal number of deconvolution layers corresponding to hazed images with different transmittance has been studied.

3.2 Optimal number of inverse convolutional layers corresponding to haze transmittance

In order to obtain the best recovery results more accurately and faster, the optimal number of reverse convolutional layers corresponding to different transmittance has been studied. First, a haze with a transmittance of 0.1 to 1 is generated in an indoor laboratory environment with a step size of 0.1. Then, hazed images with various transmittance were collected in this environment, as shown in Fig. 5.

Fig. 5. Various transmittance hazed images taken in the laboratory

Download Full Size | PDF

In the experiments, the imaging target is placed in a fog box and fog is generated using an atomizer. The driving frequency of the atomizer is 1.7 MHz, and the droplet size it generates is between 2 and 3 μm, which is basically consistent with naturally occurring fog. The imaging targets are standard color boards, standard black and white boards, and car models. At the same time as imaging, the transmittance of fog is measured using a monitoring optical path directly behind the target. After the haze transmittance in the fog box is less than 0.1, turn off the atomizer and take photos as the haze gradually dissipates to ensure the uniformity of the fog.

Further, Eq. (11) is used for the layered deconvolution of each image. PSNR, SSIM, and image entropy are used to evaluate the quantitative relationship between the number of deconvolution layers and the recovery effect, as shown in Fig. 6.

Fig. 6. Relationship between the recovery effect of haze images with various transmittance and the number of inverse convolution layers: (a) Results under PSNR; (b) Results under SSIM; (c) Results under image entropy; (d) The optimal number of inverse convolution layers corresponding to different transmittance

Download Full Size | PDF

Each curve in Fig. 6 (a), (b), and (c) corresponds to a medium transmittance of 0.9∼0.1, respectively. The relationship between curve color and transmittance is shown in Fig. 6 (a). As the number of deconvolution layers increases, the recovery effect gradually improves until the optimal number of recovery layers is reached. Afterward, the recovery effect decreases as the number of deconvolution layers increases. Figure 6 (d) shows the number of inverse convolutional layers corresponding to the optimal restoration results of hazed images under various transmittance conditions. Under the three evaluation criteria of PSNR, SSIM, and image entropy, the number of inverse convolutional layers corresponding to the best restoration effect is basically the same. The optimal deconvolution layers corresponding to haze transmittance of 0.9 to 0.1 (step size 0.1) are 2∼4, 11∼12, 14∼16, 19∼23, 24∼27, 30∼34, 45∼50 and 50∼54, respectively.

The above results provide the basis for selecting the initial value of the inverse convolution layer number. The experimental results show that the optimal number of inverse convolution layers is positively correlated with $1 - t({x,y} )$, which is basically consistent with our theory.

In this experiment, only the number of inverse convolutional layers changes in Eq. (11), and other parameters remain unchanged, where ${\eta _3}$ equal to 0.03, ${\eta _1}$ equal to 0.01, ${\eta _2}$ equal to 0.01.

3.3 Image evaluation method

To evaluate the results of image restoration objectively, five evaluation indexes, PSNR, SSIM, image entropy, NIQE, and BRISQUE, are used to evaluate the defogging results. For images with corresponding ground-truth, the most commonly used metrics are PSNR and the SSIM. For the assessment of images without ground-truth, image entropy, NIQE, and BRISQUE are often used [23–25].

PSNR can directly reflect the similarity between the de-blurred image and the non-fogged image [26]. Firstly, the mean square error (MSE) of the two images was calculated according to Eq. (14).

(14)$$MSE = \frac{1}{{mn}}\sum\limits_{i = 0}^{m - 1} {\sum\limits_{j = 0}^{n - 1} {[{I({i,j} )- {I_{\textrm{noise}}}({i,j} )} ]} } .$$

And then the PSNR was calculated according to Eq. (15).

(15)$$PSNR = 10{\log _{10}}\left( {\frac{{{{255}^2}}}{{MSE}}} \right).$$

In addition, SSIM is used to evaluate the dehazing effect [27]. SSIM can well reflect the statistical characteristics of the image. As shown in Eq. (16).

(16)$$SSIM({{I_x},{I_y}} )= \frac{{({2{\mu_x}{\mu_y} + {c_1}} )({2{\sigma_{xy}} + {c_2}} )}}{{({\mu_x^2 + \mu_y^2 + {c_1}} )({\sigma_x^2 + \sigma_y^2 + {c_2}} )}},$$

where $\mu$ is the mean value of the image, $\sigma$ is the variance of the image, and ${\sigma _{xy}}$ is the covariance of the two images. ${c_1}$ and ${c_2}$ are two constants with very small values to prevent the denominator in the formula from being zero. In this paper, both ${c_1}$ and ${c_2}$ are assigned a value of 0.01.

And also, the image entropy method is often used in the field of image restoration [18]. Image entropy can represent the information richness of an image to some extent. For the hazed image, fog greatly reduces the effective information in the image, but significantly increases the information in the dehazed image. Therefore, image entropy can be used to evaluate the results of image dehazing. The calculation method of image entropy is shown in Eq. (17).

(17)$$S ={-} \sum\limits_{i = 1}^{255} {{p_i}} \ln {p_i},$$

where S is the image entropy, i is the gray value, and ${p_i}$ is the probability of the gray appearing.

4. Results and analysis

The proposed method is tested using fog images from the HSTS database, including 10 synthetic fog images and 3 natural fog images. The reason for using synthetic fog images is that these types of images have haze-free images as a comparison, which can calculate the most objective evaluation indicators such as PSNR and SSIM, while natural fog images can only use non-reference evaluation indicators.

Using the proposed strategy, we recovered multiple fog images and compared them with eight of the most advanced and effective single-image dehazing methods. The eight methods are DCP method [18], single image dehazing algorithm based on transmission map estimation with image fusion (TMEWIF) method [19], Color Attenuation Prior method (CAP) [20], single image recovery in scattering medium by propagating deconvolution (PD) [21], Image Dehazing and Exposure Using an Enhanced Atmospheric Scattering Model (IDE) [28], Image dehazing algorithm based on optimized dark channel and haze-line priors of adaptive sky segmentation (DCPHLP) [29], End-to-End System for Single Image Haze Removal method (DehazeNet) [8], and A Weakly Supervised Refinement Framework for Single Image Dehazing (RefineDNet) [9].

The dehazing effect of images are shown in Fig. 7. The experimental results indicate that the proposed method has achieved competitive recovery results. Table 1 shows the evaluation results of synthetic fog images after dehazing. The data in the table represents the mean of each image evaluation indicator. In addition, according to the order from top to bottom in the table, the average calculation time of each method is 1.2048s, 1.8532s, 1.3319s, 1.0954s, 2.0913s, 1.4628s, 1.4493s, 0.2106s and 0.8257s. It can be seen that the proposed method has good real-time performance.

Fig. 7. Dehazing results of synthetic fog images

Download Full Size | PDF

Table 1. The evaluation results of synthetic fog images after dehazing

View Table | View all tables in this article

The data in the table is the average of the evaluation results of ten images. The higher the PSNR, SSIM, and image entropy, the better the restoration quality of the image. Meanwhile, the lower the NIQE and BRISQUE indicators, the better the dehazing effect of the image. The proposed method achieved the best recovery performance under three evaluation indicators: PSNR, SSIM, and NIQE, and ranked second under the other two evaluation indicators.

Figure 8 shows the defogging effect of three natural fog images. The quantitative evaluation indicators for this group of images are shown in Table 2.

Fig. 8. Dehazing results of natural fog images

Download Full Size | PDF

Table 2. The evaluation results of natural fog images after dehazing

View Table | View all tables in this article

Due to the lack of ground-truth images in this group of images, only three non-reference evaluation indicators can be used to evaluate the dehazing results. The proposed method is optimal under the image entropy index and ranks second under the other two indicators. The areas within the green and blue borders in Fig. 8 are truncated and enlarged, as shown in Fig. 9. The green box area represents the sky and distant targets, while the blue box area represents nearby targets.

Fig. 9. Dehazing results for long range and close range targets

Download Full Size | PDF

The proposed method can achieve high-quality recovery of two regions simultaneously. The buildings and other targets in the distant area can be clearly seen, and the restoration effect of the nearby target area is also good. In contrast, the DCP method has the most significant effect on improving the contrast of distant targets, but there is also an excessive enhancement phenomenon, resulting in the sky area being too bright. In addition, the DCP method also exhibits the phenomenon of target darkening caused by transmittance estimation bias for nearby targets. Other comparison methods have poor recovery effects on long-range targets.

In comparison methods, the DCP method, TMEWIF method, CAP method, and DCPHLP overly rely on prior knowledge and estimation of transmittance. If prior knowledge is no longer applicable or the transmittance estimation is inaccurate, the recovery effectiveness of these methods will be greatly reduced. In contrast, the proposed method utilizes iterative methods to find the best recovery results, which can effectively avoid situations of over enhancement and insufficient recovery.

The PD method uses iterative methods to find the optimal solution, but this method uses the exact same degradation function for the entire image, which cannot balance the restoration effect of high and low transmittance areas in the image. In addition, the PD method does not consider the influence of atmospheric light, so the result of haze removal is still white and the contrast is poor. The result of the proposed method does not have the phenomenon of whitening, which effectively removes the influence of atmospheric light.

Two deep learning methods, DehazeNet and RefineDNet, rely on training samples [8,9]. Although these methods have good defogging effects, they are not very helpful for studying imaging models in scattering media. Moreover, these methods have poor restoration performance for image regions with very low transmittance, as shown in the long-distance target area in Fig. 9.

In summary, the proposed method can achieve the best defogging effect in the far, medium, and close-range regions of the image simultaneously.

5. Conclusions

To sum up, we propose a regional hierarchical image degradation model and design a complete image dehazing algorithm based on this model. Specifically, we segment the hazed image according to its transmittance map and restore different sub-regions independently. The DCP method is used to obtain the transmission map of the image. The histogram of the transmission map is then calculated and analyzed. According to the filtered histogram curve, the target of different distances is divided into regions. Iterative deconvolution of different layers is carried out for each sub-image after region division. Finally, the sub-images of each region are fused to obtain the dehazing image. In addition, we also study the optimal deconvolution layers for haze images with various transmittances. Based on this data, the best recovery results can be obtained quickly. Through the restoration experiment of fog images, the proposed method is proved to have a good effect on the hazed image. The quantitative evaluation index also proves the effectiveness of the method. In terms of applicability, the degradation model and the dehazing method proposed are more suitable for hazed images in large depth-of-field complex scenes. In other words, when a hazed image contains multiple transmittance sub-regions, the proposed method has strong competitiveness. In future work, we will devote ourselves to research and propose more universal degradation models and dehazing methods.

Funding

National Natural Science Foundation of China (61675002).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. C. Zhang, J. Zhang, X. Wu, et al., “Numerical analysis of light reflection and transmission in poly-disperse sea fog,” Opt. Express 28(17), 25410–25430 (2020). [CrossRef]

2. J. S. Jaffe, “Underwater Optical Imaging: The Past, the Present, and the Prospects,” IEEE J. Oceanic Eng. 40(3), 683–700 (2015). [CrossRef]

3. T. Li, S. Rong, W. Zhao, et al., “Underwater image enhancement using adaptive color restoration and dehazing,” Opt. Express 30(4), 6216–6235 (2022). [CrossRef]

4. J. Vazquez-Corral, G. Finlayson, and M. Bertalmío, “Physical-based optimization for non-physical image dehazing methods,” Opt. Express 28(7), 9327–9339 (2020). [CrossRef]

5. M. Song, R. Li, R. Guo, et al., “Single image dehazing algorithm based on optical diffraction deep neural networks,” Opt. Express 30(14), 24394–24406 (2022). [CrossRef]

6. S. An, X. Huang, L. Wang, et al., “Unsupervised single-image dehazing using the multiple-scattering model,” Appl. Opt. 60(26), 7858–7868 (2021). [CrossRef]

7. W. Ren, L. Ma, J. Zhang, et al., “Gated Fusion Network for Single Image Dehazing,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2018), pp. 3253–3261.

8. B. Cai, X. Xu, K. Jia, et al., “DehazeNet: An End-to-End System for Single Image Haze Removal,” IEEE Trans. on Image Process. 25(11), 5187–5198 (2016). [CrossRef]

9. S. Zhao, L. Zhang, Y. Shen, et al., “RefineDNet: A Weakly Supervised Refinement Framework for Single Image Dehazing,” IEEE Trans. on Image Process. 30, 3391–3404 (2021). [CrossRef]

10. Z. Zhu, Y. Luo, H. Wei, et al., “Atmospheric light estimation based remote sensing image dehazing,” Remote Sens. 13(13), 2432 (2021). [CrossRef]

11. Z. Zhu, H. Wei, G. Hu, et al., “A novel fast single image dehazing algorithm based on artificial multiexposure image fusion,” IEEE Trans. Instrum. Meas. 70, 1–23 (2021). [CrossRef]

12. L. Zhang, Z. Yin, K. Zhao, et al., “Lane detection in dense fog using a polarimetric dehazing method,” Appl. Opt. 59(19), 5702–5707 (2020). [CrossRef]

13. Y. Schechner, S. Narasimhan, and S. Nayar, “Polarization-based vision through haze,” Appl. Opt. 42(3), 511–525 (2003). [CrossRef]

14. J. Fade, S. Panigrahi, A. Carré, et al., “Long-range polarimetric imaging through fog,” Appl. Opt. 53(18), 3854–3865 (2014). [CrossRef]

15. R. Sun, T. Liao, Z. Fan, et al., “Polarization dehazing method based on separating and iterative optimizing airlight from the frequency domain for different concentrations of haze,” Appl. Opt. 61(35), 10362–10373 (2022). [CrossRef]

16. J. Liang, L. Ren, H. Ju, et al., “Polarimetric dehazing method for dense haze removal based on distribution analysis of angle of polarization,” Opt. Express 23(20), 26146–26157 (2015). [CrossRef]

17. H. Wang, H. Hu, J. Jiang, et al., “Automatic underwater polarization imaging without background region or any prior,” Opt. Express 29(20), 31283–31295 (2021). [CrossRef]

18. K. He, J. Sun, and X. Tang, “Single Image Haze Removal Using Dark Channel Prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]

19. S. Cheng and B. Yang, “An efficient single image dehazing algorithm based on transmission map estimation with image fusion,” Eng. Sci. Technol. Int. J. 19, 101190 (2022). [CrossRef]

20. Q. Zhu, J. Mai, and L. Shao, “A Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior,” IEEE Trans. on Image Process. 24(11), 3522–3533 (2015). [CrossRef]

21. R. Wang and G. Wang, “Single image recovery in scattering medium by propagating deconvolution,” Opt. Express 22(7), 8114–8119 (2014). [CrossRef]

22. S. Emberton, L. Chittka, and A. Cavallaro, “Underwater image and video dehazing with pure haze region segmentation,” Comput. Vis. Image Underst. 168, 145–156 (2018). [CrossRef]

23. X. Guo, Y. Yang, C. Wang, et al., “Image dehazing via enhancement, restoration, and fusion: A survey,” Inf Fusion. 86-87, 146–170 (2022). [CrossRef]

24. A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘Completely Blind’ Image Quality Analyzer,” IEEE Signal Process. Lett. 20(3), 209–212 (2013). [CrossRef]

25. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. on Image Process. 21(12), 4695–4708 (2012). [CrossRef]

26. Y. H. Shiao, T. J. Chen, K. S. Chuang, et al., “Quality of Compressed Medical Images,” J. Digit. Imaging 20(2), 149–159 (2007). [CrossRef]

27. Z. Wang, A. C. Bovik, H. R. Sheikh, et al., “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]

28. M. Ju, C. Ding, W. Ren, et al., “Image Dehazing and Exposure Using an Enhanced Atmospheric Scattering Model,” IEEE Trans. on Image Process. 30, 2180–2192 (2021). [CrossRef]

29. G. Cui, Q. Ma, J. Zhao, et al., “Image dehazing algorithm based on optimized dark channel and haze-line priors of adaptive sky segmentation,” J. Opt. Soc. Am. A 40(6), 1165–1182 (2023). [CrossRef]

	PSNR↑	SSIM↑	S↑	NIQE↓	BRISQUE↓
DCP	21.1340	0.9634	6.9744	3.1788	21.1734
TMEWIF	22.2821	0.9707	7.1640	3.1970	18.8637
CAP	23.9982	0.9825	7.4571	3.0961	15.2024
PD	24.0696	0.9829	7.4973	3.2411	21.1308
IDE	24.4961	0.9835	7.6696	3.0323	17.1733
DCPHLP	24.3178	0.9837	7.3011	3.0261	19.2712
DehazeNet	23.9525	0.9862	7.6152	2.8918	17.4137
RefineDNet	21.2669	0.9733	7.5832	2.9564	8.6572
proposed	27.4301	0.9918	7.6564	2.8151	14.0475

	E↑	NIQE↓	BRISQUE↓
DCP	6.6347	3.6968	32.7707
TMEWIF	7.2919	3.6701	28.4177
CAP	7.3436	3.5634	30.3221
PD	7.3598	3.6508	29.2927
IDE	7.5088	3.4658	34.5401
DCPHLP	7.1923	3.4764	30.1790
DehazeNet	7.1761	3.6499	33.9609
RefineDNet	7.4315	2.9870	25.0765
proposed	7.6194	3.4427	27.8291

	PSNR↑	SSIM↑	S↑	NIQE↓	BRISQUE↓
DCP	21.1340	0.9634	6.9744	3.1788	21.1734
TMEWIF	22.2821	0.9707	7.1640	3.1970	18.8637
CAP	23.9982	0.9825	7.4571	3.0961	15.2024
PD	24.0696	0.9829	7.4973	3.2411	21.1308
IDE	24.4961	0.9835	7.6696	3.0323	17.1733
DCPHLP	24.3178	0.9837	7.3011	3.0261	19.2712
DehazeNet	23.9525	0.9862	7.6152	2.8918	17.4137
RefineDNet	21.2669	0.9733	7.5832	2.9564	8.6572
proposed	27.4301	0.9918	7.6564	2.8151	14.0475

	E↑	NIQE↓	BRISQUE↓
DCP	6.6347	3.6968	32.7707
TMEWIF	7.2919	3.6701	28.4177
CAP	7.3436	3.5634	30.3221
PD	7.3598	3.6508	29.2927
IDE	7.5088	3.4658	34.5401
DCPHLP	7.1923	3.4764	30.1790
DehazeNet	7.1761	3.6499	33.9609
RefineDNet	7.4315	2.9870	25.0765
proposed	7.6194	3.4427	27.8291

Hierarchical deconvolution dehazing method based on transmission map segmentation

Abstract

1. Introduction

2. Principle

2.1 Image degradation model in haze

2.2 Hierarchical degradation model for images in scattering media

2.3 Hierarchical scattering imaging model based on transmission map segmentation

2.4 Calculation method of transmission map

3. Hierarchical deconvolution dehazing method based on transmission map segmentation

3.1 Image hierarchical dehazing process based on transmission image segmentation

3.2 Optimal number of inverse convolutional layers corresponding to haze transmittance

3.3 Image evaluation method

4. Results and analysis

5. Conclusions

Funding

Disclosures

Data availability

References

Data availability

Cited By

Figures (9)

Tables (2)

Equations (17)

Optics Express