Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Dark channel prior based video dehazing algorithm with sky preservation and its embedded system realization for ADAS applications

Open Access Open Access

Abstract

Dark Channel Prior (DCP) is one of the significant dehazing methods based upon the observation of the key features of the haze-free images. But it has disadvantages; high computational complexity, over-enhancement in the sky region, flickering artefacts in video processing, and poor dehazing. Therefore, we propose improved solutions to solve the aforementioned drawbacks. First, we adopt the fast one-dimensional filter, look-up table, and program optimization to reduce the computational complexity. Next, we follow by using a part of the guided filter for sky detection and to preserve the sky region from noise by avoiding over recovery. Then, we propose an airlight update strategy and adjust the radius of a guided filter to reduce the flickering artifacts and also propose an airlight estimation method to produce the better dehazing result as the final step of our algorithm. The improved results from our proposed algorithm are stable and are obtained from the real-time processing suitable for ADAS, surveillance, and monitoring systems. The implementation of the proposed algorithm has yielded a processing speed of 75 fps and 23 fps respectively on an NVIDIA Jetson TX1 embedded platform and Renesas R-Car M2, both on D1 (720x480) resolution videos.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

1. Introduction

In recent years, as the disasters due to hazy environments are increasing, development of real-time dehazing technique require immediate action. Fundamentally, intelligent vision systems such as in Advance Driver Assistance System (ADAS), surveillance and monitoring systems etc., rely on the attributes of the input images gravely affecting the precision of the objects recognition. Although the single-image-haze-removal algorithms have become mature, the state-of-the-art dehazing algorithms still face certain earnest problems like high computational complexity and temporal coherence issue when extended to video dehazing process.

The absorption and/or scattering of natural light due to atmospheric particles present as a result of air pollution, varied weathers and water droplets causing mist, fog, haze in most parts of the world, lowers contrast of the images mostly captured in the outdoors and appear dimmer. Thus debased images unceasingly influence the adeptness of the surveillance and tracking systems including the contemporary Advanced Driving Assistance Systems (ADAS) and self-driving cars.

With rapid technological advancements, surveillance and monitoring systems, remote sensing systems, ADAS etc., are in place in almost every other technology utilization. The accuracy of these systems largely depends on the quality of the input images. Therefore, the demands for the improved quality of the input images are increasing and is the dire need of the hour. Although the physics of weather degraded image optics was discovered by Koschmieder [1] in 1740 and computer vision technologies are steadily making notable progress, we still face numerous challenges. The single image haze removal has made phenomenal progress in the recent decades. Besides, extending the state-of-the-art methods to video dehazing suffers from the computational complexity and temporal coherence issue in the practical real-time applications. There exist some cases of disasters caused by the haze in modern life such as TESLA’s self-driving car had an accident in China due to the hazy weather [2] costing the life of the driver. Furthermore, we also encounter a foggy scenario while executing surgeries, surveillance in the foggy, hazy weathers, utilizing ADAS in hazy road condition and so on. Since all these circumstances are a matter of life and death, it is critical to develop a concurrent video dehazing technique which can be employed in all the applications requiring de-hazed input images including the ADAS.

2. Background

Haze is an atmospheric appearance where dust, smoke, fog, and other dry particles obscure clear scene. The images/videos captured in haze is blended with airlight and object scene radiance. In computer vision researches, haze removal is considered as an ill-disposed problem, moreover, the scene degradation is spatial-variant. Hence, removal of haze is still a challenging problem these days.

2.1 Atmospheric scattering model

The atmospheric scattering model proposed by McCartney [3], is widely used to outline the formation of a hazy image as in Fig. 1 and is illustrated by the Eq. (1) and Eq. (2).

I(x)=J(x)t(x)+A(1t(x))
t(x)=eβd(x)
In Eq. (1), x is an index of the pixel within an image, I is the hazy image observed from the camera, J is the scene radiance representing the haze-free image, A is the atmospheric light, t is the medium transmission describing the portion of the light that is non-scattered and reaches the camera, β is the scattering coefficient of the atmosphere and d is the scene depth. I, J, and A are three-dimensional vectors in the RGB color space. The first term, J(x)t(x), is called the direct attenuation indicating the decay of the scene radiance in the medium while airlight results from previously scattered light, and the second term, A(1-t(x)), is called airlight. The main concept of this dehazing function is to estimate A and t, and then accompany with the known I to restore the scene J from the model.

 figure: Fig. 1

Fig. 1 Atmospheric Scattering Model.

Download Full Size | PDF

2.2 Dark channel prior

He et al. [4] discovered the Dark Channel Prior (DCP) from the statistical observation of the haze-free outdoor images that in most of the pixels of the non-sky patches, at least one color channel has very low intensity at some pixels. There are five major steps in DCP namely, airlight estimation, dark channel estimation, coarse transmission estimation, transmission refinement, and scene recovery. For an image J, the estimation of the dark channel is as shown in Eq. (3), where Jc(y) represents the color channel of J and Ω(x) is a local patch anchored at x.

Jdark(x)=minc{r,g,b}(minyΩ(x)Jc(y))
According to observation in [4], the value of Jdark is low and tends to be zero except for the sky region if J is a haze-free outdoor image. Then, the dark channel prior and the imaging hazy model [3] are combined together. The transmission in a local patch Ω(x) is assumed to be a constant and the patch’s transmission is denoted by t ̃(x). Taking the minimum operation in the local patch on the haze imaging equation as in Eq. (4).
minyΩ(x)(Ic(y))=t˜(x) minyΩ(x)(Jc(y))+(1t˜(x))Ac
Then, assuming Eq. (3) to be equal to zero, it is combined with Eq. (4). Therefore, we get the coarse transmission as shown in Eq. (5), and ω is a constant parameter (0<ω≤1) to retain some haze for the distant view.
t˜(x)=1ωminc(minyΩ(x)(Ic(y)Ac))
However, using the coarse transmission t ̃(x) to recover the scene may cause blocking artefacts. Hence [4], employed the soft matting for transmission refinement and to recover the scene by using the refined transmission t(x). Equation (6) shows the recovered equation. The directly recovered scene radiance J is prone to noise. Therefore, the transmission t(x) is restricted to a lower bound, t0which means that a small amount of haze is preserved in a very dense haze region.

J(x)=I(x)Amax(t(x),t0)+A

2.3 Real-time Dehazing for Image and Video

Lv et al. [5] replaced the most complex part of DCP [4], i.e., soft matting, with the cross-bilateral filter. A feature found in the DCP is that the refined transmission map is smooth everywhere except for the sharp depth discontinuities. Hence, they utilized the edge-preserving filter to accelerate the refined process.

Equation (7), where t˜(x) and t(x) is the coarse and refined transmission, G is the common Gaussian function, σs controls the spatial weight and σr controls the range weight, shows the equation of the cross-bilateral filter, and σs=16 and σr=0.1 are used for all results in [5].

t(x)=1w(x)yΩ(x)Gσs(xy)Gσr(|ExEy|)t˜(x)
The range weight guided by the intensity E of the input image prevents pixels on one side of a strong edge from influencing pixels on the other side since they have different values. w(x) is the sum weight of the local patch centered by pixel 𝑥 as shown in Eq. (8).
w(x)=yΩ(x)Gσs(xy)Gσr(|ExEy|)
In the estimation of the airlight, they chose the highest intensity pixel as the airlight, instead of the top 0.1% intensity pixel. Hence, this may encounter the failed dehazing result when the airlight is not the highest intensity.

In [5], the temporal coherence issue was not discussed, instead, the speed-up efficiency is shown. Two versions, i.e., CPU-version, and NVIDIA-CUDA-based GPU-version, are implemented. For an input resolution with 600x400, the system is able to reach about ten to twelve frames per second in GPU-version, and two frames per second in CPU-version.

3. Proposed algorithm

Dark Channel Prior (DCP) method for dehazing images proposed by He et al. [4], is chosen as the base of our proposed algorithm. DCP exhibits four disadvantages such as, (i) high computational complexity; (ii) over-enhancement in the sky region; (iii) flickering artifacts in the video processing; (iv) dim dehazing result and is designed for single image dehazing making it incompatible for the real-time and video applications.

Our proposed method aims to apply DCP concept to video dehazing in real time environments. In the proposed algorithm, we average the values of pixel above the threshold for airlight estimation, utilize the fast guided filter [6] to replace the image matting method, detect the smooth region as the sky employing the guided filter, introduce a novel airlight update strategy for temporal coherence, and employ the larger radius of the mean filter to reduce the flickering artefacts. Besides, the real-time processing is the key value to our video dehazing algorithm. With the aim of reducing the complexity of the dehazing algorithm, we apply a one-dimensional fast filtering for the mean filter and the minimum filter. Downscaling and look-up table are involved as well. The proposed algorithm flow is as shown in Fig. 2, and the detailed algorithm is discussed in the following sections followed by the embedded system implementation details and results and comparison of our proposed algorithm.

 figure: Fig. 2

Fig. 2 The flowchart of the proposed DCP based dehazing algorithm.

Download Full Size | PDF

3.1 Airlight estimation

The issue with most of the airlight estimation is the computational complexity because of the sorting and the recursive block-based computation. Thus, the threshold equal to 255 - θairlight, above which the pixel intensity value average can be generated for airlight estimation, is proposed as illustrated in Fig. 3. The value of threshold depends on the exposure of input.

 figure: Fig. 3

Fig. 3 The proposed airlight estimation.

Download Full Size | PDF

Our proposed threshold θairlight gives better dehazing result image than DCP [4] does, which is, a dim image is always generated by original DCP and a various kinds of clear images are generated employing our algorithm with different θairlight as shown in Fig. 4.

 figure: Fig. 4

Fig. 4 Comparison between Dehazing result images. (a) Input hazy image, (b) θairlight = 10, (c) θairlight = 50 (d) DCP generated image

Download Full Size | PDF

3.2 Sky detection

DCP [4] is weak against noise, especially the sky and the dense haze regions. Hence we expect to preserve some haze in the recovered Eq as in Eq. (6). Wang et al. [7] proposed a method for the sky region segmentation. Nevertheless, it will fail while dealing with the images without the sky, and its complexity is high as well. Hence, we propose a simple idea for the sky detection based on block-based airlight estimation method [8] and the guided filter [9]. Kim et al. [8] calculated the score value by subtracting the standard deviation and the mean for evaluating a smooth region. He et al. [9] calculated the mean and the variance of input images using the specific algorithm is shown in Algorithm 1.

Tables Icon

Table 1. Computer specification of NVIDIA Jetson TX1 and Renesas R-Car M2

Tables Icon

Algorithm 1. The guided filter [9].

Therefore, we extract the part of the guided filter employing the proposed sky detection, followed by defining a threshold θS for judging if the pixel is of the sky or not. The proposed sky detection is intrinsic from the guided filter. Therefore, it does not increase the computational cost. The proposed algorithm and the result of the sky detection are as shown in Algorithm 2 and Fig. 5, respectively. By our experiments, θS = 0.6 is more suitable for most of the cases. Hence, θS is set as 0.6 in the proposed algorithm.

Tables Icon

Table 2. Processing performance comparison of TX1 and PC (Unit: fps)

 figure: Fig. 5

Fig. 5 The result of proposed sky estimation method.

Download Full Size | PDF

Tables Icon

Algorithm 2. The proposed algorithm for the sky estimation.

3.3 Dark channel estimation

Dark channel is estimated according to the Eq. (3) and it is an effective way to obtain the coarse transmission t ̃(x). The minimum filter is utilized in this step. We have tried a various sets of rmin in our experiments and rmin = 7 is found to be the best and is used in our proposed algorithm and the result is as shown in Fig. 6.

 figure: Fig. 6

Fig. 6 The result of dark channel estimation. (a) Input hazy image, (b) Result image.

Download Full Size | PDF

3.4 Coarse Transmission Estimation

The coarse transmission t ̃(x) is estimated based on the dark channel as in the Eq. (5). As the DCP is unfitting for the sky region, a lower boundtsky = 0.3 is set for the pixels labeled in the sky detection. The result is shown in Fig. 7. The parameter ω in Eq. (5) is 0.95 in our proposed algorithm chosen from the experiments.

 figure: Fig. 7

Fig. 7 The result of coarse transmission estimation. (a) Dark channel image. (b) Result image.

Download Full Size | PDF

3.5 Transmission refinement

A guided filter is utilized in our work due to its low computational complexity and the refined quality. A part of the guided filter is calculated for the sky detection. Therefore, some of the computational cost can be saved and only the rest of the guided filter needs to be calculated as shown in Algorithm 3. The refined result is as shown in Fig. 8 and Fig. 9. The guided filter is constructed by the mean filter. From our experiments, it is found that the radius of mean filter is better when it is larger than the radius of minimum filter by 4 times. Hence, the radius of the mean filter rmean is set to 28 from the experiments.

Tables Icon

Table 3. Processing performance comparison of NVIDIA Jetson TX1 and Renesas R-Car M2 (Unit: fps)

 figure: Fig. 8

Fig. 8 The refined transmission result. (a) Coarse Transmission Result. (b) Refined Transmission Result.

Download Full Size | PDF

 figure: Fig. 9

Fig. 9 The effect of adopting proposed refined transmission. (a) Without refined transmission. (b) With refined transmission.

Download Full Size | PDF

Tables Icon

Algorithm 3. The rest part of the guided filter.

3.6 Scene radiance recovery

The atmospheric light A and t(x) are obtained after the airlight estimation and the transmission refinement. The scene radiance can be recuperated from the Eq. (6) directly. The overall process of DCP as illustrated in Fig. 10.

 figure: Fig. 10

Fig. 10 The procedure and its result image of DCP. (a) Input hazy image. (b) Dark channel. (c) Coarse transmission. (d) Refined transmission. (e) Recovered scene.

Download Full Size | PDF

3.7 Airlight update strategy

The present methods of the airlight estimation evaluate the airlight independently for each frame resulting in an unbalanced change and causes the flickering results. Thus, the temporal information should be taken into consideration for the coherence issue. The update strategy by adopting the historical value is proposed in [10] and the update equation is as shown in Eq. (9), where At is the airlight value in the current frame t. At1 is the airlight value in the previous frame t-1, Anew is a linear combination of a ratio co-efficient λ, and the value of λ is between 0 and 1. Since we tend to believe more in the previous airlight value as the corresponding value, λ is set as 0.95 in the proposed algorithm.

Anew=λAt1+(1λ)At

However, an abrupt change of the airlight value still leads to the color shift, a constraint that needs to be set for restricting the difference of the adjacent frames if it is greater than a threshold θdiff. The compared results are shown in Fig. 11 and Fig. 12, and the algorithm of the proposed update strategy is as shown in Algorithm 4. By our experiments, θdiff should be approximate to 5 if we want to keep the luminance coherence in the entire video. Therefore, θdiff is set as 5 in the proposed algorithm.

 figure: Fig. 11

Fig. 11 The comparison of different airlight update strategies. (a) Input hazy images at frame #171, #172, #173, and #484. (b) Update airlight independently. (c) Airlight update strategy of [11]. (d) Our Proposed airlight update strategy.

Download Full Size | PDF

 figure: Fig. 12

Fig. 12 The line charts by different airlight update strategies. (a) Update airlight frame by frame. (b) Airlight update strategy in [10]. (c) Our proposed airlight update strategy.

Download Full Size | PDF

Tables Icon

Table 4. Comparison of the processing speed on Intel i7-6900K (Unit: fps)

Tables Icon

Algorithm 4. The rest part of the guided filter.

3.8 Transmission coherence

Extending single image dehazing to video dehazing should keep the temporal coherence consistent for a smooth video result. Many existing algorithms are used for temporal coherence, such as Markov random field (MRF) [12], optical flow [13], block matching method [11] and optimization of the cost function [8]. In our observation, moving white objects break the transmission coherence and produce flickering artefacts. The reason also comes from the prior we used in the algorithm. DCP [4] and color attenuation prior [14] will fail while facing the white object. The minimum filter is a simple solution to solve the incorrect patch produced by the white objects in the transmission map. Figure 13 and Fig. 14(c) show the results of the minimum filter.

 figure: Fig. 13

Fig. 13 The single-image dehazing result with adopting minimum filter. (a) Input hazy image. (b) Coarse transmission map. (c) Dehazing result. (d) Coarse transmission after minimum filter. (e) Dehazing result.

Download Full Size | PDF

 figure: Fig. 14

Fig. 14 Spatial incoherence issue and its solutions. (a) Input hazy images at frame #399, #400, and #401. (b) Spatial incoherence. (c) Reduce spatial incoherence result by minimum filter. (d) Reduce spatial incoherence by enlarging the radius of the mean filter.

Download Full Size | PDF

In Fig. 13, the results after adopting the minimum filter is obviously darker than the one without adopting the minimum filter. Therefore, this method will make the transmission map more over-estimated and increase the computational cost.

Abandoning the previous concepts, we propose a method to reduce flickering artefacts based on adjusting the radius of the filter. The idea comes from the receptive field of the filter. For instance, a small radius of the filter emphasizes the locality around the anchor point of the filter and a large radius of the filter will reduce the locality. Therefore, the large radius of the filter benefits the transmission coherence since it can decrease the discontinuous change by the white object in the transmission map. Figure 14(d) shows the result of the enlarged radius of the mean filter. Furthermore, enlarging the radius of the mean filter results in a drawback that it will reduce the dehazing effect. Hence, it is a tradeoff between temporal coherence and dehazing effect by harnessing the proposed method. Intending to achieve better dehazing effect in practical use, adjusting the radius of the mean filter will be adopted in the proposed algorithm. In addition, the value of the radius is related to the video resolution. That is, a larger resolution needs a larger radius. The radius of the mean filter is set as 60 in our proposed video dehazing algorithm, instead of 28 in the single-image haze removal.

3.9 Fast filtering

The proposed algorithm is based on DCP [4], and it includes three filters namely, minimum filter, mean filter, and guided filter. Hence, we apply fast filtering from [6], [15], and [16] so as to decrease the computational cost. Moreover, the scaling ratio of the fast guided filter is a tradeoff between the accuracy of the transmission refinement and the computational efficiency. Thus, the scaling ratio of the fast guided filter is set as 0.5 derived from our experiments. Furthermore, the 2-D filtering is replaced with the 1-D filtering.

4. Implementation on embedded system

The haze removal algorithm is usually suffered from the computational complexity. Hence, most of the proposed works are implemented on a desktop computer rather than on an embedded platform. In order to employ the dehazing techniques in practical applications, we have reduced the computational complexity and optimized the proposed algorithm to make it suitable for the embedded platform implementation. We chose NVIDIA Jetson TX1 and Renesas R-Car M2 as our embedded implementation platforms. The specifications are shown in Table 1. The CPU of Jetson TX1 is based on ARM core processor. Therefore, we also utilize the ARM's SIMD technique and ARM Neon for the proposed algorithm. In our experiment, we compare the sequential and the speed-up versions on the desktop computer and TX1. Table 2 shows the processing performances on the test videos of three different resolutions. The proposed method is a multithread program and requires three threads at least. The CPU resource of M2 is much lesser than TX1, hence TX1 is 3.4x faster than M2. However, the proposed method with low complexity benefits the practical use, the processing performance on R-Car M2 still yields 23 fps on an average. Table 3 shows the comparison of the processing performance between M2 and TX1. For the TX1 implementation, although it has NVidia GPU inside, we only utilize its ARM Quad-Core A57 for the power consumption reasons. We can achieve 75 fps @D1 without GPU acceleration.

5. Comparison and results

The dehazing image results processed by the proposed algorithm are compared with the previous notable algorithms of the related works. In our experiment, we utilize the parallel programming, SIMD and pthread, and employ the speed-up filtering. All the experiments and comparisons are done on an Intel i7-6900K.

Figures 15-20 shows the comparison of the single image dehazing results with the results of DLCE [17], DCP [3], Meng et al.’s [18], Tarel et al.’s [19], and our proposed method.

 figure: Fig. 15

Fig. 15 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

 figure: Fig. 16

Fig. 16 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE 1 [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

 figure: Fig. 17

Fig. 17 Comparison of results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

 figure: Fig. 18

Fig. 18 Comparison of the results different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

 figure: Fig. 19

Fig. 19 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

 figure: Fig. 20

Fig. 20 Comparison of the results from different methods on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.

Download Full Size | PDF

In the video dehazing approach, we compare the processing performance and show the dehazing results for several frames. The comparison of the results are shown in Table 4 and Fig. 20. It can be noted that compared to the dehazing results of the four different algorithms, the proposed method apparently outperforms the other methods. Our proposed algorithm generates better results than those from DCP, maintains the color tone even after dehazing, preserves the sky region from noise and color distortion, keeps the best coherence for the entire video and performs at a faster processing speed than the other previous methods. In addition, the disadvantages of the other algorithms are listed as below. Since the transformation function produced by the histogram-based method is random and unpredictable in each frame, DLCE cannot maintain the color tone and produces results with the color shift.

As we mentioned before, DCP is inappropriate for the processing the sky region and it overestimates the transmission. Therefore, the results from the DCP contain the noisy sky and appears dim. As noticed from Fig. 21, the OCE cannot remove the haze thoroughly in the Visualization 1 and produces white artefacts on the road in case of the Visualization 2. The MRF only keeps the temporal coherence for adjacent frames and not for the distant frames. Hence, the dehazing results by MRF still contain some haze and also appear dim. To sum up, the other methods reduce the computational complexity but compromise with the dehazing quality. However, our proposed algorithm not only decreases the computational cost but also harvests better dehazing quality and the temporal coherence.

 figure: Fig. 21

Fig. 21 Comparison of different approaches on Visualization 1 and Visualization 2. (a) Input hazy frames. (b) DLCE [17]. (c) DCP [4]. (d) OCE [8]. (e) MRF [12]. (f) Our proposed method.

Download Full Size | PDF

In addition, the proposed algorithm can be extended to real-time dehazing for the ADAS applications. Figure 22 shows the result of the dehazed road-sweeper to which one of the automated driver-assist system had crashed onto resulting in the fatal death of the auto pilot [2]. Figure 23 and Fig. 24 are the dehazed video frame chosen from the VIP Data set [20] and a random camcorder, respectively. Thus, the proposed algorithm can be employed in the ADAS and self-driving system in order to maximize the quality of the input images.

 figure: Fig. 22

Fig. 22 (a) The input video frame from one of the automated driver-assist systems. (b) The corresponding dehazed video frame of (a).

Download Full Size | PDF

 figure: Fig. 23

Fig. 23 (a) The input video frame from the VIP Data set. (b) The corresponding dehazed video frame of (a).

Download Full Size | PDF

 figure: Fig. 24

Fig. 24 (a) The input video frame from a random camcorder. (b) The corresponding dehazed video frame of (a).

Download Full Size | PDF

We also test our result in an Autonomous Emergency Braking (AEB) system using deep leaning technology developed by us. An object detection model by training VGG-16 [21] with Faster R-CNN [22] object detection system was developed to detecting moving objects on roads, which including pedestrian, vehicle and scooter. This model was trained with our in-house dashcam videos, and was tested in the AEB system using the original autopilot system crash video and video processed by our proposed method. As the result shown in Fig. 25, the AEB with our proposed method can detect the construction vehicle 2 seconds before the collision. With the aid of our proposed de-hazing method, the ADAS system will be more robust in different kinds of environments or weathers.

 figure: Fig. 25

Fig. 25 (a) Detection result of the original video. (b) Detection result of video processed by the proposed method.

Download Full Size | PDF

It is also found that the proposed algorithm prevents the sky distortion by detecting smooth region with the guided filter. Then, we also considered the temporal coherence problem while extending the single image dehazing to the video dehazing employing the proposed airlight update strategy and adjusting the radius of the mean filter. Lastly, for the real-time applications, we adopt parallel programming skills, like SIMD and pthread and employ fast filtering and look-up table for speeding-up the dehazing process. Thus, the proposed dehazing algorithm appears to possess better efficiency and is successfully implemented on an NVIDIA Jetson TX1 achieving a processing speed of 75 fps on D1 (720x480) resolution videos. The proposed algorithm is also implemented on a Renesas R-Car M2 yielding 23 fps on D1 resolution videos.

6. Conclusion

The proposed algorithm is developed to improve the four major drawbacks of DCP [4]: (1) dim dehazing result; (2) over-enhancement in the sky region; (3) flickering artifacts in the video processing; (4) high computational complexity. First, we propose a novel airlight estimation method for improving the dim look of the image. Second, the proposed algorithm avoids from the sky distortion by detecting smooth region with the guided filter. Third, we also consider the temporal coherence problem while extending the single image dehazing to the video dehazing by the proposed airlight update strategy and adjusting the radius of the mean filter. Fourth, for the real-time application, we adopt parallel programming skill, like SIMD and pthread, and employ fast filtering and look-up table for speed-up. Finally, we make an implementation on NVIDIA Jetson TX1 and reach 75 frames per second processing speed at D1 (720x480) resolution video. Implementation on Renesas R-Car M2 can reach 23 fps @ D1 video.

Funding

Ministry of Science and Technology, Taiwan R.O.C. (MOST107-3017-F-009-001 and MOST107-2634-F-009-008)

Acknowledgment

We appreciate the support from the “Center for mmWave Smart Radar Systems and Technologies” under the ‘Featured Areas Research Center Program’ within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE), Taiwan, R.O.C., and partially supported under MOST projects with grants MOST107-3017-F-009-001 and MOST107-2634-F-009-008 in Taiwan, R.O.C.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

References

1. H. Koschmieder, “Theorie der horizontalen sichtweite,” Beitrage zur Physik der freien Atmosphare pp. 33–53 (1924).

2. N. E. Boudette, “Autopilot cited in death of chinese driver,” The New York Times (2016).

3. E. J. McCartney, “Optics of the atmosphere: scattering by molecules and particles,” New York, John Wiley Sons, Inc., 1976. 421 p. (1976).

4. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]   [PubMed]  

5. X. Lv, W. Chen, and I.-f. Shen, “Real-time dehazing for image and video,” in Computer Graphics and Applications (PG), 2010 18th Pacific Conference on, (IEEE, 2010), pp. 62–69.

6. K. He and J. Sun, 6. K. He and J. Sun, “Fast guided filter. arxiv 2015,” arXiv preprint arXiv:1505.0099.

7. G. Wang, G. Ren, L. Jiang, and T. Quan, “Single image dehazing algorithm based on sky region segmentation,” Inf. Technol. J. 12(6), 1168–1175 (2013). [CrossRef]  

8. J.-H. Kim, W.-D. Jang, J.-Y. Sim, and C.-S. Kim, “Optimized contrast enhancement for real-time image and video dehazing,” J. Vis. Commun. Image Represent. 24(3), 410–425 (2013). [CrossRef]  

9. K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE transactions on pattern analysis & machine intelligence pp. 1397–1409 (2013).

10. M. Wang, J. Mai, Y. Liang, R. Cai, T. Zhengjia, and Z. Zhang, “Component-based distributed framework for coherent and real-time video dehazing,” in Computational Science and Engineering (CSE) and Embedded and Ubiquitous Computing (EUC), 2017 IEEE International Conference on, vol. 1 (IEEE, 2017), pp. 321–324. [CrossRef]  

11. X. Xiang, Y. Cheng, and J. Tang, “A novel video dehazing method based on temporal visual coherence,” in Proceedings of the 7th International Conference on Internet Multimedia Computing and Service, (ACM, 2015), p. 36. [CrossRef]  

12. B. Cai, X. Xu, and D. Tao, “Real-time video dehazing based on spatio-temporal mrf,” in Pacific Rim Conference on Multimedia, (Springer, 2016), pp. 315–325. [CrossRef]  

13. J. Zhang, L. Li, Y. Zhang, G. Yang, X. Cao, and J. Sun, “Video dehazing with spatial and temporal coherence,” Vis. Comput. 27(6-8), 749–757 (2011). [CrossRef]  

14. Q. Zhu, J. Mai, and L. Shao, “A fast single image haze removal algorithm using color attenuation prior,” IEEE Trans. Image Process. 24(11), 3522–3533 (2015). [CrossRef]   [PubMed]  

15. S. Rakshit, A. Ghosh, and B. U. Shankar, “Fast mean filtering technique (fmft),” Pattern Recognit. 40(3), 890–897 (2007). [CrossRef]  

16. D. Lemire, “Streaming maximum-minimum filter using no more than three comparisons per element,” arXiv preprint cs/0610046 (2006).

17. S.-W. Hsu, G.-Y. Chen, P.-C. Shen, C.-Y. Cho, and J.-I. Guo, “Dynamic local contrast enhancement for advanced driver assistance system in harsh environments,” in Consumer Electronics-Taiwan (ICCE-TW), 2014 IEEE International Conference on, (IEEE, 2014), pp. 69–70. [CrossRef]  

18. G. Meng, Y. Wang, J. Duan, S. Xiang, and C. Pan, “Efficient image dehazing with boundary constraint and contextual regularization,” in Proceedings of the IEEE international conference on computer vision, (2013), pp. 617–624. [CrossRef]  

19. J.-P. Tarel, N. Hautiere, L. Caraffa, A. Cord, H. Halmaoui, and D. Gruyer, “Vision enhancement in homogeneous and heterogeneous fog,” IEEE Intell. Transp. Syst. Mag. 4(2), 6–20 (2012). [CrossRef]  

20. IEEE Signal Processing Society, ideo and Image Processing Cup-2017,” https://ghassanalregib.com/cure-tsd/.

21. S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Adv. Neural Inf. Process. Syst. 28, 91–99 (2015).

22. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 (2014).

Supplementary Material (2)

NameDescription
Visualization 1       Comparison of different approaches on video using (a) Input hazy frames. (b) DLCE [17]. (c) DCP [4]. (d) OCE [8]. (e) MRF [12]. (f) Our proposed method. 4. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transa
Visualization 2       Comparison of different approaches on video using (a) Input hazy frames. (b) DLCE [17]. (c) DCP [4]. (d) OCE [8]. (e) MRF [12]. (f) Our proposed method. 4. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transa

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (25)

Fig. 1
Fig. 1 Atmospheric Scattering Model.
Fig. 2
Fig. 2 The flowchart of the proposed DCP based dehazing algorithm.
Fig. 3
Fig. 3 The proposed airlight estimation.
Fig. 4
Fig. 4 Comparison between Dehazing result images. (a) Input hazy image, (b) θairlight = 10, (c) θairlight = 50 (d) DCP generated image
Fig. 5
Fig. 5 The result of proposed sky estimation method.
Fig. 6
Fig. 6 The result of dark channel estimation. (a) Input hazy image, (b) Result image.
Fig. 7
Fig. 7 The result of coarse transmission estimation. (a) Dark channel image. (b) Result image.
Fig. 8
Fig. 8 The refined transmission result. (a) Coarse Transmission Result. (b) Refined Transmission Result.
Fig. 9
Fig. 9 The effect of adopting proposed refined transmission. (a) Without refined transmission. (b) With refined transmission.
Fig. 10
Fig. 10 The procedure and its result image of DCP. (a) Input hazy image. (b) Dark channel. (c) Coarse transmission. (d) Refined transmission. (e) Recovered scene.
Fig. 11
Fig. 11 The comparison of different airlight update strategies. (a) Input hazy images at frame #171, #172, #173, and #484. (b) Update airlight independently. (c) Airlight update strategy of [11]. (d) Our Proposed airlight update strategy.
Fig. 12
Fig. 12 The line charts by different airlight update strategies. (a) Update airlight frame by frame. (b) Airlight update strategy in [10]. (c) Our proposed airlight update strategy.
Fig. 13
Fig. 13 The single-image dehazing result with adopting minimum filter. (a) Input hazy image. (b) Coarse transmission map. (c) Dehazing result. (d) Coarse transmission after minimum filter. (e) Dehazing result.
Fig. 14
Fig. 14 Spatial incoherence issue and its solutions. (a) Input hazy images at frame #399, #400, and #401. (b) Spatial incoherence. (c) Reduce spatial incoherence result by minimum filter. (d) Reduce spatial incoherence by enlarging the radius of the mean filter.
Fig. 15
Fig. 15 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 16
Fig. 16 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE 1 [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 17
Fig. 17 Comparison of results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 18
Fig. 18 Comparison of the results different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 19
Fig. 19 Comparison of the results from different approaches on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 20
Fig. 20 Comparison of the results from different methods on single hazy image. (a) Input hazy image. (b) DLCE [17]. (c) DCP [4]. (d) Tarel et al.’s [19]. (e) Meng et al.’s [18]. (f) Our proposed method.
Fig. 21
Fig. 21 Comparison of different approaches on Visualization 1 and Visualization 2. (a) Input hazy frames. (b) DLCE [17]. (c) DCP [4]. (d) OCE [8]. (e) MRF [12]. (f) Our proposed method.
Fig. 22
Fig. 22 (a) The input video frame from one of the automated driver-assist systems. (b) The corresponding dehazed video frame of (a).
Fig. 23
Fig. 23 (a) The input video frame from the VIP Data set. (b) The corresponding dehazed video frame of (a).
Fig. 24
Fig. 24 (a) The input video frame from a random camcorder. (b) The corresponding dehazed video frame of (a).
Fig. 25
Fig. 25 (a) Detection result of the original video. (b) Detection result of video processed by the proposed method.

Tables (8)

Tables Icon

Table 1 Computer specification of NVIDIA Jetson TX1 and Renesas R-Car M2

Tables Icon

Table 2 Algorithm 1. The guided filter [9].

Tables Icon

Table 2 Processing performance comparison of TX1 and PC (Unit: fps)

Tables Icon

Table 4 Algorithm 2. The proposed algorithm for the sky estimation.

Tables Icon

Table 3 Processing performance comparison of NVIDIA Jetson TX1 and Renesas R-Car M2 (Unit: fps)

Tables Icon

Table 6 Algorithm 3. The rest part of the guided filter.

Tables Icon

Table 4 Comparison of the processing speed on Intel i7-6900K (Unit: fps)

Tables Icon

Table 8 Algorithm 4. The rest part of the guided filter.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

I ( x ) = J ( x ) t ( x ) + A ( 1 t ( x ) )
t ( x ) = e β d ( x )
J d a r k ( x ) = min c { r , g , b } ( min y Ω ( x ) J c ( y ) )
min y Ω ( x ) ( I c ( y ) ) = t ˜ ( x )   min y Ω ( x ) ( J c ( y ) ) + ( 1 t ˜ ( x ) ) A c
t ˜ ( x ) = 1 ω min c ( min y Ω ( x ) ( I c ( y ) A c ) )
J ( x ) = I ( x ) A max ( t ( x ) , t 0 ) + A
t ( x ) = 1 w ( x ) y Ω ( x ) G σ s ( x y ) G σ r ( | E x E y | ) t ˜ ( x )
w ( x ) = y Ω ( x ) G σ s ( x y ) G σ r ( | E x E y | )
A n e w = λ A t 1 + ( 1 λ ) A t
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.