In this study, a coaxial and multi-aperture polarization imaging system was designed to obtain dehazed images under bad weather conditions, which could effectively utilize polarization information. The system can capture four polarization azimuth images simultaneously by integrating four polarizers with different transmission directions in front of the sub-apertures. The system can utilize energy of incident light effectively, as well as remain the image resolution of the camera. In order to solve the image registration problem, a translation registration method for sub-aperture polarized images was implemented, which was based on the phase-only correlation algorithm. Moreover, a polarization dehazing model suitable for this system is employed to remove the haze effect of the captured images. Further experimental data demonstrated that the system could achieve a pleasing visual effect, and its performance on image quality was certified by several objective evaluation means including the line spread function and quantified detail information. The core advantages of this system are real-time and the good image quality in terms of contrast, details, and clarity. The system can effectively adapt to the long-distance, high-resolution, and real-time imaging in bad weather conditions such as haze weather.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Images of outdoor scenes are often degraded by bad weather conditions, and under these circumstances the atmospheric phenomena like fog and haze would degrade the visibility of the object scenes. Since in bad weather conditions the aerosol is misted by additional particles, in that case the reflected light is scattered. This process would cause image degradations because it could fade the colors and reduce the contrast of the observed objects, and such degraded images often lead to negative effects on environmental monitoring, remote sensing, target detection, etc. [1–3].
These years, restoring the degraded images by the image processing means has attracted increasing attention. Some methods are based on the consideration of removing haze effect from images, such as the dark channel prior way , the Retinex theory  and the homomorphic filtering way . However, some information of the image would be inevitably lost due to the bad weather conditions. Though, all the information of these images is very useful in dehazing applications [1,4,7]. Therefore, it is impossible to effectively and fully utilize more information to recover clear image. In addition, these methods may take a long time to remove haze effect in the actual applications [7,8]. At present, some researchers have been focused on employing polarization information to image through turbid medium [9–11]. Noteworthy, the polarization information has differences of features between the atmospheric scattered light and the target reflected light [10,12]. Therefore, more information in the polarized images can be utilized in the processing of polarization dehazing. In this field, researches have established an atmospheric dehazing model using the feature information that is obtained from polarization azimuth images [1,13,14]. The traditional method of obtaining polarization azimuth information is to rotate the polarizer in front of the lens [11,15]. However, manually rotating the polarizer may bring about a shake to the camera, and then result in the rotation errors and inaccurate polarization information. Besides, the images in a variety of scenes cannot be obtained effectively and in real time, which limits their wide applications . For that reason, researchers use a dichroic prism to obtain images at four different azimuth angles at the same time, which can eliminate the error triggered by manually rotating the polarizer . Unfortunately, the low light energy utilization makes the method hard to obtain valid images in low-light conditions. The reason is that a beam of incident light is split through the prism and then received by different detectors [18,19].
In this paper, a novel design of polarization dehazing system based on a coaxial and multi-aperture polarimetric camera was proposed. This system adopted the improved polarization dehazing algorithm and the sub-aperture polarized image registration technology to reconstruct clear images in real time. Compared with the traditional method, this system avoids rotating the polarizer and increases the utilization of light energy. Several groups of experiments were conducted to show the dehaze effects. Three objective evaluation methods were employed to evaluate the experimental results, i.e. the Mean Structural Similarity (MSSIM), the Line Spread Function (LSF) and the high-frequency information statistics, and the results certified that the dehazing system presented in our study could significantly remove the impacts of haze on images.
2. Polarization dehazing model
The degradation of image quality is a well-known problem, which is usually caused by poor atmospheric conditions such as hazy weather due to the light scattering effect. There would be unwanted scattering components that enter the imaging light path and then be received by the detectors. The information of scattered sunlight can be seen as a major source of the optical noise, which leads to the result of reducing the image clarity as shown in Fig. 1 . In such case, the intensity received by the detector is composed of target light and total airlight, which is shown in Eq. (1):1) can be explained separately using the attenuation model and the atmospheric light model shown in Fig. 2 [19,20].
According to Ref. , the information received by the detector can be divided into two parts, the atmospheric light and the attenuation light. By utilizing the polarization characteristics of light intensity, the atmospheric light can be estimated effectively . It is considered that the atmospheric light is partially polarized. So, the Degree of Linear Polarization (DoLP) information of the atmospheric light can be represented by Eq. (2):
The polarization characteristics of the imaging scene of the beam can be described by four measurable scalars . And in this study, the total airlight intensity image can be calculated directly using the captured image, which is implemented and shown in Eq. (5):6):
The sub-aperture images have pixels’ translation errors due to the displacement among four different detecting positions of the multi-aperture camera. Figures 4(a-d) show different sub-aperture polarized images taken simultaneously with the camera system shown in Fig. 4(h). The system is composed of four Point Grey’s GS3-U3-41C6C-C cameras, each with a resolution of 2048×2048 pixels. And the polarizers at 0°, 45°, 90°, and 135° (0° means that the transmission axis is parallel with the X-axis of image) are installed in front of the sub-camera lens, respectively. The brightness of these images from the four azimuths looks distinct, when the polarization azimuth of each image is different from others. In addition, the image deviations among the different sub-aperture polarized images could also be found in the direct observation of Figs. 4(a-d). The intensity image, shows in Fig. 4(e), is calculated from two orthogonal polarization azimuth images. Figure 4(f) is the DoLP result calculated by the information of Figs. 4(a-d). The DoLP information in this image has a ghosting effect, which has a great influence on the accuracy of the dehazing algorithm. Furthermore, the quantification of the reduced clarity after images superposition is shown in Fig. 4(g). The intensity curve at horizontal pixels from 360 row to 500 row and the vertical pixels from 264 column, represents the windows of the red building in the image. This horizontal line passes through the most noticeable building in Figs. 4(a) and 4(e), which provides a forceful demonstration of the reduction on image clarity for the superposition images. Therefore, solving the offset errors between pixels of sub-aperture polarized images is very necessary.
3. Sub-aperture polarized image registration model
The translation errors of corresponding pixels between two images can be corrected based on the phase-only correlation algorithm is implemented. As shown in Fig. 4(h), these sub-cameras are fixed in a steel frame, so there are no rotation and scaling errors existed among them. There is only the translation transformation among the captured images. In addition, polarized images are very sensitive to pixel errors, so a high accuracy translation registration algorithm is essential.
A sub-pixel matching algorithm based on the Fourier transform can be quickly operated with a small amount of calculation, without interference from polarization information and noise in haze images. This method mainly applies the Phase Only Correlation (POC) between two images . Then the method of curve fitting is performed using a plurality of points near the maximum POC value to finally acquire the actual maximum POC value. Then, accurate estimations of the sub-pixel translation can be achieved.
The translational motion of the signal in the time domain can be manifested by the phase change in the frequency domain while performing pixel registration between each pair of the four polarized images. So, the images are processed by the Fourier transform as Eq. (8):
Since the maximum value of the sinc function can be approximately expressed as a quadric surface, the quadratic surface fitting process based on the least square method can be applied . By fitting the phase correlation values with the surface near the relevant peak point, the parameters of pixel translation can be got. Then the information of coordinate displacement among the four polarization azimuth images can be obtained.
After obtaining the amount of image shift, align size and object pixel position of the multiple polarized images by cropping the edges of the image. By utilizing the information of these images, the clear image can be obtained without blur. Figure 5(a) is the sum of the errors for the four polarized images, and Fig. 5(b) shows the total errors after registration. It can be found with the experimental results that, most of the errors between each two polarized images are eliminated. In addition, we counted the pixels in the 939 row in these four images. The image before registration has a displacement, so the position of the maximum intensity of the light is different in Fig. 5(c). As obviously shown in Fig. 5(d), after the image registration process, the tendencies of the curve changes for the four polarized images are similar. Therefore, the results depict that, our registration method is fruitful, and more results will be shown in the next section.
4. Experiments and analysis
The coaxial and multi-aperture polarimetric camera is applied to collecting four images of different polarization azimuths simultaneously. The dehazed image can be obtained by inputting information of these images into the dehazing algorithm. Figure 6 shows the dehazing results of the images in Figs. 4(a-d) before and after registration.
Figure 6(a) illustrates the dehazed image, which is unregistered. It can be seen that, the pixel mismatching of the four-aperture polarimetric cameras causes the target blurring and degrades the clarity of the image. Moreover, noise is very serious in the sky region due to pixel mismatching errors. Figure 6(b) shows the image result after image registration, where the details of the selected region are clearly enhanced, and the color information in the scene is also restored better. From the subjective evaluation on the dehazed images, it can be found that the dehazing system can be well applied after registration. Table 1 shows the comparison of MSSIM values  between the two images before and after registration, which can reflect the structural similarity of the two images. When an image is blurred, the MSSIM values are low. As can be seen from the values in Table 1, the MSSIM values of Fig. 6(b) are much higher than those in Fig. 6(a). The experimental results can certify that the dehazed image of registration retain the original structure information to a large extent.
In addition, the Edge Spread Function (ESF) and LSF were utilized to evaluate the blurring degree of the reconstructed images. Using the high-precision Fermi function to fit the ESF of the edge region, the two-dimensional coordinate information was transformed into one-dimensional x, which represents the distance of the points in the edge area to the fitted straight line . The Fermi function in Eq. (11), used for the ESF fitting, is robust to the random noise. The LSF can be obtained by analytically differentiating the ESF function, as in Eq. (12) [25,26]:
In addition, the clear scene results of this dehazing system are demonstrated in Fig. 8. Figures 8(a) and 8(e) show the raw images. Figures 8(c) and 8(g) show the dehazed images after registration. It can be obviously seen that, the distant objects become clear, and the region of buildings in this image have better performance on contrast and details, compared to those in haze image. Moreover, the intensity images of red, green, and blue channels are shown in Figs. 8(b), 8(d), 8(f) and 8(h), which represent the intensity distribution of the reconstruction images respectively. In Figs. 8(b) and 8(f), pixel intensity distributions are in the range of approximately 150 (red and green channels value range from about 30 to 180, blue channel value range from about 40 to 200), which are relatively concentrated. The intensity distribution of the images after dehazing are shown in Figs. 8(d) and 8(h), which are about 1.7 times higher than before. Hence, dehazed images have better visual effects.
Besides, high-frequency information of images extracted by the Canny edge detection algorithm, is shown in Fig. 9 . The high-frequency information of the haze image is compared with its dehazed version under the same Canny parameters. As shown in Fig. 9, it can be intuitively seen that the detailed texture information processed by the dehazing system is more abundant. As shown in Fig. 9(b), the high-frequency information of the windows in the building is better restored compared with that in Fig. 9(a). Besides, the information of distant building and surface details in Fig. 9(d) can be reconstructed at the same time. The number of extracted pixels is employed to represent the richness of the details in the scene. The numerical values of the haze images and the dehazed images of the two scenes are 7511, 7425, 14536 and 38577 pixels respectively. The improvement of high-frequency information can make the target look more realistic, the details are raised, and it can also help to recovery the information of distant objects in the scene.
The coaxial and multi-aperture imaging system has an obvious advantage that, for both intuitive observation and objective evaluation of the intensity images, the processed images provide improved quality in terms of contrast, details, and clarity. These evaluation indicators agree well with the conclusions of the previous research results on subjective assessment.
This study presents a novel polarization dehazing system. With a coaxial and multi-aperture polarimetric camera that can capture the linearly polarized information simultaneously, the designed system can solve the problems such as the poor real-time performance on polarized image acquisition, low energy utilization and manual operation errors, thus enhancing its practicability under haze weather conditions. Along with the imaging system, the dehazing algorithm is improved to realize clear imaging of the scene without the need to acquire Imax and Imin, which enable the system free from the complex calculation process. Multiple images of different polarization azimuths can be corrected by using the sub-aperture polarized image registration model. Compared to the haze image, a dehazed image can be reconstructed with richer colors, higher contrast, approximately 3 times more detailed information and 1.5 times greater MSSIM values. These advantages make the system effectively adapt to the long-distance, high-resolution and real-time imaging in haze weather.
National Natural Science Foundation of China (NSFC) (61705175); Natural Science Foundation of Shaanxi Province (2018JQ6010); China Postdoctoral Science Foundation (2017M613063); the Fundamental Research Funds for the Central Universities (20103196515); the Innovation Fund of Xidian University (20109196769).
We thank Associate Professor Rui Gong from Xidian University for checking the English grammar of this paper. Thanks to Kui Yang from Xidian University for helping to acquire experimental data in this study.
1. Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “Polarization-based vision through haze,” Appl. Opt. 42(3), 511–525 (2003). [CrossRef]
2. R. L. Spellicy, “Optical Environmental Monitoring in Industry,” Opt. Photonics News 6(1), 22–26 (1995). [CrossRef]
3. A. Akula, R. Ghosh, S. Kumar, and H. K. Sardana, “Moving target detection in thermal infrared imagery using spatiotemporal information,” J. Opt. Soc. Am. A 30(8), 1492–1501 (2013). [CrossRef]
4. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]
5. G. Orsini and G. Ramponi, “A modified retinex for image contrast enhancement and dynamics control,” in Proceedings of IEEE Conference on International Conference on Image Processing (IEEE, 2003), pp. 393–396.
6. M. J. Seow and V. K. Asari, “Ratio rule and homomorphic filter for enhancement of digital colour image,” Neurocomputing 69(7-9), 954–958 (2006). [CrossRef]
7. J. Liang, L. Ren, H. Ju, W. Zhang, and E. Qu, “Polarimetric dehazing method for dense haze removal based on distribution analysis of angle of polarization,” Opt. Express 23(20), 26146–26157 (2015). [CrossRef]
8. F. Liu, L. Cao, X. Shao, P. Han, and X. Bin, “Polarimetric dehazing utilizing spatial frequency segregation of image,” Appl. Opt. 54(27), 8116–8122 (2015). [CrossRef]
9. X. Li, H. Hu, L. Zhao, H. Wang, Y. Yu, L. Wu, and T. Liu, “Polarimetric image recovery method combining histogram stretching for underwater imaging,” Sci. Rep. 8(1), 12430 (2018). [CrossRef]
10. X. Xiao, B. Javidi, G. Saavedra, M. Eismann, and M. Martinez-Corral, “Three-dimensional polarimetric computational integral imaging,” Opt. Express 20(14), 15481–15488 (2012). [CrossRef]
11. F. Liu, P. Han, Y. Wei, K. Yang, S. Huang, X. Li, G. Zhang, L. Bai, and X. Shao, “Deeply seeing through highly turbid water by active polarization imaging,” Opt. Lett. 43(20), 4903–4906 (2018). [CrossRef]
12. M. I. Mishchenko, Y. S. Yatskiv, V. K. Rosenbush, and G. Videen, Polarimetric Detection, Characterization, and Remote Sensing (Springer, 2011).
13. J. Mudge and M. Virgen, “Real time polarimetric dehazing,” Appl. Opt. 52(9), 1932–1938 (2013). [CrossRef]
14. L. Cao, X. Shao, F. Liu, and L. Wang, “Dehazing method through polarimetric imaging and multi-scale analysis,” Proc. SPIE 9501, 950111 (2015). [CrossRef]
15. Y. Y. Schechner and N. Karpel, “Recovery of underwater visiblilty and structure by polarization analysis,” IEEE J. Oceanic Eng. 30(3), 570–587 (2005). [CrossRef]
16. J. Liang, L. Ren, E. Qu, B. Hu, and Y. Wang, “Method for enhancing visibility of hazy images based on polarimetric imaging,” Photonics Res. 2(1), 38–44 (2014). [CrossRef]
17. W. Zhang, J. Liang, L. Ren, H. Ju, E. Qu, Z. Bai, Y. Tang, and Z. Wu, “Real-time image haze removal using an aperture-division polarimetric camera,” Appl. Opt. 56(4), 942–947 (2017). [CrossRef]
18. F. Chen, A. Henry, M. S. Brian, and R. A. Gonzalo, “Compressive spectral polarization imaging by a pixelized polarizer and colored patterned detector,” J. Opt. Soc. Am. A 32(11), 2178–2188 (2015). [CrossRef]
19. S. G. Narasimhan and S. K. Nayar, “Shedding light on the weather,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2003), pp. I-665–I-672.
20. S. G. Narasimhan and S. K. Nayar, “Vision and the atmosphere,” Int. J. Comput. Vis. 48(3), 233–254 (2002). [CrossRef]
21. M. Sulami, I. Glatzer, R. Fattal, and M. Werman, “Automatic recovery of the atmospheric light in hazy images,” International Conference on Computational Photography (IEEE, 2014), pp. 1–11.
22. C. M. Persons, D. B. Chenault, M. W. Jones, and C. A. Farlow, “Automated registration of polarimetric imagery using Fourier transform techniques,” Proc. SPIE 4819, 107–117 (2002). [CrossRef]
23. H. S. Stone, M. T. Orchard, E.-C. Chang, and S. A. Martucci, “A fast direct Fourier-based algorithm for subpixel registration of images,” IEEE Trans. Geosci. Remote Sensing 39(10), 2235–2243 (2001). [CrossRef]
24. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]
25. P. Han, F. Liu, K. Yang, J. Ma, J. Li, and X. Shao, “Active underwater descattering and image recovery,” Appl. Opt. 56(23), 6631–6638 (2017). [CrossRef]
26. P. T. Alexis and M. M. Jonathan, “Measurement of the modulation transfer function of infrared cameras,” Opt. Eng. 34(6), 1808–1817 (1995). [CrossRef]
27. J. Canny, “A Computional Approach to Edge Detection,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8(6), 679–698 (1986). [CrossRef]