1. Introduction

The main physical properties of light are the intensity, wavelength, coherence, and polarization. Among these, the human vision system can easily perceive intensity and wavelength information but cannot directly distinguish polarization. As physical properties such as those related to the object shape and surface roughness can be inferred from polarization information [1], it has been extensively applied in the fields of material classification [2], image-to-fog [3], 3D shape reconstruction [4], and biomedical imaging [5]. Polarization imaging systems can be classified as division of time (DoT) [6], division of amplitude, division of aperture, and division of focal plane (DoFP) systems [7]. The DoFP polarization imaging system has obvious advantages in terms of the system volume and imaging speed, which is the focus of research in the field of polarization imaging. Recently, newly released split-focal plane color polarizing cameras such as the FLIR BFS-U3-15S5PC and PHX050S cameras have produced more detailed and richer red-green-blue color polarization images than grayscale polarizing images. These cameras are equipped with an RGGB Bayer filter and directional polarization filter, which can form a 16-pixel computing unit as a color polarization filter array (CPFA), as shown in Fig. 1.

 

Fig. 1. Color focal plane polarized camera: (a) SONY IMX250MYRCMOS sensor, and RGGB Bayer filter and directional polarization filter. (b) Color polarization filter array (CPFA) with a 16-pixel computing unit.

Download Full Size | PDF

In the imaging process of a color focal plane polarization camera, noise interference from the outside world and the device itself, which generally includes noise from many different sources, is inevitable. As the noise characteristics tend to follow a Gaussian distribution when the number of noise sources increases, and the Gaussian noise model is easy to use in the design and analysis of the denoising algorithm [8], this study mainly examines the effect of Gaussian noise. The presence of noise degrades the image quality and produces spurious polarization information during interpolation reconstruction [9,10]. The color polarization interpolation algorithm [11–13] is crucial for obtaining full-resolution color polarization images. It mainly restores all the 12-channel $({({R,G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$ images from a CPFA image dataset. However, these interpolation algorithms are all applied on the premise of noiseless CPFA images. Although denoising can be performed using denoising methods for color images after interpolation, it is more difficult to remove noise because interpolation changes the noise characteristics. A similar problem exists in grayscale DoFP images obtained using a directional polarization filter. Existing gray DoFP image denoising algorithms include those based on principal component analysis (PCA) [14], polarimetric denoising residual dense networks (PDRDN) [15], and block-matching and 3-D filtering (DoFP-BM3D) [16]. The PCA denoising algorithm utilizes reduction and linear least mean square errors to denoise in the PCA domain. The algorithm sacrifices the image details and edge information and is unsuitable for considerable Gaussian noise. The PDRDN denoising algorithm is suitable for image restoration in complex and high-noise environments. The DoFP-BM3D denoising algorithm is a universal DoFP image denoising method that can suppress most of the noise in an image, although certain image edge details are sacrificed. All these algorithms, however, are based on the DoFP image sensor. The CPFA image sensor adds color channels and changes the original image array format, which means these algorithms cannot be directly applied to CPFA image denoising. Qiu et al. first considered the influence of noise in the reconstruction process of color polarization images [17] and proposed an algorithm for solving the Stokes vector of color polarization images by solving an inverse problem within the alternating direction method of multipliers framework. However, as the main purpose of the algorithm is to reconstruct the Stokes vector, its denoising ability is limited. Therefore, it is only suitable for CPFA image data under low-intensity noise. In this study, we first prove the correlation between the newly-added color channel and the polarization direction channel, and we then propose a CPFA image denoising algorithm, CPFA-BM3D, based on block-matching and 3-D filtering (BM3D) [18]. The proposed algorithm makes full use of the correlation between the four polarization directions and three-color channels. As shown in Fig. 1(b), based on the super pixel, it utilizes the nonlocal self-similarity of the CPFA image and the correlation of the polarization and color information for finding similar blocks to form 3D blocks and subsequently performs 3D transformation and collaborative filtering. The performance of the denoising method is evaluated using the degree of linear polarization (DoLP) images reconstructed from simulated and real CPFA images. The obtained results establish that the proposed algorithm significantly improves the visual effect, peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) [19].

The rest of the paper is organized as follows. In Section 2, we first prove the existence of correlation between the 12 channels in CPFA image data. We then introduce the block matching and 3D filtering (CPFA-BM3D) algorithm for CPFA image data and the algorithm parameter settings. In Section 3, we report our experimental results and conduct a comparison against competing denoising techniques in the literature. In Section 4, we make some concluding remarks about this work and future research.

2. Method

2.1 Correlation between color polarization channels

Interchannel correlation implies strong correlation between the high-frequency (texture or edge) components of different channels. The CPFA sensor contains not only different color channels (R, G, and B channels) but also different polarization channels (${0^ \circ },\,{45^ \circ },\,{90^ \circ }$, and ${135^ \circ }$ channels), which are arranged according to certain rules. By fully utilizing the correlation between these channels, excellent results can be achieved when the BM3D algorithm is applied in CPFA image data denoising.

We used the color polarization dataset in [17] as the test set. This dataset contains 40 carefully calibrated polarization scenes, each containing four RGB color images in the polarization directions of ${0^ \circ },\,{45^ \circ },\,{90^ \circ }$, and, respectively (detailed in Section 3.1.1). We selected 12 channels $({({R,G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$ in the center region of the image, which excluded the 15 outermost pixels, to avoid the boundary effect induced by the registration step used on the raw images. Subsequently, the Pearson correlation coefficient (PCC) [20] was used to evaluate the correlation between channels, and the 40 scenarios in the dataset were averaged, as formulated by Eq. (1):

Figure 2 shows the PCC results among 12 channels of the color polarization image. In the case of the R channel, the correlation between the four channels with different polarization (${R_{{0^ \circ }}},\,{R_{{{45}^ \circ }}},{R_{{{90}^ \circ }}},\,{R_{{{135}^ \circ }}}$) is higher than those between these four channels and the other eight channels $({({G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$. In the case of the B or G color channel, considering the ${G_{{0^ \circ }}}$ channel as an example $PCC({G_{{0^ \circ }}},{B_{{0^ \circ }}})\,>\,PCC({G_{{0^ \circ }}},{G_{{{45}^ \circ }}})\,>\,PCC({G_{{0^ \circ }}},{B_{{{45}^ \circ }}})\,>\,PCC({G_{{0^ \circ }}},{G_{{{135}^ \circ }}})\,> PCC({G_{{0^ \circ }}},{B_{{{135}^ \circ }}})\,>\,PCC({G_{{0^ \circ }}},{G_{{{90}^ \circ }}})\,>\,PCC({G_{{0^ \circ }}},{B_{{{90}^ \circ }}})$ shows that the correlation is maximum when the polarization channels are the same ($PCC({G_{{0^ \circ }}},{B_{{0^ \circ }}})$ is maximum). When the polarization channels differ, the correlation between channels of the same color is greater than that between channels of different color (e.g., $PCC({G_{{0^ \circ }}},{G_{{{45}^ \circ }}}) > PCC({G_{{0^ \circ }}},{B_{{{45}^ \circ }}})$). In general, for the 12 channels, the least PCC is $PCC({R_{{0^ \circ }}},{G_{{{90}^ \circ }}})$=0.9738, and the average PCC is 0.9916 (greater than 0.8 is correlated), indicating that the 12 channels have high correlation, and the correlation between channels can be fully utilized in CPFA image denoising.

 

Fig. 2. Correlation between 12 channels in the color polarization image.

Download Full Size | PDF

2.2 CPFA - BM3D algorithm

The CPFA image denoising algorithm based on BM3D (CPFA-BM3D) is a 3D transform domain filtering algorithm for denoising, utilizing the similarity between image blocks and the correlation between channels. It includes two main steps: basic estimation and final estimation. Each step includes image block grouping, 3D collaborative filtering, and aggregation, where 3D collaborative filtering uses hard-threshold filtering and Wiener filtering. The denoising process is illustrated in Fig. 3.

 

Fig. 3. CPFA-BM3D denoising algorithm flow chart.

Download Full Size | PDF

2.2.1 Basic estimation

We first divide the noisy CPFA image into reference blocks ${Z_{XR}}$ sized $N_1^{ht} \times N_1^{ht}$ in suitable steps. In natural images, there is a correlation between pixels of each reference block. The presence of strong correlation between the color polarization channels has been proven, so a certain correlation between the pixels of each reference block of the CPFA image is also expected. To take full advantage of the polarization information and the correlation between different channels, each reference block includes pixels of at least four polarization channels and three color channels $({({R,G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$, as shown in Fig. 1(b).

Further, we search for the number of matching blocks in search range ${n^{ht}} \times {n^{ht}}$.When the distance between the matching block and reference block is less than the fixed threshold $\tau _{match}^{ht}$, the matching block is considered as a similar block ${Z_{Xi}}$, and the set of ${Z_{Xi}}$ is $S_{XR}^{ht}$. The ${l^2}$ norm is used for calculating the similarity, as depicted in Eq. (2):

When blocks with different colors and different polarization orientation configurations are combined and thresholded (for example, as shown on the left of Fig. 4(a), when blocks on the upper-left corner of the ${R_{{{90}^ \circ }}}$ sample are combined with blocks on the upper-left corner of the ${G_{{0^ \circ }}}$ sample or of the ${B_{{{90}^ \circ }}}$ sample), severe checkerboard artifacts are observed in regions with low chromatic aberration and less polarization information in the denoised DoLP color image, as shown on the right of Fig. 4(a). This is because when finding similar blocks, two image blocks of different arrays with few differences in the color and polarization information have similar pixel values, resulting in similar values of the ${l^2}$ distance. These are judged as similar blocks, causing a deviation in the reference block estimate. To solve this problem, we limit the grouping to configuration blocks with the same color and polarization direction (when the block on the upper-left corner of the ${R_{{{90}^ \circ }}}$ sample is combined with that on the upper-left corner of the ${R_{{{90}^ \circ }}}$ sample), as shown on the left of Fig. 4(b). From the image on the right of Fig. 4(b), when the array configuration of similar blocks is considered to correspond to the color as well as polarization channels, the denoised DoLP color images are of better quality and retain more image details and polarization information.

 

Fig. 4. (a) (left) Groups of blocks with different colors and different polarization channel configurations and (right) denoised DoLP images ($\sigma \textrm{ = }20$). (b) (left) Groups of blocks with the same color and the same polarization channel configurations and (right) denoised DoLP images ($\sigma \textrm{ = }20$).

Download Full Size | PDF

All similar blocks in $S_{XR}^{ht}$ are sorted in descending order of similarity, and the top ${M^{ht}}$ similar blocks are used to build the similarity group ${Z_{S_{XR}^{ht}}}$.

Collaborative filtering in basis estimation is noise attenuation by hard thresholding in the 3D transform domain. The applied 3D linear transformation is called $\tau _{3D}^{ht}$, which is a portable 3D transformation. Wavelet transform or DCT 2D transform is performed first, followed by 1D Hadamard transform. The formula for collaborative hard-threshold filtering is as follows:

When collaborative filtering is applied to ${Z_{S_{XR}^{ht}}}$ composed of blocks with the same color and channel configuration, 3D transformation and noise removal is realized more effectively. This is because ${Z_{S_{XR}^{ht}}}$ has two unique advantages:

Collaborative filtering in basic estimation and final estimation utilizes these two types of correlation to generate a sparse representation of more real signals in similar block groups. Such sparse representation is highly effective in attenuating noise, retaining the signal characteristics, and realizing a more accurate filtering effect. Here, it is noted that filtering is applied to a 3D group containing all the polarization information and color information samples. Therefore, our method takes advantage of the correlation between all the channels.

Because multiple estimates can be found for the same pixel in each similar group, we assign smaller weights to noisy block estimates, and the basic estimate of each reference block is $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\frown$}} \over Y} _{{x_m}}^{ht,XR}$.

Finally, the basic estimation image is obtained by clustering. The aggregation process can be expressed as follows:

2.2.2 Final estimation

In the final estimation, the original noisy image is Wiener-filtered, mainly based on the basic- estimate image. This step is primarily applied to restore additional image details and improve the denoising performance of the algorithm. The parameters in the final estimation are denoted by superscript “wie”. Similar blocks are grouped in the estimated image and the noise image. The specific grouping method is the same as that used in basic estimation. We assume a set of two similar blocks: $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\frown$}} \over Y} _{S_{XR}^{wie}}^{basic}$ and $S_{XR}^{wie}$.

The contraction coefficient of the Wiener filter is defined by the energy spectrum of the 3D transformation coefficient of the basic estimation group:

All the similar blocks in $S_{XR}^{wie}$ are sorted in the order of similarity, and the 3D group ${Z_{S_{XR}^{wie}}}$ is constructed with the first ${M^{wie}}$ similar blocks in the sort. Noise data are processed using 3D transformation and Wiener coefficient ${W_{S_{XR}^{wie}}}$, followed by 3D inverse transformation to obtain the estimated value of the set of similar image blocks:

The final estimate is obtained as the weighted average of different estimates of the same reference block:

2.3 Parameter setting

Parameter selection generally affects the performance of an algorithm. Because the correlation of the 12 channels in the CPFA image imposes certain restrictions on the parameter setting, we studied the parameter setting of the CPFA-BM3D algorithm comprehensively considering the algorithm performance, running time, correlation, and so on. The parameters in the basic estimation step are represented by superscript “ht”, whereas those in the final estimation step are represented by superscript “wie” as previously mentioned. The parameter set for the basic and final estimation, respectively, are:

After numerous experiments, we found that several parameters, namely, ${n^{ht}},\,{n^{wie}},\,\tau _{match}^{ht}$, and $\tau _{match}^{wie}$, had negligible effect on the result. Therefore, the following parameter values were fixed and used throughout the study: ${n^{ht}} = 39,\,{n^{wie}} = 39,\,\tau _{match}^{ht} = 2500$, and $\tau _{match}^{wie} = 400$.

We tested all 40 scene images in the test set. To quantitatively describe the denoising results, the PSNR and SSIM (detailed in Section 3.1.2) were used as the two evaluation indices. We compared five groups of images, including the Stokes vectors (S0, S1, S2), DoLP, and the angle of polarization (AoP) obtained before and after CPFA image denoising. The results showed that the parameter setting results for the five groups of compared images in the 40 scenes were similar. For simplicity, we consider the result of the DoLP image of a plant scene as an example to illustrate the parameter setting. In addition, when other parameter values are not given, we use the optimal solution for those parameters.

2.3.1 Influence of $N_1^{ht}$ and $N_1^{wie}$

To maintain the integrity of the color polarization array in the reference block, and considering the correlation between the color polarization channels, the selection of the reference block has certain requirements, owing to the unique structure of CPFA image data. When $N_1^{ht} = 2$ or $N_1^{wie} = 2$, the denoising effect is very poor because all the channels are not selected in the reference block, and the collaborative filtering effect is poor because the reference block is very small. Therefore, $N_1^{ht},N_1^{wie} \ge 4$ were set. For a low noise standard deviation, the reference block must be relatively small to preserve image details well. However, for a larger noise standard deviation, the denoising effect is better when the reference block is larger as most of the image details are lost to noise.

As shown in Table 1, when $\sigma = 30$ and the other denoising algorithms have the same parameters, the reference block size can result in a difference of 1.625 dB in the PSNR and 0.053 dB in the SSIM. Simultaneously, the size of the reference block affects the execution speed of the algorithm. The smaller the reference block, the faster is the algorithm. For $\sigma \le 20$ and $N_1^{ht} = 8,\,N_1^{wie} = 8$ and $N_1^{wie} = 12$ are very close. Therefore, we selected $N_1^{wie} = 8$ for its faster execution time.

Table 1. Influence of $N_1^{ht}$ and $N_1^{wie}$

View Table | View all tables in this article

2.3.2 Influence of the ${M^{ht}}$ and ${M^{wie}}$

As shown in Table 2, when the noise is relatively low, the maximum number of similar blocks has negligible influence on the denoising result, whereas it has a significant influence when the noise is relatively high. The denoising effect is best when ${M^{ht}}\textrm{ = }32$ and ${M^{wie}}\textrm{ = }64$.

Table 2. Influence of ${M^{ht}}$ and ${M^{wie}}$.

View Table | View all tables in this article

2.3.3 Influence of ${p^{ht}}$ and ${p^{wie}}$

To increase the processing speed, a circular selection of reference image blocks is performed with step size p (integer) in rows and columns. For example, with $p = 1$, the processing is theoretically nine times faster than with $p = 3$. In addition, because the final estimation is based on the basic estimate that assumes no noise or at least low noise, the step-size of the final estimate can be larger than that of the basic. From Table 3, ${p^{ht}} = 3$ and ${p^{wie}} = 5$ gives the best result.

Table 3. Influence of ${p^{ht}}$ and ${p^{wie}}$.

View Table | View all tables in this article

To increase the processing speed, a circular selection of reference image blocks is performed with step size p (integer) in rows and columns. For example, with $p = 1$, the processing is theoretically nine times faster than with $p = 3$. In addition, because the final estimation is based on the basic estimate that assumes no noise or at least low noise, the step-size of the final estimate can be larger than that of the basic. From Table 3, ${p^{ht}} = 3$ and ${p^{wie}} = 5$ gives the best result.

3. Experimental results and discussion

3.1 Experimental preparation

3.1.1 Simulated and real CPFA images

In this study, the color polarization dataset in Ref. [17], containing 40 carefully calibrated polarization scenes obtained using a DoT polarization imaging system, was used as the test dataset. The DoT polarization imaging system changes the wavelength and polarization angle of light by rotating the spectral filter and polarization filter multiple times for combining different spectral and polarization filters to obtain the color polarization images. This system can directly obtain each pixel including 12 values $({({R,G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$ full-resolution image, mainly used for verification of the principle of polarization imaging. For each scene, there are four groups of raw images at polarizations of ${0^ \circ },\,{45^ \circ },\,{90^ \circ }$, and ${135^ \circ }$ with the same exposure time. Each group consists of 100 images acquired continuously under the same situation, averaged to suppress noise. The resolution of each image is 1024 ${\times} $1024 pixel and the depth are 8 bits. To construct the simulated CPFA images, the images of 12 channels $({({R,G,B} )\ast ({{0^ \circ },{{45}^ \circ },{{90}^ \circ },{{135}^ \circ }} )} )$ were down-sampled, and simulated noiseless CPFA images were generated according to the array mode shown in Fig. 1(b), whose sizes were equal to those of the four color-images. Further, Gaussian white noise was added to the simulated noiseless CPFA images, which were then denoised through means filtering, median filtering, Wiener filtering, wavelet threshold filtering [21], the BM3D algorithm, the DoFP-BM3D algorithm, the algorithm proposed by Qiu et al. (hereafter called Qiu’s method), and our proposed CPFA-BM3D algorithm. The BM3D algorithm extracts the images of the 12 channels from the CPFA image data, denoises the images of each channel separately, and then synthesize the denoised images into CPFA image data. The DoFP-BM3D algorithm extracts the images of each color channel in the CPFA image to synthesize four (R, G, G, B) DoFP images, denoising them separately, and then synthesizes the denoised subimages into CPFA image data. The other denoising algorithms are directly applied to the CPFA image data. Among them, mean filtering, median filtering, and Wiener filtering adopt a 3*3 window. Wavelet threshold filtering, the BM3D algorithm, and Qiu’s method employ algorithms with source code provided in references [21], [18], and [17], respectively. The DoFP-BM3D algorithm adopts the parameters in Ref. [16]. Finally, the denoising performance was evaluated based on the color PSNR (average value of the RGB channels), SSIM, and visual comparison.

In addition, we used a Lucid Vision Lab PHX050S-Q color polarization camera to record real CPFA images. The $2048 \times 2448$ pixel SONY IMX250MYR CMOS color polarization sensor was used, where each pixel was $3.45\mu m \times 3.45\mu m$, and the depth was 8-bit.

3.1.2 Evaluation indices

The PSNR is defined as follows:

Here, $m \times n$ is the size of the image, and $I({i,j} )$ and $K({i,j} )$ represent the simulated real image and the denoised image, respectively.

The SSIM measures the image similarity with respect to three aspects: brightness, contrast, and structure. It is defined as follows:

3.2 Experimental results

In this section, the simulated images are compared in terms of the PSNR, SSIM, and visually, whereas the real images are only compared visually. Considering the denoising effect and computational complexity, the algorithm parameters were set as follows: $N_1^{ht} = 8$, $N_1^{wie} = 8,\,{M^{ht}} = 32,\,{M^{wie}} = 64,\,{p^{ht}} = 3,\,{p^{wie}} = 5,\,{n^{ht}} = 39,\,{n^{wie}} = 39,\,\tau _{match}^{ht} = 2500$, and $\tau _{match}^{wie} = 400$.

3.2.1 Results of the simulated CPFA images

We added Gaussian white noise ($\sigma \textrm{ = 20}$) to the simulated CPFA images and denoised them using the different denoising methods mentioned in Section 3.1.1. The edge-aware residual interpolation [12] color polarization interpolation algorithm was used to interpolate the denoised simulated CPFA images to generate the color images linearly polarized in four directions. The sizes of these linearly polarized color images were equal to those of the simulated CPFA images. The DoLP color image calculated from the original color polarization images with four different polarization directions in the dataset was used as the reference image. The DoLP color image calculated from the four interpolated images was compared with the reference image. The denoising results of the DoLP color images of the “plant” scene in the test set are shown in Fig. 5.

 

Fig. 5. Simulated CPFA image denoising results: (a) original images, (b) nonuniformity corrected images, (c) average images, (d) median filtering, (e) Wiener filtering, (f) wavelet threshold filtering, (g) Qiu’s method, (h) BM3D, (i) DoFP-BM3D, and (j) our method.

Download Full Size | PDF

Figure 5(a1) displays the DoLP reference image, and Fig. 5(b1) displays the DoLP image obtained by interpolating the simulated CPFA image with Gaussian noise (σ=20). Figures 5(c1)–(j1), respectively, show the results of mean filtering, median filtering, Wiener filtering, wavelet threshold filtering, Qiu’s method, BM3D algorithm, DoFP-BM3D algorithm, and the DoLP image obtained by interpolation calculation after denoising the simulated CPFA image using the CPFA-BM3D algorithm. Figures 5(a2)–(j2) display the partial enlargement of the green box in the respective DoLP color image.

As shown in Figs. 5(c1)–(f1), after denoising the simulated CPFA image using mean filtering, median filtering, Wiener filtering, and wavelet threshold filtering, the noise in the DoLP color image is partly reduced compared to the DoLP color image with noise (Fig. 5(b1)). However, some of the areas in the image are distorted because the residual noise in the denoised image is amplified by the Stokes parameter calculation. In addition, in Figs. 5(c2)–(f2), the texture and detail information of the leaves and flower-pots are severely corrupted by noise, affecting the quality of the DoLP color images. Figures 5(g1)–(g2) indicate that when $\sigma $ is large ($\sigma \textrm{ = 20}$), the DoLP color image after denoising by Qiu’s method is filled with noise, and there is no effective polarization information in the image. From Figs. 5(h1)–(j1), the image quality of the DoLP color images denoised using the BM3D, DoFP-BM3D, and CPFA-BM3D algorithms is significantly improved, and the difference in polarization between the leaves and flowerpots can be observed. However, in Fig. 5(h2), the leaf-edge of the DoLP color image denoised by the BM3D algorithm has obvious distortion, and checkerboard artifacts can be observed on the flowerpot. As shown in Fig. 5(i2), although the DoFP-BM3D denoising algorithm reduces distortion of the leaf-edges, checkerboard artifacts are present in the leaf and flowerpot because only the polarization information is utilized in this method and color correlation is not applied. As shown in Fig. 5(j2), compared to the other algorithms, the denoised DoLP color image using our proposed CPFA-BM3D algorithm removes noise more thoroughly, preserves the edge details of the leaf and flowerpot in the enlarged region more completely, and reduces the error in the DoLP color image. Therefore, in terms of the visual effects, the denoising effect of our algorithm is significantly better than those of the other denoising algorithms.

To verify the denoising effect of different algorithms at different noise levels, Gaussian white noise with different standard deviations was added to the simulated CPFA image “plant”. The PSNR and SSIM of the DoLP color image after denoising with different algorithms are plotted in Fig. 6.

 

Fig. 6. PSNR (left) and SSIM (right) under different noise standard deviations.

Download Full Size | PDF

From Fig. 6, under small standard deviation ($\sigma < 10$), the difference in the PSNR and SSIM of the DoLP color image before and after denoising are negligible, and the denoising effect of each algorithm is almost the same. However, our denoising algorithm is always the best. When the standard deviation of noise increases, the PSNR and SSIM of the DoLP color image denoised by our CPFA-BM3D algorithm remain superior to those of the other algorithms. The advantage of our algorithm over the other algorithms becomes increasingly obvious with the increase in the standard deviation of noise. Moreover, the PSNR and SSIM of the image after denoising by our algorithm are always greater than 32 dB and 0.85, respectively, indicating that our algorithm is effective for different noise levels, has strong adaptability, and a more obvious suppression effect on excessive noise.

The polarization information of color-polarized images is generally embodied in S0, S1, S2, the DoLP, and AoP. Therefore, the performance of the denoising algorithm can be evaluated using these five types of images. We denoised all 40 simulated CPFA images in the test set ($\sigma \textrm{ = 20}$) and averaged the denoising results of the 40 sets of scenes. The PSNR and SSIM values for the five images are listed in Table 4.

Table 4. Mean PSNR and mean SSIM of the S0, S1, S2, DoLP, and AoP denoising results on the test set ($\sigma \textrm{ = 20}$).

View Table | View all tables in this article

Comparing the values of the PSNR and SSIM in Table 4, it is obvious that our algorithm outperforms the other algorithms. For S1 and S2 images, our algorithm improves the PSNR and SSIM by approximately 4 dB and 0.2, respectively, compared to the other algorithms. Compared to the noisy images, the mean value of the PSNR of DoLP color images after denoising can be increased to 200%, whereas that of AoP color images after denoising can be increased to 160%. The mean SSIM of DoLP color images is increased to 400%. The above results show that our algorithm is highly effective in preserving the polarization information of color-polarized images.

3.2.2 Results of real CPFA images

For real CPFA images, correction is first performed for nonuniformity [22], after which the noise estimation algorithm [23] is used to estimate the noise of sub-images with different polarization directions and different color channels to obtain the standard deviation of the sub-image noise. Finally, the image is denoised and interpolated to obtain the computed DoLP color image.

Figure 7(a1) shows the DoLP color image obtained by interpolation calculation of a real CPFA image, and Fig. 7(b1) shows the result of nonuniformity correction. Figures 7(c1)–(j1) depict the DoLP color image after denoising a real CPFA image, and Figs. 7(a2)–(j2) show the partial enlargement of the green box in the respective DoLP color image. In Fig. 7(a1), there are severe inhomogeneities in the contours of the sculptures and buildings in the image, and there is considerable noise. As shown in Fig. 7(b1), the image quality after nonuniformity correction is significantly improved, but considerable noise is present, resulting in no significant difference in the polarization of the object. Figures 7(c1)–(f1) indicate that the image quality of the DoLP color image after denoising by mean filtering, median filtering, Wiener filtering, and wavelet threshold filtering, respectively, is improved. However, some image edge details are lost, resulting in excessive smoothness. Figures 7(h2) and 7(i2) show that the BM3D and DoFP-BM3D algorithms can remove part of the noise, but there is still some noise left in the image and there is severe loss of polarization information at the railings. Figures 7(g2) and 7(j2) demonstrate that our CPFA-BM3D algorithm and Qiu’s algorithm have better results overall than the other algorithms. However, our CPFA-BM3D algorithm is clearly better with respect to the detailed processing of the sculpture edges compared to Qiu’s algorithm. The polarization difference of the sculptures in the enlarged region is more obvious. The polarization information of the DoLP color images is retained, whereas the DoLP images processed by Qiu’s algorithm are smooth and lack details. Moreover, the denoising results of the simulated and real CPFA images indicate that Qiu’s algorithm is unsuitable for high-intensity noise. Therefore, our algorithm is more suitable for real CPFA-image denoising compared to the other algorithms.

 

Fig. 7. Real CPFA-image denoising results: (a) original images, (b) nonuniformity corrected images, (c) average images, (d) median filtering, (e) wiener filtering, (f) wavelet threshold filtering, (g) Qiu’s method, (h) BM3D, (i) DoFP-BM3D, and (j) our method.

Download Full Size | PDF

4. Conclusion

In this study, we proved the existence of correlation between 12 CPFA image data channels and proposed a BM3D-based denoising method for CPFA image data (CPFA-BM3D). This method makes full use of the correlation between different polarization channels and different color channels to limit the search for similar blocks and groups blocks corresponding to the same color channel and polarization channel to reduce chromatic aberration and checkerboard artifacts. Subsequently, 3D transformation and filtering were performed for similar block groups. Application of the correlation between different channels adds sparse representation of the real signals of similar block groups in the transform domain, further improving the filtering effect. Finally, the filtering results were aggregated, and the CPFA image was restored. The denoising performance of the proposed algorithm was evaluated using simulated and real CPFA images. The obtained results demonstrated that the proposed algorithm was superior to the existing algorithms in terms of the visual effect, PSNR, and SSIM, and that it could retain edge details and texture information. Moreover, the real CPFA image quality was obviously improved after processing with the proposed algorithm. The PSNR and SSIM of the DoLP color images obtained by interpolation of CPFA images processed by the proposed algorithm are increased to 200% and 400% of their original values, respectively. However, the detailed processing of this method is not ideal for non-Gaussian noise. Therefore, we intend to study more general CPFA image denoising methods and CPFA image interpolation denoising methods in the future.