Polarization image demosaicking using polarization channel difference prior

Rongyuan Wu; Rongyuan Wu; Yongqiang Zhao; Yongqiang Zhao; Ning Li; Ning Li; Seong G. Kong

doi:10.1364/OE.424457

1. Introduction

1.1 Background

Polarization imaging provides useful information in the perception of objects in challenging environments. Polarization imaging inspired by the polarization vision of animals [1], has been applied to road detection [2], reflection removal [3], object segmentation [4] and image dehazing [5]. Division of focal plane (DoFP) polarization camera has become a mainstream imaging technology among various polarization imaging systems. To perceive polarization states, DoFP cameras integrate a micro-polarizer array with four polarization direction (0$^{\circ }$, 45$^{\circ }$, 90$^{\circ }$, 135$^{\circ }$) units into the super-pixels of the focal plane array sensor, as shown in Fig. 1 (mosaic pattern). It leads to getting the snapshot acquisition easily. 0$^{\circ }$ pixel only records the intensity of light coming from the 0$^{\circ }$ polarizer. DoFP polarization imagers make a trade-off between spatial resolution and sensing speed. Every pixel of DoFP only records one channel of raw image data, while four polarization components are needed to describe polarization states completely. This means that we may lose three-quarters of the polarization information. Polarization demosaicking recovers missing components to restore complete full-resolution polarization images in four polarization orientations.

Fig. 1. Polarization filter array imaging overview and polarization visualization.

Download Full Size | PDF

Despite successful demosaicking algorithms [6–10] for color images have been proposed, they may not be suitable for polarization image demosaicking due to the inherent difference of the filter arrays between color filter array (CFA) and polarization filter array (PFA). In the case of color images, demosaicking is implemented among three color channels (red, green, blue). As the human visual system is more sensitive to the green component, green filters are twice as many as the other two channels in the Bayer pattern [11]. A PFA is composed of pixel-size linear polarizers oriented in four different angles periodically. For polarization demosaicking, interpolation methods such as bilinear, bicubic and gradient-based algorithms have been proposed [12,13]. The reconstructed images by interpolation-based methods are often lack of the high frequency information, which leads to the generation of artifacts in intensity images and the reconstruction of inaccurate polarization information. Li et al. [14] estimate interpolation errors with Newton’s polynomial in the polarization difference domain and alleviate this problem partly. Mihoubi et al. [15] transfer pseudo panchromatic image difference method [16] proposed for multispectral demosaicking to PFA demosaicking and get great results. Recently, some methods based on residual interpolation are proposed. Ahmed et al. [17] first apply a guided filter [18] to interpolate in a "residual" domain. Morimatsu et al. [19] propose an edge-aware residual interpolation, which incorporates a new edge detector to generate a guide image for better channel restoration. Deep learning has been widely applied to low-level computer vision tasks due to the powerful representation ability of convolutional neural networks (CNN) [20]. Zhang et al. [21] propose a network that learns an end-to-end mapping between the PFA mosaic images and full-resolution ones with skip connections [22]. Zeng et al. [23] propose an end-to-end framework to restore Stokes images directly. Besides requiring expensive computing resources, a huge amount of data is also needed to train the CNN. The previous polarization demosaicking methods often first detect potential edges and then determine the interpolation direction, while accurate edge detection requires a lot of computational costs, which means they cannot get a good trade-off between high-quality recovery and time consumption. A faithful PFA demosaicking algorithm should enable obtaining good quality images with low hardware cost, which is desirable in the DoFP camera industry, as current camera resolution can be very high.

In most cross-edge regions, the difference between the orthogonal channels changes drastically than the non-orthogonal cases. We theoretically prove the universality of this effect and apply it to augment polarization demosaicking. Most demosaicking methods perform interpolation in the difference domain due to the fewer high frequency information in the difference domain [14]. Their ability to recover edges is up to the result of the fusion of recovered channel differences in different directions. This paper presents a polarization demosaicking technique using the polarization channel difference prior to obtain a satisfactory trade-off between accuracy and speed. The differences between two polarization channels have such a relationship: the difference between the orthogonal channels changes more drastically than the non-orthogonal cases. Thus the recovered results from non-orthogonal channels are more accurate than those from orthogonal channels. To reconstruct the 0$^{\circ }$ channel, for example, higher weights should be assigned to the results from the 0$^{\circ }$-45$^{\circ }$ channel difference and 0$^{\circ }$-135$^{\circ }$ channel difference than that from the 0$^{\circ }$-90$^{\circ }$ channel difference. The proposed scheme does not include any detection of the edge direction. Therefore, our method is suitable for implementations on graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) to achieve fast and high-precision polarization information recovery in high-resolution polarization digital cameras.

1.2 Linear polarization calculation

The first three Stokes polarization parameters are defined as: [24]

(1a)$$ S_{0} = 0.5(I_{0}+I_{45}+I_{90}+I_{135}) {,} $$

(1b)$$ S_{1} = I_{0}-I_{90} {,}$$

(1c)$$ S_{2} = I_{45}-I_{135} {,}$$

where $I_{0}$, $I_{45}$, $I_{90}$ and $I_{135}$ are captured intensity images at four polarization orientations (0$^{\circ }$, 45$^{\circ }$, 90$^{\circ }$, 135$^{\circ }$), respectively. The first Stokes parameter $S_{0}$ is the total intensity of the light. $S_{1}$ and $S_{2}$ describe the linear polarization states. To observe polarization, two polarization properties are of most interest, degree of linear polarization (DoLP) and angle of linear polarization (AoLP):

(2a)$$ DoLP = \frac{\sqrt{S_{1}^{2}+S_{2}^{2}}}{S_{0}} {,}$$

(2b)$$ AoLP = 0.5\arctan \left( \frac{S_{2}}{S_{1}} \right) {,}$$

Figure 1 illustrates the outline of polarization demosaicking and polarization visualization. Polarization information reflects the stress distribution of the ruler.

2. Polarization channel difference prior (PCDP)

2.1 Across edges analysis

The polarization channel difference prior is based on the following observation on the polarization images: the inter-channel differences show different changes when across the edges. We select four typical edge cases from the color-polarization dataset [25] (here we only use the green channel for simplicity). Taking the difference between the $I_{0}$ channel and other channels as an example, we record the changes of the pixel values across the edges on the three difference images. Figure 2 shows the fluctuations of channel differences when across edges, where the counted pixels are marked with green, orange and blue lines, respectively. To facilitate comparison, we set the starting points of the three lines to the same position. We can see that the difference between $I_{0}$ and $I_{90}$ changes most drastically when across the edges. The same experiments are performed on other color channels (red channel and blue channel) and similar results are obtained: in most cases, the difference between orthogonal channels ($I_{0}$ and $I_{90}$, $I_{45}$ and $I_{135}$) varies more sharply than that between non-orthogonal channels when across edges. In Fig. 2, scenes (a) – (c) are illuminated by unpolarized light and scene (d) is illuminated by polarized light. Figures 2 (a) – (c) show that the channel differences are very close in smooth areas with low DoLP. As scene (d) is illuminated by polarized light, there is a significant difference among $I_{45}$, $I_{90}$ and $I_{135}$, which causes the curve at the right end to be different from the scenes illuminated by unpolarized light.

Fig. 2. Fluctuations of channel differences when across edges. Scenes (a) – (c) are illuminated by unpolarized light and scene (d) is illuminated by polarized light.

Download Full Size | PDF

2.2 Theoretical proof

The light captured by DoFP sensors can be represented with a superposition of unpolarized light and completely polarized light [24]. We consider the completely polarized light as linear polarized light because we cannot get a circular polarization component from DoFP sensors. Assuming that the intensity of light is $I$, the linear polarized light component is $I_{linear}$ and the unpolarized light component is $I_{unpolar}$. The relationship between them can be described as:

(3)$$I = I_{linear}+I_{unpolar} {,}$$

With Malus’s Law [24], the four intensity measurements $I_{\theta }$ of the light filtered by linear polarizer oriented at $\theta$ ($\theta$ = 0, 45, 90, 135 degree) from the DoFP sensor are defined as:

(4a)$$ I_{0} = I_{linear}\cos^{2}\varphi + I_{unpolar}/2 {,} $$

(4b)$$ I_{45} = I_{linear}\cos^{2}(\pi/4-\varphi) + I_{unpolar}/2 {,} $$

(4c)$$ I_{90} = I_{linear}\cos^{2}(\pi/2-\varphi) + I_{unpolar}/2 {,} $$

(4d)$$ I_{135} = I_{linear}\cos^{2}(3\pi/4-\varphi) + I_{unpolar}/2 {,} $$

where $\varphi$ is AoLP ($\varphi \in (0,\pi ]$). Let’s take the 0$^{\circ }$ channel as an example and analyze the correlation of differences between the 0$^{\circ }$ channel and the other three channels. Other cases can be inferred similarly. From Eq. (4), the channel differences $\Delta ^{0,j}$ between the 0$^{\circ }$ channel and other three $j$-channels ($j\in$ {45, 90, 135}) are :

(5a)$$ \Delta^{0,45} = I_{0}-I_{45} = I_{linear}(\cos(2\varphi)-\sin(2\varphi))/2 {,} $$

(5b)$$ \Delta^{0,90} = I_{0}-I_{90} = I_{linear}\cos(2\varphi) {,}$$

(5c)$$ \Delta^{0,135} = I_{0}-I_{135} = I_{linear}(\cos(2\varphi)+\sin(2\varphi))/2 {,} $$

Equation (5) elucidates that the difference between two polarization channels relies on $I_{linear}$ and $\varphi$. Let us consider two arbitrary adjacent pixels $p$ and $q$, their intensities of linear polarized light are $I_{p}$ and $I_{q}$, respectively. And their corresponding angles of linear polarization are $\varphi _{p}$ and $\varphi _{q}$. The distance $\sigma$ between $p$ and $q$ in three channel difference images is defined as:

(6a)$$ \sigma^{0,45} = |\Delta^{0,45}_{p}-\Delta^{0,45}_{q}| = |I_{p}(\cos(2\varphi_{p})-\sin(2\varphi_{p})) - I_{q}(\cos(2\varphi_{q})-\sin(2\varphi_{q}))|/2 {,} $$

(6b)$$ \sigma^{0,90} = |\Delta^{0,90}_{p}-\Delta^{0,90}_{q}| = |I_{p}\cos(2\varphi_{p}) - I_{q}\cos(2\varphi_{q})| {,}$$

(6c)$$ \sigma^{0,135} = |\Delta^{0,135}_{p}-\Delta^{0,135}_{q}| = |I_{p}(\cos(2\varphi_{p})+\sin(2\varphi_{p})) - I_{q}(\cos(2\varphi_{q})+\sin(2\varphi_{q}))|/2 {,} $$

Assuming that linear polarized light and AoLP are uniformly distributed over the image, $\sigma$ can be integrated to reflect the smoothness of the difference domain from a global perspective as:

(7a)$$ \Sigma^{0,45} = \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi} \sigma^{0,45} dI_{p}dI_{q}d\varphi_{p}d\varphi_{q} {,} $$

(7b)$$ \Sigma^{0,90} = \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi} \sigma^{0,90} dI_{p}dI_{q}d\varphi_{p}d\varphi_{q} {,} $$

(7c)$$ \Sigma^{0,135} = \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi} \sigma^{0,135} dI_{p}dI_{q}d\varphi_{p}d\varphi_{q} {,} $$

The analytical solutions of Eq. (7) are difficult to calculate, but we can find the ratio of $\Sigma ^{0,45}$, $\Sigma ^{0,90}$ and $\Sigma ^{0,135}$ as:

(8)$$\Sigma^{0,45}:\Sigma^{0,90}:\Sigma^{0,135} = 1:\sqrt{2}:1 {,}\\$$

The detailed derivation process of Eq. (8) is presented in the Appendix A. The ratio in Eq. (8) is a global estimation. Other orthogonal channels and non-orthogonal channels have the same relationship.

2.3 Verification of PCDP on the polarization dataset

The scenes of the polarization dataset [25] are grouped into polarized and unpolarized illumination cases, where 30 scenes are illuminated by unpolarized light and 10 scenes are illuminated by polarized light. The experiments are conducted according to whether the scene is illuminated by polarized light or not. The average distance between adjacent pixels in the channel difference is computed, which is defined as:

(9)$$d_{ave}=\frac{1}{8^{*}(M-2)^{*}(N-2)^{*} S} \sum_{scenes=1}^{S} \sum_{i=2}^{M-1} \sum_{j=2}^{N-1} \sum_{m=({-}1,0,1) \atop n=({-}1,0,1)}\left|P_{center}^{(i, j)}-P_{neighbor }^{(i+m, j+n)}\right| {,}$$

where $M$ and $N$ are the width and height of the image, respectively. We calculate the average distance between the center pixel ($P_{center}$) and the eight pixels ($P_{neighbor}$) around it in the $S$ scenes ($S$=10 in polarized illumination, $S$=30 in unpolarized illumination). The statistical results in Table 1 show that both statistical results coincide with the proposed prior. Table 1 also shows that the average distance ratio is close to 1:1.2:1 in the unpolarized illumination cases, and the fluctuations of the difference of the two non-orthogonal channels are inconsistent in the polarized illumination cases. These discrepancies between the statistical ratio and Eq. (8) may be caused by the background regions. Equation (8) is obtained from the assumption that the energy of linear polarization light and angles of linear polarization in adjacent pixels are evenly distributed and independent of each other. However, the existence of a large area of single AoLP and single linear polarization intensity background in the dataset [25] breaks this assumption. For the polarized illumination scenes, the background is a highly polarized screen to create the polarized illumination (as shown in scene "carrier" in Fig. 3). From Eq. (6), the background area of the screen with highly linear polarized light would have a significant impact on statistics, though the variation of AoLP is quite small. This may lead to the inconsistency between the statistical of the average distance in channel differences of the two non-orthogonal channels. For the unpolarized illumination scenes, for example, the scene "ball" in Fig. 3, the background is a white wall with very low polarization, where the linear polarization components on the channel difference are very small and the linear polarization components of adjacent pixels are very close. Thus the ratio of these areas is close to 1:1:1. Hence, the fluctuation of the difference of the orthogonal channel in the statistical analysis is smaller than that in the theoretical analysis. The statistical ratio of these smooth areas is inconsistent with Eq. (8), but this barely affect the proposed method, including other demosaicking methods, since the smooth areas contain only low frequency information. All above statistics are performed on the green channel of dataset [25].

Fig. 3. Background contrast between polarized and unpolarized illumination scenes.

Download Full Size | PDF

Table 1. The average distance of adjacent positions in unpolarized and polarized illumination scenes.

View Table | View all tables in this article

3. Polarization demosaicking using PCDP

A DoFP polarization camera with a non-redundant PFA provides a raw image $I^{PFA}$ with $m$ rows and $n$ columns. Let $P$ be the set of all pixels (with the number of $m \times n$) and $P_{i}$ be the pixel subset of $i$-channel ($i$ $\in$ {0, 45, 90, 135}). Our algorithm can be described in the following steps:

1) For each channel $i$, a sparse raw image $\hat {I}_{i}$ that contains the available values from $I^{PFA}$ and zeroes is obtained as: $(10)$$\hat{I}_{i}=I^{PFA}\odot mask_{i} {,}$$$ where $\odot$ represents Hadamard product and $mask_{i}$ is a binary matrix defined at each pixel $p$ as: $(11)$$mask_{i}^{p}=\left\{ \begin{array}{cc} 1, & \textrm{if} \textit{p} \in P_{i} \\ 0, & \textrm{otherwise} \end{array} \right. {,}$$$
2) Each channel is initially estimated by bilinear interpolation. Here we use convolution to quickly implement bilinear interpolation as: $(12)$$\tilde{I}_{i} = \hat{I}_{i} \ast F {,}$$$ where $\ast$ is the convolution operator. For a non-redundant PFA, the bilinear interpolation filter $F$ is defined as: $(13)$$F=\frac{1}{4}{ \left[ \begin{array}{ccc} 1 & 2 & 1\\ 2 & 4 & 2\\ 1 & 2 & 1 \end{array}\right ]} {,}$$$
3) The channel difference of $i$-channel and $j$-channel is computed as: $(14)$$\hat{\Delta}^{i,j}=(\hat{I}_{i}-\tilde{I}_{j}\odot mask_{i})*F {,}$$$
4) For each channel $i$, the final estimation $I_{i}$ is obtained by $(15)$$I_{i}=\sum_{j\neq i} w_{i,j}(\tilde{I}_{j}+\hat{\Delta}^{i,j}) {,}$$$ where $w_{i,j}$ is the fusion weight and determined by the relationship between the $i$-channel and the $j$-channel from Eq. (8) as: $(16)$$w_{i,j}=\left\{\begin{array}{l} \frac{\sqrt{2}}{1+\sqrt{2}+1}, \rm{if} \; |\textit{i}-\textit{j}| \neq 90^{{\circ}} \\ \frac{1}{1+\sqrt{2}+1}, \rm{if} \; |\textit{i}-\textit{j}| = 90^{{\circ}} \end{array}\right. {,}$$$

Ideally, the weights should be determined adaptively based on the AoLP of each pixel, but the ground-truth AoLP is hard to obtain in practice and the AoLP is strongly sensitive to noise [26,27]. Thus, if we use the estimated local AoLP to determine the weights, undesired errors might be introduced. Here we instead use fixed weights which are globally optimal as presented in Eq. (8). Figure 4 shows the recovery process of the $I_{0}$ channel, and the other channels can be recovered similarly.

Fig. 4. Outline of the interpolation process of $I_{0}$ channel.

Download Full Size | PDF

4. Experiments

4.1 Experimental settings

Implementation details In our experiments, we use the color-polarization dataset proposed by Qiu et al. [25], which contains 40 scenes (30 unpolarized illumination scenes and 10 polarized illumination scenes) with the size of 1024$\times$1024. For each scene, raw images of polarization orientations of 0$^{\circ }$, 45$^{\circ }$, 90$^{\circ }$ and 135$^{\circ }$ are captured by a color camera. Our method is for monochrome PFA, so we choose the green channel as the ground truth in our experiments. Synthetic DoFP mosaic images are generated by down-sampling the four captured polarization images as the pattern in Fig. 1. Performing different demosaicking algorithms on testing mosaic images, we obtain four complete polarization images ($I_{0}$, $I_{45}$, $I_{90}$, $I_{135}$). Then the intensity and polarization images ($S_{0}$, DoLP, AoLP) are calculated for each algorithm. Finally, the demosaicking performance of each method is evaluated by comparing the recovered results with the ground truth.

Comparison and metrics Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [28] are used as quality assessment metrics in our experiments. The higher PSNR and SSIM values mean better performances. In metrics computation, we exclude 8 pixels in each of the four borders of the image to avoid the inherent border effect related to the demosaicking processing. Besides, the running time of each demosaicking method is also tested.

4.2 Comparison with state-of-the-arts

The proposed PCDP is compared with other 6 existing demosaicking methods, including bilinear interpolation (BI), intensity correlation among polarization channels (ICPC) [29], residual interpolation on PFA (PRI) [17], pseudo-panchromatic image difference (PPID) method [15], Newton’s polynomial interpolation (NP) algorithm [14] and edge-aware residual interpolation (EARI) [19]. NP, ICPC and EARI provide the source code. The codes of PRI and PPID are reproduced according to the instructions of the corresponding articles.

Accuracy comparison Table 2 shows the average PSNR and SSIM of different demosaicking methods in unpolarized illumination scenes and polarized illumination scenes. PPID and EARI are relatively stable in both unpolarized and polarized illumination scenes. NP performs well in unpolarized illumination but poorly in polarized illumination scenes. In PPID and EARI, $S_{0}$ is first estimated and then used to compute differences with four intensity channels, while NP interpolates directly in the channel difference domain. In a strong polarized illumination case, the high frequency energy of channel difference may be larger than that of the intensity channel itself, which leads to the under-performance of the difference domain based methods. BI and ICPC interpolate in the intensity domain, so their performances are better than most other methods in polarized illumination scenes. Our method also interpolates in the channel difference domain, but with the proposed prior, our performance is better than NP in polarized illumination scenes. Here we only show the experimental results in the green channel, and similar results are obtained in red and blue channels.

Table 2. Performance comparison between our approach and other method. Best results are highlighted in bold.

View Table | View all tables in this article

Visual comparison Figs. 5 and 6 show that the proposed method PCDP obtains compelling performances in both unpolarized (scene "kettle") and polarized (scene "cellphonecases") illuminations. For more intuitive comparison, the regions within the red rectangles are enlarged and corresponding error images are also normalized and visualized. In the "kettle" scene, the recovery errors obtained by our method are minimal, especially on the edges. Without detecting edges, PCDP exceeds NP in the recovery of DoLP, while NP is the best method in the recovery of intensity image ($S_{0}$). This demonstrates the effectiveness of the proposed prior. In the "cellphonescases" scene, PCDP still performs properly with less reconstruction errors at complex textures than NP, PRI and EARI.

Fig. 5. Visual comparison of different methods in the polarized illumination scene "kettle" and the error image is the absolute difference map between the reconstructed result and the ground-truth image.

Download Full Size | PDF

Fig. 6. Visual comparison of different methods in the polarized illumination scene "cellphonecases" and the error image is the absolute difference map between the reconstructed result and the ground-truth image.

Download Full Size | PDF

Time consumption comparison Table 3 shows the running time of each method. The proposed PCDP is significantly fast: 2 times faster than NP, 6 times faster than EARI, and 36 times faster than PPID. Thanks to a simple calculation process, our method can speed up nearly 60 times on the RTX 2700 SUPER GPU. Other methods except for BI entail calculating the gradient pixel by pixel, so the acceleration of these method on GPU is very limited and some are even slower than CPU.

Table 3. Running time of different methods.

View Table | View all tables in this article

4.3 Ablation study

To verify the validity of polarization channel difference prior, we carry out an ablation experiment. Step 4 in our method is replaced with

(17)$$I_{i} = \hat{I}_{i} + \sum_{j\neq i}(I^{PFA}+\hat{\Delta}^{i,j}) \odot mask_{j} {,}$$

In Eq. (17), $I_{i}$ is estimated by adding the raw value and channel difference directly. This is used as a comparison to test the effectiveness of the proposed PCDP. Table 4 illustrates that PCDP is of great help to the recovery of polarization information in both polarized and unpolarized illumination scenes. In the unpolarized case, there is a slight drop in $S_{0}$ with the prior while a significant increase in both DoLP and AoLP. On the other hands, there is no such trade-off in the polarized case, which proves the superiority of the proposed polarization prior, particularly in the polarized illumination. Figure 7 shows the results of the reconstruction of the areas within the red rectangles in the scene "screen" (unpolarized illumination) and the scene "plate" (polarized illumination). The proposed method with PCDP reconstructs the edges more precisely on both $S_{0}$ and DoLP without additional computational cost.

Fig. 7. Test of the validity of PCDP in the unpolarized illumination scene "screen" (left) and polarized illumination scene "plate" (right). The error image is the absolute difference map between the reconstructed result and the ground-truth image.

Download Full Size | PDF

Table 4. Performance evaluation of the polarization channel difference prior. Best results are highlighted in bold.

View Table | View all tables in this article

4.4 Test in an unpolarized area

In a completely unpolarized case (such as diffuse surfaces), the intensity difference between polarization channels would be all zeros. Then, the prior does not hold, but our method is still valid. Our method would degenerate to an ordinary interpolation method in the difference domain. In this case, $I_{mosaic} = 0.5S_{0} = I_{0} = I_{45} = I_{90} = I_{135}$, which means that DoLP equals to 0 and demosaicking is no longer needed. However, the polarization state of each pixel is unknown in practice, so a demosaicking process is still of necessity for a completely unpolarized case. To evaluate the performance of different demosaicking methods in a completely unpolarized case, we assume that the scenes in the database are all completely unpolarized, i.e. $I_{mosaic} = 0.5S_{0}$. We still test all algorithms on the green channel. Table 5 shows the average PSNR values of reconstruction of $S_{0}$. PCDP still has good performance compared with other methods.

Table 5. Experiments for demosaicking in the completely unpolarized cases.

View Table | View all tables in this article

5. Conclusions and discussions

This paper presents a DoFP polarization image demosaicking technique using polarization channel difference prior. Based on the statics of polarization channel difference, PCDP is proved to be physics valid assuming that the polarization angle and the intensity of linear polarized light are uniform distributed. With this prior, a weighted fusion strategy is designed to recover the missing polarization information. The proposed demosaicking algorithm achieves comparable performances to the state-of-the-art methods. Our method is highly parallelizable, so it can be implemented on GPU or FPGA to perform DoFP image demosaicking at a much faster speed. The proposed method can also be applied to color polarization demosaicking, as a separate module connected to the color demosaicking module as in [18]. Although PCDP has shown obvious advantages in polarization demosaicking, it still has some limitations at this stage. The modeling of PCDP is based on the assumption that the polarization angle and linear polarization are uniformly distributed, which is not well satisfied in processing specific images and scenes that do not meet PCDP. Adaptively using PCDP or combining it with other polarization demosaicking methods will be our future work.

Appendix A

The detailed derivation of Eq. (8) is as follow:

(A1)$$\begin{aligned}\Sigma^{0,45} &=\int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi} \sigma^{0,45} d I_{p} d I_{q} d \varphi_{p} d \varphi_{q} \\ &=\frac{1}{2} \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi}\left|I_{p}\left(\cos \left(2 \varphi_{p}\right)-\sin \left(2 \varphi_{p}\right)\right)-I_{q}\left(\cos \left(2 \varphi_{q}\right)-\sin \left(2 \varphi_{q}\right)\right)\right| d I_{p} d I_{q} d \varphi_{p} d \varphi_{q} \\ &=\frac{1}{2} \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi}\left|\sqrt{2} I_{p} \cos \left(2 \varphi_{p}+\frac{\pi}{4}\right)-\sqrt{2} I_{q} \cos \left(2 \varphi_{q}+\frac{\pi}{4}\right)\right| d I_{p} d I_{q} d \varphi_{p} d \varphi_{q} \\ &=\frac{\sqrt{2}}{2} \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi}\left|I_{p} \cos \left(2 \varphi_{p}+\frac{\pi}{4}\right)-I_{q} \cos \left(2 \varphi_{q}+\frac{\pi}{4}\right)\right| d I_{p} d I_{q} d \varphi_{p} d \varphi_{q}, \end{aligned}$$

Let $t_{p}=\varphi _{p}+\pi /8, t_{q}=\varphi _{q}+\pi /8$, we have $dt_{p}=d\varphi _{p}, dt_{q}=d\varphi _{q}$. Thus $t_{p}=\pi /8, t_{q}=\pi /8$ when $\varphi _{p} = 0, \varphi _{q}=0$, and $t_{p}=9\pi /8, t_{q}=9\pi /8$ when $\varphi _{p} = \pi , \varphi _{q} = \pi$. Then, Eq. (A1) can be rewritten as:

(A2)$$\Sigma^{0,45}=\frac{\sqrt{2}}{2} \int_{0}^{1} \int_{0}^{1} \int_{\frac{\pi}{8}}^{\frac{9 \pi}{8}} \int_{\frac{\pi}{8}}^{\frac{9 \pi}{8}}\left|I_{p} \cos \left(2 t_{p}\right)-I_{q} \cos \left(2 t_{q}\right)\right| d I_{p} d I_{q} d t_{p} d t_{q},$$

According to the periodicity criterion of indefinite integral, we have

(A3)$$\begin{aligned}\Sigma^{0,45} &=\frac{\sqrt{2}}{2} \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi}\left|I_{p} \cos \left(2 t_{p}\right)-I_{q} \cos \left(2 t_{q}\right)\right| d I_{p} d I_{q} d t_{p} d t_{q} \\ &=\frac{\sqrt{2}}{2} \int_{0}^{1} \int_{0}^{1} \int_{0}^{\pi} \int_{0}^{\pi}\left|I_{p} \cos \left(2 \varphi_{p}\right)-I_{q} \cos \left(2 \varphi_{q}\right)\right| d I_{p} d I_{q} d \varphi_{p} d \varphi_{q} \\ &=\frac{\sqrt{2}}{2} \Sigma^{0,90}, \end{aligned}$$

$\Sigma ^{0,135} = \frac {\sqrt {2}}{2} \Sigma ^{0,90}$ can also be obtained by following the above derivation similarly. Thus, we have $\Sigma ^{0,45}:\Sigma ^{0,90}:\Sigma ^{0,135} = 1:\sqrt {2}:1$

Funding

National Research Foundation of Korea (NRF-2015K2A2A2000886); Natural Science Foundation of Shaanxi Province (2018JM6056); Key Research and Development Projects of Shaanxi Province (2020ZDLGY07-11); National Natural Science Foundation of China (61771391); Science, Technology and Innovation Commission of Shenzhen Municipality (JCYJ20170815162956949, JCYJ20180306171146740).

Acknowledgement

We thank Master Jiaxiang Liu for the constructive discussions. We also thank Junchao Zhang, and Yusuke Monno for sharing the executable code. We also thank Qiang Fu for sharing the database.

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. G. Horváth, G. Horváth, and D. Varju, Polarized light in animal vision: polarization patterns in nature (Springer Science & Business Media, 2004).

2. N. Li, Y. Zhao, Q. Pan, S. G. Kong, and J. C.-W. Chan, “Full-time monocular road detection using zero-distribution prior of angle of polarization,” in European Conference on Computer Vision, (Springer, 2020), pp. 457–473.

3. N. Li, Y. Zhao, Q. Pan, and S. G. Kong, “Removal of reflections in lwir image with polarization characteristics,” Opt. Express 26(13), 16488–16504 (2018). [CrossRef]

4. A. Kalra, V. Taamazyan, S. K. Rao, K. Venkataraman, R. Raskar, and A. Kadambi, “Deep polarization cues for transparent object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), pp. 8602–8611.

5. L. Shen, Y. Zhao, Q. Peng, J. C.-W. Chan, and S. G. Kong, “An iterative image dehazing method with polarization,” IEEE Trans. Multimedia 21(5), 1093–1107 (2019). [CrossRef]

6. J. F. Hamilton Jr and J. E. Adams Jr, “Adaptive color plan interpolation in single sensor color electronic camera,” (1997). US Patent 5, 629, 734.

7. S.-C. Pei and I.-K. Tam, “Effective color interpolation in ccd color filter arrays using signal correlation,” IEEE Trans. Circuits Syst. Video Technol. 13(6), 503–513 (2003). [CrossRef]

8. L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. on Image Process. 14(12), 2167–2178 (2005). [CrossRef]

9. K.-H. Chung and Y.-H. Chan, “Color demosaicing using variance of color differences,” IEEE Trans. on Image Process. 15(10), 2944–2955 (2006). [CrossRef]

10. D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Residual interpolation for color image demosaicking,” in 2013 IEEE International Conference on Image Processing, (IEEE, 2013), pp. 2304–2308.

11. B. E. Bayer, “Color imaging array,” (1976). US Patent 3, 971, 065.

12. S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,” Opt. Express 19(27), 26161–26173 (2011). [CrossRef]

13. S. Gao and V. Gruev, “Gradient-based interpolation method for division-of-focal-plane polarimeters,” Opt. Express 21(1), 1137–1151 (2013). [CrossRef]

14. N. Li, Y. Zhao, Q. Pan, and S. G. Kong, “Demosaicking dofp images using newton’s polynomial interpolation and polarization difference model,” Opt. Express 27(2), 1376–1391 (2019). [CrossRef]

15. S. Mihoubi, P.-J. Lapray, and L. Bigué, “Survey of demosaicking methods for polarization filter array images,” Sensors 18(11), 3688 (2018). [CrossRef]

16. S. Mihoubi, O. Losson, B. Mathon, and L. Macaire, “Multispectral demosaicing using pseudo-panchromatic image,” IEEE Trans. Comput. Imaging 3(4), 982–995 (2017). [CrossRef]

17. A. Ahmed, X. Zhao, V. Gruev, J. Zhang, and A. Bermak, “Residual interpolation for division of focal plane polarization image sensors,” Opt. Express 25(9), 10651–10662 (2017). [CrossRef]

18. K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). [CrossRef]

19. M. Morimatsu, Y. Monno, M. Tanaka, and M. Okutomi, “Monochrome and color polarization demosaicking using edge-aware residual interpolation,” in 2020 IEEE International Conference on Image Processing (ICIP), (IEEE, 2020), pp. 2571–2575.

20. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521(7553), 436–444 (2015). [CrossRef]

21. J. Zhang, J. Shao, H. Luo, X. Zhang, B. Hui, Z. Chang, and R. Liang, “Learning a convolutional demosaicing network for microgrid polarimeter imagery,” Opt. Lett. 43(18), 4534–4537 (2018). [CrossRef]

22. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 770–778.

23. X. Zeng, Y. Luo, X. Zhao, and W. Ye, “An end-to-end fully-convolutional neural network for division of focal plane sensors to reconstruct s 0, dolp, and aop,” Opt. Express 27(6), 8566–8577 (2019). [CrossRef]

24. D. H. Goldstein, Polarized light (CRC University, 2017).

25. S. Qiu, Q. Fu, C. Wang, and W. Heidrich, “Polarization demosaicking for monochrome and color polarization focal plane arrays,” in International Symposium on Vision, Modeling and Visualization, 2019, (The Eurographics Association, 2019).

26. S. Roussel, M. Boffety, and F. Goudail, “Polarimetric precision of micropolarizer grid-based camera in the presence of additive and poisson shot noise,” Opt. Express 26(23), 29968–29982 (2018). [CrossRef]

27. Dai Jun and Goudail Franois, “Precision analysis of arbitrary full-stokes polarimeters in the presence of additive and poisson noise,” J. Opt. Soc. Am. A 36(7), 1229–1240 (2019). [CrossRef]

28. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]

29. J. Zhang, H. Luo, B. Hui, and Z. Chang, “Image interpolation for division of focal plane polarimeters with intensity correlation,” Opt. Express 24(18), 20799–20807 (2016). [CrossRef]

$d_{a v e}$		$I_{0}$	$I_{45}$	$I_{90}$	$I_{135}$
Unpolarized Illumination $(30$ scenes $)$	$I_{0}$	0	$2.68$	$3.19$	$2.66$
	$I_{45}$	$2.68$	0	$2.54$	$3.12$
	$I_{90}$	$3.19$	$2.54$	0	$2.66$
	$I_{135}$	$2.66$	$3.12$	$2.66$	0
Polarized Illumination $(10$ scenes $)$	$I_{0}$	0	$7.10$	$8.79$	$6.54$
	$I_{45}$	$7.10$	0	$4.55$	$8.03$
	$I_{90}$	$8.79$	$4.55$	0	$7.04$
	$I_{135}$	$6.54$	$8.03$	$7.04$	0

Average PSNR (dB)		BI	ICPC	PRI	PPID	NP	EARI	PCDP (ours)
Unpolarized Illumination $(30$ scenes $)$	$S_{0}$	45.90	46.74	47.01	48.59	50.08	49.24	48.55
	DoLP	37.71	38.51	38.86	41.09	40.44	41.66	40.43
	AoLP	23.70	23.71	24.24	24.85	25.44	25.06	24.80
Polarized Illumination $(10$ scenes $)$	$S_{0}$	49.07	49.60	46.36	49.44	47.80	48.75	48.86
	DoLP	39.30	40.50	36.00	39.51	34.34	39.09	37.92
	AoLP	33.47	34.13	31.96	34.00	32.00	33.34	33.69
Average SSIM		BI	ICPC	PRI	PPID	NP	EARI	PCDP (ours)
Unpolarized Illumination $(30$ scenes $)$	$S_{0}$	0.9856	0.9855	0.9873	0.9898	0.9928	0.9909	0.9900
	DoLP	0.9193	0.9226	0.9290	0.9416	0.9351	0.9484	0.9378
	AoLP	0.6040	0.5917	0.6240	0.6489	0.6777	0.6593	0.6494
Polarized Illumination $(10$ scenes $)$	$S_{0}$	0.9897	0.9903	0.9854	0.9897	0.9861	0.9814	0.9882
	DoLP	0.9595	0.9609	0.9431	0.9566	0.9170	0.9514	0.9505
	AoLP	0.9321	0.9356	0.9170	0.9322	0.9146	0.9273	0.9308

Average PSNR (dB)	Unpolarized Illumination (30 scenes)		Polarized Illumination (10 scenes)
Average PSNR (dB)	w/o prior	w prior	w/o prior	w prior
$S_{0}$	48.91	48.55	47.08	48.86
DoLP	38.27	40.43	34.69	37.92
AoLP	24.27	24.80	32.41	33.69

$d_{a v e}$		$I_{0}$	$I_{45}$	$I_{90}$	$I_{135}$
Unpolarized Illumination $(30$ scenes $)$	$I_{0}$	0	$2.68$	$3.19$	$2.66$
	$I_{45}$	$2.68$	0	$2.54$	$3.12$
	$I_{90}$	$3.19$	$2.54$	0	$2.66$
	$I_{135}$	$2.66$	$3.12$	$2.66$	0
Polarized Illumination $(10$ scenes $)$	$I_{0}$	0	$7.10$	$8.79$	$6.54$
	$I_{45}$	$7.10$	0	$4.55$	$8.03$
	$I_{90}$	$8.79$	$4.55$	0	$7.04$
	$I_{135}$	$6.54$	$8.03$	$7.04$	0

Average PSNR (dB)		BI	ICPC	PRI	PPID	NP	EARI	PCDP (ours)
Unpolarized Illumination $(30$ scenes $)$	$S_{0}$	45.90	46.74	47.01	48.59	50.08	49.24	48.55
	DoLP	37.71	38.51	38.86	41.09	40.44	41.66	40.43
	AoLP	23.70	23.71	24.24	24.85	25.44	25.06	24.80
Polarized Illumination $(10$ scenes $)$	$S_{0}$	49.07	49.60	46.36	49.44	47.80	48.75	48.86
	DoLP	39.30	40.50	36.00	39.51	34.34	39.09	37.92
	AoLP	33.47	34.13	31.96	34.00	32.00	33.34	33.69
Average SSIM		BI	ICPC	PRI	PPID	NP	EARI	PCDP (ours)
Unpolarized Illumination $(30$ scenes $)$	$S_{0}$	0.9856	0.9855	0.9873	0.9898	0.9928	0.9909	0.9900
	DoLP	0.9193	0.9226	0.9290	0.9416	0.9351	0.9484	0.9378
	AoLP	0.6040	0.5917	0.6240	0.6489	0.6777	0.6593	0.6494
Polarized Illumination $(10$ scenes $)$	$S_{0}$	0.9897	0.9903	0.9854	0.9897	0.9861	0.9814	0.9882
	DoLP	0.9595	0.9609	0.9431	0.9566	0.9170	0.9514	0.9505
	AoLP	0.9321	0.9356	0.9170	0.9322	0.9146	0.9273	0.9308

Polarization image demosaicking using polarization channel difference prior

Abstract

1. Introduction

1.1 Background

1.2 Linear polarization calculation

2. Polarization channel difference prior (PCDP)

2.1 Across edges analysis

2.2 Theoretical proof

2.3 Verification of PCDP on the polarization dataset

3. Polarization demosaicking using PCDP

4. Experiments

4.1 Experimental settings

4.2 Comparison with state-of-the-arts

4.3 Ablation study

4.4 Test in an unpolarized area

5. Conclusions and discussions

Appendix A

Funding

Acknowledgement

Disclosures

Data availability

References

Data availability

Cited By

Figures (7)

Tables (5)

Equations (32)

Optics Express