1. Introduction

Shadows, a physical phenomenon observed in most natural scenes, are problematic to many computer vision algorithms. They often bring some wrong results in edge detection, segmentation, target recognition and tracking. For these reasons, shadow processing and illumination invariant on an image are of great practical significance and have attracted a lot of attentions [1–6]. Some previous works dealt with this problem by utilizing additional information from multiple images [7–9], time-lapse image sequences [2, 10, 11] or user inputs [9, 12, 13]. Although these methods achieve impressive results for illumination invariant and shadow-free image, they may not be applied automatically to a single image. In general, a robust method for illumination invariant and shadow-free on a single image is more favored and challenging. The recent works related to a shadow-free image on a single image can be divided into two categories: methods that need shadow detection and methods that do not need shadow detection.

Methods with shadow detection. Most works for shadow-free image often undergo two basic steps: shadow detection and shadow removal [14]. A fair proportion of shadow detection methods are based on shadow features with statistical learning [15–18]. The most commonly used shadow features are intensity, histograms, texture, color ratio, geometry property, and gradient. Lalonde et al. [16] obtained shadow removal results in consumer-grade photographs by a conditional random field (CRF) learning approach on color ratio, texture and intensity distributions. Based on initial segmentation, the method proposed by Guo et al. [15] detects shadows using SVM on shadow features (e.g, texture, chromaticity, and pairwise relation) and then relights each pixel to obtain a shadow-free image. These shadow features may not be robust enough in some applications. For example, shadowed regions are often dark, with less texture and little gradient, but some non-shadowed regions may also have similar characteristics. Due to the lack of robust shadow features, these shadow feature and learning based methods are not always effective, especially in soft shadow regions or some complex scenarios. Besides, all these learning-based methods are usually time-consuming since large complex features are needed to be analyzed for classifiers. Some other works for shadow-free image with shadow detection are based on computing physically-based illumination invariants [19–22]. With a physically-based illumination invariant, Finlayson et al. [19, 20] proposed a method to obtain a shadow-free image by shadow edge detection and solving a Poisson’s equation. In [22], we deduced a linear relationship between pixel values of a surface in and out of shadow regions, and then proposed a shadow removal method based on shadow edge detection. In the condition of correct shadow edge detection, the method proposed by Arbel and Hel-Or [14] yields a shadow-free image by finding scale factors to cancel the effect of shadows. All these mentioned methods require shadow edges to be correctly detected. However, automatic edge detection is known to be non-robust [23], especially in complex scenarios and texture (such as the shadows of leaves and grasses). The defect of these shadow detection based methods is revealed in [24]. Even when the shadows are detected rightly, retaining the original color and texture in a shadow-free image is also a challenging problem [14, 25].

Methods without shadow detection. Another effective approach for shadow-free image is intrinsic image via image decomposition. Tappen et al. [17] proposed a learning-based approach to predict the derivatives of reflectance and illumination images. Although this method successfully separates reflectance and shading for a given illumination direction used for training, it is not designed for arbitrary lighting directions [24]. Assuming that the neighboring pixels that have similar intensity values should have similar reflectance, Shen et al. [26] proposed both an automatic and user scribbles method to separate intrinsic image from a single image with optimization. Although their automatic method is able to extract most of the illumination variations, residual shadows remain in the reflectance image and they stated that user scribbles cannot improve the result significantly. Finlayson et al. [20] derived a grayscale shadow-free image by finding the special direction in a 2-D chromaticity feature space. In [22], we obtained a grayscale shadow-free image based on a physical deduced linear model from the view of atmospheric transmittance effects. These derived grayscale shadow-free images have some limitations in real applications due to the loss of color information. Applying narrowband camera sensors in deduction, Maxwell et al. [27] found that a 2-D illumination invariant image can be obtained by projecting the colors (in log-RGB) onto a plane. In their method, the projecting normal vector is determined either from a user input or a 2-D entropy searching, which may have some limitations in real applications. With the assumption of chromatic surfaces, Planckian lighting and narrowband camera sensors, Yang et al. [24] obtained a shadow-free image using bilateral filtering without shadow detection. The comparison results in [24] shows that this method for shadow-free image outperforms Finlayson’s [19] (a classical method for shadow-free image that need shadow detection). However, based solely on chromaticity, this method may not be valid on neutral regions when the neighboring regions of the image are also neutral [24]. The experimental results presented in section 4 of this paper also show that bilateral filtering often makes the method in [24] fail to recover a shadow-free image from the image with darker shadows or in which shadows pervade in a scenario.

In this paper, we propose a novel, effective and efficient method to obtain a shadow-free image from a single outdoor image without feature extraction or shadow detection. The research is based on our previous work of grayscale illumination invariant image derivation from a physically-based shadow linear model [22]. With three different grayscale illumination invariant images (e.g., Fig. 1(b)) from this linear model [22], we apply them to set up three linear equations for each pixel value vector. The solution space of these linear equations is decomposed into one-dimensional nullspace (free solution space) and particular solution space. The free solution, representation of illuminants ratio, is only determined by illumination condition. The particular solution, which is perpendicular to free solutions, is unique and illumination invariant. It retains the basic color information of the original pixel value vector. Therefore, we name this process as pixel-wise orthogonal decomposition for color illumination invariant. Due to that this color illumination invariant image (e.g., Fig. 1(c)) still has some color distortion compared with the original image, combining color illumination invariant image and Lab color space, we propose an algorithm to generate a shadow-free image (e.g., Fig. 1(d)), which better preserves the color and texture information of the original image.

 

Fig. 1 One experimental result of our algorithm. (a) Original image. (b) One of the three grayscale illumination invariant images. (c) Color illumination invariant image with our pixel-wise orthogonal decomposition. (d) Shadow-free image after color restoration.

Download Full Size | PDF

The main contributions of this paper are: (1) We propose a new pixel-wise orthogonal decomposition method to obtain a shadow-free image without feature extraction or shadow detection. (2) Our method requires only once pixel-wise calculation on an image and can run in real time. (3) This pixel-wise orthogonal decomposition retains the original texture well. (4) The color illumination invariant image yields an identical reflectivity result regardless of illumination condition, which is both invariant for different illumination conditions and shadows.

The rest of the paper is organized as follows. In Section 2, we give a brief introduction of the derivation of grayscale illumination invariant image and then present our orthogonal decomposition method for color illumination invariant image from a single image. In section 3, we present a color restoration method for the generation of shadow-free image from color illumination invariant image. Some quantitative and qualitative experiments on a diverse set of shadow images are given in Section 4. We end this paper with a brief discussion and conclusion in section 5.

2. Pixel-wise orthogonal decomposition and color illumination invariant image

In this section, we first give a brief introduction of our previous work of shadow linear model and the derivation of relevant grayscale illumination invariant image. Then we present our pixel-wise orthogonal decomposition method for color illumination invariant image from a single image.

2.1. Pixel-wise orthogonal decomposition

2.1.1. Shadow linear model on an image [22]

For visible light whose spectrum ranges from 400nm to 700nm, given spectral power distributions (SPD) of illumination E(λ), object reflectance S(λ) and sRGB color matching functions Q(λ), the RGB tristimulus values (ℱ_H) in sRGB color space are computed by,

From Eq. (2), the following equation holds.

 

Fig. 2 Grayscale illumination invariant image results based on the linear model

Download Full Size | PDF

2.1.2. Pixel-wise orthogonal decomposition

The grayscale illumination invariant image has some limitations in some applications due to the loss of color and contrast information. We expect to get an illumination (including shadow) invariant image, which can retain the color information of the original image and appear the same as the original image without shadow. For this purpose, based on the model in previous section, we deduce a pixel-wise orthogonal decomposition and obtain an illumination invariant image which retains the basic color information of the original image.

For a RGB value vector (v_R,v_G,v_B)^T, we first define the following grayscale illumination invariant values I₁ according to Eq. (4):

Similar to Eq. (6), we can obtain two other grayscale illumination invariant values I₂ and I₃,

For convenience, let u = (u_R,u_G,u_B)^T defines a Log-RGB value vector of a pixel, where u_H = log(v_H + 14),H = {R,G,B}. From Eqs. (4), (7), and (8), we have the following linear equations,

From algebraic theory, any solution u of Eq. (11), can be expressed as following:

Here the free solution has no relationship with the image itself but is determined by matrix A, i.e. illumination condition. Formula 14 further reveals that this free solution is a representation of the ratio of illuminants in imaging environments.

Given a particular solution u_s and the normalized free solution u₀ of Eq. (11), according to Eq. (13) and algebraic theory, we define u_p as

From the deducing procedure above and the orthogonal decomposition, we know that u₀, representation of illuminants ratio, is only related with (β₁,β₂,β₃)^T and u_p, perpendicular to u₀, is illumination invariant. It means that for a pixel with Log-RGB value vector u, no matter whether it is on shadow or not, u_p is invariant and only α reflects the variation of pixel RGB values caused by shadow or different illuminants. The different solutions caused by different α component in Eq. (17) can be regarded as Log-RGB values of a pixel on different illumination conditions. Therefore, given an image under multiple illumination conditions, for each pixel with Log-RGB value vector u of this image, we can directly calculate its illumination invariant vector u_p as

This pixel-wise orthogonal decomposition process can be shown more clearly from the illustration in Fig. 3. For a pixel, ${\mathit{u}}_{\mathit{s}}^1$ and ${\mathit{u}}_{\mathit{s}}^2$ denote its log-RGB vector within shadow and log-RGB vector without shadow respectively, which are the solutions of Eq. (11). Here we denote any solution of Eq. (11) by vector u. According to our orthogonal decomposition Eqs. (14), (15), and (17), ${\mathit{u}}_{\mathit{s}}^1$, ${\mathit{u}}_{\mathit{s}}^2$, and u can be projected along the vector u₀ into an specific solution of Eq. (13), u_p, which is perpendicular to u₀. Therefore, for a pixel on an image, it means that, no matter how different the values of the pixel are with different illumination conditions (within shadow, without shadow or other illuminated), we can get an unique invariant u_p by our orthogonal decomposition.

 

Fig. 3 An illustration of pixel-wise orthogonal decomposition.

Download Full Size | PDF

2.2. Analysis of Parameters β₁,β₂,β₃ and color illumination invariant image

The estimation of parameters β₁,β₂,β₃ is important in determining whether our pixel-wise orthogonal decomposition works well or not. Luckily, from Eqs. (5), (9), and (3), we know that these parameters are reflectance irrelevant and only determined by the illumination condition (i.e, the SPD of daylight and skylight), such as zenith angle (time) and aerosol (weather). In clear weather that shadows mostly take place, these parameters are mainly determined by zenith angle. The SPD can be measured by spectrometer or calculated in atmosphere optics (software like MODTRAN and SMARTS2) with zenith angle and aerosol (for weathers) in advance. We have calculated parameters β₁,β₂,β₃ in sunny weather at different sun angles from 9:00 AM till 4:00 PM in our city and some of the values are shown in Table 1. The mean values in Table 1 are calculated by the mean SPD of daylight and skylight at 20° to 70°. From Table 1, we notice that the values of parameters β₁,β₂,β₃ from sun angle 20° to sun angle 80° are quite stable. In real applications, the sun angle (time) is known and the parameters can then be determined. In the condition that we have no information about at what sun angles the pictures were taken, the stability of parameters shows that the mean values for β₁,β₂,β₃ at 20° to 70° sun angles can be used for most situations in our decomposition, except some images taken at twilight, i.e., at sunrise or sunset, when sun angle is less than 10. Also, for more precise parameter values, an automatic entropy minimization method similar to Finlayson et al. [20] is employed. In addition to the factors (zenith angle and aerosol) mentioned above that may affect parameters β₁,β₂,β₃, some other aspects are also worthy of attention. For example, cloud and fog will affect the SPD of daylight and skylight and further affect the parameters β₁,β₂,β₃ in our algorithm. From clear weather to cloudy, as the clouds increase (aerosol increases), shadows vary from strong to weak. When shadows are weak, the error of the color illumination invariant results caused by parameters (β₁,β₂,β₃) deviation from cloud is also weaker. Therefore, the influence caused by the cloud distribution would be small and can be corrected by automatic entropy minimization [20]. This similar situation also goes for foggy weather.

Table 1. Parameters (PRMs) from representative sun angles

View Table | View all tables in this article

The above analysis shows that our pixel-wise orthogonal decomposition can work well on most situations. Our experimental results on many images in different scenes and reflectance also support this analysis. Two original images, their color illumination invariant images and α information are presented in Fig. 4. The shadows disappear and the main color information is maintained in our color illumination invariant images.

 

Fig. 4 Pixel-wise orthogonal decomposition on images. Left: Original image. Middle: color illumination invariant image. Right: α information.

Download Full Size | PDF

Compared with prior works [20,27,28], our 2-D color illumination invariant image has three main advantages:

3. Color restoration and shadow-free image

By the orthogonal decomposition on an image, some image color information gets lost (Fig. 4). As shown in Table 1 and Eqs. (14) and (15), the directions of free solution in different sun angles (from sun angle 20 to sun angle 80) are all near to (1,1,1)^T. From the orthogonal decomposition equation, we have, if u ≈ αu₀, the illumination invariant u_p ≈ (0,0,0)^T. It implies that the closer to the neutral surface on RGB space a pixel RGB value vector is, the more color information is got lost. To restore this loss, we present a color restoration method to generate a shadow-free image by correcting color vectors on the illumination invariant image which are near RGB color neutral surface.

Let v(x,y) = (v_R(x,y),v_G(x,y),v_B(x,y))^T be an RGB image, where x = 1,2,...,M;y = 1,2,...,N; M and N denote the image width and height respectively. We define

Then a shadow-free image is obtained by transferring the right luminance component of ${\mathit{u}}_{\mathit{c}}^{\mathit{RGB}}$ to the color illumination invariant image. We adopt Lab color space (L, the luminance component; a, b, two chrominance components) for this transfer. First we convert ${\mathit{u}}_{\mathit{c}}^{\mathit{RGB}}$ and ${\mathit{u}}_{\mathit{p}}^{\mathit{RGB}}$ from RGB space into Lab space and get ${\mathit{u}}_{\mathit{c}}^{\mathit{Lab}}$ and ${\mathit{u}}_{\mathit{p}}^{\mathit{Lab}}$ respectively,

An overview of our shadow-free image generation pipeline is presented in Algorithm 1. Figure 5 shows the stages of the algorithm implemented on a shadowed image. More shadow-free images can be seen in the experiment section and Appendix B. From the results, we can see that our method for shadow-free image effectively removes shadows from images while retaining the color and texture information of the original image. It is worth noting that, unlike shadow removal method that only recovers pixels in the shadow region, our shadow-free image, without the process of shadow detection, is operating on the entire image. In our algorithm, all the brighter pixels and darker pixels are pulled to the same illumination intensity and it may reduce the contrast of the image. Therefore, our shadow-free image may sometimes looks a bit dull compared with the original image.

Algorithm 1. Algorithm pipeline: Pixel-wise orthogonal decomposition for shadow-free image

View Table | View all tables in this article

 

Fig. 5 The stages of our algorithm implemented on a shadowed image. Note that the arrow in the first row represents the color space conversion, that is converting from RGB space to Lab space. These three components have no actual use in our algorithm. They are only used to act as the reference for the Lab component of our shadow-free image O.

Download Full Size | PDF

4. Experiments

In our experiment, we evaluate the results of color illumination invariant image and the relevant shadow-free image on more outdoor real images. These images consist of common scenes or objects under a variety of illumination conditions without shadow or with different types of shadows, such as soft shadows, cast shadows, and self shadows. We first compare our method with two state-of-the-art full-automatic methods (a statistical learning method [15] and a physical-based method [24] (Figs. 6 and 7)) and a shadow removal method [14, 30] with human intervention (Fig. 8), respectively. Then the results obtained by our method and Shen’s method [26] on images under different illumination conditions show that our color illumination invariant image is much closer to the intrinsic feature than Shen’s from the view of yielding an identical reflectivity result regardless of illumination condition (Fig. 9).

 

Fig. 6 Comparisons with two state-of-the-art methods (Guo et al. [15] and Yang et al. [24]). (b1), (b2) are close-ups of the red rectangles in (a1), (a2).

Download Full Size | PDF

 

Fig. 7 Comparisons with two state-of-the-art methods (Guo et al. [15] and Yang et al. [24]) on more images.

Download Full Size | PDF

 

Fig. 8 Comparison with Arbel’s shadow removal method [14, 30]. (a) Original image. (b) Examples of user-guided shadow and non-shadow observations in Arbel’s method denoted by red circles and white circles respectively. (c) The extracted shadow masks in Arbel’s method. (d) Arbel’s shadow-free image. (e) Our shadow-free image.

Download Full Size | PDF

 

Fig. 9 Images under different illumination conditions. For rows: (r1) original image, (r2) intrinsic image by Shen et al. [26], (r3) our color illumination invariant image, (r4) our shadow-free image. For columns: (a) (reference image) in daylight, (b) and (c) partly in daylight and partly in skylight, (d) in skylight.

Download Full Size | PDF

4.1. Comparison of shadow-free images

Comparison with full-automatic methods. In our experiment, we compare our method with two state-of-the-art methods ([15] and [24]) in terms of shadow-free results and running time. These two methods are based on shadow feature with statistical learning and physical model respectively. These methods are executed with the codes published by the authors of [15] and [24].

In Fig. 6, we show two shadow-free image results. For the first image (Fig. 6(a1)), both our method and Yang’s method [24] remove the soft-shadow of leaves successfully. However, Guo’s method [15] fails to recover a shadow-free image due to the wrong detection on shadows. Due to lack of robust shadow features, this shadow feature and learning based method [9] are not always effective, especially in soft shadow region or some complex scenarios (e.g., Figs. 7(d) and 7(e)). Close-ups of the first image (Fig. 6(b1)) show that, compared with our result, Yang’s method [24] fails to retain the white arrow on the ground. This is because that Yang’s method is not solid for neutral regions when its neighbor regions are also neutral [24]. For the second image (Fig. 6(a2)), our method gets a shadow-free image with texture preserved, while Guo’s method [15] fails to retain the original texture even when it detects the shadows successfully due to the inappropriate relighting on pixels. Specially, some pixels are relighted serious wrongly, such that it looks like that some objects that do not exist in the original image appear (e.g., the tiny yellow flowers shown in Guo’s shadow-free image in Fig. 6(b2)). Close-ups of the red rectangle (Fig. 6(b2)) show that some of the grasses are blurred in Yang’s method [24] due to bilateral filtering, while our method, without any filtering operation, retains the texture well.

Some other comparison results are shown in Fig. 7. The images in Fig. 7 show that our method removes shadows effectively from a single image with texture well-preserved. However, Guo’s method [15] fails to recover a shadow-free image either due to the wrong detection on shadow regions (Figs. 7(b), 7(d), and 7(e)) or the inappropriate relighting on pixels (Figs. 7(a) and 7(c)). Due to the complex of image reflectance and the irregular distribution of shadows in Fig. 7(e), none of the shadow regions are correctly detected in Guo’s shadow-free image. For images with dark shadows or in which shadows pervade in a scenario (Figs. 7(a)–7(e)), Yang’s method [15] blurs the shadow-free images. For example, the clear water in the original image in Fig. 7(e) is seriously blurred in Yang’s method. Some small and tiny objects such as leaves are also removed in Yang’s results.

Figures 6 and 7 verify that our method removes shadows more effectively from a single image, and performs steadily on images with complex outdoor scenes. Our shadow-free image also better preserves the original texture compared to the other two methods. In running time, our method also outperforms Guo’s [15] and Yang’s [24]. In Table 2, we compare the running time of these three methods on the images shown in Fig. 7. All experiments were conducted on a computer with Intel (R) Core (TM) i3-2120 3.3GHz CPU and 2G RAM memory. The source codes that Guo and Yang provided are implemented in Matlab and C++ respectively. For better comparison, we implement our method both in Matlab and C++. Without any statistical learning or complex filtering operation, our method is faster than Guo’s [15] and Yang’s [24]. Note that, when the reflectance of image becomes complex and shadows on the image get intricate, it will be extremely hard for Guo’s method to detect shadow regions and then remove them, and the running time rose up to 6.4827e+003 second, almost 2 hours in Fig. 7(e). All these tested examples and quantitative results show that our method performs better in terms of accuracy, consistency, and efficiency than the other two methods on shadow images. Table 2

Table 2. Comparison of running time of the three methods on images shown in Fig. 7 (measured in seconds)

View Table | View all tables in this article

Comparison with shadow removal method with human intervention. Apart from previous comparisons with full-automatic methods for shadow-free image, in this section, we also compare our method with a shadow removal method [14, 30]. In Arbel’s method, the shadow mask is firstly extracted based on region growing and SVM learning with the user inputs (Figs. 8(b) and 8(c)). Then the shadow-free image is constructed by multiplying a constant scalar to cancel the effect of shadows in umbra regions, while on penumbra regions, the shadows are removed based on the intensity surface construction.

Figure 8 gives the comparison of our method with Arbel’s method on shadow-free image. Figure 8(r1,e) shows that even though Arbel’s method effectively removes shadows from image with dark shadows, the texture in the original shadow area is not consistent with that in the non-shadow area. Figures 8(r2,d) and 8(r3,d) further show that Arbel’s method fails to reconstruct a shadow-free image from images with complex shadows. In these complex scenarios, the constant scalar in the umbra area in Arbel’s method no longer works. These wrong constant scalar relights the shadow regions either excessively or insufficiently. As a contrast, our method can not only remove the shadows, but also keep the texture consistency between the shadow and non-shadow area as shown in Figs. 8(r1,d), 8(r2,e), and 8(r3,e).

4.2. The “intrinsic” of color illumination invariant image

Both Guo’s [15] method, Yang’s [24] method, and Arbel’s method [14, 30] are designed for removing shadows from an image. They cannot yield an identical reflectivity result regardless of illumination condition. In this section we will show that our color illumination invariant image has intrinsic feature. We apply our algorithm to a series of images taken in outdoor scenes at different times on a sunny day to testify the intrinsic feature of our color illumination invariant image.

Shown in Fig. 9(r1), these tested images were taken under four different illumination conditions, one under total daylight (Fig. 9(r1,a)), two under partly daylight and partly skylight (Figs. 9(r1,b) and 9(r1,c)), and one under total skylight (Fig. 9(r1,d)). Taking the image under total daylight as reference image, we compare the differences between the corresponding processed results by different methods. For an intrinsic image derivation algorithm, the difference of the reference intrinsic image and other three intrinsic images should be approximately the same or at least much closer to each other than the original image sequences. In our experiment, we calculated the differences both for our color illumination invariant images and shadow-free images. As a comparison, we also present the intrinsic images obtained by Shen et al. [26] and the corresponding differences. The root mean square error (RMSE) is used as the measurement. Since being compressed, the range of our color illumination is much lower compared with original images and shadow-free images. Therefore, besides RMSE, we also adopt Relative Error ( $\frac{\text{RMSE}}{\text{The}\hspace{0.17em}\text{Range}\hspace{0.17em}\text{of}\hspace{0.17em}\text{Measured}\hspace{0.17em}\text{Image}}$) as our measurement. All these measured images are 24 bit RGB images.

As shown in Fig. 9(r3), the four color illumination invariant images generated by our method are basically the same. It verifies that our orthogonal decomposition on an image yields an identical reflectivity result regardless of illumination condition. The quantitative measurement is given in Table 3. Taken the images under total daylight and total skylight as an example, the RMSE difference of the original images is decreased from 101.85 to 4.67 in our color illumination invariant images. Also, when considering the Relative Error, this value is decreased from 39.94% to 5.7%. Even after a color restoration operation on illumination invariant images, the RMSE difference (24.88) between our shadow-free images is still much smaller than original images. However, the intrinsic images obtained by Shen et al. [26] still have a big difference, 86.36. The quantitative measurements demonstrate that our color illumination invariant images are considerably closer to generate an intrinsic representation than Shen et al. [26]. The corresponding shadow-free images also maintain this “intrinsic” to some extent.

Table 3. MSE and relative error between images

View Table | View all tables in this article

5. Discussion and conclusion

In this paper, we propose a novel, effective and fast method to obtain a shadow-free image from a single outdoor image. Different from state-of-the-art methods that either need shadow detection or statistical learning, our shadow-free image is obtained by a pixel-wise orthogonal decomposition and color restoration. Also, our shadow-free image better preserves the texture information compared to the state-of-the-art method. Intrinsic to the illumination condition, our color illumination invariant image is useful for relighting by adjusting α information. Besides, only requiring pixel-wise calculation, our method for color illumination invariant image and shadow-free image can be further applied to the real-time applications to resist illumination variation.

Despite that our pixel-wise orthogonal decomposition is designed for outdoor shadow images, it is also valid for indoor situation by the correction of physical parameters β₁,β₂,β₃. Figure 10(a) gives two results for our shadow-free images on the indoor shadow images. It shows that our pixel-wise orthogonal decomposition method can be also applied in indoor situation. According to our experiments, we also found two main limitations of our method: 1) Shadow-free image has somewhat color distortion, especially near neutral surface; 2) It does not work well on images with overexposure regions. The main reason for the failure on over-exposure regions is: the pixel values on overexposure region are not physically correct values, they do not satisfy the linear model and we cannot get their correct orthogonal decomposition. Figure 10(b) shows two failure examples for our shadow-free images. The modification and improvement of the model and method for an over-exposed regions, indoor image and more accurate color restoration are in process.

 

Fig. 10 (a) Two original indoor images and the corresponding shadow-free images. (b) Two failure cases: original images with over-exposed regions and the corresponding shadow-free images.

Download Full Size | PDF

See Appendix for supporting content.

Appendix A. The uniqueness of particular solution u_p

The Theorem. Let A denote a 3 by 3 matrix, rank(A) = 2 and b be a non-zero three dimensional vector. For linear equations

(26)$$A\mathit{u}=\mathit{b}$$

there exists a unique particular solution u_p of Eq. (26) that is perpendicular to its normalized free solution.

Proof. 1) Proof of the existence

Due to rank(A) = 2 and b ≠ 0, we know that the dimension of the nullspace of Eq. (26) is one. Let u_s be a particular solution of Eq. (26) and u₀ be the normalized free solution of Eq. 26, i.e., Au_s = b, Au₀ = 0 and ‖ u₀ ‖ = 1. We obtain a new solution of Eq. (26) by

(27)$${\mathit{u}}_{\mathit{p}}={\mathit{u}}_{\mathit{s}}-〈{\mathit{u}}_{\mathit{s}},{\mathit{u}}_{\mathbf{0}}〉{\mathit{u}}_0$$

By inner product operation on both sides of Eq. (27) with u₀, we have

$$\begin{array}{ll}〈{\mathit{u}}_{\mathit{p}},{\mathit{u}}_{\mathbf{0}}〉\hfill & =〈{\mathit{u}}_{\mathit{s}},{\mathit{u}}_{\mathbf{0}}〉-〈{\mathit{u}}_{\mathit{s}},{\mathit{u}}_{\mathbf{0}}〉〈{\mathit{u}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}}〉\hfill \\ \hfill & =〈{\mathit{u}}_{\mathit{s}},{\mathit{u}}_{\mathbf{0}}〉-〈{\mathit{u}}_{\mathit{s}},{\mathit{u}}_{\mathbf{0}}〉\hfill \\ \hfill & =0\hfill \end{array}$$

Then we have u_p ⊥ u₀. Because that the dimension of the nullspace of Eq. (26) is one, we get that u_p is perpendicular to any non-zero free solution of Eq. (26).

2) Proof of the uniqueness

Similar due to rank(A) = 2 and b ≠ 0, the solution space of Eq. (26) can be decomposed into a particular solution plus one-dimensional nullspace solutions. Without loss of generality, for u_p in Eq. (27) and the normalized free solution u₀, any solution u of Eq. (26) can be expressed as following,

(28)$$\mathit{u}={\mathit{u}}_{\mathit{p}}+a{\mathit{u}}_{\mathbf{0}}$$

where a ∈ R. Suppose there exists another particular solution u′_p of Eq. (26), such that u′_p⊥ u₀ and u′_p ≠ u_p. According to Formula 28, we can have

(29)$${\mathit{u}}_{\mathit{p}}^{\prime }={\mathit{u}}_{\mathit{p}}+a\prime {\mathit{u}}_{\mathbf{0}}$$

By inner product operation on both sides of Formula 29 with u₀, we have

$$〈{\mathit{u}}_{\mathit{p}}^{\prime },{\mathit{u}}_{\mathbf{0}}〉=〈{\mathit{u}}_{\mathit{p}},{\mathit{u}}_{\mathbf{0}}〉+a^{\prime }〈{\mathit{u}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}}〉$$

Because u_p ⊥ u₀, u′_p⊥ u₀, and ‖ u₀ ‖ = 1, we obtain 0 = 0 + a′. Then a′ = 0.

From Eq. (29) and a′ = 0, we can get that the solution of Eq. (26), which is perpendicular to any non-zero free solution of Eq. (26), is unique.

For the case b= 0, we set u_p = 0.

Appendix B. Our results on more outdoor images

In this section we show our results on more outdoor images. For each case, we provide the original image, color illumination invariant image, α information, and the final shadow-free image.

 

Fig. 11 Our results on some cast shadow images. Both images with soft shadows and dark shadows are given.

Download Full Size | PDF

 

Fig. 12 Our results on more complex outdoor scenes, which mainly contain cast shadows and self shadows.

Download Full Size | PDF

Acknowledgments

This work was supported by the Natural Science Foundation of China under Grant No. 61102116, 61473280, and 61333019. We thank Eli Arbel for his provision of the source code of [14, 30].

References and links

1. E. H. Land and J. J. McCann, “Lightness and retinex theory,” J. Opt. Soc. Am. A 61, 1–11 (1971). [CrossRef]  

2. Y. Weiss, “Deriving intrinsic images from image sequences,” in Proc. ICCV, (IEEE, 2001), pp. 68–75.

3. G. D. Finlayson and S. D. Hordley, “Color constancy at a pixel,” J. Opt. Soc. Am. A 18, 253–264 (2001). [CrossRef]  

4. S.-S. Lin, K. M. Yemelyanov, E. N. Pugh Jr, and N. Engheta, “Separation and contrast enhancement of overlapping cast shadow components using polarization,” Opt. Express 14, 7099–7108 (2006). [CrossRef]   [PubMed]  

5. C. Jung, W. Kim, and C. Kim, “Detecting shadows from a single image,” Opt. Lett. 36, 4428–4430 (2011). [CrossRef]   [PubMed]  

6. J. T. Barron and J. Malik, “Intrinsic scene properties from a single rgb-d image,” in Proc. CVPR, (IEEE, 2013), pp. 17–24.

7. G. Finlayson, C. Fredembach, and M. S. Drew, “Detecting illumination in images,” in Proc. ICCV, (IEEE, 2007), pp. 1–8.

8. P.-Y. Laffont, A. Bousseau, S. Paris, F. Durand, and G. Drettakis, “Coherent intrinsic images from photo collections,” ACM Trans. Graph.31, (2012). [CrossRef]  

9. P. Laffont, A. Bousseau, and G. Drettakis, “Rich intrinsic image decomposition of outdoor scenes from multiple views,” IEEE Trans. Vis. Comput. Graphics 19, 210–224 (2013). [CrossRef]  

10. Y. Matsushita, K. Nishino, K. Ikeuchi, and M. Sakauchi, “Illumination normalization with time-dependent intrinsic images for video surveillance,” IEEE Trans. Pattern Anal. Machine Intell. 26, 1336–1347 (2004). [CrossRef]  

11. I. Huerta, M. Holte, T. Moeslund, and J. Gonzalez, “Detection and removal of chromatic moving shadows in surveillance scenarios,” in Proc. ICCV, (IEEE, 2009), pp. 1499–1506.

12. T.-P. Wu and C.-K. Tang, “A bayesian approach for shadow extraction from a single image,” in Proc. ICCV, (IEEE, 2005), pp. 480–487.

13. A. Bousseau, S. Paris, and F. Durand, “User-assisted intrinsic images,” ACM Trans. Graph. 28, 130 (2009). [CrossRef]  

14. E. Arbel and H. Hel-Or, “Shadow removal using intensity surfaces and texture anchor points,” IEEE Trans. Pattern Anal. Machine Intell. 33, 1202–1216 (2011). [CrossRef]  

15. R. Guo, Q. Dai, and D. Hoiem, “Paired regions for shadow detection and removal,” IEEE Trans. Pattern Anal. Machine Intell. 35, 2956–2967 (2013). [CrossRef]  

16. J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan, “Detecting ground shadows in outdoor consumer photographs,” in “Proc. ECCV,” (Springer, 2010), pp. 322–335.

17. M. F. Tappen, W. T. Freeman, and E. H. Adelson, “Recovering intrinsic images from a single image,” IEEE Trans. Pattern Anal. Machine Intell. 27, 1459–1472 (2005). [CrossRef]  

18. J. Zhu, K. G. Samuel, S. Z. Masood, and M. F. Tappen, “Learning to recognize shadows in monochromatic natural images,” in “Proc. CVPR,” (IEEE, 2010), pp. 223–230.

19. G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew, “On the removal of shadows from images,” IEEE Trans. Pattern Anal. Machine Intell. 28, 59–68 (2006). [CrossRef]  

20. G. D. Finlayson, M. S. Drew, and C. Lu, “Entropy minimization for shadow removal,” Int. J. Comput. Vis. 85, 35–57 (2009). [CrossRef]  

21. J. Tian, J. Sun, and Y. Tang, “Tricolor attenuation model for shadow detection,” IEEE Trans. Image Process. 18, 2355–2363 (2009). [CrossRef]   [PubMed]  

22. J. Tian and Y. Tang, “Linearity of each channel pixel values from a surface in and out of shadows and its applications,” in “Proc. CVPR,” (IEEE, 2011), pp. 985–992.

23. R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision (McGraw-Hill, Inc., 1995).

24. Q. Yang, K. Tan, and N. Ahuja, “Shadow removal using bilateral filtering,” IEEE Trans. Image Process. 21, 4361–4368 (2012). [CrossRef]   [PubMed]  

25. F. Liu and M. Gleicher, “Texture-consistent shadow removal,” in Proc. ECCV, (Springer, 2008), pp. 437–450.

26. J. Shen, X. Yang, Y. Jia, and X. Li, “Intrinsic images using optimization,” in Proc. CVPR, (IEEE, 2011), pp. 3481–3487.

27. B. A. Maxwell, R. M. Friedhoff, and C. A. Smith, “A bi-illuminant dichromatic reflection model for understanding images,” in Proc. CVPR, (IEEE, 2008), pp. 1–8.

28. M. S. Drew, G. D. Finlayson, and S. D. Hordley, “Recovery of chromaticity image free from shadows via illumination invariance,” in Proc. ICCV, (IEEE, 2003), pp. 32–39.

29. S. A. Shafer, “Using color to separate reflection components,” Color Res. Appl. 10, 210–218 (1985). [CrossRef]  

30. E. Arbel and H. Hel-Or, “Texture-preserving shadow removal in color images containing curved surfaces.” in Proc. CVPR, (IEEE, 2007), pp. 1–8.