Open Access
Add to BibSonomyAbstract
The Air Force Research Laboratory has developed a new microgrid polarization imaging system capable of simultaneously reconstructing linear Stokes parameter images in two colors on a single focal plane array. In this paper, an effective method for extracting Stokes images is presented for this type of camera system. It is also shown that correlations between the color bands can be exploited to significantly increase overall spatial resolution. Test data is used to show the advantages of this approach over bilinear interpolation. The bounds (in terms of available reconstruction bandwidth) on image resolution are also provided.
©2011 Optical Society of America
1. Introduction
Passive Polarimetric Imaging (PI) is a remote sensing discipline that exploits the vector properties of incoherent electromagnetic radiation. Among other applications, passive PI has been used to address problems in clutter suppression, material characterization, and surface orientation estimation. A recent survey of this topic area is provided in [1] and addressed in several texts, including [2].
Polarization states are not sensed directly at optical and infrared wavelengths. Instead, they are inferred from combinations of polarization sensitive intensity measurements. For present purposes, the scope of polarization imaging is limited to inferring linear polarization states. These states can be described in terms of the first three Stokes parameter images with parameter S 0 representing total intensity and parameters S 1 and S 2 containing all scene polarization information. The exact relationship between the intensity channels and Stokes parameters will be defined fully in Section 2.
In single camera passive polarimetric imaging schemes, these multiple measurements are achieved by sacrificing either temporal resolution, spatial resolution, or field-of-view. Among systems that sacrifice spatial resolution, microgrid polarization imagers find favor with us because all measurements of the scene state are simultaneous, channel alignment is fixed permanently, and microgrid focal plane arrays are compatible with many existing camera optical designs. These advantages are achieved in microgrid systems via a repeating pattern of polarization analyzers that are matched and bonded to electro-optical detectors in a manner similar to Bayer color filter arrays. In this way, the overall spatial resolution of a microgrid system is strongly influenced by the spacing between like analyzer samples in addition to simple detector-to-detector spacing.
Until now, the color sensitivity of microgrid imaging systems has been fixed globally by filters external to the array itself. In [3], we describe the first microgrid imaging system with both color and polarization sensitivity at the detector array level. In what follows, a highly effective method for extracting color sensitive Stokes parameter images from the raw two-color microgrid data is presented, bounds on reconstructed image resolution are provided, and it is shown that sensor mission flexibility may be increased further by trading back color sensitivity for additional spatial resolution. In addition, example Stokes image reconstructions are provided for both cases along with a quantitative comparison to a common image space reconstruction method.
As stated previously, a characteristic of microgrid systems is that spatial resolution is traded for sensitivity to polarization. Consequently, the Stokes parameter images are formed at a lower resolution than what would be ordinarily achievable at the native sampling of the detector array. In terms of minimizing interpolation errors, the best possible reconstruction of Stokes parameter images from traditional (i.e. one-color) microgrid data is achieved using a linear systems approach as described in [4] (this reference, along with [5], summarizes the prior art). In this new two-color system, additional spatial sampling is sacrificed for spectro-polarimetric sensitivity. To compensate, the work in [4] is generalized here to include terms for the two-color pattern.
In the Discrete Space Fourier Transform (DSFT) domain, the lost samples in the two-color array restrict the bandwidth available for image reconstruction by decreasing the separation between replicas of the original spectra. However, when color sensitivity is not needed or when spatial resolution has higher priority, some of this lost image bandwidth may be regained by exploiting the correlations between color channels to form a higher resolution (but color insensitive) set of Stokes parameter images. As will be shown, this new DSFT interpretation of the two-color microgrid can be used to demonstrate how retrieval of additional bandwidth is possible.
This new two-color microgrid arrangement is shown schematically in Fig. 1. Like a traditional microgrid array, a unit cell in this pattern is referred to as a superpixel. Each superpixel element is a polarization analyzer matched in size to an underlying detector element. Each analyzer element takes on one of four possible orientations: horizontal (H), vertical (V), or ±45 degrees.
All analyzer elements with the same orientation and color are referred to as a channel. Channel images, ix, are combined algorithmically to form Stokes parameter images:
This paper is organized as follows. In Section 2, the linear systems description of the microgrid imager is reviewed for the one-color case and then extended to the two-color case. Next, the two-color description is used to show that, by exploiting the correlation between bands, color sensitivity may be traded back for additional spatial resolution (i.e. resolution beyond what is available in either color channel alone). These theoretical results are then demonstrated in Section 4 where Stokes parameter images are reconstructed from a simulated two-color microgrid test case.
2. Linear systems description of one and two-color microgrid images
In this section, the DSFT representation of the two-color microgrid array is built upon the description of the single-color microgrid array originally proposed in [4]. A review of this previous work is presented first.
2.1. Review of the one-color case
A raw microgrid intensity image consists of a base S 0 image underlying the spatially modulated content in S 1 and S 2. Each image consists of M × N samples. The intensity at each point (m,n) in a raw one-color microgrid array image is given by
where each of the Sx(m, n) terms represent one of the Stokes parameter images defined over domain m = (0 : M – 1) and n = (0 : N – 1). The polarization analyzer pattern in Fig. 1 is assumed. By requiring that the Stokes images are properly sampled, the DSFT of I is given by: where each of ξ and η are real-valued spatial frequency coordinates defined over one period of the spectrum, i.e. −π ≤ ξ, η ≤ π.The physical interpretation of Equation (3) is that the portion of the microgrid DSFT containing polarization information (S 1 and S 2) is modulated to the edges of the spectrum while the DSFT of the total intensity image (S 0) remains centered on the origin. This distribution is shown schematically in Fig. 2a. From this result, the Stokes parameter images may be retrieved through demodulation and low pass filtering on the microgrid DSFT.
The advantage of Tyo’s approach is that it can minimize Stokes image artifacts caused by interpolation error (often referred to as instantaneous field-of-view errors in microgrid literature) through judicious filter shaping, especially when compared to other common techniques such as Nearest Like Polarization Neighbor (NLPN) interpolation, which is the primary test case in the source paper. It is for this reason that the method in [4] is selected for extension to the two-color case.
2.2. Extension to the two-color case
When reconstructing Stokes images for one of the colors in a two-color microgrid array, the deletion of every other superpixel represents a discrete sampling operation which will give rise to folded-in replicas of the original spectrum in the DSFT domain. In this section, the form of this color-deleted spectrum is found and it’s consequences are discussed.
First, some additional notation is required. Define M to be the two-color microgrid intensity array and Ix to be the one-color microgrid intensity image for color x ∈ {1, 2} corresponding to Equation 2. Note that only half of Ix exists in the two-color array. This half is defined to be Mx so that the full two-color array is M = M 1 + M 2. Consequently,
where Cx is the color sampling pattern: and the (−1)(1+x) term is simply an artifice for selecting the correct sign on the bracketed term that follows.The DSFT of Mx may be written as
withConsequently, the DSFT of Mx is a superposition of Ĩx centered at the origin and four off-axis replica spectra. The magnitude of either M̃ 1 (ξ, η) or M̃ 2 (ξ, η) is shown schematically in Fig. 2b. Only the origin centered spectrum is useful for Stokes reconstruction as the replica spectra are inextricably mixed with each other. The Stokes parameter images may be recovered from the center spectrum through demodulation and low pass filtering, subject to bandwidth restrictions as outlined in the next section.
2.3. Conditions to avoid aliasing
The overlap of an image spectrum with a replica of itself is referred to as aliasing. In the two-color microgrid case, this may occur between the centered S 0 spectrum and any one of its off-axis replicas. Similarly, overlap may occur between the spectra of different Stokes images (e.g. S 0 and S 1 ± S 2). The kind of overlap is sometimes referred to as spectral mixing. In both cases, spectral overlap results in undesirable image artifacts. Consistent with [4] and [9] both varieties of spectral overlap are collectively referred to as aliasing in the development that follows.
Tyo et al [4] established a sufficient condition to avoid aliasing in the one-color microgrid Stokes image reconstruction case in terms of a band radius on S 0 and S 1±S 2. This condition can be expressed as
where each rx is the radius, in radians per sample, of the smallest circle that encloses the band limits on S 0 or S 1 ± S 2. As intuition would suggest, the conditions to avoid aliasing are more stringent in the two-color microgrid case because of the replica spectra. To be precise, define rmax to be the larger of r S0 or r S1±S2. In the two-color microgrid case, aliasing is avoided when these two conditions are met: andIt may not be obvious why rmax must be considered in the two-color case. Here, the replica spectra contribute aliased content to each other by virtue of their spacing. For instance, the replica centered at (π/2, π/2) contains S̃ 0 spectral content overlapping with S̃ 1 + S̃ 2 content from the replica at (–π/2, π/2) and S̃ 1 – S̃ 2 content from the replica at (π/2, –π/2). Therefore, aliasing conditions in the two-color case are set by whichever contributor to the replica spectra is widest. Empirically, S 0 appears to be the dominate contributor.
3. Image resolution beyond the limit imposed by color sampling
Under conditions where I 1 and I 2 are highly correlated, the spectral sensitivity of the two-color array may be traded back to suppress the off-axis replica spectra and therefore allow for wider filters capable of providing additional image resolution. This trade of spectral sensitivity for spatial resolution is most useful for providing a high quality context image (i.e. just S 0) over which the lower resolution polarimetric information in the underlying color channels may be overlaid. An example of this kind of image fusion using the method from [8] is presented in the results section.
As stated in Section 2.3, the off-axis replica spectra determine the bandwidth available for Stokes image reconstruction for either color channel in the two-color microgrid array. If these replica spectra can be suppressed or eliminated then additional bandwidth is available to form higher resolution images. To understand how the replica spectra may be suppressed, consider M̃ to be the DSFT of the entire two-color array
and note that the first two terms evaluate as the sum of the on-axis centered spectra while the last two terms are the difference of the replica spectra with centers found off-axis. As should be expected, if Ĩ 1 = Ĩ 2 then Equation (10) reduces to Equation (3), the DSFT of a one-color microgrid image (i.e. the off-axis terms are eliminated entirely). More generally, when the I 1 and I 2 spectra are highly correlated then their difference will tend to cancel and the bandwidth occupied by the off-axis replica spectra will shrink. At the same time, each color channel reinforces the other at the center. As a result, the reconstruction filters used on the center spectrum may be made wider without introducing appreciable aliasing from the (now diminished) replica spectra.It is worth noting that correlations between color bands have been exploited to increase the bandwidth of luminance image estimation in Bayer color filter array processing (see [9] for instance). The extent to which this suppression is effective is scene dependent and so no immutable rules for filter bandwidth (as were provided in Section 2.3) may be defined. A demonstration will be provided in the following section.
4. Stokes reconstruction examples
To demonstrate the proposed Stokes image reconstruction approach, high resolution test data is collected using the red and blue channels of a Nikon D300S SLR equipped with a rotating linear polarizer. These data are registered and resampled into a two-color microgrid test scene. Using synthesized data is preferable for the present purpose because the reconstructed microgrid Stokes images can be compared visually and quantitatively to the higher resolution truth data. For comparison, results achieved using bilinear interpolation are also provided. Bilinear interpolation will always be as good or better than the NLPN method (see Section 2.1) and is representative of all image space reconstruction methods (see the example in [5]) that could be extended from the prior-art in traditional single color microgrid processing. To reduce clutter, only results from the red and color correlated bands are shown below. Results from the blue band are largely the same in terms of the differences between reconstruction methods.
The log magnitude DSFT of the red channel is shown in Fig. 3a. According to the derivation above, the log magnitude of the blue channel will look much the same. Note that all of the regions defined in Fig. 2b are also well defined here and most of the power spectrum is concentrated in the S 0 component. This knowledge is exploited to inform the shapes of the reconstruction filters. In addition, the log magnitude spectrum of the entire two-color array (henceforth referred to as the color correlated case) is provided in Fig. 3b. As predicted in Section 3, the correlation between color bands has suppressed the off-axis replica spectra thereby freeing up additional bandwidth for image reconstruction.
Cross-sections of the circularly symmetric reconstruction filters used in this analysis are shown in Fig. 4. From wide to narrow, the 3db widths of these filters are π, 0.707π, and 0.424π radians/sample. For single color Stokes reconstruction, a filter wider than H 2 (the 0.707π radians/sample case) guarantees that aliasing will be present in the reconstructed image per the rules established in Section 2.3. Each of these filters is used in the reconstruction of one or more Stokes images as shown in Table 1. Two trends demonstrated in this table hold generally (i.e. they are not specific to this data set): (a) S 0 reconstruction benefits most from wider filters without introducing appreciable aliasing and (b) wider filters may always be used in color correlated reconstruction.
Table 1:. Reconstruction filter matrix
The differences between reconstruction methods are borne out quantitatively. Table 2 compares the root mean squared error between the true and reconstructed images. These results are normalized by the average value of the true image so that the red band and color correlated band results may be compared directly. These values should not be compared across Stokes parameter images. For instance, the RMSE for both the S 1 and S 2 cases says something about differences in overall signal level but nothing about relative goodness of fit. RMSE values in S 2 are largest because, in this scene, there is relatively little S 2 signal while noise levels remain consistent across all three Stokes images. In all cases, the color correlated results are best and the bilinear results are worst.
Table 2:. Normalized RMSE for the various reconstruction techniques
The S 0 reconstructions for the red and color correlated bands are shown in Fig. 5. As expected, the red band reconstruction is blurred somewhat when compared to the truth image with a small amount of aliasing present (a good indication that the H 2 filter should be made no wider). This result is slightly softer than the bilinearly interpolated result but it is also less prone to artifacts. The color correlated image is substantially sharper than either of the red band reconstructions. These same patterns follow in the S 1 and S 2 cases (Figs. 6 and 7) with more pronounced differences between reconstruction approaches.
4.1. An application using color fusion
For visualization purposes, it would be useful to retain both the color sensitivity of the red and blue band reconstructions along with the high spatial resolution of the full band reconstruction. An example of how this can be done is shown in Fig. 8 via extension of the Tyo et al HSV color mapping for polarization imagery [8]. In Fig. 8c, the low resolution H 3 reconstructed Stokes parameter images are mapped to saturation and hue (as degree and angle of polarization) while the underlying context image is taken to be the color correlated S 0 image reconstructed with filter H 1. This combination mapping is virtually indistinguishable from the true red band color mapped image shown in 8a. For comparison, the red band color mapped image reconstructed with filter H 2 is shown in Fig. 8b.
5. Conclusion
This paper has demonstrated an effective method for extracting color sensitive Stokes parameter images at low spatial resolution and color insensitive Stokes parameter images at higher spatial resolution from raw two-color microgrid polarization array data. Using a simulated but representative test scene, this method has qualitatively and quantitatively demonstrated fewer image artifacts than bilinear interpolation when reconstructing the color channels individually. Furthermore, the color insensitive case (a.k.a the color correlated case) demonstrates significantly higher spatial resolution than either single color method. The bounds on reconstructed image resolution were also provided. Unlike the traditional one-color microgrid case, replicas of the image spectrum (particularly the S 0 spectrum) dominate the unaliased bandwidth available for image reconstruction filters.
A useful extension to this work would include more intelligent reconstruction filter design. For S 0, the Wiener filter is an obvious choice (even when aliasing is unavoidable [10]) since power spectrum models for natural scenes are well understood. It is not immediately clear, however, that such models extend to S 1 and S 2 which may not be imaged directly.
Another useful extension to this work could include filters that are adaptive to the presence or absence of color correlation and/or the degree of polarization in the image. As an extreme example of how adaptive bandwidth would be useful, consider a scene that is totally unpolarized and fully correlated between color bands. Here, the best filter for the color insensitive S 0 image would encompass the entire spectrum and the resulting image resolution would only be limited by the optics and detector array. Such adaptive techniques exist already for color filter arrays [11]. Either of these extensions are applicable to traditional (one-color) microgrid arrays as well.
Acknowledgments
This paper is written with kind support from Bob Mack, program manager for the Air Force Research Laboratory’s SPITFIRE program. Thanks also to Brad Ratliff of Space Computer Corporation along with Joe Meola and Barry Karch from AFRL for their comments and suggestions. Finally, thanks to the reviewers for their excellent comments including a significant simplification to the derivation in Section 2.2.
References and links
1. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45, 5453–5469 (2006). [CrossRef] [PubMed]
2. J. Schott, Fundamentals of polarimetric remote sensing (SPIE Press, 2009). [CrossRef]
3. D. LeMaster, R. Mack, D. Forrai, J. Harris, B. Ratliff, and J. Middendorf, “A snapshot two-color MWIR polarimetric imaging system,” in “Proceedings of the Military Sensing Symposium (MSS) Specialty Group on Passive Sensors,” (2011).
4. J. S. Tyo, C. F. LaCasse, and B. M. Ratliff, “Total elimination of sampling errors in polarization imagery obtained with integrated microgrid polarimeters,” Opt. Lett. 34, 3187–3189 (2009). [CrossRef] [PubMed]
5. B. M. Ratliff, C. F. LaCasse, and J. S. Tyo, “Interpolation strategies for reducing IFOV artifacts in microgrid polarimeter imagery,” Opt. Express 17, 9112–9125 (2009). [CrossRef] [PubMed]
6. A. V. Oppenheim, A. S. Willsky, and S. H. Nawab, Signals and Systems (Prentice Hall, 1997).
7. D. LeMaster, “Image reconstruction for two-color microgrid polarimetric imagers,” in “Aerospace conference, 2011 IEEE,” (2011), pp. 1 –7.
8. J. Tyo, E. Pugh Jr, and N. Engheta, “Colorimetric representations for use with polarization-difference imaging of objects in scattering media,” JOSA A 15, 367–374 (1998). [CrossRef]
9. D. Alleysson, S. Susstrunk, and J. Herault, “Linear demosaicing inspired by the human visual system,” IEEE Trans. Image Processing 14, 439 –449 (2005). [CrossRef]
10. S. T. Thurman and J. R. Fienup, “Wiener reconstruction of undersampled imagery,” J. Opt. Soc. Am. A 26, 283–288 (2009). [CrossRef]
11. E. Dubois, “Frequency-domain methods for demosaicking of bayer-sampled color images,” IEEE Signal Processing Lett. 12, 847 – 850 (2005). [CrossRef]
