Abstract

Imagery from microgrid polarimeters is obtained by using a mosaic of pixel-wise micropolarizers on a focal plane array (FPA). Each distinct polarization image is obtained by subsampling the full FPA image. Thus, the effective pixel pitch for each polarization channel is increased and the sampling frequency is decreased. As a result, aliasing artifacts from such undersampling can corrupt the true polarization content of the scene. Here we present the first multi-channel multi-frame super-resolution (SR) algorithms designed specifically for the problem of image restoration in microgrid polarization imagers. These SR algorithms can be used to address aliasing and other degradations, without sacrificing field of view or compromising optical resolution with an anti-aliasing filter. The new SR methods are designed to exploit correlation between the polarimetric channels. One of the new SR algorithms uses a form of regularized least squares and has an iterative solution. The other is based on the faster adaptive Wiener filter SR method. We demonstrate that the new multi-channel SR algorithms are capable of providing significant enhancement of polarimetric imagery and that they outperform their independent channel counterparts.

© 2011 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006).
    [CrossRef] [PubMed]
  2. B. M. Ratliff, C. F. LaCasse, and J. S. Tyo, “Interpolation strategies for reducing IFOV artifacts in microgrid polarimeter imagery,” Opt. Express 17(11), 9112–9125 (2009).
    [CrossRef] [PubMed]
  3. J. S. Tyo, C. F. LaCasse, and B. M. Ratliff, “Total elimination of sampling errors in polarization imagery obtained with integrated microgrid polarimeters,” Opt. Lett. 34(20), 3187–3189 (2009).
    [CrossRef] [PubMed]
  4. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
    [CrossRef]
  5. B. M. Ratliff, J. S. Tyo, W. T. Black, and C. F. LaCasse, “Exploiting motion-based redundancy to enhance microgrid polarimeter imagery,” Proc. SPIE 7461, 74610K (2009).
    [CrossRef]
  6. D. A. Lemaster, “Resolution enhancement by image fusion for microgrid polarization imagers,” in IEEE Aerospace Conference, Big Sky, MT (2010).
  7. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
    [CrossRef]
  8. R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16(12), 2953–2964 (2007).
    [CrossRef] [PubMed]
  9. R. M. A. Azzam and N. M. Bashara, Ellipsometry and Polarized Light (North-Holland, 1977).
  10. J. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1968).
  11. R. R. Shannon, “Aberrations and their effect on images,” Proc. SPIE 531, 27–37 (1985).
  12. T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” in IEEE Conference on Computer Vision and Pattern Recognition , vol. 2, pp. 600–607 (Los Alamitos, CA, 2004).
  13. S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
    [CrossRef] [PubMed]
  14. L. Condat, “A generic variational approach for demosaicking from an arbitrary color filter array,” in Proceedings of IEEE ICIP , pp. 1625–1628 (2009).
  15. R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
    [CrossRef] [PubMed]
  16. B. M. Ratliff, J. S. Tyo, J. K. Boger, W. T. Black, D. L. Bowers, and M. P. Fetrow, “Dead pixel replacement in LWIR microgrid polarimeters,” Opt. Express 15(12), 7596–7609 (2007).
    [CrossRef] [PubMed]

2010 (1)

D. A. Lemaster, “Resolution enhancement by image fusion for microgrid polarization imagers,” in IEEE Aerospace Conference, Big Sky, MT (2010).

2009 (4)

B. M. Ratliff, C. F. LaCasse, and J. S. Tyo, “Interpolation strategies for reducing IFOV artifacts in microgrid polarimeter imagery,” Opt. Express 17(11), 9112–9125 (2009).
[CrossRef] [PubMed]

J. S. Tyo, C. F. LaCasse, and B. M. Ratliff, “Total elimination of sampling errors in polarization imagery obtained with integrated microgrid polarimeters,” Opt. Lett. 34(20), 3187–3189 (2009).
[CrossRef] [PubMed]

B. M. Ratliff, J. S. Tyo, W. T. Black, and C. F. LaCasse, “Exploiting motion-based redundancy to enhance microgrid polarimeter imagery,” Proc. SPIE 7461, 74610K (2009).
[CrossRef]

L. Condat, “A generic variational approach for demosaicking from an arbitrary color filter array,” in Proceedings of IEEE ICIP , pp. 1625–1628 (2009).

2007 (2)

2006 (2)

J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006).
[CrossRef] [PubMed]

S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
[CrossRef] [PubMed]

2003 (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

1998 (1)

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

1997 (1)

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[CrossRef] [PubMed]

1985 (1)

R. R. Shannon, “Aberrations and their effect on images,” Proc. SPIE 531, 27–37 (1985).

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[CrossRef] [PubMed]

Azzam, R. M. A.

R. M. A. Azzam and N. M. Bashara, Ellipsometry and Polarized Light (North-Holland, 1977).

Barnard, K. J.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[CrossRef] [PubMed]

Bashara, N. M.

R. M. A. Azzam and N. M. Bashara, Ellipsometry and Polarized Light (North-Holland, 1977).

Black, W. T.

B. M. Ratliff, J. S. Tyo, W. T. Black, and C. F. LaCasse, “Exploiting motion-based redundancy to enhance microgrid polarimeter imagery,” Proc. SPIE 7461, 74610K (2009).
[CrossRef]

B. M. Ratliff, J. S. Tyo, J. K. Boger, W. T. Black, D. L. Bowers, and M. P. Fetrow, “Dead pixel replacement in LWIR microgrid polarimeters,” Opt. Express 15(12), 7596–7609 (2007).
[CrossRef] [PubMed]

Boger, J. K.

Bognar, J. G.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

Bowers, D. L.

Chenault, D. B.

Condat, L.

L. Condat, “A generic variational approach for demosaicking from an arbitrary color filter array,” in Proceedings of IEEE ICIP , pp. 1625–1628 (2009).

Elad, M.

S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
[CrossRef] [PubMed]

Farsiu, S.

S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
[CrossRef] [PubMed]

Fetrow, M. P.

Goldstein, D. L.

Goodman, J.

J. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1968).

Gotoh, T.

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” in IEEE Conference on Computer Vision and Pattern Recognition , vol. 2, pp. 600–607 (Los Alamitos, CA, 2004).

Hardie, R.

R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16(12), 2953–2964 (2007).
[CrossRef] [PubMed]

Hardie, R. C.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[CrossRef] [PubMed]

Kang, M. G.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

LaCasse, C. F.

Lemaster, D. A.

D. A. Lemaster, “Resolution enhancement by image fusion for microgrid polarization imagers,” in IEEE Aerospace Conference, Big Sky, MT (2010).

Milanfar, P.

S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
[CrossRef] [PubMed]

Okutomi, M.

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” in IEEE Conference on Computer Vision and Pattern Recognition , vol. 2, pp. 600–607 (Los Alamitos, CA, 2004).

Park, M. K.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Park, S. C.

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

Ratliff, B. M.

Shannon, R. R.

R. R. Shannon, “Aberrations and their effect on images,” Proc. SPIE 531, 27–37 (1985).

Shaw, J. A.

Tyo, J. S.

Watson, E. A.

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

Appl. Opt. (1)

IEEE Aerospace Conference, Big Sky, MT (1)

D. A. Lemaster, “Resolution enhancement by image fusion for microgrid polarization imagers,” in IEEE Aerospace Conference, Big Sky, MT (2010).

IEEE Signal Process. Mag. (1)

S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003).
[CrossRef]

IEEE Trans. Image Process. (3)

R. Hardie, “A fast image super-resolution algorithm using an adaptive Wiener filter,” IEEE Trans. Image Process. 16(12), 2953–2964 (2007).
[CrossRef] [PubMed]

S. Farsiu, M. Elad, and P. Milanfar, “Multi-frame demosaicing and super-resolution of color images,” IEEE Trans. Image Process. 15, 141–159 (2006).
[CrossRef] [PubMed]

R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6(12), 1621–1633 (1997).
[CrossRef] [PubMed]

Opt. Eng. (1)

R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, “High resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system,” Opt. Eng. 37(1), 247–260 (1998).
[CrossRef]

Opt. Express (2)

Opt. Lett. (1)

Proc. SPIE (2)

B. M. Ratliff, J. S. Tyo, W. T. Black, and C. F. LaCasse, “Exploiting motion-based redundancy to enhance microgrid polarimeter imagery,” Proc. SPIE 7461, 74610K (2009).
[CrossRef]

R. R. Shannon, “Aberrations and their effect on images,” Proc. SPIE 531, 27–37 (1985).

Proceedings of IEEE ICIP (1)

L. Condat, “A generic variational approach for demosaicking from an arbitrary color filter array,” in Proceedings of IEEE ICIP , pp. 1625–1628 (2009).

Other (3)

T. Gotoh and M. Okutomi, “Direct super-resolution and registration using raw CFA images,” in IEEE Conference on Computer Vision and Pattern Recognition , vol. 2, pp. 600–607 (Los Alamitos, CA, 2004).

R. M. A. Azzam and N. M. Bashara, Ellipsometry and Polarized Light (North-Holland, 1977).

J. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1968).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (15)

Fig. 1
Fig. 1

Microgrid pattern for the polarimetric imager considered here. Four polarization images are acquired using a 2 × 2 repeating pattern of micropolarizers on the FPA.

Fig. 2
Fig. 2

Block diagram illustrating the observation model relating p to the observed frames g(k), for k = 1,2,...,K.

Fig. 3
Fig. 3

Alternative observation model valid for translational motion (and rotational motion when the PSF is circularly symmetric). Here the motion model is commuted with the PSF and integrated into what becomes a nonuniform sampling operator.

Fig. 4
Fig. 4

(a) Cross sections of theoretical 2-D MTF and its components for the microgrid polarimetric imager used here. (b) Theoretical continuous PSF for L = 4.

Fig. 5
Fig. 5

Observation window for the microgrid AWF algorithm for polarimetric data with L = 4. For simplicity, the window is shown populated with pixels from only a single frame.

Fig. 6
Fig. 6

AWF filter weights for K = 1 frame, P = 4 channels, Wx = Wy = 10 observation window, Dx = Dy = 2 estimation window and L = 2 upsampling (single frame demosaicing with restoration). The weights correspond to estimating the pixel location with the red plus sign. The columns from left to right correspond to polarization angles of θ = 0°, 45°, 90° and 135°, respectively. The rows from top to bottom correspond to α = 0, 0.7, 1.0.

Fig. 7
Fig. 7

LWIR microgrid imagery results. (a) Raw microgrid image with bad pixel mask shown in red. Estimates of p 0 using an upsampling factor of L = 4 are shown for (b) bicubic interpolation of polarimetric channel g 0(1) (c) Tyo et al method from [3] (d) LSF method (e) single frame AWF with correlation model parameters of ρ = α = 0.7 (f) single frame RLS.

Fig. 8
Fig. 8

LWIR microgrid imagery results continued. (a) Single frame AWF with α = 0.0 (b) single frame AWF with α = 1.0 (c) independent-channel single-frame RLS (d) independent-channel 20 frame RLS (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Fig. 9
Fig. 9

Microgrid RLS SR algorithm cost function and its components for the single frame LWIR result in Fig. 7(e).

Fig. 10
Fig. 10

DoLP images for LWIR microgrid data. (a) Bicubic interpolation (b) Tyo et al (c) LSF method (d) independent-channel 20 frame RLS (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Fig. 11
Fig. 11

Visible polarimetric data with simulated microgrid sampling. (a) Simulated microgrid frame (b) true {s 0, j } image for an upsampling factor of L = 4 (c) estimate using bicubic interpolation (d) Tyo et al method (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Fig. 12
Fig. 12

Cross-sections of the true image {s 0, j } and corresponding estimates along Row 234 and about Column 67 from Fig. 11. These plots represent approximate line spread functions and provide some insight into the resolution of the various estimates.

Fig. 13
Fig. 13

DoLP images for simulated microgrid data. (a) True DoLP image (b) bicubic interpolation (c) Tyo et al method (d) LSF method (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Fig. 14
Fig. 14

MSE versus number of input frames for the Stokes images from the simulated microgrid imagery (a) {s 0, j } (b) {s 1, j } (c) {s 2, j }.

Fig. 15
Fig. 15

MSE versus α for the 20 frame microgrid AWF output with ρ = 0.7.

Equations (40)

Equations on this page are rendered with MathJax. Learn more.

s 0 , j = ( p 0 , j + p 45 , j + p 90 , j + p 135 , j ) / 2 s 1 , j = p 0 , j p 90 , j s 2 , j = p 45 , j p 135 , j .
DoLP j = s 1 , j 2 + s 2 , j 2 s 0 , j .
g θ , i ( k ) = j = 1 N h θ , i , j ( k ) p θ , j + n θ , i ( k ) ,
g = Hp + n ,
H ( u , v ) = H dif ( u , v ) H abr ( u , v ) H det ( u , v ) ,
H dif ( u , v ) = { 2 π [ cos 1 ( ρ / ρ c ) ( ρ / ρ c ) 1 ( ρ / ρ c ) 2 ] for ρ < ρ c 0 else ,
H abr ( u , v ) = { 1 ( W RMS / 0.18 ) 2 ( 1 4 ( ρ / ρ c 0.5 ) ) 2 for ρ < ρ c 0 else ,
p ^ = argmin p C ( p ) .
C ( p ) = 1 σ n 2 θ k = 1 K i = 1 M ( g θ , i ( k ) j = 1 N h θ , i , j ( k ) p θ , j ) 2 + m = 0 2 1 σ s m 2 i = 1 N ( j = 1 N w i , j s m , j ) 2 . + 1 σ r 2 j = 1 N ( p 0 , j + p 90 , j p 45 , j p 135 , j ) 2
w i , j = { 1 i = j 1 / 4 j is cardinal neighbor of i .
p C ( p ) = [ p 0 C ( p ) T , p 45 C ( p ) T , p 90 C ( p ) T , p 135 C ( p ) T ] T ,
p θ C ( p ) = [ C ( p ) p θ , 1 , C ( p ) p θ , 2 , , C ( p ) p θ , N , ] T .
C ( p ) p 0 , q = H ( 0 , q ) + S ( 0 , q ) + S ( 1 , q ) + R ( q ) ,
C ( p ) p 45 , q = H ( 45 , q ) + S ( 0 , q ) + S ( 2 , q ) R ( q ) ,
C ( p ) p 90 , q = H ( 90 , q ) + S ( 0 , q ) S ( 1 , q ) + R ( q ) ,
C ( p ) p 135 , q = H ( 135 , q ) + S ( 0 , q ) S ( 2 , q ) R ( q ) ,
H ( θ , q ) = 2 σ n 2 k = 1 K i = 1 M h θ , i , q ( k ) e θ , i ( k ) ,
e θ , i ( k ) = j = 1 N h θ , i , j ( k ) p θ , j g θ , i ( k ) .
S ( m , q ) = 2 σ s m 2 i = 1 N w i , q ( j = 1 N w i , j s m , j ) .
R ( q ) = 2 σ r 2 ( p 0 , q + p 90 , q p 45 , q p 135 , q ) .
p ^ n = p ^ n 1 + ɛ n p C ( p ) | p = p ^ n 1 ,
ɛ n = N 1 + N 2 + N 3 D 1 + D 2 + D 3 .
N 1 = 1 σ n 2 θ k = 1 K i = 1 M ( j = 1 N h θ , i , j ( k ) γ θ , j ) e θ , i ( k ) ,
N 2 = m = 0 2 1 σ s m 2 i = 1 N ( j = 1 N w i , j s m , j ) ( j = 1 N w i , j S m , j ) ,
N 3 = 1 σ r 2 j = 1 N ( p 0 , j + p 90 , j p 45 , j p 135 , j ) ( γ 0 , j + γ 90 , j γ 45 , j γ 135 , j ) ,
D 1 = 1 σ n 2 θ k = 1 K i = 1 M ( j = 1 N h θ , i , j ( k ) γ θ , j ) 2 ,
D 2 = m = 0 2 1 σ s m 2 i = 1 N ( j = 1 N w i , j S m , j ) 2 ,
D 3 = 1 σ r 2 j = 1 N ( γ 0 , j + γ 90 , j γ 45 , j λ 135 , j ) 2 ,
γ θ , j = C ( p ) p θ , j | p = p ^ n
S 0 , j = ( γ 0 , j + γ 45 , j + γ 90 , j + γ 135 , j ) / 2 S 1 , j = γ 0 , j γ 90 , j S 2 , j = γ 45 , j γ 135 , j .
d ^ i = W i T g i ,
W i = R i 1 P i ,
R i = E { g i g i T } = E { f i f i T } + σ n 2 I
P i = E { g i d i T } = E { f i d i T } .
r d θ d ϕ ( x , y ) = σ θ , ϕ 2 ρ θ , ϕ x 2 + y 2 ,
σ θ , ϕ 2 = { σ p 2 θ = ϕ α σ p 2 otherwise .
r d θ f ϕ ( x , y ) = r d θ d ϕ ( x , y ) * h ( x , y ) .
r f θ f ϕ ( x , y ) = r d θ d ϕ ( x , y ) * h ( x , y ) * h ( x , y ) .
b ( i , j ) = a ( i , j ) ā ( i , j ) ã ( i , j ) .
b ( i , j , k ) = { a ( i , j ) j = k ã j ( i , k ) a ( i , j ) ā ( i , j ) ã ( i , j ) + ā j ( i , k ) otherwise .

Metrics