Abstract

The TOMBO system (thin observation module by bound optics) is a multichannel subimaging system over a single electronic imaging device. Each subsystem provides a low-resolution (LR) image from a unique lateral point of view. By estimating the image’s lateral position, a high-resolution (HR) image is restored from the series of the LR images. This paper proposes an multistage algorithm comprised of successive stages, improving difficulties in previous suggested schemes. First, the registration algorithm estimates the subchannel shift parameters and eliminates bias. Second, we introduce a fast image fusion, overcoming visual blockiness artifacts that characterized previously suggested schemes. The algorithm fuses the set of sampled subchannel images into a single image, providing the reconstruction initial estimate. Third, an edge-sensitive quadratic upper bound term to the total variation regulator is suggested. The complete algorithm allows the reconstruction of a clean, HR image, in linear computation time, by the use of the linear conjugate gradient optimization. Finally, we present a simulated comparison between the proposed method and a previously suggested image restoration method. The results show that the proposed method yields better reconstruction fidelity while eliminating spatial speckle artifacts associated with the previously suggested method.

© 2013 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, “Image reconstruction for thin observation module by bound optics by using the iterative back projection method,” Appl. Opt. 45, 2893–2900 (2006).
    [CrossRef]
  2. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kodou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001).
    [CrossRef]
  3. Y. Kitamura, R. Shogenji, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D. Miyazaki, J. Tanida, and Y. Ichioka, “Reconstruction of a high-resolution image on a compound-eye image-capturing system,” Appl. Opt. 43, 1719–1727 (2004).
    [CrossRef]
  4. A. V. Kanaev, J. R. Ackerman, E. F. Fleet, and D. A. Scribner, “TOMBO sensor with scene-independent superresolution processing,” Opt. Lett. 32, 2855–2857 (2007).
    [CrossRef]
  5. M. Shankar, R. Willet, N. Pitslanis, T. Schulz, R. Gibbons, R. T. Kolste, J. Carrier, C. Chen, D. Prather, and D. Brady, “Thin infrared imaging systems through multichannel sampling,” Appl. Opt. 47, B1–B10 (2008).
    [CrossRef]
  6. T. Q. Pham, “Spatiotonal adaptivity in super-resolution of under-sampled image sequences,” Ph.D. dissertation (aan de Technische Universiteit Delft, 2006).
  7. D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Process 13, 1185–1199 (2004).
    [CrossRef]
  8. P. Jorge and S. G. Ferreira, “Interpolation and the discrete Papoulis–Gerchberg algorithm,” IEEE Trans. Signal Process. 42, 2596–2606 (1994).
    [CrossRef]
  9. J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto, and S. Miyamoto, “Color imaging with an integrated compound imaging system,” Opt. Express 11, 2109–2117 (2003).
    [CrossRef]
  10. G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Texture preserving variational denoising using an adaptive fidelity term,” Presented at the VLSM 2003, Nice, France (2003) 137–144.
  11. S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
    [CrossRef]
  12. J. M. Bioucas-Dias, M. A. T. Figueiredo, and J. P. Oliveira, “Adaptive total variation image de-convolution: a majorization-minimization approach,” Presented at the European Signal Processing Conference (EUSIPCO 2006) (2006).
  13. Y. C. Eldar, “Uniformly improving the Cramér-Rao bound and maximum-likelihood estimation,” IEEE Trans. Signal Process 54, 2943–2956 (2006).
    [CrossRef]
  14. O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
    [CrossRef]
  15. T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
    [CrossRef]
  16. K. Choi and T. Schulz, “Signal-processing approaches for image-resolution restoration for TOMBO imagery,” Appl. Opt. 47, B104–B116 (2008).
    [CrossRef]
  17. D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” J. Optim. Theory Appl. 155, 227–238 (2012).
    [CrossRef]

2012 (1)

D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” J. Optim. Theory Appl. 155, 227–238 (2012).
[CrossRef]

2009 (1)

S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
[CrossRef]

2008 (2)

2007 (2)

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

A. V. Kanaev, J. R. Ackerman, E. F. Fleet, and D. A. Scribner, “TOMBO sensor with scene-independent superresolution processing,” Opt. Lett. 32, 2855–2857 (2007).
[CrossRef]

2006 (2)

2005 (1)

T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
[CrossRef]

2004 (2)

2003 (1)

2001 (1)

1994 (1)

P. Jorge and S. G. Ferreira, “Interpolation and the discrete Papoulis–Gerchberg algorithm,” IEEE Trans. Signal Process. 42, 2596–2606 (1994).
[CrossRef]

Ackerman, J. R.

Babacan, S. D.

S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
[CrossRef]

Bioucas-Dias, J. M.

J. M. Bioucas-Dias, M. A. T. Figueiredo, and J. P. Oliveira, “Adaptive total variation image de-convolution: a majorization-minimization approach,” Presented at the European Signal Processing Conference (EUSIPCO 2006) (2006).

Brady, D.

Carrier, J.

Chan, T. F.

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

Chen, C.

Choi, K.

Christiansen, O.

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

Eldar, Y. C.

Y. C. Eldar, “Uniformly improving the Cramér-Rao bound and maximum-likelihood estimation,” IEEE Trans. Signal Process 54, 2943–2956 (2006).
[CrossRef]

Ferreira, S. G.

P. Jorge and S. G. Ferreira, “Interpolation and the discrete Papoulis–Gerchberg algorithm,” IEEE Trans. Signal Process. 42, 2596–2606 (1994).
[CrossRef]

Figueiredo, M. A. T.

J. M. Bioucas-Dias, M. A. T. Figueiredo, and J. P. Oliveira, “Adaptive total variation image de-convolution: a majorization-minimization approach,” Presented at the European Signal Processing Conference (EUSIPCO 2006) (2006).

Fleet, E. F.

Gibbons, R.

Gilboa, G.

G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Texture preserving variational denoising using an adaptive fidelity term,” Presented at the VLSM 2003, Nice, France (2003) 137–144.

Han, D.

D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” J. Optim. Theory Appl. 155, 227–238 (2012).
[CrossRef]

Ichioka, Y.

Ishida, K.

Jorge, P.

P. Jorge and S. G. Ferreira, “Interpolation and the discrete Papoulis–Gerchberg algorithm,” IEEE Trans. Signal Process. 42, 2596–2606 (1994).
[CrossRef]

Kanaev, A. V.

Katsaggelos, A. K.

S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
[CrossRef]

Kitamura, Y.

Kodou, N.

Kolste, R. T.

Kondou, N.

Kumagai, T.

Lee, T. M.

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

Lie, J.

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

Masaki, Y.

Milanfar, P.

D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Process 13, 1185–1199 (2004).
[CrossRef]

Miyamoto, M.

Miyamoto, S.

Miyatake, S.

Miyazaki, D.

Molina, R.

S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
[CrossRef]

Morimoto, T.

Nitta, K.

Oliveira, J. P.

J. M. Bioucas-Dias, M. A. T. Figueiredo, and J. P. Oliveira, “Adaptive total variation image de-convolution: a majorization-minimization approach,” Presented at the European Signal Processing Conference (EUSIPCO 2006) (2006).

Pham, T. Q.

T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
[CrossRef]

T. Q. Pham, “Spatiotonal adaptivity in super-resolution of under-sampled image sequences,” Ph.D. dissertation (aan de Technische Universiteit Delft, 2006).

Pitslanis, N.

Prather, D.

Robinson, D.

D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Process 13, 1185–1199 (2004).
[CrossRef]

Schulz, T.

Schutte, K.

T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
[CrossRef]

Scribner, D. A.

Shankar, M.

Shogenji, R.

Sinha, U.

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

Sochen, N.

G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Texture preserving variational denoising using an adaptive fidelity term,” Presented at the VLSM 2003, Nice, France (2003) 137–144.

Tanida, J.

van Vliet, L. J.

T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
[CrossRef]

Willet, R.

Yamada, K.

Yuan, X.

D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” J. Optim. Theory Appl. 155, 227–238 (2012).
[CrossRef]

Zeevi, Y. Y.

G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Texture preserving variational denoising using an adaptive fidelity term,” Presented at the VLSM 2003, Nice, France (2003) 137–144.

Appl. Opt. (5)

IEEE Trans. Image Process (2)

S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Variational Bayesian blind deconvolution using a total variation prior,” IEEE Trans. Image Process 18, 12–26 (2009).
[CrossRef]

D. Robinson and P. Milanfar, “Fundamental performance limits in image registration,” IEEE Trans. Image Process 13, 1185–1199 (2004).
[CrossRef]

IEEE Trans. Signal Process (1)

Y. C. Eldar, “Uniformly improving the Cramér-Rao bound and maximum-likelihood estimation,” IEEE Trans. Signal Process 54, 2943–2956 (2006).
[CrossRef]

IEEE Trans. Signal Process. (1)

P. Jorge and S. G. Ferreira, “Interpolation and the discrete Papoulis–Gerchberg algorithm,” IEEE Trans. Signal Process. 42, 2596–2606 (1994).
[CrossRef]

Int. J. Biomed. Imaging (1)

O. Christiansen, T. M. Lee, J. Lie, U. Sinha, and T. F. Chan, “Total variation regularization of matrix-valued images,” Int. J. Biomed. Imaging 2007, 27432 (2007).
[CrossRef]

J. Optim. Theory Appl. (1)

D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” J. Optim. Theory Appl. 155, 227–238 (2012).
[CrossRef]

Opt. Express (1)

Opt. Lett. (1)

Proc. SPIE (1)

T. Q. Pham, L. J. van Vliet, and K. Schutte, “Influence of signal-to-noise ratio and point spread function on limits of superresolution,” Proc. SPIE 5672, 169–180 (2005).
[CrossRef]

Other (3)

J. M. Bioucas-Dias, M. A. T. Figueiredo, and J. P. Oliveira, “Adaptive total variation image de-convolution: a majorization-minimization approach,” Presented at the European Signal Processing Conference (EUSIPCO 2006) (2006).

T. Q. Pham, “Spatiotonal adaptivity in super-resolution of under-sampled image sequences,” Ph.D. dissertation (aan de Technische Universiteit Delft, 2006).

G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Texture preserving variational denoising using an adaptive fidelity term,” Presented at the VLSM 2003, Nice, France (2003) 137–144.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1.
Fig. 1.

Multichannel imaging system basic architecture. (a) The scenery, (b) the microlens array, (c) the CMOS imager, and (d) a single subchannel.

Fig. 2.
Fig. 2.

Multiplexed linear system modeling. Each channel consists of translation, blur, decimation, and additive noise.

Fig. 3.
Fig. 3.

Complete image restoration process, consisting of three separate stages: common grid subchannel image registration, image fusion, and finally, image restoration.

Fig. 4.
Fig. 4.

(a) Original color Lena image, (b) original HR color Mandrill image, and (c) original HR GL Cat image.

Fig. 5.
Fig. 5.

Registration error scatter plots for various decimation factors. The left and right plots show the estimation error scattering before and after bias correction. Each color represents a different set of coupled images, with a different translation. Error means are marked with white circles.

Fig. 6.
Fig. 6.

Registration parameter estimation process for various scale ratio factors. (a) HR propagated subpixel misalignment error scatter plots before and after bias correction, showing an increase in STD as the scale ratio increases and bias increases, for an increase in both scale ratio and subchannel distance and (b) the total misalignment error reduction due to bias correction.

Fig. 7.
Fig. 7.

Examples of three restoration processes are presented in each of the three rows, at a ratio of 67. (a), (d), (g) In the first column, one of the sample LR images; (b), (e), (h) in the second column, the single fused image; and (c), (f), (i) in the third column, the reconstructed HR images.

Fig. 8.
Fig. 8.

Comparison of restorations, with a noise level of 5 GL-STD, scale ratio of 6, and blur STD width of 0.5 pixel. Each row presents a comparison of two restoration schemes: the pixel rearrangement, followed by the LMS restoration algorithm, then the proposed method, fast image fusion, followed by the MMTV restoration algorithm. (a), (e), (j) Cat, Lena, and mandrill pixel rearrangement fusion; (b), (f), (k) cat, Lena, and mandrill fast fusion; (c), (g), (l) cat, Lena, and mandrill LMS restoration with the pixel rearrangement as an initial solution; and (d), (h), (m) cat, Lena, and mandrill MMTV restoration with fast fusion as an initial solution. Noticeable visual speckle exists globally in the LMS restoration, suppressed by the MMTV regulator.

Fig. 9.
Fig. 9.

Local PSNR maps of the restorations. (a)–(c) Original noncorrupted images, (d)–(f) pixel rearrangement followed by LMS algorithm, (g), (h), (k) local PSNR maps for the pixel rearrangement with LMS algorithm reconstruction, (l)–(n) proposed method, and (o)–(q) proposed method local PSNR maps.

Fig. 10.
Fig. 10.

Local PSNR zoom image. THE local PSNR was evaluated in an area of 6×6 HR pixels, which is equal to the area of a single LR (coarse) pixel.

Fig. 11.
Fig. 11.

Proposed method, fast image fusion, followed by the MMTV restoration algorithm used to restore the target suggested in [17].

Fig. 12.
Fig. 12.

Proposed method IBP solution with two different initial conditions: (a) fast image fusion and (b) random vector.

Tables (1)

Tables Icon

Table 1. Restoration PSNR Results: Comparison of the Proposed Solution and the Alternate Algorithma

Equations (32)

Equations on this page are rendered with MathJax. Learn more.

x̲=vec(I̲̲HR).
H̲̲=D̲̲(1)B̲̲T̲̲,
z̲i=H̲̲x̲i+n̲i0<iNSMS,
VAR(t̲)tr(CRLBMVUE(t̲))=tr(F̲̲1(t̲)),
MSE(t̲)b̲(t̲)b̲T(t̲)+CRLBBIASED(t̲),
CRLBBIASED(t̲)=(1̲̲+b̲(t̲)t̲)F̲̲1(t̲)(1̲̲+b̲(t̲)t̲)T.
I̲̲(x+Δx,y+Δy)I̲̲(x,y)+[I̲̲X(x,y)I̲̲Y(x,y)]·t̲,
t^̲=argmint̲z̲tJ̲̲t̲2=GST̲̲(z̲)1J̲̲Tz̲t,
J̲̲=[z̲Xz̲Y].
z̲t=vec(I̲̲(x+Δx,y+Δy)I̲̲(x,y)),
GST̲̲(z̲)J̲TJ̲.
P(I̲̲|t̲)=12πσN2i=12n,mexp[(I̲̲[n,m]I̲̲i[ni,mi])22σN2].
VAR(t^̲)GST̲̲(z̲)1·σn2·[zYTzYzXTzX]T.
H˜̲̲=D̲̲(0)A̲̲
x^̲(0)=i=1MS·NST̲̲iTA̲̲iD̲̲i(0)Tz̲i,
o^̲[k,l]=E{[o^̲1[k,l]o^̲n1[k,l],z̲n[k/MS,l/NS],o^̲n+1[k,l]o^̲MSNS1[k,l]]}.
limn(D̲̲(0)TD̲̲(0)A̲̲)n1/n1.
S(x̲,r̲x̲,r̲)=λ·ρFID(x̲,)+ρREG(x̲,r̲x̲,r̲),
L^S(x̲,r̲x̲,r̲)=(x̲+r̲·r̲x̲)S(x̲,r̲x̲,r̲)=0̲.
ρFID(x̲,z̲)=z̲H̲̲x̲2/2σ2|Ω|,
ρTV(r̲x̲)=r̲x̲Δ̲̲x̲.
L^ρTV(r̲x̲)=r̲·(r̲x̲/r̲x̲).
ρMMTV(x̲(k),x̲(k1))=TV2(x̲(k))+TV2(x̲(k1))2·|TV(x̲(k1))||TV(x̲(k))|.
S(x̲(k),x̲(k1),r̲)=λ2σ2|Ω|z̲H̲̲x̲(k)2+12x̲(k)TΔ̲̲TW̲̲Δ̲̲x̲(k)+12x̲(k1)TΔ̲̲TW̲̲Δ̲̲x̲(k1),
λ=σ2|ΩL|(L^ρFID)T·(L^ρFID).
minα(k)S(x^̲(k+1))=minα(k)S(x^̲(k)+α(k)g̲(k)),
α(k)=d̲(k)Tg̲(k)/(d̲(k)TH̲̲TH̲̲d̲(k1))
e̲(t^̲,t̲)t^̲t̲.
z̲=H̲̲x̲+n̲,
x^̲LMS=argminx^̲(H̲̲x̲z̲)TW̲̲(H̲̲x̲z̲)=(H̲̲TW̲̲H̲̲)1H̲̲TW̲̲Tz̲.
|a|·|b|(|a|+|b|)/2.
ρMMTV(x̲(k),x̲(k1))=TV2(x̲(k))+TV2(x̲(k1))2·|TV(x̲(k1))||TV(x̲(k))|.

Metrics