Abstract

During the process of microscanning a controlled vibrating mirror typically is used to produce subpixel shifts in a sequence of forward-looking infrared (FLIR) images. If the FLIR is mounted on a moving platform, such as an aircraft, uncontrolled random vibrations associated with the platform can be used to generate the shifts. Iterative techniques such as the expectation-maximization (EM) approach by means of the maximum-likelihood algorithm can be used to generate high-resolution images from multiple randomly shifted aliased frames. In the maximum-likelihood approach the data are considered to be Poisson random variables and an EM algorithm is developed that iteratively estimates an unaliased image that is compensated for known imager-system blur while it simultaneously estimates the translational shifts. Although this algorithm yields high-resolution images from a sequence of randomly shifted frames, it requires significant computation time and cannot be implemented for real-time applications that use the currently available high-performance processors. The new image shifts are iteratively calculated by evaluation of a cost function that compares the shifted and interlaced data frames with the corresponding values in the algorithm’s latest estimate of the high-resolution image. We present a registration algorithm that estimates the shifts in one step. The shift parameters provided by the new algorithm are accurate enough to eliminate the need for iterative recalculation of translational shifts. Using this shift information, we apply a simplified version of the EM algorithm to estimate a high-resolution image from a given sequence of video frames. The proposed modified EM algorithm has been found to reduce significantly the computational burden when compared with the original EM algorithm, thus making it more attractive for practical implementation. Both simulation and experimental results are presented to verify the effectiveness of the proposed technique.

© 1998 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. E. A. Watson, R. A. Muse, F. P. Blommel, “Aliasing and blurring in microscanned imagery,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed., Proc. SPIE1689, 242–250 (1992).
    [CrossRef]
  2. J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
    [CrossRef]
  3. A. Schaum, M. McHugh, “Analytic methods of image registration: displacement estimation and resampling,” NRL Rep. 9298 (Naval Research Laboratory, Washington, D.C., 28February1991).
  4. M. Irani, S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991).
    [CrossRef]
  5. E. Kaltenbacher, R. C. Hardie, “Infrared image registration and high-resolution reconstruction,” in Proceedings of the National Aeronautics and Electronics Conference (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 702–709.
  6. M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
    [CrossRef]
  7. T. J. Schulz, “Multiframe blind deconvolution of astronomical images,” J. Opt. Soc. Am. A 10, 1064–1069 (1993).
    [CrossRef]
  8. R. R. Schulz, R. L. Stevenson, “Extraction of high resolution frames from video sequences,” IEEE Trans. Image Process. 5, 996–1002 (1996).
    [CrossRef]
  9. R. R. Schulz, R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process. 3, 233–242 (1994).
    [CrossRef]
  10. R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
    [CrossRef] [PubMed]
  11. Y. P. Hung, D. B. Cooper, “Maximum a posteriori probability 3D surface reconstruction using multiple intensity images directly,” in Sensing and Reconstruction of Three-Dimensional Objects and Scenes, B. Girod, ed., Proc. SPIE1260, 36–48 (1990).
  12. S. Cain, R. C. Hardie, “Restoration of aliased video sequences via a maximum-likelihood approach,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 377–390.
  13. Y. You, M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Process. 5, 416–428 (1996).
    [CrossRef] [PubMed]
  14. L. Shepp, Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imag. M1-1, 113–122 (1982).
    [CrossRef]
  15. M. S. Alam, “Fast infrared image registration and high-resolution reconstruction for real time applications,” in 1996 Annual Research Report (Air Force Office of Scientific Research, Bolling Air Force Base, Washington, D.C.) (in press).
  16. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).
  17. E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.
  18. W. K. Pratt, Digital Image Processing (Wiley, New York, 1992).
  19. R. C. Gonzalez, R. E. Woods, Digital Image Processing (Addison-Wesley, Reading, Mass., 1992).

1997 (1)

R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
[CrossRef] [PubMed]

1996 (2)

Y. You, M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Process. 5, 416–428 (1996).
[CrossRef] [PubMed]

R. R. Schulz, R. L. Stevenson, “Extraction of high resolution frames from video sequences,” IEEE Trans. Image Process. 5, 996–1002 (1996).
[CrossRef]

1995 (1)

J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
[CrossRef]

1994 (1)

R. R. Schulz, R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process. 3, 233–242 (1994).
[CrossRef]

1993 (1)

1991 (1)

M. Irani, S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991).
[CrossRef]

1982 (1)

L. Shepp, Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imag. M1-1, 113–122 (1982).
[CrossRef]

Alam, M. S.

M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
[CrossRef]

M. S. Alam, “Fast infrared image registration and high-resolution reconstruction for real time applications,” in 1996 Annual Research Report (Air Force Office of Scientific Research, Bolling Air Force Base, Washington, D.C.) (in press).

Armstrong, E.

E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.

Armstrong, E. E.

R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
[CrossRef] [PubMed]

Barnard, K. J.

R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
[CrossRef] [PubMed]

Blommel, F. P.

E. A. Watson, R. A. Muse, F. P. Blommel, “Aliasing and blurring in microscanned imagery,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed., Proc. SPIE1689, 242–250 (1992).
[CrossRef]

Bognar, J.

E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.

Bognar, J. G.

M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
[CrossRef]

Cain, S.

S. Cain, R. C. Hardie, “Restoration of aliased video sequences via a maximum-likelihood approach,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 377–390.

Cooper, D. B.

Y. P. Hung, D. B. Cooper, “Maximum a posteriori probability 3D surface reconstruction using multiple intensity images directly,” in Sensing and Reconstruction of Three-Dimensional Objects and Scenes, B. Girod, ed., Proc. SPIE1260, 36–48 (1990).

Gillette, J. C.

J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
[CrossRef]

Gonzalez, R. C.

R. C. Gonzalez, R. E. Woods, Digital Image Processing (Addison-Wesley, Reading, Mass., 1992).

Goodman, J. W.

J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).

Hardie, R.

E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.

Hardie, R. C.

R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
[CrossRef] [PubMed]

J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
[CrossRef]

M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
[CrossRef]

E. Kaltenbacher, R. C. Hardie, “Infrared image registration and high-resolution reconstruction,” in Proceedings of the National Aeronautics and Electronics Conference (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 702–709.

S. Cain, R. C. Hardie, “Restoration of aliased video sequences via a maximum-likelihood approach,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 377–390.

Hung, Y. P.

Y. P. Hung, D. B. Cooper, “Maximum a posteriori probability 3D surface reconstruction using multiple intensity images directly,” in Sensing and Reconstruction of Three-Dimensional Objects and Scenes, B. Girod, ed., Proc. SPIE1260, 36–48 (1990).

Irani, M.

M. Irani, S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991).
[CrossRef]

Kaltenbacher, E.

E. Kaltenbacher, R. C. Hardie, “Infrared image registration and high-resolution reconstruction,” in Proceedings of the National Aeronautics and Electronics Conference (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 702–709.

Kaveh, M.

Y. You, M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Process. 5, 416–428 (1996).
[CrossRef] [PubMed]

McHugh, M.

A. Schaum, M. McHugh, “Analytic methods of image registration: displacement estimation and resampling,” NRL Rep. 9298 (Naval Research Laboratory, Washington, D.C., 28February1991).

Muse, R. A.

E. A. Watson, R. A. Muse, F. P. Blommel, “Aliasing and blurring in microscanned imagery,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed., Proc. SPIE1689, 242–250 (1992).
[CrossRef]

Peleg, S.

M. Irani, S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991).
[CrossRef]

Pratt, W. K.

W. K. Pratt, Digital Image Processing (Wiley, New York, 1992).

Schaum, A.

A. Schaum, M. McHugh, “Analytic methods of image registration: displacement estimation and resampling,” NRL Rep. 9298 (Naval Research Laboratory, Washington, D.C., 28February1991).

Schulz, R. R.

R. R. Schulz, R. L. Stevenson, “Extraction of high resolution frames from video sequences,” IEEE Trans. Image Process. 5, 996–1002 (1996).
[CrossRef]

R. R. Schulz, R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process. 3, 233–242 (1994).
[CrossRef]

Schulz, T. J.

Shepp, L.

L. Shepp, Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imag. M1-1, 113–122 (1982).
[CrossRef]

Stadmiller, T. M.

J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
[CrossRef]

Stevenson, R. L.

R. R. Schulz, R. L. Stevenson, “Extraction of high resolution frames from video sequences,” IEEE Trans. Image Process. 5, 996–1002 (1996).
[CrossRef]

R. R. Schulz, R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process. 3, 233–242 (1994).
[CrossRef]

Vardi, Y.

L. Shepp, Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imag. M1-1, 113–122 (1982).
[CrossRef]

Watson, E. A.

E. A. Watson, R. A. Muse, F. P. Blommel, “Aliasing and blurring in microscanned imagery,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed., Proc. SPIE1689, 242–250 (1992).
[CrossRef]

Woods, R. E.

R. C. Gonzalez, R. E. Woods, Digital Image Processing (Addison-Wesley, Reading, Mass., 1992).

Yasuda, B.

E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.

Yasuda, B. J.

M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
[CrossRef]

You, Y.

Y. You, M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Process. 5, 416–428 (1996).
[CrossRef] [PubMed]

CVGIP: Graph. Models Image Process. (1)

M. Irani, S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991).
[CrossRef]

IEEE Trans. Image Process. (4)

R. R. Schulz, R. L. Stevenson, “Extraction of high resolution frames from video sequences,” IEEE Trans. Image Process. 5, 996–1002 (1996).
[CrossRef]

R. R. Schulz, R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process. 3, 233–242 (1994).
[CrossRef]

R. C. Hardie, K. J. Barnard, E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process. 6, 1621–1628 (1997).
[CrossRef] [PubMed]

Y. You, M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Process. 5, 416–428 (1996).
[CrossRef] [PubMed]

IEEE Trans. Med. Imag. (1)

L. Shepp, Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imag. M1-1, 113–122 (1982).
[CrossRef]

J. Opt. Soc. Am. A (1)

Opt. Eng. (1)

J. C. Gillette, T. M. Stadmiller, R. C. Hardie, “Reduction of aliasing in staring infrared imagers utilizing subpixel techniques,” Opt. Eng. 34, 3130–3137 (1995).
[CrossRef]

Other (11)

A. Schaum, M. McHugh, “Analytic methods of image registration: displacement estimation and resampling,” NRL Rep. 9298 (Naval Research Laboratory, Washington, D.C., 28February1991).

E. Kaltenbacher, R. C. Hardie, “Infrared image registration and high-resolution reconstruction,” in Proceedings of the National Aeronautics and Electronics Conference (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 702–709.

M. S. Alam, J. G. Bognar, B. J. Yasuda, R. C. Hardie, “High-resolution infrared image reconstruction using multiple randomly shifted, low-resolution, aliased frames,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VIII, G. C. Holst, ed., Proc. SPIE3063, 102–112 (1997).
[CrossRef]

Y. P. Hung, D. B. Cooper, “Maximum a posteriori probability 3D surface reconstruction using multiple intensity images directly,” in Sensing and Reconstruction of Three-Dimensional Objects and Scenes, B. Girod, ed., Proc. SPIE1260, 36–48 (1990).

S. Cain, R. C. Hardie, “Restoration of aliased video sequences via a maximum-likelihood approach,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 377–390.

E. A. Watson, R. A. Muse, F. P. Blommel, “Aliasing and blurring in microscanned imagery,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed., Proc. SPIE1689, 242–250 (1992).
[CrossRef]

M. S. Alam, “Fast infrared image registration and high-resolution reconstruction for real time applications,” in 1996 Annual Research Report (Air Force Office of Scientific Research, Bolling Air Force Base, Washington, D.C.) (in press).

J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1996).

E. Armstrong, J. Bognar, B. Yasuda, R. Hardie, “The application of Wiener filters to microscan imaging,” in Proceedings of the 1996 IRIS Specialty Group Meeting on Passive Sensors, E. Williams, ed. (Infrared Information Analysis Center, Ann Arbor, Mich., 1996), Vol. 1, pp. 359–375.

W. K. Pratt, Digital Image Processing (Wiley, New York, 1992).

R. C. Gonzalez, R. E. Woods, Digital Image Processing (Addison-Wesley, Reading, Mass., 1992).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1
Fig. 1

(a) High-resolution image. (b) Low-resolution image.

Fig. 2
Fig. 2

Image-formation model.

Fig. 3
Fig. 3

(a) Original input scene. (b) Staring frame sampled by the simulated detector array.

Fig. 4
Fig. 4

Theoretical (X) and estimated (O) registration parameters for 16 simulated frames.

Fig. 5
Fig. 5

Simulated FLIR Images: (a) High-resolution frame reconstructed by use of the original EM approach. (b) High-resolution frame reconstructed by use of the modified EM approach.

Fig. 6
Fig. 6

Real FLIR Images: (a) Staring frame. (b) Staring frame after bicubic interpolation. (c) High-resolution image reconstructed from 16 real FLIR data frames by use of the original EM algorithm. (d) High-resolution image reconstructed from 16 real FLIR data frames by use of the modified EM algorithm.

Tables (1)

Tables Icon

Table 1 Comparison of Processing Speeds of the Original EM Algorithm with the Modified EM Algorithm

Equations (26)

Equations on this page are rendered with MathJax. Learn more.

y ¯ m 1 ,   m 2 = i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o i 1 ,   i 2 h m 1 - i 1 ,   m 2 - i 2
f c = 2 l λ d ,
y k ( m 1 ,   m 2 ) = i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   y ¯ k ( m 1 ,   m 2 ) w ( L 1 n 1 - m 1 - h k ,   L 2 n 2 - m 2 - v k ) ,
y k ZOH ( j 1 ,   j 2 ) = i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   y ¯ k ( m 1 ,   m 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] ,
Pr X = D k j 1 ,   j 2 = y k j 1 ,   j 2 D k j 1 , j 2 exp - y k j 1 ,   j 2 D k j 1 ,   j 2 ! .
o k ( x ,   y ) = o 1 ( x + h k ,   y + v k ) ,
o k ( x ,   y ) o 1 ( x ,   y ) + h k o 1 ( x ,   y ) x + v k o 1 ( x ,   y ) y .
E k h k ,   v k 1 MN m = 1 M n = 1 N o k m ,   n - o 1 m ,   n - h k o 1 m ,   n m - v k o 1 m ,   n n 2
m = 1 M n = 1 N o 1 m ,   n m 2 m = 1 M n = 1 N o 1 m ,   n m o 1 m ,   n n m = 1 M n = 1 N o 1 m ,   n m o 1 m ,   n n m = 1 M n = 1 N o 1 m ,   n n 2   h k v k = m = 1 M n = 1 N [ o k m ,   n - o 1 m ,   n ] o 1 m ,   n m m = 1 M n = 1 N [ o k m ,   n - o 1 m ,   n ] o 1 m ,   n n .
MS = V ,
S = h k   v k T ,   M = m = 1 M n = 1 N o 1 m ,   n m 2 m = 1 M n = 1 N o 1 m ,   n m o 1 m ,   n n m = 1 M n = 1 N o 1 m ,   n m o 1 m ,   n n m = 1 M n = 1 N o 1 m ,   n n 2 ,   V = m = 1 M n = 1 N [ o k m ,   n - o 1 m ,   n ] o 1 m ,   n m m = 1 M n = 1 N [ o k m ,   n - o 1 m ,   n ] o 1 m ,   n n .
S = M - 1 V .
E [ D k ZOH ( j 1 ,   j 2 ) ] = m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   h ( m 1 - i 1 ,   m 2 - i 2 ) × o ( i 1 ,   i 2 ) × w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] = y k ZOH ( j 1 ,   j 2 ) .
D k ZOH ( j 1 ,   j 2 ) = m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   D ˆ k ( j 1 ,   m 1 ,   i 1 ,   j 2 ,   m 2 ,   i 2 ) .
E [ D ˆ k ZOH ( j 1 ,   j 2 ) ] = h ( m 1 - i 1 ,   m 2 - i 2 ) o ( i 1 ,   i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] .
Pr [ X = D ˆ k ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) ] = { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T ( j 2 ) - m 2 - v k ] } D ˆ k ( j 1 , j 2 , m 1 , m 2 , i 1 , i 2 ) D ˆ k ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) ! × exp ( - { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T ( j 2 ) - m 2 - v k ] } ) .
Pr [ D ˆ k ) = k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2 { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T ( j 2 ) - m 2 - v k ] } D ˆ k ( j 1 , j 2 , m 1 , m 2 , i 1 , i 2 ) D ˆ k ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) ! × exp ( - { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T ( j 2 ) - m 2 - v k ] } ) .
L ( o ,   h k ,   v k ) = k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2 × D ˆ k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) × ln { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) × w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] } - KH   i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o ( i 1 ,   i 2 ) + o . t . ,
E [ L ( o ,   h k ,   v k ) | D k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) , o old ( i 1 ,   i 2 ) ,   h k old ,   v k old ] = k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2 × E [ D ˆ k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) × | D k ZOH ( j 1 ,   j 2 ) ,   o old ( i 1 ,   i 2 ) ,   h k old ,   v k old ] × ln { o ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) × w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] } - KH   i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o ( i 1 ,   i 2 ) + o . t .
d E [ L ( o ,   h k ,   v k ) | D k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) ,   o old ( i 1 ,   i 2 ) ,   h k old ,   v k old ] d o ( i 1 ,   i 2 ) = 0 .
o ( i 1 ,   i 2 ) = k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   E [ D ˆ k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) | D k ZOH ( j 1 ,   j 2 ) ,   o old ( i 1 ,   i 2 ) ,   h k old ,   v k old ] KH .
E [ D ˆ k ZOH ( j 1 ,   j 2 ,   m 1 ,   m 2 ,   i 1 ,   i 2 ) | D k ZOH ( j 1 ,   j 2 ) ,   o old ( i 1 ,   i 2 ) ,   h k old ,   v k old ] = o old ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k old ,   T 2 ( j 2 ) - m 2 - v k old ] D k ZOH ( j 1 ,   j 2 ) m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o old ( i 1 ,   i 2 ) h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k old ,   T 2 ( j 2 ) - m 2 - v k old ] .
o new ( i 1 ,   i 2 ) = o old ( i 1 ,   i 2 ) k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 × h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k old ,   T 2 ( j 2 ) - m 2 - v k old ] D k ZOH ( j 1 ,   j 2 ) × KH   m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o old ( i 1 ,   i 2 ) × h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k old ,   T 2 ( j 2 ) - m 2 - v k old ] - 1 ,
h k new = arg h k min j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 D k ZOH j 1 ,   j 2 y k ZOH new j 1 ,   j 2 - 1 ,
v k new = arg v k min j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 D k ZOH j 1 ,   j 2 y k ZOH new j 1 ,   j 2 - 1 .
o new ( i 1 ,   i 2 ) = o old ( i 1 ,   i 2 ) k = 1 K j 1 = 1 L 1 N 1 j 2 = 1 L 2 N 2 m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 × h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] D k ZOH ( j 1 ,   j 2 ) × KH   m 1 = 1 L 1 N 1 m 2 = 1 L 2 N 2 i 1 = 1 L 1 N 1 i 2 = 1 L 2 N 2   o old ( i 1 ,   i 2 ) × h ( m 1 - i 1 ,   m 2 - i 2 ) w [ T 1 ( j 1 ) - m 1 - h k ,   T 2 ( j 2 ) - m 2 - v k ] - 1 .

Metrics