Abstract

In this paper, we propose a new motion estimation algorithm to be used for motion-compensated frame rate up-conversion. The proposed algorithm independently carries out motion estimations in both forward and backward directions, and selects a more reliable one between forward and backward motion vectors by evaluating the motion vector reliability from the viewpoint of the interpolated frame. The proposed algorithm smooths and refines both the forward and backward motion vectors before selecting the reliable one. This procedure helps to select the reasonable motion estimation direction. In identifying the motion vector outliers, the proposed algorithm uses a circular range of which center is located at the mean of the eight neighboring motion vectors of the motion vector being processed. Experimental results using 1720 test images show that the proposed motion estimation algorithm improves the average peak signal-to-noise ratio and the average structural similarity of the interpolated frames by up to 5.31 dB and 0.053, respectively, compared to conventional motion estimation algorithms.

© 2013 IEEE

PDF Article

References

  • View by:
  • |
  • |

  1. C. Cafforio, F. Rocca, S. Tubaro, "Motion Compensated Image Interpolation," IEEE Trans. Communications 38, 215-222 (1990).
  2. K. Hilman, Y. W. Park, Y. Kim, "Using motion compensated frame rate up conversion for the correction of 3:2 pulldown artifacts in video sequences," IEEE Trans. Circuits Syst. Video Technol. 10, 869-877 (2000).
  3. S.-J. Kang, S. Yoo, Y. H. Kim, "Multiframe-based bilateral motion estimation with emphasis on stationary caption processing for frame rate up-conversion," IEEE Trans. Consumer Electron. 54, 1830-1839 (2008).
  4. K.-M. Yang, M.-T. Sun, L. Wu, "A family of VLSI designs for the motion compensation block-matching algorithm," IEEE Trans. Circuits Syst. 36, 1317-1325 (1989).
  5. B.-W. Jeon, G.-I. Lee, S.-H. Lee, R.-H. Park, "Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure," IEEE Trans. Consumer Electron. 49, 499-508 (2003).
  6. B. D. Choi, J. W. Han, C. S. Kim, S. J. Ko, "Frame rate up-conversion using perspective transform," IEEE Trans. Consumer Electron. 52, 975-982 (2006).
  7. G. de Haan, P. W. A. C. Biezen, H. Huijgen, O. A. Ojo, "True-motion estimation with 3-D recursive search block matching," IEEE Trans. Circuits and System for Video Technology 3, 368-379 (1993).
  8. T. Ha, S. Lee, J. Kim, "Motion compensated frame interpolation by new block-based motion estimation algorithm," IEEE Trans. Consumer Electron. 50, 752-759 (2004).
  9. F. Shi, V. R. Raveendran, Method and apparatus for frame rate up conversion with multiple reference frames and variable block sizes U.S. Patent Appl. 2006/0017843 (2006).
  10. S. H. Chang, S.-J. Kang, D.-G. Yoo, Y. H. Kim, S. K. Lee, C. G. Kim, D. H. Cha, S. J. Koo, Motion compensation method, motion compensation device, liquid crystal display device having the same, and driving method thereof Korea Patent 10-2009-0068502 (2009).
  11. B.-D. Choi, J.-W. Han, C.-S. Kim, S.-J. Ko, "Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation," IEEE Trans. Circuits Syst. Video Technol. 17, 407-416 (2007).
  12. S.-J. Kang, K.-R. Cho, Y. H. Kim, "Motion compensated frame rate up-conversion using extended bilateral motion estimation," IEEE Trans. Consumer Electron. 53, 1759-1767 (2007).
  13. S.-J. Kang, S. Yoo, Y. H. Kim, "Dual motion estimation for frame rate up-conversion," IEEE Trans. Circuits Syst. Video Technol. 20, 1909-1914 (2010).
  14. D. Y. Kim, D. S. Jun, H. W. Park, "An efficient side information generation using seed blocks for distributed video coding," 28th Picture Coding Symp. (2010).
  15. H. Lim, H. W. Park, "A symmetric motion estimation method for motion-compensated frame interpolation," IEEE Trans. Image Process. 20, 3653-3658 (2011).
  16. B.-T. Choi, S.-H. Lee, S.-J. Ko, "New frame rate up-conversion using bi-directional motion estimation," IEEE Trans. Consumer Electron. 46, 603-609 (2000).
  17. T. Meier, K. N. Ngan, "Automatic segmentation of moving objects for video object plane generation," IEEE Trans. Circuits Syst. Video Technol. 8, 525-537 (1998).
  18. O. Gedik, E. Turetken, A. Alatan, High-Quality Visual Experience (Springer, 2010) pp. 219-244.
  19. E. B. Bellers, J. W. van Gurp, J. G. W. M. Janssen, R. Braspenning, R. Wittebrood, "Solving occlusion in frame rate up-conversion," Int. Conf. on Consumer Electron. (2007).
  20. A. Heinrich, C. Bartels, R. J. van der Vleuten, C. N. Cordes, G. de Haan, "Optimization of hierarchical 3DRS motion estimators for picture rate conversion," IEEE J. Sel. Topics in Signal Process. 5, 262-274 (2011).
  21. C.-C. Cheng, Y.-L. Huang, Y.-C. Lai, W.-T. Suen, Y.-R. Chen, Method of occlusion handling U.S. Patent 2011/0249870 (2011) et al..
  22. S.-W. Wu, A. Gersho, "Joint estimation of forward and backward motion vectors for interpolative prediction of video," IEEE Trans. Image Process. 3, 684-687 (1994).
  23. T. Q. Vinh, Y.-C. Kim, S.-H. Hong, "Frame rate up-conversion using forward-backward jointing motion estimation and spatio-temporal motion vector smoothing," Int. Conf. on Comput. Eng. Syst. (2009).
  24. G. Dane, T. Q. Nguyen, "Motion vector processing for frame rate up-conversion," IEEE Int. Conf. on Acoust., Speech, Signal Process. (ICASSP) (2004).
  25. D.-G. Yoo, S.-J. Kang, S. K. Lee, Y. H. Kim, "Motion vector smoothing for motion-compensated frame rate up-conversion," Int. SoC Design Conf. (ISOCC) (2010) pp. 358-359.
  26. S.-J. Kang, D.-G. Yoo, S.-K. Lee, Y. H. Kim, "Design and implementation of median filter based adaptive motion vector smoothing for motion compensated frame rate up-conversion," 13th IEEE Int. Symp. on Consumer Electron. (2009).
  27. J. Ascenso, C. Brites, F. Pereira, "Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding," Proc. Eur. Signal Process. Conf. (2005).
  28. J. Guo, J. Kim, "Adaptive motion vector smoothing for improving side information in distributed video coding," J. Inf. Process. Syst. 7, (2011).
  29. S.-J. Kang, D.-G. Yoo, S.-K. Lee, Y. H. Kim, "Hardware implementation of motion estimation using a sub-sampled block for frame rate up-conversion," Int. SoC Design Conf. (ISOCC) (2008).
  30. C. Zhu, X. Lin, L.-P. Chau, "Hexagon-based search pattern for fast block motion estimation," IEEE Trans. Circuits Syst. Video Technol. 12, 349-355 (2002).
  31. S.-C. Hsia, "VLSI implementation for low-complexity full-search motion estimation," IEEE Trans. Circuit Syst. Video Technol. 12, 613-619 (2002).
  32. M. T. Orchard, C. J. Sullivan, "Overlapped block motion compensation: An estimation-theoretic approach," IEEE Trans. Image Process. 3, 693-699 (1994).

2011 (3)

H. Lim, H. W. Park, "A symmetric motion estimation method for motion-compensated frame interpolation," IEEE Trans. Image Process. 20, 3653-3658 (2011).

J. Guo, J. Kim, "Adaptive motion vector smoothing for improving side information in distributed video coding," J. Inf. Process. Syst. 7, (2011).

A. Heinrich, C. Bartels, R. J. van der Vleuten, C. N. Cordes, G. de Haan, "Optimization of hierarchical 3DRS motion estimators for picture rate conversion," IEEE J. Sel. Topics in Signal Process. 5, 262-274 (2011).

2010 (1)

S.-J. Kang, S. Yoo, Y. H. Kim, "Dual motion estimation for frame rate up-conversion," IEEE Trans. Circuits Syst. Video Technol. 20, 1909-1914 (2010).

2008 (1)

S.-J. Kang, S. Yoo, Y. H. Kim, "Multiframe-based bilateral motion estimation with emphasis on stationary caption processing for frame rate up-conversion," IEEE Trans. Consumer Electron. 54, 1830-1839 (2008).

2007 (2)

B.-D. Choi, J.-W. Han, C.-S. Kim, S.-J. Ko, "Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation," IEEE Trans. Circuits Syst. Video Technol. 17, 407-416 (2007).

S.-J. Kang, K.-R. Cho, Y. H. Kim, "Motion compensated frame rate up-conversion using extended bilateral motion estimation," IEEE Trans. Consumer Electron. 53, 1759-1767 (2007).

2006 (1)

B. D. Choi, J. W. Han, C. S. Kim, S. J. Ko, "Frame rate up-conversion using perspective transform," IEEE Trans. Consumer Electron. 52, 975-982 (2006).

2004 (1)

T. Ha, S. Lee, J. Kim, "Motion compensated frame interpolation by new block-based motion estimation algorithm," IEEE Trans. Consumer Electron. 50, 752-759 (2004).

2003 (1)

B.-W. Jeon, G.-I. Lee, S.-H. Lee, R.-H. Park, "Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure," IEEE Trans. Consumer Electron. 49, 499-508 (2003).

2002 (2)

C. Zhu, X. Lin, L.-P. Chau, "Hexagon-based search pattern for fast block motion estimation," IEEE Trans. Circuits Syst. Video Technol. 12, 349-355 (2002).

S.-C. Hsia, "VLSI implementation for low-complexity full-search motion estimation," IEEE Trans. Circuit Syst. Video Technol. 12, 613-619 (2002).

2000 (2)

B.-T. Choi, S.-H. Lee, S.-J. Ko, "New frame rate up-conversion using bi-directional motion estimation," IEEE Trans. Consumer Electron. 46, 603-609 (2000).

K. Hilman, Y. W. Park, Y. Kim, "Using motion compensated frame rate up conversion for the correction of 3:2 pulldown artifacts in video sequences," IEEE Trans. Circuits Syst. Video Technol. 10, 869-877 (2000).

1998 (1)

T. Meier, K. N. Ngan, "Automatic segmentation of moving objects for video object plane generation," IEEE Trans. Circuits Syst. Video Technol. 8, 525-537 (1998).

1994 (2)

M. T. Orchard, C. J. Sullivan, "Overlapped block motion compensation: An estimation-theoretic approach," IEEE Trans. Image Process. 3, 693-699 (1994).

S.-W. Wu, A. Gersho, "Joint estimation of forward and backward motion vectors for interpolative prediction of video," IEEE Trans. Image Process. 3, 684-687 (1994).

1993 (1)

G. de Haan, P. W. A. C. Biezen, H. Huijgen, O. A. Ojo, "True-motion estimation with 3-D recursive search block matching," IEEE Trans. Circuits and System for Video Technology 3, 368-379 (1993).

1990 (1)

C. Cafforio, F. Rocca, S. Tubaro, "Motion Compensated Image Interpolation," IEEE Trans. Communications 38, 215-222 (1990).

1989 (1)

K.-M. Yang, M.-T. Sun, L. Wu, "A family of VLSI designs for the motion compensation block-matching algorithm," IEEE Trans. Circuits Syst. 36, 1317-1325 (1989).

IEEE J. Sel. Topics in Signal Process. (1)

A. Heinrich, C. Bartels, R. J. van der Vleuten, C. N. Cordes, G. de Haan, "Optimization of hierarchical 3DRS motion estimators for picture rate conversion," IEEE J. Sel. Topics in Signal Process. 5, 262-274 (2011).

IEEE Trans. Circuits Syst. (1)

K.-M. Yang, M.-T. Sun, L. Wu, "A family of VLSI designs for the motion compensation block-matching algorithm," IEEE Trans. Circuits Syst. 36, 1317-1325 (1989).

IEEE Trans. Circuits Syst. Video Technol. (1)

T. Meier, K. N. Ngan, "Automatic segmentation of moving objects for video object plane generation," IEEE Trans. Circuits Syst. Video Technol. 8, 525-537 (1998).

IEEE Trans. Consumer Electron. (1)

B.-W. Jeon, G.-I. Lee, S.-H. Lee, R.-H. Park, "Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure," IEEE Trans. Consumer Electron. 49, 499-508 (2003).

IEEE Trans. Image Process. (1)

H. Lim, H. W. Park, "A symmetric motion estimation method for motion-compensated frame interpolation," IEEE Trans. Image Process. 20, 3653-3658 (2011).

IEEE Trans. Circuit Syst. Video Technol. (1)

S.-C. Hsia, "VLSI implementation for low-complexity full-search motion estimation," IEEE Trans. Circuit Syst. Video Technol. 12, 613-619 (2002).

IEEE Trans. Circuits and System for Video Technology (1)

G. de Haan, P. W. A. C. Biezen, H. Huijgen, O. A. Ojo, "True-motion estimation with 3-D recursive search block matching," IEEE Trans. Circuits and System for Video Technology 3, 368-379 (1993).

IEEE Trans. Image Process. (1)

M. T. Orchard, C. J. Sullivan, "Overlapped block motion compensation: An estimation-theoretic approach," IEEE Trans. Image Process. 3, 693-699 (1994).

IEEE Trans. Circuits Syst. Video Technol. (1)

C. Zhu, X. Lin, L.-P. Chau, "Hexagon-based search pattern for fast block motion estimation," IEEE Trans. Circuits Syst. Video Technol. 12, 349-355 (2002).

IEEE Trans. Circuits Syst. Video Technol. (3)

K. Hilman, Y. W. Park, Y. Kim, "Using motion compensated frame rate up conversion for the correction of 3:2 pulldown artifacts in video sequences," IEEE Trans. Circuits Syst. Video Technol. 10, 869-877 (2000).

B.-D. Choi, J.-W. Han, C.-S. Kim, S.-J. Ko, "Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation," IEEE Trans. Circuits Syst. Video Technol. 17, 407-416 (2007).

S.-J. Kang, S. Yoo, Y. H. Kim, "Dual motion estimation for frame rate up-conversion," IEEE Trans. Circuits Syst. Video Technol. 20, 1909-1914 (2010).

IEEE Trans. Communications (1)

C. Cafforio, F. Rocca, S. Tubaro, "Motion Compensated Image Interpolation," IEEE Trans. Communications 38, 215-222 (1990).

IEEE Trans. Consumer Electron. (1)

T. Ha, S. Lee, J. Kim, "Motion compensated frame interpolation by new block-based motion estimation algorithm," IEEE Trans. Consumer Electron. 50, 752-759 (2004).

IEEE Trans. Consumer Electron. (4)

S.-J. Kang, S. Yoo, Y. H. Kim, "Multiframe-based bilateral motion estimation with emphasis on stationary caption processing for frame rate up-conversion," IEEE Trans. Consumer Electron. 54, 1830-1839 (2008).

B. D. Choi, J. W. Han, C. S. Kim, S. J. Ko, "Frame rate up-conversion using perspective transform," IEEE Trans. Consumer Electron. 52, 975-982 (2006).

S.-J. Kang, K.-R. Cho, Y. H. Kim, "Motion compensated frame rate up-conversion using extended bilateral motion estimation," IEEE Trans. Consumer Electron. 53, 1759-1767 (2007).

B.-T. Choi, S.-H. Lee, S.-J. Ko, "New frame rate up-conversion using bi-directional motion estimation," IEEE Trans. Consumer Electron. 46, 603-609 (2000).

IEEE Trans. Image Process. (1)

S.-W. Wu, A. Gersho, "Joint estimation of forward and backward motion vectors for interpolative prediction of video," IEEE Trans. Image Process. 3, 684-687 (1994).

J. Inf. Process. Syst. (1)

J. Guo, J. Kim, "Adaptive motion vector smoothing for improving side information in distributed video coding," J. Inf. Process. Syst. 7, (2011).

Other (12)

S.-J. Kang, D.-G. Yoo, S.-K. Lee, Y. H. Kim, "Hardware implementation of motion estimation using a sub-sampled block for frame rate up-conversion," Int. SoC Design Conf. (ISOCC) (2008).

C.-C. Cheng, Y.-L. Huang, Y.-C. Lai, W.-T. Suen, Y.-R. Chen, Method of occlusion handling U.S. Patent 2011/0249870 (2011) et al..

T. Q. Vinh, Y.-C. Kim, S.-H. Hong, "Frame rate up-conversion using forward-backward jointing motion estimation and spatio-temporal motion vector smoothing," Int. Conf. on Comput. Eng. Syst. (2009).

G. Dane, T. Q. Nguyen, "Motion vector processing for frame rate up-conversion," IEEE Int. Conf. on Acoust., Speech, Signal Process. (ICASSP) (2004).

D.-G. Yoo, S.-J. Kang, S. K. Lee, Y. H. Kim, "Motion vector smoothing for motion-compensated frame rate up-conversion," Int. SoC Design Conf. (ISOCC) (2010) pp. 358-359.

S.-J. Kang, D.-G. Yoo, S.-K. Lee, Y. H. Kim, "Design and implementation of median filter based adaptive motion vector smoothing for motion compensated frame rate up-conversion," 13th IEEE Int. Symp. on Consumer Electron. (2009).

J. Ascenso, C. Brites, F. Pereira, "Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding," Proc. Eur. Signal Process. Conf. (2005).

O. Gedik, E. Turetken, A. Alatan, High-Quality Visual Experience (Springer, 2010) pp. 219-244.

E. B. Bellers, J. W. van Gurp, J. G. W. M. Janssen, R. Braspenning, R. Wittebrood, "Solving occlusion in frame rate up-conversion," Int. Conf. on Consumer Electron. (2007).

D. Y. Kim, D. S. Jun, H. W. Park, "An efficient side information generation using seed blocks for distributed video coding," 28th Picture Coding Symp. (2010).

F. Shi, V. R. Raveendran, Method and apparatus for frame rate up conversion with multiple reference frames and variable block sizes U.S. Patent Appl. 2006/0017843 (2006).

S. H. Chang, S.-J. Kang, D.-G. Yoo, Y. H. Kim, S. K. Lee, C. G. Kim, D. H. Cha, S. J. Koo, Motion compensation method, motion compensation device, liquid crystal display device having the same, and driving method thereof Korea Patent 10-2009-0068502 (2009).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.