Abstract

In this paper, a novel optimization-based stitching method is presented. It minimizes an energy function defined with derivatives up to the second order. We have identified some appropriate choices for its parameters, allowing it to reduce artifacts such as ghosting, color inconsistency, and misalignment. To accelerate the computation, a multi-resolution technique is introduced. The significant speedup and memory saving make it possible for use in hand-held capturing devices.

©2007 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Saliency of color image derivatives: a comparison between computational models and human perception

Eduard Vazquez, Theo Gevers, Marcel Lucassen, Joost van de Weijer, and Ramon Baldrich
J. Opt. Soc. Am. A 27(3) 613-621 (2010)

Registration of OCT fundus images with color fundus photographs based on blood vessel ridges

Ying Li, Giovanni Gregori, Robert W. Knighton, Brandon J. Lujan, and Philip J. Rosenfeld
Opt. Express 19(1) 7-16 (2011)

Multi-focus image fusion algorithm based on Laplacian pyramids

Jianguo Sun, Qilong Han, Liang Kou, Liguo Zhang, Kejia Zhang, and Zilong Jin
J. Opt. Soc. Am. A 35(3) 480-490 (2018)

More Recommended Articles

References

  • View by:
  • Article Order
  • |
  • Year
  • |
  • Author
  • |
  • Publication
  1. F. P. Araújo and N. J. Leite, “A morphological algorithm for photomosaicking,” in Proceedings of 8th European Signal Processing Conference (EURASIP, 1996), pp. 1881–1884.
  2. F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. 1, 21–46 (1990).
    [Crossref]
  3. M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 509–516.
  4. P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. 28, 673–683 (2006).
    [Crossref]
  5. A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.
  6. C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. 42, 981–990 (1996).
    [Crossref]
  7. M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in Proceedings of International Conference on Image Processing (IEEE, 2001), pp. 245–245.
  8. M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
    [Crossref]
  9. A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques (ACM, 2001), pp. 341–346.
  10. R. Gonzalez and R. Woods, Digital Image Processing (NJ: Prentice Hall, 2002).
  11. S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).
  12. S. Boyd and L. Vandenberghe, Convex Optimization (UK: Cambridge University Press, 2004).
  13. S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.
  14. G. Strang and T. Nguyen, Wavelets and Filter Banks (MA: Wellesley-Cambridge Press, 1996).
  15. “The Panorama Factory V4.4 for Windows XP,” http://www.panoramafactory.com/.

2006 (2)

P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. 28, 673–683 (2006).
[Crossref]

S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).

2004 (1)

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
[Crossref]

1996 (1)

C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. 42, 981–990 (1996).
[Crossref]

1990 (1)

F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. 1, 21–46 (1990).
[Crossref]

Araújo, F. P.

F. P. Araújo and N. J. Leite, “A morphological algorithm for photomosaicking,” in Proceedings of 8th European Signal Processing Conference (EURASIP, 1996), pp. 1881–1884.

Beucher, S.

F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. 1, 21–46 (1990).
[Crossref]

Boyd, S.

S. Boyd and L. Vandenberghe, Convex Optimization (UK: Cambridge University Press, 2004).

Cheng, K. Y.

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
[Crossref]

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in Proceedings of International Conference on Image Processing (IEEE, 2001), pp. 245–245.

Eden, A.

M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 509–516.

Efros, A. A.

A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques (ACM, 2001), pp. 341–346.

Freeman, W. T.

A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques (ACM, 2001), pp. 341–346.

Gonzalez, R.

R. Gonzalez and R. Woods, Digital Image Processing (NJ: Prentice Hall, 2002).

Hsu, C. T.

C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. 42, 981–990 (1996).
[Crossref]

Hwang, W. L.

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
[Crossref]

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in Proceedings of International Conference on Image Processing (IEEE, 2001), pp. 245–245.

Lam, E. Y.

S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).

S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.

Leite, N. J.

F. P. Araújo and N. J. Leite, “A morphological algorithm for photomosaicking,” in Proceedings of 8th European Signal Processing Conference (EURASIP, 1996), pp. 1881–1884.

Levin, A.

A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.

Meyer, F.

F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. 1, 21–46 (1990).
[Crossref]

Nguyen, T.

G. Strang and T. Nguyen, Wavelets and Filter Banks (MA: Wellesley-Cambridge Press, 1996).

Peleg, S.

A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.

Soille, P.

P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. 28, 673–683 (2006).
[Crossref]

Strang, G.

G. Strang and T. Nguyen, Wavelets and Filter Banks (MA: Wellesley-Cambridge Press, 1996).

Su, M. S.

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
[Crossref]

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in Proceedings of International Conference on Image Processing (IEEE, 2001), pp. 245–245.

Suen, S. T.

S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).

S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.

Szeliski, R.

M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 509–516.

Uyttendaele, M.

M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 509–516.

Vandenberghe, L.

S. Boyd and L. Vandenberghe, Convex Optimization (UK: Cambridge University Press, 2004).

Weiss, Y.

A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.

Wong, K. K.

S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).

S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.

Woods, R.

R. Gonzalez and R. Woods, Digital Image Processing (NJ: Prentice Hall, 2002).

Wu, J. L.

C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. 42, 981–990 (1996).
[Crossref]

Zomet, A.

A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.

IEEE Trans. Cons. Elec. (1)

C. T. Hsu and J. L. Wu, “Multiresolution mosaic,” IEEE Trans. Cons. Elec. 42, 981–990 (1996).
[Crossref]

IEEE Trans. Imag. Proc. (1)

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Analysis on multiresolution mosaic images,” IEEE Trans. Imag. Proc. 13, 952–959 (2004).
[Crossref]

IEEE Trans. Pat. Anal. and Mach. Intell. (1)

P. Soille, “Morphological image compositing,” IEEE Trans. Pat. Anal. and Mach. Intell. 28, 673–683 (2006).
[Crossref]

J. Vis. Comm. and Imag. Rep. (1)

F. Meyer and S. Beucher, “Morphological segmentation,” J. Vis. Comm. and Imag. Rep. 1, 21–46 (1990).
[Crossref]

Proc. SPIE (1)

S. T. Suen, E. Y. Lam, and K. K. Wong, “Digital photograph stitching with optimized matching of gradient and curvature,” Proc. SPIE 6069, 139–154 (2006).

Other (10)

S. Boyd and L. Vandenberghe, Convex Optimization (UK: Cambridge University Press, 2004).

S. T. Suen, E. Y. Lam, and K. K. Wong, “Photographic mosaic for camera phones based on minimization of curvature value variations,” Tech. rep., Department of Electrical and Electronic Engineering, The University of Hong Kong (2006), http://www.eee.hku.hk/research/research_reports.htm.

G. Strang and T. Nguyen, Wavelets and Filter Banks (MA: Wellesley-Cambridge Press, 1996).

“The Panorama Factory V4.4 for Windows XP,” http://www.panoramafactory.com/.

F. P. Araújo and N. J. Leite, “A morphological algorithm for photomosaicking,” in Proceedings of 8th European Signal Processing Conference (EURASIP, 1996), pp. 1881–1884.

M. Uyttendaele, A. Eden, and R. Szeliski, “Eliminating ghosting and exposure artifacts in image mosaics,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2001), pp. 509–516.

A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching in the gradient domain,” in Proceedings of 8th European Conference on Computer Vision (2004), pp. 377–389.

A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of SIGGRAPH ’01: 28th Annual Conference on Computer Graphics and Interactive Techniques (ACM, 2001), pp. 341–346.

R. Gonzalez and R. Woods, Digital Image Processing (NJ: Prentice Hall, 2002).

M. S. Su, W. L. Hwang, and K. Y. Cheng, “Variational calculus approach to multiresolution image mosaic,” in Proceedings of International Conference on Image Processing (IEEE, 2001), pp. 245–245.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1.
Fig. 1. Input photographs with moving objects warped by commercial software. The rectangular boxes roughly indicate their overlap region.

Download Full Size | PPT Slide | PDF

Fig. 2.
Fig. 2. (a) (b) Results of pixel selection according to the curve generated by Araújo and Leite, and Soille, respectively. The cutting curves are indicated by black lines. The rectangular boxes mark the regions where the curve passes through a moving object.

Download Full Size | PPT Slide | PDF

Fig. 3.
Fig. 3. (a) Result of pixel selection according to the curve generated by our method. No moving object is cut. (b) Binary mask α(x, y) corresponds to the curve.

Download Full Size | PPT Slide | PDF

Fig. 4.
Fig. 4. Stitching Result of (a) Exposure Compensation, (b) Wavelet Blending, (c) Gradient Stitching and (d) Our method, assigning 1 to the middle 85% vertical portion of β 2(x,y).

Download Full Size | PPT Slide | PDF

Fig. 5.
Fig. 5. A zoom-in region of the result (a) Wavelet Blending, (b) Gradient Stitching and (c) Our method. They illustrate the transition across the cutting curve.

Download Full Size | PPT Slide | PDF

Fig. 6.
Fig. 6. (a) and (b) Input photographs for the stitching example described in [13].

Download Full Size | PPT Slide | PDF

Tables (3)

Tables Icon

Table 1. Summary of the smoothness metric (Ms ) and fidelity metric (Mf ) for the stitching methods: (a) Exposure Compensation, (b) Wavelet Blending, (c) Gradient Stitching, (d) Our method minimizing second derivatives over the middle 65% vertical portion, and (e) minimizing over the middle 85% portion.

View Table | View all tables in this article
Tables Icon

Table 2. Summary of the smoothness metric (Ms ) and fidelity metric (Mf ) for the stitching methods: (a) Exposure Compensation, (b) Wavelet Blending, (c) Gradient Stitching, (d) Our method minimizing second derivatives over the overlap region and (e) Minimizing throughout the mosaic.

View Table | View all tables in this article
Tables Icon

Table 3. Summary of the average memory usage (M), total running time (Tt ) and the ratio of time consumed by optimization (To ) to the total running time when stitching with different number of decomposition levels.

View Table | View all tables in this article

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

Ψ { g ̂ x y } = ∑ i = 0 2 ( ∑ x y ∈ Ω 1 α x y β i x y ∣ D i ( g ̂ x y − g 1 x y ) ∣ + ∑ x y ∈ Ω 2 ( 1 − α x y ) β i x y ∣ D i ( g ̂ x y − g 2 x y ) ∣ ) .
a x y = ∣ g 1 x y − g 2 x y ∣ .
b x y = ρ A ( g 1 x y ) ∧ ρ A ( g 2 x y ) ,
c x y = b x y a x y .
g ̂ 0 x y = α x y g 1 x y + ( 1 − α x y ) g 2 x y .

Metrics