Abstract

We report on the image formation pipeline developed to efficiently form gigapixel-scale imagery generated by the AWARE-2 multiscale camera. The AWARE-2 camera consists of 98 “microcameras” imaging through a shared spherical objective, covering a 120° x 50° field of view with approximately 40 microradian instantaneous field of view (the angular extent of a pixel). The pipeline is scalable, capable of producing imagery ranging in scope from “live” one megapixel views to full resolution gigapixel images. Architectural choices that enable trivially parallelizable algorithms for rapid image formation and on-the-fly microcamera alignment compensation are discussed.

© 2012 OSA

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. D. J. Brady and N. Hagen, “Multiscale lens design,” Opt. Express 17(13), 10659–10674 (2009).
    [CrossRef] [PubMed]
  2. E. J. Tremblay, D. L. Marks, D. J. Brady, and J. E. Ford, “Design and scaling of monocentric multiscale imagers,” Appl. Opt. 51(20), 4691–4702 (2012).
    [CrossRef] [PubMed]
  3. D. L. Marks, E. J. Tremblay, J. E. Ford, and D. J. Brady, “Microcamera aperture scale in monocentric gigapixel cameras,” Appl. Opt. 50(30), 5824–5833 (2011).
    [CrossRef] [PubMed]
  4. H. S. Son, D. L. Marks, J. Hahn, J. Kim, and D. J. Brady, “Design of a spherical focal surface using close-packed relay optics,” Opt. Express 19(17), 16132–16138 (2011).
    [CrossRef] [PubMed]
  5. R. Szeliski, Computer Vision: Algorithms and Applications (Springer, 2010).
  6. J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Commun. ACM 51(1), 107–113 (2008).
    [CrossRef]
  7. H. Haggrén, “Photogrammetric machine vision,” Opt. Lasers Eng. 10(3-4), 265–286 (1989).
    [CrossRef]
  8. C. Fraser, “Digital camera self-calibration,” ISPRS J. Photogramm. Remote Sens. 52(4), 149–159 (1997).
    [CrossRef]
  9. S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, (Prentice Hall, 1993), Chapter 7.
  10. R. Willson and S. Shafer, “What is the center of the image?” J. Opt. Soc. Am. A 11(11), 2946–2955 (1994).
    [CrossRef]
  11. R. L. Graham, G. M. Shipman, B. W. Barrett, R. H. Castain, G. Bosilca, and A. Lumsdaine, “Open MPI: a high-performance, heterogeneous MPI,” in Proceedings of IEEE International Conference on Cluster Computing (IEEE, 2006),1–9, 25–28.
  12. NVIDIA Corporation, 2012. NVIDIA GeForce GTX 680 [White paper]. Retrieved from http://www.geforce.com/Active/en_US/en_US/pdf/GeForce-GTX-680-Whitepaper-FINAL.pdf
  13. R. Szeliski and H.-Y. Shum, “Creating full view panoramic image mosaics and environment maps,” in Proceedings of SIGGRAPH 1997, (New York, NY, 1997), 251–258.
  14. M. Brown and D. Lowe, “Automatic panoramic image stitching using invariant features,” Int. J. Comput. Vis. 74(1), 59–73 (2007).
    [CrossRef]
  15. B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).
    [CrossRef]
  16. L. G. Brown, “A survey of image registration techniques,” ACM Comput. Surv. 24(4), 325–376 (1992).
    [CrossRef]
  17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
    [CrossRef]
  18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
    [CrossRef]
  19. D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
    [CrossRef] [PubMed]
  20. J. Kopf, M. Uyttendaele, O. Deussen, and M. F. Cohen, “Capturing and viewing gigapixel images,” in Proceedings of SIGGRAPH 2007 (New York, NY, 2007).
  21. M. Ben-Ezra, “A digital gigapixel large-format tile-scan camera,” IEEE Comput. Graph. Appl. 31(1), 49–61 (2011).
    [CrossRef]
  22. E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (2005).
  23. E. M. Vera, D. R. Golish, D. S. Kittle, D. J. Brady, and M. E. Gehm, “A parallel processing approach for efficient rendering of high dynamic-range gigapixel images,” in preparation for submission to Image and Vision Computing (2012).
  24. O. Cossairt, D. Miau, and S. K. Nayar, “Gigapixel computational imaging,” in IEEE International Conference on Computational Photography (IEEE, 2011).

2012 (2)

E. J. Tremblay, D. L. Marks, D. J. Brady, and J. E. Ford, “Design and scaling of monocentric multiscale imagers,” Appl. Opt. 51(20), 4691–4702 (2012).
[CrossRef] [PubMed]

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

2011 (3)

2009 (1)

2008 (2)

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Commun. ACM 51(1), 107–113 (2008).
[CrossRef]

2007 (1)

M. Brown and D. Lowe, “Automatic panoramic image stitching using invariant features,” Int. J. Comput. Vis. 74(1), 59–73 (2007).
[CrossRef]

2004 (1)

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[CrossRef]

2003 (1)

B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).
[CrossRef]

1997 (1)

C. Fraser, “Digital camera self-calibration,” ISPRS J. Photogramm. Remote Sens. 52(4), 149–159 (1997).
[CrossRef]

1994 (1)

1992 (1)

L. G. Brown, “A survey of image registration techniques,” ACM Comput. Surv. 24(4), 325–376 (1992).
[CrossRef]

1989 (1)

H. Haggrén, “Photogrammetric machine vision,” Opt. Lasers Eng. 10(3-4), 265–286 (1989).
[CrossRef]

Bay, H.

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

Ben-Ezra, M.

M. Ben-Ezra, “A digital gigapixel large-format tile-scan camera,” IEEE Comput. Graph. Appl. 31(1), 49–61 (2011).
[CrossRef]

Brady, D. J.

Brown, L. G.

L. G. Brown, “A survey of image registration techniques,” ACM Comput. Surv. 24(4), 325–376 (1992).
[CrossRef]

Brown, M.

M. Brown and D. Lowe, “Automatic panoramic image stitching using invariant features,” Int. J. Comput. Vis. 74(1), 59–73 (2007).
[CrossRef]

Dean, J.

J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Commun. ACM 51(1), 107–113 (2008).
[CrossRef]

Ess, A.

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

Feller, S. D.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Flusser, J.

B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).
[CrossRef]

Ford, J. E.

Fraser, C.

C. Fraser, “Digital camera self-calibration,” ISPRS J. Photogramm. Remote Sens. 52(4), 149–159 (1997).
[CrossRef]

Gehm, M. E.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Ghemawat, S.

J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Commun. ACM 51(1), 107–113 (2008).
[CrossRef]

Golish, D. R.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Hagen, N.

Haggrén, H.

H. Haggrén, “Photogrammetric machine vision,” Opt. Lasers Eng. 10(3-4), 265–286 (1989).
[CrossRef]

Hahn, J.

Kim, J.

Kittle, D. S.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Lowe, D.

M. Brown and D. Lowe, “Automatic panoramic image stitching using invariant features,” Int. J. Comput. Vis. 74(1), 59–73 (2007).
[CrossRef]

Lowe, D. G.

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[CrossRef]

Marks, D. L.

Shafer, S.

Son, H. S.

Stack, R. A.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Tremblay, E. J.

Tuytelaars, T.

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

Van Gool, L.

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

Vera, E. M.

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Willson, R.

Zitová, B.

B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).
[CrossRef]

ACM Comput. Surv. (1)

L. G. Brown, “A survey of image registration techniques,” ACM Comput. Surv. 24(4), 325–376 (1992).
[CrossRef]

Appl. Opt. (2)

Commun. ACM (1)

J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Commun. ACM 51(1), 107–113 (2008).
[CrossRef]

Comput. Vis. Image Underst. (1)

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).
[CrossRef]

IEEE Comput. Graph. Appl. (1)

M. Ben-Ezra, “A digital gigapixel large-format tile-scan camera,” IEEE Comput. Graph. Appl. 31(1), 49–61 (2011).
[CrossRef]

Image Vis. Comput. (1)

B. Zitová and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).
[CrossRef]

Int. J. Comput. Vis. (2)

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).
[CrossRef]

M. Brown and D. Lowe, “Automatic panoramic image stitching using invariant features,” Int. J. Comput. Vis. 74(1), 59–73 (2007).
[CrossRef]

ISPRS J. Photogramm. Remote Sens. (1)

C. Fraser, “Digital camera self-calibration,” ISPRS J. Photogramm. Remote Sens. 52(4), 149–159 (1997).
[CrossRef]

J. Opt. Soc. Am. A (1)

Nature (1)

D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature 486(7403), 386–389 (2012).
[CrossRef] [PubMed]

Opt. Express (2)

Opt. Lasers Eng. (1)

H. Haggrén, “Photogrammetric machine vision,” Opt. Lasers Eng. 10(3-4), 265–286 (1989).
[CrossRef]

Other (9)

S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, (Prentice Hall, 1993), Chapter 7.

R. Szeliski, Computer Vision: Algorithms and Applications (Springer, 2010).

R. L. Graham, G. M. Shipman, B. W. Barrett, R. H. Castain, G. Bosilca, and A. Lumsdaine, “Open MPI: a high-performance, heterogeneous MPI,” in Proceedings of IEEE International Conference on Cluster Computing (IEEE, 2006),1–9, 25–28.

NVIDIA Corporation, 2012. NVIDIA GeForce GTX 680 [White paper]. Retrieved from http://www.geforce.com/Active/en_US/en_US/pdf/GeForce-GTX-680-Whitepaper-FINAL.pdf

R. Szeliski and H.-Y. Shum, “Creating full view panoramic image mosaics and environment maps,” in Proceedings of SIGGRAPH 1997, (New York, NY, 1997), 251–258.

J. Kopf, M. Uyttendaele, O. Deussen, and M. F. Cohen, “Capturing and viewing gigapixel images,” in Proceedings of SIGGRAPH 2007 (New York, NY, 2007).

E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (2005).

E. M. Vera, D. R. Golish, D. S. Kittle, D. J. Brady, and M. E. Gehm, “A parallel processing approach for efficient rendering of high dynamic-range gigapixel images,” in preparation for submission to Image and Vision Computing (2012).

O. Cossairt, D. Miau, and S. K. Nayar, “Gigapixel computational imaging,” in IEEE International Conference on Computational Photography (IEEE, 2011).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1

The microcameras are tiled on the surface of a hemisphere to optimally cover the full system FoV without gaps. The projection of the microcamera FoVs into object space is shown in (a), where the microcameras populated in AWARE-2 have been highlighted. The machined aluminum geodesic dome, approximately 11.5” in diameter, which holds the microcameras in this configuration, is shown in (b).

Fig. 2
Fig. 2

The MapReduce approach breaks the image formation process into two parts. The map step transforms a list of key/value pairs that represent the intensity value for a given pixel on a given microcamera into an intermediate list of key/value pairs which represent the intensity value for a given location in object space. This location corresponds directly to a pixel in the output image. The reduce step combines key/value pairs sharing the same key to form an estimate the intensity that was present at that single location in object space.

Fig. 3
Fig. 3

Parametric models are used to predict the distortion and relative illumination as a function of radial position on a sensor. (a) Comparison of several polynomial functions to the distortion found in a ZEMAX simulation, demonstrating that a 9th order polynomial is sufficient to achieve a pixel-accurate distortion prediction. (b) Fit for the relative illumination model using an 8th order polynomial.

Fig. 4
Fig. 4

Values from every pixel from every microcamera are mapped into object space. Pixels that overlap in the shared coordinate system are reduced to a single value. These operations can run in parallel on every pixel to quickly form a stitched output image. The set of images on the left depicts a collection of detector outputs. The image on the right is a portion of the final image generated from this group of individual microcamera images that have been positioned with an understanding of the geometry of the array.

Fig. 5
Fig. 5

(a) Deviations from the as-designed illumination model produce microcamera imagery with stray light artifacts. (b) Compositing with this data creates rings in the stitched image. (c) Flat-field measurements can be taken on a per-microcamera basis. (d) Camera data can be corrected with the flat fields to reduce stray light effects. (e) Compositing with the corrected data produces a nearly-seamless composite image.

Fig. 6
Fig. 6

AWARE utilizes the Message Passing Interface (MPI) framework to distribute compositing work among a pool of workers. Each processing core in each server is designated a worker. The root node receives commands via some user interface and distributes the jobs to the workers.

Fig. 7
Fig. 7

The relative time to composite an image decreases as more workers are used in the computation. This experiment was done on an NVIDIA GTX 570 with 480 cores, thus requesting more workers than are available results in a reduced performance gain.

Fig. 8
Fig. 8

(a) Registration errors are not noticeable when viewing a wide FoV. (b) In narrow FoV composites, misalignment between neighboring microcameras can be clearly seen.

Fig. 9
Fig. 9

SIFT and SURF algorithms are used to identify clusters of features (shown as markers in the images) in neighboring microcameras. The clusters are transformed into object space and compared to calculate a registration error. The transformation parameters are adjusted to minimize the error.

Fig. 10
Fig. 10

(a) A composite formed with an unregistered camera angles will have stitching errors due to mechanical and thermal drift, as shown in this overlap region between three cameras. (b) The extracted features can be used to find a globally optimal registration, leading to an improved composited image.

Fig. 11
Fig. 11

A composited, tone-mapped HDR image from the AWARE-2 camera using the proposed image formation architecture. Each microcamera in the array automatically chooses a focal position and exposure time optimized for the distances and intensities found in the portion of the scene it is imaging.

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

 Θ= j=1 5 K j r 2j1
m k = s k I (θ,ϕ) + n k ,
s k = b k t k .
I ^ (θ,ϕ) = k=1 n b k t k m k k=1 n b k 2 t k 2 .
 b= j=1 8 C j r α

Metrics