Abstract

Conventional imaging techniques adopt a rectilinear sampling approach, where a finite number of pixels are spread evenly across an entire field of view (FOV). Consequently, their imaging capabilities are limited by an inherent trade-off between the FOV and the resolving power. In contrast, a foveation technique allocates the limited resources (e.g., a finite number of pixels or transmission bandwidth) as a function of foveal eccentricities, which can significantly simplify the optical and electronic designs and reduce the data throughput, while the observer's ability to see fine details is maintained over the whole FOV. We explore an approach to a foveated imaging system design. Our approach approximates the spatially variant properties (i.e., resolution, contrast, and color sensitivities) of the human visual system with multiple low-cost off-the-shelf imaging sensors and maximizes the information throughput and bandwidth savings of the foveated system. We further validate our approach with the design of a compact dual-sensor foveated imaging system. A proof-of-concept bench prototype and experimental results are demonstrated.

© 2008 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |

  1. G. Scotti, L. Marcenaro, C. Coelho, F. Selvaggi, and C. S. Regazzoni, "Dual camera intelligent sensor for high definition 360 degrees surveillance," IEE Proc. Vision Image Signal Process. 152, 250-257 (2005).
    [CrossRef]
  2. T. Ienaga, K. Matsunaga, K. Shidoji, K. Goshi, Y. Matsuki, and H. Nagata, "Stereoscopic video system with embedded high spatial resolution images using two channels for transmission," in Proceedings of ACM Symposium on Virtual Reality Software and Technology (2001), pp. 111-118.
    [CrossRef]
  3. A. Ude, C. Gaskett, and G. Cheng, "Foveated vision systems with two cameras per eye," in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2006), pp. 3457-3462.
  4. C. Wang, P. Shumyatsky, F. Zeng, M. Zevallos, and R. R. Alfano, "Computer-controlled optical scanning tile microscope," Appl. Opt. 45, 1148-1152 (2006).
    [CrossRef] [PubMed]
  5. B. Potsaid, Y. Bellouard, and J. T. Wen, "Adaptive scanning optical microscope (ASOM): a multidisciplinary optical microscope design for large field of view and high resolution imaging," Opt. Express 13, 6504-6518 (2005).
    [CrossRef] [PubMed]
  6. M. Böhme, M. Dorr, T. Martinetz, and E. Barth, "Gaze-contingent temporal filtering of video," in Proceedings of ACM Symposium on Eye Tracking Research & Applications (2006), pp. 109-116.
    [CrossRef]
  7. W. Zhou and A. C. Bovik, "Embedded foveation image coding," IEEE Trans. Image Process. 10, 1397-1410 (2001).
    [CrossRef]
  8. W. S. Geisler and J. S. Perry, "Real-time foveated multiresolution system for low-bandwidth video communication," Proc. SPIE 3299, 294-305 (1998).
    [CrossRef]
  9. A. T. Duchowski and A. Çöltekin, "Foveated gaze-contingent displays for peripheral LOD management, 3D visualization, and stereo imaging," ACM Trans. Multimedia Comput. Commun. Appl. 3, 1-21 (2007).
    [CrossRef]
  10. H. Murphy and A. T. Duchowski, "Hybrid image-/model-based gaze-contingent rendering," in Proceedings of ACM Symposium on Applied Perception in Graphics and Visualization (2007), pp. 1-8.
  11. D. Luebke and B. Hallen, "Perceptually driven simplification for interactive rendering," in Proceedings of the 2001 Eurographics Workshop on Rendering (2001), pp. 223-234.
    [CrossRef]
  12. G. Sandini, P. Questa, D. Scheffer, and A. Mannucci, "A retina-like CMOS sensor and its applications," in Proceedings of IEEE Workshop on Sensor Array and Multichannel Signal Processing (IEEE, 2000), pp. 514-519.
  13. J. P. Rolland, A. Yoshida, L. D. Davis, and J. H. Reif, "High-resolution inset head-mounted display," Appl. Opt. 37, 4183-4193 (1998).
    [CrossRef]
  14. D. V. Wick, T. Martinez, S. R. Restaino, and B. R. Stone, "Foveated imaging demonstration," Opt. Express 10, 60-65 (2002).
    [PubMed]
  15. T. Martinez, D. V. Wick, and S. R. Restaino, "Foveated, wide field-of-view imaging system using a liquid crystal spatial light modulator," Opt. Express 8, 555-560 (2001).
    [CrossRef] [PubMed]
  16. E. M. Reingold, L. C. Loschky, G. W. McConkie, and D. M. Stampe, "Gaze-contingent multiresolutional displays: an integrative review," Hum. Factors 45, 307-328 (2003).
    [CrossRef] [PubMed]
  17. L. C. Loschky and G. S. Wolverton, "How late can you update gaze-contingent multiresolutional displays without detection?," ACM Trans. Multimedia Comput. Commun. Appl. 3 (2007).
    [CrossRef]
  18. W. S. Geisler, J. S. Perry, and J. Najemnik, "Visual search: the role of peripheral information measured using gaze-contingent displays," J. Vision 6, 858-873 (2006).
    [CrossRef]
  19. L. C. Loschky, G. W. McConkie, H. Yang, and M. E. Miller, "The limits of visual resolution in natural scene viewing," Visual Cognition 12, 1057-1092 (2005).
    [CrossRef]
  20. M. Dorr, M. Böhme, T. Martinetz, and E. Barth, "Visibility of temporal blur on a gaze-contingent display," in Proceedings of the ACM Symposium on Applied Perception, Graphics & Visualization (2005), pp. 33-36.
  21. D. J. Parkhurst and E. Niebur, "Variable-resolution displays: a theoretical, practical, and behavioural evaluation," Hum. Factors 44, 611-629 (2002).
    [CrossRef]
  22. L. C. Loschky and G. W. McConkie, "Investigating spatial vision and dynamic attentional selection using a gaze-contingent multiresolutional display," Q. J. Exp. Psychol. A 8, 99-117 (2002).
  23. R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, and M. Z. Zhang, "A foveated silicon retina for two-dimensional tracking," IEEE Trans. Circuits Syst. II 47, 504-517 (2000).
    [CrossRef]
  24. G. Godin, P. Massicotte, and L. Borgeat, "High-resolution insets in projector-based stereoscopic displays: principles and techniques," Proc. SPIE 6055, 60550F (2006).
    [CrossRef]
  25. S. J. D. Prince, J. H. Elder, Y. Hou, and M. Sizinstev, "Pre-attentive face detection for foveated wide-field surveillance," in IEEE Workshops on Application of Computer Vision (IEEE, 2005), pp. 439-446.
  26. Peter G. J. Barten, Contrast Sensitivity of the Human Eye and Its Effects on Image Quality (SPIE Optical Engineering Press, 1999).
    [CrossRef]
  27. C. Gao, N. Ahuja, and H. Hua, "Active aperture control and sensor modulation for flexible imaging," in Proceedings of International Conference on Computer Vision and Pattern Recognition (2007).
  28. J. E. Greivenkamp, Field Guide to Geometrical Optics (SPIE Press, 2004).
    [CrossRef]
  29. Camera calibration toolbox, http://www.vision.caltech.edu/bouguetj/calib_doc/index.html.

2007 (2)

A. T. Duchowski and A. Çöltekin, "Foveated gaze-contingent displays for peripheral LOD management, 3D visualization, and stereo imaging," ACM Trans. Multimedia Comput. Commun. Appl. 3, 1-21 (2007).
[CrossRef]

L. C. Loschky and G. S. Wolverton, "How late can you update gaze-contingent multiresolutional displays without detection?," ACM Trans. Multimedia Comput. Commun. Appl. 3 (2007).
[CrossRef]

2006 (3)

W. S. Geisler, J. S. Perry, and J. Najemnik, "Visual search: the role of peripheral information measured using gaze-contingent displays," J. Vision 6, 858-873 (2006).
[CrossRef]

G. Godin, P. Massicotte, and L. Borgeat, "High-resolution insets in projector-based stereoscopic displays: principles and techniques," Proc. SPIE 6055, 60550F (2006).
[CrossRef]

C. Wang, P. Shumyatsky, F. Zeng, M. Zevallos, and R. R. Alfano, "Computer-controlled optical scanning tile microscope," Appl. Opt. 45, 1148-1152 (2006).
[CrossRef] [PubMed]

2005 (3)

B. Potsaid, Y. Bellouard, and J. T. Wen, "Adaptive scanning optical microscope (ASOM): a multidisciplinary optical microscope design for large field of view and high resolution imaging," Opt. Express 13, 6504-6518 (2005).
[CrossRef] [PubMed]

L. C. Loschky, G. W. McConkie, H. Yang, and M. E. Miller, "The limits of visual resolution in natural scene viewing," Visual Cognition 12, 1057-1092 (2005).
[CrossRef]

G. Scotti, L. Marcenaro, C. Coelho, F. Selvaggi, and C. S. Regazzoni, "Dual camera intelligent sensor for high definition 360 degrees surveillance," IEE Proc. Vision Image Signal Process. 152, 250-257 (2005).
[CrossRef]

2003 (1)

E. M. Reingold, L. C. Loschky, G. W. McConkie, and D. M. Stampe, "Gaze-contingent multiresolutional displays: an integrative review," Hum. Factors 45, 307-328 (2003).
[CrossRef] [PubMed]

2002 (3)

D. J. Parkhurst and E. Niebur, "Variable-resolution displays: a theoretical, practical, and behavioural evaluation," Hum. Factors 44, 611-629 (2002).
[CrossRef]

L. C. Loschky and G. W. McConkie, "Investigating spatial vision and dynamic attentional selection using a gaze-contingent multiresolutional display," Q. J. Exp. Psychol. A 8, 99-117 (2002).

D. V. Wick, T. Martinez, S. R. Restaino, and B. R. Stone, "Foveated imaging demonstration," Opt. Express 10, 60-65 (2002).
[PubMed]

2001 (2)

2000 (1)

R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, and M. Z. Zhang, "A foveated silicon retina for two-dimensional tracking," IEEE Trans. Circuits Syst. II 47, 504-517 (2000).
[CrossRef]

1998 (2)

W. S. Geisler and J. S. Perry, "Real-time foveated multiresolution system for low-bandwidth video communication," Proc. SPIE 3299, 294-305 (1998).
[CrossRef]

J. P. Rolland, A. Yoshida, L. D. Davis, and J. H. Reif, "High-resolution inset head-mounted display," Appl. Opt. 37, 4183-4193 (1998).
[CrossRef]

ACM Trans. Multimedia Comput. Commun. Appl. (1)

A. T. Duchowski and A. Çöltekin, "Foveated gaze-contingent displays for peripheral LOD management, 3D visualization, and stereo imaging," ACM Trans. Multimedia Comput. Commun. Appl. 3, 1-21 (2007).
[CrossRef]

ACM Trans. Multimedia Comput. Commun. Appl. (1)

L. C. Loschky and G. S. Wolverton, "How late can you update gaze-contingent multiresolutional displays without detection?," ACM Trans. Multimedia Comput. Commun. Appl. 3 (2007).
[CrossRef]

Appl. Opt. (2)

Hum. Factors (2)

D. J. Parkhurst and E. Niebur, "Variable-resolution displays: a theoretical, practical, and behavioural evaluation," Hum. Factors 44, 611-629 (2002).
[CrossRef]

E. M. Reingold, L. C. Loschky, G. W. McConkie, and D. M. Stampe, "Gaze-contingent multiresolutional displays: an integrative review," Hum. Factors 45, 307-328 (2003).
[CrossRef] [PubMed]

IEE Proc. Vision Image Signal Process. (1)

G. Scotti, L. Marcenaro, C. Coelho, F. Selvaggi, and C. S. Regazzoni, "Dual camera intelligent sensor for high definition 360 degrees surveillance," IEE Proc. Vision Image Signal Process. 152, 250-257 (2005).
[CrossRef]

IEEE Trans. Circuits Syst. (1)

R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, and M. Z. Zhang, "A foveated silicon retina for two-dimensional tracking," IEEE Trans. Circuits Syst. II 47, 504-517 (2000).
[CrossRef]

IEEE Trans. Image Process. (1)

W. Zhou and A. C. Bovik, "Embedded foveation image coding," IEEE Trans. Image Process. 10, 1397-1410 (2001).
[CrossRef]

J. Vision (1)

W. S. Geisler, J. S. Perry, and J. Najemnik, "Visual search: the role of peripheral information measured using gaze-contingent displays," J. Vision 6, 858-873 (2006).
[CrossRef]

Opt. Express (3)

Proc. SPIE (2)

G. Godin, P. Massicotte, and L. Borgeat, "High-resolution insets in projector-based stereoscopic displays: principles and techniques," Proc. SPIE 6055, 60550F (2006).
[CrossRef]

W. S. Geisler and J. S. Perry, "Real-time foveated multiresolution system for low-bandwidth video communication," Proc. SPIE 3299, 294-305 (1998).
[CrossRef]

Q. J. Exp. Psychol. A (1)

L. C. Loschky and G. W. McConkie, "Investigating spatial vision and dynamic attentional selection using a gaze-contingent multiresolutional display," Q. J. Exp. Psychol. A 8, 99-117 (2002).

Visual Cognition (1)

L. C. Loschky, G. W. McConkie, H. Yang, and M. E. Miller, "The limits of visual resolution in natural scene viewing," Visual Cognition 12, 1057-1092 (2005).
[CrossRef]

Other (12)

M. Dorr, M. Böhme, T. Martinetz, and E. Barth, "Visibility of temporal blur on a gaze-contingent display," in Proceedings of the ACM Symposium on Applied Perception, Graphics & Visualization (2005), pp. 33-36.

S. J. D. Prince, J. H. Elder, Y. Hou, and M. Sizinstev, "Pre-attentive face detection for foveated wide-field surveillance," in IEEE Workshops on Application of Computer Vision (IEEE, 2005), pp. 439-446.

Peter G. J. Barten, Contrast Sensitivity of the Human Eye and Its Effects on Image Quality (SPIE Optical Engineering Press, 1999).
[CrossRef]

C. Gao, N. Ahuja, and H. Hua, "Active aperture control and sensor modulation for flexible imaging," in Proceedings of International Conference on Computer Vision and Pattern Recognition (2007).

J. E. Greivenkamp, Field Guide to Geometrical Optics (SPIE Press, 2004).
[CrossRef]

Camera calibration toolbox, http://www.vision.caltech.edu/bouguetj/calib_doc/index.html.

M. Böhme, M. Dorr, T. Martinetz, and E. Barth, "Gaze-contingent temporal filtering of video," in Proceedings of ACM Symposium on Eye Tracking Research & Applications (2006), pp. 109-116.
[CrossRef]

H. Murphy and A. T. Duchowski, "Hybrid image-/model-based gaze-contingent rendering," in Proceedings of ACM Symposium on Applied Perception in Graphics and Visualization (2007), pp. 1-8.

D. Luebke and B. Hallen, "Perceptually driven simplification for interactive rendering," in Proceedings of the 2001 Eurographics Workshop on Rendering (2001), pp. 223-234.
[CrossRef]

G. Sandini, P. Questa, D. Scheffer, and A. Mannucci, "A retina-like CMOS sensor and its applications," in Proceedings of IEEE Workshop on Sensor Array and Multichannel Signal Processing (IEEE, 2000), pp. 514-519.

T. Ienaga, K. Matsunaga, K. Shidoji, K. Goshi, Y. Matsuki, and H. Nagata, "Stereoscopic video system with embedded high spatial resolution images using two channels for transmission," in Proceedings of ACM Symposium on Virtual Reality Software and Technology (2001), pp. 111-118.
[CrossRef]

A. Ude, C. Gaskett, and G. Cheng, "Foveated vision systems with two cameras per eye," in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2006), pp. 3457-3462.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1

(Color online) Sampling efficiency of a SRI system as a function of the overall FOV.

Fig. 2
Fig. 2

(Color online) Optimization of a dual-resolution foveated imaging system: (a) eccentricity of visual acuity and dual-resolution sampling scheme; (b) bandwidth saving ratio as a function of peripheral and foveated FOVs; (c) maximum bandwidth saving in percentile (blue curve with square marker) and optimal ratio of foveated FOV to overall FOV (green curve with + marker).

Fig. 3
Fig. 3

(Color online) Contrast sensitivity and image bit depth: (a) the HVS contrast sensitivity as a function of spatial frequency and eccentricity; (b) maximum contrast modulation change as a function of reduced bit depth.

Fig. 4
Fig. 4

(Color online) Bandwidth saving ratio of a DRDB system as a function of overall FOV and reduced bit depth.

Fig. 5
Fig. 5

(Color online) Simulated foveated images: (a) original SRI ( 2400 × 1800 24 bit pixels); (b) DRDB image ( 691 × 518 24 bit pixels in the center and 766 × 574 15 bit pixels in the peripheral); (c) DRDB image ( 691 × 518 24 bit pixels in the center and 766 × 574 8 bit gray-scale pixels in the peripheral); (d)–(f) enlarged views of a narrow region of images (a)–(c).

Fig. 6
Fig. 6

(Color online) Schematic design of a dual-sensor foveated imaging system.

Fig. 7
Fig. 7

(Color online) (a) Bench prototype of a dual-sensor foveated imaging system and the pupil conjugation on the MEMS chip (inset); (b) the MEMS chip.

Fig. 8
Fig. 8

(Color online) Registration and mosaic of (a) a peripheral image with (b) a foveated image.

Fig. 9
Fig. 9

(Color online) Mosaic of a sequence of foveated and peripheral images.

Fig. 10
Fig. 10

(Color online) Image quality characterization: MTF50 versus the field angle of foveated image center.

Equations (13)

Equations on this page are rendered with MathJax. Learn more.

B = θ X / 2 θ X / 2 θ Y / 2 θ Y / 2 F 2 ( e x , e y ) d e x d e y ,
E = θ X / 2 θ X / 2 θ Y / 2 θ Y / 2 F HVS 2 ( e x , e y ) d e x d e y θ X / 2 θ X / 2 θ Y / 2 θ Y / 2 F 2 ( e x , e y ) d e x d e y .
S DRI = B SRI B DRI B SRI = θ X θ Y ( θ X θ Y θ X F θ Y F ) F DRI 2 ( θ X F , θ Y F ) θ X F θ Y F θ X θ Y .
CSF ( e , ν ) = 1 C T ( e , ν ) = CSF ( 0 , ν ) exp ( α ν e 2 + e e 2 ) ,
Δ C i , j ( N , M ) | max < C T ( θ F , ν 60 μ f + μ p )
( i = 0 , 1 ,   …   2 N 1 , j = 0 , 1 ,   …   2 M 1 ) ,
S DRDB = B SRI B DRDB B SRI = 2 N θ X θ Y 2 M ( θ X θ Y θ X F θ Y F ) F DRI 2 ( θ X F , θ Y F ) 2 N θ X F θ Y F 2 N θ X θ Y .
θ c = arctan [ f 1 f 2 tan ( 2 ϕ m ) f 1 2 + Δ t Δ t ] ,
θ r = arctan [ D F ( f 1 2 + Δ t Δ t ) 2 f 1 2 f 2 ] ,
θ F | max = 2 arctan [ f 1 f 2 tan ( 2 ϕ max + θ r ) f 1 2 + Δ t Δ t ] .
y F = f 1 2 f 2 tan ( 2 ϕ m θ 2 ) f 1 2 Δ t Δ t ,
L X P = f 2 2 Δ t f 1 2 + Δ t Δ t .
D EP D MEMS | m p | ,

Metrics