Abstract

We present a novel head-mounted display setup that uses the pinhole imaging principle coupled with a low-latency dynamic pupil follower. A transmissive LCD is illuminated by a single LED backlight. LED illumination is focused onto the viewer’s pupil to form an eyebox smaller than the average human pupil, thereby creating a pinhole display effect where objects at all distances appear in focus. Since nearly all the light is directed to the viewer’s pupil, a single low-power LED for each primary color with 0.42 lumens total output is sufficient to create a bright and full-color display of 360 cd/m2 luminance. In order to follow the viewer’s pupil, the eyebox needs to be steerable. We achieved a dynamic eyebox using an array of LEDs that is coupled with a real-time pupil tracker. The entire system is operated at 11 msec motion-to-photon latency, which meets the demanding requirements of the real-time pupil follower system. Experimental results effectively demonstrated our head-mounted pinhole display with 37° FOV and very high light efficiency, equipped with a pupil follower with low motion-to-photon latency.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Dual focal plane augmented reality interactive display with gaze-tracker

Uğur Yekta Başak, Seyedmahdi Kazempourradi, Cemalettin Yilmaz, Erdem Ulusoy, and Hakan Urey
OSA Continuum 2(5) 1734-1745 (2019)

Design and prototype of an augmented reality display with per-pixel mutual occlusion capability

Austin Wilson and Hong Hua
Opt. Express 25(24) 30539-30549 (2017)

High-performance integral-imaging-based light field augmented reality display using freeform optics

Hekun Huang and Hong Hua
Opt. Express 26(13) 17578-17590 (2018)

References

  • View by:
  • |
  • |
  • |

  1. J. P. Rolland, K. P. Thompson, A. Bauer, H. Urey, and M. Thomas, “See-through head-worn display (HWD) architectures.” Handbook of Visual Display Technology (Springer, 2011), pp. 2929–2961.
  2. J. F. Koretz and G. H. Handelman, “How the human eye focuses,” Sci. Am. 259(1), 92–99 (1988).
    [Crossref] [PubMed]
  3. G. Wyszecki and W. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, (John Wiley & Sons, 1982).
  4. J. F. Koretz and G. H. Handelman, “Modeling age-related accomodative loss in the human eye,” Math. Model. 7(5–8), 1003–1014 (1986).
    [Crossref]
  5. J. Yang, W. Liu, W. Lv, D. Zhang, F. He, Z. Wei, and Y. Kang, “Method of achieving a wide field-of-view head-mounted display with small distortion,” Opt. Lett. 38(12), 2035–2037 (2013).
    [Crossref] [PubMed]
  6. W. Song, D. Cheng, Z. Deng, Y. Liu, and Y. Wang, “Design and assessment of a wide FOV and high-resolution optical tiled head-mounted display,” Appl. Opt. 54(28), E15–E22 (2015).
    [Crossref] [PubMed]
  7. G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).
  8. S. Choi, Y. Takashima, and S. W. Min, “Improvement of fill factor in pinhole-type integral imaging display using a retroreflector,” Opt. Express 25(26), 33078–33087 (2017).
    [Crossref]
  9. C. Yao, D. Cheng, T. Yang, and Y. Wang, “Design of an optical see-through light-field near-eye display using a discrete lenslet array,” Opt. Express 26(14), 18292–18301 (2018).
    [Crossref] [PubMed]
  10. K. Akşit, J. Kautz, and D. Luebke, “Slim near-eye display using pinhole aperture arrays,” Appl. Opt. 54(11), 3422–3427 (2015).
    [Crossref] [PubMed]
  11. G. Westheimer, “The maxwellian view,” Vision Res. 6, 669–682 (1966).
    [Crossref] [PubMed]
  12. M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).
  13. R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
    [Crossref] [PubMed]
  14. T. Hideya and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” Proc. SPIE 6803, 68031N (2008).
  15. J. H. Park and S. B. Kim, “Optical see-through holographic near-eye-display with eyebox steering and depth of field control,” Opt. Express 26(21), 27076–27088 (2018).
    [Crossref] [PubMed]
  16. A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
    [Crossref]
  17. P. Frank L. and L. S. Pedrotti, Introduction to Optics, 3nd edition (Prentice Hall, 1993).
  18. C. H. Morimoto and M. R. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. 98(1), 4–24 (2005).
    [Crossref]
  19. G. Coley, Beaglebone Black System Reference Manual (Texas Instruments, 2013).
  20. J. W. Goodman, Introduction to Fourier Optics (Roberts and Company Publishers, 2005).
  21. A. E. Siegman, “How to (maybe) measure laser beam quality,” in Diode Pumped Solid State Lasers: Applications and Issues (Optical Society of America, 1998) paper MQ1.
  22. W. J. Smith, Modern Optical Engineering (Tata McGraw-Hill Education, 1966).

2018 (3)

2017 (2)

S. Choi, Y. Takashima, and S. W. Min, “Improvement of fill factor in pinhole-type integral imaging display using a retroreflector,” Opt. Express 25(26), 33078–33087 (2017).
[Crossref]

G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).

2015 (2)

2013 (1)

2008 (1)

T. Hideya and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” Proc. SPIE 6803, 68031N (2008).

2005 (2)

R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
[Crossref] [PubMed]

C. H. Morimoto and M. R. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. 98(1), 4–24 (2005).
[Crossref]

1988 (1)

J. F. Koretz and G. H. Handelman, “How the human eye focuses,” Sci. Am. 259(1), 92–99 (1988).
[Crossref] [PubMed]

1986 (1)

J. F. Koretz and G. H. Handelman, “Modeling age-related accomodative loss in the human eye,” Math. Model. 7(5–8), 1003–1014 (1986).
[Crossref]

1966 (1)

G. Westheimer, “The maxwellian view,” Vision Res. 6, 669–682 (1966).
[Crossref] [PubMed]

Aksit, K.

Bakholdin, A. V.

G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).

Beer, R. D.

R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
[Crossref] [PubMed]

Chen, L.

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Cheng, D.

Choi, S.

Chu, J.

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Coley, G.

G. Coley, Beaglebone Black System Reference Manual (Texas Instruments, 2013).

Deng, Z.

Frank L., P.

P. Frank L. and L. S. Pedrotti, Introduction to Optics, 3nd edition (Prentice Hall, 1993).

Georgiou, A.

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Goodman, J. W.

J. W. Goodman, Introduction to Fourier Optics (Roberts and Company Publishers, 2005).

Handelman, G. H.

J. F. Koretz and G. H. Handelman, “How the human eye focuses,” Sci. Am. 259(1), 92–99 (1988).
[Crossref] [PubMed]

J. F. Koretz and G. H. Handelman, “Modeling age-related accomodative loss in the human eye,” Math. Model. 7(5–8), 1003–1014 (1986).
[Crossref]

He, F.

Hideya, T.

T. Hideya and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” Proc. SPIE 6803, 68031N (2008).

Hirooka, S.

T. Hideya and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” Proc. SPIE 6803, 68031N (2008).

Inami, M.

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

Kang, Y.

Kautz, J.

Kawakami, N.

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

Kim, S. B.

Kollin, J.

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Koretz, J. F.

J. F. Koretz and G. H. Handelman, “How the human eye focuses,” Sci. Am. 259(1), 92–99 (1988).
[Crossref] [PubMed]

J. F. Koretz and G. H. Handelman, “Modeling age-related accomodative loss in the human eye,” Math. Model. 7(5–8), 1003–1014 (1986).
[Crossref]

Liu, W.

Liu, Y.

Luebke, D.

Lv, W.

MacLeod, D. I.

R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
[Crossref] [PubMed]

Maeda, T.

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

Miller, T. P.

R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
[Crossref] [PubMed]

Mimica, M. R.

C. H. Morimoto and M. R. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. 98(1), 4–24 (2005).
[Crossref]

Min, S. W.

Morimoto, C. H.

C. H. Morimoto and M. R. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. 98(1), 4–24 (2005).
[Crossref]

Park, J. H.

Pedrotti, L. S.

P. Frank L. and L. S. Pedrotti, Introduction to Optics, 3nd edition (Prentice Hall, 1993).

Romanova, G. E.

G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).

Siegman, A. E.

A. E. Siegman, “How to (maybe) measure laser beam quality,” in Diode Pumped Solid State Lasers: Applications and Issues (Optical Society of America, 1998) paper MQ1.

Smith, W. J.

W. J. Smith, Modern Optical Engineering (Tata McGraw-Hill Education, 1966).

Song, W.

Tachi, S.

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

Takashima, Y.

Travis, A. R.

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Vasilyev, V. N.

G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).

Wang, Y.

Wei, Z.

Westheimer, G.

G. Westheimer, “The maxwellian view,” Vision Res. 6, 669–682 (1966).
[Crossref] [PubMed]

Yanagida, Y.

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

Yang, J.

Yang, T.

Yao, C.

Zhang, D.

Appl. Opt. (2)

Behav. Res. Methods (1)

R. D. Beer, D. I. MacLeod, and T. P. Miller, “The Extended Maxwellian View (BIGMAX): A high-intensity, high-saturation color display for clinical diagnosis and vision research,” Behav. Res. Methods 37(3), 513–521 (2005).
[Crossref] [PubMed]

Comput. Vis. Image Underst. (1)

C. H. Morimoto and M. R. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. 98(1), 4–24 (2005).
[Crossref]

J. Soc. Inf. Disp. (1)

A. R. Travis, L. Chen, A. Georgiou, J. Chu, and J. Kollin, “Wedge guides and pupil steering for mixed reality,” J. Soc. Inf. Disp. 26(9), 526–533 (2018).
[Crossref]

Math. Model. (1)

J. F. Koretz and G. H. Handelman, “Modeling age-related accomodative loss in the human eye,” Math. Model. 7(5–8), 1003–1014 (1986).
[Crossref]

Opt. Express (3)

Opt. Lett. (1)

Proc. SPIE (2)

G. E. Romanova, A. V. Bakholdin, and V. N. Vasilyev, “Optical schemes of the head-mounted displays,” Proc. SPIE 10374, 103740I (2017).

T. Hideya and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” Proc. SPIE 6803, 68031N (2008).

Sci. Am. (1)

J. F. Koretz and G. H. Handelman, “How the human eye focuses,” Sci. Am. 259(1), 92–99 (1988).
[Crossref] [PubMed]

Vision Res. (1)

G. Westheimer, “The maxwellian view,” Vision Res. 6, 669–682 (1966).
[Crossref] [PubMed]

Other (8)

M. Inami, N. Kawakami, T. Maeda, Y. Yanagida, and S. Tachi, “A stereoscopic display with large field of view using Maxwellian optics,” in Proceedings of The 7th International Conference on Artificial Reality and Tele-Existence (Tachi Lab, 1997), pp. 71–76).

G. Wyszecki and W. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, (John Wiley & Sons, 1982).

P. Frank L. and L. S. Pedrotti, Introduction to Optics, 3nd edition (Prentice Hall, 1993).

G. Coley, Beaglebone Black System Reference Manual (Texas Instruments, 2013).

J. W. Goodman, Introduction to Fourier Optics (Roberts and Company Publishers, 2005).

A. E. Siegman, “How to (maybe) measure laser beam quality,” in Diode Pumped Solid State Lasers: Applications and Issues (Optical Society of America, 1998) paper MQ1.

W. J. Smith, Modern Optical Engineering (Tata McGraw-Hill Education, 1966).

J. P. Rolland, K. P. Thompson, A. Bauer, H. Urey, and M. Thomas, “See-through head-worn display (HWD) architectures.” Handbook of Visual Display Technology (Springer, 2011), pp. 2929–2961.

Supplementary Material (3)

NameDescription
» Visualization 1       The pupil follower is in operation. As the eye rotates the pupil position is calculated and an appropriate LED is turned on in real time.
» Visualization 2       The pupil follower in operation. The pupil position is calculated using image processing, which is marked with the red circle. The LED is driven synchronously with the pupil tracker software. Whole operation implemented is in real time.
» Visualization 3       The pinhole display is demonstrated with a navigation application.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1 Objects that are closer than the near-point of the eye (~20cm) cannot be focused due to the refractive limitations of the eye lens (top). With a pinhole aperture in front of the eye lens, objects at all depths appear focused on the retina (bottom). Limited depth of field and fixed viewing depth is one of the fundamental limitations of all stereoscopic head mounted displays.
Fig. 2
Fig. 2 Pupil follower display system operation. (a) The illumination beam is focused at the eye pupil by a focusing lens, which effectively creates pinhole imaging and small eyebox. If the eye rotates slightly no light enters the eye pupil, thus the image is lost. As a solution we propose using multiple light sources that are spaced closer than the minimum pupil size so that all pupil positions can be addressed. (b) We synchronize the array of light sources with a pupil tracker and turn on only the required source so that the display can be seen in an extended eyebox.
Fig. 3
Fig. 3 The LED array is synchronized with our camera-based pupil tracker software and it turns on only the required LED so that the display can be seen in an extended eyebox. We call this system the pupil follower display. A detailed operation of the pupil follower can be seen in Visualization 1 and Visualization 2.
Fig. 4
Fig. 4 (a) Zemax model of the optical system. LED array is imaged on the pupil plane with unit magnification. The LED spacing is set on the PCB such that their images are 2mm apart at the pupil plane. (b) The optical setup on the bench. (c) Image of a single LED unit. Each unit has individually addressable RGB LEDs. (d) The LED array. LED units are 1x1mm in size and are placed 2mm apart in each direction (e) Image of the LEDs at the pupil plane when all of them are turned on (in actual operation only one RGB triad is turned on at a time).
Fig. 5
Fig. 5 (a) Physical optics simulations show the optimal pixel size for the LCD. Different pixel sizes are simulated using the angular spectrum method and the FWHM spot size at the retina is calculated. (b) Cross-section of the smallest spot size at the retina for a 250μm pixel size, where the red dots mark the FWHM beam diameter.
Fig. 6
Fig. 6 Demonstration of the system outdoors. We can show bright images with a single low power LED as all the light from the LCD is captured by the eye. FOV is roughly circular, 37°.
Fig. 7
Fig. 7 Demonstration of the system for an indoor navigation scenario. The full version can be viewed in Visualization 3.
Fig. 8
Fig. 8 (a) Simulated MTF of the pinhole display. The maximum frequency is limited by the LCD pixel pitch. (b) Experimental verification of the MTF. Spatial frequencies marked on the MTF curve are displayed as bar patterns on the LCD and the images are taken by a camera. On the simulated MTF curve the cutoff frequency is calculated about 2 cyc/deg, which is confirmed by the experimental results.

Metrics