Abstract

State-of-the-art optical see-through head-mounted displays for augmented reality (AR) applications lack mutual occlusion capability, which refers to the ability to render correct light blocking relationship when merging digital and physical objects, such that the virtual views appear to be ghost-like and lack realistic appearance. In this paper, using off-the-shelf optical components, we present the design and prototype of an AR display which is capable of rendering per-pixel mutual occlusion. Our prototype utilizes a miniature organic light emitting display coupled with a liquid crystal on silicon type spatial light modulator to achieve an occlusion capable AR display offering a 30° diagonal field of view and an angular resolution of 1.24 arcminutes, with an optical performance of > 0.4 contrast over the full field at the Nquist frequency of 24.2 cycles/degree. We experimentally demonstrate a monocular prototype achieving >100:1 dynamic range in well-lighted environments.

© 2017 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Design of a bright polarized head-mounted projection display

Hong Hua and Chunyu Gao
Appl. Opt. 46(14) 2600-2610 (2007)

Engineering of head-mounted projective displays

Hong Hua, Axelle Girardot, Chunyu Gao, and Jannick P. Rolland
Appl. Opt. 39(22) 3814-3824 (2000)

References

  • View by:
  • |
  • |
  • |

  1. J. P. Rolland and H. Fuchs, “Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization,” Presence (Camb. Mass.) 9(3), 287–309 (2000).
  2. O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006).
  3. J. P. Rolland and H. Hua, “Head-mounted display systems,” in Encyclopedia of Optical Engineering, R. B. Johnson and R. G. Driggers, eds. (Marcel Dekker 2005), pp.1–13.
  4. H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).
  5. E. Tatham, “Getting the best of both real and virtual worlds,” Commun. ACM 42(9), 96–98 (1999).
  6. A. Malmone and H. Fuchs, “Computational augmented reality eyeglasses,” Proc. of 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.29–38, (2013).
  7. G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar, “Tensor Displays: Compressive light field synthesis using multilayer displays with directional backlighting,” Proc. ACM SIGGRAPH (ACM Transaction on Graphics), 31(4), (2012).
  8. K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.
  9. K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).
  10. O. Cakmakci, Y. Ha, and J. P. Rolland, ” “A compact optical see-through head-worn display with occlusion support,” Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.16–25, (2004).
  11. C. Gao, Y. Lin, and H. Hua, “Occlusion capable optical see-through head-mounted display using freeform optics,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282, (2012)
  12. C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).
  13. D. Cheng, Y. Wang, H. Hua, and M. M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009).
    [PubMed]
  14. D. Cheng, Y. Wang, H. Hua, and J. Sasian, “Design of a wide-angle, lightweight head-mounted display using free-form optics tiling,” Opt. Lett. 36(11), 2098–2100 (2011).
    [PubMed]
  15. H. Hua, X. Hu, and C. Gao, “A high-resolution optical see-through head-mounted display with eyetracking capability,” Opt. Express 21(25), 30993–30998 (2013).
    [PubMed]
  16. X. Hu and H. Hua, “High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics,” Opt. Express 22(11), 13896–13903 (2014).
    [PubMed]
  17. R. Zhang and H. Hua, “Design of a polarized head-mounted projection display using ferroelectric liquid-crystal-on-silicon microdisplays,” Appl. Opt. 47(15), 2888–2896 (2008).
    [PubMed]

2014 (1)

2013 (2)

H. Hua, X. Hu, and C. Gao, “A high-resolution optical see-through head-mounted display with eyetracking capability,” Opt. Express 21(25), 30993–30998 (2013).
[PubMed]

C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).

2011 (1)

2009 (1)

2008 (1)

2006 (1)

O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006).

2001 (1)

K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).

2000 (1)

J. P. Rolland and H. Fuchs, “Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization,” Presence (Camb. Mass.) 9(3), 287–309 (2000).

1999 (1)

E. Tatham, “Getting the best of both real and virtual worlds,” Commun. ACM 42(9), 96–98 (1999).

Ahuja, N.

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

Billinghurst, M.

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.

Brown, L. D.

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

Cakmakci, O.

O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006).

O. Cakmakci, Y. Ha, and J. P. Rolland, ” “A compact optical see-through head-worn display with occlusion support,” Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.16–25, (2004).

Campbell, B.

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.

Cheng, D.

Fuchs, H.

J. P. Rolland and H. Fuchs, “Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization,” Presence (Camb. Mass.) 9(3), 287–309 (2000).

A. Malmone and H. Fuchs, “Computational augmented reality eyeglasses,” Proc. of 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.29–38, (2013).

Gao, C.

H. Hua, X. Hu, and C. Gao, “A high-resolution optical see-through head-mounted display with eyetracking capability,” Opt. Express 21(25), 30993–30998 (2013).
[PubMed]

C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).

C. Gao, Y. Lin, and H. Hua, “Occlusion capable optical see-through head-mounted display using freeform optics,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282, (2012)

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

Ha, Y.

O. Cakmakci, Y. Ha, and J. P. Rolland, ” “A compact optical see-through head-worn display with occlusion support,” Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.16–25, (2004).

Hu, X.

Hua, H.

X. Hu and H. Hua, “High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics,” Opt. Express 22(11), 13896–13903 (2014).
[PubMed]

H. Hua, X. Hu, and C. Gao, “A high-resolution optical see-through head-mounted display with eyetracking capability,” Opt. Express 21(25), 30993–30998 (2013).
[PubMed]

C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).

D. Cheng, Y. Wang, H. Hua, and J. Sasian, “Design of a wide-angle, lightweight head-mounted display using free-form optics tiling,” Opt. Lett. 36(11), 2098–2100 (2011).
[PubMed]

D. Cheng, Y. Wang, H. Hua, and M. M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009).
[PubMed]

R. Zhang and H. Hua, “Design of a polarized head-mounted projection display using ferroelectric liquid-crystal-on-silicon microdisplays,” Appl. Opt. 47(15), 2888–2896 (2008).
[PubMed]

C. Gao, Y. Lin, and H. Hua, “Occlusion capable optical see-through head-mounted display using freeform optics,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282, (2012)

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

Kiyokawa, K.

K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.

Kurata, Y.

K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).

Lin, Y.

C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).

C. Gao, Y. Lin, and H. Hua, “Occlusion capable optical see-through head-mounted display using freeform optics,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282, (2012)

Malmone, A.

A. Malmone and H. Fuchs, “Computational augmented reality eyeglasses,” Proc. of 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.29–38, (2013).

Ohno, H.

K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).

Rolland, J.

O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006).

Rolland, J. P.

J. P. Rolland and H. Fuchs, “Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization,” Presence (Camb. Mass.) 9(3), 287–309 (2000).

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

O. Cakmakci, Y. Ha, and J. P. Rolland, ” “A compact optical see-through head-worn display with occlusion support,” Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.16–25, (2004).

Sasian, J.

Talha, M. M.

Tatham, E.

E. Tatham, “Getting the best of both real and virtual worlds,” Commun. ACM 42(9), 96–98 (1999).

Wang, Y.

Woods, E.

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.

Zhang, R.

Appl. Opt. (2)

Commun. ACM (1)

E. Tatham, “Getting the best of both real and virtual worlds,” Commun. ACM 42(9), 96–98 (1999).

Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” (1)

K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-through Display for Mutual Occlusion with a Real-time Stereo Vision System,” Elsevier Computer & Graphics Special Issue on “Mixed Realities - Beyond Conventions,” 25(5), 2765–2779 (2001).

J. Disp. Technol. (1)

O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006).

Opt. Express (2)

Opt. Lett. (1)

Presence (Camb. Mass.) (1)

J. P. Rolland and H. Fuchs, “Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization,” Presence (Camb. Mass.) 9(3), 287–309 (2000).

Proc. SPIE (1)

C. Gao, Y. Lin, and H. Hua, “Optical see-through head-mounted display with occlusion capability,” Proc. SPIE 8735, 87350F (2013).

Other (7)

J. P. Rolland and H. Hua, “Head-mounted display systems,” in Encyclopedia of Optical Engineering, R. B. Johnson and R. G. Driggers, eds. (Marcel Dekker 2005), pp.1–13.

H. Hua, C. Gao, L. D. Brown, N. Ahuja, and J. P. Rolland, “A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD),” Proceedings IEEE Virtual Reality, Orlando, FL, pp. 81–89. (2002).

O. Cakmakci, Y. Ha, and J. P. Rolland, ” “A compact optical see-through head-worn display with occlusion support,” Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.16–25, (2004).

C. Gao, Y. Lin, and H. Hua, “Occlusion capable optical see-through head-mounted display using freeform optics,” IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282, (2012)

A. Malmone and H. Fuchs, “Computational augmented reality eyeglasses,” Proc. of 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.29–38, (2013).

G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar, “Tensor Displays: Compressive light field synthesis using multilayer displays with directional backlighting,” Proc. ACM SIGGRAPH (ACM Transaction on Graphics), 31(4), (2012).

K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion capable optical see-through head mount display for supporting co-located collaboration,” Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–9, 2003.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1
Fig. 1

Superimposing a virtual airplane in a well-lit real world environment: AR view captured through a typical OST-HMD without occlusion capability.

Fig. 2
Fig. 2

Schematic diagram of the proposed OCOST-HMD design based on two-layer folded architecture.

Fig. 3
Fig. 3

Optical system layout of OCOST-HMD using stock lenses.

Fig. 4
Fig. 4

Modulation transfer function of 0, 5, 10 and 15 degree transverse (Tan) and radial (Rad) fields evaluated with a 3mm pupil diameter and a cutoff spatial frequency of 36 cycles/degree for the (a) OCOST-HMD eyepiece design and 60 cycles/degree for the (b) OCOST-HMD see-through (i.e. eyepiece + objective) design.

Fig. 5
Fig. 5

(a) Solidworks design of the fully assembled OCOST-HMD (b) Monocular prototype of the OCOST-HMD.

Fig. 6
Fig. 6

Experimental demonstration of mutual occlusion capability in our OCOST-HMD prototype with photographs captured with a digital camera placed at the exit pupil of the system: (a) view of a natural background scene through the occlusion model for light pass-through with the SLM turned on; (b) view of the virtual scene through the eyepiece with the see-through path being blocked by the SLM; (c) augmented view of the natural and virtual scenes without occlusion capability enabled; (d) View of the natural scene with an occlusion mask rendered on SLM; (e) augmented view with occlusion capability enabled where the virtual teapot is inserted in front of the background scene; (f) augmented view with occlusion capability enabled where the virtual teapot is inserted between two real objects for mutual occlusion demonstration.

Fig. 7
Fig. 7

Measured MTF performance of the OCOST-HMD prototype for the on-axis field of the virtual display, see-through view as well as the camera used for measurement.

Fig. 8
Fig. 8

Sample images of a grating target of 12 cycles/degree displayed by the virtual display superimposed onto a bright background of 350cd/m2 (a) with occlusion enabled to block the see-through light and (b) without occlusion.

Fig. 9
Fig. 9

Image contrast degradation of the virtual target of different spatial frequencies as a function of background scene brightness for (a) occlusion-disabled; and (b) occlusion-enabled displays.