We propose a design of a retinal-scanning-based near-eye display for augmented reality. Our solution is highlighted by a laser scanning projector, a diffractive optical element, and a moist eye with gradient refractive indices. The working principles related to each component are comprehensively studied. Its key performance is summarized as follows. The field of view is 122°, angular resolution is 8.09′, diffraction efficiency is 57.6%, transmittance is 80.6%, uniformity is 0.91, luminance is 323 cd/m2, modulation transfer functions are above 0.99999 at 3.71 cycle/degree, contrast ratio is 4878, and distortion is less than 24%.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
For a very long time, near-eye display (NED), also known as head-mounted display, remained as a marginal technology in the display community . Until recently, in the wake of augmented and virtual realities , it theatrically morphs into a superstar sought after by a crowd of researchers, engineers, investors, bloggers etc. Technically, NED is a type of wearable projection display . Pursuant to the criterion that whether the retina is the image plane, NED can be divided into two categories, i.e. indirect projection and direct or retinal projection. For the former, the image is first projected onto a virtual image plane, and then received by the retina. To do so, the optical path needs to be folded via either a combiner [4–11] or waveguide [12–22]. As far as the field of view (FOV) is concerned, both combiner and waveguide based NEDs have limited FOVs, usually below 50° . Another restriction is the paradox that FOV is inversely proportional to the exit pupil . There must be a trade-off between the FOV and exit pupil. Direct or retinal projection based NED, on the other hand, refers to the case when the image is directly projected to the retina, for which the image plane coincides with the retina. Virtual retinal display (VRD) [23–25], developed by Human Interface Technology Lab of University of Washington, is a pioneering retinal projection based NED, known for its concept of retinal scanning. iOptik [26,27], a proprietary technology of Innovega, is characterized by a contact lens embedded with a zone plate. Pinlight display , co-developed by University of North Carolina at Chapel Hill and Nvidia, mimics a pinhole camera by an array of point light sources in front of eye. Intel―through acquisitions of Composyt Light Labs and Lemoptix―built a smart glasses prototype dubbed Vaunt with holographic optical elements (HOEs) . Though the above retinal projection based NEDs are able to achieve extremely large FOVs, each has its own pros and cons. The optical setup of VRD is bulky and sophisticated, making it unwearable. iOptik has been struggling for many years in persuading the consumers to wear the contact lens. Pinlight display is vulnerable to the change of eye state, including the diopter of eye, pupil size, and rotation of eyeball. Lasting for only 4 odd years, Vaunt was aborted in part due to manufacturability of HOEs. Inspired from the said issues, we would like to present a lensless retinal scanning display (RSD), which is compact in design and immune to the change in the pupil size and diopter of eye. In what follows, its structure, working principles, and overall performance are to be elaborated.
2.1 Proposed structure
The proposed RSD can be decomposed into three major components, i.e. a laser scanning projector, a diffractive optical element (DOE), and an eye, as shown in Fig. 1, where dp is the distance between projector and glass, ddoe the distance between projector and DOE, dlens the distance between DOE and lens center, W the horizontal dimension of DOE, x the distance starting from the left edge of DOE, θi the angle of incident light, and θm the angle of diffracted light. Inside the temple is housed the laser scanning projector for saving the room. On the inner surface of a flat glass substrate is fabricated the DOE, which is able to converge the light coming out of the projector towards the center of lens of eye. Analogous to the Maxwellian view , such configuration ensures that the image formed on the retina will remain intact no matter how the eye accommodates its diopter to the distance of object. Moreover, as long as the beam size of the laser is way smaller than the pupil―minimally 2 mm across ―the brightness of image could be maintained regardless of the variation of pupil size. For the sake of symmetry, the DOE and eye are center-aligned.
2.2 Eye model
Unlike our previous eye models [32–34], in which the eye is dry―in the absence of tear―and lens of eye has a constant refractive index, the current model factors into not only the tear but also a gradient lens  studied by Goncharov et. al. Figure 2 is a cross-sectional view of our schematic moist eye, consisting of tear, cornea (anterior and posterior), aqueous chamber filled with aqueous humor, iris with an opening known as pupil, lens (anterior and posterior), vitreous chamber filled with vitreous humor, and retina. Approximately, two-thirds of the eye’s optical power is derived from the cornea―including the tear―and one-third from the lens . While the thickness of tear is negligibly thin―ranging from 6 to 20 μm―the irregularities of tear can cause significant visual aberrations and distortions .
To model the gradient lens, an assistive virtual plane is inserted to separate the lens into the anterior and posterior halves, as shown in Fig. 3. The center of lens is―coinciding with the nodal point ―designated as the origin of y-z coordinate. Invoking the Goncharov’s gradient lens model , refractive indices na for the anterior lens and np for the posterior lens could be computed with35], which are itemized in Table 1, refractive indices of lens are calculated along the horizontal and vertical directions, respectively, as shown in Fig. 4. It can be seen that the refractive index maximizes at the lens center, from which it starts to decrease towards the outermost surface.
2.3 Laser scanning projector
Figure 5 is a schematic drawing of the laser scanning projector, inside which are mounted a laser diode, a circular polarizer, a mirror, and a scanning mirror controlled by a biaxial micro-electromechanical system (MEMS) . Laser is transformed to circular polarization after passing through the circular polarizer. Since the laser scanning projector is lensless, the image projected onto the retina will be always in focus―in other words, its depth of focus is infinite―even when the eye is defocused. In this regard, the laser scanning projector is essentially a pinhole camera . This is a big advantage over the lens-based projector, whose image plane is at a certain distance and depth of focus is finite . The depth cue of the projected image, on that other hand, shall be coupled with the distance of real object, depending on the accommodation of eye . A major downside of laser scanning projector, among others, is about its low resolution―e.g. the best resolution of Microvision’s laser scanning projector is merely 848 × 480 ―restricted by the scanning rate of MEMS. Due to the limited choices of laser scanning projector available in the market, a set of parameters customized to our design are given in Table 2, where the resolution is 640 × 640, scanning angle (horizontal) is 17°, beam diameter at the waist D0 is 0.4 mm, luminous flux Φ is 2 lm, and contrast ratio (CR) is 5000. Being treated as a Gaussian TEM00 beam, beam diameter D of the laser enlarges as it propagates at a distance L from the waist, which is described as 
For M2 = 1.1, λ = 532 nm, L = 50 mm, and D0 = 0.4 mm, D = 0.411 mm and θdiv = 0.93 mrad or 0.05°. As a rule of thumb, for a laser with a beam divergence less than 1 mrad, its beam can be approximated as the perfectly collimated one. Upon the reflection of MEMS, both the shape and size of the said laser beam will be altered. To match the shape and size of the subsequent DOE, MEMS is square in shape and 60 µm wide, as will be discussed later. To avoid the loss of light, a beam shaper could be inserted on top of the laser diode. Plus, in case the projector and DOE are misaligned during the mounting, it is suggested to make both of them adjustable for calibration. Incidentally, speaking of the speckle effect, a random laser made from disordered materials  is among the most desirable solutions.
2.4 Diffractive optical element
DOE is an array of diffraction gratings, the number of which shall match with the resolution of laser scanning projector. The size of each diffraction grating shall match with that of the beam emerging from MEMS, i.e. 60 μm. For enhancing the diffraction efficiency (DE) and eliminating the undesired diffraction orders, each grating is designed to be a slanted grating , as depicted in Fig. 6, where p is the grating period, hg the grating depth, wg the grating width, and β the slant angle relative to the normal. In order for the image to be immune to the change in diopter of eye, all the diffracted beams are supposed to pass through the lens center. For a grating located at a distance x from the left edge of DOE, its period p, diffraction order m, wavelength λ, incident angle θi and diffraction angle θm should satisfy both the reflection grating equation , i.e.Fig. 1) given by
3. Results and discussion
3.1 Simulation setting
The performance of our RSD is quantitatively analyzed with Code V (Synopsys) and COMSOL Multiphysics (COMSOL). Capable of ray tracing, Code V lends itself to analyzing the imaging properties, such as modulation transfer function (MTF), distortion, and imaging simulation. Based on the finite element method , COMSOL Multiphysics is a powerful tool in dealing with the diffraction grating. The design wavelength is 532 nm.
Figure 7 outlines the optical surfaces used in Code V. The object is placed at 3 m ahead of the eye. Surfaces 1 to 8 (S1 to S8) constitute the moist eye, of which, S1 is tear, S2 anterior cornea, S3 posterior cornea, S4 iris with pupil, S5 anterior lens, S6 virtual plane, S7 posterior lens, and S8 retina. To imitate the laser scanning projector as a pinhole camera, the semi-aperture of virtual plane―where the lens center is located―is set as 30 µm, i.e. radius of the laser beam.
The original parameters of eye are adopted from , wherein the eye is focused to the infinity. In our model, with the tear being added and the object distance being assigned as 3 m, those parameters need to be tweaked through an optimization, which is carried out under a constraint that the length of eye be 24 mm . As a result, Table 3 summarizes the optimized parameters, where AL and PL, in turn, denote the gradient refractive indices of anterior and posterior lenses. Besides, detailed parameters for aspherical surfaces and gradient lens are provided in Table 4 and Table 5, respectively.
3.2 Ray tracing diagram
Figure 8 shows the ray tracing diagram for the fields of 0°, 10°, 20°, 30°, 40°, 50°, and 61°, from which it can be seen that all rays are converged at the lens center.
3.3 Field of view
As illustrated in Fig. 9, FOV, whose vertex is situated at the center of entrance pupil, is the angle subtended by DOE. If measured diagonally, it could be determined by
3.4 Angular resolution
Angular resolution (AR) in arcminute (′) is calculated by dividing FOV in degree (°) by the number of pixels N along the diagonal, which can be written as 
3.5 Diffraction efficiency and transmittance
In modeling the grating, wave optics module of COMSOL Multiphysics using the interface of electromagnetic wave, frequency domain is employed. The boundary condition is Floquet periodicity. The incident light is linearly polarized as transverse electric (TE) mode. The diffraction order m is + 1. The glass substrate of DOE is chosen as N-BK7 (Schott), whose refractive index is 1.5195 at 532 nm. Without loss of generality, 9 gratings are picked for simulation, as shown in Fig. 10. Say dp = 10 mm, ddoe = 20 mm, and dlens = 16.73 mm, according to Eqs. (5) and (6), the incident/diffraction angles θi/θm can be calculated. Based on the optimization, the optimal grating parameters, DEs, and transmittance T for normal incidence (θi = 0°) are obtained as in Table 6. The average DE and T are 57.6% and 80.6%, respectively.
As DE of each grating of DOE more or less differs, a figure of merit Γ to evaluate the uniformity is introduced as Table 6, the uniformity Γ is 0.91.
Luminance is a measure of the luminous intensity per unit area of light that is diffracted from the DOE within the entire FOV. By this definition, we could express the luminance in terms of the luminous flux Φ, average DE DEavg of DOE, solid angle Ω subtended by DOE, and area A of DOE, i.e.
3.8 Spot diagram
Figure 11 plots the spot diagram for the fields of 0°, 10°, 20°, 30°, 40°, 50°, and 61°, from which it can be seen that the spot of each field is smaller than the Airy disk.
As shown in Fig. 12, MTFs are calculated as a function of spatial frequency in cycle/degree for the fields of 0° and 61° when the distances between DOE and eye are offset by 0 mm, 3 mm and 6 mm from the target eye relief of 12 mm, respectively. At 3.71 cycle/degree, which corresponds to the angular resolution of 8.09′, MTFs are above 0.99999 for all fields and eye relief offsets. This agrees with the foregoing statement on the laser scanning projector in analogy to a pinhole camera. This also indicates that even if the laser beams are not perfectly converged at the center of lens, the image quality will not be much affected.
3.10 Contrast ratio
CR―the ratio of maximum intensity to minimum intensity ―can be deduced as
Distortion, which measures the difference between the paraxial and actual image heights, is defined as Fig. 13, the distortion is 24%, give or take.
3.12 Simulated imaging
Figure 14 shows the original image alongside the see-through retinal and projected retinal images. Compared to the original one, the projected retinal image is, while distorted, sharp and bright as a whole. In particular, the uncompromised brightness is extremely critical for the outdoors usage.
A lensless RSD and its working principles have been proposed. Its structure is highlighted by a laser scanning projector, a DOE, and a moist eye. To precisely model the eye, tear and gradient lens are taken into account. Based on the simulation, FOV is 122°, angular resolution is 8.09′, average DE of DOE is 57.6%, average transmittance of DOE is 80.6%, uniformity is 0.91, luminance is 323 cd/m2, MTFs are above 0.99999 at 3.71 cycle/degree for all fields and offsets, CR is 4878, and distortion is 24%. As opposed to other retinal projection based NEDs [32–34], our RSD exhibits several unique features. First, no lens―except the lens of eye―is involved. Second, the projected retinal image is focus free. If constructed as a binocular RSD, it would be inherently free of the vergence-accommodation conflict . Third, DOE is used as a combiner, making the device compact in design and suitable for see-through augmented reality. Fourth, the retinal image is immune to the change in the diopter of eye and pupil size. Instead, it will be subject to the rotation of eye, thereby fixating the eye to look straight ahead. That being said, a user can still look around by rotating his/her head if the head tracking is enabled.
National Natural Science Foundation of China (61831015); Science and Technology Commission of Shanghai Municipality (19ZR1427200, 1801H163000, 1701H169200); Shanghai Rockers Inc. (15H100000157).
Special thanks to Dr. Site Zhang (LightTrans, Germany) for his insightful comments on the manuscript.
1. J. E. Melzer and K. Moffitt, Head-Mounted Displays: Designing for the User (McGraw-Hill, 1997).
2. W. Barfield, Fundamentals of Wearable Computers and Augmented Reality 2nd Edition (Chemical Rubber Company, 2015).
3. M. S. Brennesholtz and E. H. Stupp, Projection Displays 2nd Edition (Wiley, 2008).
4. J. P. Rolland, “Wide-angle, off-axis, see-through head-mounted display,” Opt. Eng. 39(7), 1760–1767 (2000). [CrossRef]
5. S. Liu, H. Hua, and D. Cheng, “A novel prototype for an optical see-through head-mounted display with addressable focus cues,” IEEE Trans. Vis. Comput. Graph. 16(3), 381–393 (2010). [CrossRef] [PubMed]
6. H.-S. Chen, Y.-J. Wang, P.-J. Chen, and Y.-H. Lin, “Electrically adjustable location of a projected image in augmented reality via a liquid-crystal lens,” Opt. Express 23(22), 28154–28162 (2015). [CrossRef] [PubMed]
9. Q. Gao, J. Liu, X. Duan, T. Zhao, X. Li, and P. Liu, “Compact see-through 3D head-mounted display based on wavefront modulation with holographic grating filter,” Opt. Express 25(7), 8412–8424 (2017). [CrossRef] [PubMed]
10. A. Maimone, A. Georgiou, and J. S. Kollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. Graph. 36(4), 85 (2017). [CrossRef]
12. Y. Amitai, “Extremely compact high-performance HMDs based on substrate-guided optical element,” in SID Symposium Digest of Technical Papers (2004), pp. 310–313. [CrossRef]
13. T. Levola, “Diffractive optics for virtual reality displays,” J. Soc. Inf. Disp. 14(5), 467–475 (2006). [CrossRef]
14. H. Mukawa, K. Akutsu, I. Matsumura, S. Nakano, T. Yoshida, M. Kuwahara, and K. Aiki, “A full-color eyewear display using planar waveguides with reflection volume holograms,” J. Soc. Inf. Disp. 17(3), 185–193 (2009). [CrossRef]
15. D. Cheng, Y. Wang, H. Hua, and M. M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009). [CrossRef] [PubMed]
16. Q. Wang, D. Cheng, Y. Wang, H. Hua, and G. Jin, “Design, tolerance, and fabrication of an optical see-through head-mounted display with free-form surface elements,” Appl. Opt. 52(7), C88–C99 (2013). [CrossRef] [PubMed]
19. J. Yang, P. Twardowski, P. Gérard, and J. Fontaine, “Design of a large field-of-view see-through near to eye display with two geometrical waveguides,” Opt. Lett. 41(23), 5426–5429 (2016). [CrossRef] [PubMed]
21. Y. Wu, C. P. Chen, L. Zhou, Y. Li, B. Yu, and H. Jin, “Near-eye display for vision correction with large FOV,” in SID Display Week (2017), pp. 767–770.
22. Z. Liu, Y. Pang, C. Pan, and Z. Huang, “Design of a uniform-illumination binocular waveguide display with diffraction gratings and freeform optics,” Opt. Express 25(24), 30720–30731 (2017). [CrossRef] [PubMed]
23. Wikipedia, “Virtual retinal display,” https://en.wikipedia.org/wiki/ Virtual_retinal_display.
24. T. A. Furness III and J. S. Kollin, “Virtual retinal display,” US Patent 5,467,104 (1992).
25. S. C. McQuaide, E. J. Seibel, J. P. Kelly, B. T. Schowengerdt, and T. A. Furness III, “A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror,” Displays 24(2), 65–72 (2003). [CrossRef]
26. R. B. Sprague, “Method and apparatus to process display and non-display information,” US Patent 8,520,309 B2 (2008).
27. R. Sprague, A. Zhang, L. Hendricks, T. O’Brien, J. Ford, E. Tremblay, and T. Rutherford, “Novel HMD concepts from the DARPA SCENICC program,” Proc. SPIE 8383, 838302 (2012). [CrossRef]
28. A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs, “Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources,” ACM Trans. Graph. 33(4), 89 (2014). [CrossRef]
29. E. Tremblay, M. Guillaumee, and C. Moser, “Method and apparatus for head worn display with multiple exit pupils,” US Patent 9,846,307 B2 (2017).
31. H. K. Walker, W. D. Hall, and J. W. Hurst, Clinical Methods: The History, Physical and Laboratory Examinations 3rd Edition (Butterworth-Heinemann, 1990), Chap. 58.
32. C. P. Chen, L. Zhou, J. Ge, Y. Wu, L. Mi, Y. Wu, B. Yu, and Y. Li, “Design of retinal projection displays enabling vision correction,” Opt. Express 25(23), 28223–28235 (2017). [CrossRef]
33. L. Mi, W. Zhang, C. P. Chen, Y. Zhou, Y. Li, B. Yu, and N. Maitlo, “A retinal-projection-based near-eye display for virtual reality,” Proc. SPIE 10676, 106761C (2018).
34. Y. Wu, C. P. Chen, L. Mi, W. Zhang, J. Zhao, Y. Lu, W. Guo, B. Yu, Y. Li, and N. Maitlo, “Design of retinal-projection-based near-eye display with contact lens,” Opt. Express 26(9), 11553–11567 (2018). [CrossRef] [PubMed]
36. M. Bass, C. DeCusatis, J. Enoch, V. Lakshminarayanan, G. Li, C. MacDonald, V. Mahajan, and E. V. Stryland, Handbook of Optics 3rd Edition Volume III: Vision and Vision Optics (McGraw-Hill Education, 2009).
37. M. Farid, “The tear film: the neglected refractive interface,” https://www.eyeworld.org/supplements/EW-September-supplement-2014.pdf.
38. Wikipedia, “Cardinal point (optics),” https://en.wikipedia.org/wiki/Cardinal_point_(optics).
39. S. E. Lyshevski, Nano- and Micro-Electromechanical Systems: Fundamentals of Nano- and Microengineering 2nd Edition (Chemical Rubber Company, 2005).
40. F. L. Pedrotti, L. M. Pedrotti, and L. S. Pedrotti, Introduction to Optics 3rd Edition (Pearson, 2006).
42. O. Svelto, Principles of Lasers 5th Edition (Springer, 2010).
45. E. G. Loewen and E. Popov, Diffraction Gratings and Applications (Marcel Dekker, 1997).
46. COMSOL, “An introduction to the finite element method,” https://www.comsol.com/multiphysics/finite-element-method.
47. W. Zhang, C. P. Chen, L. Mi, Y. Lu, M. Zhu, X. Ren, R. Tang, and N. Maitlo, “A retinal-projection-based near-eye display with contact lens for mixed reality,” Proc. SPIE 11040, 1104005 (2019). [CrossRef]
48. L. Mi, C. P. Chen, Y. Lu, W. Zhang, M. Zhu, R. Tang, X. Ren, and N. Maitlo, “A lensless retinal scanning display for augmented reality,” in SID Display Week (2019), pp. 1583–1586.
49. R. E. Fischer, B. Tadic-Galeb, and P. R. Yoder, Optical System Design 2nd Edition (McGraw-Hill Education, 2008).
50. H. Hua, “Enabling focus cues in head-mounted displays,” Proc. IEEE 105(5), 805–824 (2017). [CrossRef]