Open Access
Add to BibSonomy
Abstract
In this paper, we present a fully-customized AR display design that considers the user’s prescription, interpupillary distance, and taste of fashion. A free-form image combiner embedded inside the prescription lens provides augmented images onto the vision-corrected real world. The optics was optimized for each prescription level, which can reduce the mass production cost while satisfying the user’s taste. The foveated optimization method was applied which distributes the pixels in accordance with human visual acuity. Our design can cover myopia, hyperopia, astigmatism, and presbyopia, and allows the eye-contact interaction with privacy protection. A 169g dynamic prototype showed a 40° × 20° virtual image with a 23 cpd resolution at center field and 6 mm × 4 mm eye-box, with the vision-correction and varifocal (0.5-3m) capability.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Augmented reality (AR) displays present virtual images at real-world scenes while preserving the viewer’s natural vision. Numerous AR head mounted displays (HMDs) have been introduced in both commercial prototypes [1,2] and research literature [3–8] since Google glass, but no device has achieved wide adoption by consumers. In contrast to market expectations on AR displays represented by “holographic” images, there is no viable optical structure meeting both of the slim form factor and high visual standards including high-resolution, large field-of-view (FOV), large eye-box, and variable focus.
The diversity of human head shape and eye structure aggravates this challenge further. Every user has different interpupillary distance (IPD, 50 - 75 mm) [9] and nose shape, which raises the bar on eye-box and eye relief coverage beyond the requirement for a single user. Besides, since more than 40% of the population uses special aids for vision correction caused by myopia, hyperopia, astigmatism, and presbyopia, AR display should consider the viewer’s prescription [10–12]. AR display manufacturers produced either an eye-glasses compatible design or an additional ophthalmic-lens option, but both methods increase the weight and the size of the system significantly.
Several approaches have been introduced to include vision correction in an AR display structure [5,13,14]. The “Focals” by North Inc. demonstrated the most ergonomic eye-glasses like AR displays by measuring customer’s prescription and IPD. However, previous methods were mostly based on the retinal projection display using holographic optical elements (HOEs) attached to the rear surface of the ophthalmic lens, which inherently causes limited FOV at given eye relief and narrow eye-box [14]. Considerable dynamic eye-box expansion methods have been proposed [15,16], but they required a additional spatial light modulator or linear actuator and are not suitable for the form factor of an eyeglasses. Above all, there are no previous customized AR display design which considers the observer’s prescription, IPD, and facial appearance.
In this paper, we propose a fully-customized, prescription-embedded AR display. We propose an optical design that utilizes the prescription lens as a wave-guide for the AR display. A free-form image combiner is inserted into the prescription lens, so one lens piece can deliver virtual scenes and correct the vision of the real scene simultaneously. Based on a modified myopia eye model, we first design the shape of the prescription lens. Then the free-form image combiner, in-coupling prism, and beam shaping lens are optimized for the individual prescription lens. In addition, a customized ergonomic eye-glasses design is achieved by using a 3D facial scan. A Prescription AR prototype with a 5-mm thick lens provides 1 diopter (1D) vision correction, 23 cycles per degree (CPD) angular resolution at center, 6 mm eye-box, and varifocal (0.33D - 2D) capability. The prototype is lightweight (169$g$ for dynamic and 79$g$ for static prototype), has 70% transparency, protects user’s privacy, and enables eye-contact interaction with surroundings.
2. Design
2.1 System overview
As its core, our Prescription AR display optically corrects the user’s vision with a prescription lens and utilizes that lens as a wave-guide in an AR display system. As shown in Fig. 1(a), the top surface of the prescription lens of thickness $t_l$ is used as an entrance of the wave-guide. The light rays from a micro display of size $w_d\times h_d$ and resolution $N_{x} \times N_{y}$ located in front of user’s forehead with an angle $\theta _d$ are refracted by a bi-convex ($R_{l1}$, $R_{l2}$) beam shaping lens (refractive index: $n_1$, thickness: $t_{BSL}$) located at $a$ from the micro display with the tilted angle $\theta _l$ and entered to the wave-guide through an in-coupling prism (refractive index: $n_1$) located at $d_p$ from the beam shaping lens with the tilted angle $\theta _p$. The in-coupling prism is composed by a set of a plano-concave and a convex-plano cylindrical lens. Then, the rays are refracted by a cylindrical lens ($R_{cy}$, refractive index: $n_2$) located at $t_c$ from the prism surface with the tilted angle $\theta _c$ and travel in the wave-guide (refractive index: $n_3$, tilted angle: $\theta _w$) located at $t_w$ from the cylindrical surface as shown in Fig. 1(c). The light rays are total internal reflected (TIR) twice by the frontal surface ($S_f$) and the rear surface ($S_r$) of the prescription lens, reflected by a free-form half-mirror coated surface ($S_{free}$, tilted angle: $\theta _f$), and arrived at the pupil of the eye. Note that the in-coupling prism, cylindrical lens, upper part of the prescription lens, and lower part of the prescription lens are bonded by an optical adhesive, so the prototype consists of only two lens pieces: the main lens and the beam shaping lens. The enlarged virtual image of size $w_i\times h_i$ is located at distance $d_i$ from the eye in the vision-corrected real scene. The virtual image depth can be dynamically adjusted ($\Delta d_i$) by moving the micro display back and forth ($\Delta a$).
Fig. 1. Schematic diagram of Prescription AR: (a) The side view and the beam path of the AR image of the proposed system. The prescription lens works both for vision-correction and for wave-guide of the AR image. Light rays from a micro display (MD) refracted by a beam shaping lens (BSL) enter to the prescription lens through an in-coupling prism (ICP) and create a magnified virtual image located a distance $d_i$ from the lens. The focal image depth can be dynamically changed from 0.33D to 2D by moving MOLED axially, $\Delta a$, in experimental results. (b) The detailed diagram for geometric parameters in the Prescription AR. (c) The 3D diagram of optical components.
The optical design process is divided into two steps: the prescription lens design ($S_f$, $S_r$) and the AR display path design (rest of all) using a commercial optics simulation tool, Zemax OpticStudio. The overall optical path is infeasible to be investigated by an analytic form because of the free-form surface and the multiple off-axis components. Nevertheless, we propose a universal designing and optimization method which is valid for any kind of prescription including myopia, astigmatism, hyperopia, and presbyopia. Figure 2 shows a flow chart of the universal 2-step optimization process in Prescription AR, which is started from the user’s eyeglasses prescription including spherical correction(SPH), cylinder correction(CYL), axis of astigmatism(AXIS), and add power (ADD).
Fig. 2. Flow chart of the two-step optimization for Prescription AR: The frontal and rear surfaces ($S_f$, $S_r$) are optimized at given lens thickness $t_l$ in the first step based on user’s prescription. Other geometric parameters ($R_{l1}$, $R_{l2}$, $R_{cy}$, $a$, $d_p$, $t_{BSL}$, $t_c$, $t_w$, $\theta _d$, $\theta _l$, $\theta _p$, $\theta _c$, $\theta _w$, and $\theta _{free}$) are optimized in the second step based on target foveated resolution and eye relief range, $d_e$.
2.2 Prescription lens design from modified myopia eye model
The first step is the optimization of the prescription lens, especially the frontal ($S_f$) and rear ($S_r$) surface profile of the prescription lens [17]. Figure 3 shows how to design the prescription lens for myopic eye. Figures 3(a) and 3(b) show the principle of vision correction for myopic eye. The normal vision whose amplitude of accommodation is 4D has a far point at 0D and near point at 4D. The 1D myopic eye with the same amplitude of accommodation has a far point at 1D and near point at 5D. The observer cannot perceive full-resolution image of the objects located at 0.6D because the object is out of accommodation range, as shown in Fig. 3(a). The prescription lens shifts the object at infinity to the myopic eye’s far point, 1D location, so the objects are imaged at 1.6D plane, inside the accommodation range of 1D myopic eye. Similarly, Prescription AR display offer the clear enlarged virtual image inside the accommodation range of myopic eye. In additionally, the prescription lens compensates the astigmatism, by adding inverse cylinder power to the given axis.
Fig. 3. Principle of the Prescription AR: (a) A 1D myopic eye and its accommodation range of 1D to 5D. The 1D myopic eye cannot resolve the real objects located at 0.6D. (b)The prescription lens design for 1D myopic eye. The lens effectively shifts the object from infinity (0D) to the myopia’s far point (1D). The object located at 0.6D is imaged at 1.6D plane with the correct prescription lens. (c) Prescription AR design. The virtual image plane should be located inside the accommodation range of the myopic eye. Note that the depth of field (DOF, accommodation range) of normal vision is from 0D to 4D and the DOF of $X$D myopic eye is from $X$D to ($X+4$)D.
Instead of using the direct calculation of surface profiles from the SPH, CYL, AXIS, and ADD values, we optimized both surfaces using a human eye model. This optimization method can minimize the aberration at the given thickness $t_l$, refractive index $n_3$, and given eye relief $d_e$. Previously, Atchison built human myopic eye model based on the measured data from 121 subjects [18]. And it is known that the total astigmatism is the sum of the corneal and internal astigmatism [19]. However, there isn’t a general human eye model covering both myopia and astigmatism. In this work, we assumed corneal astigmatism only and modified the corneal surface property of the Atchison’s model. This simple assumption is valid in this case because the prescription lens is only affected by the sum of astigmatism, not the source. The cornea surface profile $C_{rv}$ and $C_{rh}$ are calculated from the cylindrical power, CYL, and AXIS value, and the modified eye model is achieved with SPH value as shown in Table 1 where the $r_x$ and $r_y$ are the radius value of bifocal system in horizontal and vertical respectively, $k_x$ and $k_y$ are the conic constant of bifocal system in horizontal and vertical respectively, $N_d$ is the reflective index of material, and $V_d$ is the Abbe number of material. Note that the radius of cornea surface, $r_x^*$, is calculated by adding the CYL power into another direction as $D_x$ = $D_y$ + CYL, where the $D_y$ = (Nd-1)/$r_y$.where the $r_x$ and $r_y$ are the radius value of bifocal system in horizontal and vertical respectively, $k_x$ and $k_y$ are the conic constant of bifocal system in horizontal and vertical respectively, $N_d$ is the reflective index of material, and $V_d$ is the Abbe number of material. Note that the radius of cornea surface, $r_x^*$, is calculated by adding the CYL power into another direction as $D_x$ = $D_y$ + CYL, where the $D_y$ = (Nd-1)/$r_y$. The complete equation of $r_x^*$ can be expressed as follows:
Based on this modified myopia eye model, we calculated $S_f$ and $S_r$. $S_f$ is set as a spherical surface of radius $r_f$ while $S_r$ is set as a bifocal surface of radii $r_{ro}$, $r_{re}$ and rotation angle $\theta _r$, to correct the myopia and the astigmatism. All the values were optimized iteratively with the merit function for the range of 12 to 20 mm eye relief, $d_e$. The Fig. 4 shows the spot diagram change axially around the retina plane of myopic astigmatism eye (SPH: -2, CYL: -2, and AXIS: 30) without and with the prescription lens. Compared to the naked eye focusing at infinite object in Fig. 4(a), and the myopia-only correction lens in Fig. 4(b), the designed prescription lens forms a smaller focal point at the retinal plane in Fig. 4(c).
Table 1. The modified myopia eye model based on Atchison’s model.
Fig. 4. Analysis of the spot diagram on the retina through focus shifting: (a) the blurred spot on the retina from the infinite object without a prescription lens. (b)The focused spot on the retina only optimized for SPH, and (c) optimized for SPH, CYL, and AXIS. Note that there is serious astigmatism even the spherical optical power is compensated in (b).
2.3 Prescription-embedded AR display design
Based on the prescription lens design, all other geometric parameters ($R_{l1}$, $R_{l2}$, $R_{cy}$, $a$, $d_p$, $t_{BSL}$, $t_c$, $t_w$, $\theta _d$, $\theta _l$, $\theta _p$, $\theta _c$, $\theta _w$, $\theta _{free}$, and $S_{free}$) were optimized in the second step. Although actual numbers are calculated by Zemax OpticStudio, the geometry of optics, the materials, the constraints should be carefully considered at the design stage for the optimal performance.
2.3.1 Geometry creation and materials
There have been several research efforts on free-form image combiner based AR displays [20–23] and on general free-form optimization methodology [24]. We designed a free-form image combiner for the best imaging quality and the compensation of highly compact off-axis optical design. Figure 1(b) shows the detailed diagram of the AR display path. In the wave-guide, the light rays are reflected at the positive power surface ($S_f$) first, and at the negative power surface ($S_r$) second. So it is reasonable to choose positive power image combiner ($S_{free}$) for the free-form surface for the flatter focal plane, symmetric power distribution, and less lens aberration. The free-form surface can be characterized by an extended polynomial equation including conic aspherical surfaces and extended polynomial terms as follows:
where $c$ is the curvature for the base sphere, $r$ is the normal radius expressed as $r = \sqrt {x^2+y^2}$, $k$ is the conic constant, $N$ is the number of polynomial terms, and $A_{i}$ is the coefficient of the $i^{th}$ extended polynomial terms. In our optimization, up to $4^{th}$ polynomials were considered.The bi-convex beam shaping lens increases the system’s numerical aperture (NA) for higher resolution and compactness of the shorter optical path. The in-coupling prism guides the light rays into the wave-guide with the TIR condition. The y-axis only cylindrical surface ($R_{cy}$) inside the in-coupling prism compensates some astigmatism and the tilted image plane, which are caused by the off-axis folded path. The tilted angle of the beam shaping lens is identical to the tilted angle of the micro-display for the symmetric magnification ($\theta _d$ = $\theta _l$), but the angles of other components were freely decided by the optimizer to maximize FOV and minimize the aberration. The materials for the beam shaping lens and the upper part of the in-coupling prism ($n_1$, $v_1$), the lower part of that ($n_2$,$v_2$), and the prescription lens ($n_3$,$v_3$), where $n$ and $v$ refer to index of refraction and Abbe number respectively, were carefully chosen to minimize the thicknesses and the chromatic aberration using the different dispersion characteristics. The distances ($a$, $d_p$, $t_{BSL}$, $t_c$, $t_w$) were calculated to some non-negative values based on the following constraints and the priorities.
2.3.2 Physical constraints
The optical configuration for the AR system is limited by giving the constraints in the merit function. These constraints are determined by the comprehensive consideration of lens implementation, distance from the forehead, total internal reflection condition, and boundary on display panel. In detail, the center thickness of each lens, $t_{BSL}$, $t_c$, $t_w$, and edge thickness must be over 1 $mm$ for manufacturability. The gaps between optical component, $a$ and $d_p$, should be longer than 0.2 $mm$ to avoid overlapping. The sum of thicknesses $a$, $d_p$, $t_{BSL}$, $t_c$, and $t_w$ are limited up to 8.5 $mm$ to minimize the total thickness of AR system so the micro-display can be located in front of the forehead. The light rays used in the optimization were started from the object and limited to the size of the micro display as the final surface.
2.4 Design trade space
2.4.1 FOV vs thickness and eye-box vs thickness
The FOV and the thickness of the prescription lens has a trade-off relation as shown in Fig. 5(a). Especially, a thicker prescription lens allows a larger vertical FOV because of the larger size of the free-form image combiner. We chose 5 mm thickness in our design for a comparable FOV with a slim and lightweight form factor. Similarly, the eye-box is also decided by the thickness; the thicker prescription lens the larger eye-box. The coupling between thickness and display characteristics is reasonable since the height of the micro display $h_d$ is limited by the thickness of the prescription lens. Note that the horizontal FOV and horizontal eye-box is determined by the width of the micro display $w_d$ and can be enlarged by choosing long rectangular displays.
Fig. 5. Design trade-off space for the Prescription AR. The micro display $w_d\times h_d$ = 10.08 $\times$ 7.56 mm and pixel pitch 6.3 $\mu m$, virtual image plane $d_i = (X+1)D$, and thickness $t_l$ = 5 mm correspond to the ones in the prototype (red circles), where $X$ represents the diopter of myopia. (a) Thickness vs. FOV and eye-box. Both FOV and eye-box are propotional to the $t_l$. (b) Eye relief vs. FOV. Smaller eye relief provides larger FOV. (c) Focus cue change. The virtual image plane, $d_i$, can be changed with the axial movement of the micro display, $\Delta a$.
2.4.2 FOV vs eye relief
Generally, the more extended eye relief causes the smaller FOV of the near-eye display. Due to the embedded-prescription, the Prescription AR can provide a smaller eye relief (12$\sim$20 mm) compared to the commercially available AR displays, and a comparable FOV with a 0.6-inch display. Since the minimum achievable eye relief might vary per user based on the facial structure, we optimized the optics for 5 different eye relief cases (12, 14, 16, 18 and 20 mm) using the multi-configuration function. The 5-mm thickness prototype can provide 20$\sim$23 degrees vertical FOV based on the user’s eye relief.
2.4.3 Focus cue change
In addition, Prescription AR provides focal cues. The virtual image plane can be changed by moving the micro display back and forth. The neutral position of the micro display ($a$) was optimized for a 1D focal plane in the vision-corrected scene, which is $(X+1)$D in the real scene for $X$ D myopia (Fig. 5(c)). By axially shifting the micro display around the original position ($\Delta a$), the virtual image plane can be controlled. Due to the compact optical structure, the overall magnification is large and the depth range from 0D to 2.5D in the vision-corrected scene can be covered with the 0.4 mm axial movement of the micro display. This varifocal characteristic is similarly valid for different myopia levels.
2.4.4 Resolution: foveated optimization
Because today’s micro display pixel numbers are not sufficient to provide 20/20 visual acuity (30 cpd) angular resolution over the full FOV, the foveated display was introduced which distributes the pixels in accordance with the human visual acuity [7,25]. Since our display is customized for each user which allows the precise alignment of the eye pupil and the display center, we adopted the foveated optimization. Figure 6(a) shows a comparison between a psychophysical visual acuity data and the angular resolution with the uniform and the foveated optimization as a function of horizontal eccentricity [26]. Although uniform optimization can achieve higher resolution in the peripheral region, the user cannot perceive most of the information due to the bandwidth limit. The foveated optimization provides 20/20 visual acuity resolution at the center. Since this system provides fixed foveation, the foveal region and the highest resolution region become misaligned as the eye rotates. Therefore, we sacrificed the resolution of the periphery more and secured higher resolution around the fovea to cover some gaze angle change. The user can perceive 22 cpd resolution image at the 6 degrees gaze angle. The optimized resolution over the FOV is illustrated in Fig. 6(b). The Angular resolution for the vertical eccentricity was optimized at the slightly lower gaze because people tend to look down naturally.
Fig. 6. Foveated optimization of angular resolution in Prescription AR: (a) Human visual acuity over eccentricity (black dotted line), Prescription AR design results (for 1D myopia) with the foveated optimization (blue line), and uniform optimization (red line). Note that each component and the free-form image combiner was optimized for the human visual acuity (fixed foveation) to minimize the wasted information. (b) Angular resolution over FOV in 1D myopia Prescription AR prototype.
3. Implementation
Prescription AR can cover most of the myopia, astigmatism, presbyopia, hyperopia, and any combinations of those. The Prescription AR display for a normal vision case (0D), multiple myopia cases (SPH = -1D, -2D, -3D, -4D, and -5D), a hyperopia case (SPH = 1D) and a myopic astigmatism case (SPH = -2, CYL = -2, AXIS = 30) has been designed, as shown in Tables 2 and 3. Among those designs, one case was manufactured, 1D myopia, for the verification. All the optical components including the mold for the free-form image combiner, the in-coupling prism, and the beam shaping lens were fabricated by ILLUCO. Then the glasses frames were customized for each user. The dynamic and static prototypes were demonstrated.
Table 2. The examples of the optimized values in Prescription AR including the normal vision, myopia, hyperopia, and myopic astigmatism case.
Table 3. The examples of the optimized coefficients for the free-form image combiner including the normal vision, myopia, hyperopia, and myopic astigmatism case.
3.1 Prescription-embedded free-form image combiner
The fabrication of the customized optical components includes the free-form image combiner are the key for the optical performance. The free-form image combiner was made through the molding processing by ILLUCO. The plastic material, Zeonex Cyclo Olefin polymer, was used for the ophthalmic lens. It is lightweight and supports the implementation of a free-form surface for molding processing. Figure 7(a) shows one mold of two lenses since the free-form image combiner separates the ophthalmic lens into upper and bottom lens. Figure 7(b) shows the complete prescription-embedded free-from combiner after the coating (anti-reflection, half-mirror, and neutral density) and bonding. Note that the overall transparency of the glasses is controlled by the coating process (70% in our prototypes). The final lenses were cut for the target eyeglasses frame. The assembled AR display engine is small and lightweight, as shown in Fig. 7(c).
Fig. 7. Prescription-embedded free-from image combiner made by the injection molding process (1D myopia): (a) The manufactured mold, (b) the complete prescription-embedded free-form image combiner, and (c) the final AR display engine after coating, bonding, and cutting. Note that the final lens shape can be changed depending on the user’s frame choice.
3.2 Customized eye-glasses prototype
Because every facial structural is unique, the ergonomic frame design is as important as the optics design. Since the optics was optimized for the eye relief of 12 mm to 20 mm, the prototype will work within that range. However, smaller eye relief can provide a larger FOV as Fig. 5(b) and more comfortable fit (closer center of mass). Further, the center of the pupil should be aligned with the optical axis for the best foveated experience as Fig. 6. Therefore, the glasses frame design should consider the user’s IPD too.
Figure 8 shows the customization process of the Prescription AR display. The facial structure was 3D scanned with the Kinect [27], and imported to the 3D rendering software, Fusion 360, where the glasses frame was designed and optimized for each user. The glasses frame designs were parameterized with the input of the IPD and the width of the head, followed by manual fitting for nose part. Figure 8(c) shows the results of the customization for the individual users.
Fig. 8. Customized Prescription AR display: (a) the scanned 3D model of a user, (b) fitting the glasses frame design with the scanned 3D model, and (c) the results of customization for users.
3.3 Prototypes
Two Prescription AR prototypes were implemented: a dynamic prototype and a static prototype.
LVT-based static prototype. Figure 9(a) shows the static prototype. Two sets of a 10.08$\times$7.56 mm, 3048 pixel per inch light valve technology (LVT) film with an Electro-Luminescent (EL) film backlight were used for the static display. A CR-2032 coin cell powered both EL films. A 3D printed housing aligned all of the optics, statics display modules, and the battery for wearable eyeglasses form factor. The weight of the static prototype was 79$g$.
Fig. 9. Implemented prototypes of Prescription AR: (a) static prototype based on LVT films (79 $g$) and (b) dynamic prototype based on micro OLED. 164 $g$ including the driving board.
OLED-based dynamic prototype. Figure 9(b) shows the dynamic prototype. All of the display results were from this setup. Two 10.08$\times$7.56 $mm^2$ Sony micro OLED (ECX339A) displays were used as binocular micro displays, where each display has 1600$\times$1200 resolution, 6.3 $\mu m$ pixel pitch, and maximum brightness 1000 $cd/m^2$. The free-form optics with the 70% transparency for 1D myopia were fabricated by Illuco. 3D printed frame housed and aligned all of the optical structures including the main lens attached with an in-coupling prism, beam shaping lens, micro display, and driving board. A 3D printed gear was also applied to change the IPD. The weight of the dynamic prototype including the driving board was 164 $g$. For display, a set of binocular images with 1200 $\times$ 1600 resolution were rendered in real-time. Over 100 binocular frames were generated every second in a local personal computer with a NVIDIA RTX 2080Ti. The FOV, DOF, IPD were selected carefully in accordance with the actual prototype.
Software implementation. We used a C++ open source innovation engine, called G3D, for rendering [28]. The binocular images with 1200 $\times$ 1600 resolution were rendered in real-time. Over 100 binocular frames were generated every second in a local personal computer with a NVIDIA RTX 2080Ti. The FOV, DOF, IPD were selected carefully in accordance with the actual prototype. Since the resolution was optimized with foveation, the foveated rendering can be considered [7,25]. We did not apply foveated rendering in this work, but it can be directly applied to the current prototype for reduced computation.
4. Display assessment
To properly illustrate the performance of the prototypes of Prescription AR, the experimental results are presented with photographs and videos by an iPhone X camera with the F-number of 2.4 and a 3024 x 4032 resolution. The detailed experimental setup with the camera is shown in Fig. 10.
Fig. 10. Photograph of the experimental setup: (a) setup for the experimental results. Note that both the real scene and the virtual image (resolution target) are clearly observed on the phone screen in the normal light condition. The close-up photos at the (b) side view and (c) top view.
Field of view and angular resolution. To measure the FOV, the entire display was illuminated with a white image and was captured with the background FOV panel located at a 15 cm distance as shown in Fig. 11(a). The measured FOV of the prototype was 40$^\circ$ $\times$ 20$^\circ$. The angular resolution is measured from the slant-edge MTF method [29]. Figure 11(b) shows the measured MTF graph and a close-up photo of a slanted edge. The normalized MTF of the Prescription AR prototype was greater than 0.5 at 23 cpd at the center, which was identical to the angular resolution calculated from the pixel pitch. Note that the angular resolution was limited by not the optics but the display. The optical structure can provide up to 32 cpd at the center with the smaller pixel size as shown in Fig. 6(a).
Fig. 11. Experimental results of the FOV and resolution measurement: (a) The AR image covers 20$^\circ$ by 40$^\circ$ of the checking board in vertical and horizontal direction respectively. (b) The analysis of slant-edge measurement at center field. Note that the angular resolution of 23 CPD is realized in our prototype.
Eye-box. Figure 12 shows the experimental results of the eye-box. The full FOV image was displayed and the camera was shifted at the pupil plane to measure the eye-box. The Prescription AR prototype provided 6 mm of horizontal and 4 mm of vertical eye-box while preserving the $40^\circ \times 20^\circ$ FOV.
Fig. 12. Eye-box measurement results: The $40^\circ \times 20^\circ$ image captured at (a) the left-most, (b) the center, (c) the right-most position. Note that the FOV panel was captured together to show the FOV. The measured eye-box was 6$\times$4 $mm^2$.
AR display results with the prescription. Figure 13 shows the captured images without and with the Prescription AR prototype to show the vision-correction results. The real objects were located at different depths: a car, a horse doll, and an eye chart were located at 0.5, 1, and 3 $m$ from the camera respectively in the real world. A fixed focus camera mimicked a 1D myopia eye: the camera was focused at 0.5 $m$ (2D), which is the middle of the 1D myopic eye’s DOF. Without the Prescription AR prototype, the camera was focused at the car, was able to resolve some of the details of the horse doll, and was not able to see the details of the eye chart. With the Prescription AR prototype, the camera focus was shifted to the horse doll (1D), and was able to achieve good resolution from both of the car and the eye chart. In the real myopic eye where the user can change accommodation, the observer can focus on all three objects with the Prescription AR. The virtual image located at the 1 $m$ distance was also clearly observed in Fig. 13(b). Figure 13(c) shows the video with the rendered augmented contents generated by our rendering software (Visualization 1).
Fig. 13. Experimental results of Prescription AR: (a) Captured image without Prescription AR display. The myopic eye is mimicked as a 1D myopic eye with a fixed focus camera and (b) Captured image with the Prescription AR display. Note that the far object (eye chart) looks sharper with the Prescription AR as well as the virtual image is observed. (c) Experimental video result with the rendered augmented contents (Visualization 1)
Focus cue change. Vergence-accommodation conflict (VAC) [30] is one of the visual fatigue while the user perceives the virtual image from binocular vision since the lack of monocular focus cue. Varifocal function offers the monocular focus cue through the adjustable virtual image plane of the display system. As the same concept of that, the AR image plane is adjustable by moving the micro display back and forth ($\Delta a$), as shown in Fig. 14. A car, a horse doll, and a can were located at 0.5 ($d_1$), 1 ($d_2$), and 3 $m$($d_3$) respectively, as shown in Fig. 14(a). The micro display was located accordingly to provide the focus cue. As shown in Fig. 14, the in-focus virtual images were observed at 0.5, 1, and 3 $m$ by moving the micro display. The required traveling distance of MD was 0.27 $mm$ to cover the depth range from 0.33D to 2D through the experiment. In addition, the micro display is lightweight (< 3$g$) which can be further implemented with the small electric motor under the sub-millimeter traveling.
Fig. 14. Focus cue change results: (a)schematic diagram of the experimental setup. Three real objects were placed at 0.5 $m$, 1 $m$, and 3 $m$ respectively which related to three AR images from the certain position of MD within 0.3 $mm$ of shifting, $\Delta a$. The captured image with camera focus at (b) 0.5 $m$, (c) 1 $m$, and (d) 3 $m$. The clear AR images were perceived at 0.5 $m$, 1 $m$, and 3 $m$ by corrected position of MD.
5. Discussion
Presbyopia and hyperopia. The presbyopia is usually corrected by a bi-focal lens or a progressive lens, which adds an optical power to the lower gaze, for close viewing point. Since the Prescription AR display only utilizes the upper half of the ophthalmic lens, the bi-focal or progressive solution can be directly applied to the current design at the lower half of the ophthalmic lens. For hyperopia, there is no proper hyperopia eye model, but it can be directly calculated the ophthalmic lens profiles ($S_f$, $S_r$), and performed the same optimization. The simulated resultant FOV was smaller in 1D hyperopia Prescription AR display because the ophthalmic lens should have a positive optical power and it is difficult to maintain TIR condition. Other than that, the proposed Prescription AR display worked well also in the hyperopia case.
Fixed foveated display with off-axis gaze. As shown in Fig. 6(a), the optics was optimized based on foveation. The resolution at the center (32 cpd) was higher than the far periphery (2 cpd at 20 degrees). Compared to the uniformly optimized optics (12 cpd over the FOV), the foveated optimization matches the human visual acuity well at the center gaze. Although, Prescription AR is a fixed foveated display and it cannot provide optimal resolution for off-axis gaze [31], we believe that the fixed foveated display provides better visual experience in overall due to the following reasons.
First, the foveated optimization can provide higher resolution to the fovea within $\pm$8 degrees gaze angle. As shown in Fig. 8(a), the prototype provides higher angular resolution for larger eccentricity than the foveal region. Therefore, the user will perceive higher resolution when they rotate their eyes a bit compared to the uniform optimization. Since it is known that humans tend to rotate their head for the far off-axis gaze objects (>$\pm$ 15 degrees in horizontally, > 8 degrees upward, and > 12 degrees downward) [32], the user will be able to see sharper images most of the time.
Second, most of the pixels are wasted regardless of user’s gaze in the uniform optimization. Due to the human visual acuity difference over eccentricity, the user cannot perceive high resolution images in periphery. Therefore in the uniform optimization, significant amount of pixels are not perceived all the time. In the fixed foveation, the effective perceived pixels numbers are increased as it accords well with the human visual acuity system.
Finally, one can wisely choose the target angular resolution curve by controlling the merit function in the optimization stage. For better stability, it is possible to sacrifice some resolution at the center to achieve more resolution in the off axis. Finding an optimal angular resolution distribution for a fixed foveated near-eye display is still an open problem.
Brightness and transparency: privacy and eye-contact interaction. Another essential feature of the AR display is privacy and eye-contact interaction. The privacy is the ability to hide the user’s AR contents from outside and the eye-contact interaction is the capability to observe the user’s eye from outside while the user is watching AR contents. These issues are closely related to brightness, transparency, and optical configuration. The transparency of the Prescription AR prototype was 70%, and the measured luminance at the pupil plane was 40 $cd/m^2$ while the display brightness was 200 $cd/m^2$. By taking advantages of high light efficiency of the system and the diverging light rays, Prescription AR allows eye-contact interaction while preserving user’s privacy. Figure 15 shows the observed user’s face while watching the resolution target contents. The contents were only directly observed from the micro OLED, not through the main lens. Therefore, the user’s privacy can be protected by covering the micro OLED region with the non-transparent frames. Furthermore, the user’s eye was clearly observed from outside at the different point of views, which enables the eye-contact interaction. Note that both the brightness at the pupil plane and the contrast ratio can be improved by increasing the reflectance, which can be easily controlled by changing the coating. For example, higher reflectance of the lenses can be helpful for outdoor use cases.
Fig. 15. The captured image of the prototype with the human model. The eye was clearly captured from different at the top, bottom, center, right, and left viewpoint.
Productivity. This Prescription AR approach follows the grammar of the conventional eyeglasses industry. The free-form lens mold for each prescription is prepared as bigger size and the final lens shape is customized for the specific frame chosen by the customer. Due to the finite range of the human prescription, the mass production can reduce the cost significantly while satisfying user’s taste.
6. Conclusion
As the internet of things becomes common, the objects around human are getting smarter and interactive. However, unlike other everyday personal items, only the eyeglasses have remained a passive, non-electronic device. Although recent AR display products provide nice augmented experiences with reasonable form factors, they cannot upgrade/substitute eyeglasses without vision-correction. Besides, since people’s taste of glasses is very individual, offering a small number pre-designed several options, as true with smartphones, is not a suitable approach for smart glasses business.
In this paper, we showed a fully-customized AR display design approach that considers the user’s prescription, IPD, facial structure, and taste of fashion. We established a prescription-embedded AR display optical design method as well as the customization method for individual users. This optical design can cover myopia, hyperopia, astigmatism, and presbyopia, and allows the eye-contact interaction with privacy protection. The glasses and the frames were ergonomically customized using a Kinect-based 3D scanning method. A 169$g$ dynamic prototype showed a 40$^\circ$ $\times$ 20$^\circ$ virtual image with a 23 cpd resolution at center field and 6 mm $\times$ 4 mm eye-box, with the vision-correction and varifocal capability. We believe our work is a start of the paradigm shift in AR display research: from universal design to personalized design.
Acknowledgments
We thank Peter Shirley, Kaan Akşit, Zander Majercik, Morgan McGuire, and David Luebke for helpful discussions and advice.
Disclosures
The authors declare no conflicts of interest.
References
1. M. Billinghurst, A. Clark, and G. Lee, “A survey of augmented reality,” FNT in Human-Computer Interaction 8(2-3), 73–272 (2015). [CrossRef]
2. B. C. Kress, “Digital optical elements and technologies (EDO19): applications to ar/vr/mr,” Proc. SPIE11062, 1106201 (2019).
3. K. Akşit, W. Lopes, J. Kim, P. Shirley, and D. Luebke, “Near-eye varifocal augmented reality display using see-through screens,” ACM Trans. Graph. 36, 189 (2017). [CrossRef]
4. A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs, “Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources,” ACM Trans. Graph. 33(4), 1–11 (2014). [CrossRef]
5. A. Maimone, A. Georgiou, and J. S. Kollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. Graph. 36(4), 1–16 (2017). [CrossRef]
6. S. Lee, J. Cho, B. Lee, Y. Jo, C. Jang, D. Kim, and B. Lee, “Foveated retinal optimization for see-through near-eye multi-layer displays,” IEEE Access 6, 2170–2180 (2018). [CrossRef]
7. J. Kim, Y. Jeong, M. Stengel, K. Akşit, R. Albert, B. Boudaoud, T. Greer, J. Kim, W. Lopes, Z. Majercik, P. Shirley, J. Spjut, M. McGuire, and D. Luebke, “Foveated ar: Dynamically-foveated augmented reality display,” ACM Trans. Graph. 38(4), 1–15 (2019). [CrossRef]
8. D. Dunn, C. Tippets, K. Torell, P. Kellnhofer, K. Akşit, P. Didyk, K. Myszkowski, D. Luebke, and H. Fuchs, “Wide field of view varifocal near-eye display using see-through deformable membrane mirrors,” IEEE Trans. Vis. Comput. Graph. 23(4), 1322–1331 (2017). [CrossRef]
9. N. A. Dodgson, “Variation and extrema of human interpupillary distance,” Proc. SPIE 5291, 36–46 (2004). [CrossRef]
10. B. A. Holden, T. R. Fricke, D. A. Wilson, M. Jong, K. S. Naidoo, P. Sankaridurg, T. Y. Wong, T. J. Naduvilath, and S. Resnikoff, “Global prevalence of myopia and high myopia and temporal trends from 2000 through 2050,” Ophthalmology 123(5), 1036–1042 (2016). [CrossRef]
11. C.-W. Pan, M. Dirani, C.-Y. Cheng, T.-Y. Wong, and S.-M. Saw, “The age-specific prevalence of myopia in asia: a meta-analysis,” Optom. Vis. Sci. 92(3), 258–266 (2015). [CrossRef]
12. H. Hashemi, A. Fotouhi, A. Yekta, R. Pakzad, H. Ostadimoghaddam, and M. Khabazkhoob, “Global and regional estimates of prevalence of refractive errors: systematic review and meta-analysis,” J. Curr. Ophthalmol. 30(1), 3–22 (2018). [CrossRef]
13. Y.-J. Wang, P.-J. Chen, X. Liang, and Y.-H. Lin, “Augmented reality with image registration, vision correction and sunlight readability via liquid crystal devices,” Sci. Rep. 7(1), 433 (2017). [CrossRef]
14. L. Zhou, C. P. Chen, Y. Wu, Z. Zhang, K. Wang, B. Yu, and Y. Li, “See-through near-eye displays enabling vision correction,” Opt. Express 25(3), 2130–2142 (2017). [CrossRef]
15. C. Jang, K. Bang, G. Li, and B. Lee, “Holographic near-eye display with expanded eye-box,” ACM Trans. Graph. 37, 1 (2018). [CrossRef]
16. C. Jang, K. Bang, S. Moon, J. Kim, S. Lee, and B. Lee, “Retinal 3d: augmented reality near-eye display via pupil-tracked light field projection on retina,” ACM Trans. Graph. 36(6), 1–13 (2017). [CrossRef]
17. W.-S. Sun, C.-L. Tien, C.-C. Sun, M.-W. Chang, and H. Chang, “Ophthalmic lens design with the optimization of the aspherical coeffiecients,” Opt. Eng. 39(4), 978–989 (2000). [CrossRef]
18. D. A. Atchison, “Optical models for human myopic eyes,” Vis. Res. 46(14), 2236–2250 (2006). [CrossRef]
19. P. C. Hoffmann and W. W. Hütz, “Analysis of biometry and prevalence data for corneal astigmatism in 23 239 eyes,” J. Cataract Refractive Surg. 36(9), 1479–1485 (2010). [CrossRef]
20. H. Hua and B. Javidi, “A 3d integral imaging optical see-through head-mounted display,” Opt. Express 22(11), 13484–13491 (2014). [CrossRef]
21. D. Cheng, Y. Wang, H. Hua, and M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009). [CrossRef]
22. D. Cheng, Y. Wang, H. Hua, and J. Sasian, “Design of a wide-angle, lightweight head-mounted display using free-form optics tiling,” Opt. Lett. 36(11), 2098–2100 (2011). [CrossRef]
23. A. Wilson and H. Hua, “Design and demonstration of a vari-focal optical see-through head-mounted display using freeform alvarez lenses,” Opt. Express 27(11), 15627–15637 (2019). [CrossRef]
24. A. Bauer, E. M. Schiesser, and J. P. Rolland, “Starting geometry creation and design method for freeform optics,” Nat. Commun. 9(1), 1756 (2018). [CrossRef]
25. A. Patney, M. Salvi, J. Kim, A. Kaplanyan, C. Wyman, N. Benty, D. Luebke, and A. Lefohn, “Towards foveated rendering for gaze-tracked virtual reality,” ACM Trans. Graph. 35(6), 1–12 (2016). [CrossRef]
26. S. M. Anstis, “A chart demonstrating variations in acuity with retinal position,” Vis. Res. 14(7), 589–592 (1974). [CrossRef]
27. J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3d full human bodies using kinects,” IEEE Trans. Vis. Comput. Graph. 18(4), 643–650 (2012). [CrossRef]
28. M. McGuire, M. Mara, and Z. Majercik, “The G3D innovation engine,” (2017). Online Document, https://casual-effects.com/g3d.
29. P. D. Burns, “Slanted-edge mtf for digital camera and scanner analysis,” in Proc. IS&T 2000 PICS Conference, 135–138 (2000).
30. D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks, “Vergence–accommodation conflicts hinder visual performance and cause visual fatigue,” J. Vis. 8(3), 33 (2008). [CrossRef]
31. J. Spjut, B. Boudaoud, J. Kim, T. Greer, R. Albert, M. Stengel, K. Aksit, and D. Luebke, “Toward standardized classification of foveated displays,” arXiv preprint arXiv:1905.06229 (2019).
32. T. Hatada, H. Sakata, and H. Kusaka, “Psychophysical analysis of the sensation of reality” induced by a visual wide-field display,” J. SMPTE 89(8), 560–569 (1980). [CrossRef]
