Open Access
Add to BibSonomy
Abstract
A foveated display is a technology that can solve the problem of insufficient angular resolution (relative to the human eye) for near-eye display. In a high-resolution foveated display, a beam steering element is required to track the human gaze. An electrowetting prism array is a transmissive non-mechanical beam steering device, that allows a light and compact optical system to be configured and a large aperture possible. However, the view is obstructed by the sidewall of the prism array. When the size of the cell prism is 7mm, the prism array has an 87% fill-factor. To push the fill-factor to 100%, the cell prisms were magnified using a lens array. Image processing was performed such that the image produced by the lens array was identical to the original. Beam steering by refraction is accompanied by chromatic dispersion, which causes chromatic aberration, making colors appear blurry. The refractive index condition to reduce chromatic dispersion was obtained using the doublet structure of the electrowetting prism. The chromatic dispersion was reduced by 70% on average.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Using a binocular display close to the eye, a near-eye display (NED) [1–3] creates a wide viewing angle and three-dimensional (3D) display. It is possible to implement virtual reality or augmented reality, which cannot be achieved with conventional displays; recently, this technology has been attracting attention for use in a variety of applications (e.g., education, games, leisure, medical care). Despite the advantages of near-eye diplays, however, there are several factors that prevent commercialization. The most important factor is that most display devices are of head-mounted (HMD) [4–6] type, so they are limited in size and weight. If a near-eye display device is too heavy, even if it shows good performance, it is inconvenient for users.
Low angular resolution is another limiting factor. When the total number of pixels is fixed, angular resolution decreases as the field of view (FOV) increases. If two displays having the same number of pixels are used, a near-eye display has a wide FOV, and so the angular resolution is lower than with a TV or mobile phone. When using a display with full-high definition (HD) resolution and FOV of 100°, the angular resolution is 20 pixels per degree (ppd). This is a low value compared to human sight (average angular resolution of 60 ppd). To bring the angular resolution of the NED closer to that of human vision, it is necessary to increase the resolution of the display or to decrease the FOV. If the FOV is reduced, the reason for using the near-eye display disappears, so a display with high resolution should be used. However, it is difficult to implement high-resolution display for NED due to issues of size or price (or both).
The use of a foveated display [7–10] can solve the angular resolution problem. Foveated display technology involves using the characteristics of the human field of vision, which is divided into two parts: central vision (called foveal vision, with FOV of 8–10 degrees from center) and peripheral vision (FOV of 180–200 degrees). Because most human optic nerves are concentrated in the center of the retina, foveal vision provides high resolution and color sensitivity, whereas peripheral vision has relatively low resolution and color values. Even if high angular resolution is provided only in the foveal vision, the human subject has a sense of high resolution overall.
A foveated display uses a high-resolution display with low FOV for foveal vision and a low-resolution display with large FOV for peripheral vision. This allows viewers to have the sense of wide FOV and high resolution. The position of the foveal vision changes according to the direction of the viewer’s gaze. Because a foveated display is high-resolution only for foveal vision, it is necessary to adapt the position of the foveal display according to the gaze direction.
Moving the position of the foveal display can be done by directly moving the display device or by using a beam steering element to move the display image. The method of directly moving the display device is intuitive and does not require an additional optical system, so such a system is easy to design. However, direct physical display movement requires a device such as a motorized stage. Therefore, this method is not suitable for head mounted display because the operating speed is too slow and the volume and weight are too large. The beam steering method has the advantages of lower volume and weight of the NED device.
The beam steering elements are divided into those using mechanical [11–15] and non-mechanical methods [16–24]. The mechanical beam steering device has accurate and reliable performance; however, it is too large and heavy. Moreover, its power consumption is high. Therefore, it also is unsuitable for HMD. Non-mechanical beam steering solves these problems. Non-mechanical methods have compact structure, fast operating speed and low power consumption, all possible because there is no mechanical movement. In addition, most of the non-mechanical beam steering methods are transmissive type through refraction using a prism structure. Transmissive methods have the advantage that the optical axis is maintained during steering. Additional elements are required to maintain the optical axis when using a reflective type device. Therefore, not only the size of the device but also the size of the optical system can be reduced. However, various aberrations (chromatic, astigmatism, coma, distortion) may occur depending on the conditions of refraction through the prism structure [25]. Therefore, solving these aberrations is an important.
In this work, using an electrowetting prism array, we moved the position of the displayed image via transmissive non-mechanical beam steering. While other non-mechanical methods such as OPA [26] and LC [27] have low light efficiency due to polarization, reflection, and absorption, electrowetting has good light efficiency because there is no polarization element and light absorption is small in the visible region. In addition, the prism array allows large-aperture beam steering, making this technique suitable for use in near-eye displays.
2. Adaptive foveated display
2.1 Optical system with electrowetting prism array
The structure of each cell prism is shown in Fig. 1(a). Four sidewall electrodes are positioned vertically, and a different voltage is applied to each sidewall to form a refracting interface with prism shape. The square structure is suitable for arrays because it has a high fill factor. The electrowetting prism array was fabricated assembled using the sidewall method, as was done in the previous work [28]. The sidewall piece was manufactured as shown in Fig. 1(b). Electrodes and dielectric layers were deposited on both sides of the sidewall. After that, the sidewall pieces were assembled to make the electrowetting prism array chamber. Finally, the electrowetting prism array was fabricated by dosing and sealing the liquid. The beam steering angle range of the fabricated electrowetting prism array was ±13°, and the RMS wavefront error of the liquid-liquid interface was 0.181λ.
Fig. 1. Electrowetting prism array: (a) Principle of beam steering, (b) Fabrication process of electrowetting prism array chamber, and (c) Chamber sample.
However, if the electrowetting prism array was directly applied to the foveated display as a beam steering element, the view was obstructed by the sidewalls of the prism array chamber. Even if the thickness of the sidewall was reduced, there was still an obstacle on the display that caused inconvenience to viewers. Therefore, by using a lens array having the same size as the electrowetting prism array, it was possible to enlarge the prism and make the sidewall invisible. Figure 2 shows the optical system of the proposed adaptive foveated display using the electrowetting prism array and lens array.
Fig. 2. (a) Optical system of adaptive foveated display with electrowetting prism array and lens array and (b) Expected image of foveated display.
Using the proposed adaptive foveated display, beam steering was performed to track the viewer’s gaze. Then, the electrowetting prism array was operated according to the gaze information. If the frames per second (fps) of a universal video is 30 fps, the time between frames is 33ms and the speed of commercial gaze tracking technology is 8ms, so the operation speed of the electrowetting prism array should be within 25ms. When the size of the cell prism was 7mm, the operating speed was 25ms, so the size of the utilized cell prism was 7mm.
2.2 Fill-factor enhancement in prism array
Two kinds of lens arrays were used, a convex lens array and a concave lens array, with focal lengths of 51 and −39 mm, respectively, in a telecentric structure so that the magnification did not change with distance. Therefore, the magnification of the lens array was 1.3:1. The size of the cell prism was 7 mm, the thickness of the sidewall was 0.5 mm, and the fill-factor was 87%. The magnification to simply make the fill-factor 100% is 1.07:1, but the focal length of the lens was set in consideration of the viewing angle and the height of the prism. The distance of the two displays utilized in the foveated display were determined by considering the position as changed by the lens array and other optical parameters. The positions of the two displays were set to ${d_1}$= 73 mm and ${d_2}$= 56 mm considering that the position of the foveal display is changed by the lens array.
The lens array not only magnifies the prism, it also magnifies the display behind the prism. Therefore, the display image is also enlarged for each cell of the lens. Even if the display is magnified by the lens array, image processing is required for this image to look like the original image. First, the image processing needs to find the area that the lens magnifies in the display. In the simulation, the magnification area of an ideal paraxial lens is uniformly distributed, but in a real lens, the magnification area is distributed farther away from the center of the lens array, as shown in Fig. 3(a). Since this area changes depending on the degree of beam steering of the prism, the magnification area within the steering range must be calculated in advance. After finding the magnification area, the image is reduced to the same degree as the magnification of the lens array around the magnification area. Figure 3(b) shows the part of the image obscured by the sidewalls, the image magnified by the lens array, and the image processing method for lens array. Image processing is performed by reducing the image in advance to offset the magnification of the lens array.
Fig. 3. (a) Simulation to find area magnified by lens array and (b) Image processing: image obscured by sidewalls, image magnified by lens array, and image processing for lens array.
3. Achromatic electrowetting prism
3.1 Chromatic dispersion of prism
In general, optical elements that perform beam steering through refraction have different deviation angles according to the wavelengths refracted, as shown in Fig. 4(a). Because the refractive index of short wavelength beam is higher than that of long wavelength beam, chromatic dispersion occurs. In applications that use a single wavelength, such as lidar, chromatic dispersion is not a concern. However, in the case of a display, it is important to reduce chromatic dispersion and use an electrowetting prism because a broadband wavelength in the region of visible light is used.
Fig. 4. (a) Deviation angle and chromatic dispersion of prism, (b) Achromatic doublet prism and (c) Structure of the electrowetting doublet prism
The chromatic dispersion of a prism can be reduced by making two prisms into a doublet structure [29] (see Fig. 4(b)). Because the direction of the apex angles of the two prisms differ, they refract in opposite directions like convex and concave lenses. If a different material is used for each prism, the deviation angle of the two different wavelengths can be made equal by adjusting the refractive index relationship according to the wavelengths of the two materials. Chromatic dispersion can be minimized by making the deviation angle of a short wavelength beam equal to the deviation angle of a long wavelength beam. In the visible light range, an achromatic prism can be created by equalizing the blue light (486 nm) and red light (656 nm) deviation angles.
3.2 Deviation angle and achromatic condition
When the apex angle of the prism is A and the incident angles of both sides are ${\theta _1}$ and ${\theta _2}$, the deviation angle D is as follows.
If the apex angle and the incident angle are small, the deviation angle can be simplified as follows, and $\Delta {D_{B,R}}$. which is the deviation difference of blue light and red light, is as follows.
and ${n_R}$ are the refractive indices of blue and red light. Figure 4(c) shows the structure of the proposed electrowetting prism. A conductive liquid prism and a non-conductive liquid prism form a doublet structure. The total deviation difference of the doublet prism is equal to the sum of the conductive liquid prism and the non-conductive liquid prism of the deviation difference. When the refractive indices of the blue and red light of the conductive liquid are ${n_{c,B}}$ and ${n_{c,R}}$ and the refractive indices of the F and C lines of the non-conductive liquid are ${n_{n,B}}$ and ${n_{n,R}}$, the deviation angle difference of the doublet prism $\Delta {D_{doublet}}$ is as follows.Here, $\Delta {D_c}$ and $\Delta {D_n}$ are the deviation differences between the conductive liquid prism and the non-conductive liquid prism, and ${A_c}$ and ${A_n}$ are the apex angles of the conductive liquid prism and non-conductive liquid prism. If $\Delta {D_{doublet}}$ is zero, chromatic dispersion is minimal. Because the apex angle of the conductive and the non-conductive liquid prisms are always the same, the condition for $\Delta {D_{doublet}}=0$ is as follows.
In the electrowetting operation, the conductive and non-conductive liquids must have the same density, but a large difference in refractive index. In the previous work, for density matching, water was used as a conductive liquid and a mixture of two oils was used as a non-conductive liquid. To satisfy conditions of both density matching and Eq. (5), the ratios of the two conductive liquids and the two non-conductive liquids were adjusted. For the conductive liquid, a distilled water and ethylene glycol mixture was used, and for the non-conductive liquid, liquids with various refractive indexes from Cargille Lab (Series A) were used. Figure 5(a) shows the optical and physical properties of the conductive and non-conductive liquids. Among the various combinations of non-conductive liquids, a pair was selected to have a large difference of refractive index between the conductive and the non-conductive mixtures. When the difference in refractive index between the conductive and the non-conductive liquids is large, the steering angle is large even for the same apex angle. The refractive index differences between the conductive and the non-conductive liquids, according to the liquid combinations, are shown in Fig. 5(b). Because operation speed is slow when viscosity is high, Series A 1.48 and Series A 1.60 were used. These have large differences in refractive index under condition of moderate viscosity. The ratio of the conductive liquids (water to ethylene glycol) was 56:44, and the ratio of the non-conductive liquids (Series A 1.48 to Series A 1.60) was 39:61. The final refractive index of the conductive liquid ${n_c}$ was 1.38, ${n_{c,B}} - {n_{c,R}}$ is 0.018, the final refractive index of the non-conductive liquid is $n_{n}$ 1.55, and ${n_{n,B}} - {n_{n,R}}$ is also 0.018.
Fig. 5. (a) Optical and physical properties of conductive and non-conductive liquids and (b) Refractive index differences of conductive (${n_c}$) and non-conductive liquids (${n_n})$ according to combinations of two non-conductive liquids.
3.3 Optical simulation and experimental results
The achromatic effect was confirmed through image simulation using the conditions of the conductive and non-conductive liquids, obtained above. The optical engineering software Zemax was used for the simulation. For comparison, a prism meeting only the density condition was also tested.
The apex angle of the electrowetting prism was 30° and the distance to the object was 100 mm. Figure 6(a) is the original image, and Fig. 6(b) is a combination of water and non-conductive liquid prepared without considering chromatic dispersion. Compared to Fig. 6(b), it can be seen that the achromatic prism in Fig. 6(c) exhibits significantly reduced chromatic dispersion. Based on the simulation results, an electrowetting prism was made under the same conditions and tested. Figure 7(a) shows the chromatic dispersion results for the chromatic prism; Fig. 7(b) shows the results for the achromatic prism. Chromatic dispersion was calculated by measuring the size ratio of the beam steered image to that of the original image, and the chromatic dispersion reduction was calculated by comparing the chromatic dispersions. In simulations, the chromatic dispersion was reduced by 83.7%. Although in the actual experiment the chromatic dispersion was not reduced as much as in the simulation, it was reduced by 70% on average compared to the non-achromatic prism. According to Eq. (2), the deviation angles of both of the achromatic prism and the chromatic prism are proportional to the apex angle. Thus, the degree of chromatic dispersion reduction in the achromatic prism not related to the apex angle.
Fig. 6. Optical simulation of prism: (a) Original image, (b) Without achromatic condition prism, and (c) Achromatic prism.
Fig. 7. Experimental result of prisms: (a) Without achromatic prism at apex angle 30°, (b) Achromatic prism at apex angle 30°, and (c) Chromatic dispersion reduction according to apex angle.
4. Final experimental results of foveated display
In a foveated display, the angular resolution increase ratio is determined by the resolution and FOV of the foveal and peripheral displays. In this experiment, the peripheral display used 1K resolution and final viewing angle was 60°, and so the angular resolution was 17 ppd; the foveal display also uses 1K resolution and final viewing angle is 15°, so the angular resolution was 67 ppd, four times higher than that of the peripheral display. Figure 8 shows results of foveated display experiment using USAF chart image for resolution comparison. Figure 8(a) is the peripheral display that becomes the background display, Fig. 8(b) shows the foveated display with chromatic electrowetting prism array, and Fig. 8(c) shows a foveated display with achromatic electrowetting prism array. When an achromatic electrowetting prism array is used, the angular resolution increase rate is the same, but the image quality is better due to the reduced chromatic dispersion.
Fig. 8. Experimental results of USAF chart image: (a) Peripheral display, (b) Foveated display with chromatic electrowetting prism array, and (c) Foveated display with achromatic electrowetting prism array.
Fig. 9. Experimental results of natural image with high-frequency color detail: (a) Peripheral display, (b) Foveated display with chromatic electrowetting prism array, and (c) Foveated display with achromatic electrowetting prism array.
In addition, to check whether the foveated display is well-implemented for arbitrary content and arbitrary color transition, a natural image with high-frequency color detail was also used (Fig. 9).
5. Conclusions
We demonstrated an adaptive foveated display using an electrowetting prism array. In the foveated display, a beam steering element is required to move the position of the high-resolution foveal display according to the viewer’s gaze. Because the electrowetting prism is a transmissive, non-mechanical beam-steering device, it was possible to configure a light and compact optical system, and large aperture beam steering was made possible by using an array. This appears to be the most suitable beam steering device for near-eye displays. However, in the process of making the array, the fill-factor of the device cannot achieve a value of 100% due to the sidewalls, and the display is partially obscured. This problem was resolved by magnifying the prism using a lens array. Problems arising from magnification of the display by the lens array were solved through image processing to resize each cell image.
We also demonstrated the design, simulation, fabrication, and optical properties of an achromatic electrowetting prism array to reduce the chromatic dispersion of the electrowetting prism. Using the doublet structure of the electrowetting prism, the refractive index conditions of the conductive and non-conductive liquids that minimize chromatic dispersion were obtained. Both the achromatic condition and the density condition were satisfied by mixing two conductive liquids and two non-conductive liquids. The achromatic effect was verified through simulation and experiment. For comparison, a general prism to which the achromatic condition was not applied was also tested. When measuring the degree of chromatic dispersion compared to that of the original image, the achromatic prism was found to decrease chromatic dispersion by 70% compared to the general prism.
Funding
Ministry of Science and ICT, South Korea; National Research Foundation of Korea.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF2020R1F1A107408912) and the BK21 FOUR program.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. G. A. Koulieris, K. Akşit, M. Stengel, R. K. Mantiuk, K. Mania, and C. Richardt, “Near-eye display and tracking technologies for virtual and augmented reality,” Comput. Graph. Forum 38(2), 493–519 (2019). [CrossRef]
2. W. Vorraber, S. Voessner, G. Stark, D. Neubacher, S. DeMello, and A. Bair, “Medical applications of near-eye display devices: an exploratory study,” INT. J. SURG. 12(12), 1266–1272 (2014). [CrossRef]
3. M. C. Tsai and T. X. Lee, “Design of an ultra-thin near-eye display with geometrical waveguide and freeform optics,” Opt. Express 22(17), 20705–20719 (2014). [CrossRef]
4. H. Hua and B. Javidi, “A 3D integral imaging optical see-through head-mounted display,” Opt. Express 22(11), 13484–13491 (2014). [CrossRef]
5. H. Li, L. Trutoiu, K. Olszewski, L. Wei, T. Trutna, P. L. Hsieh, and C. Ma, “Facial performance sensing head-mounted display,” ACM Trans. Graph. 40(4), 1–15 (2021). [CrossRef]
6. W. Song, Y. Wang, D. Cheng, and Y. Liu, “Light field head-mounted display with correct focus cue using micro structure array,” Chin. Opt. Lett. 12(6), 060010 (2014). [CrossRef]
7. J. Kim, Y. Jeong, M. Stengel, K. Akşit, R. Albert, B. Boudaoud, and D. Luebke, “Foveated AR: dynamically-foveated augmented reality display,” ACM Trans. Graph. 38(4), 1–15 (2019). [CrossRef]
8. G. Tan, Y. H. Lee, T. Zhan, J. Yang, S. Liu, D. Zhao, and S. T. Wu, “Foveated imaging for near-eye displays,” Opt. Express 26(19), 25076–25085 (2018). [CrossRef]
9. J. Chen, L. Mi, C. P. Chen, H. Liu, J. Jiang, and W. Zhang, “Design of foveated contact lens display for augmented reality,” Opt. Express 27(26), 38204–38219 (2019). [CrossRef]
10. B. Bastani, E. Turner, C. Vieri, H. Jiang, B. Funt, and N. Balram, “Foveated pipeline for AR/VR head-mounted displays,” Inform. Display 33(6), 14–35 (2017). [CrossRef]
11. U. Hofmann, J. Janes, and H. Quenzer, “High-q mems resonators for laser beam scanning displays,” Micromachines 3(2), 509–528 (2012). [CrossRef]
12. C. Knoernschild, C. Kim, F. P. Lu, and J. Kim, “Multiplexed broadband beam steering system utilizing high speed MEMS mirrors,” Opt. Express 17(9), 7233–7244 (2009). [CrossRef]
13. P. Zhang, P. M. Goodwin, and J. H. Werner, “Fast, 3D imaging via confocal line scanning of a Bessel beam using a single galvo mirror,” Proc. SPIE 8947, 89471K (2014). [CrossRef]
14. M. S. Devadas, Z. Li, T. A. Major, S. S. Lo, N. Havard, K. Yu, and G. V. Hartland, “Detection of single gold nanoparticles using spatial modulation spectroscopy implemented with a galvo-scanning mirror system,” Appl. Opt. 52(32), 7806–7811 (2013). [CrossRef]
15. X. Wu, S. Chen, B. Shi, W. Chen, and X. Xiong, “High-powered voice coil actuator for fast steering mirror,” Proc. SPIE 50(2), 023002 (2011). [CrossRef]
16. Z. He, F. Gou, R. Chen, K. Yin, T. Zhan, and S. T. Wu, “Liquid crystal beam steering devices: Principles, recent advances, and future developments,” Crystals 9(6), 292 (2019). [CrossRef]
17. J. Kim, C. Oh, M. J. Escuti, L. Hosting, and S. Serati, “Wide-angle nonmechanical beam steering using thin liquid crystal polarization gratings,” Proc. SPIE 7093, 709302 (2008). [CrossRef]
18. D. P. Resler, D. S. Hobbs, R. C. Sharp, L. J. Friedman, and T. A. Dorschner, “High-efficiency liquid-crystal optical phased-array beam steering,” Opt. Lett. 21(9), 689–691 (1996). [CrossRef]
19. A. Yaacobi, J. Sun, M. Moresco, G. Leake, D. Coolbaugh, and M. R. Watts, “Integrated phased array for wide-angle beam steering,” Opt. Lett. 39(15), 4575–4578 (2014). [CrossRef]
20. J. K. Doylend, M. J. R. Heck, J. T. Bovington, J. D. Peters, L. A. Coldren, and J. E. Bowers, “Two-dimensional free-space beam steering with an optical phased array on silicon-on-insulator,” Opt. Express 19(22), 21595–21604 (2011). [CrossRef]
21. S. Terrab, A. M. Watson, C. Roath, J. T. Gopinath, and V. M. Bright, “Adaptive electrowetting lens-prism element,” Opt. Express 23(20), 25838–25845 (2015). [CrossRef]
22. J. Lee, J. Lee, and Y. H. Won, “Nonmechanical three-dimensional beam steering using electrowetting-based liquid lens and liquid prism,” Opt. Express 27(25), 36757–36766 (2019). [CrossRef]
23. J. Cheng and C. L. Chen, “Adaptive beam tracking and steering via electrowetting-controlled liquid prism,” Appl. Phys. Lett. 99(19), 191108 (2011). [CrossRef]
24. J. Lee, J. Lee, and Y. H. Won, “Image stitching using an electrowetting-based liquid prism with a fabrication method,” Opt. Express 29(2), 729–739 (2021). [CrossRef]
25. J. M. Sasián, “Aberrations from a prism and a grating,” Appl. Opt. 39(1), 34–39 (2000). [CrossRef]
26. H. W. Rhee, J. B. You, H. Yoon, K. Han, M. Kim, B. G. Lee, and H. H. Park, “32 Gbps data transmission with 2D beam-steering using a silicon optical phased array,” IEEE Photon. Technol. Lett. 32(13), 803–806 (2020). [CrossRef]
27. S. R. Davis, S. D. Rommel, S. Johnson, M. H. Anderson, and W. Y. Anthony, “Liquid crystal clad waveguide laser scanner and waveguide amplifier for LADAR and sensing applications,” Proc. SPIE 9365, 93650N (2015). [CrossRef]
28. J. Lee, J. Lee, and Y. H. Won, “Beam steering and forming in compact electrowetting prism array with separate electrode control,” OSA Contin. 4(9), 2400–2409 (2021). [CrossRef]
29. W. S. Sun, C. L. Tien, C. C. Sun, and C. C. Lee, “A low-cost optimization design for minimizing chromatic aberration by doublet prisms,” J. Opt. Soc. Korea 16(4), 336–342 (2012). [CrossRef]
