Open Access
Add to BibSonomy
Abstract
A super-multiview light-field display with horizontal and vertical parallax is realized by time-division and color multiplexing to deliver full-color images to each viewpoint. In the conventional study, an image of a different color is delivered to each viewpoint to induce focal accommodation. In the proposed method, we deliver images of different colors sequentially to generate a full-color image by an after-image effect. Though the number of time-divisions increases in the proposed method, perceived flicker is suppressed by showing different colors at different timings. We compare the observed images given by the proposed method with those given by the conventional method to find out that the former reproduces a natural blur effect when the image is defocused. We also confirm with a psychophysical experiment using a refractometer that the proposed method induces a stronger focal accommodation than other super-multiview methods with a smaller number of time-divisions or with a stronger flicker. The proposed optics is applicable to a near-eye display with a natural focal effect.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Corrections
Yuta Watanabe and Hideki Kakeya, "Time-division and color multiplexing light-field display using liquid-crystal display panels to induce focal accommodation: publisher’s note," Appl. Opt. 60, 2517-2517 (2021)https://opg.optica.org/ao/abstract.cfm?uri=ao-60-9-2517
1 March 2021: Corrections were made to Table 2.
1. INTRODUCTION
Binocular convergence and focal accommodation are the two major physiological factors that constitute human depth perception. As shown in Fig. 1, the focusing point of each eye matches the depth of binocular convergence under a natural environment. When the viewer is observing a 3D image given by a standard stereoscopic display, however, the focus of the eye is adjusted onto the screen while the binocular convergence is adjusted to the 3D image away from the screen. This discrepancy is called vergence-accommodation conflict, which is one of the main causes of eye fatigue or sickness specific to stereoscopic vision. Since most of the conventional stereoscopic displays realize stereoscopy by displaying two different images with parallax for the left eye and the right eye, vergence-accommodation conflict occurs inevitably.
A super-multiview display is one of the methods to solve the problem of vergence-accommodation conflict [1–4], as shown in Fig. 2. Super-multiview displays project multiple images to each eye by displaying parallax images at intervals smaller than the size of the pupil. In other words, they generate a dense light field around the eyes. In order to prevent the image from becoming double or multiple images and blurred when two or more light rays going through an aerial 3D point are projected onto the retina, the focusing position is induced to that aerial point, which eliminates vergence-accommodation conflict. To realize a practical super-multiview display, however, a huge number of views is required to be displayed to cover a wide viewing zone.
One way to decrease the number of views to be presented is to generate light fields only around the eyes by use of a face-tracking system. Takaki et al. have realized it by use of a lenticular lens [5]. Kakeya et al. and Watanabe et al. have realized a light field around the eye maintaining high spatial resolution of the presented image [6–8] based on the time-division multiplexing parallax barrier technique [9–11].
Generation of a light field around the eye is easier to attain when we use a head mounted display or a near-eye display. Takaki et al. proposed a super-multiview near-eye display system that combines a high speed spatial light modulator (SLM) and a 2D light source array [12]. Though this method can provide many views with its extremely fast refresh rate, the device is still too expensive to be commercialized.
As a more inexpensive solution, Watanabe et al. have proposed a system using liquid-crystal displays (LCDs) with a high refresh rate instead of an SLM [13]. To express many views with a limited refresh rate, a combination of time-division multiplexing and color multiplexing is applied in this system, where each view for different viewpoints is expressed with different colors. Since many views are projected to a pupil, the colors are merged to perceive a full-color image. However, the induction of focal accommodation is not sufficient in this system.
In this paper, we propose an LCD-based light-field display that delivers images of different colors sequentially to generate a full-color image by an after-image effect. It is known that perceived flicker is suppressed by time-division multiplexing anaglyph technology, where image components of different colors are shown sequentially instead of showing a full-color image with a long interval [14–17].
This paper is organized as follows. The principle of the propose method is explained in Section 2. The experimental setup is explained in Section 3, and the results of the experiments are shown in Section 4. We discuss the obtained results in Section 5, and the paper is concluded in Section 6.
2. PRINCIPLE
The optical setup of the proposed system is shown in Fig. 3. Here we use an achromatic lens whose focal length is $f$. We use an LCD panel for dot-matrix control of a backlight, which is placed $a$ mm behind the lens. Suppose the relationship $1/a + 1/b = 1/f$ holds. The eye of an observer is placed $b$ mm in front of the lens so that the real image of the rear panel may be generated on the pupil, where the magnification ratio of the image is $b/a$.
In the proposed system, we show different colors at different places on the backlight. Then different colors are projected on different parts of the pupil as shown in Fig. 4. When we show blue, green, and red images with parallax on the front LCD panel, the images with different color and parallax are projected to different parts of the pupil. If we keep on changing the positions of the colored backlight sequentially and alternating the image on the front LCD panel accordingly to show the image with proper horizontal and vertical parallax for each viewpoint, a natural light field is reconstructed around the eye.
Fig. 4. Optical configuration to generate a light field around the eye. Horizontal parallax is given in each frame, and vertical parallax is given by the after-image effect of time-multiplexing.
In the previous study [13], Watanabe et al. used time-division quadruplexing to generate a light field with 12 (3 rows and 4 columns) viewpoints, where only one of the three colors is observed. In the proposed system, we apply time-division multiplexing anaglyph technology, where image components of different colors are shown sequentially to reconstruct a full-color image at each viewpoint, as shown in Fig. 5. We generate 9 (3 rows and 3 columns) viewpoints instead of 12 viewpoints in this study.
Fig. 5. Ninefold time-division to realize a full-color super-multiview image with nine viewpoints. A white image is reproduced for all viewpoints using an after-image effect.
3. EXPERIMENT
For comparison, we make four different experimental settings to evaluate the generated light field. The detail of the four settings is shown in Fig. 6, where $t$ is the frame number.
In the first setting, we simulate the previous study, where a single color image is delivered to each viewpoint. Here time-division triplexing is applied to generate nine viewpoints.
In the second setting, we apply sixfold time-division, where the blue backlight and the green backlight are shown alternately to generate a cyan backlight next to a red backlight with the after-image effect. The choice of red and cyan is to keep the luminance at each viewpoint as close as possible.
The third setting is the method we propose in this paper. We apply ninefold time-division, where a full-color image is delivered to each viewpoint by using three frames. In this system, we need nine frames in total to reconstruct a full-color image. Regardless of the long time sequence, the flicker is expected to be suppressed because of the time-division anaglyph scheme.
The fourth setting is the simple ninefold time-division, where a full-color image is delivered to each viewpoint sequentially. Since the time-division anaglyph technique is not applied, more distinct flicker is expected in this method.
We used a pair of 24 inch LCD panels (AUO M240HW01 V8) whose resolution was ${{1920}} \times {{1080}}$ (the pixel pitch was 0.276 mm). The optical parameters were as follows: $f = {{215}}$; $a = {{430}}$; $b = {{430}}$ (mm). The size of the backlight segment (one of the nine squares) was ${1.1}\;{\rm{mm}} \times {1.1}\;{\rm{mm}}$ (4 by 4 pixels, each of which was composed of three color subpixels), and the real image of the same size was generated around the pupil because $a = b$.
The light fields with different amounts of parallax were generated with the above system and were observed by changing the focus of a camera, which was set on the point where the pupil was supposed to be located. The camera we used was Sony DSC-RX100M4. We also measured the diopter of viewers with a refractometer (Grand Seiko WAM-5500) to measure the focal accommodation response to the light field generated by the above system. The diopter was measured continuously with time intervals ranging from 0.12 s to 0.19 s.
4. RESULTS
Figure 7 shows the photos of the light field taken with different focal settings. The asterisk in the upper right was located 100 mm in front of the panel with the corresponding amount of parallax (0.33 mm), that in the upper left was located 50 mm in front of the panel (0.14 mm parallax), that in the lower right was located 65 mm away from the panel (0.14 mm parallax in the opposite direction), and that in the lower left was located 187 mm away from the panel (0.33 mm parallax in the opposite direction). The red arrows point at the asterisks on which the camera was focused.
Fig. 7. Photos with different focal conditions. The red arrows point at the focused images. The upper right asterisk is 100 mm in front; the upper left is 50 mm in front; the lower right is 65 mm behind; and the lower left is 187 mm behind the display.
As the figure shows, the farther away the asterisk is located from the focal point, the more blurred the image becomes. The image in focus includes some blurs when the image includes larger parallax to show the image away from the display panel.
To further clarify the difference of blurs under different conditions, we also showed a text image on this system. An enlarged text image on the front display panel with parallax in different colors is shown in Fig. 8, and the enlarged observed images when the number of time-division $T$ is 3 (single color for each viewpoint), 6 (cyan and red), and 9 (full-color) are shown in Fig. 9. When $T = {{3}}$, separation of colors in the blurred image is distinct, which is hard to perceive as a natural blur. When $T = {{6}}$, the color of the blur is suppressed to a certain extent. When $T = {{9}}$, the blur does not have color separation, which gives a whitish and natural image blur. A natural blur observed when the viewer is out of focus is expected to trigger focal accommodation more effectively.
The result of the experiment measuring focal accommodation of the subjects observing the light field generated in the proposed system is shown in Fig. 10. Here eight male subjects in their 20s and 40s were tested. In this experiment, an X-shaped image, where each line segment was 7.4 mm long, was displayed 100 mm in front of the display panel (0.33 m from the eye), and after 10 s, the image was moved to 187 mm behind the display panel (0.617 m from the eye). For reference, the same X-shaped images were physically displayed at 0.33 m and 0.671 m from the eyes, which were switched after 10 s, and the accommodation response was measured.
Fig. 10. Focal accommodation of eight subjects when the light field is generated by four kinds of super-multiview methods (color multiplexing ${\rm{T}} = {{3}}$, 6, 9, and no color multiplexing named “white”). Reference data given by a physical display is labeled “Real.”
The white luminance and the black luminance of the display were ${{118}}\;{\rm{cd/}}{{\rm{m}}^2}$ and ${{10}}\;{\rm{cd/}}{{\rm{m}}^2}$, respectively. Note that the luminance of the presented image is reduced in accordance with the number of time-divisions. The refresh rate was 144 Hz. The subjects saw the image with a single eye to eliminate the effect of binocular convergence, which is known to affect focal accommodation [18,19]. The subjects were allowed to blink, and the missing data while blinking were eliminated.
$D$ is the diopter, which is supposed to be the inverse of the distance from the eye to the focused object. In this experiment, the diopter is supposed to be 3 ($= \;{{1}}\;{\rm{m}}/0.33\;{\rm{m}}$) when the presented image is close (in the first 10 s) and 1.6 ($= \;{{1}}\;{\rm{m}}/0.617\;{\rm{m}}$) when the image is far (after 10 s), theoretically. Note that this value varies from person to person, and the change in the diopter value is more important than the value itself.
In Fig. 10, the red dots show the diopters of the subjects when the ninefold time and color division is applied. As the figure shows, subjects B, D, and F show crisp change of diopters after 10 s, when the distance of image is changed, while similar responses are not observed for subjects B and D in other conditions. Subjects G and H show a delayed crisp response, which shows it takes time to adjust to the moved image. Subjects A, C, and E show gradual change, and the amount of change is notable for subject E, while the other two subjects show smaller shifts of diopters. Compared with the focal accommodation response to a physical display, subjects A, D, F, and G showed less accommodation to the super-multiview image supposed to be presented close to the eye.
The mean diopter values from 0 to 10 s and those from 11 to 21 s for each subject are shown in Table 1. The data from 10 to 11 s was excluded for analysis because the timing of the image switch can include some delays due to manual control in the experiment. The numbers with asterisks mean that the focal accommodation response is reversed. As this table shows, only the ninefold time and color division includes no asterisks.
Table 1. Mean Diopter Values in Each Experiment
To see whether the difference of average is statistically significant, t-tests were applied to the obtained data. The results of t-tests applied to the data in Table 1 are shown in Table 2. The numbers with asterisks mean that the difference is statistically significant (5% significance level), but the significant difference is reversed from the intended focal effect. The numbers with pluses mean that the difference is not statistically significant. As the table shows, the ninefold time and color division includes no asterisks or plus marks, meaning that the proposed method induces intended focal accommodation with significant difference for all subjects.
Table 2. Results of t-Tests ($p$-Values)
Fig. 11. Observed asterisk image (left) and the real images of backlights generated around the eye (center and right). The focus of the photo is adjusted to the asterisk displayed 187 mm behind the panel (the same as the bottom left picture at $T = {{9}}$ in Fig. 6), where the horizontal line is clear and the vertical line is doubled and blurred. As the real images of backlights show, diffraction is stronger in the horizontal direction.
5. DISCUSSION
As the results of the experiments show, the proposed method succeeds in inducing proper focal accommodation stably for all subjects, while the other methods sometimes fail, including ninefold time-division without dividing colors, which means that suppression of flickers with color time-division contributes to the inducement of proper focal accommodation. The superiority of the ninefold time-division system over threefold and sixfold time-division systems is considered to come from the natural blur as shown in Fig. 9, which can give stronger incentive to decrease the perceived blur.
As shown in Fig. 7, the image is harder to focus when the presented image is farther from the display panel and the parallax becomes larger. The blur is especially distinct in the horizontal direction as shown in Fig. 11 (left), which is considered to come from diffraction, causing blur in the real image of backlight generated around the eye (the center right of Fig. 11).
The diffraction is stronger in the horizontal direction because the color filters of the LCD panel used in this experiment are aligned in the horizontal direction. (The effect of diffraction by LCD panels has been studied to evaluate cross talk caused by an active parallax barrier [20].) Further study is needed to suppress the effect of diffraction to realize a more ideal light field around the eye. Suppression of diffraction can lead to a more natural focal accommodation, which is closer to that when the distance of the image is physically changed.
6. CONCLUSION
This paper has presented a super-multiview light-field display with horizontal and vertical parallax using time-division and color multiplexing to deliver full-color images to each viewpoint. The images of different colors are delivered sequentially to generate a full-color image by an after-image effect. Though the number of time-divisions increases in the proposed method, perceived flicker is suppressed by showing different colors at different timings. The proposed method has succeeded in reproducing a natural blur effect when the image is defocused. We have also confirmed with a psychophysical experiment using a refractometer that the proposed method stimulates stronger focal accommodation of some subjects than the other methods with smaller numbers of time-divisions. Further study is needed to suppress the effect of diffraction to realize a near-eye display with a less blurry light field.
Funding
Core Research for Evolutional Science and Technology (JPMJCR18A2); Japan Society for the Promotion of Science (17H00750).
Disclosures
The authors declare that there are no conflicts of interest related to this paper.
REFERENCES
1. Y. Kajiki, H. Yoshikawa, and T. Honda, “Hologram-like video images by 45-view stereoscopic display,” Proc. SPIE 3012, 154–166 (1997). [CrossRef]
2. Y. Takaki, “Thin-type natural three-dimensional display with 72 directional images,” Proc. SPIE 5664, 56–63 (2005). [CrossRef]
3. H. Nakanuma, H. Kamei, and Y. Takaki, “Natural 3D display with 128 directional images used for human-engineering evaluation,” Proc. SPIE 5664, 28–35 (2005). [CrossRef]
4. Y. Takaki and N. Nago, “Multi-projection of lenticular displays to construct a 256-view super multi-view display,” Opt. Express 18, 8824–8835 (2010). [CrossRef]
5. Y. Takaki, K. Tanaka, and J. Nakamura, “Super multi-view display with a lower resolution flat-panel display,” Opt. Express 19, 4129–4139 (2011). [CrossRef]
6. H. Kakeya, “A full-HD super-multiview display with time-division multiplexing parallax barrier,” J. Soc. Inf. Disp. 49, 259–262 (2018). [CrossRef]
7. H. Kakeya and Y. Watanabe, “A full-HD super-multiview display with a deep viewing zone,” in IS&T Electronic Imaging (2019), paper 628. 1-6.
8. Y. Watanabe and H. Kakeya, “A full-HD super-multiview display based on adaptive time-division multiplexing parallax barrier,” ITE Trans. Media Technol. Appl. 8, 230–237 (2020). [CrossRef]
9. H. Kakeya, K. Okada, and H. Takahashi, “Time-division quadruplexing parallax barrier with subpixel-based slit control,” ITE Trans. Media Technol. Appl. 6, 237–246 (2018). [CrossRef]
10. A. Hayashishita and H. Kakeya, “Time-division multiplexing parallax barrier with sub-subpixel phase shift,” SID Dig. Tech. Pap. 49, 1515–1518 (2018). [CrossRef]
11. H. Kakeya, A. Hayashishita, and M. Ominami, “Autostereoscopic display based on time-multiplexed parallax barrier with adaptive time-division,” J. Soc. Inf. Disp. 26, 595–601 (2018). [CrossRef]
12. T. Ueno and Y. Takaki, “Super multi-view near-eye display to solve vergence–accommodation conflict,” Opt. Express 26, 30703–30715 (2018). [CrossRef]
13. Y. Watanabe and H. Kakeya, “A super-multiview display with horizontal and vertical parallax by time division and color multiplexing,” SID Symp. Dig. Tech. Pap. 51, 1017–1020 (2020). [CrossRef]
14. Q. Zhang and H. Kakeya, “An autostereoscopic display system with four viewpoints in full resolution using active anaglyph parallax barrier,” Proc. SPIE 8648, 86481R (2013). [CrossRef]
15. Q. Zhang and H. Kakeya, “Time-division multiplexing parallax barrier based on primary colors,” Proc. SPIE 9011, 90111F (2014). [CrossRef]
16. Q. Zhang and H. Kakeya, “A high quality autostereoscopy system based on time-division quadplexing parallax barrier,” IEICE Trans. Electron. E97-C, 1074–1080 (2014). [CrossRef]
17. Q. Zhang and H. Kakeya, “Time-division quadruplexing parallax barrier employing RGB slits,” J. Disp. Technol. 12, 626–631 (2016). [CrossRef]
18. T. Shiomi, K. Uemoto, T. Kojima, and S. Sano, “Simultaneous measurement of lens accommodation and convergence in natural and artificial 3D vision,” J. Soc. Inf. Disp. 21, 120–128 (2013). [CrossRef]
19. H. Mizushina, J. Nakamura, Y. Takaki, and H. Ando, “Super multi-view 3D displays reduce conflict between accommodative and vergence responses,” J. Soc. Inf. Disp. 24, 747–756 (2016). [CrossRef]
20. A. Hayashishita, T. Matsumoto, K. Kusafuka, and H. Kakeya, “An active barrier autostereoscopic display with less crosstalk,” in International Display Workshops (IDW) (2019), pp. 1066–1069.
