A three-dimensional (3D) windshield display can display driving information in the vicinity of objects in the driver’s front scene. We propose a super multi-view windshield display that can present the information in a wide depth range. The super multi-view display technique provides a smooth motion parallax. Motion parallax is the only physiological cue for perceiving the depths of 3D images displayed at far distances; these cannot be perceived by other physiological cues such as vergence, binocular disparity, and accommodation. A prototype system, which generates 36 viewing zones with a horizontal interval of 3.61 mm, was constructed. The smoothness of the motion parallax and the accuracy of the depth perception were evaluated.
©2011 Optical Society of America
A windshield display (WSD) [1–3] is a large head-up display (HUD) used for automobiles, which superimposes images over a driving scene. In the present study, a super multi-view windshield display (SMV-WSD) has been developed, which enables the presentation of driving information in the vicinity of objects in the front scene.
The WSD permits the display of information in the driver’s forward field of view as a virtual image. A windshield is used as a half mirror so that the driving information can be viewed along with the outside scene. This placement decreases the need to shift gaze from the outside scene to obtain information from displays inside the vehicle . Moreover, the WSD images may also be presented at a distance further into the forward scene than allowed by traditional HUDs, thereby reducing the reaccommodation time between the WSD information and elements in the external scene . The WSD has two types of applications: one is the presentation of codified information such as speed, route navigation symbology, and collision avoidance warnings ; the other is vision enhancement for use in conditions of restricted visibility such as night-time darkness or fog [7–9].
When three-dimensional (3D) display technology is applied to the WSD, the information that is strongly related to an object in the front scene could be displayed in the vicinity of the object, as shown in Fig. 1 . Drivers are therefore not required to change their gaze direction or the accommodation of their eyes to see the information while driving. The gaze movements and the reaccommodation time are minimized. The 3D images can be displayed from just in front of the vehicle to more than 100 m ahead of the vehicle in the depth direction of the front driving scene.
The development of the three-dimensional windshield display (3D-WSD) was previously reported by Nakamura et al. . They demonstrated the 3D-WSD using a two-view-type 3D display. This prototype system employed two projectors to display two parallax images to the left and right eyes. It is well-known that the human visual system uses four physiological factors to perceive depth, namely, vergence, binocular disparity, motion parallax, and accommodation . When 3D images are displayed by a two-view-type 3D display, the depth is perceived by vergence and binocular disparity. These two physiological factors are effective for relatively short distances so that observers cannot correctly perceive the depth of 3D images displayed at far distances. Vergence, which can perceive absolute depth, is effective for a depth of ~10 m or less, and binocular disparity perceives the relative depth with respect to the absolute depth perceived by vergence . Nakamura et al. experimentally showed that the perception of a depth of 20 m or more by the factors of vergence and binocular disparity was difficult with their prototype system. Moreover, the two-view system has another problem: The perceived horizontal and vertical positions of the 3D images change depending on the horizontal and vertical positions of the driver’s eyes, as shown in Fig. 2(a) , because the two images displayed on the screen, corresponding to both eyes, do not change with the eye position.
In the present study, to overcome the above problems, the super multi-view (SMV) display technique [11–14] was applied to the 3D-WSD. The SMV display technique was originally developed in order to construct a natural 3D display that is free from the accommodation-vergence conflict that causes visual fatigue during the observation of 3D images. The SMV display technique makes the interval of multiple viewing zones of a multi-view display smaller than the pupil diameter of the eye to allow accommodation to work for 3D images. Therefore, the eye can focus on the 3D images and the accommodation-vergence conflict does not occur. The interval is made smaller than 5 mm, which is the average pupil diameter of a human being.
The small view interval also provides a smooth motion parallax. Motion parallax is the physiological factor that is effective for long-distance depth perception. Therefore, the smooth motion parallax provided by the SMV display technique has the possibility to enable the display of images at far distances. Moreover, even when the driver’s eye positions change, the perceived position of the 3D images does not change because the two images seen by both eyes change depending on the eye positions, as shown in Fig. 2(b). Accommodation is effective for short-distance perception, typically within 1 ~2 m, so that the SMV display technique provides a smooth motion parallax rather than the evocation of accommodation, when used with the 3D-WSD.
The present study constructs a prototype SMV-WSD system to study its feasibility; it consists of a multi-view display and a projection system. A 3D image generated by a lenticular-type multi-view display is imaged by a projection system to produce a virtual image. The combination of a 3D display and a projection system was previously reported [15–17]. An integral imaging (II) display was used as the 3D display. The 3D images were projected near the observers, i.e., within several meters, because the researchers aimed to construct an ordinary 3D display for desktop and home use. The II display offers both horizontal and vertical parallaxes, although the resolution of the 3D images is low. In contrast, although the SMV display used in the present study provides only a horizontal parallax, the resolution is high. For automobile use, because the driver’s vertical eye position can be assumed to be fixed toward the windshield, the vertical motion parallax is less important than the higher resolution used to display rich driving information.
The head-mount-type SMV display  was previously developed by Kim et al. This system is a monocular display and does not offer motion parallax because a limited number of viewing zones are generated around the pupil.
In the present study, a prototype SMV-WSD display was constructed and the ability to display long-distance 3D images was evaluated.
2. Long-distance depth presentation
Prior to describing the system design of the SMV-WSD, the long-distance depth presentation of 3D images is discussed. First, depth reproduction by 3D displays is considered. Then, the depth perception by physiological factors is considered.
2.1 Long-distance depth reproduction by 3D displays
The precision of the depth reproduction by 3D displays depends on the display parameters and the viewing conditions. Figure 3 shows the geometry of the depth reproduction. The display plane, on which the 3D-WSD projects the virtual image of the 3D display screen, is located in front of the viewer. The distance between the display plane and the viewer is denoted by l. The distance between a 3D image and the viewer is denoted by z. The inter-ocular distance is denoted by P. The disparity on the display plane is represented by ξ. From the similarity of triangles,Eq. (2) can be rewritten as
The precision of long-distance depth reproduction is evaluated. We assume that the field of view for 3D images is 30°. The calculation was done for 3D image formats of 640 × 480 (VGA), 1,024 × 768 (XGA), 1,280 × 1,024 (SXGA), and 1,920 × 1,080 (full HD). The dependence of the depth uncertainty, Δz, on the viewing distance z was calculated using Eq. (3) and is shown in Fig. 4 . The uncertainty can be reduced by reducing the angular resolution, Δθp. However, the precision of the depth reproduction is not satisfactorily high even when 3D images are displayed with the full HD image format. Therefore, depth reproduction based on the disparity of left and right images is difficult when 3D images are displayed at a long distance.
2.2 Long-distance depth perception by physiological factors
As mentioned above, accurate long-distance depth reproduction using the disparity of left and right images is impossible for 3D displays. Therefore, vergence and binocular disparity cannot work. Essentially, depth perception by vergence is effective for ~10 m or less, and binocular disparity perceives the depth relative to the absolute depth perceived by vergence. Either way, accommodation is ineffective for long-distance depth perception. The last physiological factor is motion parallax, which is effective for both short-distance and long-distance depth perception.
Here, the motion parallax produced by a multi-view 3D display is considered. The multi-view display generates multiple viewing zones where corresponding parallax images are viewed. Because the viewing zones are discrete, the parallax image viewed by the eye does not change until the eye moves to adjacent viewing zones. Therefore, the motion parallax of a multi-view display is discontinuous. This discontinuity might affect the depth perception of 3D images.
Figure 5 explains the discontinuity of the motion parallax. The parallax images are displayed on the display plane. The width of the viewing zone for each viewpoint is denoted by d. For simplicity, the pitch of the viewing zones is assumed to be equal to the width of the viewing zones, i.e., crosstalk between the viewing zones is zero. When the eye moves horizontally within one viewing zone, the viewer sees the 3D image move toward the opposite horizontal direction. Then, the eye moves to the adjacent viewing zone and the 3D image jumps horizontally.
Now, consider that we display a point P as a 3D image. The parallax image corresponding to the viewing zone #n contains a point Pn. When the eye moves from endpoint A to the other endpoint B on the viewing zone #n, the perceived 3D point moves from A’ to B’. Then, as the eye moves to the adjacent viewing zone #n+1, the perceived 3D point jumps to A’. The discontinuity of the motion parallax is presented by Δθm, which is the visual angle of the line A’B’. Because the length of A’B’ is given by d (z − l) / l, the visual angle Δθm is given by
The requirements for the SMV-WSD are: (1) a virtual image of the 3D display screen that is projected through the windshield at an appropriate distance from the driver, preferably ~2zn; and (2) multiple viewing zones that are produced just in front of the driver with an appropriate horizontal pitch, preferably having the same magnitude as that of the pixel pitch of the virtual image.
The schematic of the proposed SMV-WSD system is illustrated in Fig. 6 . The SMV-WSD consists of a multi-view display and projection optics. Both the multi-view display and the lens are placed under the windshield, and a driver sees 3D images through the windshield. The screen of the multi-view display is imaged by a projection lens to produce a virtual image at an appropriate distance from the driver. A real image of the multiple viewing zones generated by the multi-view display is also imaged by the lens to generate multiple viewing zones just in front of the driver. The pitch of the projected viewing zones should be appropriately small to provide a smooth motion parallax.
The details of the imaging system are explained using Fig. 7 . The actual reflection imaging system is shown as a transmission imaging system for simplicity. The projection lens performs two image formations: one is the formation of the virtual image and the other is that of the driver’s viewing zones. The screen width of the multi-view display is denoted by D0, the distance between the screen and its viewing zones is denoted by l0, and the total width of the viewing zones is denoted by W0. The width of the virtual image is denoted by D, the distance between the virtual image plane (display plane) and the driver’s viewing zones is l, and the total width of the driver’s viewing zones is denoted by W. The focal length of the projection lens is denoted by f ′, the distance between the lens and the driver’s viewing zones is denoted by h, and the distance between the lens and the multi-view display is denoted by s. The image formation of the virtual image requires the following two relations:
When the number of viewing zones is represented by N, the pitch of the driver’s viewing zones is given by d = W / N. When the total number of pixels of the 3D image is denoted by X × Y, the pixel pitch of the virtual image is given by p = D / X.
4. Prototype system
A prototype SMV-WSD system was constructed to examine the feasibility of long-distance depth presentation.
First, the requirements for the system are described. The viewing angle should preferably be 30° or more to cover a large amount of the driver’s field of view. We assumed that the movement of the driver’s head while driving is limited in width to twice as large as the inter-ocular distance (65 mm), i.e., W = 130 mm. The distance between the projection lens and the viewer was set to h = 800 mm to maintain a sufficient space in front of the driver. The length l, the distance between the virtual image and the driver, is the critical parameter that determines the depth range of 3D images without the discontinuous motion parallax. This distance was also determined by another reason: The focal length of the projection lens must be increased in order to increase l. The spherical radius of the lens becomes larger with a longer focal length until the degradation of images caused by the complex curvature of the windshield cannot be neglected. Therefore, the distance l cannot be very large; it must be 2 m or less (l ≤ 2 m).
A lenticular-type 3D display was used as the multi-view display; it consists of a lenticular lens and a high-resolution flat panel display. The slanted lenticular technique  was used to increase the number of views. A liquid crystal display (LCD) was used as the flat panel display. The resolution was 1,920 × 1,200, the pixel pitch was 0.1905 mm, and the screen size was 17 inches (D0 = 366 mm). By slanting the lenticular lens, the rays emitted from each 6 × 6 pixels were given different horizontal proceeding directions to produce 36 viewing zones (N = 36) . The slant angle of the lenticular lens was 9.46° and the lens pitch was 1.13 mm. The distance between the display screen and the viewing zones was l0 = 29.1 m. The width of the total viewing zones was W0 = 4.65 m and the horizontal pitch of the viewing zones was do = 129 mm. The number of pixels in the 3D image was 320 × 200.
A Fresnel lens was used as the projection lens because a large lens diameter is required to provide the wide field of view. The Fresnel lens was designed using an optical design software. The focal length was f = 0.823 m. The distance to the lenticular display was s = 480 mm, and the distance between the display plane and the driver was l = 1.95 m. The width of the virtual image was D = 878 mm so that the pixel pitch of the virtual image was p = 2.74 mm. The distance to the driver’s viewing zones was h = 800 mm. The width of the driver’s viewing zones was W = 130 mm so the pitch of the driver’s viewing zones was d = 3.61 mm. The viewing angle was 25°, which is unfortunately smaller than the target value.
Instead of using a real windshield, a half mirror was used. Since a large-size half mirror was required, we used a half-silvered mirror, which is a glass plate with thin metal coating. Because of light abortion in the metal layer, the transmittance and reflectance were both 30%.
Figure 8 shows the constructed SMV-WSD. A driver’s seat was placed behind the windshield in order to fix the distance between the driver’s eyes and the Fresnel lens. When seated, the width of the driver’s horizontal head movement was limited so that 3D images could be viewed easily. A photograph of the 3D image captured from the center position of the viewing zones is shown in Fig. 9 ; a movie is provided. Figure 10 shows the photographs of a diamond-shaped mark displayed at 5, 10, 20, 30, 40, and 50 m; a movie is also provided. The photographs were captured in the hallway of a building because the display was not bright enough to be used outside. The length of the hallway was ~50 m. The brightness of the display could be increased by replacing the backlight of the LCD panel with a brighter one.
The crosstalk among the driver’s viewing zones was measured. A cooled charge coupled device (CCD) camera was placed at the plane of the driver’s viewing zones to measure the light intensity distributions of the viewing zones. A white image was displayed to one viewing zone in which the intensity distribution was measured and black images were displayed to the other viewing zones. The cooled CCD camera was mounted on a horizontal translation stage, and was appropriately moved in the horizontal direction to cover all the viewing zones. The measured results are shown in Fig. 11 . The average pitch of the viewing zones was 3.6 mm. The crosstalk existed among the viewing zones because the slanted lenticular technique intrinsically generates crosstalk  and the image formation by the Fresnel lens increases the crosstalk. When the crosstalk is defined as the ratio of the light intensity at the center of one viewing zone to the sum of the light intensities generated by other viewing zones at the center position, the average crosstalk was 62.8%.
5. System evaluation
The 3D resolution of the constructed SMV-WSD system is 320 × 200 pixels; considering the results for VGA resolution in Fig. 4, the uncertainty of the depth reproduction due to vergence and binocular disparity is very large. The long-distance 3D-image presentation using motion parallax produced by the constructed system was evaluated.
5.1 Smoothness of motion parallax
First, we examined the results derived in Sec. 2.2 regarding the smoothness of the motion parallax. Equation (5) predicts that the depth range of 3D images with smooth motion parallax depends on the pixel pitch p of the virtual image, the pitch d of the driver’s viewing zones, and the distance l to the virtual image. In this evaluation, the pitch of the viewing zones was changed and the smoothness of the motion parallax was evaluated.
The pitch of the viewing zones was changed between 3.61 mm, 7.22 mm, and 10.8 mm by displaying identical parallax images for one, two, and three succeeding viewing zones. It can be considered that the number of views of the prototype display was changed between 36, 18, and 12. We performed preparatory experiments to determine the depth range and the depth step for each viewing zone pitch. The test image was “Lenna,” which is a widely used standard test image. The test image was displayed at several different depths and the smoothness of the motion parallax was evaluated. Subjects were asked whether they were aware of the discontinuity of the motion parallax. The depth of the test images was changed in a random order. Five trials were done for each display condition. The number of subjects was 10.
The results are shown in Fig. 12 . The vertical axis shows the probability of the perception of the discontinuous motion parallax. From Figs. 10(a) and (b), the probability is 50% at the distances of 2.7 m and 3.6 m for the 12-view and the 18-view modes, respectively. Figure 10(c) shows that, for the 36-view mode, the discontinuity was not perceived even when the test image was displayed at 100 m.
Equation (5) in Sec. 2.2 predicts that the depth range with smooth motion parallax is 1.56 ~2.61 m when d = 10.8 mm, 1.41 ~3.14 m when d = 7.22 mm, and 1.11 ~8.09 m when d = 3.61 mm, respectively. The former two ranges coincide well with the experimental results. However, the last one does not agree with the experimental results. When d = 3.61 mm, i.e., the 36-view mode, the test image looked blurred when it was displayed at further than ~5 m. There is a possibility that the effective pixel pitch increased. Therefore, we examined the blurring of the 3D images.
Line pairs consisting of white and black lines whose line widths were one, two, and three pixels were displayed; the captured images are shown in Fig. 13 . The lines were slanted at the same angle as that of the lenticular lens. Straight vertical lines were not used because they became jaggy and their width varies in the vertical direction when displayed by the slanted lenticular lens. These photographs show that the image blurring was not obvious for the 12-view and 18-view modes. The image blurring was obviously observed in the 36-view mode. The image blurring was caused by the crosstalk among the viewing zones. As described in Sec. 4, the crosstalk was 62.8% for the 36-view mode. Therefore, considerable amount of parallax images from the adjacent viewing zones was superimposed onto the parallax image for each viewing zone so that the image was blurred. From Fig. 11, the crosstalk for the 12-view and 18-view modes was negligible. If we assumed that the effective pixel pitch was 1.5 times as large as the original one, the depth range would increase to 0.91 m ~∞.
5.2 Long-distance depth perception
The accuracy of the depth perception for long-distance 3D images was evaluated. The displayed depth was changed and the perceived depth was measured. For comparisons, we also measured the two-view mode (binocular mode) and the 18-view mode in addition to the 36-view mode. The two-view mode displays a left image to 18 viewing zones on the left-hand side and a right image to 18 viewing zones on the right-hand side.
As a test target, the diamond-shaped mark shown in Fig. 10 was used. To avoid depth perception by psychological factors, especially the depth perception by image size, the viewing angle of the test target was fixed at 1.78°, independent of the displayed depths. The 3D images shown in Fig. 8 were synthesized considering the image size factor so that the viewing angle varied. The displayed depth range was from 10 m to 50 m, with a depth interval of 10 m. A reference real object was moved to the depth at which the subjects felt the 3D image was located. The maximum depth was 50 m because the experiments were done in the hallway of a building and the length of the hallway was about 50 m. The depth of the test images was randomly changed. Five trials were done for each display condition. The number of subjects was seven.
The results are shown in Fig. 14 . Figure 14(a) shows that the depth perception by the two-view mode is effective for 20 m or less. These results coincide with the results reported in . From Fig. 14(b), the results for the 18-view mode are similar to those obtained for the two-view mode. For the 18-view mode, the motion parallax was smooth for the distance of 3.6 m or less, as described in Sec. 5.1. This means that the depth perception by motion parallax was ineffective, so the results were similar to the two-view mode. Figure 14(c) shows that the subjects correctly perceived the depth of the 3D image even when the image was displayed as far away as 50 m. As described in Sec. 5.1, the motion parallax was smooth for the distance of 100 m or less for the 36-view mode. This means that the motion parallax is effective for long-distance depth perception while the discontinuity of motion parallax is not perceived. Moreover, the error bars are the shortest for the 36-view mode compared to the other two modes. Thus, the smooth motion parallax improved the precision of the depth perception.
We proposed a super multi-view windshield display (SMV-WSD), which provides a smooth motion parallax in order to present driving information in the vicinity of distant objects. We discussed the depth range in which the discontinuity of the motion parallax is not perceived. A prototype SMV-WSD that has 36 viewing zones with an interval of 3.61 mm was constructed. The smoothness of the motion parallax and the long-distance depth perception were evaluated using the prototype system. We experimentally showed that the discontinuity of the motion parallax was not perceived when 3D images were displayed at 100 m. We also showed that the depth of the 3D image could be perceived when the image was displayed as far away as 50 m.
References and links
1. K. Nakamura, J. Inada, M. Kakizaki, T. Fujikawa, S. Kasiwada, H. Ando, and N. Kawahara, “Windshield Display for Intelligent transport system,” in Proc. 11th World Congress on Intelligent Transportation Systems, 85, pp. 3058–3065 (2004).
2. R. J. Kiefer, and A. W. Gellatly, “Quantifying the Consequences of the “Eyes-on-Road” Benefit Attributed to Head-Up Displays,” in Proc. SAE International Congress & Exposition, Paper No. 960946 (Society of Automotive Engineers, Warrendale, PA,1996).
3. N. Aoki, “Perspective for Head-Up Display,” in Proc. 11th World Congress on Intelligent Transportation Systems, 85, p3149–3156 (2004).
4. A. Stokes, C. Wickens, and K. Kite, “Human Factors Concepts,” in Display Technology, (Society of Automotive Engineers, Warrendale, PA, 1990).
5. D. Weintraub, and M. Ensing, “A head-up display state of the art report,” in Human Factors Issues in Head-Up Display Design: The Book of HUD, (CSERIAC, Wright-Patterson Air Force Base, Dayton, OH, 1992).
6. T. Todoriki, J. Fukano, S. Okabayashi, M. Sakata, and H. Tsuda, “Application of head-up displays for in-vehicle navigation/route guidance,” in Proc. Vehicle Navigation and Information Systems Conference, pp.479–484 (1994).
7. S. Nancy, Martinelli and Scott A. Boulanger, “Cadillac DeVille Thermal Imaging Night Vision System,” in Proc. SAE 2000 World Congress, Paper No. 2000–01–0323 (Society of Automotive Engineers, Warrendale, PA, 2000).
8. Y. Hagisato, Y. Iwata, K. Toyofuku, Y. Goto, H. Senoo, and T. Kumasaka, “The “Night View System” Using Near-Infrared Light,” TOYOTA Technical Review. 52, 48–53 (2002).
9. T. Tsuji, H. Hattori, M. Watanabe, and N. Nagaoka, “Development of night-vision system,” IEEE Trans. Intell. Transp. Syst. 3(3), 203–209 (2002). [CrossRef]
10. T. Okoshi, Three-Dimensional Imaging Techniques,Academic Press, New York, 1976.
11. Y. Kajiki, H. Yoshikawa, and T. Honda, “Hologram-like video images by 45-view stereoscopic display,” Proc. SPIE 3012, 154–166 (1997). [CrossRef]
12. T. Honda, Y. Kajiki, K. Susami, T. Hamaguchi, T. Endo, T. Hatada, and T. Fujii, “Three-dimensional display technologies satisfying ‘super multiview condition,” SPIE. Crtical. Reviews. CR76, p.218–249.
13. T. Honda, Y. Kajiki, S. Susami, T. Hamaguchi, T. Endo, T. Hatada, and T. Fujii, “A display system for natural viewing of 3-D images,” in Three-dimensional television, video and display technologies, B. Javidi, F. Okano ed. (Springer-Verlag, Berlin Heidelberg, Germany, 2002) p.461–487.
16. J. Kim, S.-W. Min, and B. Lee, “Viewing region maximization of an integral floating display through location adjustment of viewing window,” Opt. Express 15(20), 13023–13034 (2007). [CrossRef] [PubMed]
19. C. van Berkel and J. A. Clarke, “Characterization and optimization of 3D-LCD module design,” Proc. SPIE 3012, 179–186 (1997). [CrossRef]
20. Y. Takaki, “Thin-type natural three-dimensional display with 72 directional images,” Proc. SPIE 5664, 56–63 (2005). [CrossRef]
21. Y. Takaki, “Multi-view 3-D display employing a flat-panel display with slanted pixel arrangement,” J. Soc. Inf. Disp. 18(7), 476–482 (2010). [CrossRef]