1. Introduction

A light field 3D display approximates the continuous light field generated by real-world objects by presenting a subsample of the continuous light field and producing different images to various angular positions [1], thus allowing multiple viewers to perceive binocular disparity, motion parallax and occlusion depth cues simultaneously without special eyeglasses.

In the past decade, projector arrays have been demonstrated to be capable of producing convincing 3D displays of very high resolution, large-format images, wide FOV and flexibility to adapt to non-planar display geometries [2–6]. However, the fundamental issue of projector array displays is that a dense array of projectors is needed in order to provide high-density rays, consequently a number of problems arise: high cost, projector’s physical size constrain, complex and bulky setup, laborious geometric and radiometric calibration. These limitations make projector arrays difficult and exorbitant to be reproduced. We are inspired to remedy these gaps by introducing plane mirrors to projection-based light field display systems.

In this work, we propose an easily reproducible and cost-effective approach to realize light field display using a mirror array of special geometry to redirect light from a single, high-resolution projector. The optical paths of projector pixels reflected by the mirror array can be unfolded to mimic a large collection of rays projected from a dense array of virtual projectors in many different directions.

Mirror array has a number of advantages over projector array, including low cost, flexible reconfiguration, simple setup and easy calibration. Specifically, mirrors can create virtual projectors that are densely packed or even (virtually) overlapping with each other regardless of the physical size of the real projector, thus creating an ultra high-density light field that is not achievable by projector arrays. Such high-density light field allows more than one ray depicting the same 3D point enters the pupil of the viewer’s eye at slightly different angles simultaneously, thus naturally driving the eye’s lens to focus at the depicted depth rather than the screen surface and thereby effectively alleviating the vergence-accommodation conflict (VAC), which is a very common problem of conventional stereoscopic and multiview 3D displays.

The key challenges of realizing mirror array light field display systems are 1. how to properly design such a mirror apparatus that can accurately redirect light from the projector while maintaining the angular divergence of different pixels; and 2. how to determine numerous system parameters and balance trade-offs to optimize the display performance under realistic constraints. The goal of this work is to propose methodologies to address these challenges.

The primary contributions of this work include the following:

2. Related works

2.1. Projector arrays

The idea of using multiple projectors to create horizontal-parallax-only (HPO) display was first proposed by Ives in 1931 [7]. He focused 39 film projectors on a vertically aligned lenticular screen coated with diffuse paint on the back to reflect projector rays back to the same angular directions horizontally. Matusik et al. [8] used similar technique to demonstrate real-time 3D video streaming using 16 cameras and 16 projectors, where single-lenticular screen was used in front-projection and double-lenticular screen was used in rear-projection. Several convincing systems proposed recently have demonstrated that projector arrays are well-suited to 3D displays because of their ability to generate dense and flexible arrangements of pixels [2, 4–6, 8–10]. Most of these works, as well as our proposed mirror array displays, used vertical anisotropic diffuser as the screen material.

In recent years, researchers have used a massive number, as many as 300 units [4], of projectors to create 3D displays [4–6, 9]. However, the cost and calibration efforts required by these systems are very high. Apart from the massive number of projectors, these systems often involve a number of computers, special graphic cards, video expanding modules, cooling fans, connectors and cables, which lead to high cost, high power consumption, complex and bulky setup. As such, we propose to use inexpensive mirrors to simulate projector array to alleviate the above problems. Digital fabrication technologies make the construction of mirror array cheap and effortless. Unlike projector array, radiometric calibration is not needed by mirror array system as only a single projector is used. Moreover, the mirror-array-based configuration will reduce the system size by folding the projection path.

2.2. Mirror array imaging and display systems

Traditionally mirror plays an important role in optics. In the last decades, mirror systems have found new applications in computational photography [11]. Several computational camera systems proposed use arrays of plane [12] or curved mirrors [13, 14] to shape and rearrange camera rays for light field [15] imaging with a single camera.

Researchers have also used a camera, a projector and an array of mirrors for aperture synthesis in confocal imaging [16, 17]. Several integral displays have been produced using concave [18] and convex [19] mirror arrays. Hong et al. proposed a 3D-2D convertible display using a concave half-mirror array based on integral imaging [20]. However, concave or convex micro-mirrors of high precision are expensive to fabricate. Moreover, the spatial or angular resolution of integral displays are limited as the display pixels are divided into both horizontal and vertical views. Otsuka et al. [21] proposed a circularly-arranged mirror array which reflects multiview images onto a spinning screen to create 360 degrees-viewable images. However, a large amount of projector pixels are not used in their system. Moreover, there is a physical limitation on the display volume due to the existence of rotating parts. Jung et al. [22] used an one-dimensional (1D) mirror array to create an integral display. Arai et al. [23] used a mirror array and a gradient-index lens array to solve the image deterioration and depth reverse problems in integral imaging display systems. However, no precise geometry model of mirror array and image rendering method were presented in both works. The image capturing and reconstruction processes have to be bounded together. Moreover, only low-quality image reconstruction results have been achieved by these systems.

2.3. Light field display

A light field display presents a large number of angular rays to approximate the continuous light field. Light field displays have many advantages over multiview 3D displays. First, a light field display does not have a fixed viewing distance or distinct viewing zones, which gives more freedom to the viewers. Second, as a multiview display has distinct viewing zones, image flipping will arise when the viewer’s position is not aligned properly with the intended viewing zone or viewing angle [1]. On the contrary, a light field display has continuous viewing position, thus not suffering from the image flipping artifact [24].

Many kinds of light field display systems have been proposed. Jones et al. [3] applied a high-speed projector to emit multiple-center-of-projection (MCOP) images onto a spinning anisotropic screen to achieve an interactive 360 degree viewable light field display. By stacking up several LCD panels, Wetzstein et al. [25] designed a compact light field display device that shows good quality 3D images. However, their system suffers from brightness degradation when adding more LCD layers. There are projector-array-based light field displays theoretically similar to our system. Holografika [26] used a projector array to emit rays onto a large vertical diffuse screen to approximate a continuous light field. Two side mirrors are used to enlarge the field-of-view (FOV) to 70 degree. Lee et al. [4] arranged 300 projectors to create a 300-Mpixel multi-projection 3D display. They tested different types of projector arrangements and analyzed the luminance and ray distribution in detail. However, their system is very bulky, expensive and difficult to calibrate. Most importantly, projector arrays fail to provide a flexible trading between spatial and angular resolutions and high-density ray distribution due to physical limitation.

2.4. Vergence-accommodation conflict

Vergence and accommodation are two important occulomotor depth cues of 3D displays. Vergence is binocular and driven by retinal disparity while accommodation is monocular and driven by retinal blur [27]. A widely-accepted model describes these two depth cues as dual parallel feedback control systems interacting with each other via cross-links [28]. Conventional 3D displays, including stereoscopic projection systems, head-mounted devices and multiview displays, often unintentionally force the viewers’ eyes to focus on the screen regardless of the depths of the 3D objects being depicted, thus causing the VAC problem which may give rise to visual discomfort, fatigue and reduced visual clarity [29]. Although volumetric and holographic displays can provide natural focus cues, practical implementation of these displays has a large number of technical challenges.

To alleviate the VAC problem, researchers have proposed a technique called super multiview (SMV) which reduces the interval between viewpoints to be smaller than the pupil diameter, so that more than one ray passing through the same point in space enter the pupil simultaneously. Several promising SMV systems have been developed in the past [30–36]. However, these systems require numerous, densely-packed projectors or LCDs and complicated setups, thus are very difficult to put into practice. Moreover, it is hard to scale up the display specifications due to hardware constraints.

Similar to the SMV approach, our proposed light field 3D display can solve the VAC problem by facilitating correct accommodation of the eye. With proper design parameters, our light field 3D display can present a set of rays that is sufficiently dense so that more than one ray from the same 3D point to be depicted can be projected into the pupil simultaneously, thus the eye focuses on the depicted point instead of the display screen. The viewers are able to perceive accommodation depth cue and high fidelity three dimensionality without the VAC problem. We have constructed a 11 × 7 mirror array light field display and conducted a user study to demonstrated this ability in Section 7.

3. Display principle

The light field display system proposed in this work consists of a high-resolution video projector, a vertical anisotropic diffuser screen and an array of plane mirrors. We propose an ellipsoidal model to position all the system components and define the geometric shape of the mirror array.

Figure 1 illustrates the principle of the mirror array display. We use a video projector to emit light rays onto an array of plane mirrors. Each mirror reflects an image slice of the entire projection onto a common screen. The mirrors are tangent to an ellipsoid of revolution that has the projector and the screen center located at its two focal points respectively. Ellipsoid inherits an important geometric property of ellipse, that is the total length from one focal point to the other through any surface points is constant. Therefore, the total length of any light ray emitted from the projector reflected off the mirror array onto the screen, which is also the projection distance, is constant. This geometric arrangement can ensure uniform scale, size and sharpness of every reflected image slice.

 

Fig. 1 The proposed light field display uses a mirror array to create a dense array of virtual projectors. Each mirror is tangent to the ellipsoid of revolution that has a real projector and a screen located at its foci. Thus, all virtual projectors have the same distance from the screen center. The vertical anisotropic diffuser screen has small horizontal and large vertical angles of diffusion. The display’s angular density is multiplied by shifting different rows of the mirrors horizontally with a fixed angular interval.

Download Full Size | PDF

The mirror array setup can also be interpreted as an optical system that reproduces an array of virtual projectors that exist in the virtual space and have equal distance from the screen. All virtual projectors are distributed on the surface of the director sphere of the ellipsoid with the screen at its center. The number of virtual projectors is identical to the number of mirrors in the array. Each virtual projector is projecting a different image slice from a specific angle onto the screen through a window - the mirror that produces the virtual projector itself.

All image slices overlap on the screen that is made with a sheet of vertical anisotropic diffuser with small horizontal and large vertical angles of diffusion. The narrow horizontal scattering profile will maintain the divergence of different projector pixels to various horizontal directions.

The wide vertical scattering profile can deliver the same pixel to multiple heights so that 3D imagery can be seen regardless of viewer’s height. Vertical anisotropic diffusers are commercially available.

Each viewing position will receive a unique set of rays coming from many different virtual projectors and allow a mosaicked image to be observed only at that viewing angle. Thus, the display can achieve autostereoscopy by synthesizing a stereopair to the viewer’s left and right eyes and allowing viewer to perceive three-dimensionality through binocular parallax. When the viewer moves horizontally in front of the screen, a number of multi-perspective images are fused in the viewer’s brain which induces motion parallax and further enhances the 3D experience. Multiple viewers at distinct locations can perceive the depth cues simultaneously. Unlike multiview systems, our light field display does not have fixed viewing distance. When designed with proper parameters, our light field display can produce rays that are sufficiently dense to naturally evoke eye focus, so that the viewer can perceive accommodation depth cue and the VAC problem can be avoided.

4. System design

The above-mentioned ellipsoidal model provides a geometric framework. However, there exist infinitely many candidate ellipsoids. The display has numerous design parameters that are inextricably intertwined, as well as a set of realistic constraints due to fabrication and physical setup. We have extensively explored the design space. In this section, we systematically describe the suggested design strategies and procedures for our mirror array system.

4.1. Step 1: define preliminary target parameters

Without the loss of generality, we assume these system parameters are known:

We then decide the following target parameters of the display:

Based on the above parameters, the followings will be obtained.

The projection distance d_p, which is equivalent to the ellipsoid’s major axis length, can also be used as a design parameter to control the image size. The mirror numbers $m\times n$ play a key role in controlling the display properties. More mirrors will create more virtual projectors thus enhancing the display’s angular resolution and ray density in the expense of spatial resolution and image size.

4.2. Step 2: build the ellipsoidal model

In this step, we attempt to build the ellipsoidal model that can achieve the design targets. Figure 2 illustrates the intermediate geometry of our display system, in which the screen S, the projector P and the mirror array M lie on the same line, and the projector is pointing at the center of the mirror array. Suppose the projection frustum fully covers the mirror array.

The size of the mirror array will affect two key and opposing display specifications: display FOV and ray density. The display FOV increases with the mirror array size. Thus, if the design goal is to develop a light field display with wide viewing angle that does not need to provide accommodation cue, a larger mirror array should be made. On the contrary, reducing the mirror array size can enhance the ray density and angular pixel density, which are important for solving the VAC problem. We will discuss in Section 4.4 how to evaluate whether the selected set of parameters can yield suitable ray density distribution and angular pixel density to solve the VAC. The following paragraphs first assume accommodation is not needed and the design goal is to optimize the display FOV.

Since we fabricate the mirror array by generating the 3D model of mirror support structure and printing it out with 3D printer, the build envelop of the printer limits the overall size of the mirror array. In order to maximize the display FOV, the selected width of the mirror array m_x should be close to the fabrication limit. The distance between the projector and the mirror array, i.e. PM, can then be determined. Therefore, the screen is settled at point S and $PM+SM=d_p$. The viewing angle is bounded by the rightmost ray of the leftmost projector and the leftmost ray of the rightmost projector (red and blue solid lines in Fig. 2). The display FOV θ_d can be easily obtained from the geometric relationship.

 

Fig. 2 Top and side views of the system geometry.

Download Full Size | PDF

The intermediate model shown in Fig. 2(a) is a special case when the screen, projector and mirror lie on a straight line. However, apparently, the model cannot be directly used as some of the rays reflected off the mirrors are blocked by the real projector and cannot reach the screen. The projector has to be moved away from the reflected rays while maintaining the optimal distance from the mirror. As shown in Fig. 2(b), the projector is rotated about the mirror array center M to leave the light path. Note that the projector and the screen may collide if they are placed too close or when their physical sizes are too large. This should also be taken into consideration.

4.3. Step 3: design the mirror array

The mirror apparatus is a faceted ellipsoidal reflector composed of a number of small plane mirrors. Each mirror is tangent to the optimal ellipsoid obtained in Step 2 with the mirror center located at the point of tangency. The mirror boundary can be obtained by intersecting each tangent plan and the frustum of the corresponding virtual projector (a sub-frustum of the entire projection, as shown in Fig. 1.

To distribute the loss of spatial resolution into both the horizontal and vertical directions and to proliferate a large number of horizontal views, we align multiple rows of mirrors and offset each row horizontally with a uniform angular interval. This strategy is analogous to the technique of using slanted lenticular screen on multiview autostereoscopic LCD displays and the apparatus arrangements in some other multiview systems [4, 37, 38]. Due to the anisotropic diffusion property of the screen (large vertical diffusion profile), the vertical positions and directions of the virtual projectors are irrelevant to the perceived light field. As such, the mirror array arrangement is equivalent to populating all virtual projectors horizontally and evenly along a circular arc at a uniform angular spacing.

Note that, for simplicity and ease of calibration, only $n\left(m-1\right)+1$ full-size mirrors are used. In fact, the $2\left(n-1\right)$ incomplete mirrors due to the offsetting arrangement could also be used to further increase the usable pixels and display FOV.

4.4. Evaluate ray density and angular pixel density

The mirror array size selected in Section 4.2 maximizes the display FOV. However, if the rays are spread over a wide angle, the eye cannot receive more than one ray depicting the same space point simultaneously. The display would then fail to provide focus cue, thus causing the VAC problem. It is possible to enhance the accommodation-evoking capability of the display by trading some of the parameters, such as the display FOV, image size and spatial resolution. Reducing the display FOV can be done by selecting a smaller mirror array. Reducing the spatial resolution can be done by increasing the mirror number.

We formulate the ray density and angular pixel density to quantitatively evaluate whether the light field generated by the candidate display can evoke accommodation response. The ray density is a scalar field representing the number of rays passing through a unit area in the viewing space, while the angular pixel density is a scalar value representing the average number of pixels projected on the screen per unit length (mm) of the screen and within 1° angular interval.

As shown in Fig. 3(a), the virtual projectors are populated on a circular arc with radius equal to the projection distance d_p. Using the polar coordinate system with the origin located at the screen center, the polar angle of virtual projector P_i can be represented as ${\gamma }_i=\frac{\pi -\beta }{2}+\frac{i\beta }{n\left(m-1\right)}$, where $i=0,1,\mathrm{...},n\left(m-1\right)+1$ and β is the angular interval between the two outermost virtual projectors, which can be obtained from the system geometry. Point $\left(x,y\right)$ is an arbitrary point in the viewing area (in Cartesian coordinate system). If we connect this point $\left(x,y\right)$ to both end points of the screen, the lines can be extended to intersect the circular arc at points A $\left({\gamma }_a,d_p\right)$ and B $\left({\gamma }_b,d_p\right)$. Then, we can sort out a subset of virtual projectors $\left\{P_i\right\}$ within AB that can deliver rays to point $\left(x,y\right)$, i.e. $\left\{P_i\right|{\gamma }_b<{\gamma }_i<{\gamma }_a\}$. Consequently, the ray density at point $\left(x,y\right)$ can be computed by adding the ray/pixel dose per unit distance at point $\left(x,y\right)$ by all $\left\{P_i\right|{\gamma }_b<{\gamma }_i<{\gamma }_a\}$, which is formulated based on similar triangles as follows:

The ray density distribution of our three prototype systems are visualized in Fig. 7.

 

Fig. 3 Illustrations for computation of ray density.

Download Full Size | PDF

Next, we formulate the average angular pixel density APD:

We have implemented the proposed methodologies using parametric modelling technique. We developed a software toolkit in 3D modelling software Rhinoceros and its plugin Grasshopper, as well as Python scripts. The toolkit allows users to specify the target parameters, then automatically computes the remaining parameters based on the user input and generates the 3D models of all system components. Users can visualize and verify the system geometry interactively. Our parametric modelling tool can automatically generate the necessary computer files for digital fabrication, including the 3D model for printing the support structure of the mirror array, as well as the 2D vector files for cutting the optical mirrors with CNC. The mirrors are then attached to the 3D printed structure. The fabricated mirror arrays are shown in Fig. 6.

 

Fig. 4 The light field rendering procedure.

Download Full Size | PDF

5. Light field rendering

We render the image for each virtual projector based on the light field rendering technique proposed by Levoy et al. [15]. It is an image-based light field rendering technique. The advantage of using image-based rendering technique is that the image acquisition process is made flexible as it can be done by capturing images of either real objects or computer models. As our proposed system is a HPO display, the perspective views are created in horizontal direction only. The optical geometry shown in Fig. 4(a) illustrates our light field rendering process. The screen is on the XY-plane and the viewpoints/virtual cameras are on the ST-plane. The projection distance and viewing distance are denoted by d_p and d_v respectively. The width of the viewing zone is computed based on the chosen value of d_v. Suppose the viewpoints/virtual cameras are evenly distributed along the S-axis within the viewing zone. The captured view of each virtual camera C_j is denoted by $I_c{ }^j$, where the total number of virtual cameras should be made sufficient to ensure the quality of the projection images. The projection image of each virtual projector P_i is denoted by $I_p{ }^i$ for i = 1 to $n\left(m-1\right)+1$. Each column of pixels $I_p{ }^i\left(x\right)$ for virtual projector P_i is rendered by tracing the pixels captured from the corresponding view on the ST-plane. Then, all pixel columns $I_p{ }^i\left(x\right)$ are mosaicked together to compose the projection image $I_p{ }^i$. As shown in Fig. 4(a), $I_p{ }^i\left(x\right)$ (red line) intersects the screen and the viewing plane at x and s respectively. A new camera view $I_c{ }^{{ }^{\text{'}}}$ (dot lines) of camera $C^{\text{'}}$ at point s is synthesized by interpolating the captured views of the neighboring cameras. As all $I_p{ }^i$ and $I_c{ }^j$ cover the entire screen, $I_p{ }^i\left(x\right)$ can be mapped from $I_c{ }^{{ }^{\text{'}}}$ directly. Repeating this process for all columns of pixels and for all virtual projectors, the required light field is generated. Our rendering method can be easily extended to the case when the viewpoints/virtual cameras are distributed on a circular curve.

As shown in Fig. 4(b), the ray distribution of our light field display is similar to that of an integral imaging display system. The ray distribution can also be represented in $\left(X,S\right)$ coordinates (Fig. 4(c)), where each column of pixels is represented as a dot. We can see that the rays distribute continuously on the viewing plane (S-axis). Integral imaging rendering, including the technique reported in [40], can also be used to synthesize the projection images for our system.

6. Calibration

Despite the high precision in digital fabrication process, gluing of individual mirrors may induce geometric misalignments which would distort the image slices reflected on the screen and affect the display performance. Unlike projector array, our mirror array system does not involve massive radiometric variations as only a single projector is used. Thus, only geometric calibration is necessary and radiometric calibration is not needed.

 

Fig. 5 Calibration workflow.

Download Full Size | PDF

We have developed a fully automatic calibration method to correct the geometric deviations due to mirror misalignments. As a number of mirrors are involved, we use an uncalibrated camera in the loop to automate the calibration process. The camera is facing to the rear side of the vertical diffuser screen as shown in Fig. 6(a). A 2D rectification process similar to the projector calibration method described in [10] is applied. A printed 4 × 4 chessboard is attached to the screen and a reference photo is taken for benchmarking. The purpose of the rectification process is to ensure all projected image slices will coincide with the printed chessboard after calibration.

We then place a diffuse surface on the screen and project the same chessboard pattern as an sub-image of the entire projection onto each of the mirrors sequentially. The camera captures a set of distorted chessboard patterns reflected off the mirrors on the diffuse screen.

The key of the calibration process is the homography between each image slice (sub-image of the entire projection) and the corresponding photo capturing the reflected image on the screen. As shown in Fig. 5(a), the homography matrix between the calibration photo plane F_i and the sub-image plane S_i is calculated using the correspondence between the detected corner points on both images. As the homography between the reference photo (capturing the printed chessboard pattern) F_r and the projection sub-image S_i is identical to the homography between F_i and S_i, the calibrated sub-image $S_i{ }^{{ }^{\text{'}}}$ can be easily obtained by multiplying F_r and H_i (Fig. 5(b)). We yield the final calibrated projection image by adding all the calibrated sub-images $S_i{ }^{{ }^{\text{'}}}$ for $i=1,2,3,\mathrm{...},n\left(m-1\right)+1$, as shown in Fig. 5(c). The calibration result demonstrates high geometric accuracy as shown in Fig. 5(d).

7. Results

We have implemented three display systems of different specifications to accommodate different requirements. The simulated geometric models and the actual experimental setups are shown in Fig. 6. A high-resolution projector with 4K UHD $\left(3840\times 2160\right)$ resolution and 40° horizontal FOV is used. We have fabricated a custom projector stand with high payload that can support a heavy projector tilting up to 60°. The vertical anisotropic screen is 300 × 300 mm and is made with the elliptical light shaping diffuser by Luminit with 1° horizontal and 60° vertical diffusion profile. The support structures for mirrors are printed using SLA 3D printers with 0.1 mm precision. Besides the 3D models of support structures, our modelling software also generates the 2D vector files for cutting the optical mirrors with CNC machine.

 

Fig. 6 Simulated and real system setups.

Download Full Size | PDF

The three fabricated prototypes are of different mirror numbers: 5 × 3, 7 × 5 and 11 × 7. When designing the 5 × 3 system, we intended to use a small number of mirrors and a 3D printer of build capacity 400 × 400 mm. We set the target image width I_x to around 300 mm and horizontal spatial resolution $w=768$ pixels. Owing to the limited size of the mirror array, the display FOV yielded by the 5 × 3 mirror array is ${17.7}^{\circ }$. When designing the 7 × 5 system, we used another 3D printer of larger build capacity 800 × 800 mm in order to attain a larger display FOV. We set the target image width I_x = 260 mm and horizontal spatial resolution $w=548$ pixels. The display FOV yielded by the $7s5$ is ${33.7}^{\circ }$. The first two displays do not generate high ray density light fields as they are not intended to provide accommodation depth cue. When designing the 11 × 7 system, we intended to present high-density light rays to demonstrate the capacity of our mirror array light field display technique in solving the VAC problem. The high ray density is achieved at the expense of image size, resolution and FOV. The resultant image width, horizontal resolution and horizontal FOV of the 11 × 7 system are 156 mm, 349 pixels and ${13.4}^{\circ }$ respectively. The system parameters of the three implemented displays are summarized in Table 1.

Table 1. System Parameters of the 5×3, 7×5 and 11×7 Mirror Array Displays.

View Table | View all tables in this article

To evaluate the ray distribution of the three different configurations, we compute and visualize the ray density at different viewing positions based on Eq. (1). As shown in Fig. 7, the width of viewing zone increases with the viewing distance, but the ray density will decrease. Comparing among the three configurations, the 5 × 3 mirror array has the most uniform distribution of light rays. The 7 × 5 mirror array has relative uniform ray distribution when the viewing distance is larger than 800 mm, and has the widest viewing zone. Despite the 11 × 7 mirror array yields a narrow viewing zone, it achieves the highest ray density which can overcome the VAC problem. The ray distribution maps provide a good reference for researchers to decide the system parameters to fulfill their specific needs.

 

Fig. 7 Ray density distribution of the three light field displays implemented.

Download Full Size | PDF

We show the display results of the three setups in ^{Fig. 8, Fig. 9} and Fig. 10. For the 5 × 3 and 7 × 5 mirror array setups, the same set of examples are used. To demonstrate the flexibility in image acquisition of our rendering method, both computer models and real objects are used. The first three examples (Stanford bunny, Asian dragon and dice) are computer models captured in software Blender, while the last example is a set of real 3D objects (toy models consisting a deer, a rabbit and plants) captured with a smartphone camera while being placed on a rotation platform.

 

Fig. 8 Four 3D examples viewed from the left to the right of the 5×3 mirror array display. Three examples are captured from computer models and the last one is captured from a set of real objects.

Download Full Size | PDF

 

Fig. 9 Four 3D examples viewed from the left to the right of the 7×5 mirror array display. Three examples are captured from computer models and the last one is captured from a set of real objects.

Download Full Size | PDF

In Fig. 8 and Fig. 9, the images on the three right columns are captured from the left, front and right viewing directions of the display screen, and the corresponding ground truth (GT) views are also shown on the left three columns. Readers may refer to the supplementary videos ( Visualization 1 and Visualization 2) for the continuous view of the 3D examples displayed on both systems.

We show that the 11 × 7 mirror array system is able to arouse focus cue and solve the VAC problem. As illustrated in Fig. 10(a), when the ray density of the light field is sufficiently high, multiple rays depicting the same point in space enters the pupil simultaneously. In order to obtain a sharp image, the eye is evoked to focus at the depicted point rather than the screen surface. The vergence and accommodation are stimulated in a consistent manner. To demonstrate our 11 × 7 system is capable of stimulating focus cue, we captured the display image with a Lytro Illum light field camera. The results are shown in Fig. 10(b), where we refocused the light field photos at different depths. As shown in Visualization 3, our rendering method is capable of creating a correct light field of the 8 vertical lines at different depth positions.

 

Fig. 10 Presenting accommodation depth cue with our mirror array light field system: (a) The display schematic shows the high-density light field can solve the VAC by allowing two or more rays depicting the same 3D point enters the eye simultaneously, thus naturally evoking the eye’s lens to focus at the depicted depth rather than the screen surface. (b - dragon and light stripes) A light field photo of each model displayed on the 11×7 system is captured with the Lytro Illum light field camera. The depth features can be observed by refocusing. This demonstrates the display can successfully present focus cue.

Download Full Size | PDF

In addition, a user study is conducted to further evaluate the performance of the 11 × 7 mirror array light field display. Twelve volunteers (age range $18\sim 40$, 2 males) participated in the user study. The experimental setting is shown in Fig. 11(a). The display screen was placed 1 D (1 m) from the viewers. A Grand Seiko WAM-5500 auto refractor with open field was use to measure the accommodation response of the viewers’ eyes dynamically. The users visually tracked a target oscillating step wisely while their accommodation responses were measured. The visual target was a sphere with red/white chessboard texture oscillating between 0.9 D (110 cm) and 1.1 D (90 cm) at 1/12 Hz for 4 cycles. The sphere stayed at 0.9 D, 1 D and 1.1 D each for 3 seconds. Orthogonal projection was used for rendering the sphere to eliminate the size cue. Each participant were asked to repeat the task for 5 times and the average accommodation response was computed. As shown in Fig. 11(b), the individual responses of all users (black lines) are averaged and smoothed with Gaussian to obtain the average response (blue line), which shows a good coherence to the stimulus (red line). This demonstrates the ability of our 11 × 7 display system in stimulating accommodative responses thus providing focus cue.

 

Fig. 11 User study for accommodation response to the visual stimulus presented by our 11×7 mirror array display. (a) An autorefractor was use to measure the accommodation response of the users’ eyes dynamically. (b) The individual (black) and average (blue) accommodative responses to the stepwisely oscillating visual stimulus (red) are shown. The average response shows a degree of coherence with the stimulus.

Download Full Size | PDF

These results have verified that our mirror array technique and ellipsoidal geometry can successfully create light field 3D displays using only a single projector. The display performance of the three prototypes have verified our design pipeline, rendering and calibration methods. Mirror array approach allows flexible trading among different parameters to accommodate different needs, including solving the challenging VAC problem, and provides a truly practical solution to realize projection-based light field display.

8. Discussion

8.1. Comparison of system specifications with projector arrays

We compare our system specifications with other existing light field projector array systems in Table 2. Despite using a single projector, our displays demonstrate appealing quality. The fundamental problem of projector array displays lies in the fact that the number of views is controlled by the number of projectors. The system cost, complexity, bulky size and laborious calibration make projector arrays difficult to be reproduced in practice. Our displays use only one projector and one moderate computer. No special graphic cards, video splitters, structured cabling system or cooling fans are needed. Mirrors further help reduce the physical size of our display systems as the projector rays are reflected.

Table 2. Comparison of Our System Specifications with Other Light Field Projector Arrays.

View Table | View all tables in this article

The image size yielded by our mirror array systems is comparable to that of the projector arrays in [6] and [10] (in the scale of $10\sim 30$ cm). The 360° display in [6] is composed of 12 modules. Each module consists of 24 projectors and provides 30° horizontal FOV, which is similar to our 7 × 5 system which uses only one projector. Similarly, if we divide the projector array in [10] into 4 modules, each 18-projector module gives around 30° FOV, which is again similar to our 7 × 5 mirror array.

The projector arrays proposed in [2] and [4] produce images of very large format by trading the pixel density. The image size of our displays is limited by two factors: projector FOV and mirror array size. Thus, in order to scale up the image size, we may increase either the projector FOV or the mirror array size. The former can effectively increase the virtual projector’s FOV thus shortening the projection distance required by a larger image (short throw), where the latter permits a longer projection distance thus producing a larger image.

Our 11 × 7 system yields the highest angular pixel density. Although Yoshida et al. [6] has a high pixel density, the pixels are diverging to cover a 360° view. Lee et al. [4] ranks the second in angular pixel density. However, as its viewing distance is long and the ray density decreases with the viewing distance, the viewers cannot receive high-density angular rays at the viewing positions. Thus, these systems fail to produce high-density angular rays that can evoke the lens focus. Only our 11 × 7 system can reduce the VAC problem.

8.2. Vertical FOV

Vertical FOV refers to the vertical range of the viewable angle of the display. It is affected by the vertical diffusion angle of the screen material as well as the vertical angular divergence of all the transmitted rays. The maximum vertical diffusion angle of commercial anisotropic diffuser is typically around 60°. If the incident rays reach the screen at drastically different vertical angles, it will limit the vertical viewing zone of the display. The vertical FOV varies with the focal distance of the ellipsoid and the projector tilt angle. Therefore, during the system design process, we should ensure the vertical FOV is wide enough to accommodate viewers of different height.

8.3. Limitations and future work

As we fabricate the mirror arrays by manually gluing individual mirrors on the 3D printed support structure, there is noticeable misalignment among the reflected images. Despite the calibration process can correct the distortions and misalignments, the number of effective pixels is reduced. To remedy this issue, the faceted ellipsoidal reflector can be produced by customization service from professional optics suppliers.

Similar to existing autostereoscopic LCD displays, the proposed mirror array is based on spatial multiplexing approach to produce multiviews. It shares the disadvantage of spatial multiplexing systems, that is the trade-off between spatial and angular resolution. Even with a 4K projector, the pixel count is limited. Therefore, it is impractical to create a mirror array display with a very large image size, high angular and spatial resolutions and wide FOV using only a single projector. Mirror array and multi-projector approaches can be combined to enhance the image size, spatial and angular resolution as well as FOV of the display. Rather than adding a massive number of projectors, a few high-resolution projectors may be used to guarantee sufficient number of pixels.

9. Conclusions

The continuous growth of pixel counts and price reduction of high-resolution video projectors offer a great opportunity for achieving light field display by dividing the projector resolution into angular views. Mirrors suggest a simple and effective way to accomplish the task. In addition, mirrors can easily reproduce densely arranged virtual projectors regardless of the physical size of the real projector, thus producing a display of high ray density that is unachievable by existing projector arrays. This work provides a systematic solution to tackle the key challenge of mirror array display which is to properly design the mirror apparatus that can accurately redirect light from the projector while maintaining the angular divergence of different pixels to various views. Three different mirror array setups were designed to demonstrate the capability and flexibility of proposed methodologies. Most importantly, the high-density light field produced by our mirror array approach can successfully alleviate thecritical and challenging VAC problem of 3D displays.