1. Introduction

Stereoscopic displays have found numerous applications, spanning the fields of scientific visualization, training and simulation, cinema, and home entertainment systems. Although conventional stereoscopic displays can render pictorial and binocular disparity cues fairly well and thus create compelling depth perceptions, they suffer the drawbacks of vergence accommodation conflict [1,2], distorted depth perception [3], and visual fatigue [4]. Several alternative technologies may potentially overcome these drawbacks, including holographic displays [5], volumetric displays [6–8], and multi-focal plane displays [9–16]. Among those technologies, multi-focal plane three-dimensional (3-D) displays leverage computational power, accuracy on depth perception, and conventional two-dimensional (2-D) display technologies.

Different from conventional stereoscopic displays where stereoscopic image pairs are rendered on 2-D flat surfaces at a single fixed focal distance from the eye, a multi-focal plane display consists of a series of carefully placed focal planes at discrete focal distances. These discrete focal planes sample a 3-D scene volume into multiple zones along the visual axis. Objects contained within a zone are rendered on a corresponding pair of adjacent focal planes. Multi-focal plane displays may be implemented either by spatially multiplexing a stack of 2-D displays [9–12] in the visual space or by fast switching the focal distance of a single 2-D display in synchronization with the frame rendering of multi-focal images [13–16].

Although the emergence of multi-focal plane displays have demonstrated promising potentials for reducing the visual artifacts to various extents [2,3,10–12,15,16], there have been relatively few systematic investigations on the optical design and image quality assessment of such displays. There are two fundamental issues that need to be addressed—(1) the optimal dioptric spacing between adjacent focal planes to produce acceptable image quality and depth resolution, which essentially determines the number of focal planes necessary for creating a multi-focal plane display to cover a given volumetric space; and (2) a method to render correct or near-correct focus cues for a continuous scene volume through a finite number of sparsely-spaced focal planes without creating noticeable depth aliasing effect and depth discontinuity. Several pioneering work addressed some of these issues from different aspects. For instance, in a theoretical work Rolland et al. suggested that the spacing between adjacent focal planes should be a constant dioptric distance of 1/7 diopter (D) and 14 focal planes are needed to cover the depth range from 0D to 2D [9]; Akeley et al. demonstrated a three-focal plane display prototype by optically combining three 2-D displays through beamsplitters which creates a constant dioptric separation of 0.67D for adjacent focal planes [10]; Liu and Hua presented a dual-focal plane display prototype with addressable focal distances throughout a volumetric space from 8D to 0D enabled by a liquid lens device [15]; Love et al. recently developed a four-focal plane prototype with discretely addressable focal planes enabled by birefringence lenses [16]; and Lee et al. demonstrated an immaterial two-focal plane display with the two image planes configured in an L-shape [12].

A large number of focal planes and small dioptric spacing are desirable for improving image quality and reducing perceptual effects in multi-focal plane displays. On the other hand, it is practically very challenging to achieve a large number of focal planes with the affordance of current technologies. In order to reduce the number of necessary focal planes to a manageable level, a depth-weighted blending technique may be implemented. Such method leads to a depth-fused 3-D (DFD) perception [11], where two overlapped images displayed on two transparent screens at two different depths may be perceived as a single-depth image. In addition, the luminance ratio between the two images may be modulated to change the perceived depth of the fused image. Hereafter multi-focal plane technologies that incorporate the DFD effect are referred to as depth-fused multi-focal plane displays, or DFD displays.

Due to the lack of a systematic quality metrics, in existing DFD displays, the choice of the spacing between adjacent focal planes differ significantly, ranging from 0.028D [11] to 0.67D [10]. Furthermore, a linear form of depth-weighted fusing function was mainly adopted in existing DFD prototypes [2,10,11,16], with few considerations of optimizing the function form to account for key ocular factors of the human visual system (HVS). Hoffman et al. [2] investigated the retinal image quality in a three-focal plane DFD display prototype. The contrast modulation of the retinal images was found to be affected by pupil size, ocular aberrations, and target spatial frequency.

In this paper, we present a systematic approach to address the two aforementioned fundamental issues in designing a depth-fused multi-focal plane display. Our method is based on quantitative evaluation of the modulation transfer functions (MTF) of the DFD images formed on the retina. It takes into account most of the ocular factors, such as pupil size, monochromatic and chromatic aberrations, diffraction, Stiles-Crawford effect (SCE) [17], and accommodation, and display factors, such as dioptric midpoint, dioptric spacing, depth filter, and spatial frequency of the target. Based on the MTFs of the retinal images of a DFD display and the depth of field (DOF) of the human visual system under photopic viewing conditions, we determine the optimal arrangement of the focal planes and characterize the depth-weighted fusing function between adjacent focal planes. In Section 2, we describe the generalized DFD display method. In Section 3, the simulation models and methods to predict the retinal image quality, or MTF, of a DFD display is described. In Section 4, we present the simulation results that investigated the effects of various display factors on retinal image quality and suggest guidelines for optimizing a DFD display design. Finally in Section 5, we simulate retinal images and MTFs of DFD displays using our proposed methods and compare to those generated by the prior methods.

2. Generalized DFD display method

Figure 1 illustrates the depth fusion concept of two images displayed on two adjacent focal planes separated by a dioptric distance of Δz. The dioptric distance from the eye to the front focal plane is z ₁ and to the rear plane is z ₂. When the images shown on the two-layer displays are aligned such that each pixel on the front and rear planes subtends the same visual angle to the eye, the front and back pixels (e.g. A and B, respectively) are then viewed as completely overlapped at the viewpoint and fused as a single pixel (e.g. C). The luminance of the fused pixel (L) is summed from the front and rear pixels (L ₁ and L ₂, respectively), and the luminance distribution between the front and back pixels is weighted by the rendered depth z of the fused pixel. These relationships may be expressed as:

 

Fig. 1 Schematic model of a depth fused dual-focal plane display. Pixels on the front (A) and back (B) focal planes are located at z ₁ and z ₂ respectively from the eye, and the fused pixel (C) is located at z (z ₂<z<z ₁). All distances are defined in dioptric units.

Download Full Size | PDF

Depth-fused 3-D perception effect indicates [10,11], as the depth-weighted fusing functions (w ₁ and w ₂) change, the perceived depth $\widehat{z}$ of the fused pixel will change accordingly, formulated as:

For instance, when w ₁(z) = 1, the perceived depth should be z ₁, and vice versa is z ₂ when w ₂(z) = 1.

In a generalized n-focal plane DFD system, the dioptric distances from the eye to the n-focal planes are denoted as z ₁, z ₂, ….z _n in distance order where z ₁ is the closest one to the eye. We assume that the 3-D scenes contained between a pair of adjacent focal planes are rendered only on this corresponding focal plane pair. Under this assumption, a given focal plane at z _i will render all the 3-D scenes contained between the (i-1)^th and the (i + 1)^th focal planes. Within the depth range of z _i-1≥z≥z _{i + 1}, many scene points may be projected onto the same pixel of the i ^th focal plane, among which only the closest scene point to the eye is un-occluded and thus effectively determines the depth-weighted fusing function modulating the luminance of the specific pixel. The closest scene point corresponding to a specific pixel can typically be retrieved from the z-buffer in computer graphics renderer [18]. Let us assume the depth of the closest 3-D scene point projected onto a given pixel of the i ^th focal plane is z. Based on the depth-fused 3-D perception described above, the luminance of the 3-D point is distributed between the (i-1)^th and i ^th focal planes if z _i-1≥z≥z _i, otherwise between the i ^th and (i + 1)^th focal planes if z _i≥z≥z _{i + 1}. The luminance attribution to the i ^th focal plane is weighted by the depth z. It may be characterized by the ratio of the luminance attribution L _i(z) on the i^th focal plane at z _i to that of the total scene luminance L(z), written as g _i(z) = L _i(z)/L(z), where L(z) = L _i-1(z) + L _i(z) if z _i-1≥z≥z _i or L(z) = L _i(z) + L _{i + 1}(z) if z _i≥z≥z _{i + 1}. In general, the depth-weighted fusing function, w _i(z), of the i ^th focal plane can be defined as:

In summary, by knowing the rendered depth z of a 3-D virtual scene, the luminance levels of the multi-focal plane images can be modulated accordingly by the depth-weighted fusing functions in Eq. (3) to render pseudo-correct focus cues.

3. Models and methods for DFD quality assessment

In DFD displays, the adjacent focal planes are separated in space at a considerable distance. The retinal image quality apparently is expected to become worse when the eye is accommodated at a distance in between than when focusing on either the front or back focal planes. However, both the dioptric spacing between adjacent focal planes and the depth-weighted fusing functions can be selected such that the perceived depth of the fused pixel $\widehat{z}$ closely matches with the rendered depth z and the image quality degradation is minimally perceptible as the observer accommodates to different distances between the focal planes.

The optical quality of a fused pixel in DFD displays may be quantitatively measured by the point spread function (PSF) of the retinal image, or equivalently by the modulation transfer function (MTF) which is characterized by the ratio of the contrast modulation of the retinal image to that of a sinusoidal object on the 3-D display. Without loss of generality, hereafter a dual-focal plane display is assumed and the results can be extended to n-focal planes. Based on Eq. (1), when the eye is accommodated at the rendered distance z, the PSF of the fused pixel, PSF ₁₂, may be described as:

Many factors may affect the retinal image quality — PSF ₁₂ and MTF ₁₂ — of a DFD display. Table 1 categorizes those parameters, along with their notation and typical range, into two types: ocular and display factors. Ocular factors are mostly related to the human visual system when viewing DFD images from a viewer’s perspective. These variables, including pupil size, pupil apodization, reference wavelength, and accommodation state, need to be carefully considered when we model the eye optics. Display factors are related to the practical configurations of DFD displays, such as the covered depth range, dioptric midpoint of two adjacent focal planes to the eye, dioptric spacing between two adjacent focal planes, depth-weighted fusing functions, as well as the spatial frequency of the displayed target.

Table 1. Parametric Dependence of Retinal Image Quality for n-Focal Plane DFD Displays

View Table | View all tables in this article

Instead of using observer- and display-specific measurements to evaluate the PSF and MTF of DFD displays [2,3,16], we adopt a schematic Arizona eye model to simulate and analyze the retinal image quality from simulated targets to derive generalizable results. In the fields of optical design and ophthalmology, various schematic eye models have been widely used to predict the performance of an optical system involved with human observers [19–21]. In this study, the Arizona eye model was set up in CODE V [22]. The Arizona eye model is designed to match clinical levels of aberration, both on- and off-axis fields, and can accommodate to different distances. Detailed parameters of the model can be found in [19,23]. The accommodative distance z, as shown in Fig. 1, determines the lens shape, conic constant and refractive index of the surfaces in the schematic eye. The distances of the front and back focal planes, z ₁ and z ₂ respectively, and their spacing z are varied to simulate different display configurations.

Ocular characteristics of the HVS, such as depth of field, pupil size, diffraction, Stiles-Crawford effect, monochromatic and chromatic aberrations, and accommodation, play important roles on the perceived image quality of DFD displays. Although previous literatures [2] have investigated the image quality dependence upon pupil size, high-order aberration, and accommodation, the treatment to the aforementioned factors lack generality to average observers and to a full-color DFD display with different display configurations. For instance, only monochromatic aberrations specific to one user’s eye were considered and a linear depth-weighted fusing function was assumed.

In order to accurately simulate PSF/MTF of the retinal images in DFD displays, we firstly examined the dependence of the polychromatic MTF of a fused pixel upon eye pupil diameter while fixing other ocular and display factors. Particularly, we examined the MTFs under the condition that the luminance of a rendered pixel is equally distributed between the front and back focal planes separated by 0.5D and the eye is accommodated at the midpoint between the two focal planes. The midpoint is generally expected to have the worst retinal image quality for a fused pixel. Assuming the same pupil size, we further compared the MTFs of the fused pixel against that of a real pixel that is physically placed at the dioptric midpoint of the two focal planes. For pupil diameters no larger than 4mm, we found the MTF differences of the fused pixel from a real pixel at the same distance is acceptable for spatial frequencies below 20cpd while a considerable degradation is observed for larger pupils. Therefore, we set the eye pupil diameter of the eye model to be 4mm, which in fact corresponds well to the pupil size when viewing typical HMD-like displays. Secondly, in order to account for the directional sensitivity of photoreceptors on the human retina, which commonly refers to the Stiles-Crawford effect (SCE) [17], a Gaussian apodization filter was applied to the entrance pupil with an amplitude transmittance coefficient of β = −0.116mm ⁻² [23,24]. Consequently, SCE may induce a slightly contracted effective pupil, and thus reduce spherical aberration and improve MTF. Furthermore, the image source in the model was set up with polychromatic wavelengths, including F, d, and C components as listed in Table 1, to simulate a full-color DFD display. To compensate the longitudinal chromatic aberration (LCA) which commonly exists in human eyes [23,25], we inserted a zero optical power achromat at 15mm away from the cornea vertex with the LCA opposite to the Arizona eye model. Similar approach had been used in subjective measurement of DOF of the human eye [26]. In practical DFD designs, instead of inserting an achromat directly in front of the eye, the display optics may be optimized to have equivalent chromatic aberration to compensate the LCA of the visual system. Finally, the effect of diffraction is accounted in the modeling software—CODE V while simulating PSFs, and the effect of accommodation will be discussed with the depth filters in the next section.

Based on the model setup described above, for a given eye accommodation status and display settings, PSF ₁(z,z ₁) and PSF ₂(z,z ₂) for an on-axis point source are simulated separately in CODE V. Using the relationship in Eq. (3), a series of PSF ₁₂(z) are computed by varying w ₁ from 1 to 0, which corresponds to varying the rendered depth z from z ₁ to z ₂. The corresponding MTF ₁₂(z) of the DFD display is derived by taking the FT of PSF ₁₂.

To evaluate the retinal image quality of a depth-fused pixel against a physical pixel placed at the same distance, we further simulate the PSF of a real point source placed at distance z, PSF _ideal(z), and compute the corresponding MTF _ideal(z). The degradation of MTF ₁₂(z) from MTF _ideal (z) is expected to vary with the dioptric spacing of the two adjacent focal planes, rendered depth z, as well as eye-specific parameters. Through comprehensive analysis of the retinal image quality of DFD displays, threshold values are established to ensure the degradation from a real display condition is minimally perceptible to average observers. Optimal depth-weighted fusing functions are then obtained.

4. Optimal DFD designs

As mentioned earlier, a fused pixel that is rendered to be at the dioptric midpoint of two adjacent focal planes is expected to have the worst retinal image quality compared with other points between the focal planes [2,16]. Therefore, in the analysis hereafter, we use the retinal image quality of a fused pixel rendered at the midpoint of two adjacent focal planes as a criterion to determine appropriate settings for display designs.

4.1 Optimal dioptric spacing

In this study, the overall focal range of a DFD display covers the depth varying from 3D (z ₁) to 0D (z _n). Within this range, we further assume a constant dioptric spacing between two adjacent focal planes (e.g. z _i and z _{i + 1}) independent of the dioptric midpoint of the focal plane pair relative to the eye noted as z _{i,i + 1} = (z _i + z _{i + 1})/2 in Table 1. Using the simulation method described in Section 3, we validated this assumption by examining the dependence of the MTF of a fused pixel at the midpoint of two focal planes upon the dioptic distance of the midpoint to the eye whiling fixing other ocular and display factors (i.e. w ₁ = w ₂ = 0.5, Δz = 0.5D, z = z _{i,i + 1}). As expected the MTF of a fused pixel at the midpoint varies as the midpoint gets closer to the eye due to the fact that the ocular aberrations are highly correlated to accommodation [27,28]. However, the average variation is less than 15% for spatial frequencies below 20cpd for z _{i,i + 1} within the 0D~3D range.

Under these assumptions, it is feasible to study the effect of dioptric spacing on DFD displays by setting the midpoint of a pair of adjacent focal planes at an arbitrary position within the depth range without loss of generality. We thus chose 1D as the midpoint of a focal plane pair and varied their dioptric spacing Δz from 0.2D to 1D at an interval of 0.2D. For each dioptric spacing condition, the MTF of a fused pixel at the dioptric midpoint (i.e. MTF ₁₂ (z = z _{i,i + 1})) of the two focal planes was calculated with the assumption that the luminance level is evenly divided between front and back focal planes. Figure 2 (a) plots the results corresponding to different dioptric spacings. For comparison, on the same figure we also plotted MTF _ideal, which corresponds to the MTF of a real pixel placed at the midpoint, and the MTF _+0.3D and MTF _-0.3D, which correspond to the MTF of the eye model with + 0.3D and −0.3D defocus from the midpoint focus, respectively. It is worth noting that ± 0.3D defocus was chosen to match the commonly accepted DOF of the human eye [25,26,29]. As expected, MTF ₁₂ consistently degrades with the increase of the spacing of the focal planes. However, when Δz is no larger than 0.6D, MTF ₁₂ falls within the region enclosed by MTF _ideal (green dashed line) and the ± 0.3D defocused MTFs (the overlapped blue and red dashed lines). The results indicate that the DOF of the human eye under photopic viewing conditions can be selected as the threshold value of the dioptric spacing in multi-focal plane display designs, which ensures the degradation of the retinal image quality of a DFD display from an ideal display condition is minimally perceptible to average observers. If better retinal image quality is required for certain applications, a smaller Δz may be used but at the expenses of adding more focal planes. For instance, if Δz = 0.6D is selected, 6 focal planes would be sufficient to cover the depth range from 3.0D to 0D, while 9 focal planes would be necessary to cover the same range if Δz = 0.4D is selected. At the medium spatial frequency of 20cpd, the improvement of MTF ₁₂ at the dioptric midpoint from 0.1 to 0.4 may be worth of the cost for demanding applications.

 

Fig. 2 (a) Modulation transfer functions of a depth-fused dual-focal plane display as a function of dioptric spacings of 0.2D (green circle markers), 0.4D (square), 0.6D (star), 0.8D (diamond), and 1.0D (triangle). (b) Modulation transfer functions as a function of accommodations with z = 1.3, 1.2, 1.1, 1.0, 0.9, 0.8, 0.7D. Medial focal plane is setup at 1D and the luminance ratio is L ₁/L = 0.5. MTF of the ideal viewing condition is plotted as a light green dashed line, and defocused MTFs are plotted as red ( + 0.3D) and blue (−0.3D) dashed lines respectively in (a).

Download Full Size | PDF

4.2 Optimal depth-weighted fusing function

By setting a dioptric spacing of Δz = 0.6D and a dioptric midpoint of z ₁₂ = 1D from the eye, we further examined the MTF of a fused pixel while incrementally varying the eye accommodation distance from the front focal plane (z ₁ = 1.3D) to the back focal plane (z ₂ = 0.7D) at an increment of 0.1D, as shown in Fig. 2 (b). As expected, an accommodation distance at the dioptric midpoint (z = z ₁₂ = 1D) would maximize the MTF of the fused pixel, while shifting the accommodation distance toward either front or back focal planes will always decrease the MTF. For instance, the MTF values for a target spatial frequency of 10cpd is reduced from 0.6 when z = 1D to nearly 0.when z = 1.3D or z = 0.7D. Many studies have investigated the effects of stimulus contrast and contrast gradient on eye accommodation in viewing real-world scenes [30]. Evidences from these studies suggested that the accommodative response attempts to maximize the contrast of the foveal retinal image and the contrast gradient helps stabilize the accommodation fluctuation of the eye on the target of interest. Therefore, we may conclude that pseudo-correct focus cues can be generated at the dioptric midpoint by applying an appropriate depth fusing filter even without the existence of a real focal plane.

To further demonstrate the pseudo-correct focus cues created through DFD displays, we set a dual-focal plane display model to be the same as used in the previous paragraph (i.e. z ₁₂ = 1D, and Δz = 0.6D). We simulated multiple retinal images of a Snellen E target by convolving the target with the PSF ₁₂(z) defined in Eq. (3) while the luminance of the target is evenly divided between the two focal planes (i.e. w ₁ = w ₂ = 0.5). Thus the fused target is expected to appear at the dioptric midpoint of the two focal planes. In Fig. 3 , the left to right columns correspond to the eye accommodation distances of z = 1.3, 1, 0.7D, respectively; while the top to bottom rows correspond to the target spatial frequencies of v = 2, 5, 10, 30cpd, respectively. As predicted by the results in Fig. 2 (b), the retinal image contrast is higher when the eye is focused at z = 1D rather than at either z = z ₁ = 1.3D or z = z ₂ = 0.7D. Meanwhile, at the same accommodation distance, the retinal image contrast clearly depends on the spatial frequency of the target where the targets with lower spatial frequencies (e.g. 2, 4, and 10 cpds) have better image contrast than the higher frequencies (e.g. v = 30cpd).

 

Fig. 3 Simulated retinal images of a Snellen E target in a depth-fused dual-focal plane display, with z ₁ = 1.3D, z ₂ = 0.7D, and w ₁ = 0.5. The accommodative distances are z = 1.3D in (a), (d), (g), (j); z = 1.0D in (b), (e), (h), (k); and z = 0.7D in (c), (f), (i), (l), respectively. The target spatial frequencies are v = 2cpd in (a), (b), (c); v = 5cpd in (d), (e), (f), v = 10cpd, in (g), (h), (i); and v = 30cpd in (j), (k), (l) respectively. The sizes of the above pictures are proportional to the relative sizes viewed on the retina.

Download Full Size | PDF

To derive the dependence of the rendered accommodation cue on the depth-weighted fusing function as described in Eq. (2), we extended the MTF simulation shown in Fig. 2 (b) by incrementally varying w ₁ from 1 to 0 at an increment of 0.01 while having w ₂ = 1-w ₁. For each w ₁ increment, we simulated the MTF ₁₂ of a fused pixel while incrementally varying the eye accommodation distance from the front focal plane (z ₁ = 1.3D) to the back focal plane (z ₂ = 0.7D) at an increment of 0.02D. We selected the accommodation distance that maximizes the MTF ₁₂ to be the rendered accommodation cue corresponding to the given depth-weighted fusing factor (w ₁) of the front focal plane. The accumulated results yield the optimal depth weighted luminance (L ₁ and L ₂) of the front and back focal planes to the luminance of the fused target (L) as a function of the accommodation distance (z) for a focal plane pair. We may further extend the method described above to more than two focal planes covering a much larger depth range. As an example, we chose a 6 focal plane DFD display design covering a depth range from 3D to 0D. By assuming a 0.6D dioptric spacing, 6 focal planes were placed at 3D (z ₁), 2.4D (z ₂), 1.8D (z ₃), 1.2D (z ₄), 0.6D (z ₅), and 0D (z ₆) respectively. In this display configuration, we repeated the above-described simulations independently to each adjacent pair of focal planes. The black solid curves in Fig. 4 plotted the luminance ratio g _i = L _i/L (i = 1,2,3,4,5) of the front focal plane in each focal plane pair of (i, i + 1) as a function of the rendered accommodation cue z. On the same figure we also plotted a typical box filter (blue dashed curves), which corresponds to multi-focal plane displays without applying depth-weighted fusing method [9], and a linear depth-weighted filter (green dashed curves) as proposed in [10,11]. It is clear that the fusing functions based on the maximal MTF ₁₂ values demonstrated noticeable nonlinearity. As mentioned in Section 3, since the retinal image quality is affected by defocus, we tentatively attribute the nonlinearity effect to the nonlinear degradation of the retinal image quality with defocus.

 

Fig. 4 Simulated accommodation cue versus depth filter curves (black solid line), and the nonlinearly-fitted curves (red dashed line) in a 6-focal plane DFD display, with z ₁ = 3D, z ₆ = 0D, and Δz = 0.6D. Blue and green dashed lines indicate the box and linear depth filters respectively.

Download Full Size | PDF

Based on the simulated results shown in Fig. 4, a periodical function g _i(z) can be used to describe the dependence of the luminance ratio of the front focal plane in a given pair of focal planes upon the scene depth:

Table 2. Parameters of Eq. (5) for a 6-Focal Plane DFD Display

View Table | View all tables in this article

5. Retinal image qauality analysis

Figure 5 demonstrated the simulated retinal images of a 3-D scene through a 6-focal plane DFD display with depth-weighted nonlinear fusing functions given in Eq. (5), as well as with the box and linear filters shown in Fig. 4. The six focal planes were placed at 3, 2.4, 1.8, 1.2, 0.6, 0D, and the accommodation of the observer’s eye was assumed at 0.5D. The 3-D scene consists of a planar object extending from 3D to 0.5D at a slanted angle relative to the z-axis (depth-axis) and a green grid as ground plane spanning the same depth range. The planar object is textured with a Sinusoidal grating subtending a spatial frequency of 1.5~9cpd from its left (front) to right (back) ends. The entire scene subtends a FOV of 14.2x10.7 degrees. The simulation of the DFD images takes five steps. We first render a regular 2-D perspective image of a 3-D scene using computer graphics rendering techniques. A 2-D depth map (Fig. 5a) in the same size as that of the 2-D perspective image is then generated by retrieving the depth (z) of each rendered pixel from the z-buffer in OpenGL shaders [18]. Next, a set of six depth-weighted maps is generated, one for each of the focal planes by applying the depth-weighted filtering functions in Eq. (5) to the 2-D depth map. In the fourth step, we then render six focal plane images by individually applying each of the depth-weighted maps to the 2-D perspective image rendered in the first step through an alpha-blending technique. Finally, the 6-focal plane images are convolved with the corresponding PSFs of the eye determined by the specific accommodation distance (z = 0.5D) and the focal plane distances, and the resultant retinal image is then obtained by summing up the convolved images. Figures 5 (b), (c) and (d) show the simulated retinal images of the DFD display by employing a box [10], linear [10,11], and nonlinear depth-weighted fusing function, respectively. As expected, the 3-D scene rendered by the box filter (Fig. 5b) indicates strong depth discontinuity effect around the midpoint of two adjacent focal planes, while those rendered by linear and nonlinear filters show smoothly rendered depths. In term of the contrast of these images, although we expect that the non-linear filters yielded higher image contrast in general than the linear filters, the contrast differences are barely visible by solely comparing Fig. 5 (c) and (d), partially due to the low spatial frequency of the grating target.

 

Fig. 5 (a) Depth map of a 3-D scene rendered by shaders, and retinal images of the same scene rendered by (b) box (c) linear and (d) nonlinear depth filters in a 6-focal plane DFD display.

Download Full Size | PDF

In order to quantitatively evaluate the retinal image quality differences between the linear and nonlinear fusing functions, we further evaluated the MTFs of the retinal images simulated with the method described in Section 3. A dual-focal plane display configuration, with z ₁ = 1.8D and z ₂ = 1.2D, was assumed in the simulation without loss of generality. The eye accommodation distance z was varied from z ₁ to z ₂ at an interval of 0.1D. For each eye accommodation distance, Fig. 6 plotted the MTFs of the retinal images simulated with the linear (green circle) and nonlinear (red square) depth-weighted fusing functions. As shown in Figs. 6 (a), (d), and (g), when the accommodation distance is at z ₁, z ₂, or z ₁₂, the MTFs of using the linear depth filter is nearly identical to that of using the nonlinear filters; while at all other accommodation distances, the MTFs of using the nonlinear filter is consistently better than that of using the linear filter indicated by Figs. 6 (b), (c), (e), and (f). More interestingly, even though previous literatures [2] have assumed the worst image quality occurs at the dioptric midpoint by employing a linear depth filter, our quantitative analysis shows this assumption indeed is not supported by a linear filter while being true for the nonlinear filter. For instance the green-colored MTF in Fig. 6 (b) (as z = 1.7D) is even worse than that in Fig. 6 (d) (as z = z ₁₂ = 1.5D). In summary, the nonlinear depth-weighted fusing functions shown in Fig. 4 can produce better retinal image quality compared to a linear filter. In consequence it may better approximate the real 3-D viewing condition and further improve the accuracy on depth perception.

 

Fig. 6 Comparison of MTFs in a dual-focal plane DFD display by using linear (green circle) and nonlinear (red square) depth-weighted fusing functions respectively. Front and back focal planes are assumed at z ₁ = 1.8D and z ₂ = 1.2D respectively, and accommodation distance is z = 1.8D (a), 1.7D (b), 1.6D (c), 1.5D (d), 1.4D (e), 1.3D (f), and 1.2D (g), respectively.

Download Full Size | PDF

6. Conclusion

We presented a systematic method to address two fundamental issues in designing a multi-focal plane display: (1) the appropriate dioptric spacing between adjacent focal planes; and (2) the depth-weighted fusing function to render a continuous 3-D volume. By taking account of both ocular and display factors, we determined the optimal spacing between two adjacent focal planes to be ~0.6D to ensure the MTF of a fused pixel at the dioptric midpoint to be comparable to the DOF effect of the HVS on the MTF of a real pixel at the same distance under photopic viewing conditions. We further characterized the optimal form of a set of depth-weighted fusing functions as a function of rendered accommodation cues. Based on simulation results, the proposed nonlinear form of depth filters appears to be better than a box filter in term of improved depth continuity, and better than a linear filter in term of retinal image contrast modulation. Although our method, as discussed in Section 2, does not take into account some other ocular factors such as scattering on the retina and psychophysical factors such as the neuron response, it provides a systematic framework that can objectively predict the optical quality and guide the design of DFD displays. Further validation of the proposed DFD design methods requires a supplemental approach where the optical quality of a DFD display is evaluated through objective or subjective tasks with real observers in the loop. This approach requires a DFD display platform that offers the flexibility to reconfiguring the key display parameters such as focal plane placements and requires careful control of experimental conditions to maintain generality of the measurements. Due to the slow response speed of the microdisplay and active optical elements, our previous prototypes [15,31] were unable to render as many as 6 focal planes at an acceptable refresh rate without visible flickering. In the future, we plan to build a 6-focal plane DFD display with faster microdisplay and active optical components and carry out further experiments to validate our methods.