1. Introduction

Due to the creation of immersive realistic experiences with depth perception, great variety of developments in 3D display technologies have been occurred in the last years. However, even immersive realistic experiences can be provided by 3D displays, the main challenge to hinder its extensive applications is how to provide satisfactory Quality of Experience (QoE) for users [1]. Therefore, simply extensions of the existing 2D techniques (e.g., image retargeting) to 3D applications usually fail to deliver a satisfactory QoE for users, because the QoE models in 2D and 3D semantics are different [2]. Although some promising holographic, integral imaging and Super MultiView displays [3] are being developed to offer a high QoE, the essence of QoE optimization considered in this work is to eliminate intrinsic content limitations (e.g., accommodation-vergence conflict [4,5]) by taking advantage of new means.

With the diversity of various stereoscopic 3D (S3D) display devices, such as phones, table PCs and TVs, similar to 2D media, 3D media also requires content adaptation to different display devices [6]. In addition to adapting to the device spatial resolution (retargeting along the x and y directions), for S3D displays, depth adaptation is especially critical to adapt the comfort zone of different displays [7], otherwise, undesirable visual discomfort may be occurred. Particularly, the depth ranges should be adjustable on different S3D displays to avoid accommodation-vergence conflicts of the scene. Without a specific explanation, S3D displays in this work refer to binocular 3D displays. According to the accessibility of depth information, sources for S3D displays mainly originate from two aspects, i.e., fixed-viewpoint stereoscopic images and unfixed-viewpoint multiview video plus depth (MVD) data [8]. Currently, many works have been focused on designing effective coding methods for MVD data [9–12]. To manipulate the depth according to different sources, in the first aspect, depth adaptation is achieved by changing the disparity/depth ranges of stereoscopic images [13,14]. In the second aspect, depth map is adjusted according to specific display requirements, and is aided to generate stereoscopic images by depth-image-based rendering (DIBR) technique [15]. However, these methods still rest on the original image resolution to adjust the depth information, but if the display device is changed, it is desirable to control the resolution and depth simultaneously for S3D displays. Since existing depth adaptation methods cannot retarget the size of stereoscopic image, the advantage of this work is to generate resolution and depth adjustable stereoscopic image pairs from MVD data by retargeting.

This work focuses on optimizing the resolution and depth of MVD data to adapt to the comfort zone of the display collecting camera parameters of virtual camera for view synthesis, which are not presented in fixed-viewpoint stereoscopic image system. This paper provides new insights on how the selected virtual camera affects the retargeting process and further affects the 3D perception of view synthesis. A novel method, the Multiview Video plus Depth ReTargeting (MVDRT), aiming at providing a solution to optimize the factors, such as image resolution, depth range, and the perceived visual comfort. Previous methods only consider fixed-viewpoint depth remapping or content retargeting, while the MVDRT utilizes the parameters of virtual camera and viewing conditions to optimize the 3D perception of view synthesis. This optimization provides a very valuable tool to guide the retargeting for MVD data. The major advantages of the MVDRT are as follows:

The rest of the paper is organized as follows: we first review the related work in Section 2, detail our method in Section 3, and present results in Section 4 and discussion in Section 5.

2. Related work

3D media retargeting. The problem of 2D image/video retargeting has received much attention over the recent years, followed the classification of discrete and continuous methods [16–18]. Towards retargeting for 3D media, some approaches have been proposed. Seam carving technique is to determine the seams for one view of the stereoscopic image, and find the corresponding seams in another view based on the correspondence constraint as well as the spatial coherence [19–21]. However, seam caving for stereoscopic image will cause shape distortion in or between left and right images. Also, due to depth preservation essence of the seam carving methods, their depth adaptation on different display devices is comparatively poor.

For continuous methods, warping-based retargeting is performed to solve mesh deformation problem with several deformation constraints. The key of this type of methods is to uniformly distribute more distortion on those less important regions and less distortion on more important regions. Chang et al. [22] presented a content-aware method for editing stereoscopic image to different displays, which optimizes energy function based on sparse stereo correspondences and disparity consistency constraints. Lin et al. [23] proposed an object-coherence warping method for content-aware stereoscopic image retargeting, in which the object correspondences between the left and right images are utilized to generate an object-based significance map and preserve object consistency during warping. Li et al. [24] presented a warping-based stereo retargeting framework by imposing both shape preservation and depth preservation constraints, in which region-wise depth-preserving constraints are derived to control the local warping functions for depth preservation. Lee et al. [25] proposed a layer-based stereoscopic image resizing method, in which each layer is warped by its own mesh deformation and the warped layers are composited together to form the retargeted images. However, although these methods can handle MVD data, it is limited to altering the perceived depth without the constraint from user’s visual experience.

Depth adaptation. Another way for depth adaptation is to retarget depth/disparity range, or control depth perception or visual comfort for the 3D content. Disparity shifting methods are relatively simple that adjust the zero disparity plane (ZDP) of a scene but maintain its overall disparity range [26]. Various linear and nonlinear disparity/depth remapping methods are developed to adjust the comfortable viewing zone [27], or adjust the depth perception [28], or manipulate the perceived depth [29], with the aims to enhance the 3D visual experience. However, these methods do not take into account the distortion from display resolution, and their depth adaptation to this situation is poor.

To optimize the disparity/depth by warping, Lang et al. [13] presented a set of disparity mapping operators for stereoscopic images that uses disparity and saliency estimation to compute a deformation of the input views to meet the target disparities. Yan et al. [14] used image warping to simultaneously adjust the positions of image pixels in the left and right images in order to remap the depth range of stereoscopic image. Wang et al. [30] proposed stereoscopic video disparity adjustment framework that takes disparity range, motion, and stereoscopic window violation into account. However, these methods only focus on disparity adjustment for stereoscopic contents, and develop several disparity mapping operations with consistent size constraint. If considering the distortions from different spatial resolutions and different camera geometries, these methods may fail in these situations.

Although multiple virtual views can be conveniently generated via DIBR technique, depth adaptation for view synthesis is considered in some methods. Huang et al. [31] presented a warping-based method for synthesizing virtual views from a binocular stereoscopic image based on dense feature correspondences instead of using DIBR. Li et al. [32] investigated the influences of stereoscopic parameters on multiview synthesis in S3D display and presented a parameter adjustment method for reducing visual discomforts and geometric distortions. Lei et al. [33] proposed a projection-based disparity control method to generate multiple virtual views by shifting the ZDP. However, in the practical applications, it is expected to view the virtual views naturally and comfortably under different viewing conditions. Thus, the virtual camera parameters should be taken into account in synthesizing the multiple virtual views.

In order to illustrate the limitations of the existing 3D retargeting and depth adaptation methods, we list the main properties of some representative works in Table 1. Considering resolution and depth adaptation to generate virtual views from MVD data in this work, we combine image retargeting and depth adaptation techniques to present a possible solution. The motivation of this work mainly comes from two aspects:

Table 1. Main properties of the representative works

View Table | View all tables in this article

3. The proposed method

3.1 Framework overview

We propose a retargeting framework that aims to generate resolution and depth adjustable stereoscopic images from MVD data. As shown in Fig. 1, taking two-view MVD data as input (denoted as L-Reference and R-Reference respectively), we first transform the input MVD data to the display space based on the target displays and viewing conditions. Then, warping energy function is constructed based on the constraints of shape preservation, line bending, and visual comfort. The energy function is used to optimize the mesh warping procedure to generate the retargeted reference images and the corresponding retargeted depth maps. Finally, we generate the virtual view to construct the stereoscopic images by DIBR. Different with the existing stereoscopic image retargeting methods that preserve the original depth information after retargeting, our method simultaneously optimizes the resolution and depth to match the target displays and viewing conditions.

 

Fig. 1 Illustration of our proposed MVD retargeting framework.

Download Full Size | PDF

3.2 Relationship between the depth image and the display space

Figure 2 illustrates cameras geometry in different spaces. The scene is captured by a reference camera in 3D space shown in (a), while the viewer perceives the 3D scene in the display space shown in (b). The baseline between reference and virtual cameras is not equal to the interocular distance between two eyes, and the depth of the convergence plane is not equal to the viewing distance. Considering the effects of different displays and viewing conditions, the camera arrangements used for the creation of stereoscopic contents are not unique. Therefore, transformation from the 3D space to display space is the first step to fulfill the following optimization. Given the L-Reference data ($I_L(x,y)$and$D_L(x,y)$), the real depth z_x,y of a pixel point in$I_L(x,y)$can be calculated from the quantized depth value D_x,y in$D_L(x,y)$by

 

Fig. 2 Illustration of camera geometry in different spaces.

Download Full Size | PDF

Considering the relationship between the camera coordinate and 3D world coordinate, the imaging geometry of the reference camera is formulated by projecting a matrix $[u_L,v_L,1]$in the camera coordinate to the matrix$[X,Y,Z]$in the world coordinate as follows:

From (5), it can be seen that the binocular disparity d incorporates the effects of camera distance (T₁ and T₂), focus length (A₁ and A₂) and camera rotation (R₁ and R₂), as well as the distance of objects to the camera (z). These parameters are common for a given camera arrangement. However, as demonstrated in Figs. 2(a) and 2(b), scene geometry from different camera sensors and screens will be not the same. Therefore, the distortions in image plane may be not the same with the perceived distortions on displays. Since the actual size of a real scene is generally too large or too small to be reproduced on different displays, the scaling on all dimensions besides horizontal and vertical dimensions is necessary. Taking the left and virtual images as a stereoscopic image pair, based on the principal of binocular vision, the point coordinates perceived by the viewer in the display space can be determined as follows [32]:

3.3 Construction of warping energy function

For resolution and depth adaptation, warping energy optimization is a feasible way to describe the relationship among camera arrangement, display, and stereoscopic viewing conditions. These factors are usually utilized in the energy functions to characterize the overall QoE perceived by the viewers when watching 3D content. In the practical situations, the QoE will be affected by different viewing conditions. Given the inputs $I_L(x,y)$and$D_L(x,y)$, a set of 2D grid meshes with size of 40 × 40 in the reference image plane are denoted as$U_L=\left\{U_{L,k}\right\}$, $U_{L,k}=\{u_{L,k}^1,u_{L,k}^2,u_{L,k}^3,u_{L,k}^4\}$. For each vertex (taking $u_{L,k}^1$ as example), it has$u_{L,k}^1=\{x_{L,k}^1,y_{L,k}^1\}$, where$x_{L,k}^1$and $y_{L,k}^1$ denote the horizontal and vertical coordinates in the reference image plane, respectively. By transforming the coordinates to the display space by (6), a set of 3D surface meshes$V=\left\{V_k\right\}$ are created to cover the reference and virtual cameras. For each 3D surface mesh V_k, it consists of four vertices,$V_k=\{v_k^1,v_k^2,v_k^3,v_k^4\}$, where $v_k^1$, $v_k^2$, $v_k^3$and $v_k^4$ denote the left-top, left-bottom, right-top, and right-bottom vertices, respectively. The coordinate of each vertex (taking$v_k^1$ as example) is represented as$v_k^1=\{X_k^1,Y_k^1,Z_k^1\}$, where$X_k^1$, $Y_k^1$ and$Z_k^1$denote the horizontal, vertical and depth coordinates in the display space, respectively.

1) Shape preservation energy: Since our method uses four vertices of a mesh to control the selected object, the shape and depth of the selected object may be deformed while implementing warping. This problem is solved by introducing a shape preservation energy term that preserves the shapes of the high-significance objects as much as possible, while allows to conduct large deformation for those low-significance objects. For each mesh V_k with four vertices, the distortion energy for the mesh is defined as

Considering visual importance of each mesh, the total shape preservation energy is the weighted sum of the distortions of all meshes, defined as

 

Fig. 3 Example of the 3D saliency map and the depth sensitivity map.

Download Full Size | PDF

2) Line bending energy: Refer to [22], to minimize the bending of the mesh edges, we define a line bending energy by measuring the angle between the deformed edge and its original edge. Let $e=v_i-v_j$be original edge between two vertices v_i and v_j, and $\widehat{e}={\widehat{v}}_i-{\widehat{v}}_j$be the deformed edge between two vertices ${\widehat{v}}_i$ and${\widehat{v}}_j$, the angle between e and$\widehat{e}$is approximated by

Finally, the total line bending energy is defined as

3) Visual comfort energy: According to our warping framework, the depth of each mesh will be changed together with the resolution. To ensure the naturalness and visual comfort of the retargeted color-plus-depth data, we propose to adjust the depth range to adapt the new camera geometry. Considering different displays and viewing distances, the original depth range in the 3D space is mapped to the display space, obtaining a new depth range$[Z_D^{far},Z_D^{near}]$. To ensure the depth range$[Z_D^{far},Z_D^{near}]$fall into the comfort zone, we define a comfortable depth range based on the Percival’s Zone of Comfort (PZC) as follows:

The scaling factor is defined to scale the original depth range to a new comfortable depth range by:

The depth scaling function is defined to remap the original depth Z_D to a new depth by

Since objects at the screen yield null disparity (d = 0), while objects in front of and behind the screen produce negative and positive disparity, respectively, it leaves large space for the objects at the screen to adjust their depth information. Also, since the depth scaling function in (17) linearly remaps the depth values, the scene layout will be still the same with the original one. Considering the impacts of different disparity planes on visual comfort, the visual comfort energy is defined as

3.4 Optimization for virtual color-and-depth generation

By combining the above shape preservation, line bending and visual comfort energy terms, the final optimization for the mesh warping is formulated as

Minimizing the energy function corresponds to solving a least-squares linear system$A{\tilde{V}}^T=b$and finds a set of deformed meshes$\tilde{V}$, which is satisfied the boundary conditions. However, in a typical DIBR system, typically two views of color and depth videos are used to synthesize a virtual view. That is, the quality of the virtual view will be affected by the detailed view blending in DIBR, that combines the synthesized virtual views from different reference views. To ensure the consistent deformation across different reference views, by re-projecting the deformed vertices$\tilde{V}$to the left and right reference image planes, the deformed vertices${\tilde{U}}_L$and${\tilde{U}}_R$in the left and right reference images are obtained. Further, by establishing the warping relationships between the original vertices$U_L$and the deformed vertices${\tilde{U}}_L$, and between the original vertices$U_R$and the deformed vertices${\tilde{U}}_R$, we can warp the left and right images to the target resolution, denoted as${\tilde{I}}_L(x\text{'},y)=I_L(x,y)$and${\tilde{I}}_R(x\text{'}\text{'},y)=I_R(x,y)$, respectively.

Since the depth coordinate of each vertex in changed after the above mesh optimization, we cannot directly use the same warping relationship on the depth maps. To obtain the retargeted depth map, the retargeted depth values are inversely transformed from the display space to 3D space. Then, we establish a five-parameter fitting function by minimizing the reconstruction error between the original and retargeted depth values for all vertices as [36]

Finally, with the retargeted${\tilde{I}}_L(x\text{'},y)$and${\tilde{D}}_L(x\text{'},y)$, ${\tilde{I}}_R(x\text{'}\text{'},y)$and${\tilde{D}}_R(x\text{'}\text{'},y)$, the virtual view is synthesized by DIBR technique. Different with the general procedure that synthesizes the virtual view first and retargets it to target resolution, the advantage of the framework is that the viewing condition and the location of virtual camera are propagated to the warping process, so that the virtual camera parameters are considered and stereoscopic vision is addressed in a more natural way. Especially, in multiview displays, multiple virtual views should be synthesized simultaneously from the reference cameras. In this situation, the same warping relationship derived from the two-view case can be propagated to other virtual views to add the efficiency of multi-view view synthesis. Since virtual views are synthesized from the adjacent two views (i.e., L-Reference and R-Reference in Fig. 1), we only consider two-view MVD format in this work, and it can be easily extended to multiple views. For example, for three-view MVD format, the virtual view can be synthesized from two adjacent views.

4. Experimental results

In the experiment, seven typical 3D video sequences, ‘Balloons’ (1024 × 768), ‘Kendo’ (1024 × 768), ‘Bookarrival’ (1024 × 768), ‘Newspaper’ (1024 × 768), ‘GT_Fly’ (1920 × 1088), ‘Undo_Dancer’ (1920 × 1088) and ‘Poznan_Street’ (1920 × 1088) are tested. The selections of left, right and virtual views for these sequences are shown in Table 2. For the view synthesis, view synthesis reference software (VSRS) 1D fast [37] is employed. Although different parameter setting will have certain influences on the retargeting results, based on the practical subjective tests, we set the interocular distance d_e = 65mm, the width of display W = 750mm, the horizontal resolution R = 1920 and the viewing distance L_D = 800mm.

Table 2. Selected views for experiment.

View Table | View all tables in this article

4.1 Analysis the influence of depth remapping

As an important step of the proposed method, the final retargeted depth maps are obtained by remapping the original depth values to its retargeted values. Figure 4 shows the examples of the retargeted depth maps and the corresponding synthesized virtual images with and without depth remapping step (for better display of the depth maps, the depth values are mapped to [0, 255]). The retargeted depth maps with depth remapping step have distinctly different appearances with those of the retargeted depth maps without depth remapping step, because the depth values are simultaneously optimized in our warping framework, while the retargeted depth maps without depth remapping step omit this property (still preserve the original depth values). Using the depth maps without depth remapping step, the synthesized images have significant ghost in the boundary area, while the proposed method can eliminate the phenomenon. Also, duo to the influence of incorrect depth maps in the existing 3D video sequences, which are estimated by depth estimation software, the synthesized images will introduce a certain geometric change especially at the edge areas, while optimizing the depth in the display space can somewhat reduce the influence. More results will be presented in the next sections.

 

Fig. 4 Examples of the retargeted depth maps and the corresponding synthesized images with and without depth remapping step.

Download Full Size | PDF

4.2 Comparison to other methods

Since the proposed method is very related to the stereoscopic image retargeting in the two-view case, we compare our method with state-of-the-art stereoscopic image retargeting methods, including linear scaling (SCL), stereoscopic seam carving (SSC) [20], and single-layer warping (SLW) [22]. For the SCL method, we independently apply the same scaling factor on each reference view. For the SSC and SLW methods, the left reference image and right reference image are taken as a stereoscopic image pair. The same retargeting relationships are directly performed on the depth maps for these methods. Figure 5, Fig. 6, and Fig. 7 illustrate the retargeting results on ‘Poznan_Street’, ‘Undo_Dancer’ and ‘Newspaper’ test sequences. In these figures, all images are shrunk by 40%. The first row in the figures shows the retargeted left reference images. The second row shows the retargeted left depth maps and the third row show the corresponding histograms of the depth maps. The synthesized images from the left and right reference views are shown in the fourth row. As shown in the retargeted left reference images, the SCL method may distort images due to the lack of object information, and the SSC method will suffer from more or less object shape distortion due to the essence of discrete seam removal. Although it is effective in shape preservation, the SLW method will lost the boundary contents, e.g., the car in Fig. 5 and girl in Fig. 7. As shown in the histograms of the depth maps, due to the nature of depth preservation, the SCL and SLW methods have the similar depth distributions with the original ones, while our method changes the depth distributions due to depth remapping. Using the above retargeted reference images and the retargeted depth maps to synthesize the virtual images, the synthesized images with the SCL and SLW methods have significant ghost in the boundary area. The reasons may be that, on one hand, the inconsistent deformation on the left and right reference views with the methods may change the inter-view geometry, while image retargeting in fact does not consider this factor. On the other hand, due to depth preservation essence, a reference point will be projected to the original instead of the retargeted location in the virtual view, while the matched reference points in the left and right reference views will be changed after retargeting. Even the ghost effect is not obvious for the SSC method, the synthesized images still suffer from significant shape deformation due to the poor quality of the retargeted reference images. In contrast, the proposed method preserves most of the structure information without the ghost effect in the synthesized images.

 

Fig. 5 Comparison between our method and the SCL, SSC and SLW on ‘Poznan_Street’ test sequence.

Download Full Size | PDF

 

Fig. 6 Comparison between our method and the SCL, SSC and SLW on ‘Undo_Dancer’ test sequence.

Download Full Size | PDF

 

Fig. 7 Comparison between our method and the SCL, SSC and SLW on ‘Newspaper’ test sequence.

Download Full Size | PDF

4. 3 Subjective evaluation results

A subjective assessment test is conducted to validate the proposed MVDRT model and investigate the influence of the above-mentioned factors (e.g., shape preservation energy, line bending energy and visual comfort energy) on the perceived QoE. The original motivation of this study is to optimize the MVD retargeting for multiview displays with multiple virtual views. The difference between the stereoscopic and multiview displays is that, stereoscopic displays delivers the same pair of information across the entire view zone, whereas multiview displays treats the same space as discrete pieces and provides a different view of the virtual world. However, due to the lack of multiview displays, this test is carried out on a stereoscopic display with a given stereoscopic image pair. The main characteristics of the test are detailed as follows:

Methodology: The Absolute Category Rating (ACR) [38] methodology was used to evaluate the test materials. During the test, the observers were asked to provide a score for the overall 3D QoE, consid ering shape preservation, depth perception, visual comfort, etc. They used the five-grade quality scale. After detecting and discarding outliers of all opinion scores, the final mean opinion score (MOS) for each stereoscopic image was calculated as the mean of the remaining opinion scores.

Environment: The subjective tests were conducted in the laboratory designed for subjective tests according to the recommendation BT.500-11 [39]. A 65” Samsung UA65F9000 stereoscopic display war used to carry out the subjective tests. It is an Ultra HD 3D-LED display with active shutter glasses. The viewing distance was 3 times the height of the screen.

Observers: A total of 25 undergraduate students (12 males and 13 females) in Ningbo University were participated in the subjective test, all of them having normal vision. The ages of the participants were between range 20 and 25. The participants were asked to rank the stereoscopic images based on their own judgment.

The user study results on seven MVD sequences are reported in Table 3. To construct a stereoscopic image pair, the reference left image and the synthesized virtual image are used. Considering the factors of shape preservation, depth perception and visual comfort in evaluating the QoE, we evaluate the user’s visual experience in viewing a stereoscopic image pair. For the SCL and SLW methods, due to the effect of ghost, visual discomfort is obvious in viewing the stereoscopic image pairs, leading to lower opinion scores. For the SSC method, shape deformation is particularly serious for ‘Balloons’, ‘Newspaper’ and ‘Undo_Dancer’ test sequences (also can be found in Figs. 5-7), leading to very poor visual experience, while for some sequences, e.g., ‘Balloons’ and ‘GT_Fly’, the object shape preservation performances are better than our method. Overall, our approach can achieve a better tradeoff among shape preservation, depth perception and visual comfort factors to provide a better visual experience for users.

Table 3. Quantitative subjective assessment results of different MVD sequences.

View Table | View all tables in this article

5. Conclusions

We have presented a retargeting framework for MVD data to optimize the visual experience for S3D displays. The most important technical innovation of our framework is that the parameters of virtual camera and viewing conditions are utilized to optimize the resolution and depth of MVD data for view synthesis. Our results demonstrate the effectiveness and power of our approach. Currently, we generate fixed-viewpoint S3D output from MVD data. For dense virtual view generation, other constraint conditions, such as displaying parameters, should be integrated into our MVDRT framework.