1. Introduction

Integral Imaging (InIm) which was first proposed by Lippmann in 1908, has been an effective approach for the display of Three-dimensional (3D) scene [1–5]. The perspective information of 3D scene is sampled and reconstructed by the pinhole hole array or microlens array (MiLA), which can provide full and continuous motion parallax as well as true color display. Moreover, 3D scenes’ record and reconstruction for InIm is illuminated by incoherent light, and can be perceived auto-stereoscopically without the use of any special glasses. The implementation and configuration of InIm is also relatively simple and low cost. Benefit for the rapid advancement of imaging sensor, display devices and computer graphics, the viewing resolution, the field depth and view angle of InIm has been enhanced, and InIm based dynamic stereoscopic display becomes a most promising and practical way for 3D TV broadcasting systems.

One of the key components of the conventional InIm display system is the MiLA. Up to now, a very expensive metal mold has been necessary in order to manufacture a MiLA for high imaging quality. However, using of the MiLA (pitch size < 1mm) requires the display panel to supply high pixel per inch (PPI), and it’s difficult to achieve large size display limited by the manufactured technique and cost of large size MiLA. Recently, the macro lens array (MaLA) assisted with optical diffusor has been proposed to realize InIm with high viewing resolution and large display size [6] The macro lens with 10mm aperture size is used to build the MaLA, and a holographic functional screen is used to expand light beam. The system can achieve a relatively large display size and high viewing resolution. The generation of elemental images and the characteristics of field depth has been studied. Our group has reported the enhanced viewing resolution of MaLA based InIm using holographic diffuser, which interpolates the discrete light field of the reconstructed points to approximate the original light field [7].

Compared to MiLA, MaLA can be constructed with common lens rather than precision optical machining, and the size of MaLA can be built large enough to realize large size 3D display besides its low cost. Meanwhile, the parameters of the MaLA, such as lens pitch, lens focal, duty ratio of the lens aperture, arrangement of lens, can be adjusted flexibly. However, the geometrical precision of the MaLA is lower than MiLA. The elemental lens in MaLA should be optical gluing on a base glass by manual/mechanical assembling, therefore, the elemental lens cannot be placed on the ideal positions and resulted into a rigging error. Although this rigging error is on the level of 200 um, but the pixel size of LCD panel is about 85 um (300 ppi) or much smaller, and this may lead to a significant shift between the centers of lens and elemental images, which can make the reconstructed 3D images deteriorated and distorted.

To achieve a high-quality 3D display, calibration is necessary to correct optical misalignment and optical aberrations. Although it is quite challenging to achieve correct mapping between a lens array and display, some novel research has been reported in the related topic. Arai et al. [8] first presented the effects of misarrangement of elements (elemental lenses and elemental images) that construct three dimensional (3-D) images in InIm in term of local and global positional errors, which is a foundation for many latter works. Aggoun [9] explored a pre-processing method based on hough transform to accurately correct geometric distortions caused during the capture elemental image to eliminate the banding and moiré effects on the reproduced image. Kawakita et al. [10] analyzed the relationships between the geometric distortion in elemental images caused by an elemental lens and the spatial distortion in the reconstructed 3D image. Others introduce method to find a full correspondence map from light field capture. One of such method is proposed by Li et al. [11], strict dense mapping between LCD and MLA is found by full light field capture and image analysis, their system is complicated in capture devices since one camera with a precision motion platform is adopted, so the time complexity is inevitably high. Similar work has been done by them to remove the 3D image artifacts in a tiled-lens-array InIm due to lens array misalignments [12]. Ji et al. [13] proposed a tilted elemental image array generation method for computer generated InIm to reduce Moiré patterns. Other groups [14–16] simplify the question by assuming that the lenses in the MLA share common features like pinholes free of lens aberrations and equally spaced without fabrication or mounting errors, so the dense mapping between MLA and LCD can be derived from an estimated translation and rotation of misalignment. Sang et al. [17,18] also does some innovative works in the improvement of aberration for better 3D display. Recently, a hybrid camera array-based calibration for computer-generated InIm with flexible setup is proposed [19], the method can get high precision at a reasonable time cost. However, to the best of our knowledge, no work on calibrating the inter-lens position errors has been reported.

In this work, we propose a post-calibration compensation method for the inter-lens position errors correction in InIm with a macrolens array. The inter-lens position misalignment is corrected by forcing it to image in a regular ideal reference grid. The method distinguishes itself from previous ones by finding the correct pixel-to-ray correspondence with a relatively simple setup and acceptable precision. To present the new method, we have organized this paper as follows. In Section 2, we explain the principle of the proposed post-calibration method, ray optical analysis is conducted to show the original and calibrated pixel-to-ray correspondence. Section 3 is devoted to verification of the usefulness of the proposed method by optical experiment. Finally, in Section 4, we summarize the main achievements of this reported research.

2. Principle

2.1 Misarrangement of macro lens array in InIm system

The typical structure of MaLA InIm system assisted with optical diffusor is shown in Fig. 1(a), including a high definition flat display panel, a manually or mechanically assembled MaLA with inexpensive ready-made high-quality macro lens, and a holographic diffusor. Such systems can provide better 3D experience than ever before compared with conventional InIm with MiLA in term of viewing resolution and viewing angle [20]. As the MaLA of an InIm display is still not as ideal as a mature product, macro lenses are not distributed evenly in the MaLA. For a hexagonal arranged MaLA, the inter-lens pitch is roughly set at 11 mm by default (see Fig. 1(b)). However, the actual lens pitch can vary from lens to lens due to the limited precision in fabrication and mounting (see Fig. 1(c)). So, a post-calibration is a must to get correct inter-lens position misalignment and find actual pixel-to-ray correspondence.

 

Fig. 1 (a) The sketch map of the MaLA InIm system with diffusor. The lens distribution of hexagonal arranged MaLA, (b) ideal MaLA, (c) the actual MaLA with inter-lens position deviations.

Download Full Size | PDF

Since the imaging performance of the macro lens used in the experiment is relatively good, besides, an aperture array is often used to extend the depth of field of InIm and relief aberration. Therefore, we can focus on the location of the principle point (lens center) of the lens, and it is sufficient to consider only the principal rays, i.e. a pin-hole model of the lens can be taken for simplicity.

As shown in Fig. 2, there are three paralleled planes, defined as the LCD plane where the elemental images loaded, the lens plane where the MaLA were fixed, and the center depth plane which is the image plane of the MaLA. The distance between the center depth plane and the lens plane is l while the gap between the lens plane and LCD plane is g.

 

Fig. 2 Pixel-to-ray correspondence for (a) ideal MaLA, (b) actual MaLA with inter-lens position deviations.

Download Full Size | PDF

For the regular arrangement of MaLA as shown in Fig. 2(a), the center of the elemental image is denoted as C_i (i = 1,2,3,…), while the optical center of the lens placed in directly front of the corresponded elemental image is denoted as O_i (i = 1,2,3,…). The incident beam that emitted from point C_i and passed through point O_i hits the projection plane on point R_i (i = 1,2,3,…). For the ideal regular arrangement of MaLA, these incidents beams should be paralleled to each other and should be perpendicular to the projection plane, the MaLA plane, as well as the LCD plane. The image points R_i are distributed uniformly and forms an isometric grid. However, in actual case as shown in Fig. 2(b), the position of lens’s optical center may be deviated from its ideal position (denoted as O'_i (i = 1,2,3,…)) resulted from the rigging error. Therefore, these image points R_i (j = 1,2,3,…) are then maldistributed. It means that, arisen from the irregularly arranged MaLA due to the rigging error, the uniformly distributed points grid C_i may form an unevenly distributed points grid after imaging by the MaLA, and the imaging aberration is occurred. Since the magnification of the MaLA is much larger than 1, the small rigging error of MaLA may lead to significant imaging aberration and reduce the reconstruction quality of integral image seriously. Nevertheless, if some mechanisms of compensation can be adopted to maintain a uniformly distributed R_i even with the irregular arrangement MaLA, the imaging aberration can be corrected and the display quality of the integral imaging may be improved. This is the core idea of the post-calibration method for the integral imaging system with MaLA.

2.2 Analysis of post-calibration compensation on projection plane

The principle of the post-calibration compensation is shown in Fig. 3. One lens of the MaLA is exemplified and the one-dimensional case is considered for simplicity. The ideal position of the lens is depicted with dash line while the real one is depicted with the solid line, correspond to their optical center O' and O, respectively. Point C_err denotes the ideal center of the elemental image for the lens, and its ideal image point is R. Since the ineluctable rigging error, the actual image point of C_err shifts to point R_err, which deviates far from the ideal point R. If we move the center of the elemental image to the new position C_per to make that the line is perpendicular to the LCD plane, then the image point R_per may be closer to the ideal one, R. Further on, if the elemental image’s center is placed to an optimized position C_opt where its actual image point R_opt and the ideal one R coincide, the reconstruction aberration can be corrected. In experiment, a cross sign “+” is used as the calibration elemental image, and its position on the LCD is moved one by one pixel from the ideal position C_err along the vertical and horizontal direction until the exact overlapping of R_opt and R is achieved, where R is printed on a calibration target with the ideally regular grid. Suppose for the i-th lens the offset pixel number is and when R_opt and R coincide, and for each lens its and are recorded. In the reconstruction, the elemental image for the i-th lens just should make the corresponded offset pixel number and , and repeat this process for all the elemental images to get the final post-calibration integral image.

 

Fig. 3 The principle of the proposed post-calibration compensation method.

Download Full Size | PDF

The new parameter of misalignment angle, which is defined as the angle spanned by the line $\alpha =\arctan \left(\overline{O{O}^{\prime }}/g\right)$. After the calibration, the center of the elemental image is moved to point C_opt, the misalignment angle becomes

Furthermore, the calibration can be implemented more directly. As shown in Fig. 3, if we load the center point of the test elemental image at its ideal position C_err, and it is imaged by the lens with its imaged center point R_err. After measure the length of , the calibrated value for the elemental image’s center, , can be obtained by

The deviation value of the lens’s center position can also be obtained as

As a matter of fact, this approach can bring significant benefits. We can load a regular arranged test elemental image array with their center points located at the ideal positions and place a target mask at $\overline{R{R}_{\text{err}}}$ between the ideal image and the actual image for each lens can be easily captured and computed, then the calibrated value $\overline{C_{\text{err}}{C}_{\text{per}}}$ for each elemental image can be obtained. If each elemental image is compensated by moving its position with the calibrated value, the imaging deterioration then can be eliminated.

2.3 Availability of post-calibration compensation out of projection plane

The above analysis is effective for the correction of plane images in the projection plane, however, the actual 3D object occupies a certain depth range in the space. According to this method, when the object point in the 3D scene is right located in the projection plane, the correction effect of the object point is the best, and the edge of the object point is displayed more clearly. When the object point is located away from the projection plane, the position of the space point will show some distortions. In order to facilitate the description of this distortion relationship, three sets of grid correspondences existing in the correction process are introduced, i.e. the ideal lens center grid (equal spacing, indicated by the thin-dot line in Fig. 4), the real lens center grid relationship (Constant but non-equal spacing, indicated by the dash line in Fig. 4), and the calibrated lens center grid (non-equal spacing, spacing is related to screen distance, indicated by the solid line in Fig. 4). The first two grids do not vary their spacing with depth change while the third grid varies. From the depth section view of the lens center grid we can find that in the projection plane, the calibrated lens center grid coincides with the ideal grid (see Fig. 4(c)), In the plane near to the display, the calibrated lens center grid is always between the ideal grid and the real grid(see Fig. 4(b)), In the plane close to the observer, the calibrated lens center grid is always on the side away from the real lens grid(see Fig. 4(d)). i.e. at the depth range ± d away from the depth plane l, the accumulation error in plane is larger than the plane. It can be seen from the above analysis that in order to get better display at the position away from the depth plane, we need to pay attention to the following three aspects: (1) From the perspective of the initial value, the ideal grid to should be a reasonable approximation of the actual grid; (2) The projection plane selected during calibration process should be as far as possible from display plane when other system parameters are satisfied; (3) In the reconstruction process, trying to display the object in the depth center plane and the space area near the display side. Since in the InIm with MaLA, a holographic diffusor is used to expanding the light beams, the overall calibration effect of the proposed method is visually acceptable, which will be verified by our latter experiment.

 

Fig. 4 (a)The overall lens center grid in different plane, (b) in z = l-d plane, (c) in z = l plane, and (d) in z = l + d plane.

Download Full Size | PDF

3. Experiment and discussion

3.1 Experimental setup

The experimental setup is shown in Fig. 5. A 12.5-inch 4K (3840x2160) high resolution LCD panel with the pixel pitch p = 72 μm is used to load the elemental image array. The MaLA is hexagonally arranged with the ideal lenslet pitch d = 11mm. The focal length of the circular lens is f = 10 mm, while the lens diameter is 10 mm, which gives a F-number of F = 1. Choosing lens with relatively small F-number is to realize the relatively large viewing angle. The MaLA contains (25 + 24)x8 lens and all the lens are manually cemented on a thin glass plate. The gap between the MaLA and LCD is g = 10.7 mm. To eliminate the cross-talk between the adjacent elemental image, an aperture array with 6mm-diameter is used to block the oblique incident rays from the neighboring elemental image. The aperture can also thin the ray bundles composed of directional light rays that emit from the right-opposite elemental image on LCD, and thus enhancing the depth of field (DOF) of the system. The holographic diffusers with expanding angle of 3.5° is placed l = 17 cm away from the MaLA.

 

Fig. 5 The experimental system setup.

Download Full Size | PDF

3.2 Results and discussions

The deviation value of the lenslets’ center position is measured using the method mentioned above. The result is shown in Fig. 6(a). For each of the lens, the face color value of the second quadrant denotes the deviation value of the lenslets’ center position along x direction, while the face color value of the fourth quadrant denotes that along y direction. The definition of the x-y coordinate system is shown in Fig. 1. The histogram of the distribution of deviation values is also shown in Fig. 6(b). Because of the randomicity of the fabrication and mounting error, all the deviation values have a similar characteristic of distribution both along x direction and y direction. Thus, all the x-direction’s and y-direction’s deviation values are together analyzed statistically. The mean value of the deviation is about $\overline{\Delta }\approx 8\mu m$ with the standard deviation of $\sigma \approx 150\mu m$. It can be seen that the deviation values of the lenslets’ center positions are distributed normally.

 

Fig. 6 (a) The deviation value of the lenslets’ center position where the second quadrant denotes the deviation value along x direction, while the fourth quadrant denotes that along y direction, (b) the histogram of the deviation values.

Download Full Size | PDF

A cross sign “+” array is used as the test elemental image array. Each center of the cross sign “+” is placed in its ideal position. Figure 7(a) shows the photograph of the test elemental image array imaging by the MaLA without calibration. It can be seen that the imaged “+” array indicates a seriously irregular arrangement which is resulted from the rigging error of the MaLA. The test elemental image array is then rearranged with the proposed post-calibration compensation method, and the reconstructed cross sign is near perfectly equal spaced in term of human vision, as shown in Fig. 7(b).

 

Fig. 7 The cross pattern reconstructed in the projection plane (a) from the original ideal lens center value, (b) from the calibrated lens center value.

Download Full Size | PDF

To verified effectiveness the proposed method, A virtual 3D scene of a magic cube with a side length of 12cm is used, the elemental image array is generated by the parallel light field resampling method [21]. Two elemental image arrays were generated by using the ideal and the calibrated lens center data obtained by the proposed method. A comparison of the display result of the two elemental image array is made from the left, middle, and right view, as is shown in Fig. 8. The reconstructed 3D object is broken and seriously serrated when the elemental image array generated by using the ideal lens center data is loaded to the LCD, the distortion is severe and greatly hampered our viewing experience. When the elemental image array generated by using the calibrated lens center data is loaded to the LCD, the distortion is greatly alleviated, at some view even eliminated (see Fig. 8(d)). The overall viewing experience is improved, and the 3D display become acceptable despite the fact that there are still some minor distortions. We thought these experimental results can be a solid evidence for the effectiveness of our proposed method.

 

Fig. 8 comparisons of the display result of the elemental image array generated by the ideal lens center, (a) left view, (b) central view, (c) right view, and the display result of the elemental image array generated by the calibrated lens center, (d) left view, (e) central view, (f) right view.

Download Full Size | PDF

To give an intuitive understanding of the precision of the proposed method at different reconstruction plane, three plane images originated from the 3 different planes of the magic cube located in different place in front of the display panel is used as the 3D scene. The first set of three image from left to right is located 150, 170, 190mm away from the LCD respectively. The display result before calibration can be seen in Fig. 9(a), the image reconstructed is evidently blurred and broken due to the pixel-to-ray correspondence mismatch. The corresponding calibrated display result is well integrated as well as the sharpness of the image is also improved, see Fig. 9(b). When the three images from left to right are located 130, 170, 210mm away from the LCD, the original display result is further blurred partially due to the limited DOF of the display system, the broken phenomenon is also severer, see Fig. 9(c). The display result after calibration (see Fig. 9(d)) is visually acceptable despite the fact that the left and right image are still blurred compared the images reconstructed with shallow depth from the central depth plane. The above experimental results show that the proposed method is effective in inter-lens position error calibration in term of reconstructed geometry accuracy and image quality.

 

Fig. 9 The display result for the plane images locate in 150, 170, 190mm away from the display, (a) before calibration, (b) after calibration, and the display result for the plane images locate in 130, 170, 210mm away from the display, (c) before calibration, (d) after calibration.

Download Full Size | PDF

4. Conclusion

The main contribution of the work is the proposal and feasibility verification of post-calibration compensation method for the InIm system with MaLA, which is vital to eliminate the inter-lens position errors and find correct pixel-to-ray correspondence. 3D display with large-format is an inevitable and foreseeable trend for the future display technology, MaLA assembled from off-the-shelf commodity macro lens provide a practical way to get the key component for InIm 3D display, our proposed method can compensate the image quality loss caused by the inter-lens position errors, and make the unevenly distributed MaLA available. Since the compound lens is often much better in imaging quality and larger in the FOV compared with single lens. In the future, we will test our method on more general InIm display hardware with larger-format MaLA assembled by compound lens. In this kind of implementation, the InIm after calibration can provide better 3D image quality. One of the shortcomings of our proposed method is that the reference grid is handcrafted according to the ideal lens center location, the final calibration precision is closely related the original initial data. we believe our method will help to the application and promotion of InIm based 3D display by providing a way to make the manually or mechanically assembled MaLA acceptable for display.