We propose methods of enhancing pinhole-type integral imaging ray density, resolution, and expressible depth range using a color filter pinhole array on a liquid crystal display panel with a projection scheme. A color filter structure on a liquid crystal display panel acts as pinhole array in integral imaging with separation of color channels. In conventional pinhole-type integral imaging, the resolution, viewing angle, and ray density are limited by the pinhole interval, the width and thickness of the pinhole structure, and the gap between the display panel and the pinhole array. To overcome the limitation of the pinhole interval, we use a color filter pinhole array on a display panel and a projection-type integral imaging scheme. The use of a color filter pinhole array and the projection scheme can enlarge the region of one elemental image and improve the resolution and ray density remarkably. This paper presents the experimental results of the proposed method and a comparison with conventional methods.
© 2012 OSA
1. Introduction and motivation
Integral imaging is a promising technique in glasses-free three-dimensional (3D) display [1–6]. Following the emerging display market of stereoscopic type 3D display, autostereoscopic type 3D displays, such as parallax barrier and lenticular displays, are expected to become the main stream of 3D displays. The autostereoscopic type 3D display based on a lens array and a spatial multiplexing technique can display 3D images without the use of special glasses for finite viewpoints. Integral imaging is technically very similar to lenticular and parallax barrier methods. However, integral imaging has many advantages compared with autostereoscopic displays, such as reconstructing 3D images with full-parallax and a quasi-continuous viewing points achieved by using an elemental image set and a lens array. In addition, it can induce observer accommodation cues and provide smooth motion parallax.
Pinhole-type integral imaging, which uses a pinhole array instead of a lens array, is one of the modified integral imaging systems, which can be constructed more simply than a lens array based system and can be easily converted between two-dimensional (2D) and 3D modes [7–16]. However, pinhole based integral imaging has fundamental problems, such as intensity degradation, a limited viewing angle, and a low resolution. The principle of pinhole-type integral imaging is the same as that of focal mode integral imaging [2, 3]. In the focal mode integral imaging and pinhole-type integral imaging, each lens or pinhole acts as a voxel of the reconstructed 3D image and the spatial resolution of the 3D image is the same as the number of lenses or pinholes. The major difference between focal mode integral imaging and a pinhole-type integral imaging is the use of a pinhole array instead of a lens array, which incurs additional limitations resulting from the pinhole specifications. The specifications of a pinhole-type integral imaging are derived from the characteristics of the pinhole array and other parameters. The viewing angle, resolution, ray density, and expressible depth range are limited by the diameter, thickness, and the interval of the pinhole array. Pinhole intervals, the size of the elemental image region, and system parameters limit the viewing angle, depth range, ray density, and resolution [17–19].
To overcome the limitations of pinhole-type integral imaging, we propose integral imaging based on a color filter pinhole array with a projection scheme. In pinhole-type integral imaging, the interval of the pinhole array determines the viewing angle of each pinhole when the thickness, the gap between the pinhole array and the display, and the diameter of the pinhole array are fixed. Previously, many research groups have proposed methods of enhancing the viewing angle and resolution in integral imaging by using a display panel as a time-multiplexed pinhole array and pinhole pattern control techniques [12, 14–16]. Here, for the first time to our knowledge, we propose pinhole-type integral imaging with color filtering and sub-pixel sampling for the enhancement of integral imaging. In this paper, we explain the limitations of pinhole parameters in a pinhole-type integral imaging system and propose an enhancement method using a color filter pinhole array. We show comparisons between analyses of the specifications of the proposed method and previous pinhole-type integral imaging systems and present experimental results with a resolution-enhanced system.
1.2 Limitations of viewing angle and resolution
In pinhole-type integral imaging, the characteristics of the system are limited by the specifications of the pinhole array, as shown in Fig. 1 . The parameters of pinhole-type integral imaging are composed of the interval between the pinholes Ip, the pinhole diameter d, the gap between the pinhole array and the display panel g, the thickness of the pinhole array t, and the viewing angle of the display panel θ. From the principle of pinhole-type integral imaging, the spatial resolution of the system is the same as the number of pinholes and the ray density, viewing angle, and angular resolution are limited by the pinhole specifications.
Generally, the viewing angle of an integral imaging is the same as that of each pinhole, and one elemental image region we is derived from the relation between the gap g and the viewing angle θ, as follows:Fig. 1. When the thickness of a pinhole array and the diameter are fixed, the viewing angle θ can be derived by the ray optic assumption because the pinhole diameter is much larger than the wavelength of visible light. Equation (2) shows the derived viewing angle for the pinhole array with geometric parameters:Eqs. (1) and (2), the pinhole interval between each pinhole Ip is limited as follows:Fig. 2 .
If the pinhole interval is smaller than one elemental image region, each elemental image is overlapped and the observer cannot avoid a flipped image problem as shown in Fig. 2(a). On the other hand, an elemental image region is wasted and the spatial resolution is decreased when the pinhole interval is larger than one elemental image region, as shown in Fig. 2(b). In pinhole-type integral imaging and focused mode integral imaging, the spatial resolution and the viewing angle fundamentally have trade-off relation. However, pinhole-type integral imaging has additional constraints imposed by the pinhole specifications. Therefore, other structures or methodologies are needed to overcome the limitations imposed by the pinhole interval.
2. Principles of the proposed method
2.1 Pinhole-type integral imaging using color filter pinhole array with projection scheme
To overcome the limitation imposed by the relation between the pinhole interval and viewing angle, we propose the color filter pinhole array based integral imaging using a projection scheme. As shown in Fig. 2, the characteristics of pinhole-type integral imaging are defined by the pinhole interval and the size of one elemental image region, which cannot be extended without temporal multiplexing or spatial multiplexing techniques. In previous research, moving pinhole arrays with temporal multiplexing or polarization switching techniques were proposed for enhancing the resolution or viewing angle [8, 12]. In this paper, we propose color channel multiplexing using a color filter pinhole array on a display panel and a projection scheme.
Figure 3 illustrates the concept of the proposed method and its layer structure. The scheme of the proposed method is not a complex and singular setup. In the proposed method, a sub-pixel structure consisting of color filters on a liquid crystal display (LCD) or other transmission-type display panel substitutes for the conventional pinhole array and the rear display, which shows an elemental image, is replaced by a projection screen and a projector, as shown in Fig. 3(a). Many studies have already demonstrated the use of LCD panels or other types of display panels as the pinhole array with electrical control in integral imaging [8, 12, 16]. However, they focused on the electrical control for 3D/2D convertible systems or temporal multiplexing techniques, and did not consider the sub-pixel structure of the display panel and filtering of color channels. In these studies, the pinhole array of integral imaging was generated by turning each pixel on a display panel on and off without controlling sub-pixels or its color filters.
In this paper, the proposed method controls the sub-pixels of a transmission-type display panel for filtering color channels of an elemental image set, which can enhance the resolution, the viewing angle, and the ray density of integral imaging. In conventional integral imaging, the attachment of color filters to a lens array or a pinhole array causes loss of information and a reduction in resolution. To avoid loss of information, the proposed method is based on projection-type integral imaging, which can project red, green, and blue channels onto one pixel of a projection screen. This can enhance the characteristics of pinhole-type integral imaging when used in combination with color filter pinhole array.
Figure 3(b) shows a more detailed illustration of the proposed concept with the layer structure. When an elemental image is projected onto the projection screen located at the rear of transmission-type display panel by a beam projector, the elemental image has white light information mixed from three channels into one pixel unit on the projection screen and is diffused in all directions by the diffusing angle of the projection screen. The white light rays passing through the gap and the liquid crystal layer are filtered to red, green, and blue channel elemental images by the color filter on the sub-pixel structure. To enhance the viewing angle and the intensity of the reconstructed 3D image, a transmission-type screen is attached to the back surface of the display panel. Therefore, the gap is set to the thickness of the layers of the rear glass and the polarizer, and the thickness of the pinhole array is the same as the thicknesses of the transparent electrode layer, the color filter layer, and the liquid crystal layer. Because of the limitations imposed by the pinhole structure, the viewing angle and the interval between pinholes are determined by limited size Ip, which introduces the limitations in conventional integral imaging. However, the color filter pinhole array can reduce the pinhole interval between different color channels. In the proposed method, elemental images in different color channels can be overlapped with each other, which enhances the resolution and the ray density.
Figure 4 illustrates the principle of the proposed method in detail. From Eq. (3), the interval of the pinhole array is limited by Ip, which is the same as the largest elemental image region with the maximum viewing angle. However, the use of a color filter pinhole array makes it possible for the proposed system to increase the resolution and reduce the interval of each pinhole. As shown in Fig. 4(a), from Eq. (3), the pinhole interval of one color channel must maintain the minimum pinhole interval Ip, which is defined by the specifications of the pinhole, whereas the color filter pinholes of the different color channels can be formed between the pinholes of one color channel. For example, when the red color pinhole is at the minimum pinhole interval Ip, a green and a blue color pinhole can be placed between two adjacent red pinholes at the same interval Ic. Different color channels of the elemental image can be mixed freely, whereas same color channels must maintain the minimum pinhole interval Ip. Therefore, the optimized interval of the color filter pinhole array Ic is one-third of the minimum pinhole interval Ip, which is limited by the pinhole width for the color filter pinholes, as follows:Eq. (4), it can be seen that to prevent repetition of color pinholes of the same color the minimum interval of pinholes for the same color channel Ip cannot be set to a multiple of three of the pinhole diameter and that the gap has to be larger than the thickness of pinhole array.
In the pickup process of the proposed method, spatial information of a 3D object is recorded by different color filter pinhole arrays depending on the color channels of the 3D object as shown in Figs. 4(b) and 4(c). As shown in Fig. 4(b), if each 3D object has a red, green or blue color texture, the spatial information of each 3D object is filtered by the red, green, or blue color channel filters of the pinholes. In the case of Fig. 4(b), the 3D objects are sampled by the minimum pinhole interval Ip and its rays pass through only one color of pinholes. On the other hand, when each texture of the 3D objects is composed of magenta, yellow, and cyan colors, as shown in Fig. 4(c), the rays from the 3D object pass through two different color filter pinholes and record the spatial information of the 3D object. Therefore, the sampling rate of a 3D object in the proposed method depends on the texture-color of the 3D object.
2.2 Comparison of the proposed method with conventional pinhole-type integral imaging
The main idea of the proposed method is sub-pixel sampling of a 3D object with color channel filtering using color filters on a LCD panel and a projection scheme. To demonstrate the advantages of the proposed method over previous pinhole-type integral imaging methods, we diagram all possible types of pinhole-type integral imaging using display panels or a projection scheme in Fig. 5 . Pinhole-type integral imaging is composed of a pinhole array part for spatial sampling of a 3D object and a display part for recording and displaying the elemental image. We assume that the pinhole array is of the same thickness, diameter, and pinhole interval for the same viewing angle, size of elemental image, and gap distance. In addition, the sub-pixel pitch of display panel and the pixel pitch of projection image on a projection screen are the same. To compare each pinhole-type integral imaging configuration and the proposed method, we analyze the resolution, the ray density, and the expressible depth range of each method.
We start by defining the maximum ray density, and the expressible depth range. The resolution of a 3D display is characterized by the spatial resolution and the angular resolution. In pinhole-type integral imaging, the spatial resolution Rs is defined by the number of pinholes in the pinhole array and the angular resolution Rθ is defined by the number of pixels in one elemental image region, as follows:
Figure 5(a) shows one of the conventional pinhole-type integral imaging systems composed of a passive pinhole array and a display panel, and Fig. 5(e) shows the proposed method with ray distribution. Comparison between the two different configurations reveals that the resolution, the ray density, and the expressible depth range are enhanced by the proposed scheme. In the case of Fig. 5(a), the minimum pinhole interval is derived from Eq. (3) and set as Ip, and the period of pixel Tp is the same as the pitch of three sub-pixels 3psb. However, the pinhole interval of the proposed method is set as Ic, which equals one-third of Ip, and the period of the pixels is defined by psb. These results in the spatial resolution Rs being enhanced by a factor of three, from W/3Ic to W/Ic, and the angular resolution Rθ being increased, from we/3psb to we/psb, for each color channel. The ray density of the system Rd is derived by the number of rays per unit area we, which is defined by consideration of each color channel as follows:Eq. (6), the ray density of the proposed method is enhanced by a factor of 3, from 1/psb to 3/psb. In addition, the nearest expressible depth dnear of the proposed method is reconstructed nearer the pinhole array than that in case of Fig. 5(a), and is improved from 3Icg/we to Icg/we by trigonometry. Therefore, the spatial and angular resolutions, the ray density, and the expressible depth range of the proposed method are improved compared to the method of Fig. 5(a). Comparing the cases of Figs. 5(b) and 5(e), the angular resolution and ray density are improved by a factor of three and the spatial resolution and the nearest expressible depth are unchanged.
In comparison to the conventional methods using a projection scheme, as shown in Figs. 5(c) and 5(d), the angular resolution and the ray density of the proposed method are conserved while the nearest expressible depth, the spatial resolution, and the color uniformity of the reconstructed image are improved. In the case of conventional projection-type integral imaging using a pinhole array, as shown in Fig. 5(c), the voxels are reconstructed at a periodicity of the interval of the pinhole Ip, which is three times larger than Ic of the full color expression. On the other hand, the proposed method reconstructs voxels in 3D space with Ic periodicity of red, green, and blue channels with sub-voxel sampling. Therefore, the reconstructed 3D object using the proposed method has a higher fill-factor and spatial resolution although the color expression is limited at the scale of the sub-voxel. However, the observer cannot perceive the color separation of sub-voxels at the optimized observer distance. At this distance, the observer cannot perceive the periodicity of the voxel in the conventional projection-type integral imaging using a pinhole array. Therefore, the proposed method can reconstruct 3D images at a higher spatial resolution than conventional methods. The resolution and ray density are conserved in the conventional projection-type integral imaging using pixels on a display panel whereas the nearest expressible depth and the color uniformity of a reconstructed 3D object are improved in proposed method, as shown in Figs. 5(d) and 5(e).
3. Experimental results
3.1 Experimental setup
To verify the feasibility of the proposed method, we performed an experiment with the configuration shown in Fig. 6 . The experimental setup is composed of an LCD panel for a pinhole pattern generated by color filters using sub-pixel control, a transmission-type projection screen, relay optics, and a beam projector for projection of the elemental image. In the proposed method, one of the improvements is the use of the beam projector for displaying an elemental image with a narrow pixel pitch and a multichromatic pixel unit. For the narrow pixel pitch of the elemental image from the beam projector, we choose the relay optics from the beam projector to the LCD panel to be a normal prime lens and a camera zoom lens. The elemental image from the beam projector is focused onto the CCD size of a digital camera, and then it is magnified and projected onto the transmission-type projection screen attached on the back surface of the LCD by the zoom lens without severe distortion. To enhance the resolution of a 3D image in the proposed method, the gap between pinhole array and the projection screen is set to the minimum value, which is same as the thickness of the structures behind the liquid crystal (LC) layer of the LCD panel. Therefore, the projection screen is attached to the back surface of the LCD panel, the gap is set to the minimum distance, and the 3D image is reconstructed at the highest resolution with the minimum pinhole interval.
To determine the effect of viewing angle of LCD panels, we evaluated the proposed method with two different display types: a twisted nematic (TN) panel and a super in-plane switching (S-IPS) II panel, as shown in Figs. 6(a) and 6(b), respectively. The geometrical specifications of two different LCD panels, such as thickness and pixel pitch, are approximately the same but the viewing angle, brightness, and contrast are improved in the S-IPS II panel. Table 1 shows the specifications of the respective experimental components.
To compare the conventional and proposed methods, we generated the two different pinhole patterns electrically, as shown in Fig. 7 . In the pinhole patterns of the conventional method, each pixel acts as one pinhole and the pinholes are arranged at a four-pixel pinhole interval, as shown in Figs. 7(a) and 7(c); in contrast, the proposed method uses sub-pixels as color filter pinholes and the pinhole interval of one color channel is the same as the conventional method (four pixels) and the pinhole interval of different color channels is one-third of the conventional method (four sub-pixels), as shown in Figs. 7(b) and 7(d).
Figures 8(a) and 8(b) show the elemental image sets of three white letters S, N, and U on different depth planes: 30 mm, −20 mm, and 10 mm from the display panel for the conventional and proposed methods. As shown in Figs. 8(c) and 8(d), the sampled elemental images of the conventional method are separated by the pinhole interval while the elemental images of the color filter pinhole array freely overlap on each elemental image region. Figure 9 shows the magnified elemental image of high-resolution objects: two-fish on different depth planes 30 mm in front of and behind the pinhole array for the conventional and proposed methods.
3.2 Experimental result
To our knowledge, experiments with conventional pinhole-type integral imaging have been performed only with 3D letter objects in a darkened room because of low spatial resolution and brightness [7–16]. In the experiments of the proposed method, we took pictures of and observed the reconstructed 3D object in a bright room to demonstrate the advantages of high brightness and high resolution.
Figure 10 shows the experimental results for three white letters displayed by the conventional and proposed methods. The experimental results show the different voxel separations and fill factors in the center of the view of the reconstructed 3D image. In the conventional method, the pinhole can express whole color information with an interval of pixel size but exhibits a color non-uniformity problem. However, the proposed method reconstructs a 3D image with red, green, and blue sub-voxels, which are separated by an interval of sub-pixel size, do not exhibit a color non-uniformity problem, and the fill-factor of the proposed method is higher than the case of the conventional method. Figure 11 shows the reconstructed 3D images of a high-resolution image of two fish for the conventional and proposed methods. The conventional method reconstructs high-resolution 3D images using separated voxels whereas the proposed method reconstructs 3D images using red, green, and blue sub-voxels with sub-voxel sampling and a high fill factor.
To verify the color filtering of the proposed method, we performed an experiment with three colored letters in different depth planes of 30 mm, −20 mm, and 10 mm from the color filter pinhole array. The colors of the letters were pink, spring grass, and sky-blue, respectively. The pink color is generated by different 8-bit intensity values of 255 for the red channel, 85 for the green channel, and 170 for the blue channel, the spring grass color by R, G, B values of 170, 255, and 85, respectively, and the sky-blue by R, G, B values of 85, 170, and 255, respectively. As shown in Fig. 12 , the resolution of the reconstructed 3D object in each individual color channel is the same as that of the conventional method, whereas the resolution of the reconstructed 3D object using all the channels of the color filter pinhole array together is enhanced by a factor of three. The reconstructed 3D image has high spatial and angular resolution, high ray density, and a wide expressible depth range with high brightness, as shown in the video clips of Figs. 13(a) (Media 1) and (b) (Media 2).
The viewing angle of pinhole-type integral imaging is limited by the pinhole specifications, whereas the proposed method is additionally limited by the viewing angle of the LCD panel because of the use of the color filters and an LC layer as a pinhole array. To determine the effect of the viewing angle of the LCD panel, we used two different types of LCD panels: a TN LCD panel and an S-IPS II LCD panel. The maximum viewing angle of the S-IPS II panel (178° by 178°) is larger than that of the TN LCD panel (120° by 110°). Figure 14 shows the experimental results for each panel at different observer positions. As shown in Fig. 14, the leftmost and rightmost views of the S-IPS II panel are brighter and provide more vivid 3D images than those of the TN LCD panel. Therefore, pinhole-type integral imaging based on a color filter pinhole array on an S-IPS II LCD panel using projection scheme reconstructs the best quality 3D images of various pinhole-type integral imaging systems. In addition, the proposed method can be easily converted to a 2D display mode by the use of white light images on the display panel, as shown in Fig. 15 .
We proposed a method of improving pinhole-type integral imaging using a color filter array on the display panel with a projection scheme. The use of the projection scheme leads to improvements in the angular resolution and brightness, and the use of a color filter pinhole array on the display panel results in an enhancement of the spatial resolution, ray density, and expressible depth range. The characteristics of the proposed system are improved up to three times in terms of spatial resolution, angular resolution, ray density, and expressible depth range. Experiments and analysis were performed to compare the proposed system with conventional pinhole-type integral imaging systems. The proposed method, based on an S-IPS II panel, displays high-quality 3D images by pinhole-type integral imaging even in a bright room.
This work was supported by the National Research Foundation and the Ministry of Education, Science and Technology of Korea under the Creative Research Initiative Program (#2009-0063599).
References and links
1. G. Lippmann, “La photograhie integrale,” C. R. Acad. Sci. 146, 446–451 (1908).
3. B. Lee, J.-H. Park, and S.-W. Min, “Three-dimensional display and information processing based on integral imaging,” in Digital Holography and Three-Dimensional Display, T.-C. Poon, ed. (Springer, 2006), pp. 333–378.
4. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues [Invited],” Appl. Opt. 50(34), H87–H115 (2011). [CrossRef]
7. J.-H. Park, H.-R. Kim, Y. Kim, J. Kim, J. Hong, S.-D. Lee, and B. Lee, “Depth-enhanced three-dimensional-two-dimensional convertible display based on modified integral imaging,” Opt. Lett. 29(23), 2734–2736 (2004). [CrossRef]
8. H. Choi, J. Kim, S.-W. Cho, Y. Kim, J. B. Park, and B. Lee, “Three-dimensional-two-dimensional mixed display system using integral imaging with an active pinhole array on a liquid crystal panel,” Appl. Opt. 47(13), 2207–2214 (2008). [CrossRef]
10. S.-W. Cho, J.-H. Park, Y. Kim, H. Choi, J. Kim, and B. Lee, “Convertible two-dimensional-three-dimensional display using an LED array based on modified integral imaging,” Opt. Lett. 31(19), 2852–2854 (2006). [CrossRef]
12. Y. Kim, J. Kim, J.-M. Kang, J.-H. Jung, H. Choi, and B. Lee, “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array,” Opt. Express 15(26), 18253–18267 (2007). [CrossRef]
14. J.-H. Jung, Y. Kim, Y. Kim, J. Kim, K. Hong, and B. Lee, “Integral imaging system using an electroluminescent film backlight for three-dimensional-two-dimensional convertibility and a curved structure,” Appl. Opt. 48(5), 998–1007 (2009). [CrossRef]
15. J.-H. Jung, K. Hong, G. Park, I. Chung, and B. Lee, “360°-viewable cylindrical integral imaging system using a 3-D/2-D switchable and flexible backlight,” J. Soc. Inf. Disp. 18(7), 527–534 (2010). [CrossRef]
16. D. Lanman, M. Hirsch, Y. Kim, and R. Raskar, “Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization,” ACM Trans. Graph. 29(6), 163 (2010). [CrossRef]
17. J.-Y. Son, V. V. Saveljev, Y.-J. Choi, J.-E. Bahn, S.-K. Kim, and H. Choi, “Parameters for designing autostereoscopic imaging systems based on lenticular, parallax barrier and IP plates,” Opt. Eng. 42(11), 3326–3333 (2003). [CrossRef]
18. H. Hoshino, F. Okano, H. Isono, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A 15(8), 2059–2065 (1998). [CrossRef]
19. Z. Kavehvash, M. Martinez-Corral, K. Mehrany, S. Bagheri, G. Saavedra, and H. Navarro, “Three-dimensional resolvability in an integral imaging system,” J. Opt. Soc. Am. A 29(4), 525–530 (2012). [CrossRef]