Popular range cameras are classified into stereo and time-of-flight (TOF) cameras. A stereo camera simulates a human binocular vision using two or more lenses with a separate image sensor for each lens. However, the stereo camera cannot avoid a number of inherent limitations: (1) multiple lenses and the corresponding sensors increase the cost and volume of the system; (2) computationally expensive pre-processing algorithms are need to align and match the stereo images, and (3) a large disparity between the stereo images results in occlusion or nonmatched holes. A TOF camera consists of an illumination unit to emit the light source and an image sensor to measure the time that the light travels to the object and reflected back to the focal plane array [1–3]. Although the TOF camera can quickly estimate the distance, the distance estimation range is limited, and it is impossible to estimate the distance to a transparent object.

In order to overcome the drawbacks of the stereo and TOF cameras, a light field camera is equipped with multiple microlenses, and a dual aperture camera consists of an image sensor with infrared pickup pixels and a special aperture that blocks infrared [4–6]. These two cameras can reconstruct a 3D input scene using depth information under a single-image sensor. However, a very high-precision manufacturing process is needed.

An aperture is an important optical device that controls the amount of incoming light to the camera and determines the depth of field. A circle of confusion or point spread function (PSF) is generally dependent on the size and shape of the aperture. In addition, if the center of the aperture deviates from the optical axis, the position of the image on the sensor moves according to the distance of an object. In order to use the characteristics of a special aperture, a coded aperture and color-filtered apertures were proposed in the literature [7,8]. Among various approaches to use a special aperture, an off-axis aperture can easily generate the disparity using a single camera and, as a result, can estimate the distance without additional optical devices [9,10]. Since a fixed type off-axis aperture can control neither the amount of incoming light nor change the shape of the PSF, a flexible generation of the aperture shape and location is needed.

A liquid crystal display (LCD) consists of a polarizing filter, a thin-film transistor, and liquid crystal. For this reason, it is a good candidate to produce a flexible aperture that can control the amount of incoming light, shape of the PSF at the same time, and colors. A coded aperture and light field cameras were produced using the LCD [11–13]. The major issue of the LCD aperture is the diffraction caused by the pixel grid of the LCD. In order to estimate the distance using a single camera, this Letter presents a computational camera system that can easily change the color, size, and location of the aperture using an LCD. We also analyze the diffraction of the LCD to formulate the image degradation model of the camera. Using the LCD imaging property, we can realize a special type of aperture that has variable sizes, locations, and colors.

As shown in Fig. 1(a), a cell of the color LCD consists of a pair of orthogonally polarizing filters, a pair of glass plates, and three pairs of color filters and liquid crystal cells on the thin-film transistors (TFT). The upper and lower glass plates protect liquid crystal cells, and directions of two polarizing filters are orthogonal to each other. Since the inactive liquid crystal has the codirectional molecular structure, light cannot pass through the color LCD cell with an orthogonal pair of polarizing filter, as shown in Fig. 1(a). However, when the liquid crystal becomes active by the electric signal generated by the thin-film transistor, the upper and lower molecular structures become orthogonal to each other, and the rotated light in the liquid crystal can pass through the pair of orthogonally polarizing filters.

 

Fig. 1. Structure of an LCD and diffraction phenomenon: (a) cell of a color LCD and (b) light trajectories in the LCD aperture imaging.

Download Full Size | PDF

Figure 1(b) shows an imaging system with an LCD aperture, where each cell of the LCD consists of red, green, and blue color filters. However, the light passing through the LCD aperture is diffracted by the narrow grating of an LCD cell.

Assuming that the light source is far enough from the camera, the polarization light wave is considered to be orthogonal to the direction of the propagation. Based on Fraunhofer’s diffraction grating theory, the angle of diffraction is expressed as

The diffracted light rays generate virtual images, as shown in Fig. 1(b), and the distance between the real and $n$-th virtual images is determined as

The proposed range finding camera uses an LCD to build a programmable coded aperture. As shown in Fig. 2(a), the LCD aperture is installed between the front and rear lenses of the camera. The LCD can play a role of multiple color-filtered apertures since a white pixel lets the entire light pass the LCD, and a pixel with a specific color lets the corresponding color pass the LCD. We can generate any of the colors, shapes, or positions of the LCD aperture by simply generating a desired pattern image on the LCD.

 

Fig. 2. Structure of the optical range-finding camera system: (a) proposed camera using the LCD aperture, (b) light paths of three different object positions using an off-axis aperture, and (c) light paths of three different object positions using dual color-filtered apertures.

Download Full Size | PDF

The proposed system uses multiple off-axis-type apertures for distance measurement. The off-axis aperture is located away from the optical axis, as shown in Fig. 2(b). A conventional camera with a single aperture on the optical axis makes the image of any of the points on the optical axis at the center of the image sensor. On the other hand, the off-axis aperture makes the image of a scene point on the optical axis at different positions in the image sensor depending on the distance of each point from the lens. Figure 2(c) shows the dual color-filtered aperture (DCA) imaging system. Since the DCA can be considered as a double extension of the off-axis aperture with complementary color filters, a single point on the optical axis at an unfocused area makes two differently located images on the corresponding color channels. The disparity between the two color channels is determined by the distance of the point from the in-focused position. Therefore, the distance between the camera and the object can be estimated using the relationship between the distance and disparity. An accurate estimation of the disparity between the two different color channels is a challenge because the existing image registration methods estimate the disparity between homogeneous color channels. An existing approach to solve this problem is the use of the color-shifting model at the cost of a nontrivial computational complexity [9,14].

In addition, the diffraction property of the LCD apertures generates a special image degradation model. Figure 3(a) shows the captured image from conventional camera system. The point spread function (PSF) is very close to the impulse function at the focused region, and becomes the Gaussian function at the unfocused regions. However, as shown in Fig. 3(b), the proposed system has the PSF convolved the impulse train function to the PSF of conventional camera system. The impulse train function means the mathematical representation of diffraction phenomenon caused by the patterns of LCD pixels that has the highest weight value at the zero-th order of diffracted light.

 

Fig. 3. Diffraction property of the proposed camera system: (a) acquired image using a general camera system and (b) image acquired by the proposed camera system. (a) and (b) are focused at the triangle object.

Download Full Size | PDF

The proposed computational camera system uses two horizontally symmetric off-axis apertures to solve this problem. Since a pair of color stereo images is acquired using the two LCD off-axis apertures, additional color registration and rectification are not needed.

The proposed camera system estimates disparities between the two color images using the normalized cross-correlation (NCC) [15]. Since the NCC measures the similarity between two images based on variance and deviation, the result is robust to a certain amount of intensity difference. The disparity is measured using the NCC as

Finally, the distance of the object is computed using the relationship with the disparity. Lee et al. proposed a single optical range-finding system using the optical characteristics of off-axis apertures to estimate the distance as [10]

For a precision measurement, we evaluate the obtained disparities from the optical parameters of the off-axis aperture. The ground truth of the real disparity is computed by modifying Eq. (4) as [10]

A prototype of the proposed optical range-finding system is shown in Fig. 4. We inserted a 45.7 mm TFT-LCD with a resolution of $160\times 128$ into a 50 mm fixed focus lens module after removing the backlight module. A microcontroller unit (MCU) on top of the prototype system controls the LCD to generate the desired aperture images using switches, as shown in Fig. 4(b). As a result, the prototype system can flexibly generate various types of apertures by activating the group of sub-pixels, as shown in Fig. 4(c).

 

Fig. 4. Prototype of the proposed optical range-finding system: (a) right side of the system with an LCD aperture that is inserted between the lens and the camera body, (b) MCU and switches on top of the system, and (c) front view of the system and the LCD aperture.

Download Full Size | PDF

To evaluate the performance of the prototype system, we used a test scene with a white triangular object, as shown in Fig. 5(a), and compared the estimated distance with an existing off-axis aperture proposed by Lee et al. [10]. We acquired multiple pairs of stereo images by moving the object at the distance from 320 to 920 mm with an interval of 25 mm, and estimated the disparity and distance using Eqs. (3) and (4), respectively. In order to measure the disparity without using color registration, a white object of Fig. 5(a) is used. Figure 5(b) shows a color-shifted image acquired by the DCA camera system, where the disparity is measured between the R and B channels. A pair of stereo images acquired by the proposed two off-axis apertures is shown in Figs. 5(c) and 5(d).

 

Fig. 5. Experiment to generate stereo images using the two different types of off-axis apertures: (a) input scene with a white triangular object at the distance of 520 mm, (b) acquired image using the DCA [10], and (c) and (d) pair of acquired stereo images using the proposed left and right off-axis apertures, respectively.

Download Full Size | PDF

Figure 6(a) shows the result of the estimated disparity using the stereo images shown in Fig. 5, where the x- and y-axes represent the object distance and the estimated disparity, respectively. In spite of the image degradation caused by the diffraction in the LCD aperture, the proposed system can correctly estimate the disparity that is almost equal to the actual value, and it is called the ground truth in this Letter. The estimated distance using the disparity is shown in Fig. 6(b), where the straight line represents the ground truth. The experimental results show that the average difference from the ground truth is approximately 6 mm.

 

Fig. 6. Estimated disparity and distance using the test image shown in Fig. 5: (a) ground truth and two estimated disparity curves using the DCA and the proposed system, and (b) corresponding distances computed form the disparity shown in part (a).

Download Full Size | PDF

To test whether the proposed system can measure color-invariant distance, we performed another experiment using a colorful object, as shown in Fig. 7(a), under the same conditions of the first experiment. Since each color channel has different intensity distributions for the same object, as shown in Figs. 7(b) and 7(c), the existing DCA system cannot accurately estimate the disparity, as shown in Fig. 7(f) and, thus, the estimated distance is incorrect, as shown in Fig. 7(g). On the other hand, the proposed system can generate a set of color stereo images using two off-axis apertures and successfully estimate the disparity and distance, as shown in Figs. 7(d) and 7(f), respectively.

 

Fig. 7. Distance and disparity measurement results using a color object: (a) input scene, including the colorful object, and (b) and (c) stereo images from DCA. (d) and (c) Stereo images from the proposed two off-axis apertures, and (f) and (g) estimated corresponding disparity and distance curves, respectively.

Download Full Size | PDF

This Letter presents an optical range-finding system with an LCD to generate a programmable coded aperture. The proposed system can easily change the shape, location, and color of multiple apertures by simply controlling the LCD. In addition, it can measure the distance of an input scene under various environments since the LCD apertures can adapt to change the exposure, the baseline, and the color of the aperture in real time. The experimental results showed that the proposed camera system could robustly measure the disparity between the intentionally misaligned color channels in spite of the inherent image degradation caused by the LCD’s diffraction.