1. Introduction

Applications of three-dimensional (3D) optical image sensing and visualization technologies span a wide range of areas including TV broadcasting, 3D displays, entertainment, medical sciences and robotics, to name a few [1–3]. An advantage of 3D in relation to traditional 2D imaging techniques is their capability to capture the structural information of the target.

Integral Imaging (II) is an autostereoscopic imaging method used to capture 3D information and to visualize it optically or computationally [4, 5]. We may consider that II started with the works of G. Lippmann [6] and H. E. Ives [7] in the beginning of the 20th century, but only recently these principles have seen all its potential with the advancements in optoelectronic (CMOS and CCD sensors), display devices such as liquid crystal displays (LCDs), and the tremendous increase in computing power [8–10].

Acquisition and visualization of 3D objects can be divided into two different steps, called pickup and reconstruction. In the pickup stage, multiple 2D images (elemental images) are captured through an array of small lenses (lenslet array). Each lenslet contains a unique projective transformation that maps the 3D object space onto 2D elemental images and is a function of the lenslet position and the focal length. As a result, an array of inverted real images is formed on the image sensor. Scene reconstruction can be made optically or computationally. Computational reconstruction of the 3D image can be achieved by simulating the optical backprojection of elemental images in the computer. This reconstruction method uses a computer synthesized virtual pinhole array for inverse mapping of each elemental image into the object space. All elemental images are computationally overlapped afterwards. With this process, the intensity distribution can be reconstructed at arbitrary planes inside the 3D object volume.

Multispectral imaging consists of the acquisition of images in the portion of the spectrum extending from the visible region through the nearinfrared and midinfrared. These techniques have recently found applications in remote sensing [11–13], medical imaging [14, 15], fine arts [16], just to cite a few.

Integration of 3D and multispectral information may be important in: a) underwater 3D visualization [17], since water absorption is wavelength dependent, b) dermatology [18], for instance, in skin cancer detection, because melanoma pigmented skin lesions are wavelength dependent and have a 3D structure, c) remote sensing applications, for sensors onboard airplanes or satellites that may be able to create 3D models with the inclusion of multispectral information, d) remote sensing pattern recognition (i. e., identification of the 3D structure and the spectral response of an object form the distance), e) in photon starved or ”hard to visualize” conditions (at night, in foggy conditions, etc.) [19]. Multispectral 3D reconstruction has already been applied in microscopy [20]. Other 3D acquisition and visualization techniques have already incorporated multispectral information [21–23].

In this paper we present 3D multispectral bands integral imaging, including scene capture and the 3D reconstruction results. The acquisition of the elemental images is made using a monochrome camera and a liquid crystal tunable filter (LCTF), applying the Integral Imaging technique, at different wavelengths from 480 to 680nm. Section 2 describes the methods used for the multispectral integral image acquisition. Section 3 explains the methodology applied to obtain the 3D profile of the imaged scene. Section 4 shows the 3D reconstruction results at different depth planes as well as the reconstructed scene profile. Conclusions are given in Section 5.

2. 3D Multispectral integral image acquisition

The multispectral 3D integral image acquisition system used is shown in Fig. 1. It consists of a Marlin F080B model camera, whose CCD sensor size is 4.80 × 3.62mm (and the image size is 1032 × 778 pixels) and a Liquid Crystal Tunable Filter (LCTF) covering the [400, 720]nm spectral range. It can acquire up to 33 bands with a 10nm spectral resolution. A Canon TV zoom lens is attached to the front part of the filter, and a Macro Schneider system [24] is used between the filter and the camera. The illumination system is a Dolan-Jenner MI–150 fiber optic illuminator system with a 150W bulb lamp (3250 Kelvin colour temperature). A uniform diffuser is attached to the fiber optic guide in order to homogenize the illumination. The system was adjusted to focus at a distance of 209mm from the Canon TV zoom lens. The effective focal length of the system was f = 10.35mm, the lateral magnification was M = 0.052 and the resulting angular field of view was 21.8°. The depth of field was big enough to ensure that all parts of the 3D scene were imaged sharply.

 

Fig. 1 Integral Imaging data acquisition system setup.

Download Full Size | PDF

The acquisition time per band used for the LCTF filter is assessed using an ideal reflectance diffuser object, called spectralon. Its reflectance curve is flat for the visible part of the wavelength spectrum. A Region-Of-Interest (ROI) is selected for the spectralon to avoid the influence of some defects in its surface. The mean grey scale value of the ROI per band is used for the calibration. This time is adjusted to obtain the same grey scale value for all the wavelength bands. An automatic calibration method was implemented. For one band, the acquisition time and grey level value are considered to increase linearly, if illumination and distance to object do not vary. The method consists of selecting two (time, grey – level) pairs not allocated at extreme values, and assessing the exposure time considering a linear dependence between the grey scale value and the exposure time.

2.1. Integral Imaging Image Capture

Lenslet-based II systems are diffraction-limited in spatial resolution due to the small numerical aperture of the lenslets. In essence, three parameters must be considered: the camera pixel size, the lenslet point spread function, and the lenslet depth of focus [25–27]. In addition, aberrations can negatively affect the resolution. Nevertheless, II can also be performed using a single 2D imaging sensor, scanning the aperture and capturing intermittent images over a large area. This approach is described as synthetic aperture integral imaging and overcomes some of the limitations of traditional lenslet-based integral imaging systems [28, 29]. This is the approach we followed in our paper. Other data capture approaches may be used as well. For example, see [30].

3. Volumetric reconstruction

Once the elemental images per wavelength were acquired, we applied the computational method developed in [31] to obtain the 3D reconstruction. This method starts from the elemental images and projects them directly through a virtual pinhole array. In particular, for a fixed distance z = L from this pinhole array, each elemental image is inversely projected through each synthesized pinhole in the array. Each inversely projected image is magnified applying the magnification factor M, where M is the ratio of the distance between the pinhole array and the plane of reconstruction at z = L, to the distance between the pinhole array and the elemental image plane (z = g). With this definition: $M=\frac{L}{g}$. In order to assess the volumetric information, we repeat this process for all the reconstruction planes. Let us consider EI_pq the pth row and the qth column elemental image, and EM_pq(x, y, z) the elemental image corresponding to EI_pq that has been inversely mapped at (x,y,z). It was shown in [31] that:

 

Fig. 2 Image reconstruction step using a projection of the elemental images through a virtual pinhole array

Download Full Size | PDF

The distance between the objects of the scene and the sensor can be inferred as well. Let us consider each elemental image as an information source of the light distribution at each volumetric pixel in space, and define the spectral radiation pattern (SRP, ℒ(θ, ϕ, λ)) in order to capture the radiation intensity at a certain wavelength and direction. In [32] DaneshPanah et al. define a profilometry operator, and assume that object surfaces are Lambertian or semi-Lambertian. This implies that the following deviation metric:

4. Reconstruction results

Moving the camera in an array of positions is equivalent to moving the scene in positions of the same grid. Due to the weight of the optical capture system and the specifications of the small motors, we chose to move the scene to capture the elemental images (see Fig. 3). In particular, we moved the scene in a regular 11×11 grid. Applying the calibration method described in the second paragraph of Section 2, we found that 510 seconds are needed to acquire the 33 bands for any of the positions in the grid. Therefore, we finally decided to acquire elemental images only in seven wavelengths of {480, 510, 550, 570, 600, 650, 680} nm for this grid. Acquisition time in this case was 56 seconds per position.

 

Fig. 3 Image acquisition using the Integral Imaging technique.

Download Full Size | PDF

The pitch of the grid was fixed to p = 4.2mm in both horizontal and vertical direction. On the other hand the sensor size was 4.80×3.62mm. Consequently, the elemental images overlapped in the horizontal direction but not in the vertical one. The lack of overlapping is not important for reconstruction planes far from the camera. Figure 4 shows a subset of the acquired elemental images for wavelength λ = 650nm. Inverse-projecting each elemental image through the virtual pinhole array {see [31] and Section 3}, we can obtain the corresponding images at the different perpendicular planes (z) of interest. Figure 5(a)–(c) shows false colour RGB reconstructed images (using the grey scale images and the following assumption 480nm ← B, 550nm ← G and 650nm ← R) of the 3-dices scene at values of the optical axis depth (z) equal to the positions for which the rear face of each dice is in focus. Reconstruction results are available for any wavelength inside the filter spectral range and therefore the spectral information of the scene is available as well. Figure 6(a) shows the corresponding 3D profile for the scene of Figure 6(b), which is obtained by applying the method explained in [32]. The z axis represents the distance (in mm) from the acquisition system to the objects that define the scene. The x and y axes represent the lateral pixels of the scene. We can infer from the 3D profile that each plane corresponding to closest face of each dice is at a different and well defined distance. However, there are some points in which the estimation quality is lower, which may be due in part to the specular reflections on the surface of the dice. In order to assess the quality of the recovered depth information, we considered the pixels that belong to the rear orthogonal face of each dice. We applied the Root Mean Square Error formula (defined as: $RMSE=\sqrt{\frac{1}{NP}{\sum }_i{\sum }_j{\left(D_{E,ij}-D_{L,ij}\right)}^2}$, where NP means total number of pixels, D_E,ij means the depth obtained with the minimization of variance method, and D_L,ij refers to the depth measured in the laboratory) as a measure of error. In this case the RMSE for all the pixels considered is RMSE_Total = 13.37mm. For each dice, RMSE_Blue = 13.57mm, RMSE_Green = 6.03mm, and RMSE_Red = 16.52mm.

 

Fig. 4 Elemental images corresponding to positions: (2, 1), (2, 6), (2, 11), (6, 1), (6, 6), (6, 11), (9, 1), (9, 6), and (9, 11), for λ = 650nm.

Download Full Size | PDF

 

Fig. 5 False colour RGB reconstructed images at planes corresponding to z = 230, z = 249 and z = 267mm. The colour images were created using the spectral assumption: 480nm ← B, 550nm ← G and 650nm ← R.

Download Full Size | PDF

 

Fig. 6 (a) 3D profile of the 3 dice scene for λ = 650nm. (b) Elemental image corresponding to position (6, 5).

Download Full Size | PDF

On the other hand, the mean and standard deviation of the depths are: $\overline{D_{E,\mathit{red}}}=\left(252\pm 11\right)\text{mm}$, $\overline{D_{E,\mathit{green}}}=\left(262.11\pm 6.03\right)\text{mm}$, and $\overline{D_{E,\mathit{blue}}}=\left(232.8\pm 4.5\right)\text{mm}$. The corresponding relative errors for the blue, red and green dices were: 5.8%, 6.6%, and 2.3%, respectively.

From the above results, we can see that the error is highest for the red dice. This may be because it is in the middle of the three dices and the region on the left may be blocked by the blue dice in some of elemental images. Therefore depth estimation may have a higher error.

5. Conclusion

We have shown the viability to implement a multispectral 3D integral imaging acquisition and reconstruction system. We have used a monochrome camera and a liquid crystal tunable filter, for the data acquisition stage, a computational volumetric reconstruction method [31] for the 3D reconstruction, and a minimisation of the variance algorithm among elemental images [32] for the creation of the 3D profile of the scene. These results open the way to the analysis of the relationship between the spectral and the spatial content among elemental images. These relationships are important in order to develop compression algorithms for these datasets. 3D and spectral information can also be combined to help improve classification accuracy. This would have an immediate and important effect in research areas like melanoma cancer detection, where it has already been shown that the best way to classify them is using a combination of volumetric and spectral information [33].