## Abstract

In this paper, we introduce an improved signal analysis of the computational integral imaging (CII) system having a pickup process of three-dimensional object and a volumetric computational reconstruction (VCR) process. We propose a signal model for the CII system. From the signal model and its analysis, we can define a granular noise caused by the non-uniform overlapping. We also analyze the characteristics of the granular noise. According to our model and analysis, there is a condition that the granular noise cancels out. To show the feasibility of our model, the preliminary experiments are carried out and the result is presented. This is the first time, to our knowledge, that a signal model for the analysis of CII systems is provided.

©2007 Optical Society of America

## 1. Introduction

Integral imaging (II) is a sensing and display technique for the visualization of a true three-dimensional (3-D) image. It was first proposed by Lippmann in 1908 and has been actively studied in the literature of 3-D image capturing and visualization [1–9]. The II system records an image of a 3-D object using a lenslet array and an image sensor. The recorded images that are referred to as elemental images are considered to be demagnified images with their own perspective through each lenslet. II reconstruction of a 3-D image from elemental images is a reverse of the recording process.

In II, reconstruction techniques can be classified into two categories: one is based on optical reconstruction [2–9] and the other is based on volumetric computational reconstruction (VCR) [10–17]. When the optical reconstruction techniques are utilized, there are some problems such as the degraded image quality of the displayed 3-D images caused by physical limitations of optical devices such as diffraction and aberration. To overcome these drawbacks of the optical reconstruction techniques, a VCR technique employing a pinhole array model has been introduced in the computational II (CII) system [12]. However, one serious drawback in VCR is that there is a possible artifact due to the nonuniform overlapping [12–14]. The artifact causes the degradation of the visual quality of reconstructed 3-D images. To reduce the artifact, some techniques have been proposed [13–14]. One is based on the lenslet model to improve the resolution of the reconstructed 3-D images by supplying the sufficient overlapping of elemental images [13]. The other is based on a compensation process accomplished by normalizing the number of elemental images overlapped and a moving array lenslet technique [14]. However, they did not address the detailed analysis of the artifact although they proposed the additional compensation process to remove the artifact. And the additional process may require an additional computational cost in VCR.

Recently, various VCR-based CII applications such as 3-D object recognition and a 3-D correlator are reported [15–17]. In 2006, Javidi et al. proposed a concept of VCR-based correlation to recognize 3-D objects that are occluded partially [15, 16]. They showed a good performance of the CII system. Also, in 2007, a novel VCR-based 3-D image correlator to detect the 3-D location coordinates of objects in space was proposed [17]. Even though the usefulness of the CII system was shown, its detailed analysis has not been reported yet.

In this paper, we try to analyze the computational integral imaging. A signal model for the CII system is proposed. To our knowledge, this is the first time to analyze the noise characteristics based on a signal model in the literature. From the signal model and its analysis, we define a granular noise caused by the non-uniform overlapping. According to our model and analysis, there is a condition that the granular noise cancels out. We also analyze the characteristics of the granular noise. To show the feasibility of our model, the preliminary experiments are carried out and the result is presented.

## 2. Overview of VCR in computational integral imaging

The CII system shown in Fig. 1 is composed of two processes: the pickup process of 3-D objects and the VCR from elemental images. In the pickup process of CII, elemental images are recorded by use of a lenslet array and an image sensor. On the other hand, in the VCR process 3-D images are digitally reconstructed from the elemental images by use of a computer where they can be easily reconstructed at any output plane without optical devices. As shown in Fig. 1(b), the elemental images are inversely mapped on the image plane. The elemental images are magnified by a factor of *z*/*g*, where *z* is the distance between the reconstruction image plane and the virtual pinhole array and *g* is the distance between the elemental images and the virtual pinhole array. The magnified elemental images are superimposed on each other and the reconstructed 3-D image is finally produced at the reconstruction plane *z*. To completely obtain the volumetric reconstruction of a 3-D object, this process is repeatedly performed along the *z*-axis.

## 3. Signal model for the analysis of the CII system

To analyze the overall CII system in detail, we construct a one-dimensional (1-D) signal model for CII. This model has a direct counterpart in multidimensional systems and thus it is easily applied to the CII system. Let a function *f _{z}*(

*x*) be an intensity signal in the image plane located at the distance

*z*from the lens array to the object, as depicted in Fig. 2. The signal

*f*(

_{z}*x*) are windowed and reversely mapped through the lens array into the elemental images. This process is referred to as the pickup process of the elemental images in integral imaging. To reconstruct a signal

*r*(

_{z}*x*) in the reconstructed image plane located at the distance

*z*from the virtual pinhole array, elemental images are reversely mapped and magnified by a factor of

*z*/

*g*. And then these magnified elemental images are superimposed on each other in the reconstructed image plane. This reconstruction method is so called the VCR method.

We now focus on the relationship between the signal *f _{z}*(

*x*) and the reconstructed signal

*r*(

_{z}*x*).

*r*(

_{z}*x*) is considered as a linear sum of windowed (or truncated) versions of

*f*(

_{z}*x*). This situation is illustrated in Fig. 3. The pickup process provides each elemental image that is considered to be a windowed, inverted, and downscaled version of the signal

*f*(

_{z}*x*). In the VCR process, these elemental images are again inverted and upscaled before superimposing on them. Thus, the effect of inverting and scaling disappears and that of windowing is left in our model. Hence, the lens array and virtual pinhole-array in Fig. 2 can be eliminated in our model and thus our model is simplified as shown in Fig. 3. Also, a reconstructed signal in VCR is considered to be a sum of windowed versions of the original signal.

Figure 3 provides us a relationship between the signal *f _{z}*(

*x*) and the reconstructed signal

*r*(

_{z}*x*) in the form

where

*N* denotes the number of elemental images, and *w* denotes the size of the windows. Here, Eq. (1) can be easily converted into a discrete version by substituting *x _{k}* for

*x*, where

*k*is a sampling point of

*x*. A shifted window function (SWF)

*π*(

_{i}*x*) is a shifted version of a rectangular window function

*π*

_{0}(

*x*) that equals one in the interval [0,1] and zero outside. Note that the shifting factor

*s*in a SWF is a multiple of the elemental signal (image) length

*a*, that is,

*s*=

*i*∙

*a*, as illustrated in the Fig. 3. Equation (1) states that the reconstructed signal

*r*(

_{z}*x*) can be the original signal

*f*(

_{z}*x*) if the sum of SWFs, denoted by

*S*(

_{π}*x*), is one or constant. In other words, the function

*S*(

_{π}*x*) is the ratio of a reconstructed signal to the original signal

*f*(

_{z}*x*). Hence, the function

*S*(

_{π}*x*) is considered as the response function of a CII system and the characteristics of

*S*(

_{π}*x*) deserve to be analyzed more precisely.

Figure 4 illustrates an example of the superposition or sum of nine SWFs. The size of each shifted window is *w* and the shifting factor is a multiple of the value of *a*. The resulting signal *S _{π}*(

*x*) by superposition of SWFs has two regions; one is a transition region (a portion of the non-homogeneous response function), and the other is a non-transition region (a portion of the homogeneous response function). At the boundary of

*S*(

_{π}*x*), there exists a transition status due to the lacking of the number of the overlapping. The status is treated as a trivial one in this paper.

## 4. Granular noise (GN) and GN-free condition

The response function *S _{π}*(

*x*) of the non-transition region may not be uniform or constant as illustrated Fig. 4. This non-uniformity in the function

*S*(

_{π}*x*) occurs from the fact that the number of SWFs that participate in the superposition process is different along the x axis. We call this number as the overlapping number. And the overlapping number is an integer number given by the function

*floor*(

*w*/

*a*) or

*ceil*(

*w*/

*a*). Based on this property, we define the uniform components and the granular components as depicted in Figs. 4(b) and 4(c). The uniform components have two types: lower level one and upper level one. The related granular components are positive one and negative one, respectively. The granular components are considered to be an unwanted signal because the function is desired to be constant according to Eq. (1). Thus we call these granular components as the granular noise whose detailed definition is given in the section 5.

Let the value *b* be the residual number after dividing the value *w* by the value *a*, as illustrated in Fig. 4(a). This relation is defined by

where *n* is an integer number given by the function *floor*(*w*/*a*) and the value *b* is a real number in the range [0, *a*). Note that the number *n* is the same as the level of the lower uniform component and it is related to the overlapping number. From Eq. (2), it can be easily seen that the granular noise is free when the value *b* equals zero, in other words, the size *w* is a multiple of the value *a*. Here, we consider the geometric relation between the window size *w* and the distance *z* where the reconstructed image plane is located, as depicted in Fig. 2. The relation has the form

It states that the elemental signal (image) of length *a* is magnified by a factor of *M*=*z*/*g* and thus the corresponding window size *w* has to be *aM*. From Eqs. (2) and (3), we obtain that *z*/*g*=*n*+*b*/*a*. This equation implies the value *b* be zero if the ratio *z*/*g* is an integer number. If the value *b* is zero, the levels of the two uniform components are equal to each other and the granular component, referring to Fig. 3(b), has to be zero. Therefore, we can say that the GN-free condition in VCR is achieved when the ratio *z*/*g* or the magnification factor *M* is integer. In fact, a 3D object never can be free of the granular noise in all its depths. However, the GN-free images that are generated at the distance of *ceil*(*z*/*g*) may be enough to construct a volume data if the value *g* is much less than the depth of a 3D object. Then the intermediate images between two consecutive GN-free images can be interpolated by an interpolation method. Also, the value *g* is considered to be a very small number so that it enables the GN-free condition to be useful. To facilitate the understanding of the granular noise, a set of 2D reconstructed images are shown in Fig. 5. Figure 5(c) and 5(b) show the reconstructed images when the GN-free condition is satisfied or not satisfied. It can be easily seen that the reconstructed image of Fig. 5(b) suffers from the granular noise.

If the ratio *z*/*g* is not an integer number, there should be the granular noise in the reconstructed images of the VCR. In this case, a compensation process is required to eliminate the granular noise. To derive the compensation formula, we rewrite Eq. (1) as

The compensation is accomplished by dividing the reconstructed signal *r _{z}*(

*x*) by the function

*S*(

_{π}*x*). This additional process in VCR requires an additional computational cost. And this additional dividing operation requires a higher computational cost than the summation operations of the superposition process. Although the complexity of the compensation process may be relatively small compared with that of the magnification process in VCR, it requires a higher computational cost than the superposition process. For example, assume that the magnification factor is

*M*and the pixel number of a reconstructed image is

*K*×

*K*. Then the number of adders is around

*M*×

*K*×

*K*for the superposition process. If the compensation process is required, there are two ways to implement it. One is only based on calculation. The other is based on calculation with a memory. We discuss the latter in this paper since it is easier to implement it. The memory is required to save the overlapping numbers in the reconstructed image plane. The size of the memory is required to be

*K*×

*K*because the numbers are regionally different. To evaluate the overlapping numbers, counters are required in the superposition process and the number of counters is around

*M*×

*K*×

*K*. Then the compensation is done by dividing the superimposed image by the overlapping numbers in the memory. Thus, the additional operations are

*M*×

*K*×

*K*memory access operations and the

*M*×

*K*×

*K*counter operations besides the division operations. In embedded systems, memory access operations are much more expensive than adders or counters because the speed of external memory access is much slower than the clock speed of CPU. It is desirable to avoid the use of the compensation process in the real time applications, especially embedded systems. Thus, the GN-free condition may enable us to reduce a computational cost and memories in VCR by eliminating the compensation.

## 5. Experiments and discussions

The granular components (GCs) introduced in this paper are periodic signals with period *a*, as depicted in Figs. 4(a)–4(c). It can be seen that the power of the positive GC increases as the value *b* increases whereas that of the negative GC decreases as the value *b* increases. In one case that the value *b* is in the range 0 to 0.5*a*, the power of the positive GC is less than that of the negative GC. In the other case that the value *b* is in the range 0.5*a* to *a*, the power of the negative GC is less than that of the positive GC. Based on this, we define the granular noise by

And the uniform component is easily defined by *round*(*w*/*a*) or *round*(*z*/*g*), which is related to the number of the signals overlapped. To understand the nature of the granular noise, we plot the curve of the granular noise power along the *z*-axis in Fig. 6(a) and that of the uniform component power along the *z*-axis in Fig. 6(b). Referring to Fig. 6(a), the power of the granular noise is periodic along the z-axis and equals to zero at the integer position of the scaled z-axis. The maximum of the power is every half position in the z-axis. The power of the uniform component increases with the square of the overlapping number, referring to Fig. 6(b). Figure 6(c) shows the curves of uniform-signal to noise ratio (USNR) that is the ratio of the power of the uniform component to that of the granular noise, as defined by

where the power of the uniform component is denoted by *P _{U}* and the power of the granular component is denoted by

*P*.

_{gn}Let us investigate two cases: one is a short distance case (e.g., let the ratio *z*/*g* = 3.4) and the other is a relatively long distance case (e.g., let the ratio *z*/*g* =10.4). A reconstructed image located at the short distance *z* = 3.4*g* has USNR=13.52dB whereas a reconstructed image located at the long distance *z*=10.4*g* has USNR=23.98dB. Figure 6(c) indicates this situation. Therefore, it is easily seen that the granular noise effect is more noticeable in a reconstructed image located at a short distance.

## 6. Conclusions and future work

We have introduced an improved signal analysis of computational integral imaging. A signal model for the pickup process and the VCR process in the CII system was proposed. From our signal model, a granular noise was defined and our analysis provided the granular noise-free (GN-free) condition. We also analyzed the characteristics of the granular noise in detail. The proposed model can apply to the optical pickup process of real 3D objects because the 3D objects can be considered as a set of sampled plane images. Further, it is expected that our model can provide various models to analyze various CII systems more precisely. A CII system including VCR can be expressed in terms of a response function that is a summation of SWFs. Thus, the SWF may play a key role to determine the visual quality of the CII system. For future works, hence, another SWF will be applied to VCR and be analyzed on the basis of visual quality.

## 7. Acknowledgment

The authors wish to express their sincere gratitude to the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by a grant from the Dongseo University and in part by a grant from the post Brain Korea 21 project.

## References and links

**1. **G. Lippmann, “La photographic integrale,” C. R. Acad. Sci. **146**, 446–451 (1908).

**2. **F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Three-dimensional video system based on integral photography,” Opt. Eng. **38**, 1072–1077 (1999). [CrossRef]

**3. **B. Javidi and F. Okano, eds., *Three dimensional television, video, and display technologies*, (Springer Verlag Berlin, 2002).

**4. **J.-S. Jang and B. Javidi, “Improved viewing resolution of three- dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. **27**, 324–326 (2002). [CrossRef]

**5. **B. Lee, S. Y. Jung, S.-W. Min, and J.-H. Park, “Three-dimensional display by use of integral photography with dynamically variable image planes,” Opt. Lett. **26**, 1481–1482 (2001). [CrossRef]

**6. **M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, “Multifacet structure of observed reconstructed integral images,” J. Opt. Soc. Am. A **22**, 597–603 (2005). [CrossRef]

**7. **A. Stern and B. Javidi, “3D image sensing, visualization, and processing using integral imaging,” Proc. of IEEE **94**, 591–608 (2006). [CrossRef]

**8. **J. S. Jang and B. Javidi, “Depth and size control of three-dimensional images in projection integral imaging,” Opt. Express **12**, 3778–3790 (2004). [CrossRef] [PubMed]

**9. **D. -H. Shin, B. Lee, and E. -S. Kim, “Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens,” Appl. Opt. **45**, 7375–7381 (2006). [CrossRef] [PubMed]

**10. **H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. **26**, 157–159 (2001). [CrossRef]

**11. **A. Stern and B. Javidi, “3D image sensing and reconstruction with time-division multiplexed computational integral imaging,” Appl. Opt. **42**, 7036–7042 (2003). [CrossRef] [PubMed]

**12. **S. -H. Hong, J. -S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express **12**, 483–491 (2004). [CrossRef] [PubMed]

**13. **D.-H. Shin, E.-S. Kim, and B. Lee, “Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array,” Jpn. J. Appl. Phys. **44**, 8016–8018 (2005). [CrossRef]

**14. **S. -H. Hong and B. Javidi, “Improved resolution 3D object reconstruction using computational integral imaging with time multiplexing,” Opt. Express **12**, 4579–4588 (2004). [CrossRef] [PubMed]

**15. **B. Javidi, R. Ponce-Díaz, and S. -H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. **31**, 1106–1108 (2006). [CrossRef] [PubMed]

**16. **S. -H. Hong and B. Javidi, “Distortion-tolerant 3D recognition of occluded objects using computational integral imaging,” Opt. Express **14**, 12085–12095 (2006). [CrossRef] [PubMed]

**17. **J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, “Resolution-enhanced three-dimensional image correlator using computationally reconstructed integral images,” Opt. Commun. **26**, 72–79 (2007). [CrossRef]