1. Introduction

The Gabor holography technique [1] has been widely applied to address numerous problems in optical engineering, e.g. full three-dimensional (3D) imaging and display [2–5], for biomedical applications [6], particle image analysis [7, 8], and information security [9–11]. However, despite receiving such attention holographic television, for example, remains a work in progress, because of the large quantities of data to be processed in real time and the complexity of the optoelectronic systems necessary. Thus, a more practical alternative approach to 3D imaging and visualization is highly desired. Integral imaging (II), based on Integral Photography (IP) [12], seems to be one such alternative, providing a promising approach for 3D object sensing, recording and autostereoscopic visualization [13–17]. Using this technique, a 3D scene can effectively be reconstructed by recording two-dimensional (2D) Elemental Images (EIs) from different perspectives of the same 3D scene. II is therefore analogous to an incoherent version of conventional holography [16]. Further, II systems can be used to reproduce detailed versions of 3D scenes that can be comparatively better than those produced using conventional stereoscopic based imaging techniques. Importantly, II is much easier to perform than classical hologram based 3D imaging.

Photon-Counting Imaging systems (PCI) have recently received a great deal of attention because of their many useful properties, (i.e. sparse sensing, nonlinearity, reduced power consumption and simultaneous bandwidth reduction), see for example [18–21]. Most of the studies involving PCI have been carried out for single light frequencies using achromatic data. However, such studies do not illustrate what happens in the case of real-time chromatic (full color) information. Therefore recently, a model describing the capture and visualization of multispectral 3D scenes in a photon starved environment, using a Bayer patterned CCD sensors, was proposed [22].

The rapid development of communication systems indicates the need for both higher levels of data security and intellectual property protection. Data protection techniques, such as steganography, watermarking and encryption, are in increasing demand. The simplicity and elegance of the classical Fourier based 4f Double Random Phase Encryption system, (DRPE), has led to proposals for numerous optically inspired digital encryption techniques over the past two decades [23–29]. The DRPE transforms an input signal into a noise-like complex-valued distribution. Optical implementation of this encryption technique provides additional security as coherent illumination and precise light field alignment are necessary. Further, optically it is possible to perform unambiguous information processing [26].

On the other hand, digital implementations of the DRPE have been studied in detail and have been shown to be vulnerable to various types of attack [30–32]. Numerous attempts to alleviate the security problems associated with the digital DRPE technique have been made over the past decade. Recently, a new encryption technique has been proposed that increases security by exploiting the sparse sensing and nonlinear transformation properties of PCI system combined with the conventional DRPE system (DRPE-PCI) [33, 34]. This technique introduces significant additional layers of security.

Furthermore, extension of the classical 2D based encryption techniques, which can be used to significantly increase 3D image data (scene) security, have also been demonstrated [35–37]. Recently, Cho et al. [38] have extended a combined DRPE-PCI technique to secure a 3D scene. In their method, the Fourier encrypted complex Elemental Images (EIs) were further secured using the photon counting detection model and it was shown that the resulting decrypted data could be used to realize 3D scene reconstructions. It also has been demonstrated that both higher security and 3D data authentication can be achieved by combining the complex-valued DRPE distribution and the photon counting Poisson nonlinear transformation [38]. However, in many practical applications, light intensity measured at the camera is highly sensitive to the input signal phase or the key phase. In such scenarios, the classical optical holography setup is used, in which the phase is encoded in the intensity and must be reconstructed [1]. Clearly, any encryption system that simultaneously reduces hardware requirements while providing greater security would be highly desirable.

Recently, a new encryption system employing the classical Hartley transform (HT) has been proposed [29]. Using this system, a real 2D input signal is transformed into a purely real valued 2D form (i.e. intensity image data). The resulting system neither compromises security nor reduces the system’s reliability. Furthermore, encryption can be realized using either coherent or incoherent illumination [29, 39–41]. Additionally, it has been reported [42-44] that the resulting incoherent optical system implementation outperforms the corresponding coherent based systems, since it does not suffer from scattering effects (i.e. speckle noise). Therefore, using such a system the computational complexity is reduced and no incoherent to coherent conversions are necessary. In practical terms the system can operate with a variety of inputs (i.e. luminous objects, CRT displays) [42].

In this study, we propose to combine the HT based encryption system with the Photon Counting Integral Imaging (PCII) system to achieve secured 3D scene transmission. Such a system will work for incoherent light, provide secured 3D data transmission and requires a simplified processing architecture. Furthermore, the system should allow decryption and verification of the 3D scene at different depth cues (i.e. various scene depths).

The paper is organized as follows. In Section 2, a brief overview of the optical HT based encryption system is presented. The passive multispectral PCII technique and the use of the Maximum Likelihood Estimation (MLE) for the reconstruction of the 3D scene are briefly reviewed in Section 3. The combination of the HT with PCII is discussed in detail in Section 4. Simulation results, involving the use of experimental data, which illustrate the system performance, are presented in Section 5. Finally, a brief conclusion is given in Section 6.

2. Optical Hartley Transforms (OHT)

Over the last two decades various optically inspired encryption algorithms, starting with the Fourier transformation (FT) based DRPE system, have been proposed and studied. Additionally, several techniques have been proposed in which the full phase information of a complex data set, i.e. an optical field, must be recorded. One widely used method for capturing complex optical signals involves use of a holographic setup in which the phase information is encoded as intensity data [1]. Although, the holographic recording technique is very useful, capturing and processing holographic data (e.g. encrypted images) is difficult.

Chen et al. [29] have demonstrated a HT based optical image encryption technique, which neither requires holographic image capture nor requires the use of coherent illumination. While the HT is similar to the FT, the implementation involves the use of optical interferometry, which real 2D input image signals are transformed into purely real output encrypted data. In such a case the intensity detectors used to capture the output (i.e. CCD cameras) directly record the transformed scrambled data (i.e. the encrypted data) enabling unambiguous reconstruction in both the optical and digital implementation cases. Therefore, in this sense, the HT outperforms the FT based systems. In general, the HT,$H\left(u,v\right)$, of the 2D image$g\left(x,y\right)$, is defined as follows [41]:

where the $\text{cas}$ function is defined as $\text{cas}\left(\theta \right)=\cos \left(\theta \right)+\sin \left(\theta \right)$ [41]. Based on this definition, the HT can be deduced as follows:

where $u,v$ are the coordinates in the HT output domain. $F\left(u,v\right)=FT\left\{g\left(x,y\right)\right\}$, and $F\left(-u,-v\right)$ represent the forward and inverse FT, respectively. As noted, the HT output is real, (i.e. a real-valued function is transformed into a real-valued function under the HT), and therefore no complex arithmetic is required during data processing. A number of feasible optical implementations of the HT are presented in [41]. In what follows HT based encryption systems are examined. In [29], the HT based encryption process is described mathematically as follows:

where $\psi \left(u,v\right)$ and $r\left(u,v\right)$ denote the encrypted image and a random intensity mask respectively. Decryption can be performed using the reciprocal of the intensity mask$\left(r^{-1}\left(u,v\right)\right)$, and thus the original intensity image information can be retrieved as follows:

From Eq. (3), it should be noted that encryption converts the input data into scrambled output data (i.e. cosine and sine functions). Attackers cannot easily obtain the encoded information without access to the correct decryption keys [29].

3. Multispectral PCII with MLE reconstruction

Integral Imaging (II), is a passive 3D imaging technique, in which both the irradiance and directional information of the light rays measured are recorded. The principle of operation of 3D II is illustrated in Fig. (1).

 

Fig. 1 3D II Principle: (a) Pick-up (recording) process, and (b) Reconstruction (display) process. A Bayer CCD camera (GRBG pattern) is used in the experiment illustrated.

Download Full Size | PDF

II involves two major operations: The first: (a) is the pick-up process, and the second: (b) is the reconstruction process. In the pick-up process, imaging camera is shifted laterally across a rectangular grid in steps of equal distance in $x$ and$y$,$(S{h}_x,S{h}_y)$, and at each location a 2D image is captured. These 2D images, each of which contains a different perspective view of the entire 3D scene, are commonly known as Elemental Images (EI’s). During reconstruction, the set of captured 2D EI’s are used to realize or recreate the original 3D scene. One of the possible approaches to 3D II reconstruction is to simulate the reverse of the pick-up process using a geometrical ray back propagation method, i.e. computational volumetric reconstruction. In this approach, the 3D object located at a particular distance is reconstructed at the corresponding image plane. As a consequence each EI appears magnified, depending on the distance to the desired reconstruction plane. Furthermore, only the objects originally located at a particular distance will appear clearly in focus. Any objects positioned in other planes appear blurry and out of focus. The magnification factor is given by$M_0=z_0/d$, where $d$ is the distance between the pick-up grid (i.e. camera) and the imaging plane, and $z_0$ is the object distance from the lenslet array (see Fig. 1).

A method of capturing and processing multispectral 3D photon-counted integral images, using Bayer patterned imaging sensors, has recently been demonstrated [22]. Colorful photon-counted images can be generated by limiting the expected number of incident photons in the entire Bayer patterned scene. In general, the probability of counting photons (i.e. estimating the number of photons) in an image scene is governed by the Poisson distribution [22]. Suppose the total number of photons (i.e. the photon count) in the entire encrypted image is$n_p$, then the probability of counting $L_w$ photons at any arbitrary pixel point in an image scene, at the point,$\left(x,y\right)$, is given by the following expression [34]:

In Eq. (5), the subscript variable $w$ is used to indicate either the red, green or blue channels of the captured Bayer EI, and ${\lambda }_w$ is the Poisson parameter at any arbitrary pixel point on each spectral channel, which is computed as ${\lambda }_w\left(x,y\right)={\overline{\psi }}_w\left(x,y\right)\times n_p$ [34].

In this paper, the $Poissrnd(·)$ a built–in MATLAB function is used, to generate random numbers having a Poisson distribution, in order to estimate the photon counts in the normalized encrypted elemental image. Here, the normalized irradiance of an encrypted data is defined as ${\displaystyle {\sum }_{q=1}^N\psi (·)}=1$, where $N$is the total number of pixels in the same encrypted image. Applying this function, the photon-limited Bayer EI for each color channel can be obtained as follows [22]:

$P_w(·)$ denotes the number of photons at the particular pixel$\left(x,y\right)$, $n_p$ is the expected number of photons in the Bayer EI and ${\overline{\psi }}_w\left(x,y\right)$ is the normalized encrypted image, respectively.

After appropriate decryption, photon-counted 3D object reconstruction is performed as follows: We assume that the 3D object, which is located at distance $z_0$ away from the pick-up grid, is captured at pixel $\left(x,y\right)$ of the first sensor. Computational reconstruction of the 3D object is then performed by applying the parametric Maximum Likelihood Estimator (MLE) to the decrypted photon-counted Bayer elemental image set [21, 22]:

where $k+\Delta k_{pq}=\left(x+p\left(\frac{1}{M_0}\right)S{h}_x,y+q\left(\frac{1}{M_0}\right)S{h}_y\right)$ . The subscripts $p,q$ indicate the location of elemental image in the $P\times Q$ position locations in the pick-up grid. $C_{pq}(.)$ is the photon counted pixel value of the ${\left(p,q\right)}^{th}$elemental image. We note that a detailed description of multispectral 3D PCII recording and reconstruction is presented in [22].

4. HT based 3D photon counting encryption

In this section, we discuss the combination of the HT with PCII to produce a new encryption technique, (see the schematic of the proposed system in Fig. 2). In a previous study [38], PCII was combined with the FT based DRPE technique. The resulting system produces a complex-output distribution, (i.e. amplitude and phase), for which data recording and transmission is difficult. Using the HT, real data is encrypted into an amplitude only intensity modulation facilitating unambiguous reconstruction [40]. Several feasible optical implementations for HT are proposed in [41].

 

Fig. 2 The proposed optical setup for performing the HT and capturing and securing the 3D photon limited scene.

Download Full Size | PDF

It should be noted that the output of the classical optical Hartley transform contains both positive and negative values. Since any digital camera placed in the output plane will only measure the intensity pattern, such sign changes are not directly recorded. To recover the full set of scrambled data real values, first the square root of the intensity is taken to extract the field amplitude and then appropriate sign values must be assigned at each point. We note that several methods have been proposed in the literature to solve this sign ambiguity problem, see for example [41].

It has been shown that the Hartley transform can be implemented optically using a system similar to that shown in Fig. 2 following Eq. (2). As shown the primary object beam propagates along through a thin Fourier lens of focal length (f) and along one arm to the optical mirror (M) then it reflects back to the beam splitter (BS) and finally the output, $F\left(u,v\right)$, is produced at output encrypted image plane. Simultaneously half the field, split off by BS, is incident upon the cube corner, which rotates the beam by${180}^0$, producing $F\left(-u,-v\right)$ at the output plane. An additional $\pi /2$ phase delay, see Eq. (2), can be introduced by appropriately displacing the mirror. Furthermore, we note that the Hartley transform based optical interferometry system (i.e. both the Michelson type and Mach-Zehnder type) architecture is highly sensitive to the distances between, i.e. the positions, of the components within the system. For example, both the cube corner and the plane optical mirror must be placed at the same distance relative to the Fourier domain [40]. Any errors (displacements from the correct positions) in the setup will lead to incorrect data reconstruction.

It is worth re-emphasizing that the resulting encrypted output is real valued. Figure 2 illustrates the proposed 3D photon counting optical encryption setup. As shown, the primary input light field is that of an Elemental Image Array (EIA), and the synthesized encrypted EIA are what are processed using the PCI detection model. The decrypted data is then used for 3D sectional reconstruction as explained in Section 3.

Furthermore, for the experiments discussed, the Bayer GRBG patterned elemental images captured are converted into multispectral images with greater visual quality. This is achieved by applying an adaptive interpolation algorithm, proposed by Malvar [45], to convert the resulting Bayer patterned 3D sectional images into multispectral 3D scenes. This algorithm estimates the missing spectral pixel values by combining the calculated gradient value of a particular pixel with bilinear interpolated spectral information. Detailed descriptions of the conversion of Bayer GRBG data to multispectral 3D scene data are given in [22] and [34].

It should be noted that the photon-limited decrypted 3D images cannot be recognized by direct visual inspection due to the low number of photons present. Therefore, in order to evaluate the information content present a k^th-law Nonlinear Cross-correlation (NC) filter is used. To carry out this operation efficiently the signals or images to be compared are in general first transformed from the spatial domain into the frequency domain (i.e. Fourier transformed). Following application of the nonlinearity and multiplication of the two signals an inverse Fourier transform of the synthesized product yields the NC between them. The NC is implemented as follows (1D notation is used for brevity) [38]:

where $F\left(\mu \right)$ represents the Fourier transform of the primary image [see Fig. 3(b)] and${\varphi }_{g_p^{d_0}}\left(\mu \right)$, ${\varphi }_F\left(\mu \right)$ are the phase values of the photon-counted 3D sectional image [see Fig. 4(c) and 4(d)] and the primary 2D image, respectively. The parameter $k$ defines the applied NC nonlinearity. We note that it has been estimated [34] that the best value of$k$, i.e. the value which achieves the sharpest intensity peak value, is$k=0.3$. We note that NC results presented are normalized as follows: $NC=\left|c\left(x\right)\right|/\max \left\{\left|c\left(x\right)\right|\right\},$in our experiments.

 

Fig. 3 Test images used in our experiments: (a) Bayer patterned (“GRGB”) 2D elemental image; (b) Interpolated multispectral image; (c) Photon-counted Bayer image with 1000 photons, and (d) The corresponding multispectral photon-counted image.

Download Full Size | PDF

 

Fig. 4 Multispectral visualization of reconstructed 3D sectional images: PSNR is calculated between the computational integral image and the photon-limited integral image: (a) 3D sectional image with the first object in focus; (b) 3D sectional image with the second object in focus; (c) Photon-limited sectional image with first object is focused (PSNR = 32.95 dB); and (d) Photon-limited sectional image with second object is focused (PSNR = 30.54 dB).

Download Full Size | PDF

5. Experimental results

In order to verify the proposed HT based PCII encryption system experimentally measured data and appropriate simulations are used. Following experimentally capture of the 3D object images, the encryption, decryption and MLE based 3D reconstruction are implemented digitally using virtual optical simulations. In this study, two differently colored 3D objects are used see Fig. 3.

These objects are located at two different distances, (i.e. 540 mm and 620 mm, respectively), from the center of the imaging grid. 14 × 14 Bayer-patterned 2D elemental images (8 bits) are captured by moving the imaging system in equal steps of 5 mm in the horizontal and vertical directions. The entire 3D scene was illuminated by a broad band incoherent light source (i.e. diffused incandescent bulbs). The EIs were recorded using a camera with a pixel array of$2048\left(H\right)\times 2048\left(V\right)$, (i.e. in each EI image frame).

Figure 3(a) shows one of the captured Bayer 2D EIs and Fig. 3(b) show the resulting Malvar [45] interpolated multispectral EI of the 3D scene. Similarly, Fig. 3(c) shows the photon-counted version of the primary Bayer image while Fig. 3(d) is the resting interpolated multispectral photon image. It is evident from Fig. 3(b) and 3(d) that estimating the multispectral based information is important as it reveals both the luminance and chrominance information of the captured scene even under photon starved conditions.

Figures 4(a) and 4(b) show the 3D reconstruction of the multispectral computational slice images,$I_{MCII}$, at two different depth positions. Similarly, Fig. 4(c) and 4(d) shows the 3D photon-counted sectional images,$I_{MPCII}$, at these two focused positions.

It is evident from Fig. 4 that at each distance only one of the objects is clearly in focus while the other object appears smeared (i.e. defocused). Thus, the correct depth must be utilized to clearly visualize the 3D object or to quantitative extract information. Depth can therefore act as a key value for the decryption process.

In addition to this, recognizing the objects in the 2D image format when only 1000 photons are available is quite challenging, [see Fig. 3(d)]. However, application of 3D sectional imaging using MLE calculation at low photon count, (i.e. with 1000 photons present), demonstrates that visually recognizable data can be extracted, [see Fig. 4(c) and 4(d)].

Figures 5(a) and 5(b) shows the general 2D HT encrypted and HT based photon-limited image. The corresponding decrypted images are shown in Fig. 5(c) and 5(d), respectively. Note that the photon-limited decrypted version, Fig. 5(d), cannot be easily visualized, (recognized as being present), due to the low number of photons present.

 

Fig. 5 Encrypted and decrypted data: (a) HT based encrypted 2D elemental image; (b) Photon-limited HT encrypted EI; (c) Decrypted 2D elemental image; and (d) Decrypted 3D sectional image (n_p = 1000).

Download Full Size | PDF

As noted, the photon-limited decrypted sectional images (see Fig. 5(d)) not easily recognizable due to the low number of photons. Thus, to avoid any ambiguity, (i.e. to test whether the required information is present), an NC filter is used for data authentication. The decrypted sectional images of true class, (i.e. the objects considered in this experiment), are compared with the decrypted sectional images of false class, (i.e. a different object set). Figure 6(a) shows one such false class 2D elemental image. Figure 6(b) shows the corresponding decrypted version photon-limited false class sectional image. Both the decrypted sectional images, Fig. 5(d) and Fig. 6(b), are not visible and it is therefore difficult (impossible) to differentiate between them. Therefore, we apply the NC and examine the resulting correlation characteristics, shown in Fig. 7.

 

Fig. 6 False class images: (a) the primary false class 2D image; (b) The decrypted false class 3D sectional image.

Download Full Size | PDF

 

Fig. 7 Nonlinear correlation (NC) values versus number of photons (10^x) results using a k^th law nonlinear processor (k = 0.3): Highest peak values obtained are marked within each figures using 3D surface plot in MATLAB. (a) NC values for the 2D case; (b) NC values for the 3D case; (c) NC values for 2D false class images, and (d) NC values for 3D false class data set. In Fig (b) and (d) the blue colored lines, (i.e. solid lines), refers to the reconstruction of the first object at d = 540 mm while the orange colored lines (i.e. dashed line) refers to the reconstruction of second object at d = 620 mm. The corresponding peak NC values are indicated in the figures as values inside round brackets, e.g. (0.15) in Fig. 7(c).

Download Full Size | PDF

The calculated NC values for the true class are shown in Fig. 7(a) for the 2D scenes and in Fig. 7(b) for the 3D scenes, respectively. It has previously been demonstrated that scrambled samples (i.e. encrypted data) requires more photons to validate the scene [34]. We note here that in the 2D multispectral image case a good trade-off between the photon numbers and the applied nonlinearly can be achieved when n_p = 10^3.5 and k = 0.3, in this case values of NC > 0.6 are typically found. However, for 3D scene authentication, it is found that values of n_p = 10³ and k = 0.3 provides good correlation (NC > 0.7). These results reflect the fact that the 3D scene can provide more detailed information than the conventional 2D image case under similar low illumination conditions. Figures 7(c), and 7(d) present the computed NC values for the false 2D and 3D cases, respectively. It can be seen that the NC values are less than 0.2 in both cases and thus recognition and authentication does not take place. We note that the NC used is normalized, i.e. has a maximum value of unity, in order to make comparison easier.

These results demonstrate that our combination of HT based encryption system with PCII provides robust data protection and introduces an additional layer of protection to the information security. Furthermore, the system is capable of both authenticating and discriminating between the true result and false decrypted 3D scenes recorded using different objects.

6. Conclusion

A new optical method for encrypting 2D as well as 3D multispectral photon-limited scene data using a classical Hartley transform (HT) based encryption system has been proposed. The HT converts an input signal into real valued coefficients of cosine and sine functions and can be optically implemented. The parametric maximum likelihood estimation (MLE) algorithm has been applied to decrypted photon-limited elemental 2D images to achieve 3D scene reconstruction in a Bayer format. A gradient corrected linear interpolation technique has been employed to convert Bayer patterned 3D sectional images into a multispectral 3D scene. Thus, 3D objects can be recognized and their full spectral information retained even under low light level conditions.

In summary, by combining classical HT with multispectral PCII, a color 3D scene can be encrypted without the need to extract the phase of complex light fields using holographic or interferometric phase unwrapping techniques.