1. Introduction

Holography can reconstruct three-dimensional (3D) scenes, because the light wave distribution of a scene, both intensity and phase, was completely recorded in holograms. However, the traditional holographic recording process requires a coherent light illumination and should be operated in a darkroom. Therefore, obtaining holograms under incoherent light illumination is an urgent and practical desire. Among researches on this topic, a portable approach is synthesizing computer generated holograms (CGHs) from 2D multi-view projection images of a 3D object under white light illumination [1–3]. Depending on a micro-lens array (MLA) embedded in a imaging system, integral imaging (II) can acquire a set of elemental images (EIs) at a single-shot without mechanical scanning or camera matrix [4–6], which offers a convenient acquisition of multi-view images. In fact, the integral-holography (IH) [5] approach still presents some drawbacks, such as the small size of hologram and the low resolution of reconstructed image. Every angular viewpoint image yields one pixel in the hologram, consuming huge calculation time but throwing much redundant information. To obtain a big size hologram by utilizing a small number of multi-view images, some methods which are based on holographic stereogram technique are proposed [3, 4, 7]. Unfortunately, they have the inherent drawback of holographic stereogram that the reconstructed image is composed of patches of projections. To acquire a high resolution hologram, some researchers attempt to increase the sample density on the spatial and angular domain [8, 9]. Since artificially increasing density of sample by interpolation cannot break through the physical diffraction limitation of imaging system, nor bring more details, so the resolution enhancement of these methods is unsatisfactory.

Correspondently, a novel method termed as ptychography iterative engine (PIE) [10] was developed in microscopy in the past decade. PIE was originally employed in coherent diffraction imaging (CDI) to obtain a high measurement precision [11]. It seeks a complex solution that is consistent with intensity measurements under overlap illuminations by phase reversal iteration. PIE did have one uniquely and profoundly important advantage over all other phase-retrieval or imaging techniques: it is not subject to the limitations of the coherence envelope (the information limit) [10], thus remarkable fruitions were brought and published on Nature and Science [12–15]. After replacing the illumination probe function by different angular plane waves and imposing object support constraints in Fourier domain, Guoan Zheng et al. developed a new method on Nature Photonics named as Fourier Ptychography (FP) [16–21]. FP transforms a low-NA conventional optical microscope into a high-resolution wide-FOV microscope with a great throughput. This method begins with a random guess of the high resolution target image, and then considers the measured low resolution images to correspond to various low-pass filtered target images. The initial guess converges to a unique solution which is consistent with the measured images via an iterative approach. Based on this idea, FP stitches together a number of variably illuminated, low-resolution intensity images to recover an accurate high resolution measurement, and extends the resolution beyond the physical limitation of a microscope. Prior work on implementing FP in macroscopic imaging system was demonstrated in [22]. This approach can achieve a complex hologram of an object placed at the far field requiring no interfermetric measurements, but a mechanical scanning and a coherent light illumination are still needed.

FP and IH have same features: both remove the interfermetric measurements in capture stage; both capture intensity images rather than record phase information; both have huge quantity of redundant information in captured position-related images; both try to recover the phase distribution from the correlative intensity images. For the benefit of surpassing diffraction-limitation of FP and being inspired by these similarities, we expand the application of FP into IH, developing a new approach for CGH calculation which can generate big size holograms and produce 3D scenes with high resolution, termed as Fourier ptychographic integral holography (FPIH). In this method, orthographic images (OIs) are firstly generated from EIs captured by II system, and treated as multi-view projection images with low resolution taken from small apertures. Next, OIs fuses into a high resolution complex hologram by their spatial frequency spectrums being shifted and superposed partly. Simultaneously, the hologram is equivalent to the one taken from a big aperture and is iteratively solved through a phase retrieval procedure with FP constrains. Contrary to previous approaches of hologram acquisition and resolution improvement in IH, FP does not work in spatial domain, but synthesizes holograms and enhance resolution by spatial frequency spectrum fusion. FPIH has its unique advantage: it can reconstruct a scene accurately without a complicated algorithm to render the light field of the object or to resample it in spatial domain. To the best of the authors knowledge, it has never been reported that employing FP method in IH to enhance the resolution.

The principle of hologram synthesis and resolution improvement by multi-view image spectrum fusion is explained in Section 2. Constrains required in FPIH are determined accurately and the iterative process is described in detail in Section 3. The results of optical reconstructions from the CGHs synthesized by the proposed method are demonstrated in Section 4. Finally, discussion and conclusion are given in Section 5.

2. Principle of hologram synthesis from multi-projections

Firstly, we give a brief description of II model and note these optical devices and coordinates for later use. Figure 1(a) shows the scheme of an II system. A 3D scene (or an object) is considered as a gather of point sources with each position of (x_o, y_o, z) and is represented by O. N_s × N_t lenses, localized by discrete coordinates (s, t), compose a MLA. And N_s × N_t elemental images are captured on the image plane located at an approximate distance of f₀ from the MIA, where f₀ is the focal length of micro-lenses. Each EI has N_x × N_y pixels, so that EI_s,t(x, y) expresses the intensity of a pixel located at (x, y) in the (s, t) th EI. Δ_p is the pixel pitch in an EI, and Δ_o is the sample interval on object plane. The magnification of II is defined as:

As depicted in Fig. 1(b), we assume that there were a Fourier holographic 4f system, involving a Fourier lens with focal length of f_h and a hologram plane located at the back focus of the Fourier lens, moreover, the hologram plane were at the front focus of a virtual low-NA lens array. All these planes and optical devices are parallel to each other and perpendicular to axial Z. The relationship of II system and IH system is built through a conversion of elemental image array (EIA) to orthographic image array (OIA), as shown in Fig. 1(c). It is the foundation of hologram synthesis and resolution enhancement and is expounded next.

 

Fig. 1 Schemes of II model and multi-projection holography system. (a) The model of II, in which a 3D object parrot and its EIA are shown as well. (b) The multi-projection holography system, in which a high resolution hologram is composed of a set of low resolution regions which produce multi-view projections. (c) The conversion of EIA to OIA is of a link between II and multi-projection holography.

Download Full Size | PDF

In Fig. 1(b), the complex amplitude of a depth slice of the 3D object located at z_o from the Fourier lens is O(x_o, y_o; z_o). (u, v) are the continuous coordinates of the hologram plane and P is the light wave distribution on it, or the complex Fourier hologram. According to the phase modulation for light wave of lens and ignoring constants, P is proportional to the integral of Fourier transform of O(x_o, y_o; z_o) along z_o [23].

Equation (2). means that P can be treated as the spatial frequency spectrum of the object. A complex value P(u, v) represents a plane wave component with spatial frequency of (ξ_u = u/(λf_h), ξ_v = v/(λf_h)) and modulus of |P(u, v)|. λ is the working wave length.

Defining P as a high resolution spatial frequency spectrum taken from a big aperture pupil, a part of P can be regarded as a low resolution spatial frequency spectrum taken from a small aperture. Let p is a rectangle region of P with center of (u_x, u_y) and size of a × b, and p is the output of applying a low-pass filter on P. The low-pass filter is confined within the small apertures and expressed in Fourier domain as rect(u/a)rect(v/b) * δ(u − u_x, v − v_y), where the symbol * is a convolution operation. Because of digital filter theory, an inverse Fourier transform on p implies a spectrum shift of high frequency to low frequency, which results a multiplication of complex exponential signal on O(x_o, y_o; z_o). As Eq. (3) describes:

It can be regarded as that the object O is illuminated by a plane wave exp[j2π(ξ_xx_o + ξ_yy_o)] with incidence angles of (θ_x, θ_y), and then modulated by a limited aperture imaging system, which is a transmission function denoted by a operator of L{}. The relationship between the spatial frequency and the incidence angles is denoted in Eq. (4).

The right part of Eq. (3) equals to a light accumulation of the object in direction of angles (θ_x, θ_y). Since a projection image records the power of object light emitted at an angle, the following equation is deduced:

 

Fig. 2 The principle of synthesizing high resolution holograms from low resolution multi-view images. (a) A numerical reconstruction of two point sources from a high resolution Fourier hologram (b) Reconstructions from the low-pass-filtered hologram array.

Download Full Size | PDF

In presented algorithms, OIs acquired from EIs can be treated as multi-view projections to calculate holograms. An OIA is synthesized from an EIA in following steps: the pixels are collected from each EI at the same local position, and then are arranged into an OI in the order of EI, which is expressed as OI_x,y(s, t) = EI_s,t(x, y). An OIA consisting of N_x × N_y OIs is generated, and each OI has N_s × N_t pixels. The details of projection images generation are provided in [6] and [24].

There is a fact should be noticed that: An EI captured by a real camera is always an inversion of a part of the object. In the case, the OIA formed from inversed EIs has a convex viewpoint distribution. After rotations of EIs by 180°, the OIA formed from un-inversed EIs has a concave viewpoint distribution. In FP microscopy, a specimen is transmitted by variable plane waves while the observe point is fixed. Therefore, the transmission images aligned as an array according the illumination angels are of concave viewpoint distribution. Since the OIA with concave viewpoint is a counterpart of the transmission image array, we can use this type of OIA to synthesize holograms. Additionally, addressing subregion in the target hologram is sequential access and straightforward.

3. Complex hologram generations by modified FP

The multi-view projection holography using FP algorithm is similar to the one using holographic stereogram technique: they both generate a hologram by sub-hologram tiling. But in holographic stereogram, each view image is first multiplied with a random phase mask, and then Fourier-transformed to be of a sub-hologram. These sub-holograms are tiled, neither with gap or overlap, into a big size hologram [3, 7]. There is no iteration in calculation. Since each sub-hologram produces a 2D perspective projection of a 3D scene, the reconstructed image from a stereographic hologram is composed of a set of patches and the depth perception is synthesized in human brain through stereo-vision. Whereas, FP algorithm makes good use of sufficient overlaps as a constraint to stitch together sub-holograms by iteration, creating a single hologram of a 3D scene. Because the unique wave field of an object is acquired, the reconstructed image from a FP hologram is a whole image distributed in 3D space. The reason for recovering a light wave field from correlative images was explained in [10], and the uniqueness of iterative solution of the retrieval was proved in [25]. In this section, we will deduce the constraints required in FPIH and describe the procedure of hologram generations using FP.

Intrinsically, FP is a phase retrieval algorithm with some constraints [10]. The target spectrum which is searched for in FP must fulfill two support constraints: one is modulus constraint and the other is overlap constraint [26]. Modulus constraint is provided by Eq. (5). The overlap constraint is essential to ensure iterative convergence. In this paper, overlap constraint is determined by the projection angle. Each OI_x,y can be regarded as a projection at the view angel of (tan⁻¹(xΔ_p/f₀), tan⁻¹(yΔ_p/f₀)) [6]. The projection angle is equivalent to the incidence angle of the oblique plane wave. Hence Eq. (4) is approximated to:

A discrete spectrum P is obtained by sampling (u, v) into digital coordinates (m, n) with step of Δ_h evenly. The constraints required in FPIH are listed as following:

Combining Eq. (1) and Eq. (8), the interval is expressed as another format:

The overlap region is depends on the distance and the sample interval of the object. According to sampling criterion, the sample interval of reconstructed image is determined by the highest spatial frequency contained in the hologram (For briefness, only horizontal direction is listed):

Comparing Eq. (10) and Eq. (11), the ratio of original sample interval to reconstructed sample interval can be solved:

Obviously, contrary to conventional hologram, the transverse magnification of reconstructed image to original object depends not only on the parameters of hologram but also on the area of overlapping.

The procedure of recovering a complex hologram by FP from an OIA with concave viewpoint is demonstrated in Fig. 3 and explained as follows.

 

Fig. 3 Procedure of complex hologram calculation using FP. The numbers in figure represent the steps of FP. The orange arrow presents a Fourier transform operation while the black arrow presents an inverse one. Several OIs with concave viewpoint are laid out in array format. The red, green and blue rectangles individually enclose a subregion corresponding to an OI labeled by the same color arrow.

Download Full Size | PDF

The error between multi-view intensity images and the produced images of low-pass filtered spectrums in the k-th iteration is measured quantitatively by the following equation:

In order to realize full-color holographic display, OIAs for three color channels should be generated separately. Each color channel OIA produces a hologram. And then in reconstruction, these holograms are illuminated by different color light to composite a color scene. In this paper, we calculate three color channel holograms but superpose them into one hologram and illuminate it by a single wave length laser.

4. Experimental results

We carry out some experiments to verify that a high resolution optical reconstruction can be achieved by the proposed method. To avoid the influence of a degraded EIA on reconstructions, we use computer generated EIAs of virtual 3D scenes to synthesize OIAs. The EIAs are computationally generated by the light ray sample approach [28], which simulates the II process accurately. Hence the imaging parameters can be controlled precisely and the light field of scenes can be captured by the computer generated EIA.

Three experiments are designed. The first experiment validates the capability of 3D holographic reconstruction by FPIH. The other two proves the resolution enhancement of FPIH. They share a common optical reconstruction setup but have individual CGH synthesized from different scenes. The sketch map of experimental setup is shown in Fig. 4, and the configurations of equipments are listed in Table 1.

 

Fig. 4 Optical experimental setup.

Download Full Size | PDF

Table 1. Configurations of equipments

View Table

A SLM is used to export holograms. As the SLM we used is of transmission amplitude modulated type, a complex hologram has to be encoded into an intensity picture by Burch coding:

4.1. 3D scene reconstructions from FPIH

We firstly generate a depth extended object consisting two letters located at different distances in computer. Each letter is a gray 2D picture. Figure 5(a) illustrates the spatial arrangement of two letters with marked distances and dimensions. A total 32 × 32 EIs each with 64 × 64 pixels are computationally generated. Then, a complex Fourier hologram is calculated by FPIH algorithm with N_g of 24 pixels under η = 0.5 and 10 iterations, then encoded into an intensity CGH by Eq. (16). The intensity CGH is loaded into the SLM.

 

Fig. 5 3D scene reconstruction from FPIH. (a) Space alignment of letters in II system, (b) Reconstructed ‘A’ and ‘R’ letters at different distances ( Media 1 Media 2.

Download Full Size | PDF

In this experiment, which depth slice of the 3D scene is focused relies on where the screen is positioned. Figure 5(b) shows photographs of the image on the screen being at two axial positions. The reconstructed image of ‘A’ is in-focus at a position while ‘R’ is off-focus; the ‘R’ is in-focus on another position while ‘A’ is off-focus. Moving the screen axially, we will observe letters being in-focus or off-focus at different distances. This experiment validates that the FPIH indeed acquire the light wave distributions of 3D scenes and reconstruct 3D scenes in space. Note that the diffusing screen causes a degraded photograph.

4.2. Resolution improvement of FPIH

Firstly an EIA is generated through a 3D model of ‘parrot’ rendered in computer graphics (CG) software. The ‘parrot’ image which contains fine structure has appeared in Fig. 1. A virtual pinhole camera array which acts as a MLA faces the model and captures an EIA. The amount of pinhole cameras or EIs is 50 × 50, and each EI has pixels of 32 × 32. Several central OIs synthesized from the ‘parrot’ EIA have already been laid out in Fig. 3. Secondly, complex Fourier holograms are calculated by FP algorithm with Ng of 12 pixels under different iterative parameters, then encoded into intensity CGHs. These intensity CGHs are loaded into the SLM to reconstruct the ‘parrot’ images.

The experimental results are shown in Fig. 6, which validates that FPIH can improve resolution of reconstructed image remarkably. The curve in Fig. 6(a) denotes that as the iterative times increases, the target hologram which begins with a random phase guess is closer and closer to the spatial frequency spectrum of the object; the resolution of reconstructed image is improved. Figure 6(b) is the optical reconstruction under η = 0 and non-iteration. In this case, the proposed approach is equivalent to holographic stereogram method, presented in [3] and [7], just adding a part-superposition between adjacent sub-spectrums. The image is only an overall shape of ‘parrot’. Figure 6(c) shows the optical reconstruction result under η = 0.5 and iteration of 10. It is observed that the feathers on the head of the ‘parrot’ and its feet are too blurred to be recognized in Fig. 6(b), while they emerge clearly in Fig. 6(c). A comparison between Fig. 6(b) and Fig. 6(c) indicates that since the ‘parrot’ becomes clearer and sharper and presents more details, FP improves the image resolution. In Fig. 6(c), the camera captures images being of the zero, the first and the second diffraction orders respectively, at a single-shot. Because the view angles of the fixed camera to each image are different, one can observe stereo-phenomenon.

 

Fig. 6 Reconstructions of a 3D object with fine structures from FPIH. (a) The variation curve of E with iteration times, (b) Reconstructed ‘parrot’ at the second diffraction order under η = 0 and non-iteration, (c) Reconstructed images at three diffraction orders under η = 0.5 and 10 iterations.

Download Full Size | PDF

Another experiment is performed to estimate the resolution improvement of FPIH quantitatively. A US Air force (USAF) resolution target picture, shown as Fig. 7(a), is computational calculated to an EIA. The resolution target has a size of 1750 × 1750 pixels, whose EIA contains 50 × 50 EIs each with 32 × 32 pixels. Figure 7(b) and Fig. 7(c) show the reconstructions from FPIH under different parameters, respectively. The ground noise of Fig. 7(c) is lower than that of Fig. 7(b) obviously. The number ”6” in Fig. 7(c) can be identified, whereas it is dim in Fig. 7(b). The intensity distributions crossing the 3rd, the 4th and the 5th line groups of Figs. 7(a)–7(c) (highlighted by a line on Fig. 7(a)) are plotted in Fig. 7(d). Some facts can be drawn from Fig. 7(d): The peak-valley distribution of the curve with triangle symbol is closer to that of the curve of USAF than that of the curve with circle symbol; At the region of the 5th line group, the curve with circle symbol is so oscillatory that one can not discriminate three major peaks, while the curve with triangle symbol still remains a three-peak-like shape; The resolution of reconstruction is indeed improved by FP algorithm.

 

Fig. 7 Resolution enhancement of FPIH. (a) The USAF resolution target, (b) Reconstruction at the zero diffraction order under η = 0 and non-iteration, (c) Reconstruction at the zero diffraction order under η = 0.5 and 50 iterations.

Download Full Size | PDF

5. Conclusion

In this paper, we propose and experimentally validate an innovative approach for obtaining a high resolution holographic reconstruction from II using FP algorithm. The hologram generated from OIs which are synthesized from EIs captured by II. The hologram begins with a random guess with high resolution, and converges to an accurate complex wave field by phase retrial iteration with FP constrains. The proposed approach enhances reconstruction resolution by expanding the spectrum in spatial frequency domain rather than increasing the sample intensity in spatial domain. As a future work, a real-existed scene should be captured by a light field camera or a MLA and be optically reconstructed in high resolution from a CGH synthesized by FPIH finally. FPIH is also promising in high resolution incoherent measurement.