1. Introduction

Fringe projection profilometry (FPP), as a non-contact three-dimensional (3D) shape measurement technique, has becoming more prevalently adopted in a variety of applications, including manufacturing, medical imaging, computer vision, education, bio-medicine, and virtual/augmented reality (VR/AR) [1,2]. Many mature instruments have been developed to measure static objects based on FPP. Recently, with the rapid development of image sensors and digital projection technology, it becomes possible to realize high-speed, real-time 3D shape measurement of dynamic objects by using FPP [3]. Different from those techniques for applications with static objects, the primary problem in this field is to recover the 3D information of moving objects or dynamic scenes with high speed, accuracy, and reliability. To this end, much research has been done targeting real-time, high-performance 3D shape measurement, and considerable progresses have been made during the past few years [4,6–8].

In high-speed, real-time 3D shape measurement based on FPP, the sinusoidal fringe pattern is the most frequently used projection pattern because using the phase information has the merits of robustness to sensor noise, surface reflectivity variations, and being able to achieve high spatial and/or temporal resolutions. The phase of the fringe is usually retrieved by Fourier transform algorithm [4,9,10] or phase-shifting algorithm [5,6,11]. Fourier transform algorithm extracts the phase measurement from a single fringe image by applying a properly designed bandpass filter in the frequency domain, while the phase-shifting algorithm uses a minimum of three fringe images and offers much higher measurement accuracy. 3D shape measurement methods using these two algorithms are referred as the Fourier transform profilometry (FTP) and the phase-shifting profilometry (PSP), respectively. Both methods have the capability of high-speed real-time measurement. This capability can be improved in the following two aspects: (1) increasing the speed of hardware (projector and camera); (2) increasing the efficiency of absolute phase retrieval (or matching point retrieval). The first aspect focuses on using projector defocusing technique to generate the sinusoidal fringe patterns from binary ones so that the projection speed can be increased to the maximum frame rates of the digital mirror device (DMD), e.g. kHz [10,12] or even tens of kHz [13,14]. Then with the assistance of high-speed camera, it is not difficult to reduce the motion artifacts between two adjacent captured frames. The major concern in this aspect is to explore what kind of binary pattern is suitable for achieving high-accuracy phase measurement with a slightly de-focused projector [15–19]. The second aspect tends to retrieve absolute phase or search the matching points by using as few fringe patterns as possible [14,20–22,30–42]. How to ensure the robustness of phase unwrapping with reduced patterns is the essential problem in this aspect [5]. Besides these two aspects, removing the motion artifacts and reducing the motion vulnerability of multi-shot PSP based on error compensation algorithms have also attracted increasing attention recently [22–26]. In practical applications, the above mentioned three aspects can be combined to improve the accuracy of dynamic measurement for moving parts. In this work, we focus more on the second aspect and want to enhance the robustness of absolute phase retrieval without requiring additional image acquisition.

Temporal phase unwrapping (TPU) is the most popular technique to retrieve the absolute phase map which may contain large discontinuities and isolated surfaces [27–29]. However, TPU generally requires extra fringe/graycode patterns, which decreases the efficiency of 3D measurement in high-speed, time-critical scenarios. To maximize the measurement efficiency of PSP, Weise et al. [22] introduced a novel phase unwrapping method where the geometric constraint between different views (two cameras and a projector) is used to retrieve the absolute phase. This method is termed as the stereo phase unwrapping (SPU) in our paper. There are no restrictions on the shape of the measured objects and no requirements of extra projection in SPU. This could be hardly realized in any other conventional techniques, such as spatial/temporal phase unwrapping techniques. It is well known that the low-frequency sinusoidal fringes have less phase ambiguities, tending to make the phase unwrapping more reliable. However, on the other hand, increasing the frequency of the fringe pattern is essential to achieve high-precision 3D shape reconstruction. Using conventional SPU is not enough to robustly eliminate phase ambiguities when high-frequency fringes are used. In the works of Weise et al. [22] and Garcia et al. [30], graph cut and loopy belief propagation (LBP) were employed as auxiliary means to further correct the fringe orders errors after SPU when high-frequency fringes are used. Digital image correlation (DIC) is a different spatial algorithm to enhance the robustness of SPU due to the high spatial distinguishability of speckle patterns [31,32]. Notni et al. [33] took the limited measurement volume of FPP system into account, and as a result, they set a depth volume to preclude some phase ambiguities before the process of SPU. However, since the high density of the fringes used, graph cut algorithm is inevitable in order to obtain an error-free result. The similar idea was also employed by Li et al. [34] where the fringe density is properly selected to make sure that only few candidates fall within the pre-defined measurement volume. According to Li’s work, a narrow depth range should be set to ensure the reliability of SPU when high-frequency fringes are used. The idea in Notni and Li’s work is the so-called depth constraint. Based on the depth constraint, Tao et al. [35] integrated the composite phase shifting scheme into SPU. Reliable real-time 3D measurement result was obtained over a large measurement volume at the cost of reducing the amplitude of the fringe. Liu and Kofman [36] also developed a 3D shape measurement approach that uses high-frequency background modulation fringe patterns generated based on the depth constraint. Song et al. [37] used the passive stereo matching method to generate a coarse depth map of the measured scene, which then serves as the depth volume for SPU to reconstruct a high-resolution depth map. Besides, the depth constraint has also been applied to conventional monocular FPP system (consisting of only one camera and one projector) to restrict the search range for possible fringe orders and rule out several false candidates in TPU [14, 38–40]. Besides the depth constraint, Notni’s work [41] and Tao’s recent work [42] suggest that SPU can also be further refined by optimizing the relative positions between the projector and cameras.

From above discussion, we know that SPU is usually implemented accompanying some complicated and time-consuming spatial domain processing algorithms, such as graph cut [22,30–33,41] and DIC [31,32]. However, in practical applications, these algorithms still need to be simplified in order to realize better real-time performance. On the other hand, the depth constraint is an effective approach to improve the performance of SPU and has been extensively used in combination with other phase retrieval approaches in many recent work [14,33–42]. The key problem in the depth constraint is how to set a suitable depth range. When a fixed depth range is used, there is an inherent trade-off between the measurement range and the robustness of the phase unwrapping. To guarantee the robustness of phase unwrapping, the fringe patterns are usually designed with a relatively low frequency, resulting in a low measurement precision. On the other hand, if high-frequency fringe patterns are used, the measurement range will be significantly compromised as the object has to be placed within a very limited depth range.

To overcome the above mentioned limitations, we propose an adaptive depth constraint (ADC) approach for high-speed real-time 3D shape measurement, where the measurement depth volume for geometric constraints is adaptively updated according to the current reconstructed geometry. During the real-time measurement process, we first analyze the statistical characteristics of the raw depth map, and use the analysed results to update the global depth volume. Meanwhile, some outliers can be filter out to obtain a refined depth map. Based on this refined depth map, then we focus on the depth map of a neighborhood of each pixel and create a pixel-wise depth volume map to be an input of the depth constraint in the next cycle of 3D measurement. Compared to the conventional approaches based on fixed depth volume, the pixel-wise depth volume in our method is more compact, adaptive to the object shape, and adaptively updated over time. That means our method cannot only ensure the robustness of SPU but also have a wide measurement volume. To make this adaptive depth constraint more flexible and robust, we develop two auxiliary techniques including a simplified left-right consistency check and a feedback mechanism of valid depth area. The simplified left-right consistency check tends to help ensure the reliability of raw depth map. The feedback mechanism is used to deal with the case of the abrupt depth change, such as a new object entering the field of view. All these techniques constitute the whole computational framework for enhancing the robustness of SPU. The effectiveness and real-time performance of this method are validated by several experiments.

2. Principle

In this section, we will focus on the basic principle of the proposed method. In Section 2.1, we first introduce the framework and the inadequacy of SPU to more conveniently explain the principle as well as the motivation of this work. Then we propose the core idea of our method which is called ADC in Section 2.2. In the following Sections 2.3 and 2.4, the simplified left-right consistency check and the feedback mechanism based on the valid area are detailed as two complementary algorithms for ADC.

2.1. Basic principle of SPU

A typical FPP system using SPU is composed of two cameras and a projector. The fringe patterns are projected onto the measured object by the projector, and then deformed by the object, and finally captured by two cameras. A phase map is extracted from the deformed fringe patterns to search the sub-pixel matching points and retrieving depth information. Taking three-step phase-shifting fringe patterns for example, these fringe patterns captured by the cameras can be expressed as the following formulae:

In SPU, the K^c is retrieved with geometric constraint. To the best of our knowledge, it is first used in Weise’s work [22]. Figure 1 displays the basic principle of SPU. o^c₁ is an arbitrary point in the first camera (c₁) with the coordinate (u^c₁, v^c₁) and the wrapped phase ϕ^c₁(u^c₁, v^c₁). For the sake of brevity, o^c₁ is used to substitute for (u^c₁, v^c₁) in some expressions. We first sequentially assign the integers within the interval [0, N − 1] to K^c₁(o^c₁), and notate K^c₁(o^c₁) assigned with different values as $K_0^{c_1}(o^{c_1}),K_1^{c_1}(o^{c_1})\dots K_n^{c_1}(o^{c_1})\dots K_{N-1}^{c_1}(o^{c_1})$. Each $K_n^{c_1}(o^{c_1})$ corresponds to a ${\mathrm{\Phi }}^{c_1}\left(o^{c_1},K_n^{c_1}(o^{c_1})\right)$ according to Eq. (9) ($K_n^{c_1}(o^{c_1})$ will be simplified as $K_n^{c_1}$). Then a total of N 3D points can be derived from Eqs. (4) – (7), and the n-th 3D point is notated as o^w(o^c₁, $K_n^{c_1}$) with the coordinate $\left(X^w(o^{c_1},K_n^{c_1}),Y^w(o^c,K_n^{c_1}),Z^w(o^{c_1},K_n^{c_1})\right)$. All these 3D points are called the 3D candidates of o^c₁, and the only 3D matching point is included in these candidates. The blue lines from the projector in Fig. 1 denote part of the rays with the same wrapped phase ϕ^c₁(o^c₁) but the different absolute phase Φ^c₁(o^c₁, $K_n^{c_1}$). These blue lines from the projector and camera c₁ intersect at different o^w(o^c₁, $K_n^{c_1}$). The 3D candidates can be projected into the second camera c₂ to get its corresponding 2D candidates o^c₂(o^c₁, $K_n^{c_1}$) with the coordinates (u^c₂(o^c₁, $K_n^{c_1}$), v^c₂(o^c₁, $K_n^{c_1}$)). There is also a 2D matching point within these 2D candidates. o^c₁ and its 2D matching point should have the similar properties, such as the phase, the texture and so on. Keep this in mind, the 2D candidate with the closest wrapped phase to ϕ^c₁(o^c₁) is chosen to be the matching point. Once the 2D matching point is determined, the fringe order of o^c₁ will be known. This is the basic principle of SPU.

 

Fig. 1 Diagram of SPU and conventional depth constraint (CDC).

Download Full Size | PDF

However, we must consider the effects of the discrete property of the camera, imperfect system calibration and the noise in an actual experiment. All these effects will introduce errors to our system and decrease the similarity between o^c₁ and its 2D matching point. On the other hand, there may exist more than one 2D candidates having the similar phase to ϕ^c₁(u^c₁, v^c₁) if the high-frequency fringes are used. In this case, some matching errors will emerge during the process of SPU. We can use a scheme called phase consistency check [42] to reject some 2D candidates whose phase difference with ϕ^c₁(u^c₁, v^c₁) is larger than a threshold. As shown in Fig. 1, the 3D candidates corresponding to red lines from c₂ are rejected while those candidates corresponding to blue lines are reversed. Some additional techniques are required to distinguish the matching point from the reserved candidates. Several kinds of techniques have been proposed, and the review of these techniques can be seen in the introduction. Only the depth constraint will be described in detail to explain both some basic principles and the initial motivation of our method.

2.2. Adaptive depth constraint

At the beginning of this section, we will introduce the principle of CDC. Next, to resolve the contradiction of CDC, we introduce the proposed methods including the global adaptive depth constraint (GADC) and the pixel-wise adaptive depth constraint (PWADC).

2.2.1. Global adaptive depth constraint

Considering limitations of the depth of focus and the common field of view, the measurement volume of FPP system must be restricted to a finite range. Different systems should have different measurement volumes. But the theoretical measurement volume of a specific system is difficult to determine, so it is always replaced by a larger experimental volume. The physical meanings of Z_min and Z_max in Fig. 2 represent the minimum and maximum depth boundaries in the world coordinate, respectively, and their units are mm. The volume between Z_min and Z_max is the measurement volume. In CDC, the measurement volume usually serves as the depth volume. If we can confirm that the measured object is located in the depth volume, we can use

 

Fig. 2 Diagram of the cross-section of Fig. 1

Download Full Size | PDF

In order to resolve the contradict between the measurement volume and robustness, we propose a GADC technique. The main idea of this technique is to analyze the statistical properties of the current depth map, and update a new depth volume for the next cycle of 3D measurement. In GADC, note that the measurement volume [Z_min, Z_max] will be independent of the depth volume and remain constant. We first averagely divide the measurement volume (or initial depth volume) [Z_min, Z_max] into Q intervals, and the q-th interval is notated as [Z_min+qΔZ, Z_min+(q + 1)ΔZ]. q is an integer and belongs to [0, Q −1] while $\mathrm{\Delta }Z=\frac{Z_{\mathit{max}}-Z_{\mathit{min}}}{Q}$. H(q) is used to denote the number of points in [Z_min+qΔZ, Z_min+(q + 1)ΔZ]. We can easily obtain the histogram of the current depth map by

 

Fig. 3 Diagram of GADC for measuring single object

Download Full Size | PDF

2.2.2. Pixel-wise adaptive depth constraint

The effectiveness of GADC depends on the updated depth volume while this depth volume is determined by the object. Let us consider the case of measuring a large-size object or several isolated objects. In this case, the dynamic depth volume $\left[Z_{\mathit{min}}^{\mathit{global}},Z_{\mathit{max}}^{\mathit{global}}\right]$ may not be so superior to initial depth volume [Z_min, Z_max], as shown in Fig. 4. Although the reconstructed errors in H(q − j) can still be corrected, more errors cannot be removed. The effectiveness of GADC will decrease obviously. That is a result of only two depth boundaries being used. A more flexible depth volume should be created to handle this case, and this is actually another technique called PWADC that we will introduce in the next paragraph.

 

Fig. 4 Diagram of GADC for measuring two isolated objects

Download Full Size | PDF

Just as its name implies, PWADC allocates each pixel with its own independent depth volume. We will describe its principle based on Fig. 5. As shown in Fig. 5, the green point denotes an arbitrary point o^c₁ in camera, and the red dotted rectangle is the 5 × 5 neighbourhood around o^c₁. Each small grey rectangle represents a pixel in this neighbourhood, and the green rectangle is o^c₁. The numbers in grey rectangles represent depth of the pixel. The maximum and minimum depth will be used as the boundaries of depth volume $\left[Z_{\mathit{min}}^{\mathit{pixel}}(o^{c_1}),Z_{\mathit{max}}^{\mathit{pixel}}(o^{c_1})\right]$, where $Z_{\mathit{min}}^{\mathit{pixel}}(o^{c_1})=-20-\mathrm{\Delta }{Z}^{\mathit{motion}}$, $Z_{\mathit{max}}^{\mathit{pixel}}(o^{c_1})=10+\mathrm{\Delta }{Z}^{\mathit{motion}}$. The depth difference raised by motion is addressed from two aspects: (1) the pixel-wise depth volume is created based on a neighbourhood instead of a single pixel; (2) the depth allowance ΔZ^motion is added. Note that there is an important step that should be implemented at the very start. That is we should use the global depth volume $\left[Z_{\mathit{min}}^{\mathit{global}}+\mathrm{\Delta }{Z}^{\mathit{motion}},Z_{\mathit{max}}^{\mathit{global}}-\mathrm{\Delta }{Z}^{\mathit{motion}}\right]$ to set some outliers inactive. Assuming the red rectangle in Fig. 5 is the inactive pixel, then the actual minimum boundary of depth volume of o^c₁ should be $Z_{\mathit{min}}^{\mathit{pixel}}(o^{c_1})=1-\mathrm{\Delta }{Z}^{\mathit{motion}}$. The final depth volume of o^c₁ in Fig. 5 is [1 − ΔZ^motion, 10 + ΔZ^motion]. The whole process can be summarized as following formulae

 

Fig. 5 Diagram of PWADC for measuring two isolated objects

Download Full Size | PDF

2.3. Simplified left-right consistency check

The proposed ADC technique provides a strict but accurate depth volume to enhance the robustness of SPU. However, there still exist a few unreliably reconstructed points within the pixel-wise depth volume, especially around the contours of the moving object. Besides, it is possible that the number of outliers in the current depth map is larger than H_min, and in this case, the outliers will not be set inactive. To present the occurrence of these cases, a simplified left-right consistency check technique is proposed.

Left-right consistency check is a frequently used technique in stereo vision to detect and remove matching errors. For SPU, we should independently calculate the fringe orders of all valid points in c₁, and c₂, and then check the consistency between fringe orders of points in c₁ and those of their matching points in c₂. The points without the consistency of fringe orders will be finally set invalid and removed from subsequent processing. Though the left-right consistency check often works quite well, a double computation time of SPU is required, which imposes a heavy burdens on real-time measurement. In this paper, we proposed a more efficient left-right consistency check to achieve better real-time performance without compromising the accuracy.

Almost all the errors of SPU are induced due to the existence of two or more candidates being reserved after the depth constraint and phase consistency check. This has been explained in detail in Section 2.1. There does exist some cases that only false matching points are reserved after phase consistency check, but these cases are very rare, especially in a multi-view system. In this paper, we do not consider these rare cases. Figure 6(a) displays the distribution of the number of candidates. In the orange area, there is only one candidate while in the red area the number of candidates is more than one. The points located in orange area in Fig. 6(a) are first projected into c₂. The orange area in Fig. 6(b) denotes the matching points of that in Figure 6(a). The white area in Fig. 6(b) is the blank area without any matching points for the points in orange area in Fig. 6(a). The green point o^c₁ is an arbitrary point in the red area. Two green points o^c₂(o^c₁, k) and o^c₂(o^c₁, k − 2) in Fig. 6(b) are the 2D reserved candidates of o^c₁. There are two basic principles for the simplified left-right consistency check: (1) it is an one-to-one correspondence between the points in c₁ and the points in c₂; (2) the area whose points have only one candidate is the reliable area, as shown in Fig. 6(a). Point o^c₂(o^c₁, k − 2) is obviously not consistent with principle (1), and we confirm that the correct matching point of o^c₂(o^c₁, k − 2) is located in the orange area in Fig. 6(a) instead of o^c₁. On the other hand, o^c₂(o^c₁, k) located in the blank area satisfies the one-to-one relationship with o^c₁. As a result, o^c₂(o^c₁, k) is chosen to be the matching point. There exist some special cases that all the candidates of o^c₁ satisfy the principle (1), or none of the candidates satisfy the principle (1). In these cases, o^c₁ will be set invalid. Let us assume that the value of the reliable area and blank area in Fig. 6(b) is 0 and 1, respectively, and the number of reserved candidates is N^r. Then for an arbitrary point o^c₁ whose N^r ≥ 2, if it satisfies

 

Fig. 6 Diagram of simplified left-right consistency check

Download Full Size | PDF

2.4. Feedback mechanism based on the valid area

Now let us consider another special case that a new object enter the adaptive depth volume $\left[Z_{\mathit{min}}^{\mathit{pixel}}(u^{c_1},v^{c_1}),Z_{\mathit{max}}^{\mathit{pixel}}(u^{c_1},v^{c_1})\right]$ but within the initial depth volume [Z_min, Z_max]. Obviously, this new object will not be correctly measured based on the above procedures. Therefore, an additional feedback mechanism is required to detect and handle this case. In this paper, we calculate the valid area within $\left[Z_{\mathit{min}}^{\mathit{pixel}}(u^{c_1},v^{c_1}),Z_{\mathit{max}}^{\mathit{pixel}}(u^{c_1},v^{c_1})\right]$ and [Z_min, Z_max], respectively. The so-called valid area is the region having at least one candidate after the depth constraint and the phase consistency check. The numbers of the points in these two valid areas are notated as S₁ and S₂. Then we can detect the appearance of the new object according to whether the following inequality is satisfied or not,

In order to clearly show the process of the proposed method, a flowchart is given in Fig. 7. The green modules denote the outputs, while the red modules denote the main algorithms.

 

Fig. 7 Flowchart of the proposed method

Download Full Size | PDF

3. Experimental results

Several experiments are designed to verify the validity of the proposed techniques, including ADC, the simplified left-right consistency check and the feedback mechanism based on the valid area. We also separately use quad-camera and dual-camera systems to implement the real-time measurement to display the compatibility of our method. All the cameras used in our system is Basler acA640–750um with the highest 750fps at the full resolution 640 × 480. The cameras are outfitted with 12mm Computar lenses. The projector is LightCrafter 4500Pro with the resolution of 912 × 1140 and projection speed of 120Hz. In our experiments, the projection speed is 100Hz, and all the cameras are synchronized by the trigger signal from the projector. The related parameters and thresholds are set as Z_min = −200 mm, Z_max = 200 mm, ΔZ^motion = 10 mm, H_min = 500, r = 5, S_min = 500.

3.1. Measurement of single object

In the first experiment, a David statue with complex surface was measured to verify the robustness of SPU. A quad-camera system was used with the projection of 48-period phase-shifting fringe patterns. The retrieved absolute phase maps and the reconstruction results are shown in Fig. 8. In the measuring process, the statue was first moved toward the measuring system, and then left static for few seconds. During the static stage, the absolute phase map acquired by multi-frequency PSP in Fig. 8(a) serves as the ground truth to detect the errors in Fig. 8(b) – 8(d). The quantitative numbers and ratios of falsely unwrapped points are counted and displayed in the right bottom in Fig. 8(b) – 8(d). Here the missing points beyond the common field of views are set inactive and removed from the statistics. The erroneous regions are marked by the red dotted circles. We can easily find that: (1) compared to CDC, the robustness of ADC increases obviously; (2) there is no difference between the performance of GADC and PWADC when measured such a small single object. Besides, The comparison between Fig. 8(f) and Fig. 8(e) suggests that the falsely reconstructed points are far from their true positions. All the experimental results are consistent with our analysis in Section 2.2. More detailed measurement results of the moving statue are shown in Visualization 1.

 

Fig. 8 Measured results of David statue (see Visualization 1 for the whole results). (a)–(d) The absolute phase maps acquired by conventional multi-frequency PSP, CDC, GADC, and PWADC. (e)–(h) The 3D reconstruction corresponding to (a)–(d).

Download Full Size | PDF

3.2. Measurement of two isolated objects

In Section 2.2.2, we have pointed out the problem of GADC under the case of measuring a large-size object or two isolated objects. The second experiment is designed to explain this problem and verify the superiority of PWADC. Two isolated objects including a ping-pong and a geometry model were measured in this experiment, as shown in Fig. 9(i). The ping-pong was arranged far from the camera and remained static while the model was shifted from the position near the camera to the position far from the camera. Since the valid depth volume (about 300 mm) is much larger than that in the first experiment (about 150 mm), the robustness of GADC is obviously inferior to that of PWADC, although it has been enhanced compared to that of CDC. From the Fig. 9(j) – 9(l), it is found that GADC has removed some redundancies in CDC but more redundancies still exist compared to adaptive PWADC. The more detailed results are displayed in Visualization 2. This experiment provides a strong evidence for the principle in Sections 2.2.1 and 2.2.2. Note that the X and Y coordinates in Fig. 9(j) – 9(l) are pixel coordinates, and the invalid area of Fig. 9(l) is hidden to outstand the valid area.

 

Fig. 9 Measured results of a plastic ball and a geometry model (see Visualization 2 for the whole results). (a)–(d) The absolute phase maps acquired by conventional multi-frequency PSP, CDC, GADC, and PWADC. (e)–(h) The 3D reconstruction corresponding to (a)–(d). (i)–(l) Depth volumes corresponding to (e)–(h).

Download Full Size | PDF

In the previous two experiments, we do not use any left-right consistency check and related algorithms. Next, the simplified left-right consistency check is added to our system to re-analyze the data of the second experiment. Figure 10 displays the final results. Comparing Fig. 10(b) with Fig. 9(b), we can find that the simplified left-right consistency check can reduce the error ratio obviously. However, the missing ratio in Fig. 10(b) after the simplified left-right consistency check increases obviously. That is because that there are too many unreliable areas which should be checked by Eq. (14). That means more points have the possibility of being not satisfied with Eq. (14) and set inactive. To reduce the error ratio and simultaneously not increase the missing ratio, we must have the precondition that the unreliable area cannot be too large. The small missing ratio in Fig. 10(c) and nearly perfect missing ratio in Fig. 10(d) just benefit from their less and less unreliable areas. It is worth decreasing the error ratio at the cost of increasing missing ratio because the low error rate of depth map will generate a correct compact depth volume which will conversely decrease the unreliable areas and the missing ratio. Visualization 3 further displays more related details about the experiment.

 

Fig. 10 Measured results with the simplified left-right consistency check (see Visualization 3 for the whole results). (a)–(h) The results corresponding to Fig. 9(a) – Fig. 9(h). (i)–(l) enlarged details corresponding to (e)–(h).

Download Full Size | PDF

3.3. Real-time experiments

In the last experiment, we focus on validating the performance of the feedback mechanism and the ability of real-time measurement and the compatibility of our method. A ping-pong and a microscope shell were measured in this experiment. The microscope shell was rotated by an electronic control turntable, the ping-pong was hung on a position far from the microscope shell. In the beginning, the ping-pong was shaded by the microscope shell, but with the continuous rotation of the microscope shell, the ping-pong revealed gradually. Because the ping-pong was far from and shaded by the microscope shell, it was not included in the adaptive depth volume. However, when it revealed, its position would be accurately detected by the feedback mechanism, and the adaptive depth volume was updated quickly. We implemented this experiment using dual-camera and quad-camera system respectively to verify the compatibility for different multi-view systems of our method. About 40fps and 60fps reconstruction speed can be achieved by the quad-camera system and the dual-camera system, respectively. The real-time measurement processes and results of quad-camera and dual-camera system can be found in Visualization 4 and Visualization 5. Two frames of Visualization 4 and Visualization 5 are shown in Fig. 11.

 

Fig. 11 The real-time measurement process and results using our method based on (a) quad-camera system (see Visualization 4 for the whole process) and (b) dual-camera system (see Visualization 5 for the whole process).

Download Full Size | PDF

4. Conclusion

We have presented a high-speed real-time 3D shape measurement approach based on ADC, where the measurement depth volume for geometric constraints is adaptively updated according to the current reconstructed geometry. The rationality of the proposed approach relies on the fact that the depth distribution of a moving object varies continuously in both spatial and temporal domain, which allows to define a compact depth volume and helps enhance the robustness of SPU. Furthermore, the adaptively updated measurement depth guarantees that the measured object can freely move within a large measurement volume. Besides, two complementary techniques, including simplified left-right consistency check and feedback mechanism based on valid area, are introduced to further remove erroneously unwrapped region and increase the robustness and flexibility of the ADC. The adaptive depth constraint approach, along with simplified left-right consistency check and feedback mechanism based on valid area constitute a complete computational framework for robust and efficient phase unwrapping in high-speed real-time 3D shape measurement. Besides, the proposed approach has low computation cost and good real-time performance. The processing speed can be further significantly improved by using graphics processing units (GPUs), as all the involved algorithms are performed pixel-wise and highly parallelizable. Experiments demonstrated the ability of the method to perform real-time 3D shape measurement with high-accuracy, for complex surfaces and spatially isolated objects, using only three high frequency fringe patterns.

There are several aspects that need to be further improved in the proposed method, which we will leave for future consideration. First, there are several parameters in the proposed approach, which should be properly selected for different kinds of motion. Currently, we empirically set these parameters according to the overall property of the test scene. It should be better that these parameters can be automatically and adaptively selected. Second, we simply use the measurement volume to replace the adaptive depth volume when a new object appears within the measurement volume but out of the adaptive depth volume, which is not an optimal selection. How to handle both cases is an another interesting direction for further investigation.