1. Introduction

Multi-Line structured Light (MLL) is a representative technique in the active 3D visual reconstruction [1,2]. It plays a vital role in reconstructing the key geometric dimensionality of the free-form 3D surface. Thus, MLL technique has been widely applied in the fields of industry, manufacture [3], transportation [4], etc. Compared with single-pattern structured light, projecting multiple line-structured lights allows large light coverage and high 3D reconstruction efficiency. However, identifying the multiple light projections in a 2D image is a challenging issue in the MLL technique. If multiple line-structured lights are projected onto a 3D non-planar surface, the light projections in the 2D image are likely to be cracked, cluttered, and deformed. The aim of the image light identification is to determine which 3D laser plane matches the line-structured light in the 2D image. Then, we can transfer the prior-calibrated position parameters to the lights for reconstructing the 3D surface.

The existing approaches for dense line stripes are mainly based on laser coding. The coding schemes can be roughly categorized as temporal or spatial coding due to the light projection mode [5,6]. The temporal coding schemes utilize multi-shot projection to generate multiple patterns at several consecutive times. One representative temporal pattern is the fringe pattern with various phase shifting modes, including sinusoidal [7,8], trapezoidal [9], etc. The temporal coding approaches are only suitable for static relative position between the sensor and the inspected object. For hand-held sensors, the multi-shot projection mode does not work.

Spatial coding schemes are more applicable in various scenarios, as they only project the pattern lights once (i.e. one-shot projection mode). The pattern can be with varying grayscales [10,11], shapes [12–14] or colors [15–17]. The shape-coded pattern typically consists of various geometrical elements, such as disc, circle and dash. For example, Tang et al [12]. designed 8 geometrical elements and applied Convolutional Neural Network (CNN) to automatically identify the pseudorandom-array-encoded lights. Jia et al [13]. defined 4 shape primitives and proposed a M-array-based spatial shape pattern. The above spatial coding strategies usually work for dense 3D reconstruction and the decoding algorithms are time-intensive. On the other hand, the majority of them depend on the neighbor information of the pattern, which would lead to a high sensitivity to line stripe ambiguities [5] caused by surface discontinuities [18].

As for active sparse 3D reconstruction, light identification should be efficient and accurate. Since multiple line-structured light projections in a 2D image present some sequentiality, even if they are modulated or cracked by a complicated 3D surface. Under this observation, we seek to solve the issue of light identification via a cue - the light projection distribution in a 2D image, and we describe the cue via a local-to-global graph representation framework. To the best of our knowledge, this novel viewpoint is still less-investigated and is a promising solution for efficient sparse light identification. The merit is that the image pattern of light distribution is independent of auxiliary laser codings. In summary, the main contributions of this paper are:

The rest of the paper is organized as follows. Section 2 recalls an overall pipeline of the 3D reconstruction via multi-line structured light, where one can locate the task of image light identification. Section 3 details the proposed algorithm based on the local-to-global graph representation. Section 4 presents the experimental results and discussion. We conclude the paper in Sec. 5.

2. Preliminary on 3D reconstruction via multi-line structured light

In this section, we present a general 3D reconstruction pipeline based on multi-line structured-light vision, as shown in Fig. 1. The light identification is a vital process before the 3D reconstruction. The overall pipeline consists of 3 main phases, including system calibration, identifying line-structured lights from a 2D image, and recovering 3D surface. We focus on the second phase. The details of the rest phases beyond the scope of this paper.

 

Fig. 1. Light identification in a general framework of 3D reconstruction via multi-line structured light. The irregular 3D surface reshapes the lights as cracked, cluttered, and deformed line patterns. The aim is to identify every line projection from the image and transfer the calibration parameters to the lights for 3D reconstruction.

Download Full Size | PDF

A multi-line structured light sensor is mainly composed of a camera and several laser projectors. Before the sensor is put into use, a calibration process is conducted to refine the sensor parameters, including internal parameters of the camera, geometrical parameters of the 3D laser planes, and structure parameters of the components. Each projector emits a laser plane that intersects with the surface of an inspected 3D object, generating a few line-structured lights. The camera captures the line-structured lights and projects them onto a 2D image (see Fig. 1(a)). The planes shaped by the irregular surface present cracked and scattered line patterns in the 2D image (Fig. 1(b)). The proposed light identification algorithm allows each light in the 2D image to be matched with its original laser projector (ID1-ID3), as shown in Fig. 1(c), so the prior-calibrated parameters of the laser planes can be transferred to the cracked line-structured lights in the image. The 2D lights with known 3D geometric cue are used for the following 3D reconstruction in Fig. 1(d). (Note: we prefer to mention line-structured light projection in a 2D image using one word “light” in the following text, just for simplification.)

3. Local-to-global graph representation for multiple line-structured lights

This section looks into the proposed local-to-global graph representation framework for image lights (Section 3.1) and details the soft overlapping strategy for local graph construction (Section 3.2). We finally establish a global graph by integrating the local graphs and sharing their hierarchies (Section 3.3).

3.1 Overview

The local-to-global graph representation framework is illustrated in Fig. 2. It mainly consists of two modules: local graph representation and global graph representation. The 2D image shows multiple line-structured light segments, with noise or undesired short lights in the background. The raw image goes through 2-step preprocessing. We firstly extract all the line-structured light candidates in a subpixel level, using an off-the-shelf line center extraction algorithm [19,20]. We then analyze the connectivity of the light candidates by the well-known Flood Fill algorithm. The connectivity analysis removes the light outliers in the background and identifies the positive light segments.

The image after the preprocessing is fed into a graph construction phase that is formulated into a local-to-global mode. The core of the local graph representation is grouping the lights. We define a soft overlapping metric that quantifies the correlation of each two lights (Fig. 2(b)) so that we can group the lights with high correlations together. We assume each light as a node $V_i$ of a local graph. For each two nodes $\{V_i, V_j\}$, we determine an edge $E_{i \rightarrow j}$ with a weight and a direction based on the soft overlapping rate (Fig. 2(d)–2(e)). The weight expresses the connection strength, and the direction reflects the hierarchy that will be used in the global graph. Through the local graph module, the lights are represented as a set of local directed and weighted graphs $\{G({V_i},{E_{i \rightarrow j}})\}$. Each local graph describes the hierarchical distribution of correlated lights in a local region. For more details on the local graph construction, please see Section 3.2.

 

Fig. 2. The proposed framework: (a) the input image goes through a 2-step preprocessing for removing outliers and identifying light segments; (b-e) constructing local graphs, where the red numbers beside the edges are weights; (f-h) constructing a global graph by integrating local graphs. The global graph represents the hierarchical distribution of all the image lights; (i) Mapping the labels in the global graph back to the input image and determining the IDs of the light segments.

Download Full Size | PDF

Given the local graphs, a global graph module integrates all of them via an important cue: depth of nodes (Fig. 2(f)). We remove the edges of non-adjacent nodes without successive depth (Fig. 2(g)), so the nodes with the same depth share one level in the global graph (Fig. 2(h)). Finally, we map the level labels of the nodes back to the light counterparts in the raw image (Fig. 2(i)), thus the light segments from the same laser plane can be matched. The details of the global graph are given in Section 3.3.

3.2 Local graph construction

We cast the issue of constructing local graphs as a process of light grouping, based on a grouping metric $-$ the overlapping rate of two lights. Each two overlapped lights are hierarchically correlated and are within a group. We define an overlapping rate for any two adjacent lights and a soft overlapping rate for two non-adjacent lights, respectively. The definitions of the overlapping metrics are illustrated in Fig. 3.

 

Fig. 3. Soft overlapping rate of two line-structured light projections in an image: (a) projecting three line-structured lights in a 2D image onto a reference axis; (b) calculating an overlapping rate for the adjacent lights; (c) calculating a soft overlapping rate for the non-adjacent lights $\{Line 1,Line 3\}$.

Download Full Size | PDF

According to the distribution property of multi-line lasers, line-structured lights from multiple laser planes are typically sequential along a principal direction. Thus, we project all the lights to a 1D reference axis that is vertical to the principal direction (Fig. 3(a)). The reference axis is designed to capture the direction of the original line-structured light without being reshaped by the surface. In practice, we put a smooth hardboard facing frontal to the multiple lasers, and captured an image of the line-structured lights intersecting with the board. We calculated the vertical direction of all the parallel laser lines in the image as the principal direction. Thus, the parallel laser lines are just the reference axis. Both the reference axis and the projection direction are global for all the light segments. For each two adjacent lights $\{Line 1,Line 2\}$ in Fig. 3(b), the overlapping rate $OR_{1-2}$ is calculated as

When the overlapping rate $OR$ of two lights is larger than a pre-defined threshold (here we use 0.5), the two lights are considered as overlapping, and vice-versa. We group all the overlapped lights into a graph, which means that each two overlapped lights are hierarchically correlated in the local graph.

3.2.1 Soft overlapping rate

The overlapping rate is related to the projection direction. When the direction slightly varies, the overlapping rate of two lights varies as well, especially for two non-adjacent lights. Thus, we expand the calculation of the overlapping rate to make it robust against a slight variation of the projection direction. This strategy we call it Soft Overlapping Rate that focuses on non-adjacent lights, as illustrated in Fig. 3(c).

The connectivity of the non-adjacent lights can be represented by consecutive adjacent lights as well. The overlapping rate $OR_{1-2}$ of the $Line 1$ and $Line 2$ is $A_1B_2$, and that of the $Line 2$ and $Line 3$ is $OR_{2-3} = A_2B_3$. For the non-adjacent lights $Line 1$ and $Line 3$, we define a soft overlapping rate $OR_{1-3}$ as Eqn.2, instead of using a direct overlapping rate $A_1B_3$. The soft overlapping rate can weaken the variation of the projection direction by choosing the strongest adjacent connectivity as an alternative.

The core of the soft overlapping metric is to enhance the connectivity of two non-adjacent lights, which is beneficial to the following two steps: local graph refinement (Section 3.2.3) and calculating depth of nodes (Section 3.3). We will discuss them in the corresponding sub-sections.

3.2.2 Edge weight and direction

Each two overlapped nodes $\{V_i,V_j\}$ are connected using an edge $E_{i-j}$. Then, a light group is transformed into a local graph. We weight the edge using the overlapping rate (OR) of its two endpoints, as OR measures the strength of the connectivity. For a local graph, combining all the edge weights forms a weight matrix $\textbf {{F}}(i,j) = max\{W_{ij},0\}$, where $W_{ij}$ is the weight of the edge $E_{i-j}$.

The direction of an edge is determined via a projection height. As shown in Fig. 3(b), the projection height of the $Line 1$ is the distance of the end-point $A_1$ to the 1D reference axis. For two overlapping nodes $\{V_i,V_j\}$, we define that the node with a larger projection height is the starting-node of the edge, while the other node is the ending-node of the edge. Similarly, in a local graph, we combine all the directions of the edges and form a direction matrix $\textbf {D}(i,j) = sgn(d_i, d_j)$, where $d_i, d_j$ are the projection heights of the nodes $\{V_i, V_j\}$, respectively. The matrix describes the direction of an edge $E_{i-j}$ using a label of 1 or -1.

3.2.3 Local graph refinement

There are usually weak edges or cutting edges in the local graphs, as shown in Fig. 4(a). The weak edges can be filtered out by the weight threshold. The cutting edge is a bridge that connects two connected components. In our graph, there would be cutting edges that connect two independent local graphs and disturb the structure of the local graphs, e.g. the red edge in Fig. 4(a-2). We adopt the Tarjan algorithm to detect the strong-connected components of the local directed graphs and thus remove the cutting edges. Overall, the scattered lights in the raw 2D image are formulated into several local graphs whose connectivity is weighted by the (soft) overlapping rate. The directions of the connections represent the hierarchy of the local nodes, which will be used in the following global graph.

 

Fig. 4. Local graph refinement: (a-1) calculating soft overlapping rate; (a-2) removing weak connectivity; (a-3) removing cutting edges and calculating the depths of the nodes; (b) a comparison pipeline without using the soft overlapping rate metric. The connectivity of non-adjacent nodes are enhanced by the soft overlapping rate, e.g. the edges $E_{1 \rightarrow 5}$, $E_{2 \rightarrow 6}$. Note: the number beside each edge is the weight of the edge.

Download Full Size | PDF

To illustrate the superiority of the soft overlapping metric, we show a process of the local graph refinement without the soft overlapping metric in Fig. 4(b), as a contrast. The non-adjacent nodes (e.g. the edges $E_{1 \rightarrow 5}$ and $E_{2 \rightarrow 6}$) are very important for enhancing the robustness of a local graph. The absence of the connections of the non-adjacent nodes can result in false local graphs and node depths, as shown in Fig. 4(b-3).

3.3 Global graph representation

The local graphs represent the local distribution of the lights. We integrate them as a global graph via an important cue – the depth of the node. The global graph representation consists of two steps, including calculating the depth and global graph refinement.

3.3.1 Calculating the depth of nodes

The definition of the depth is that the top father node is with the depth of 0, and the depth of a child node is 1 larger than its father node. If a child node has multiple father nodes whose depths are different (e.g. the depth of one father node is 0 and that of the other one is 2), the child node follows the “deeper” father node, with the depth of 3 instead of 1.

Based on this rule, we first calculate the depths of the nodes in the local graphs using the Breadth First Search (BFS) algorithm [21]. The BFS algorithm provides a searching path to count the depth of every node. The depth calculation can be seen in Fig. 4(a-3). The process without the soft overlapping metric leads to a different global graph, as Fig. 4(b-3), where the depth of the left local graph drifts entirely.

3.3.2 Global graph refinement

To simplify the graphs, we remove the edges whose two end-points are non-adjacent, as illustrated in Fig. 2(g). That is, the depth difference between the two end-points is larger than 1. It is worth noting that emphasizing the connections of the non-adjacent nodes in the local graphs is to separate the individual local graphs better. However, when the depths of the nodes are known, those connections are no longer necessary. According to the depth, we unify the nodes of the local graphs in one hierarchical structure that is the global graph we expect.

Since each node in the global graph corresponds to a light segment in the raw image, we map the depth of each node back to the light counterpart in the raw image. Finally, we identify the lights with the same depth in the global graph are from the same laser in the 3D space.

4. Results and discussion

In this section, we first go through the experimental procedure and the evaluation metrics. Then, we report both qualitative and quantitative performance of the proposed algorithm on the rugged surface of a 3D aluminum bar. We finally show its application to a case on 3D section profile reconstruction.

4.1 Experimental equipment

We put up a multi-line structured light system that contains three line-structured laser projectors and a camera, as shown in Fig. 5(a). The 3D object used for section profilometry is an aluminum 3D bar (Fig. 5(b)). The reason we adopt this object is that the normal section profile of the bar is rugged and challenging, as shown in Fig. 5(c). The surface of the object typically separates a line-structured light as multiple light segments. The relative pose between the vision system and the measured object varies to 3 different situations, as shown in Fig. 5(d).

 

Fig. 5. Experimental equipment: (a) a three-line structured light system; (b) a 3D aluminum bar for profilometry; (c) the normal section profile of the 3D aluminum bar; (d) 3 different relative poses: parallel, tilted, and lifted.

Download Full Size | PDF

4.2 Evaluation metrics

We adopt both segment-wise and length-wise measures to quantitatively evaluate the performance of the proposed light identification algorithm. For the segment-wise assessment, we calculate the accuracy (S-Acc) and the recall rate (S-Recall), where

We take into account the contribution of the identified lights to the 3D reconstruction, thus we investigate the length-wise identification performance. In an image, we represent a line-structured light $l_i$ as a set of center points $\{\textbf {p}_{ij}\in R^2\}(j=1\cdots m)$, and we utilize the amount of the center points of the light as a length measurement. We calculate the length-wise accuracy (L-Acc) and the length-wise recall rate (L-Recall), where

4.3 Qualitative performance

The distribution of light segments in the image varies with the relative pose between the sensor and the measured object. To test the robustness of the algorithm to varying light segment distributions, we arranged 3 different poses as shown in Fig. 5(d). We use the cylinder axis and the sensor baseline to represent the poses of the sensor and the object, respectively. In the normal pose, the cylinder axis is roughly parallel with the sensor baseline. We tilted the sensor baseline to form the second pose. In the third pose, we lifted the measured object, which allows for a tilt angle between the cylinder axis and the sensor baseline.

In each pose, we rotated the object around its axial direction to 5 different positions and captured an image at each rotation. We finally collected 15 images in all the three poses. The number of light segments at each position is listed in Fig. 6(a). It shows that the original 3 line-structured lights are cracked up to at least 9 segments. Three example raw images can be seen in Fig. 7(a) with the zoom-up of the center points in Fig. 7(b).

 

Fig. 6. (a) The number of light segments in all the images; (b) the lengths of all the light segments. We define four borderlines to separate them as 4 classes; (c) the percentages of the 4 light classes.

Download Full Size | PDF

 

Fig. 7. Light identification results at 3 example positions. Each row shows the result at one position. (a) raw images; (b) extracting line centers; (c) identifying light segments (each segment is marked using one color and with an ID number); (d) grouping lights as local graphs (each group is marked using one color); (e) final identification result via the global graph (the lights from the same laser projector are marked using one color).

Download Full Size | PDF

Overall, there are 176 light segments captured in all the 15 positions, and the lengths of the segments are shown in Fig. 6(b). We separate all the segments as 4 classes according to the length, that is, very short, short, medium, and long segments. The percentages of the segment samples are shown in Fig. 6(c), where the very short segments are challenging in the identification.

The results of the proposed algorithm performed at three example positions are shown in Fig. 7. The light identification goes through pre-processing and center extraction in Fig. 7(b), segment identification in Fig. 7(c), local clustering in Fig. 7(d), and final global identification in Fig. 7(e). The results demonstrate the effectiveness of each step. The algorithm is robust to varying distributions of cracked lights and thus is free of relative pose constraints between the measured object and the sensor. The most challenging cases lie at the very short light segments which are likely to be mixed or linked with other longer lights. Our algorithm can distinguish the local graphs due to the proposed soft overlapping strategy.

The overlapping rate matrices of the light segments in an image are shown in Fig. 8. We firstly calculate the overlapping index of each two light segments, which leads to an overlapping rate matrix, as Fig. 8(b). The overlapping rate threshold is set as 0.5. Then we enhance the connectivity of non-adjacent segments and removed the weak connectivity, resulting in a soft overlapping rate matrix, as Fig. 8(c). According to the soft overlapping index, the light segments are grouped as 5 clusters, as Fig. 8(d). It is obviously observed that the light segments with high overlapping correlations are grouped into a local cluster.

 

Fig. 8. Overlapping matrix and local light grouping: (a) lights segments in an image; (b) overlapping matrix of each two segments; (c) soft overlapping matrix; (d) grouping the segments as 5 local clusters (Each point is the center of a light segment).

Download Full Size | PDF

4.4 Quantitative performance

We manually labelled all the light segments in the images as ground truth. Then, we calculated both the segment-wise and length-wise evaluation metrics performed by the proposed algorithm. The results are shown in Table 1 and Table 2.

Table 1. Segment-wise performance of our algorithm

View Table | View all tables in this article

Table 2. Length-wise performance of our algorithm^a

View Table | View all tables in this article

For the total 176 light segments in all the images, the proposed algorithm can achieve an average segment identification precision of $99.4\%$ and an average recall rate of $96\%$. The performance at tilted and lifted poses slightly degrade, because some very short light segments are mixed up with other local graphs. In terms of the length-wise evaluation, $99.2\%$ of the light length is correctly identified and can be used for the reconstruction. Among all the identified lights, $99.9\%$ of the light length is correct. Overall, the proposed algorithm is very effective at dealing with image light identification in varying poses.

The false cases usually happen on the very short lights, the reasons are two-fold: a) projecting the very short light to an image would generate possible false segment connection; b) the overlapping rate of very short lights is sensitive to the projection direction, therefore the lights are much likely to be missed or grouped into a wrong local cluster. Overall, the false cases can be summarized as two types: missed segments and false-connected segments. Sometimes the two kinds of false cases are coupled. We exhibit a failure example in Fig. 9. The three very short light segments are supposed to be from a local cluster, however, one is false-connected with another long light No.8 and thus is grouped into a wrong cluster. The rest two form two independent groups that are further filtered out in the global graph.

 

Fig. 9. Example false identification: (a) raw images with red-marked line centers; (b) separating light segments, where one short light is false-connected with the light 8; (c) the two short lights form two independent groups respectively, which are filtered out in the global identification in (d).

Download Full Size | PDF

4.5 Performance on 3D section profile reconstruction

After the lights in the image are identified and transferred with known 3D prior-calibrated geometrical information, they can be applied to reconstruct 3D surfaces. To further verify the applicability of the proposed algorithm, we design a 3D reconstruction case as below. We reconstruct the 3D normal section profile of the object using multiple general section profiles [22]. First, multiple general profiles of the object are point-wise registered, then the point correspondences are used to estimate the normal section profile via a profile transformation model. The general section profiles of the object are diverse, while the normal section profile is unique. Thus, we utilize the normal section profile as a reference for verifying all the general section profiles.

In practice, the normal section profile is estimated in an iterative framework. As initialization, the point-wise registering is performed using curvature features, and we can get an initial profile transformation model. At each iteration, the general profiles are transformed into the normal section profile via the model, and we search for the closest point pairs on each two transformed profiles to update the point correspondences. Finally, we can get an optimal normal section profile as well as a set of point correspondences of multiple profiles. Since we do not have the ground truth of the normal section profile, we quantitatively assess the reconstruction using the overall point-wise registration precision. If the general section profiles are reconstructed correctly, the overall registration error is supposed to be low.

Fig. 10 shows the procedure of normal section profile reconstruction at one position. The whole reconstruction framework includes 4 main stages: light identification using the proposed algorithm (Fig. 10(b)), 3D general profile reconstruction, point-wise registering general profiles (Fig. 10(c)), and normal section profile estimation in Fig. 10(d). Correct light identification allows the following accurate 3D profile reconstruction. In practice, one can repeat the process shown in Fig. 10 from different viewpoints and then register all the partial profiles as a whole. Here, we just present the reconstruction process at one viewpoint.

 

Fig. 10. Procedure of 3D section profile reconstruction: (a) raw image with red-marked line centers; (b) three sets of general profiles are identified; (c) reconstructing 3D general section profiles and point-wise registering the three profiles for a normal section profile (black lines are correspondences); (d) calculating 3D normal section profile; (e) its frontal view and comparison to the real section.

Download Full Size | PDF

Fig. 11(a) shows the iterative optimization processes of calculating the normal section profile in three poses, where the performance at each iteration is measured using the Root Mean Square (RMS) of all the point registration errors. We can see that all the three registration processes converge at the iteration number of around 20. For each case, we investigated the distribution of all the final point registration errors. The statistical results are shown in Fig. 11(b). The RMS of the final registration errors in the three cases are $0.033mm$, $0.022mm$, and $0.020mm$, respectively. The average registration precision at the aforementioned 15 positions is $0.025mm$.

 

Fig. 11. Registration error of reconstructing 3D section profile: (a) RMS of the registration errors vs. iteration number; (b) final registration error statistics of the three cases.

Download Full Size | PDF

5. Conclusions

This paper proposes a line-structured light identification algorithm via a local-to-global graph representation. We design a soft overlapping metric and a corresponding light grouping strategy to establish local graphs for the lights in a 2D image. The local graphs are then integrated as one global graph via the depth of the node. The global graph describes the spatial distribution of the lights in a hierarchical fashion. The experiments were performed on a complicated 3D surface, which demonstrates the applicability of the proposed framework and its robustness to varying relative poses between the sensor and the inspected object. We further integrate the light identification module into a 3D reconstruction case that aims at recovering a 3D section profile. The results show that the comprehensive pipeline can achieve an overall reconstruction precision of $0.025mm$. Our light identification algorithm benefits from image pattern, and thus it is free of any auxiliary coding scheme or multi-shot strategy. It can deal with flexible application scenes and is especially applicable in sparse 3D reconstruction, e.g. recovering 3D key geometric dimensionality of an object.