1. Introduction

3D depth sensing is growing widely used in various applications such as augmented reality [1], man machine interaction [2] and self-driving automobile [3]. The mainstream depth sensing methods can be divided into passive 3D imaging [4] and active 3D imaging [5]. Compared with active 3D imaging, passive 3D imaging has more stringent requirements on imaging conditions, especially is not suitable for low illumination environment. Active illumination imaging is more robust to ambient light than passive illumination. The active 3D imaging technique light detection and ranging(LiDAR) adopts pulses of light to illuminate the scene and use a sensor to record the time-of-flight information of back-scattered light from the scene. LiDAR is emerging as an appealing choice for diverse depth imaging scenes.

In the context of LiDAR, depth perception capability in low light flow environments is mainly promoted by pulsed laser illumination coupled with nascent single-photon avalanche diodes (SPAD) capable of recording the round-trip time of individual photon with picosecond time accuracy, such as Long-range imaging [6], non-line-of-sight imaging [7], and imaging through Scattering medium [8]. The principle of SPAD for scene 3D reconstruction is to accumulate the photon counting histogram by recording time-of-flight(ToF) data under multiple pulse emission. However, restrictions on optical flow and imaging time prevent adequate collection of signal photons. As a consequence, the raw measurements with extremely low signal photon counts and low signal-to-background ratio (SBR) impose tremendous challenge on the depth reconstruction algorithms.

To tackle this challenge, a large amount of recent works have concerned on developing robust reconstruction algorithms from noisy SPAD measurements. Recent works can be broadly divided into three directions. Kirmani et al. and Shin et al. [9–11] utilized scene structures to establish probabilistic models for back-scattered light detected by SPAD and employed pixel-by-pixel processing to remove the background noise. Rapp et al. and Chen et al. [12,13] combined with the adaptive super-pixel approach to reconstruct sparse ToF Data by exploiting distribution characteristics and space domain correlation of signal photon. Lindell et al. and Sun et al. [14,15] used convolution neural networks to fuse data from multiple sensors for depth estimation by employing a multi-sensors fusion strategy. The echo signal will be weaker at edge locations than at non-edge locations due to light scattering for active 3D imaging system. Fig. 1(b) shows the ToF data acquired by the SPAD array and the echo signal at the edge of the object is significantly weaker. Considering that the range images have more simpler structures than the intensity images, the range images can be seen as a patchwork of smooth regions divided by edges [4,16]. However, how to reconstruct depth images from the perspective of improving edge reconstruction accuracy has not been investigated.

 

Fig. 1. (a) Schematic of SPAD-based pulsed LiDAR imaging system including a pulsed laser source, synchronous signal generator and a SPAD detector with a integrated TCSPC module. The TCSPC module timestamp photon arrival event to generate a 3D spatial-temporal data. Each data cube contains a histogram, which is generated by accumulating echo photons. (b) Figure(I) represents the imaging scene, figure(II) denotes the histogram data recorded at background pixel, figure(III) denotes the histogram data recorded at the edge pixel of imaged object, figure(IV) denotes the histogram data recorded at non-edge pixel of imaged object. As we can see, the recorded echo photons at the edge position are significantly less than the non-edge position.

Download Full Size | PDF

A most promising direction for depth reconstruction from noisy SPAD measurements depends on neural networks, which are based on rich representation capability and strong feature extraction capability. Lindell et al. [14] and Sun et al. [15] adopted multi-sensor fusion strategy, including a conventional intensity sensor and SPAD sensor, to reconstruct depth images from noisy SPAD measurements. However, the conventional intensity sensor is not suitable for low light flow environments. Peng et al. [17] introduced a non-local block into neural network model to exploit the long-range correlation of SPAD data in spatial-temporal domain. Nevertheless, the non-local block has a large model size, which is not easily portable. Besides, the non-local block does not take full advantage of the correlation on the feature channel. Zang et al. [18] proposed a lightweight neural network, which improves integration of neural network without employing sensor fusion approach. However, the network would introduce redundant information when directly cascading the feature maps at the encoding and decoding ends. Therefore it is valuable to improve the accuracy of edge reconstruction by enhancing the performance of neural network for depth reconstruction in low optical flow environment.

In this paper, we propose a performance-enhanced neural network by improving the reconstruction accuracy of depth edges to promote depth reconstruction accuracy. We adopt the multi-scale integrated U-Net network [18] as the backbone of our network. The multi-scale integrated U-Net network achieves reconstruction performance comparable to the performance of sensor fusion methods [14,15] by recovering edge information lost due to down-sampling through cascading the encoder and decoder with long and short skip connection. However, multiple skip connection can introduce redundant information (e.g. noise and pseudo-edges) from the encoder to the decoder. It will ultimately affect the reconstruction of the depth map. This is because the features in the encoder are low-level features due to shallow convolution layers, while the corresponding features in the decoder are higher-level features due to deeper convolution layers, and they are semantically different. Therefore it is not appropriate to splice them directly through skip connection. Attention-gate(AG) structure is introduced to replace the skip connection. The features of encoder related to the decoder are retained and the irrelevant features are filtered out by the AG module. Meanwhile, in order to enhance weak edge reconstruction, a lightweight attention module is also employed to capture long-range correlations of the spatial-temporal domain of SPAD measurements in the process of up-sampling and feature refinement. In addition, we employ hybrid loss functions including Kullback-Leibler (KL) divergence, total variation (TV) spatial regularization and gradient regularization functions to further obtain accurate edge reconstruction of objects in the scene. We validate our proposed method under various signal-to-background ratio(SBR). We also evaluate the impact of the edge reconstruction on the depth reconstruction using the evaluation metrics we first proposed. The experimental results show that our method is well suited for the SPAD-LiDAR imaging under low SBR environment.

2. Forward model

In a representative SPAD-based pulsed LiDAR imaging system, the pulsed laser triggered by the synchronization signal emits multiple pulses of laser light to irradiate the imaging scene. Then the received pluses from the scene return to the SPAD sensor, which would be recorded by the time-correlated single-photon counting (TCSPC) module. Finally the system would generate 3D spatial-temporal histogram data, as shown in Fig. 1(a).

It is worth noting that previous literature [10–12,14,17,18] generally assumes that the SPAD-based system operates under low optical flow environment. It means that in each illumination cycle, the returned photons are very rare(much less than 1 photon). Thus the pile-up effect [19–21] and other effects which would distorted the histogram, can be negligible. The low flux environment allows photon arrival events recorded to be modeled as independent between adjacent timestamps. Thus the number of photons returned to the sensor for the time stamp n and imaging position (i,j), can be depicted as:

For the SPAD sensor, the sensor’s quantum efficiency $\eta$ indicated photoelectric conversion efficiency [22] and the dark count $b_{d}$ (error detection rate) are taken into account. Consequently, the number of photons captured by the SPAD sensor within N illumination cycles can be denoted as a time-domain histogram:

3. Edge enhancement model

In this section, the proposed network architecture, the ADAG(Attention-Directed Attention Gate) module, and hybrid loss functions would be introduced.

3.1 Network architecture

We use the multi-scale integrated U-Net network, proposed by Zang et al. [18], as the backbone of our neural network. The U-net++ [23] was used as the backbone of the multi-scale integrated U-Net network. Through U-net++ network, the edge information lost due to down-sampling was recovered by multi-level up-sampling and multiple long- and short-range cascades on the same level. Consequently, depth reconstruction accuracy comparable to that of sensor fusion network was obtained under low optical flow environment. However, the multi-scale integrated U-Net network did not filter the redundant information when adopting skip connection to cascade the features from the encoder and the decoder directly. The features in the encoder and the decoder are semantically different. Therefore it is not appropriate to splice them by skip connection. Moreover the multi-scale integrated U-Net network used only maximum pooling in down-sampling, which leads to more information lost in the down-sampling process. To tackle these problems, we introduce ADAG module into network to remove redundant information from the encoder and enhance reconstruction accuracy of weak edge from the decoder. Meanwhile, we use maximum pooling and average pooling to down-sample in a parallel manner to reduce lost information.

Overall network structure is shown in Fig. 2. $X_{i, j}$ denotes the feature map where $\mathrm {i} \in \{0,1,2,3\}$ is the levels of down-sampling, $\mathrm {j} \in \{0,1,2,3\}$ is levels of refinement,l is the depth of down-sampling. The network takes the noisy 3D spatial-temporal data as the input firstly. Then network extracts features by the feature extraction module including down-sampling as the encoding side and transposed convolution as the decoding side. Then the extracted feature flow to the feature fusion module named ADAG, which improves the accuracy of edge reconstruction. Considering the trade-off between model size and performance improvement efficiency, we only add the ADAG module for feature fusion in the uppermost layer (the feature with the largest spatial resolution). Because large spatial resolution means richer feature information and the features at both ends of the encoding and decoding ends differ the most compared to other scales for the U-net structure. Then the feature refinement module CBAM is adopted to refine feature after fusion. Finally, convert the denoised 3D estimated depth to 2D depth map through the soft argmax operator.

 

Fig. 2. Schematic of the proposed network’s architecture. We adopt the multi-scale integrated U-Net network proposed by Zang et al. [18] as the backbone of our network. Unlike the multi-scale integrated U-Net network to directly cascading of features, we introduce ADAG module into the node of the cascade of convolutional output features(the dashed arrows) in the first row and up-sampling features(the green arrows) from the second row. Meanwhile, we employ the hybrid loss functions to constrain the train process. Consequently, the proposed network can obtain stronger performance in edge information extraction. The reconstructed 2D depth map obtains a higher accuracy of edge reconstruction.

Download Full Size | PDF

3.2 ADAG module

For the U-net network, if the encoder end and decoder end are cascaded directly through a skip connection, redundant information unrelated to the features from the decoder end would be introduced [24]. We introduce Attention Gate(AG) module [25] ever used in medical image segmentation into proposed network. The AG module has the advantage of being able to learn task-relevant features with attention and can be integrated into the network model in a plug-and-play manner. The AG module is originally used in the field of medical image segmentation to process 2D medical image data. In this paper, the AG module is used to SPAD-LiDAR depth reconstruction to process 3D SPAD measurement data. We introduce AG module into the U-net++ model to replace the skip connection between the encoder end and decoder end. The purpose is to enhance the information related to the features from the decoder end. To further enhance the edge enhancement performance of AG module, the attention module CBAM and the AG module are merged to form ADAG module.

The ADAG module is shown in Fig. 3(a). The module input consists of two branches. Along the upper branch, the first node named "Data Transfer" denotes input features passed from the encoding ends. The features(X) is obtained as the features to be filtered after cascading the features from the coding end. Along the lower branch, the first node named "CBAM" denotes input features passed from the attention module(CBAM). These input features are the up-sampled features refined by the self-attention module as gating signal(G). After the input features are convolved by 1*1*1 convolution kernel($w_{g}, w_{x}$), the number of channels changes from $F_{g}$ to $F_{c}$. After the operation of summing the two inputs, the activation is performed by ReLU activation function($\sigma _{1}$). Subsequently the features are convolved by 1*1*1 convolution kernel($\Psi$). Then activated by sigmoid activation function($\sigma _{2}$), the normalized coefficients are multiplied with the input features at the encoding end by elements. Then the obtained edge-enhanced output are cascaded with the refined features from the decoding end. The operation procedure for ADAG can be modeled as:

 

Fig. 3. (a) The overview of ADAG module. The output of CBAM module is adopted as the gating signal(G). The attention coefficients($\alpha$) is obtained by calculating the correlation between the input features(X) and the gating signal(G). Input features(X) are scaled by attention coefficients calculated in ADAG. The output(Y) of the ADAG module is obtained by cascading the refined input features and the output of CBAM. (b) The overview of CBAM module. The module contains two sequential sub-modules:channel attention module($M_{c}$) and spatial-temporal attention module($M_{s}$). We can obtain the long-range correlation of feature channels and SPAD measurements through these modules. Aiming to improve extraction of weak signals, the final refinement results are obtained by summing attention coefficients and original input features by element.

Download Full Size | PDF

The captured light pulse signal photons have the same temporal domain distribution and the arrival time of the signal photons can be any timestamp within the light pulse period [17]. Therefore the signal photons have long-range temporal domain correlation. For most natural scenes, neighboring pixels with similar geometric structures are spatially correlated. It is similar to the temporal domain, such structures may appear at arbitrary spatial locations. Therefore it is also necessary to consider the long-range correlation in the spatial domain. In [17], a non-local block [26,27] was introduced into the network to exploit the long-range correlation in the spatial-temporal domain of SPAD data. However, the non-local block has a high computational overhead. Besides it is not easily transplanted.

To tackle these problems, we introduce a plug-and-play lightweight self-attention module Convolutional Block Attention Module(CBAM) [28], which is employed to extract long-range correlations of features in spatial-temporal and channel domain. We place CBAM module after transposed convolution to refine the features after up-sampling. Aiming to improve the weak edge reconstruction and does not affect the original up-sampling results, we add CBAM into the network through skip connect. The CBAM module is shown in Fig. 3(b). The refined features are extracted by a sequence attention extraction operation including the channel attention module and the spatial-temporal domain attention module. Finally the output results are obtained by adding the refined features to the input features using skip connection. The operation procedure for CBAM can be modeled as:

3.3 Loss function

Aiming to make network training more effective, we adopt hybrid loss functions to perform a two-stage depth estimation process, similar to other neural network based methods [15,17,18,28]. Considering the similarity of the temporal-domain distribution between the noisy SPAD histogram and the ground truth histogram, we employ a combination of the Kullback-Leibler (KL) divergence as the objective function to constrain the first-stage estimation process, which estimates the denoised 3D histogram. The network perform projection from 3D histogram into 2D depth map after the first-stage. Aiming to maintain edge sharpness and improve robustness against noise, we utilize the total variation(TV) spatial regularization to constrain the projected 2D depth map. Besides we use a gradient regularization for joint constraints with TV regularization to reconstruct more accurate edges.

3.3.1 Kullback-Leibler (KL) divergence

The Kullback-Leibler (KL) divergence [15] is employed to estimate the similarity of the probability distribution at each imaging position (i,j) between the denoised histogram and the normalized ground-truth histogram. This loss function can be expressed as:

3.3.2 Total variation(TV) spatial regularization

Aiming to improving denoising performance and edge retention capability, the total variation(TV) spatial regularization is introduced to constrain the projected depth map. This loss function can be written as:

3.3.3 Gradient regularization

Natural images contain a large number of edges and details. Aiming to reconstruct a depth map with fine details and accurate boundaries, it is crucial to penalize the boundary error [4,29]. We introduce the following loss function to constrain the gradients of depth:

3.4 Evaluation metrics

3.4.1 Depth estimation accuracy

In order to make a quantitative comparison with the previous work [14,18], we choose the depth reconstruction accuracy metric as one of the evaluation metrics:

3.4.2 Edge estimation accuracy

Previous works only use depth reconstruction accuracy as the evaluation metric to reflect the performance of the algorithm on depth reconstruction from noisy SPAD measurements. In order to show the edge reconstruction accuracy, we first introduce quantitative metrics to evaluate edge reconstruction accuracy. For this purpose, we firstly employ the Sobel operator [30] to calculate the gradient maps of the estimated depth map and the true depth map respectively. Then calculate the RMSE between the gradient map of the estimated depth map and the gradient map of the true depth map(GradRMSE).

4. Experiments

4.1 Implementation Details

We employ a simulated dataset to perform network training and compare performance with other methods on a simulated test data and a dataset of captured real scenarios.

The training set of SPAD measurements are simulated from the NYU v2 data set [31]. In simulation, randomly crop the depth maps of the training set to a resolution size of 32*32, and the histogram of 896 time bins is generated by sampling the inhomogeneous Poisson process of Eq. (2 according to the depth values of the pixels. A total of 16,061 images and 3,633 images were used for training and validation respectively. In training process, we employ the ADAM [32] solver as the optimizer with a batch size of 4 and a learning is initialized as $10^{-5}$ with a decay rate of 0.9 after 100 iterations. The network is realized by PyTorch and trained on NVIDIA 2080Ti GPU for 4 epochs. Each epoch takes about 8.5 hours and iterates 16k times in total.

4.2 Simulated data

we compare the performance differences between our proposed method and other methods for depth reconstruction of simulated datasets from both qualitative and quantitative perspectives.

4.2.1 Quantitative evaluation

We assess our proposed network EeNet on the 6 test scenes of the Middlebury dataset under 2 low signal-to-noise ratio(SBR) environment(2:50, 2:100) with comparison of approximate maximum likelihood estimator(MLE) [33], Lindell et al. [14], Peng et al.[17], and Zang et al. [18]. We retrain the network for the two different noisy scenes. We list the root mean squared error(RMSE) of depth map and the RMSE of the gradient map between the estimated depth map and the true depth map (GradRMSE) in Table 1 for the 6 Middlebury scenes reconstructed with the various methods. As can be seen from the results, our method obtains a better performance for deep reconstruction over both evaluation metrics than MLE, Lindell’s method and Zang’s method. Besides, our method is comparable to Peng’s method. It demonstrates the effectiveness of our method for enhancing the accuracy of edge reconstruction. Meanwhile, the result also shows that improving the edge reconstruction accuracy is meaningful for improving the overall depth reconstruction accuracy.

Table 1. Quantitative comparison of the different reconstruction methods on 6 Middlebury scenes. Two tables for two low SBR levels - 0.04 and 0.02. The evaluation metrics RMSE and GradRMSE indicates the depth reconstruction accuracy and edge reconstruction accuracy, respectively. The lower the value of the evaluation metrics, the higher the reconstruction accuracy. Bold numeric characters represent the best performance, underlined numbers represent the second best.

View Table | View all tables in this article

4.2.2 Qualitative evaluation

We use two exemplar scenes to qualitatively compare the performance of different methods for deep reconstruction under different noise conditions, as shown in Fig. 4. The top two rows of the figure denote the results of various methods to reconstruct the scene named Art and the below two rows denotes the results of various methods to reconstruct the scene named Reindeer. As we can see from the closeup views, MLE leads to poor reconstruction and signal is almost drowned by noise. For Lindell’s method, we adopt the sensor-fusion model to conduct comparison. Although there is a better denoising effect than MLE and the main structure can be recovered, fine edges of depth map are lost. For Peng’s method, the depth map is better reconstructed, but some details are partially lost. For Zang’s method, we adopt uncompressed model to conduct comparison. The fine edges have been restored, but blurred details have also been introduced. This means that noise is introduced in the process of performing feature fusion. For the proposed method, the reconstructed depth map has sharp and clean edges.

 

Fig. 4. Exemplar scenes "Art" and "Reindeer" from Middlebury dataset are adopted for qualitatively compare the performance of different methods. (a) shows the ground truth 2D depth image of exemplar scenes "Art". (b) shows the reconstructed 2D depth map with MLE [33]. (c) shows the reconstructed 2D depth map with lindell’s method [14]. (d) shows the reconstructed 2D depth map with Peng’s method [17]. (e) shows the reconstructed 2D depth map with Zang’s method [18]. (f) shows the reconstructed 2D depth map with our proposed method.(a*), (b*), (c*), (d*), (e*) and (f*) are the closeup views of (a)-(f). (h)-(m) are the corresponding images from another exemplar scenes "Reindeer" with (h*)-(m*) Indicating the closeup views.

Download Full Size | PDF

4.3 Captured data

Apart from the simulated data, we use real scene data collected by the experimental setup to verify the performance of different methods. The captured data is obtained by conducting the indoor experiments with the experimental setup showed in Fig. 5. The experimental setup is divided into two main parts: the illumination light path and the imaging light path. We adopt the flood illumination method, where the laser (A.L.S. EIG1000AF Picosecond Diode Laser, PIL063F) emits laser pulses at a repetitive rate of 20 MHz and wavelength of 637nm through a diffuser to cover the imaging scene. The 2mW average power of the laser is affordable for the illumination method at a close distance(1.5m). The imaging optical path consists of a SPAD array sensor (Photon Force PF32) with the spatial resolution size of 32*32 and an imaging lens. All pixels of the SPAD array sensor can timestamp synchronously photon arrival time to generate the histogram data with a temporal resolution of 55 picosecond.

 

Fig. 5. Experimental setup for static scenes captured using the SPAD camera. The pulsed laser flood illuminates the scene through the diffuser triggered by synchronization signal. The SPAD camera timestamps the echo photon from the scene triggered by synchronization signal simultaneously. The experiments were conducted in a dark environment due to the low laser power.

Download Full Size | PDF

We use the experimental setup to capture three static scenes, including a windmill, two letters, hat and cup as the multi-object scene. Here the networks of Lindell, Peng, Zang and ours are all trained on the aforementioned simulated dataset in Sec.4.1, which are employed to reconstruct the captured data directly. The qualitative comparisons with other methods are shown in Fig. 6(c). As we can see, for the approximate maximum likelihood estimator(MLE), the method fails to filter out noise and the reconstructed object edges appear jagged. For lindell’s method, although there is a better denoising effect than MLE, but they fail to reconstruct the finer edge structure of objects. For Peng’s method, the main structure of the target has been effectively reconstructed, but the weak echo region marked by the blue boxes can’t be reconstructed, such as the thin rod of the windmill. For Zang’s method, compared with the first three methods, the weak echo edge region is reconstructed, but the recovered depth image is not sharp enough at the edges compared with the intensity image, such as the upper part of the windmill and the inner edge part of the letter N marked by red boxes. As a comparison, the reconstructed depth map by our method is more similar to the intensity map compared to other methods. Our method not only reconstructs the weak echo region, but also reconstructs the depth map with sharp edges. Through qualitative comparison, it is found that the method by improving the edge reconstruction accuracy achieves more superior performance for depth reconstruction, compared to the previous method.

 

Fig. 6. (a) Intensity images of the three scenes including letter, windmill and hat-cup.(b) SPAD data captured by experimental setup shown in Fig.5. (c) Reconstructed 2D depth maps computed from the captured data. The first column is the reconstruction of the depth with MLE [32], the second column is the reconstruction of the depth with Lindell’s method [14], the third column is the reconstruction of the depth with Peng’s method [17], the fourth column is the reconstruction of the depth with Zang’s method [18], and the final column is the reconstruction of the depth with our proposed method. Blue boxes in figures mark out regions with extremely weak signal return, red boxes in figures mark out regions with sharp edges.

Download Full Size | PDF

4.4 Ablation study

In this section, we investigate what effect the addition of the ADAG module, AG module and the addition of gradient regularization play on the performance of the network, respectively. For comparison, we use four versions of EeNet: one without the gradient regularization(EeNet w/o gradient), one without the ADAG module(EeNet w/o ADAG), one without the AG module(EeNet w/o AG), and one with all of these components(EeNet). We train each version under the SBR level 0.04.

Quantitative results of different versions from the Middlebury dataset are showed in Table 2. As can be seen from the results, the complete version of EeNet obtains the best result over both evaluation metrics. This result again proves that the improvement of edge accuracy is helpful for depth reconstruction.

Table 2. Quantitative comparison from the ablation study of proposed EeNet. The reconstructed result of the three versions on 6 Middlebury scenes with SBR level 0.04. The evaluation metrics RMSE and GradRMSE indicate the depth reconstruction accuracy and edge reconstruction accuracy, respectively. The lower the value of the evaluation metrics, the higher the reconstruction accuracy. Bold numeric characters represent the best performance.

View Table | View all tables in this article

We use exemplar scene "Art" to qualitatively compare the performance of different versions for deep reconstruction, as shown in Fig. 7. For the version lack of gradient regularization(EeNet w/o gradient), reconstructed depth maps have false edges. For example, the long pens have burrs at the edge as shown in Fig. 7(b*). For the version lack of ADAG module(EeNet w/o ADAG), details of the reconstructed depth map are lost. For example, the top of the long pens is missing as shown in Fig. 7(c*). For the version lack of AG module(EeNet w/o AG), the top of the long pens is restored, but the torso is missing as illustrated in Fig. 7(d*). It benefit from introduction of the CBAM module. The CBAM module captures the non-local spatial-temporal correlations within 3D SPAD measurements, which is beneficial for depth reconstruction to enhance structural edge information. For the complete version of EeNet, it can be seen from the reconstruction results as shown in Fig. 7(e*). Firstly the reconstruction of fine edges is improved compared to the version lack of AG module(EeNet w/o AG), because of adding the AG module into the the process of data transfer from encoder end to decoder end within UNet++. Secondly cleaner and sharper edge is achieved than the version lack of gradient regularization(EeNet w/o gradient). It benefit from the addition of the gradient regularization, which impose constraints on edges in order to obtain higher reconstruction accuracy.

 

Fig. 7. Exemplar scenes "Art" from Middlebury dataset is adopted for qualitatively compare from the ablation study of proposed EeNet. (a) shows the ground truth 2D depth image of exemplar scenes "Art". (b) shows the reconstructed 2D depth map with the version lack of gradient regularization(EeNet w/o gradient). (c) shows the reconstructed 2D depth map with the version lack of ADAG module(EeNet w/o ADAG). (d) shows the reconstructed 2D depth map with the version lack of AG module(EeNet w/o AG). (e) shows the reconstructed 2D depth map with the complete version of EeNet. (a*), (b*), (c*), (d*) and (e*) are the closeup views of (a)-(e).

Download Full Size | PDF

5. Conclusion

We analyze the distribution of SPAD data at different locations in the imaging scene and find that the echo signal recorded at the edge location of the object is weaker than that at the non-edge location due to light scattering. The range images can be seen as a patchwork of smooth regions divided by edges. Therefore we propose the edge reconstruction accuracy enhancement method to improve the depth map reconstruction accuracy. The method is implemented by adding an attention-directed attention gate module to the U-Net++ network. Besides we add the gradient regularization function to constrain edge reconstruction. Extensive experiments on simulated data demonstrate that our proposed method achieves significant reconstruction accuracy improvements compared to other state-of-the-art methods. Meanwhile, it is verified that improvement of the edge reconstruction accuracy promotes the overall depth reconstruction accuracy by our proposed evaluation metrics. In addition to the excellent performance shown on the simulated data, our method also shows optimal performance in deep reconstruction of real scenes compared to other methods. Therefore, we believe that the proposed edge enhancement method can be extended to other applications of active 3D imaging with strict restrictions on optical flow.