Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Refraction correction for deep-water three-dimensional visual measurement based on multi-objective optimization

Open Access Open Access

Abstract

Refraction-induced errors affect the accuracy of three-dimensional visual measurements in deepwater environments. In this study, a binocular camera refractive imaging model was established, and a calibration method for the refraction parameters was proposed for high-accuracy shape and deformation measurements in deep-water environments. First, an initial estimate of the refractive axis was obtained using a three-dimensional calibration target. Then, the errors in the distance between the spatial point pairs and the reprojection errors are taken as the dual optimization objectives, and the Non-dominated Sorting Genetic Algorithm II is applied to optimize the refraction parameters. To efficiently calculate the reprojection error, an improved numerical computation method is proposed to accelerate the calculation of the analytical forward projection. Underwater experiments were conducted to verify the method’s effectiveness. The results showed that the average error of the absolute position of the reconstructed points was less than 1.1 mm and the average error of the displacement was less than 0.04 mm. This study provides a sound solution for accurate three-dimensional visual measurement in deep-water environments.

© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

1. Introduction

Underwater acoustic sensing for long-distance and wide-range target detection typically suffers from low resolution [14]. To address this dilemma, underwater-vision-based sensing, especially in deep-water environments, has recently attracted extensive attention [57]. Vision-based methods can obtain rich information about underwater targets within a close range with high resolution and low cost. Therefore, it has been applied in various fields including water engineering inspection, ROV mechanical arm navigation, marine archeology, underwater search, and marine bioscience. In deep-water environments, a camera is typically enclosed in a cylindrical waterproof housing with a thick observation window to withstand the tremendous hydrostatic pressure. Therefore, light rays must pass through multiple media before being captured by a camera. If a pinhole camera model and parameters calibrated in air are employed, significant measurement errors are introduced because of refraction [8].

Numerous studies have attempted to eliminate the effect of refraction on underwater visual measurements. Early studies approximated the effects of refraction by treating it as a change in camera focal length or lens distortion based on a pinhole camera model [911]. Lavest suggested that if the calibration method used in air were directly applied to water, the focal length would be multiplied by the water index, while the image principal point seemed unchanged, and the radial distortion would be significantly different [9]. However, owing to the strong nonlinearity of underwater imaging, the systematic errors in these methods are large and cannot be ignored. Recently, the establishment of accurate underwater imaging models and the development of calibration methods has attracted significant interests [1225]. Kunz et al. developed a theory for an underwater hemispherical refraction model and a camera calibration method [12]. Treibitz et al. established a simple physical model to describe camera refractive imaging, and derived a forward projection applicable to a model based on Snell’s law [13]. However, the thickness of the observation window was not considered and the optical axis was assumed to be perpendicular to the window. Agrawal et al. established an innovative theory for multi-layer flat refractive geometries [14]. A refractive axis estimation method based on coplanar constraints was proposed, and other refraction parameters were calculated; however, only a single-camera refractive imaging model was used, and the derived forward projection was quite complex. Chen et al. proposed a method for calculating the thickness of all refractive layers with a known refractive axis and relative pose of the binocular camera [15]. Su et al. established a binocular refractive imaging model with a shared refractive surface [16,17]. When calculating the refractive axis, marks must be manually laid out on the refractive surface, which limits the application of this method in deep-water environments. To solve this problem, some researchers have adopted a complete camera refractive imaging model for refraction correction. Jordt-Sedlazeck and Koch developed the concept of a virtual camera to calculate the virtual reprojection error as an objective function [18,19]. Both Zhang [20] and Kong [21] used spatial features related to a calibration target to establish an objective function. In fact, they all attempted to avoid the complexity of forward projection; therefore, minimization of the reprojection error was not considered. Refraction correction methods for deepwater three-dimensional (3D) visual measurements, particularly for high-accuracy shape and deformation measurements, have not been well studied.

To develop a sound 3D visual method applicable to deep-water environments, an underwater binocular camera imaging model was established, and a calibration method for the refraction parameters was proposed accordingly. Calibration of the refractive parameters can be divided into two steps: initial estimation and optimization based on a heuristic algorithm. Three experiments were conducted to test the performance of the proposed method in underwater 3D reconstruction. The main contributions of this study are as follows:

  • (1) By using a 3D calibration target, a more accurate initial estimation of the refractive axis was obtained. It should be noted that 3D calibration targets are inexpensive and easily obtained through photogrammetry.
  • (2) Reasonable and explicit dual optimization objectives were constructed, including reprojection and distance errors. We point out that because of the nonlinearity of the underwater imaging model, the errors on both the image and space sides should be minimized simultaneously.
  • (3) A numerical method for forward projection is proposed that significantly reduces complexity and accelerates calculation.
  • (4) An underwater displacement experiment was conducted to verify the displacement measurement accuracy of this method, which has rarely been reported in previous studies.

The remainder of this paper is organized as follows. Section 2 introduces the proposed calibration method, including the initial estimation and detailed optimization of the refractive parameters. Section 3 presents the experiments and an analysis of the results. In Section 4, we discuss the proposed method. Finally, the conclusions are presented in Section 5.

2. Method

2.1 Binocular camera refractive imaging model in deep water

Considering the hydrostatic pressure in deepwater environments, waterproof housing is typically designed to be compact. Therefore, binocular cameras are typically enclosed in a separate waterproof housing with a thick flat observation window. As shown in Fig. 1, a ray starting from Point ${P_W}$ travels through three different media: water, observation window, and air, undergoing refraction twice before being imaged by the camera sensor.

 figure: Fig. 1.

Fig. 1. Binocular camera refractive imaging model in deep water

Download Full Size | PDF

For the left camera, the interface between water and observation window is ${\pi _2}$ and the interface between air and observation window is ${\pi _1}$. The intersections of the ray path with ${\pi _1},{\pi _2}$ are $P_l^1,P_l^2$, respectively. ${p_l}$ is the projection of ${P_W}$ on the left camera sensor. The normal vector of ${\pi _1},{\pi _2}$ is ${{\boldsymbol n}_l}$ in the left camera coordinate system ${O_l} - {X_l}{Y_l}{Z_l}$. ${d_l}$ is the distance from the optical center of left camera to ${\pi _1}$, and ${h_l}$ is the thickness of the observation window. The refractive index of air, observation window and water are ${u_1},{u_2},{u_3}$ respectively. The relative pose between the two cameras is described by the rotation matrix R and translation vector T. If the effect of refraction is not considered and triangulation is directly performed, the reconstructed point $P_W^{\prime}$ would differ significantly from the true position as shown in Fig. 1.

For the reconstruction of 3D points, it is assumed that the intrinsic parameters of the two cameras have been calibrated in advance, so the vector ${O_l}{p_l}$ and ${O_r}{p_r}$ can be determined. When light rays pass through media with different refractive indices ${u_i},{u_{i + 1}}$ as shown in Fig. 2(a), according to Snell's law

$${{\boldsymbol v}_{i + 1}} = \frac{{{u_i}}}{{{u_{i + 1}}}}{{\boldsymbol v}_i} + \frac{{ - {u_i}{\boldsymbol v}_i^T{\boldsymbol n} - \sqrt {u_i^2{{({\boldsymbol v}_i^T{\boldsymbol n})}^2} - (u_i^2 - u_{i + 1}^2){\boldsymbol v}_i^T{{\boldsymbol v}_i}} }}{{{u_{i + 1}}}}{\boldsymbol n}$$
where ${\boldsymbol n}$ is the normal vector of media interface. Therefore, once the variables highlighted in red in Fig. 1 $\Omega ({d_l},{d_r},{h_l},{h_r},{{\boldsymbol n}_l},{{\boldsymbol n}_r},R,T)$ are determined, vectors in observation windows and water $P_l^1P_l^2,P_r^1P_r^2$and$P_l^2{P_W},P_r^2{P_W}$ can be determined as well. The reconstructed points ${P_W}$ will be the point with the smallest sum of distances between $P_l^2{P_W}$ and $P_r^2{P_W}$. Variables $\Omega ({d_l},{d_r},{h_l},{h_r},{{\boldsymbol n}_l},{{\boldsymbol n}_r},R,T)$ are defined as the refraction parameters that need to be calibrated.

 figure: Fig. 2.

Fig. 2. (a) Schematic diagram of Snell's law, (b) analytical forward projection (AFP) on refractive plane

Download Full Size | PDF

According to Agrawal et al. [14], the analytical forward projection (AFP) is a 12th degree equation when three different media exist. The AFP can be used to compute the projection of a known 3D point on the camera sensor; however, the conventional computation method has the shortcomings of complexity and time consumption. To realize comprehensible and fast computation of the AFP, an improved numerical computation method was proposed, as shown in Eq. (2) to (6), respectively.

In multi-layer flat refractive geometry, all rays on the propagation path are in the same plane, which is defined as refractive plane. As shown in Fig. 2(b), considering the coordinate system $O - {X_n}{Y_n}$ established on the refraction plane, O is the optical center of camera, $O{Y_n}$ is parallel to n. A point $P(X,Y,Z)$ is in the camera coordinate system $O - XYZ$, and ${P^{\prime}}(s,z)$ in $O - {X_n}{Y_n}$ is the transformed coordinate of P. The refractive angles in air, observation window, and water are ${\theta _1},{\theta _2},{\theta _3}$ respectively. According to Snell's law:

$${u_1}\sin {\theta _1} = {u_2}\sin {\theta _2} = {u_3}\sin {\theta _3} = a$$
where a is a constant. According to the geometric constraints shown in Fig. 2(b),
$$d\tan {\theta _1} + h\tan {\theta _2} + (z - d - h)\tan {\theta _3} = s$$

There are other equations as follows:

$$\left\{ \begin{array}{l} \tan {\theta_i} = \frac{{\sin {\theta_i}}}{{\sqrt {1 - \sin {\theta_i}^2} }}\\ \sin {\theta_i} = \frac{a}{{{u_i}}} = {k_i}a \end{array} \right.,i = 1,2,3$$

Substituting Eq. (4) into Eq. (3), we get

$$f(a) = d\frac{{{k_1}a}}{{\sqrt {1 - {k_1}^2{a^2}} }} + h\frac{{{k_2}a}}{{\sqrt {1 - {k_2}^2{a^2}} }} + (z - d - h)\frac{{{k_3}a}}{{\sqrt {1 - {k_3}^2{a^2}} }} - s = 0$$

The problem is transformed into finding the solution when $f(a) = 0$.

Consider the following function:

$$g(x) = \frac{x}{{\sqrt {1 - {x^2}} }},x \in [{0,1} )$$
$g(x)$ is monotonically increasing within [0,1). Because ${\theta _i} \in [0,9{0^\circ })$, ${k_i}a = \sin {\theta _i} \in [0,1)$, $f(a) = d\ast g({k_1}a) + h\ast g({k_2}a) + (z - d - h)\ast g({k_3}a) - s$, $f(a)$ is also monotonically increasing and has a unique zero point. The value of a is obtained by iteration using the Newton's method. Consider ${\theta _0}$ as the angle between $O{Y_n}$ and $O{P^{\prime}}$, as shown in Fig. 2(b). The initial estimation of a will be $\sin {\theta _0}$, since ${u_1}\sin {\theta _1} = a$, ${u_1}$ is usually 1 and ${\theta _0}$ is very close to ${\theta _1}$. Due to the monotonically increasing function properties and good initial value estimation, a can be calculated in 4-5 iterations with a final error less than 10−5 pixels, which is fast and accurate. Subsequently, the coordinate of ${P^1}$ in $O - {X_n}{Y_n}$ is calculated and converted back to camera coordinate system $O - XYZ$. At last, the pixel coordinates p will be obtained by performing a perspective transformation to the converted ${P^1}$.

2.2 Initial estimation of refraction parameters

As shown in Fig. 2(b), ${{\boldsymbol v}_0}$ is the unit vector in the $Op$ direction. Assuming a point ${P_W}$ in the world coordinate system and is transformed to camera coordinate system as $P = {R_1}{P_W} + {T_1}$. Notably, ${{\boldsymbol v}_0}$ and n are both in the camera coordinate system. According to Agrawal et al. [14], the coplanarity constraint can be written as

$${\boldsymbol v}_{\bf 0}^T \cdot ({\boldsymbol n} \times ({R_1} \cdot {P_w} + {T_1})) = 0$$

The coplanarity constraint is re-written as

$${\boldsymbol v}_{\bf 0}^TE{P_w} + {\boldsymbol v}_{\bf 0}^Ts = 0$$
where $E = {[{\boldsymbol n}]_ \times }{R_1}$ and $s = {\boldsymbol n} \times {T_1}$. If there are N correspondences in ${P_W}$ and ${{\boldsymbol v}_0}$, we obtain
$$\left[ {\begin{array}{cc} {{P_w}{{(1)}^T} \otimes {{\boldsymbol v}_{\bf 0}}{{(1)}^T}}&{{{\boldsymbol v}_{\bf 0}}{{(1)}^T}}\\ \vdots & \vdots \\ {{P_w}{{(N)}^T} \otimes {{\boldsymbol v}_{\bf 0}}{{(N)}^T}}&{{{\boldsymbol v}_{\bf 0}}{{(N)}^T}} \end{array}} \right]\left[ \begin{array}{c} E(1:9)\\ s \end{array} \right] = B\left[ \begin{array}{c} E(1:9)\\ s \end{array} \right] = 0$$

During $N \ge 12$, the solutions of E and s are given by the eigenvector corresponding to the minimum eigenvalue of matrix B after SVD. Since

$${{\boldsymbol n}^T} \cdot E = {{\boldsymbol n}^T} \cdot {[{\boldsymbol n}]_ \times }R = {E^T} \cdot {\boldsymbol n} = 0$$

The solution of n is given by the eigenvector corresponding to the minimum eigenvalue of matrix ${E^T}$ after SVD decomposition. If there is only one set of correspondences, then only one matrix ${E^T}$ will be obtained. If there is significant random noise in the correspondences of ${P_W}$ and ${{\boldsymbol v}_0}$, it will affect the accuracy of n. Thereby, we created a 3D calibration target as shown in Fig. 3. The 3D coordinate of the encoded points on the target is obtained by the close-range photogrammetry method, the distance between two points can be calculated. Taking M underwater photos of the calibration target in different poses, we can get M matrix ${E^T}$. A more robust results of n is given by the eigenvector corresponding to the minimum eigenvalue of matrix C after SVD decomposition as shown in Eq. (11).

$$\left[ \begin{array}{l} E_1^T\\ \ldots \\ E_M^T \end{array} \right]{\boldsymbol n} = C{\boldsymbol n} = 0$$

Because observation windows are typically made in advance, the thickness h of the observation window was measured using the Vernier scale as a constant. The extrinsic parameters between the two cameras, R and T are obtained by Zhang’s calibration using a planar calibration board in air. Although there was an observation window, its impact on the calculation of the relative pose was very small. d is calculated based on Chen’s method [15], which calculates the thickness of all refractive layers with a known refractive axis and the relative pose of the binocular camera.

 figure: Fig. 3.

Fig. 3. 3D calibration target

Download Full Size | PDF

 figure: Fig. 4.

Fig. 4. Experimental setup for axis estimation.

Download Full Size | PDF

2.3 Optimization of refraction parameters based on NSGA-II

Considering the strong nonlinearity of underwater camera imaging, we used heuristic algorithms to optimize refraction parameters. NSGA-II (Non-dominated Sorting Genetic Algorithm II) is a multi-objective optimization algorithm developed by Kalyanmoy Deb [26]. NSGA-II uses a fast and efficient sorting technique to rank solutions based on their non-dominance and diversity, and employs a crowding distance measure to preserve a diverse set of solutions. The choice of the optimization objective is the most important issue in the NSGA-II. To ensure minimal errors on both the image and space sides, the optimization objective f1 was defined as the average reprojection error of all points, and f2 was defined as the average distance error of all selected point pairs. The objective functions are expressed as follows:

$${f_1} = \frac{1}{{2MN}}\sum\limits_{m = 1}^M {\sum\limits_{n = 1}^N {(||{p_l^{mn} - \bar{p}_l^{mn}} ||} } + ||{p_r^{mn} - \bar{p}_r^{mn}} ||)$$
$${f_2} = \frac{1}{{MI}}\sum\limits_{m = 1}^M {\sum\limits_{i = 1}^I {|{d_i^m - {{\bar{d}}_i}} |} }$$
where M is the number of 3D calibration target images, N is the number of coded points on the 3D calibration target, I is the number of selected 3D point pairs, p is the re-projection coordinate of the reconstructed point, $\overline p$ is the detected coordinate of the feature point coordinates, d is the distance between the reconstructed point pairs, and $\overline d$ is the true distance. To reduce the number of parameters to be optimized, replace nz in the refractive axis using $\sqrt {1 - {{({n^x})}^2} - {{({n^y})}^2}}$. The rotation matrix R was converted to angles using ${\omega _1},{\omega _2},{\omega _3}$. Finally, 12 parameters were optimized using ${d_l},{d_r},n_l^x,n_l^y,n_r^x,n_r^y,{\omega _1},{\omega _2},{\omega _3},{t_1},{t_2},{t_2}$.

3. Experiments and results

3.1 Results of refraction parameters

First, the accuracy of the refractive axis estimation was verified. Because it is difficult to directly obtain the true orientation of the refractive axis of the camera coordinate system, we designed the experiment shown in Fig. 4, where a high-precision electric-driven rotary stage rotated a fixed camera 2.5° four times and 10-20 photos of the 3D calibration target were taken each time. The accuracy of the proposed method was verified by calculating the change in the angle along the refractive axis. We compared the results of our algorithm with those of the 11-point algorithm proposed by Agrawal et al. [14]. Table 1 shows that our algorithm is significantly superior to the 11-point algorithm and that the measurement error of the relative rotation angle is less than 3%.

Tables Icon

Table 1. Results of refractive axis rotation angle

Figure 5(a) shows the waterproof housing and underwater binocular camera system. The entire system operated stably for several hours under a pressure of 3.75 MPa, as shown in Fig. 5(b). We conducted a series of experiments based on the underwater binocular camera system, including calibration and underwater 3D reconstruction. Fifteen underwater images of 3D calibration targets in different poses were captured using binocular cameras, which were used for both initial value estimation and optimization. Calibration method is described in Section 2. Table 2 lists the initial values of the refractive parameters obtained from the proposed method and the range used for optimization.

 figure: Fig. 5.

Fig. 5. (a) Waterproof housing, (b) pressure experiment

Download Full Size | PDF

Tables Icon

Table 2. Values of refraction parameters

Figure 6 shows the changes in the values of the optimization objectives during the optimization process, where each point represents an individual in the population. As the iterations progressed, the value of the objective function rapidly decreased and tended to stabilize around the 50th generation, with little change until the 100th generation. Specifically, when the iteration tends to stabilize, owing to the effect of nondominated sorting, some individuals make f1 very small but f2 large, whereas others make the opposite. The selected individuals in the red circle c shown in Fig. 6 make both f1 and f2 relatively small; therefore, the final results will be obtained from these individuals. Based on the experimental results and considering computational efficiency and accuracy, the recommended population size was 100, and the number of iterations was 50.

 figure: Fig. 6.

Fig. 6. Optimization process based on NSGA-II

Download Full Size | PDF

Part of the detailed optimization results were presented in Fig. 7. We selected point pairs 129-221, 207-223, and 156-195, and obtained the distance error between the true distance $\overline d$ and the distance d calculated by reconstructed points, $\Delta = d - \overline d$. Coded points 207, 223, and 117 were selected and reprojection error were calculated as $\Delta = |p - \overline p |$. The results indicate that among the 15 calibration poses, all the distance errors of the three-point pairs lie in -0.3∼0.3 mm, and all the reprojection errors are less than 0.25 pixel. The final results of refraction parameters are listed in the last column in Table 2.

 figure: Fig. 7.

Fig. 7. Optimization results: (a) distance error, (b) reprojection error

Download Full Size | PDF

3.2 Performance of underwater 3D measurement

Three experiments were conducted to demonstrate the performance of the proposed method in underwater 3D reconstruction. First, the absolute positional error of the reconstructed point cloud was studied. We placed a planar calibration board in the water tank to ensure that the relative position between the binocular camera and calibration board was unchanged and calculated the 3D coordinates of the feature points in air and water. When in air, the observation window of the waterproof housing was removed to ensure measurement accuracy (In principle, the waterproof housing of camera used in deep water environments should not be opened after being sealed properly). As shown in Fig. 8(b), considering the 3D coordinates in air as the ground truth, the positions of the underwater calibration board feature points obtained using the proposed method were very close to the ground truth, with an average of 1.081 mm. Furthermore, the re-projection error of the reconstructed points was very small, with an average of 0.506 pixels, as shown in Fig. 8(a).

 figure: Fig. 8.

Fig. 8. Performance of the optimized refraction parameters: (a) reprojection error, (b) absolute position error

Download Full Size | PDF

The experimental setup shown in Fig. 9 was designed to test the displacement measurement accuracy of the proposed method. A plate with speckles was fixed to an electrically driven translation stage to generate horizontal and vertical displacements. The horizontal direction x was parallel to the camera baseline, and the vertical direction y was perpendicular to the camera baseline, as shown in Fig. 9. The translation stage was moved by 1 mm 10 times. The calculated full-field displacements are listed in Table 3. In both the vertical and horizontal directions, the average displacement error was less than 0.040 mm, and the standard deviation was less than 0.032 mm, which illustrates the reliability of the method.

 figure: Fig. 9.

Fig. 9. Experimental setup for underwater displacement experiments

Download Full Size | PDF

Tables Icon

Table 3. Results of displacement experiments (mm)

Finally, the three-dimensional surface morphology of the concrete bricks was measured. As shown in Fig. 10(b), we used a laser projector to project a random speckle onto the surface of the measured object, and then obtained the 3D point cloud of the concrete brick surface using the proposed method. The morphology provides rich information and high accuracy, and the Chinese characters on the concrete bricks are clearly visible, as shown in Fig. 10(c).

 figure: Fig. 10.

Fig. 10. Underwater 3D reconstruction: (a) concrete brick, (b) projected speckle, (c) reconstructed point cloud

Download Full Size | PDF

4. Discussion

We attempt to demonstrate the reason for adopting a 3D calibration target and dual optimization objectives. For a planar calibration board, the constraints on corner spacing, parallelism, and perpendicularity between lines must be considered simultaneously. Excessive optimization objectives will obviously slow down the optimization or easily trap the optimization results in local optima. Using a 3D calibration target facilitates the construction of simple and strong constraints on the spatial side. In Section 2, we demonstrate that multiple images of a 3D calibration target can be used for more accurate estimation of the refractive axis. In fact, 3D calibration targets based on close-range photogrammetry are not only cheap but also have high precision and are easy to make. Thus, a 3D calibration target was selected for the proposed method.

Examining the 3D reconstruction performance of individuals in the red circle a in Fig. 6, the average absolute position error was 7.168 mm, and the average reprojection error was 0.492 pixels. For the individuals in red circle b, the average absolute position error was 1.052 mm and the average reprojection error was 2.110 pixels, as shown in Fig. 11. This indicates that if only one single-side optimization objective is adopted, the optimization results may be satisfactory; however, the parameters do not truly match the model. This is probably due to the strong nonlinearity of underwater imaging. Therefore, errors on both the image and space sides were supposed to be minimized simultaneously to achieve a stronger bundling adjustment. Furthermore, the proposed numerical method for forward projection significantly accelerated the calculation, making it feasible to use the reprojection error as the optimization objective.

 figure: Fig. 11.

Fig. 11. (a) reprojection error based on individual in red circle b, (b) absolute position error based on individual in red circle a

Download Full Size | PDF

5. Conclusion

We propose a sound refraction correction method for deep-water 3D visual measurements. Using a 3D calibration target, we obtain a more accurate initial estimate of the refractive axis. The reprojection and distance errors were defined as the optimization objectives, and the reasons for choosing them were discussed in detail. Because of the nonlinearity of the underwater imaging model, errors on both the image and space sides were assumed to be minimized simultaneously. The proposed numerical method for forward projection significantly reduces complexity and accelerates forward projection. The projected coordinates can be calculated in 4-5 iterations with a final error of less than 10−5 pixels.

Experiments verified the performance of the proposed method for underwater 3D reconstruction. The average absolute position error of the reconstructed points was 1.081 mm and the average reprojection error was 0.506 pixels. The results of the displacement experiment indicate that in both the vertical and horizontal directions, the average displacement error is less than 0.040 mm, and the standard deviation is less than 0.032 mm, which illustrates the reliability of the method.

Funding

National Key Research and Development Program of China (2020YFC1511900); National Natural Science Foundation of China (12272093).

Disclosures

The authors declare that there are no conflicts of interest related to this article.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. Y. Cong, C. J. Gu, T. Zhang, and Y. J. Gao, “Underwater robot sensing technology: A survey,” Fundam. Res. 1(3), 337–345 (2021). [CrossRef]  

2. 2. C. Y. Li, C. L. Guo, W. Q. Ren, R. M. Cong, J. H. Hou, S. Kwong, and D. C. Tao, “An Underwater Image Enhancement Benchmark Dataset and Beyond,” IEEE Trans Image Process. 29, 4376–4389 (2020). [CrossRef]  

3. A. Palomer, P. Ridao, and D. Ribas, “Inspection of an underwater structure using point-cloud SLAM with an AUV and a laser scanner,” J. Field Rob. 36(8), 1333–1344 (2019). [CrossRef]  

4. G. Telem and S. Filin, “Photogrammetric modeling of underwater environments,” ISPRS J Photogram Remote Sens. 65(5), 433–444 (2010). [CrossRef]  

5. M. P. Hayes and P. T. Gough, “Synthetic Aperture Sonar: A Review of Current Status,” IEEE J. Oceanic Eng. 34(3), 207–224 (2009). [CrossRef]  

6. D. P. Williams and Ieee, “Underwater Target Classification in Synthetic Aperture Sonar Imagery Using Deep Convolutional Neural Networks,” in 23rd International Conference on Pattern Recognition (ICPR), International Conference on Pattern Recognition (2016), pp. 2497–2502.

7. Y. C. Yu, J. H. Zhao, Q. H. Gong, C. Huang, G. Zheng, and J. Y. Ma, “Real-Time Underwater Maritime Object Detection in Side-Scan Sonar Images Based on Transformer-YOLOv5,” Remote Sens. 13(18), 3555 (2021). [CrossRef]  

8. Y. H. Kwon and J. B. Casebolt, “Effects of light refraction on the accuracy of camera calibration and reconstruction in underwater motion analysis,” Sports Biomech. 5(1), 95–120 (2006). [CrossRef]  

9. J. M. Lavest, G. Rives, and J. T. Lapresté, “Underwater Camera Calibration,” in Computer Vision — ECCV 2000 (Springer, Berlin Heidelberg, 2000), pp. 654–668.

10. R. Ferreira, J. P. Costeira, and J. A. Santos, “Stereo reconstruction of a submerged scene,” in Pattern Recognition and Image Analysis, Pt 1, Proceedings, J. S. Marques, N. PerezdelaBlanca, and P. Pina, eds. (2005), pp. 102–109.

11. L. Kang, L. D. Wu, and Y. H. Yang, “Experimental study of the influence of refraction on underwater three-dimensional reconstruction using the SVP camera model,” Appl. Opt. 51(31), 7591–7603 (2012). [CrossRef]  

12. C. KunzH. Singh, Mts, and Ieee, “Hemispherical Refraction and Camera Calibration in Underwater Vision,” in OCEANS 2008 Conference, Oceans-Ieee (2008), pp. 1097–1103.

13. T. Treibitz, Y. Y. Schechner, and H. Singh, “Flat refractive geometry,” in IEEE Conference on Computer Vision & Pattern Recognition (2008).

14. A. Agrawal, S. Ramalingam, Y. Taguchi, V. Chari, and Ieee, “A Theory of Multi-Layer Flat Refractive Geometry,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Conference on Computer Vision and Pattern Recognition (2012), pp. 3346–3353.

15. X. D. ChenY. H. Yang, and Ieee, “Two-View Camera Housing Parameters Calibration for Multi-Layer Flat Refractive Interface,” in 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Conference on Computer Vision and Pattern Recognition (2014), pp. 524–531.

16. Z. L. Su, J. Y. Pan, L. Lu, M. L. Dai, X. Y. He, and D. S. Zhang, “Refractive three-dimensional reconstruction for underwater stereo digital image correlation,” Opt. Express 29(8), 12131–12144 (2021). [CrossRef]  

17. Z. L. Su, J. Y. Pan, S. Q. Zhang, S. Wu, Q. F. Yu, and D. S. Zhang, “Characterizing dynamic deformation of marine propeller blades with stroboscopic stereo digital image correlation,” Mech Syst Sig Proces 162, 108072 (2022). [CrossRef]  

18. A. Jordt-SedlazeckR. Koch, and Ieee, “Refractive Structure-from-Motion on Underwater Images,” in IEEE International Conference on Computer Vision (ICCV), IEEE International Conference on Computer Vision (2013), pp. 57–64.

19. A. Jordt-Sedlazeck and R. Koch, “Refractive Calibration of Underwater Cameras,” in Computer Vision – ECCV 2012 (Springer, Berlin Heidelberg, 2012), 846–859.

20. C. Zhang, X. Zhang, Y. K. Zhu, J. B. Li, and D. W. Tu, “Model and calibration of underwater stereo vision based on the light field,” Meas. Sci. Technol. 29(10), 105402 (2018). [CrossRef]  

21. S. H. Kong, X. Fang, X. Y. Chen, Z. X. Wu, and J. Z. Yu, “A NSGA-II-Based Calibration Algorithm for Underwater Binocular Vision Measurement System,” IEEE Trans. Instrum. Meas. 69(3), 794–803 (2020). [CrossRef]  

22. Y. J. ChangT. H. Chen, and Ieee, “Multi-View 3D Reconstruction for Scenes under the Refractive Plane with Known Vertical Direction,” in IEEE International Conference on Computer Vision (ICCV), IEEE International Conference on Computer Vision (2011), 351–358.

23. R. Rofallski and T. Luhmann, “An Efficient Solution to Ray Tracing Problems in Multimedia Photogrammetry for Flat Refractive Interfaces,” Pfg-J Photogram Remote Sens Geoinfo Sci. 90(1), 37–54 (2022). [CrossRef]  

24. Y. P. Ma, Y. Q. Zhou, C. K. Wang, Y. Wu, Y. Zou, and S. Zhang, “Calibration of an underwater binocular vision system based on the refraction model,” Appl. Opt. 61(7), 1675–1686 (2022). [CrossRef]  

25. B. Elnashef and S. Filin, “Target-free calibration of flat refractive imaging systems using two-view geometry,” Opt. Laser Eng. 150, 106856 (2022). [CrossRef]  

26. A. P. K. Deb, “A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii,” KanGAL report, Indian Institute of Technology (2000). [CrossRef]  

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1.
Fig. 1. Binocular camera refractive imaging model in deep water
Fig. 2.
Fig. 2. (a) Schematic diagram of Snell's law, (b) analytical forward projection (AFP) on refractive plane
Fig. 3.
Fig. 3. 3D calibration target
Fig. 4.
Fig. 4. Experimental setup for axis estimation.
Fig. 5.
Fig. 5. (a) Waterproof housing, (b) pressure experiment
Fig. 6.
Fig. 6. Optimization process based on NSGA-II
Fig. 7.
Fig. 7. Optimization results: (a) distance error, (b) reprojection error
Fig. 8.
Fig. 8. Performance of the optimized refraction parameters: (a) reprojection error, (b) absolute position error
Fig. 9.
Fig. 9. Experimental setup for underwater displacement experiments
Fig. 10.
Fig. 10. Underwater 3D reconstruction: (a) concrete brick, (b) projected speckle, (c) reconstructed point cloud
Fig. 11.
Fig. 11. (a) reprojection error based on individual in red circle b, (b) absolute position error based on individual in red circle a

Tables (3)

Tables Icon

Table 1. Results of refractive axis rotation angle

Tables Icon

Table 2. Values of refraction parameters

Tables Icon

Table 3. Results of displacement experiments (mm)

Equations (13)

Equations on this page are rendered with MathJax. Learn more.

v i + 1 = u i u i + 1 v i + u i v i T n u i 2 ( v i T n ) 2 ( u i 2 u i + 1 2 ) v i T v i u i + 1 n
u 1 sin θ 1 = u 2 sin θ 2 = u 3 sin θ 3 = a
d tan θ 1 + h tan θ 2 + ( z d h ) tan θ 3 = s
{ tan θ i = sin θ i 1 sin θ i 2 sin θ i = a u i = k i a , i = 1 , 2 , 3
f ( a ) = d k 1 a 1 k 1 2 a 2 + h k 2 a 1 k 2 2 a 2 + ( z d h ) k 3 a 1 k 3 2 a 2 s = 0
g ( x ) = x 1 x 2 , x [ 0 , 1 )
v 0 T ( n × ( R 1 P w + T 1 ) ) = 0
v 0 T E P w + v 0 T s = 0
[ P w ( 1 ) T v 0 ( 1 ) T v 0 ( 1 ) T P w ( N ) T v 0 ( N ) T v 0 ( N ) T ] [ E ( 1 : 9 ) s ] = B [ E ( 1 : 9 ) s ] = 0
n T E = n T [ n ] × R = E T n = 0
[ E 1 T E M T ] n = C n = 0
f 1 = 1 2 M N m = 1 M n = 1 N ( | | p l m n p ¯ l m n | | + | | p r m n p ¯ r m n | | )
f 2 = 1 M I m = 1 M i = 1 I | d i m d ¯ i |
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.