GPLFR—Global perspective and local flow registration-for forward-looking sonar images

Abstract

Forward-looking sonar (FLS) image registration is a key step in many underwater applications such as underwater target detection, ocean observation, and mapping. However, low resolution, low signal-to-noise ratio, and the complex nonlinear transformation relationship between FLS images from two different viewpoints have brought great challenges to register them. In order to better cope with this challenge, we propose a global perspective and local flow registration (GPLFR) method for FLS images. GPLFR consists of two networks, i.e., a regression correction network (RCNet) and a deformable network (IRRDNet) with the iterative refinement of the residual. For a given pair of FLS images, RCNet is used to estimate the global transformation parameters to achieve global registration, and then, IRRDNet is used to estimate the deformation field or flow field to realize local alignment. The experimental results on real FLS image and 2D face expression image registration tasks demonstrate the effectiveness and robustness of the proposed method.

Appendix

In the sonar-based spherical coordinate system, the projection model of the sonar is shown in Fig. 9. The FLS projects the 3D point ${\mathbf {P}}$ in the scene onto the 2D image plane, and the projection point is ${\mathbf {P}}_{\mathrm {s}}$. The point ${\mathbf {P}}$ is defined as:

Fig. 9

Sonar coordinate system and projection model in mapping a 3D point onto zero-elevation plane

$$\begin{aligned}&{\mathbf {P}}=\left( \begin{array}{l} x \\ y \\ z \end{array}\right) =\left( \begin{array}{c} R \cos \varphi \sin \theta \\ R \cos \varphi \cos \theta \\ R \sin \varphi \end{array}\right) , \end{aligned}$$

(18)

$$\begin{aligned}&\left[ \begin{array}{l} R \\ \theta \\ \varphi \end{array}\right] =\left[ \begin{array}{c} \sqrt{x^{2}+y^{2}+z^{2}} \\ \tan ^{-1}(x / y) \\ \tan ^{-1}\left( z / \sqrt{x^{2}+y^{2}}\right) \end{array}\right] , \end{aligned}$$

(19)

where $\left[ \begin{array}{lll}x&y&z\end{array}\right] ^{T}$ is the Cartesian coordinates of point ${\mathbf {P}}$ and $\left[ \begin{array}{lll}R&\theta&\varphi \end{array}\right] ^{T}$ represents the distance, azimuth, and elevation angle of point ${\mathbf {P}}$ in the spherical coordinate system.

The projection point ${\mathbf {P}}_{\mathrm {s}}$ is defined as:

$$\begin{aligned} {\mathbf {P}}_{\mathrm {s}}=\left( \begin{array}{l} x_{s} \\ y_{s} \end{array}\right) =\left[ \begin{array}{l} R \sin \theta \\ R \cos \theta \end{array}\right] =\frac{1}{\cos \varphi }\left[ \begin{array}{l} x \\ y \end{array}\right] . \end{aligned}$$

(20)

Now suppose that the sonar device follows rigid body motion, and then the coordinate points ${\mathbf {P}}$ and ${\mathbf {P}}^{\prime }$ of different views of the same scene satisfy the following transformation relationship:

$$\begin{aligned} {\mathbf {P}}^{\prime }={\mathbf {R}} {\mathbf {P}}+{\mathbf {t}}, \end{aligned}$$

(21)

where ${\mathbf {R}}$ is $3 \times 3$ 3D rotation matrix and ${\mathbf {t}}$ is the 3D translation vector.

Let ${\mathbf {n}}=\left[ n_{x}, n_{y}, n_{z}\right] ^{T}$ be the scaled normal vector derived from the plane equation $Z=Z_{o}+\zeta _{x} X+\zeta _{y} Y$, and satisfying ${\mathbf {n}} \cdot {\mathbf {P}}=1$, and then [22]

$$\begin{aligned} {\mathbf {P}}^{\prime }=\left( {\mathbf {R}}+\mathbf {t n}^{T}\right) {\mathbf {P}}={\mathbf {Q}} {\mathbf {P}}, \end{aligned}$$

(22)

where

$$\begin{aligned} {\mathbf {Q}}=\left[ \begin{array}{lll} q_{11} &{} q_{12} &{} q_{13} \\ q_{21} &{} q_{22} &{} q_{23} \\ q_{31} &{} q_{32} &{} q_{33} \end{array}\right] . \end{aligned}$$

(23)

Using Eq.18, we can get

$$\begin{aligned} \left[ \begin{array}{c} R^{\prime } \cos \varphi ^{\prime } \sin \theta ^{\prime } \\ R^{\prime } \cos \varphi ^{\prime } \cos \theta ^{\prime } \\ R^{\prime } \sin \varphi ^{\prime } \end{array}\right] ={\mathbf {Q}}\left[ \begin{array}{c} R \cos \varphi \sin \theta \\ R \cos \varphi \cos \theta \\ R \sin \varphi \end{array}\right] , \end{aligned}$$

(24)

which can be rewritten:

$$\begin{aligned} \left[ \begin{array}{l} R^{\prime } \sin \theta ^{\prime } \\ R^{\prime } \cos \theta ^{\prime } \\ R^{\prime } \tan \varphi ^{\prime } \end{array}\right] =\left( \frac{\cos \varphi }{\cos \varphi ^{\prime }}\right) {\mathbf {Q}}\left[ \begin{array}{l} R \sin \theta \\ R \cos \theta \\ R \tan \varphi \end{array}\right] . \end{aligned}$$

(25)

Using Eq.20, we can get

$$\begin{aligned} \left[ \begin{array}{c} x_{s}^{\prime } \\ y_{s}^{\prime } \\ R^{\prime } \tan \varphi ^{\prime } \end{array}\right] =\left( \frac{\cos \varphi }{\cos \varphi ^{\prime }}\right) {\mathbf {Q}}\left[ \begin{array}{c} x_{s} \\ y_{s} \\ R \tan \varphi \end{array}\right] , \end{aligned}$$

(26)

and then

$$\begin{aligned} \begin{array}{l} x_{s}^{\prime }=\left( \frac{\cos \varphi }{\cos \varphi ^{\prime }}\right) \left[ q_{11} x_{s}+q_{12} y_{s}+q_{13} R \tan \varphi \right] , \\ y_{s}^{\prime }=\left( \frac{\cos \varphi }{\cos \varphi ^{\prime }}\right) \left[ q_{21} x_{s}+q_{22} y_{s}+q_{23} R \tan \varphi \right] , \end{array} \end{aligned}$$

(27)

where the projection point of point ${\mathbf {P}}^{\prime }$ on the 2D image plane is:

$$\begin{aligned} {\mathbf {P}}_{s}^{\prime }=\left( \begin{array}{c} x_{s}^{\prime } \\ y_{s}^{\prime } \end{array}\right) =\left[ \begin{array}{l} R^{\prime } \sin \theta ^{\prime } \\ R^{\prime } \cos \theta ^{\prime } \end{array}\right] =\frac{1}{\cos \varphi ^{\prime }}\left[ \begin{array}{l} x^{\prime } \\ y^{\prime } \end{array}\right] , \end{aligned}$$

(28)

Finally, sonar image points satisfy the transformation [6]:

$$\begin{aligned} \left[ \begin{array}{c} x_{s}^{\prime } \\ y_{s}^{\prime } \\ 1 \end{array}\right] ={\mathbf {H}}\left[ \begin{array}{c} x_{s} \\ y_{s} \\ 1 \end{array}\right] , \end{aligned}$$

(29)

where

$$\begin{aligned} {\mathbf {H}}=\left[ \begin{array}{ccc} \alpha q_{11} &{} \alpha q_{12} &{} \beta q_{13} \\ \alpha q_{21} &{} \alpha q_{22} &{} \beta q_{23} \\ 0 &{} 0 &{} 1 \end{array}\right] ; \alpha =\frac{\cos \varphi }{\cos \varphi ^{\prime }}, \beta =R \frac{\sin \varphi }{\cos \varphi ^{\prime }}. \end{aligned}$$

(30)

Although it appears to be an affine model, the elements of ${\mathbf {H}}$ are different throughout the image due to the dependence on elevation angles $\varphi$ and $\varphi ^{\prime }$. The sonar images from two different viewpoints show a complex nonlinear transformation relationship [22].