Multi-attributed Graph Matching with Multi-layer Random Walks

Abstract

This paper addresses the multi-attributed graph matching problem considering multiple attributes jointly while preserving the characteristics of each attribute. Since most of conventional graph matching algorithms integrate multiple attributes to construct a single attribute in an oversimplified way, the information from multiple attributes are not often fully exploited. In order to solve this problem, we propose a novel multi-layer graph structure that can preserve the particularities of each attribute in separated layers. Then, we also propose a multi-attributed graph matching algorithm based on the random walk centrality for the proposed multi-layer graph structure. We compare the proposed algorithm with other state-of-the-art graph matching algorithms based on the single-layer structure using synthetic and real datasets, and prove the superior performance of the proposed multi-layer graph structure and matching algorithm.

1 Introduction

Graph structure has been widely adopted to model various problems in computer vision such as categorization [1, 2], detection and tracking [3–5], and shape matching [6–8], because a graph can effectively reduce the ambiguity caused by various deformations and outliers thanks to the structured information. A graph structure consists of a set of vertices, edges, and attributes. Here, attributes are associated values with vertices or edges belonging to the graph. In a feature point matching application, for example, feature points can be defined as vertices, and the pairwise relations between feature points can be established as edges. Then, vertex attributes describe the appearances of feature points such as color and gradient information [9, 10], and edge attributes describe geometric relations between pairs of feature points such as angle and distance [1, 8–12]. Since an attribute assigns distinctiveness to each of vertices or edges, the definition of attributes enormously affects matching performance.

Multiple attributes are often employed simultaneously because a single attribute is not capable of representing the complex properties of image contents in general. Most of graph matching algorithms [1, 4, 6, 8–10, 12] integrate multiple attributes to construct a single attribute, and there are two common ways of attribute integration: attribute-level and affinity-level integration. The attribute-level integration combines multiple attributes at an attribute description step in order to establish a new attribute containing the whole information of multiple attributes. After that, affinity values of matching candidates are computed by using the integrated attributes. An example for this type of integration is Histogram-Attributed Relational Graph (HARG) proposed by Cho et al. [1]. In HARG representation, each edge is defined with the log-polar histogram that is constructed by concatenating a log-distance histogram and a polar-angle histogram. On the other hand, the affinity-level integration is performed at an affinity computation step. Firstly, each of affinity values is computed for each attribute, then multiple affinity values are integrated to construct a unified affinity value. For example, Zhou et al. [10] and Yan et al. [9] define two types of affinity based on the differences in distance and angle between vertices, and then linearly integrate the values to construct affinity between matching candidates. However, the problem is that, although integrations are performed at various stages, information from multiple attributes are not fully exploited in most cases; because the integration process is likely to oversimplify the characteristics of attributes.

Fig. 1.

Two types of formulation for graph matching problems with multiple attributes. (a) The multiple attributes are integrated, and then an affinity matrix is constructed. (b) The multiple attributes are separately represented, and affinity matrices are constructed for multiple layers that are linked to one another (rank-4 tensor form).

In this paper, we propose a multi-layer graph matching algorithm that considers multiple attributes jointly while preserving the characteristics of each attribute. The main contribution of this paper is twofold. First, we propose a multi-layer structure to represent the multiple attributes as described in Fig. 1. The proposed structure consists of multiple layers in which each layer represents a single attribute and multiple layers are closely linked to one another. While the single-layer structure can lose important information during the attribute integration process, the proposed structure can preserve the characteristics of each attribute in multiple separated layers. Second, we propose a novel multi-attributed graph matching algorithm based on the random walk centrality concept for the proposed multi-layer graph structure. To obtain centrality values of matching candidates, the random walkers traverse the multi-layer association graph according to the transition probability that is derived from pairwise attributes.

The rest of this paper is organized as follows. First, Sect. 2 gives an overview of related works. Section 3 provides explanations on the problem formulation and the proposed multi-layer structure. Then, we explain details of the proposed matching algorithm in Sect. 4. Finally, experimental results are presented in Sect. 5.

2 Related Works

The multi-layer network structure has been studied in the field of network theory for several decades [13] to represent complicated relations in the natural environment. The network consists of multiple layers that describe various aspects of relations and are closely connected to each other. In order to analyze multi-layer networks, several diagnostics have been proposed in previous researches [13], such as vertex degree and path. Especially, centrality is one of the most frequently used concepts, and is employed to identify the importance of each vertex (e.g. degree or sum of distances from other vertices). In order to compute the centrality for each vertex, random walk-based methods have attracted appreciable interest as in [13–16], because it is intuitive and easy to interpret various centrality concepts. For example, PageRank [15] employs the random walk centrality to measure the ranks of webpages. Each random walker moves along contained hyperlinks, then the stationary distribution of the walkers on the webpage network is defined as their ranks. Moreover, Domenico et al. [17] and Solé-Ribalta et al. [16] proposed its generalized versions for an undirected multiplex network structure. In these algorithms, random walkers can move from any vertex to any other vertices in different layers with given transition probabilities.

Interestingly, Cho et al. [11] provided a connection between the concept of random walk centrality and graph matching problems, which is called Reweighted Random Walk Matching (RRWM). In order to formulate the graph matching problem, they employed an association graph structure. As described in Fig. 1(a), each of possible matching candidates is defined as the vertex of the association graph, and the pairwise relation between two matches is described as an edge. Then, similarly to the random walk centrality computation process, random walkers traverse the association graph along the edges. Finally, the correct correspondences can be obtained from the stationary distribution of the walkers. They presented that the random walk view of a graph matching problem has the same form with a classical spectral graph matching formulation [6, 8, 12].

Our work is inspired by the random walk centrality measure for the multi-layer structure and RRWM for graph matching. Based on these, we propose a multi-layer association graph structure for multiple attributes, and then identify trustful correspondences based on the random walk centrality concept.

3 Multi-layer Graph Structure

In this section, we propose a multi-layer graph structure to consider multiple attributes while preserving their characteristics independently. First, we introduce single-layer graph matching formulations for conventional graph structures, and then generalize the formulation for the multi-layer graph structure.

3.1 Single-Layer Graph Matching Problem

In single-layer graph matching problems, an attributed graph $\mathcal {G}$ is defined as $\mathcal {G}\left( \mathcal {V},\mathcal {E},\mathcal {A}\right)$, where $\mathcal {V}$ and $\mathcal {E}$ represent vertices and edges respectively, and $\mathcal {A}$ is a set of single-type attributes that describe vertices and edges. Most applications need multiple attributes to describe complex properties. Nonetheless, the single-layer structure can handle the cases, because multiple attributes can be regarded as a single-type attribute by various integration methods [1, 4, 6, 8–10, 12] as described in Fig. 1(a).

In order to define a correspondence problem for given two graphs $\mathcal {G}^{P}$ and $\mathcal {G}^{Q}$, Lawler’s quadratic assignment problem (QAP) formulation [8, 10–12] is widely employed, which consists of an assignment matrix $\mathbf {X}$ and an affinity matrix $\mathbf {W}$. $\mathbf {X}$ is an ${N}^{P}\times {N}^{Q}$ matrix, where ${N}^{P}$ and ${N}^{Q}$ are numbers of vertices in $\mathcal {G}^{P}$ and $\mathcal {G}^{Q}$ respectively, and represents possible correspondences. Each element of the matrix, $\mathbf {X}_{i;a}$, is a binary value indicating the correspondence relation between ${v}_{i}^{P}\in \mathcal {V}^{P}$ and ${v}_{a}^{Q}\in \mathcal {V}^{Q}$. For example, $\mathbf {X}_{i;a}=1$ when ${v}_{i}^{P}$ is matched with ${v}_{a}^{Q}$, and $\mathbf {X}_{i;a}=0$ otherwise. On the other hand, $\mathbf {W}$ is an ${N}^{P}{N}^{Q}\times {N}^{P}{N}^{Q}$ matrix that describes structural information of matching candidates. A non-diagonal elements of the matrix, $\mathbf {W}_{ia;jb}$, indicates a pairwise affinity between two matching candidates $({v}_{i}^{P},{v}_{a}^{Q})$ and $({v}_{j}^{P},{v}_{b}^{Q})$, and each diagonal element, $\mathbf {W}_{ia;ia}$, indicates a unary affinity of a matching candidate $({v}_{i}^{P},{v}_{a}^{Q})$. Then, a graph matching problem can be formulated as a QAP as follows:

$$\begin{aligned} \begin{array}{c} \hat{ \mathbf {x} } = \underset{ \mathbf {x} }{arg\max } {\left( { \mathbf {x} }^{ \top } \mathbf {W} \mathbf {x} \right) }, \\ s.t.\quad \mathbf {x} \in { \left\{ 0,1 \right\} }^{ { N }^{ P }{ N }^{ Q } }, \quad \forall i\sum \limits _{ a=1 }^{ { N }^{ Q } }{ { \mathbf {x} }_{ ia } } \le 1, \quad \forall a\sum \limits _{ i=1 }^{ { N }^{ P } }{ { \mathbf {x} }_{ ia } } \le 1, \end{array} \end{aligned}$$

(1)

where $\mathbf {x}$ is an ${N}^{P}{N}^{Q}$ dimensional vector that is originated from $\mathbf {X}$ by columnwise vectorization, and the inequality constraints represent the one-to-one matching constraint between $\mathcal {G}^{P}$ and $\mathcal {G}^{Q}$. Since the QAP is an NP-hard problem, efficient algorithms which guarantee an optimal solution are unknown. For this reason, recent researches [8, 10–12, 18] have aimed at finding an approximated solution by relaxing the constraints in various ways.

As mentioned earlier, Cho et al. [11] presented the viewpoint which interprets the single-layer graph matching problem as a random walk centrality computation problem by constructing a single-layer association graph $\mathcal {G}^{S}(\mathcal {V}^{S},\mathcal {E}^{S},\mathcal {A}^{S})$. In this viewpoint, each matching candidate $({v}_{i}^{P},{v}_{a}^{Q})$ is considered as a node ${v}^{S}_{ia}\in \mathcal {V}^{S}$, and a relation between two vertices ${v}_{ia}^{S}$ and ${v}_{jb}^{S}$ is considered as an edge ${e}^{S}_{ia;jb}\in \mathcal {E}^{S}$. Their attributes ${a}^{S}_{ia;jb}\in \mathcal {A}^{S}$ are defined from the affinity values $\mathbf {W}_{ia;jb}$. Because these attributes do not directly reflect the node-to-node transition probability, the probabilistic transformation process for the association graph is required to describe the traversal of random walkers. For this purpose, Cho et al. [11] proposed an affinity preserving normalization method that adopts a concept of absorbing Markov chain. First, the maximum degree ${d}_{max}$ is computed as Eq. (2) to preserve the original relations among affinity values while constructing the probabilistic network. Then, $\mathbf {W}$ is normalized by using ${d}_{max}$ as following:

$$\begin{aligned} \begin{array}{c} \widetilde{\mathbf {W}}=\frac{1}{{d}_{max}} \mathbf {W}, \quad s.t.\quad {d}_{max}=\mathop {\max }\nolimits _{i}\mathbf {d}_{i}=\mathop {\max }\nolimits _{i}\mathop {\sum }\nolimits _{k}\mathbf {W}_{ik}. \end{array} \end{aligned}$$

(2)

In order to describe the centrality value of each vertex, a distribution vector of random walkers $\mathbf {x}$ is defined as an ${N}^{P}{N}^{Q}$ dimensional vector. Then, all of the random walkers simultaneously and iteratively traverse the network based on the transition probability defined in $\widetilde{\mathbf {W}}$ until reaching to the stationary distribution $\bar{\mathbf {x}}$. Finally, the optimal solution $\hat{\mathbf {x}}$ can be obtained from $\bar{\mathbf {x}}$ by adopting any discretization methods such as Hungarian algorithm [19].

3.2 Multi-layer Graph Matching Problem

In this section, we propose a multi-layer association graph structure to jointly describe multiple attributes while preserving characteristics of each attribute. The proposed multi-layer association graph $\mathcal {G}^{M}(\mathcal {V}^{M},\mathcal {E}^{M},\mathcal {L}^{M},\mathcal {A}^{M})$ is inspired from the multiplex network structure [13, 16, 17] that is frequently adopted to describe the complex networks which have multiple types of connections, such as social network. The multiplex network structure consists of multiple layers that share an index set of nodes (e.g. the IDs of people in a social network), and the relations between the nodes are independently defined for each layer. Each node can be connected to other nodes through two types of links: intra-layer and inter-layer links. The intra-layer links indicate the connection between two nodes in the same layer, and the inter-layer links indicate the connection between two nodes in different layers.

Similarly, the proposed association graph $\mathcal {G}^{M}$ consists of multiple layers that share the same index set of matching candidates as described in Fig. 1(b). Each vertex ${v}_{ia}^{\alpha }\in \mathcal {V}^{M}$ is distinguished from other vertices according to the layer index (e.g. ${v}_{ia}^{\alpha } \ne {v}_{ia}^{\beta }$, where $\alpha ,\beta \in \mathcal {L}^{M}$ are layer indices when $\mathcal {L}^{M}$ denotes a set of layer indices). Each edge ${e}_{ia;jb}^{\alpha ;\beta }\in \mathcal {E}^{M}$ should be represented with the layer indices because the vertices are identified by not only the vertex indices but also the layer indices. Similarly to the multiplex network structure, the intra-layer edges ${e}_{ia;jb}^{\alpha ;\alpha }$ are defined for the connections between the vertices in the same layer, and the inter-layer edges ${e}_{ia;ia}^{\alpha ;\beta }$ are defined for the connections between the same-indexed vertices in different layers.

According to the definition of multi-layer association graph structure, the affinity information can be represented as a rank-4 tensor $\mathbf {\Pi }$. Each element of the tensor, $\mathbf {\Pi }^{\alpha ;\beta }_{ia;jb}$, describes an affinity value between two matching candidates ${v}^{\alpha }_{ia}$ to ${v}^{\beta }_{jb}$. Similarly, the assignment information can be also represented as a rank-2 tensor $\mathbf {T}$. Because the high-rank tensor form is hard to understand intuitively, an unfolding method that converts the high-rank tensor to a lower rank matrix is frequently adopted [16, 20–22]. For example, an ${N}^{P}{N}^{Q}\times L$ dimensional rank-2 assignment tensor (matrix) $\mathbf {T}$ can be unfolded to an ${N}^{P}{N}^{Q}L\times 1$ dimensional rank-1 assignment tensor (vector) $\mathbf {t}$, where L is the number of layers. In the same manner, $\mathbf {\Pi }$ can be also unfolded to an ${N}^{P}{N}^{Q}L\times {N}^{P}{N}^{Q}L$ dimensional rank-2 tensor $\mathbf {P}$. This flattened matrix $\mathbf {P}$ is called supra-adjacency matrix [16, 20, 21], which consists of two types of block adjacency matrices as illustrated in Fig. 2. The detailed information about the block matrix construction is presented in Sect. 3.3. Finally, the multi-attributed graph matching problem can be formulated as

$$\begin{aligned}&\hat{ \mathbf {t} } = \underset{ \mathbf {t} }{arg\max } {\left( { \mathbf {t} }^{ \top } \mathbf {P} \mathbf {t} \right) }, \nonumber \\ s.t.&{\left\{ \begin{array}{ll} \quad \mathbf {t} \in { \left\{ 0,1 \right\} }^{ { N }^{ P }{ N }^{ Q }L}, \\ \forall i \sum \limits _{\alpha =1} ^{L} \sum \limits _{ a=1 }^{ { N }^{ Q } }{ { \mathbf {t} }^{\alpha }_{ ia } } \le 1, \quad \forall a \sum \limits _{\alpha =1} ^{L}\sum \limits _{ i=1 }^{ { N }^{ P } }{ { \mathbf {t} }^{\alpha }_{ ia } } \le 1, \end{array}\right. } \end{aligned}$$

(3)

where the one-to-one constraints are adopted to the layer-wise integrated solution because matching candidates (vertices) $\mathcal {V}^{M}$ are shared across all layers. In Sect. 4, we propose a novel graph matching algorithm to solve this problem.

Fig. 2.

Supra-adjacency matrix $\mathbf {P}$ constructed from the multi-layer association graph $\mathcal {G}^{M}$ and the adjacency tensor $\mathbf {\Pi }$

3.3 Supra-Transition Matrix Construction

The supra-adjacency matrix consists of two types of block matrices: intra-layer and inter-layer adjacency matrix. The intra-layer adjacency matrix $\mathbf {P}^{\alpha ;\alpha }$ describes the edges between vertices in the same layer, and the inter-layer adjacency matrix $\mathbf {P}^{\alpha ;\beta }$ describes the edges between vertices in different layers. Each of block matrices is constructed according to the definitions of proposed multi-layer graph structure. Then, the supra-adjacency matrix $\mathbf {P}$ can be expressed by arranging the block matrices according to their layer index as following:

$$\begin{aligned} \mathbf {P}= \begin{pmatrix} \mathbf {P}^{\alpha ;\alpha } &{} \mathbf {P}^{\beta ;\alpha } &{} \cdots &{} \mathbf {P}^{\omega ;\alpha } \\ \mathbf {P}^{\alpha ;\beta } &{} \mathbf {P}^{\beta ;\beta } &{} \cdots &{} \mathbf {P}^{\omega ;\beta } \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \mathbf {P}^{\alpha ;\omega } &{} \mathbf {P}^{\beta ;\omega } &{} \cdots &{} \mathbf {P}^{\omega ;\omega } \end{pmatrix}. \end{aligned}$$

(4)

In order to apply the random walk centrality concept, $\mathbf {P}$ should be converted into a transition matrix $\widetilde{\mathbf {P}}$ that describes the transition probabilities of all vertices. Since $\mathbf {P}$ consists of multiple types of layers that contain distinctive characteristics (e.g. different scales), the stochastic normalization of whole matrix is a difficult process. Fortunately, however, thanks to the block structure of the supra-adjacency matrix, the separate normalization of each block matrix can be applied. Therefore, we first adopt block-wise normalization for the affinity scaling to get $\mathbf {P}^{'}$, and then construct the supra-transition matrix $\widetilde{\mathbf {P}}$ by applying the affinity preserving normalization [11] again as follows:

$$\begin{aligned} \begin{array}{c} \widetilde{\mathbf {P}}=\frac{1}{{d}_{max}} \mathbf {P}^{'} =\frac{1}{{d}_{max}} \begin{pmatrix} \widetilde{\mathbf {P}}^{\alpha ;\alpha } &{} \widetilde{\mathbf {P}}^{\beta ;\alpha } &{} \cdots &{} \widetilde{\mathbf {P}}^{\omega ;\alpha } \\ \widetilde{\mathbf {P}}^{\alpha ;\beta } &{} \widetilde{\mathbf {P}}^{\beta ;\beta } &{} \cdots &{} \widetilde{\mathbf {P}}^{\omega ;\beta } \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \widetilde{\mathbf {P}}^{\alpha ;\omega } &{} \widetilde{\mathbf {P}}^{\beta ;\omega } &{} \cdots &{} \widetilde{\mathbf {P}}^{\omega ;\omega } \end{pmatrix}, \\ s.t.\quad {d}_{max}=\mathop {\max }\nolimits _{i}\mathbf {d}_{i}=\mathop {\max }\nolimits _{i}\mathop {\sum }\nolimits _{k}\mathbf {P}^{'}_{ik}, \end{array} \end{aligned}$$

(5)

where $\widetilde{\mathbf {P}}^{\alpha ;\alpha }$ and $\widetilde{\mathbf {P}}^{\alpha ;\beta }$ are intra-layer and inter-layer transition matrices. The details about the stochastic normalization methods for each type of block matrices are presented in following subsections.

Intra-layer Transition Matrix Construction. Each intra-layer adjacency matrix $\mathbf {P}^{\alpha ;\alpha }$ is constructed based on the definition of an affinity value for each attribute. Since the intra-layer connections are only defined for the vertices of the same layer, the intra-layer transition matrix $\widetilde{\mathbf {P}}^{\alpha ;\alpha }$ can be constructed by using conventional affinity preserving normalization methods [11] as follows:

$$\begin{aligned} \widetilde{\mathbf {P}}^{\alpha ;\alpha } = \frac{1}{{d}^{\alpha }_{max}} \mathbf {P}^{\alpha ;\alpha },\quad s.t.\quad {d}^{\alpha }_{max}=\max _{i}\mathbf {d}^{\alpha }_{i}=\max _{i}\sum _{k}\mathbf {P}^{\alpha ;\alpha }_{ik}. \end{aligned}$$

(6)

Inter-layer Transition Matrix Construction. The affinity values of inter-layer connections are uniformly set to 1 because the relative importance of the edges is unknown. Therefore, the inter-layer adjacency matrix $\mathbf {P}^{\alpha ;\beta }$ can be initially defined as an ${N}^{P}{N}^{Q}\times {N}^{P}{N}^{Q}$ all-ones matrix. Then, the row-wise probabilistic normalization is applied to construct the inter-layer transition matrix $\widetilde{\mathbf {P}}^{\alpha ;\beta }$. However, since the proposed multi-layer graph structure does not consider the inter-layer connections between vertices that have different indices, non-diagonal elements should be set to zero in the normalization process. Moreover, each layer should have the same transition probability because the relative importance of the layers is also unknown without any prior knowledge. Finally, the inter-layer transition matrix $\widetilde{\mathbf {P}}^{\alpha ;\beta }$ can be defined as follows:

$$\begin{aligned} \widetilde{\mathbf {P}}^{\alpha ;\beta } = \frac{1}{{N}^{P}{N}^{Q}L} \mathbf {P}^{\alpha ;\beta }=\frac{1}{{N}^{P}{N}^{Q}L} \mathbf {I}_{{N}^{P}{N}^{Q}}, \end{aligned}$$

(7)

where $\mathbf {I}_{{N}^{P}{N}^{Q}}$ denotes an ${N}^{P}{N}^{Q}\times {N}^{P}{N}^{Q}$ identity matrix. Unfortunately, the above uniform transition probability distribution can decrease the distinctiveness among matching candidates in practical environments. To handle this problem, we employ an assumption that strongly connected vertices with others have more valuable and reliable information to propagate than other vertices. The assumption is reasonable because true correspondences usually organize a strongly connected cluster in practical graph matching applications. Moreover, this assumption is one of the theoretical bases of the spectral matching scheme [8], which is frequently adopted in recent graph matching researches. Based on the assumption, we can design a weight vector using the degrees of intra-layer connections for computing relative importance among vertices. Then, the reinforced inter-layer transition matrix can be defined as follows:

$$\begin{aligned} \begin{array}{c} \widetilde{\mathbf {P}}^{\alpha ;\beta } = \text {diag}(\frac{\mathbf {d}^{\alpha }}{{d}^{\alpha }_{max}}) \circ \left( \frac{1}{{N}^{P}{N}^{Q}L} \mathbf {P}^{\alpha ;\beta }\right) , \end{array} \end{aligned}$$

(8)

where $\text {diag}\left( \mathbf {d}^{\alpha }/{d}^{\alpha }_{max}\right)$ is a diagonal matrix that describes the normalized degrees of the vertices, and the operator $\circ$ indicates the Hadamard product.

3.4 Advantages Against the Single-Layer Structure

The main difference between the proposed multi-layer structure $\widetilde{\mathbf {P}}$ and the integrated single-layer structure $\widetilde{\mathbf {W}}$ is in how the relations between multiple attributes are described. While $\widetilde{\mathbf {W}}$ describes the relations between multiple attributes at the affinity definition level, $\widetilde{\mathbf {P}}$ describes it in the multi-layer structure. By embedding the relations in the multi-layer structure, the proposed structure achieves two advantages against the single-layer structure. First, $\widetilde{\mathbf {P}}$ provides a probabilistic viewpoint of multi-attributed graph matching problems. Generalizing the single-attributed (or integrated-attributed) graph matching problem to the multi-attributed problem is difficult because of the complex relations among multiple attributes. In this sense, the probabilistic viewpoint of the multi-layer network can provide an efficient way to solve the complicated problem. Second, $\widetilde{\mathbf {P}}$ can adaptively describe various combinations of multiple attributes. Of course, $\widetilde{\mathbf {W}}$ may be appropriately designed to deal with a specific type of the problem by using various methods such as machine learning. However, the specially designed $\widetilde{\mathbf {W}}$ does not guarantee satisfactory performance for other types of the problem. On the other hand, $\widetilde{\mathbf {P}}$ can adaptively describe the relation between attribute pairs using the transition probability that is iteratively adjusted during the proposed matching process. In Sect. 4, we propose a multi-layer graph matching algorithm that takes these advantages into account based on the proposed structure.

4 Multi-layer Random Walk Graph Matching Algorithm

The proposed framework for computing random walk centrality of vertices is rather similar to the general random walk framework. In the framework, a random walker starts a traversal from any arbitrary node and then randomly moves to the new node according to the transition probability of given network. Suppose the current distribution of random walkers is given as $\mathbf {t}_{k}$, then the next distribution $\mathbf {t}_{k+1}$ can be obtained by multiplying the transition matrix $\widetilde{\mathbf {P}}$ as follows:

$$\begin{aligned} \mathbf {t}_{k+1}^{\top } = \mathbf {t}_{k}^{\top } \widetilde{\mathbf {P}}. \end{aligned}$$

(9)

When the distribution $\mathbf {t}_{k+1}$ is equal to $\mathbf {t}_{k}$ after the iterative traversal, $\mathbf {t}_{k}$ is called stationary distribution $\bar{\mathbf {t}}$. Because $\bar{\mathbf {t}}$ should not be changed by the transition, $\bar{\mathbf {t}}$ has to satisfy the following equation,

$$\begin{aligned} \lambda \bar{\mathbf {t}}^{\top } = \bar{\mathbf {t}}^{\top } \widetilde{\mathbf {P}}. \end{aligned}$$

(10)

Since $\bar{\mathbf {t}}$ is a non-negative vector and $\widetilde{\mathbf {P}}$ is irreducible, $\lambda$ is the maximal eigenvalue of $\widetilde{\mathbf {P}}$ and $\bar{\mathbf {t}}$ is a normalized eigenvector of $\widetilde{\mathbf {P}}$ corresponding to $\lambda$ according to the Perron-Frobenius theorem. Finally, the normalized eigenvector (the stationary distribution $\bar{\mathbf {t}}$) can be obtained by iteratively updating the vector based on the power iteration method [8, 11] as follows:

$$\begin{aligned} \mathbf {t}_{k+1}^{\top } = \frac{1}{{\left\| \mathbf {t}_{k}^{\top } \widetilde{\mathbf {P}} \right\| }_{2}} \mathbf {t}_{k}^{\top } \widetilde{\mathbf {P}}. \end{aligned}$$

(11)

By generalizing this basic random walk framework, we propose the novel multi-layer graph matching algorithm as described in Algorithm 1. Initially, a uniform distribution vector $\mathbf {t}_{0}$ and a supra-transition matrix $\widetilde{\mathbf {P}}$ are generated. Then, for each iteration step, a next distribution vector $\bar{\mathbf {t}}$ is computed by multiplying the previous distribution $\mathbf {t}$ and the transition matrix $\widetilde{\mathbf {P}}$. The update is repeated until the distribution vector $\bar{\mathbf {t}}$ reaches to the stationary state.

To encourage the matching constraints in Eq. (3.2), the proposed algorithm employed a reweighting process in the middle of iteration steps similarly to the single-layer random walk graph matching algorithm [11]. Random walkers traverse according to the transition probability during the iteration steps, however, the traversal does not consider about the matching constraints. By applying the reweighting process [11] controlled by the reweighting factor $\theta$, random walkers can jump to the constrained nodes regardless of the transition probability. The reweighting process consists of two steps: inflation and bistochastic normalization. The purpose of our inflation procedure is to filter out unreliable matching candidates, which is indicated in Line 9 of Algorithm 1. During the inflation step, the large assignment values are amplified, and the small assignment values are attenuated. Line 10 of Algorithm 1 indicates a bistochastic normalization step using the Sinkhorn method [23], which encourages the one-to-one matching constraint by transforming the assignment matrix to the bistochastic matrix. During this normalization step, the assignment distribution vector is transformed to an assignment matrix, and then the rows and columns of the matrix are alternatively normalized until converges. As a result, the contradictions among the matching candidates in the assignment distribution could be removed. These steps are separately applied for each layer to preserve the distinctiveness of multiple layers.

After the reweighting process, the reweighted distribution vectors $\mathbf {u}$ should be integrated to prevent a contradiction among the layers. One of the possible methods is gathering the distribution vectors according to any predefined priority of layers at once, and then propagating again to each layer. However, it is difficult to decide the most important layer without any prior knowledge, because the attributes have various characteristics hard to compare with each other. For that reason, we empirically define a layer importance measure as the intersection of reweighted distribution and current assignment distribution. The proposed measure is based on the observation that the large difference between the current assignment distribution and the reweighted distribution means that the reweighting process causes large information loss. Our extensive experiments with this measure show the meaningful performance enhancement. After calculating each layer importance value, the importance vector $\mathbf {s}$ is normalized into the interval $\left[ \tau ,1\right]$, where $\tau$ is a minimum importance value. At the end of iteration steps, the reweighted distribution is integrated with the current assignment distribution to generate the biased assignment distribution (reweighting jump in [11]). Finally, the converged distribution vector $\mathbf t$ is discretized to obtain a binary solution $\hat{\mathbf{t }}$ by using Greedy mapping [8] or Hungarian method [19].

5 Experimental Results

In order to evaluate the proposed algorithm, we perform experiments with the synthetic dataset and the WILLOW object class dataset¹ [1]. We compare our algorithm with the widely used graph matching algorithms including Reweighted Random Walk Matching (RRWM) [11], Factorized Graph Matching (FGM) [10], Spectral Matching (SM) [8], Max Pooling Matching (MPM) [18], Graduated Assignment Graph Matching (GAGM) [24], and Integer Projected Fixed Points matching (IPFP) [25]. In all experiments, we fix the reweighting factor $\theta$ to 0.2, the inflation factor $\rho$ to 30, and the minimum layer importance value $\tau$ to 0.1. Moreover, the parameters of other algorithms are also fixed to those given in the original papers. Our evaluation framework is based on the open MATLAB programs of [11], and the source codes of other matching methods are obtained from the authors.

5.1 Performance Evaluation for Synthetic Dataset

In this experiment, we generate a pair of synthetic graphs which contain outliers and deformations in similar manners represented in [11, 12]. We first define an initial graph $^{0}\mathcal {G}$ which has several types of attributes $^{0}\mathcal {A}$. To reflect the characteristics according to attribute types, we randomly assign affinity values with different variances for each layer. Then we generate two graphs $^{1}\mathcal {G}$ and $^{2}\mathcal {G}$ with small attribute deformation based on a Gaussian function $\mathcal {N}(0,{\epsilon }^{2})$ and randomly defined outliers ${v}_{out}$. The affinity matrix $\mathbf {W}^{\alpha }$ of layer $\alpha$ is defined by $\exp (-{|^{1}{a}^{\alpha }_{ij}-^{2}{a}^{\alpha }_{ab}|}_{2}/{\sigma }^{2})$, where $^{1}{a}^{\alpha }_{ij}$ and $^{2}{a}^{\alpha }_{ab}$ are edge attributes that are normalized into the interval [0, 1], and ${\sigma }^{2}$ is a scaling factor which is set to 0.3. For the attribute integration of single-layer graph matching methods, we aggregate the normalized affinity matrices from multiple layers by summing them.

Table 1. Parameter setting for the synthetic graph matching experiments

Fig. 3.

Synthetic graph matching results. (Color figure online)

Then, we design three experiments by changing parameters: the magnitude of deformation $\epsilon$, the number of outliers ${n}_{out}$, and the number of attributes ${n}_{att}$. First, in the deformation experiment, the magnitude of deformation $\epsilon$ is varied and other variables are fixed. Similarly, in the outlier experiment, only the number of outliers ${n}_{out}$ is changed. In the last experiment, both parameters, $\epsilon$ and ${n}_{out}$, are fixed and the number of attributes ${n}_{att}$ is varied. For all experiments, the number of inliers ${n}_{in}$ is fixed to 10, and each test is iterated 100 times. Details of the parameters are represented in Table 1.

As shown in Fig. 3, the proposed algorithm, Multi-Layer Random Walk Matching (MLRWM-multi), shows comparable performance with other methods under deformation.² However, ‘MPM-integrated’ and ‘FGM-intgrated’ yield better performance than ours in the outlier and attribute experiments. Since MPM selects correspondences that have the maximum affinity values, this method is highly robust against the variation caused by outliers. On the other hand, ‘MPM-integrated’ is relatively weak for the attribute deformation as shown in Fig. 3(a). In addition, when the portion of outliers is small, the proposed method is still better than ‘MPM-integrated’ and the others except ‘FGM-integrated’ as shown in Fig. 3(c). In contrast with MPM, ‘FGM-integrated’ shows superior performance in all of the synthetic graph matching experiments. These results are caused by the experimental design that does not consider the credibility of attributes. In order to reflect practical environments, the credibility of each attribute should be considered because each attribute has different powers of description for various applications. Although we took account of this issue by adopting different variances of affinity values, it is not enough to effectively reflect realistic scenarios. Furthermore, since our algorithm defines the layer importance by observing characteristics of real images as mentioned in Sect. 4, the importance value could be inaccurately estimated in this experiment. For that reason, our method yields the relatively declined performance than ‘FGM-integrated’. However, the proposed method outperforms ‘FGM-integrated’ in a realistic scenario; because real data generally contains both unreliable and reliable attributes whereas synthetic images contain either reliable attributes or unreliable attributes. Details are represented in Sect. 5.2.

5.2 Performance Evaluation for WILLOW Dataset

In this experiment, we evaluate the proposed algorithm with graphs which are constructed from general images in the WILLOW object class dataset [1]. The dataset consists of five categories (face, motorbike, car, duck, winebottle), and all images are manually annotated. In oder to construct a graph, we extract interest points by using the Hessian detector [26]. Among the extracted features, 10 nearest points from the annotated points are selected as inliers. Outliers are randomly selected from the rest of interest points. To define the multi-attribute problem, we use four types of attributes: SIFT descriptor [27], color histogram in RGB color space, relative distance histogram, and relative angle histogram which are the parts of HARG [1]. Edge attributes are defined by concatenating descriptors (or histograms) of two vertices. Exceptionally, the relative angle- and distance- histogram layers define edge attributes by following the definition of HARG for better matching performance. To compute the pairwise affinity value between two attribute vectors, we adopt the normalized Hamming distance measure [1]. Because the attributes have different scales according to the definitions, we normalize the affinity matrices by $(\mathbf {W}/\max (\mathbf {W}))$ before integrating the layers. Finally, the integrated affinity matrices are constructed by summing the normalized affinity matrices.

Table 2. Performance on the WILLOW dataset [1]. Red and blue bold numbers denote the best and the second-best performance in each category.

Fig. 4.

Qualitative comparison of feature graph matching results for all categories in the WILLOW dataset. The blue lines indicate true correspondences, and the red lines indicate false correspondences or outliers. From top to bottom: each of object categories in the WILLOW dataset. From left to right: graph matching algorithms – MLRWM(proposed), RRWM [11], MPM [18], SM [8]. (Color figure online)

Based on the above criteria, we generate 100 pairs of graphs for each category and evaluate the performance of matching algorithms. As shown in Table 2 and Fig. 4, our algorithm shows robust performance in all categories except the face category. Especially, the proposed method outperforms FGM in most of the categories unlike the synthetic graph matching experiments. This proves the superiority of the proposed multi-layer graph structure and matching algorithm for multiple attributes. Our algorithm presents relatively lower performance for the face category (but still comparable to others) because the failure case of the layer importance estimation is occurred more frequently than other categories. In those cases, the uniform layer importance shows better performance in our additional experiments. Since our current layer importance estimation approach does not use any prior knowledge, we expect that it is possible to improve the estimation accuracy by using the additional information or techniques such as machine learning algorithm. We will focus on this issue in the future to improve the proposed algorithm.

6 Conclusion

In this paper, we have presented a novel multi-attributed graph matching algorithm based on the multi-layer structure. We first designed a multi-layer structure to consider multiple attributes jointly while preserving the characteristics of each attribute in separated layers. Based on the proposed structure, we also proposed the novel multi-attributed graph matching algorithm by adopting the random walk centrality concept. In our extensive experiments, the proposed graph matching algorithm showed the better performance than the state-of-the-art algorithms based on the single-layer graph structure.