3D model retrieval using constructive-learning for cross-model correlation

Abstract

With the advance of 3D technology and digital image processing technique, there have been a great number of applications of 3D models, such as virtual reality, computed aided design, and entertainment. Under such circumstance, much research attention has been spent on 3D model retrieval in recent decades. Although extensive research efforts have been dedicated to this task, it is a difficult task to explore the correlation among 3D models, which is the key issue in 3D model retrieval. In this paper, we design and implement a constructive-learning for cross-model correlation algorithm for 3D model retrieval. In this method, we first extract view features from multi-views of 3D models. To exploit the cross-model correlation, we formulate the correlation of 3D models in a hypergraph structure, where both the vertex correlation and the edge correlation are simultaneously learned in a constructive-learning process. Then, the correlation of each model to the query can be used for retrieval. To justify the performance of our proposed algorithm, we have implemented the method and tested on two datasets. We have compared it with recent state-of-the-art methods, and the results have shown superior performance of our proposed method.

Introduction

In recent decade, 3D technology has rapid progress and the digital image processing technique has also been developed extensively. With the advance on both hardware and software, a great number of applications have employed 3D models, such as virtual reality, entertainment, computed aided design [1], molecular biology [2], and other applications [3]. 3D movies, tele-medicine and 3D games have become much popular in recent years. All these applications lead to a booming increase of 3D models [4], [5], [6], which make it a urgent requirement to conduct effective 3D model retrieval from large scale dataset [7], [8], [9], [10]. In recent decades, multimedia information retrieval [11], [12], [13], [14], [15], [16] has attracted much attention. In such 3D era, 3D model retrieval [17], [18], [19] becomes even more important, and the importance of retrieving 3D models can be illustrated in the example of industrial design. Previous study shows that only 20% of designs require completely new designs, while other 80% of designs can be combined or revised from existing designs. Therefore, an accurate 3D model retrieval method can significantly improve the industrial design performance and reduce the cost.

In recent decades, much research attention has been spent on 3D model retrieval, and thus it becomes a hot research topic nowadays. The task of retrieving 3D models can be defined as follows: For the query 3D model, the objective of 3D model retrieval is measuring the similarity/distance between each model and the query. Therefore, how to calculate the distance/similarity between two 3D models is the key in the 3D model retrieval task. Regarding this task, existing methods [20], [21] can be mainly classified into two types, model-based 3D model retrieval methods [22], [23], [24] and view-based 3D model retrieval methods [25], [26], [27], [28], [29], [30], based on the different 3D model representation methods.

In model-based 3D model retrieval method, each model is described by a corresponding virtual 3D model, such as point cloud data or mesh data. In this type of methods, the features of 3D models are extracted from the 3D model and the comparison is based on the feature matching. In model-based method, typical representative features include low-level features [23], [31], [32], [22], [33], [34], [24], and high-level features. In model-based 3D model retrieval method, each model is described by a corresponding virtual 3D model, such as point cloud data or mesh data. In this type of methods, the features of 3D models are extracted from the 3D model and the comparison is conducted using feature matching. In model-based method, typical representative features include low-level features [23], [31], [32], [22], [33], [34], [24], and high-level features. Low level features mainly employ the direct description of 3D model information, such as the distribution of surface [23], the geometric moments [31] and volumetric information of 3D model. High level features represent 3D models from a context level, such as skeletons [35]. The advantage of model-based methods comes from the direct representation of 3D model information. While, the main drawback of these methods is the mandatory requirement of 3D models. In many practical applications, the 3D models are not explicit available, which limits the application of model-based methods.

With the development of cameras and image processing methods, it has been much easier to acquire multi-views of 3D models, which leads to the progress of 3D model retrieval methods using multiple views [25], [27], [36], [37]. In these methods, a group of multiple views are used for 3D model representation, captured from different directions by real or virtual cameras. Different from model-based methods, view-based methods do not need the virtual model information, making these methods easier to be applied in various of applications. In view-based 3D model retrieval methods, first a group of views are generated and then the visual features are extracted on these views. The comparison between 3D models is based on the matching between two sets of multi-views. Although there have been much work on view-based 3D model retrieval, it is difficult to explore the correlation among 3D models, which is the key issue in 3D model comparison. Recently, hypergraph-based methods have been introduced into 3D model retrieval, in which the correlation among 3D models is modeled by a hypergraph. Although these methods have shown better performance compared with existing methods, all these methods have just build the initial level model relevance, which is not optimal to reflect the underneath correlation among 3D models.

Under such circumstance, it is important to jointly explore the high-order correlation among 3D models and the relationship among links on the hypergraph, which can bring in deeper investigation on the data modelling. Hypergraph is one type of graph. In hypergraph, each edge is able to link two or more vertices. The flexible structure of hypergraph makes it fit for high order relationship modelling. Regarding the hypergraph based data modelling [38], [39], [40], [41], it has been employed in plenty of computer vision tasks, such as image retrieval [42], [43], [44], model segmentation [45], and hyperspectral image classification. To conduct model recognition, Xia et al. [46] presented a class-specific hypergraph (CSHG) to jointly employ local SIFT features and global geometric constraints. In this work, a selected category of models with multiple appearance instances was modeled by hypergraph. Huang et al. [43] proposed to employ hypergraph structure to formulate the relationship among images, and the transductive learning was conducted to retrieve images. In this method, each vertex denotes one image and the visual feature-based distance is used for edge construction. Zhu et al. [47] presented a multimodal hypergraph learning method for landmark analysis. In this method, the edges were generated based on the visual features of landmark images. It is noted that the initial edges may be not optimal for data representation. We note that the edge weights are just simple set in the learning objective function, indicating that the correlation among edges has not been taken into consideration.

In our task, to handle the issue of high-order correlation among 3D models, we design and implement a constructive-learning for cross-model correlation algorithm for 3D model retrieval. In this method, we first extract view features from multi-views of 3D models. To exploit the cross-model correlation, we formulate the correlation of 3D models in a hypergraph structure. More specifically, the vertex on the hypergraph denotes on 3D model, and the corresponding edges on the hypergraph are built based on the feature-based distance among 3D models. On this hypergraph structure, both the vertex correlation and the edge correlation are simultaneously learned in a constructive-learning process. Then, the correlation of each model to the query can be used for retrieval. To justify the performance of our proposed algorithm, we have conducted experiments on two datasets, including the National Taiwan University dataset and the ETH-80 dataset. We have compared it with recent state-of-the-art methods, and the results have shown superior performance of our proposed method.

The main contributions of our work are two-fold:

• We propose a constructive-learning for cross-model correlation targeting the task of 3D model retrieval. This method is able to take both the model correlation and the correlation of model connections into consideration simultaneously and yet achieves better performance on 3D model comparison.

• To measure the performance of the proposed constructive-learning method, we have conducted experiments on two datasets. The experimental results and comparison with existing methods have shown superior results of proposed method.

The rest of this paper is organized as follows. Section 2 provides related work on 3D model retrieval. Section 3 introduces the proposed method and Section 4 provides detailed experimental results and the comparisons with state-of-the-art methods. We finally conclude this paper in Section 5.