Elsevier

Pattern Recognition

Volume 47, Issue 1, January 2014, Pages 169-177
Pattern Recognition

A new proposal for graph-based image classification using frequent approximate subgraphs

https://doi.org/10.1016/j.patcog.2013.07.004Get rights and content

Highlights

  • We propose a new framework for image classification, which uses frequent approximate subgraph patterns as features.

  • We propose to compute automatically the substitution matrices needed in the process, instead of using expert knowledge.

  • We propose to use a new graph-based image representation.

  • We propose a criterion for selecting isomorphism threshold for the graph mining process.

Abstract

Graph-based data representations are an important research topic due to the suitability of this kind of data structure to model entities and the complex relations among them. In computer vision, graphs have been used to model images in order to add some high level information (relations) to the low-level representation of individual parts. How to deal with these representations for specific tasks is not easy due to the complexity of the data structure itself. In this paper we propose to use a graph mining technique for image classification, introducing approximate patterns discovery in the mining process in order to allow certain distortions in the data being modeled. We are proposing to combine a powerful graph-based image representation adapted to this specific task and frequent approximate subgraph (FAS) mining algorithms in order to classify images. In the case of image representation we are proposing to use more robust descriptors than our previous approach in this topic, and we also suggest a criterion to select the isomorphism threshold for the graph mining step. This proposal is tested in two well-known collections to show the improvement with respect to the previous related works.

Introduction

In many research fields, graphs have been largely used to model data due to their representation expressiveness and their suitability for applications where some kind of entities and their relationships must be encoded within some data structure. Also, a vast graph theory has been developed in order to work with graphs and process the information they represent. In this paper, we intend to explore and combine two research fields where graphs are involved, in order to exploit both their advantages.

The first field is related to Computer Vision. Our intention is to classify images using a graph-based representation. The first step for image classification is to extract low-level features that will encode relevant information for the task, but it has been shown that low-level information by itself cannot provide the high-level perception cues that exist in human minds to describe objects or images in general (this is the well-known semantic gap problem) [16]. Within the range of low-level features developed so far, graphs are one of the representations that can provide some kind of high-level information implicitly, making them a desirable representation choice for researchers to find new solutions. Many works have represented images as graphs (see Section 2) with this purpose, and have developed methods for classification using this type of data structure. One major concern in this area is that although graphs are powerful representational tools, they are hard to work with, leading usually to algorithms with high computational costs, or simplifying the data structure, thus losing some of the embedded information. Having a collection of images represented as graphs rises the question of whether graph mining techniques can be used to discover beneficial information and to perform certain tasks such as image classification.

Now we will approach the other research field that we are aiming to explore: Data Mining. Several authors have developed graph-based techniques and methods for satisfying the need to convert large volumes of data into useful information [4], [24], [45]. Frequent subgraph discovery is an example of such techniques [15]. An important problem in graph mining tasks is classifying information, such as image [2], [13], [25], [35], [36], text [26], and molecular [11], [23] datasets. Although using graph mining for classification purposes has been widely explored, these approaches may not always produce the optimal results in all applications and several authors [22], [27], [7] have expressed the necessity to use approximate graph matching for frequent subgraph mining. These authors defend the idea that, by using approximation, more interesting subgraphs can be found for many applications, for instance, when processing graph databases that have distortions (in terms of different geometric, topological or semantic variations) of similar structures in several objects [2], [23]. Distortion in data is one of the challenges for developing classifiers based on frequent subgraphs in several domains of science [2], [9], [19], [23], [29], [40], [42], [43], [46], [47]. Frequent approximate subgraph (FAS) mining is an important problem in graph mining, where the mined patterns are detected taking into account such semantic distortions. Thus, such approximate solutions achieve classification results which are different from the other graph mining methods.

As mentioned before, in this paper we aim at combining and exploiting both research fields (image classification and frequent approximate subgraph mining) by proposing an image representation that can be used in a classification framework. Although graph mining techniques for image classification have been explored before (see Section 2.3), our main contribution resides precisely in the approximation part of the subject. This work is, in fact, an extension of a previous work [3] where we make use of a powerful graph-based representation adapted to the conditions of the problem. The new contributions and changes with respect to that work are the following:

  • We use a different visual description of the regions in order to add context information to it. We employ a visual descriptor already reported in the literature, making some changes to take advantage of the structure of irregular pyramids.

  • We propose a criterion to obtain the isomorphism threshold needed in the FAS mining step, which is the parameter employed to determine whether two graphs are similar enough to be considered in the frequency count.

We performed new experiments to show how these improvements largely and positively influence the classification results, while comparing it also with other state-of-the-art methods in image classification.

The remainder of the paper is distributed as follows. Section 2 is a summary of related works in the fields of graph-based image classification, graph mining and the combination of both. Section 3 presents some basic notions regarding graph mining techniques and some specific details of the FAS mining process. The graph-based image representation used in our proposal is described in Section 4. Section 5 depicts the classification framework, where both topics discussed in Section 3 and 4 are combined. In order to validate our approach, we present experimental results in Section 6 and conclusions are given finally in Section 7.

Section snippets

Related work

In this section, we start by providing a brief on classification methods using graph-based representations. Next, we present a review of previous works related to approximate graph mining and finally, we present a brief on classification using frequent patterns, which is the subject of this paper.

Background on FAS mining

In this section we intend to provide the reader with the basic tools for understanding the principles of graph mining techniques and, specifically in this context, how approximation methods work in order to take into account possible data distortions.

Graph-based image representation

In the present work we choose to explore the approach proposed in [31], [32] to obtain a graph-based image representation that can serve as input to a graph mining algorithm. In this case we construct an irregular pyramid for each image [5], which provides a hierarchy of partitions at different levels of resolution. Each level is a RAG and the whole pyramid is built from bottom to top, being the base level (level 0) the entire image (i.e. each vertex of the base level represents one pixel in

Classification framework

After presenting the basic concepts and details regarding the graph-based image representation and the FAS mining methods used, we can finally describe the overall classification framework where we combine these tools.

First of all, we obtain the graph-based image representation of a given set of pre-labeled real images, which gives us a graph collection to work with. Next, we label all the vertices and edges, and create the corresponding substitution matrices as presented in Section 4.3. Once

Experimental results

Development in the field of approximate graph mining is still incipient when it comes to applying it to problems with real images. Tests to validate this approach have been performed so far in simple collections [25], [2]. Actually, in the graph mining community, the standards to perform tests are synthetic datasets or molecular datasets [41]. When we try to use these techniques in real images, they have to deal with bigger graphs, with sizes ranging from 200 to 300 edges per graph, and the

Conclusions

In this work we proposed an image representation using FASs as features that can be used in a classification framework. The FASs are obtained by means of FAS miners reported in the literature. The FAS miners are able to identify FAS patterns in graph collections allowing distortions in the data (in terms of edge and vertex label). We propose to automatically compute substitution matrices and the isomorphism threshold for the mining process, based on image features embedded in the framework,

Conflict of interest

None declared.

Niusvel Acosta-Mendoza is graduated of Computer Science Engineering from University of Informatic Sciences (UCI) of Havana, in 2007. Currently, he is working as a “Junior Researcher” at the Data Mining Department at the Advanced Technologies Application Center (CENATAV), Cuba.

References (47)

  • L. Brun et al.

    Contains and inside relationships within combinatorial pyramids

    Pattern Recognition

    (2006)
  • C. Borgelt, M.R. Berthold, Mining molecular fragments: finding relevant substructures of molecules, in: Proceedings of...
  • H. Bunke, K. Riesen, Graph classification based on dissimilarity space embedding, in: Proceedings of the 2008 Joint...
  • C. Chen, X. Yan, F. Zhu, J. Han, gApprox: mining frequent approximate patterns from a massive network, in: IEEE...
  • D. Conte, P. Foggia, C. Sansone, M. Vento, Thirty years of graph matching in pattern recognition, International Journal...
  • M. Deshpande et al.

    Frequent substructure-based approaches for classifying chemical compounds

    IEEE Transactions on Knowledge Data Engineering

    (2005)
  • O. Duchenne, A. Joulin, J. Ponce, A graph-matching kernel for object categorization, in: IEEE International Conference...
  • M. Everingham, L. Van Gool, C.K.L. Williams, J. Winn, A. Zisserman, The PASCAL Visual Object Classes Challenge 2011...
  • A. Gago-Alonso, J.A. Carrasco-Ochoa, J.E. Medina-Pagola, J.F. Martínez-Trinidad, Full duplicate candidate pruning for...
  • R. Glantz, M. Pelillo, W.G. Kropatsch, Feature selection for graph-based image classifiers, in: Proceedings of the...
  • R. Glantz et al.

    Matching segmentation hierarchies

    International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI)

    (2004)
  • L.B. Holder, D.J. Cook, H. Bunke, Fuzzy substructure discovery, in: Proceedings of the Ninth International Workshop on...
  • X. He, M. Ji, H. Bao, Graph embedding with constraints, in: Proceedings of the 21st International Joint Conference on...
  • Cited by (24)

    • Learning to transfer focus of graph neural network for scene graph parsing

      2021, Pattern Recognition
      Citation Excerpt :

      In recent years, this type of graphic inference has attracted increasing attention because it provides a deeper understanding of images [8]. Compared with object detection [9,10], human-object interaction [11,12], and scene classification [13], scene graph parsing presents a unique challenge because it requires inferring the complex dependencies among all these components. The existing methods adopt a two-stage pipeline process [14,15].

    • Robust graph representation of images with underlying structural networks. Application to the classification of vascular networks of mice's colon

      2017, Pattern Recognition Letters
      Citation Excerpt :

      It has been illustrated here with three given choices for the segmentation strategy (single parameters methods with global thresholding, local thresholding [10] and mean shift clustering [17]), a given choice for the skeletonization [11], and two given choices for the graph extraction [12,15]. It would be interesting to investigate variants of this framework with other segmentation strategies including multiple parameters, other skeletonization strategies which bear some properties of robustness [16,23,24] or other graph coding approaches [25] or topological coding of images such as topological maps for instance [26]. Structural networks are present in a large variety of life sciences.

    • Improving graph-based image classification by using emerging patterns as attributes

      2016, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      Later, in Acosta-Mendoza et al. (2012c), a way for automatically computing vertex and edge label substitutions based on image features is proposed. Finally, Acosta-Mendoza et al. (2012c) was extended in Morales-González et al. (2014), where a new criterion for selecting the similarity threshold for the mining step is also suggested. These works are key steps for the image classification approach based on FAS mining, since both vertex and edge label substitutions and the similarity threshold are parameters, which are difficult to be fixed by the user.

    • Finding the best not the most: Regularized loss minimization subgraph selection for graph classification

      2015, Pattern Recognition
      Citation Excerpt :

      Recent years have witnessed an increasing number of applications involving objects with structural relationships, including chemical compounds in Bioinformatics [1], brain networks [2], image structures [3], and academic citation networks [4].

    View all citing articles on Scopus

    Niusvel Acosta-Mendoza is graduated of Computer Science Engineering from University of Informatic Sciences (UCI) of Havana, in 2007. Currently, he is working as a “Junior Researcher” at the Data Mining Department at the Advanced Technologies Application Center (CENATAV), Cuba.

    Annette Morales-González is a PhD student currently at the Advanced Technologies Application Center. She is graduated of Software Engineering from the Polytechnic University “José Antonio Echeverría” (CUJAE) in 2005. Her research interests include image segmentation and classification, automatic image annotation and content-based image retrieval.

    Andrés Gago-Alonso received a BSc in Computer Science from the Havana University in 2004. He holds the MS degree in Mathematics from the same university in 2007. He completed his PhD Degree in Computational Sciences at the National Institute of Astrophysics, Optics and Electronics (INAOE) in January 2010. His research interests include but not restricted to Knowledge Discovery and Data Mining in graph-based content. Currently, he is a Researcher and the Head of the Data Mining's Department in the Advanced Technologies Application Centre (CENATAV), Cuba.

    Edel B. García-Reyes is graduated of Mathematic and Cybernetic from University of Havana, in 1986 and received the Dr. degree in Technical Sciences at the Technical Military Institute “Jose Marti” of Havana, in 1997. At the moment, he is working as a researcher in the Advanced Technologies Application Center. Dr. Edel has focused his researches on digital image processing of remote sensing data, biometrics and video surveillance. He has participated as a member of technical committees and experts groups and has been reviewer for different events and journals such as Pattern Recognition Letters, Journal of Real-Time Image Processing, etc. Dr. Edel worked in the Cuban Institute of Geodesy and Cartography (1986–1995) and in the Enterprise Group GeoCuba (1995–2001) where he directed the Agency of the Centre of Data and Computer Science of Geocuba – Investigation and Consultancy (1998–2001).

    José Eladio Medina-Pagola received his BS in Cybernetic Mathematics from the Havana University in 1977 and his PhD from the Higher Polytechnic Institute “José A. Echeverría” (ISPJAE) in 1996. His research interests include but not restricted to knowledge discovery and data mining, association rules, clustering, computational linguistic, information retrieval and text mining. He is currently a Senior Researcher and Research Deputy Director of the Advanced Technologies Application Centre (CENATAV), Cuba.

    View full text