Elsevier

Neurocomputing

Volume 398, 20 July 2020, Pages 108-116
Neurocomputing

On a granular functional link network for classification

https://doi.org/10.1016/j.neucom.2020.02.090Get rights and content

Abstract

In this paper, we present a new granular classifier in two versions (iterative and non–iterative), by adopting some ideas originating from a kind of Functional Link Artificial Neural Network and the Functional Network schemes. These two architectures are substantially the same: they both use a function basis instead of the usual activation function, but they are different for the learning algorithm. We augment them from the perspective of Granular Computing and information granules, designing a new kind of classifier and two learning algorithms, by taking into account granularity of information. The proposed classifier exhibits the advantages of the granular architectures, that is higher accuracy and transparency. We formally discuss the convergence of the iterative learning scheme. We carry out some numerical experiments using publicly available data, by comparing the results against those results produced by the state-of-the-art methods. In particular, we achieved sound results by invoking the iterative learning scheme.

Introduction

Originally, Functional Link Artificial Neural Networks (FLANNs) were proposed by Pao et al. [28], [29], as flat networks without hidden layers, with the usual activation function processing a linear combination of input values with a bias and trained using the delta rule. Such kind of networks appeared very promising for function approximation and pattern classification, apparently resulting in a faster convergence rate and lesser computational cost than encountered in multi-layer perceptrons. In [32] a variant of this scheme was discussed, by adopting a functional expansion (e.g. by the polynomial basis) and processing the input directly by means of a function basis. This is different from passing the linear combination of the input values to an activation function, which in [28], [29] and related works represents an enhanced pattern in a flat network. The learning algorithm in [32] was based on the back-propagation. The architecture with a functional expansion inspired several studies (e.g. [10], [41], [44]). Such scheme has not been extensively applied to classification problems. In this context, it has been mainly combined with evolutionary techniques or similar, in order to optimize the topology, such as genetic algorithms [10], particle swarm optimization and harmony search [27]. Some recent works use similar ideas [9], [26]. In particular, in [9], FLANNs are combined with chemical reaction optimization for classification problems dealing with datasets with missing values, inconsistent records, and noisy instances. There is another architecture very similar to FLANN based on functional expansion, that is the Functional Network (FN) [7]: it uses a function basis as in [32], but unlike that, the learning algorithm is a least squares procedure. A recent review on FNs can be found in [45].

We introduce in such architecture a granular layer, built by means of fuzzy granules, giving rise to a Granular Functional Network (GFN) or a Granular Functional Link Artificial Neural Network (GFLANN), depending on the learning algorithm. The kind of adopted granules was recently introduced in [21], for a granular FN with delay. The study of the latter architecture is motivated by the need to limit the number of the adopted information granules when the number of features is high. Hence we develop a delta rule to achieve the needed accuracy. For this new proposed scheme, we formally discuss the convergence. In our algorithm the weights connecting the granular and the internal layer are randomly generated (under a suitable condition), affecting the membership degrees of the input values into the fuzzy sets forming the granules. This strategy allows to avoid a constrained optimization as in [20]. This is different from the original idea of the random vector functional-link (RVFL) net as proposed in [30], [31], where weights and biases affecting the input values are randomly selected [31]. The interested readers can find a comprehensive discussion on RVFL networks in [42].

Proposing a new kind of randomized mechanism against the state of the art is beyond the scope of this paper. The aim here is to exploit granularity to get an accurate interpretable model. Interpretable models are particularly important in the medical field [14]. The interpretability of fuzzy systems, meant as the ability to explain the behavior of the system in an understandable way, is not a new topic. It has attracted many researchers in the last decade (e.g. see [36] and references therein), with the age-old question of the accuracy achieved by Takagi–Sugeno–Kang systems and the interpretability of Mamdami systems [8]. Over the last decade, the formal definition of interpretability has been underway [14], [19]. Only recently, some features of interpretability seem to have been fixed in the literature. Those features are: transparency, intelligibility, simplicity, comprehensibility and meaningfulness [14]. In a transparent model, it is possible to deduce clearly how the model works. This requires a clear mathematical model. In intelligible models, the influence of each model input on the final decision can be deduced (as in additive models, for instance). Regarding simplicity, in the model there should be as less as possible characteristic parameters. There exist very simple models with one or a low number of attributes, even though simplicity does not guarantee comprehensibility [14]. By following the ”comprehensibility postulate”, a model should be interpretable in natural language. This is somewhat related to the information granules and to the fact that fuzzy sets are suitable either for a qualitative representation (in linguistic terms) or a quantitative one [24]. Meaningfulness is mostly referred to the knowledge of the experts, who may choose a qualitative or quantitative representation [24]. It is clear that the proposed model, based on information granules with the fuzzy formalism, allows both a qualitative and a quantitative representation. Besides it addresses also the transparency and the intelligibility issues, being an additive model with a clear mathematical formulation. The ”link” between the classical mathematics, to cover transparency and intelligibility, and the fuzzy mathematics to cover the other above-mentioned aspects with the extraction of IF-THEN rules, is represented by a newly defined information granule [21]. With regard to simplicity, we tried to keep the number of involved parameters as small as possible. This has motivated the iterative computing scheme (GFLANN). As mentioned before, when the number of input features is high, the number of granules needed to get an accurate solution may proportionally grow. In order to achieve a good accuracy, by containing the number of granules, an iterative scheme based on a delta rule has been developed.

On the other hand, there are classifiers which are considered as interpretable to some extent, but they mostly try to find a kind of interpretation of deep learning, leaving some open mathematical issues, e.g. [11], or are based on linearized systems with empirical evaluation (e.g. [3], [18]). Finally, it is the case to mention that some types of Granular Neural Networks were proposed for classification (e.g. [12], [43]), but their convergence was not discussed.

The main contribution of this paper lies in the theoretical framework defining a new granular classifier. Thanks to the adopted formalism, we have been able to introduce a mathematical model to be formally investigated, by proving the convergence. Even though granular computing is meant as a design methodology, the mathematics behind the granular schemes has not found a deeper discussion yet.

Several numerical experiments have been performed, moving from limited datasets, for the sake of comparison with former FLANN-based classifiers, towards high-dimesional datasets, considering state-of-the-art techniques used with those datasets, such as [1], [35], [40]. The emphasis is on the GFLANN classifier, since, as mentioned before, the GFN classifier exhibits some limitations when dealing with a high number of features in datasets. This is reflected in the numerical results, which show the better performance of the GFLANN classifier.

The paper is structured as follows: Section 2 is devoted to some preliminary notation; in Section 3, the new classifier model is introduced; in Section 4, the learning algorithm and its accuracy are formally discussed; Section 5 is referred to numerical experiments and finally in Section 6, some conclusions are offered.

Section snippets

Preliminaries

In this section we recall all the notions related to fuzzy sets, since in this paper we refer to data granulation with fuzzy sets.

Let I=[ξ0,ξm+1] be a closed interval and ξ={ξ0,ξ1,,ξm+1}, with m ≥ 3, be a sequence of points of I, called nodes, such that ξ0<ξ1<<ξm+1. A fuzzy partition of I is defined as a sequence A={A1,A2,,Am} of fuzzy sets Ai: I → [0, 1], with i=1,,m such that

  • Ai(ξ) ≠ 0 if ξ(ξi1,ξi+1) and Ai(ξi)=1;

  • Ai is continuous and has its unique maximum at ξi;

  • i=1mAi(ξ)=1,ξI.

The

The proposed model

Before introducing the model, we recall some basic notions on information granules, FNs and FLANNs.

The learning algorithm

As mentioned before, the proposed granular classifier has two types of weights. The first one, here named the granular weights, denoted as wij, are randomly chosen, as detailed in the next section. The second one, here called output weights, denoted as w¯lm, are unknown. In the GFN scheme, they are learnt through a least squares (LS) approach as explained in the following. The random assignment of some weights combined with the LS approach may be also found in deep architectures (e.g. see [15])

Numerical studies

The numerical experiments are divided into two parts: in the first part, we consider small datasets in order to compare the results against the ones by some recent variants of FLANN (based on certain function bases), such as [10], [27]. In the second part, we consider medium and high-dimensional datasets, referring to the state-of-the-art techniques for the considered datasets such as [1], [35], [40]. Even though our approach cannot be regarded as a variant of the RVFL, we discuss also a

Conclusions

In this paper, we revised FLANNs and FNs (two computational schemes using function bases) from a granular perspective. We defined a new architecture, by introducing a granular layer. We deduced a delta rule based learning algorithm in opposition to a least squares one, leading to an iterative computing scheme in the first case. We formally proved the convergence, under some conditions. The aim here is not to propose a new more accurate classifier, since nowadays, there is a copious literature

CRediT authorship contribution statement

Francesco Colace: Conceptualization, Methodology, Software, Validation, Investigation. Vincenzo Loia: Conceptualization, Methodology. Witold Pedrycz: Conceptualization, Methodology. Stefania Tomasiello: Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Dr. Tomasiello acknowledges support from the European Social Fund via IT Academy programme.

Francesco Colace is Associate Professor in Computer Science in the Department of Industrial Engineering, University of Salerno, Salerno, Italy. His main research directions involve Knowledge Management, Recommender System, Context Aware and Computing, Affective Computing and Sentiment Analysis, e-Learning, ICT for Cultural Heritage. He has published numerous papers in these areas. He is also an author of research monographs and edited volumes covering various aspects of ICT and Cultural

References (46)

Cited by (10)

  • Neighborhood rough set-based three-way clustering considering attribute correlations: An approach to classification of potential gout groups

    2020, Information Sciences
    Citation Excerpt :

    The other is similarity measures between heterogeneous information granules within the framework of granular computing, including distance function [23] membership function [47]. Meanwhile, clustering model is widely used in data dimensionality reduction [1,3,16], event prediction [21], pattern recognition [6]. Although there are many clustering theories and methods to deal with classification with incomplete information, most of the existing suggestions cannot make up all the shortcomings of multi-source heterogeneous information classification problems.

  • A New Class of Efficient Adaptive Filters for Online Nonlinear Modeling

    2023, IEEE Transactions on Systems, Man, and Cybernetics: Systems
View all citing articles on Scopus

Francesco Colace is Associate Professor in Computer Science in the Department of Industrial Engineering, University of Salerno, Salerno, Italy. His main research directions involve Knowledge Management, Recommender System, Context Aware and Computing, Affective Computing and Sentiment Analysis, e-Learning, ICT for Cultural Heritage. He has published numerous papers in these areas. He is also an author of research monographs and edited volumes covering various aspects of ICT and Cultural Heritage, Pervasive Computing and Sentiment Analysis. Professor Colace is managing, as principal investigator, various research projects founded by national and international institutions. He serves as reviewer for many international journal, as IEEE Transactions on Knowledge and Data Engineering, Knowledge-Based Systems, IEEE Transaction on Education and is a member of various editorial boards of international journals.

Vincenzo Loia received B.S. degree in computer science from University of Salerno, Italy in 1985 and the M.S. and Ph.D. degrees in computer science from University of Paris VI, France, in 1987 and 1989, respectively. From 1989 he is Faculty member at the University of Salerno where he teaches Safe Systems, Situational Awareness, IT Project & Service Management. His current position is as President of University of Salerno, Italy. He is the founder and editor-in-chief of Ambient Intelligence and Humanized Computing Springer, and editor-in-chief of Journal of Evolutionary Intelligence, Springer. He is an Associate Editor of various journals IEEE Transactions Journals. His research interests include soft computing, agent technology for technologically complex environments Web intelligence, Situational Awareness, Cognitive Dedense, Artificial Intelligence. He hold in the last years several role in IEEE Society in particular for Computational Intelligence Society (Chair of Emergent Technologies Technical Committee, IEEE CIS European Representative, Vice-Chair of Intelligent Systems Applications Technical Committee).

Witold Pedrycz (IEEE Fellow, 1998) is Professor and Canada Research Chair (CRC) in Computational Intelligence in the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada. He is also with the Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland. In 2009 Dr. Pedrycz was elected a foreign member of the Polish Academy of Sciences. In 2012 he was elected a Fellow of the Royal Society of Canada. In 2007 he received a prestigious Norbert Wiener award from the IEEE Systems, Man, and Cybernetics Society. He is a recipient of the IEEE Canada Computer Engineering Medal, a Cajastur Prize for Soft Computing from the European Centre for Soft Computing, a Killam Prize, a Fuzzy Pioneer Award from the IEEE Computational Intelligence Society, and 2019 Meritorious Service Award from the IEEE Systems Man and Cybernetics Society. His main research directions involve Computational Intelligence, fuzzy modeling and Granular Computing, knowledge discovery and data science, pattern recognition, data science, knowledge-based neural networks, and control engineering. He has published numerous papers in these areas; the current h-index is 111 (Google Scholar) and 82 on the list top-h scientists for computer science and electronics http://www.guide2research.com/scientists/. He is also an author of 21 research monographs and edited volumes covering various aspects of Computational Intelligence, data mining, and Software Engineering. Dr. Pedrycz is vigorously involved in editorial activities. He is an Editor-in-Chief of Information Sciences, Editor-in-Chief of WIREs Data Mining and Knowledge Discovery (Wiley), and Co-editor-in-Chief of Int. J. of Granular Computing (Springer) and J. of Data Information and Management (Springer). He serves on an Advisory Board of IEEE Transactions on Fuzzy Systems and is a member of a number of editorial boards of international journals.

Stefania Tomasiello, Ph.D. in computer science (University of Salerno, Italy) is currently a lecturer of Artificial Intelligence with the Institute of Computer Science, University of Tartu, Estonia, where she is responsible for the course of Fuzzy Logic and Soft Computing. Formerly, permanent researcher with CO.RI.SA. (Research Consortium on Agent Systems), University of Salerno, Italy and Senior Research Fellow with the Dept. of Management and Innovation Systems (DISA-MIS), University of Salerno. She holds the Italian Scientific Qualification to function as associate professor in computer science and numerical analysis. Workpackage leader in several funded projects and expert evaluator (ex-ante and ex-post) of applied research projects for the Italian Ministry of Economic Development and, formerly, the Italian Ministry of University and Research. She has been adjunct professor of Fundamentals of Computer Science, Human-Computer Interaction, Computational Methods and Finite Element Analysis in the University of Basilicata, Italy. TPC member in many international conferences, here included ACM and IEEE sponsored events. Her research interests lie in scientific and soft computing, AI, fuzzy mathematics, nonlinear dynamics. She authored and co-authored numerous papers in the above mentioned areas. She is managing editor of Evolutionary Intelligence (Springer), associate editor of International Journal of Computer Mathematics (Taylor&Francis), and editorial board member of some journals. ACM, ECMI, EUSFLAT, IEEE and INNS member.

View full text