On a granular functional link network for classification
Introduction
Originally, Functional Link Artificial Neural Networks (FLANNs) were proposed by Pao et al. [28], [29], as flat networks without hidden layers, with the usual activation function processing a linear combination of input values with a bias and trained using the delta rule. Such kind of networks appeared very promising for function approximation and pattern classification, apparently resulting in a faster convergence rate and lesser computational cost than encountered in multi-layer perceptrons. In [32] a variant of this scheme was discussed, by adopting a functional expansion (e.g. by the polynomial basis) and processing the input directly by means of a function basis. This is different from passing the linear combination of the input values to an activation function, which in [28], [29] and related works represents an enhanced pattern in a flat network. The learning algorithm in [32] was based on the back-propagation. The architecture with a functional expansion inspired several studies (e.g. [10], [41], [44]). Such scheme has not been extensively applied to classification problems. In this context, it has been mainly combined with evolutionary techniques or similar, in order to optimize the topology, such as genetic algorithms [10], particle swarm optimization and harmony search [27]. Some recent works use similar ideas [9], [26]. In particular, in [9], FLANNs are combined with chemical reaction optimization for classification problems dealing with datasets with missing values, inconsistent records, and noisy instances. There is another architecture very similar to FLANN based on functional expansion, that is the Functional Network (FN) [7]: it uses a function basis as in [32], but unlike that, the learning algorithm is a least squares procedure. A recent review on FNs can be found in [45].
We introduce in such architecture a granular layer, built by means of fuzzy granules, giving rise to a Granular Functional Network (GFN) or a Granular Functional Link Artificial Neural Network (GFLANN), depending on the learning algorithm. The kind of adopted granules was recently introduced in [21], for a granular FN with delay. The study of the latter architecture is motivated by the need to limit the number of the adopted information granules when the number of features is high. Hence we develop a delta rule to achieve the needed accuracy. For this new proposed scheme, we formally discuss the convergence. In our algorithm the weights connecting the granular and the internal layer are randomly generated (under a suitable condition), affecting the membership degrees of the input values into the fuzzy sets forming the granules. This strategy allows to avoid a constrained optimization as in [20]. This is different from the original idea of the random vector functional-link (RVFL) net as proposed in [30], [31], where weights and biases affecting the input values are randomly selected [31]. The interested readers can find a comprehensive discussion on RVFL networks in [42].
Proposing a new kind of randomized mechanism against the state of the art is beyond the scope of this paper. The aim here is to exploit granularity to get an accurate interpretable model. Interpretable models are particularly important in the medical field [14]. The interpretability of fuzzy systems, meant as the ability to explain the behavior of the system in an understandable way, is not a new topic. It has attracted many researchers in the last decade (e.g. see [36] and references therein), with the age-old question of the accuracy achieved by Takagi–Sugeno–Kang systems and the interpretability of Mamdami systems [8]. Over the last decade, the formal definition of interpretability has been underway [14], [19]. Only recently, some features of interpretability seem to have been fixed in the literature. Those features are: transparency, intelligibility, simplicity, comprehensibility and meaningfulness [14]. In a transparent model, it is possible to deduce clearly how the model works. This requires a clear mathematical model. In intelligible models, the influence of each model input on the final decision can be deduced (as in additive models, for instance). Regarding simplicity, in the model there should be as less as possible characteristic parameters. There exist very simple models with one or a low number of attributes, even though simplicity does not guarantee comprehensibility [14]. By following the ”comprehensibility postulate”, a model should be interpretable in natural language. This is somewhat related to the information granules and to the fact that fuzzy sets are suitable either for a qualitative representation (in linguistic terms) or a quantitative one [24]. Meaningfulness is mostly referred to the knowledge of the experts, who may choose a qualitative or quantitative representation [24]. It is clear that the proposed model, based on information granules with the fuzzy formalism, allows both a qualitative and a quantitative representation. Besides it addresses also the transparency and the intelligibility issues, being an additive model with a clear mathematical formulation. The ”link” between the classical mathematics, to cover transparency and intelligibility, and the fuzzy mathematics to cover the other above-mentioned aspects with the extraction of IF-THEN rules, is represented by a newly defined information granule [21]. With regard to simplicity, we tried to keep the number of involved parameters as small as possible. This has motivated the iterative computing scheme (GFLANN). As mentioned before, when the number of input features is high, the number of granules needed to get an accurate solution may proportionally grow. In order to achieve a good accuracy, by containing the number of granules, an iterative scheme based on a delta rule has been developed.
On the other hand, there are classifiers which are considered as interpretable to some extent, but they mostly try to find a kind of interpretation of deep learning, leaving some open mathematical issues, e.g. [11], or are based on linearized systems with empirical evaluation (e.g. [3], [18]). Finally, it is the case to mention that some types of Granular Neural Networks were proposed for classification (e.g. [12], [43]), but their convergence was not discussed.
The main contribution of this paper lies in the theoretical framework defining a new granular classifier. Thanks to the adopted formalism, we have been able to introduce a mathematical model to be formally investigated, by proving the convergence. Even though granular computing is meant as a design methodology, the mathematics behind the granular schemes has not found a deeper discussion yet.
Several numerical experiments have been performed, moving from limited datasets, for the sake of comparison with former FLANN-based classifiers, towards high-dimesional datasets, considering state-of-the-art techniques used with those datasets, such as [1], [35], [40]. The emphasis is on the GFLANN classifier, since, as mentioned before, the GFN classifier exhibits some limitations when dealing with a high number of features in datasets. This is reflected in the numerical results, which show the better performance of the GFLANN classifier.
The paper is structured as follows: Section 2 is devoted to some preliminary notation; in Section 3, the new classifier model is introduced; in Section 4, the learning algorithm and its accuracy are formally discussed; Section 5 is referred to numerical experiments and finally in Section 6, some conclusions are offered.
Section snippets
Preliminaries
In this section we recall all the notions related to fuzzy sets, since in this paper we refer to data granulation with fuzzy sets.
Let be a closed interval and with m ≥ 3, be a sequence of points of I, called nodes, such that . A fuzzy partition of I is defined as a sequence of fuzzy sets Ai: I → [0, 1], with such that
- •
Ai(ξ) ≠ 0 if and ;
- •
Ai is continuous and has its unique maximum at ξi;
- •
.
The
The proposed model
Before introducing the model, we recall some basic notions on information granules, FNs and FLANNs.
The learning algorithm
As mentioned before, the proposed granular classifier has two types of weights. The first one, here named the granular weights, denoted as wij, are randomly chosen, as detailed in the next section. The second one, here called output weights, denoted as are unknown. In the GFN scheme, they are learnt through a least squares (LS) approach as explained in the following. The random assignment of some weights combined with the LS approach may be also found in deep architectures (e.g. see [15])
Numerical studies
The numerical experiments are divided into two parts: in the first part, we consider small datasets in order to compare the results against the ones by some recent variants of FLANN (based on certain function bases), such as [10], [27]. In the second part, we consider medium and high-dimensional datasets, referring to the state-of-the-art techniques for the considered datasets such as [1], [35], [40]. Even though our approach cannot be regarded as a variant of the RVFL, we discuss also a
Conclusions
In this paper, we revised FLANNs and FNs (two computational schemes using function bases) from a granular perspective. We defined a new architecture, by introducing a granular layer. We deduced a delta rule based learning algorithm in opposition to a least squares one, leading to an iterative computing scheme in the first case. We formally proved the convergence, under some conditions. The aim here is not to propose a new more accurate classifier, since nowadays, there is a copious literature
CRediT authorship contribution statement
Francesco Colace: Conceptualization, Methodology, Software, Validation, Investigation. Vincenzo Loia: Conceptualization, Methodology. Witold Pedrycz: Conceptualization, Methodology. Stefania Tomasiello: Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Dr. Tomasiello acknowledges support from the European Social Fund via IT Academy programme.
Francesco Colace is Associate Professor in Computer Science in the Department of Industrial Engineering, University of Salerno, Salerno, Italy. His main research directions involve Knowledge Management, Recommender System, Context Aware and Computing, Affective Computing and Sentiment Analysis, e-Learning, ICT for Cultural Heritage. He has published numerous papers in these areas. He is also an author of research monographs and edited volumes covering various aspects of ICT and Cultural
References (46)
- et al.
Ensemble decision forest of RBF networks via hybrid feature clustering approach for high-dimensional data classification
Comput. Stat. Data Anal.
(2019) Comparison theorems on moduli of continuity
Comput. Math. Appl.
(1995)- et al.
Interpretable regularized class association rules algorithm for classification in a categorical data space
Inf. Sci.
(2019) - et al.
Approximation properties of fuzzy transforms
Fuzzy Sets Syst.
(2011) - et al.
Evolutionarily optimized features in functional link neural network for classification
Expert Syst. Appl.
(2010) - et al.
Specifics of medical data mining for diagnosis aid: asurvey
Expert Syst. Appl.
(2019) Adaptive nonlinear system identification with echo state networks
Proceedings of the 15th International Conference on Neural Information Processing Systems – NIPS02
(2002)- et al.
Evolving granular neural networks from fuzzy data streams
Neural Netw.
(2013) - et al.
Interpretable time series classification using linear models and multi-resolution multi-domain symbolic representations
Data Min. Knowl. Discov.
(2019) - et al.
On the role of interpretability in fuzzy data mining
Int. J. Uncertain. Fuzziness Knowl.-Based Syst.
(2007)
A best upper bound for the 2-norm condition number of a matrix
Linear Algebra Appl.
A self adaptive harmony search based functional link higher order ANN for non-linear data classification
Neurocomputing
Neural-net computing and intelligent control systems
Int. J. Control
Abstraction and specialization of information granules
IEEE Trans. Syst. Man Cybern.B
Adaptability, interpretability and rule weights in fuzzy rule-based systems
Inf. Sci.
On the complexity of matrix inversion
Mathematical Note
A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare
Inf. Fusion
Active learning through density clustering
Expert Syst. Appl.
A comprehensive evaluation of random vector functional link networks
Inf. Sci.
Granular neural networks with evolutionary interval learning
IEEE Trans. Fuzzy Syst.
Improved functional link artificial neural network via convex combination for nonlinear active noise control
Appl. Soft Comput.
Functional networks and applications: a survey
Neurocomputing
Principles of Mathematical Analysis
Cited by (10)
Neighborhood rough set-based three-way clustering considering attribute correlations: An approach to classification of potential gout groups
2020, Information SciencesCitation Excerpt :The other is similarity measures between heterogeneous information granules within the framework of granular computing, including distance function [23] membership function [47]. Meanwhile, clustering model is widely used in data dimensionality reduction [1,3,16], event prediction [21], pattern recognition [6]. Although there are many clustering theories and methods to deal with classification with incomplete information, most of the existing suggestions cannot make up all the shortcomings of multi-source heterogeneous information classification problems.
A fuzzy comprehensive evaluation method based on the joint distribution dynamic membership degree for classification problems
2024, Journal of Intelligent and Fuzzy SystemsA New Class of Efficient Adaptive Filters for Online Nonlinear Modeling
2023, IEEE Transactions on Systems, Man, and Cybernetics: SystemsAn improved parameter learning methodology for RVFL based on pseudoinverse learners
2023, Neural Computing and ApplicationsHigher order ANN parameter optimization using hybrid opposition-elitism based metaheuristic
2022, Evolutionary Intelligence
Francesco Colace is Associate Professor in Computer Science in the Department of Industrial Engineering, University of Salerno, Salerno, Italy. His main research directions involve Knowledge Management, Recommender System, Context Aware and Computing, Affective Computing and Sentiment Analysis, e-Learning, ICT for Cultural Heritage. He has published numerous papers in these areas. He is also an author of research monographs and edited volumes covering various aspects of ICT and Cultural Heritage, Pervasive Computing and Sentiment Analysis. Professor Colace is managing, as principal investigator, various research projects founded by national and international institutions. He serves as reviewer for many international journal, as IEEE Transactions on Knowledge and Data Engineering, Knowledge-Based Systems, IEEE Transaction on Education and is a member of various editorial boards of international journals.
Vincenzo Loia received B.S. degree in computer science from University of Salerno, Italy in 1985 and the M.S. and Ph.D. degrees in computer science from University of Paris VI, France, in 1987 and 1989, respectively. From 1989 he is Faculty member at the University of Salerno where he teaches Safe Systems, Situational Awareness, IT Project & Service Management. His current position is as President of University of Salerno, Italy. He is the founder and editor-in-chief of Ambient Intelligence and Humanized Computing Springer, and editor-in-chief of Journal of Evolutionary Intelligence, Springer. He is an Associate Editor of various journals IEEE Transactions Journals. His research interests include soft computing, agent technology for technologically complex environments Web intelligence, Situational Awareness, Cognitive Dedense, Artificial Intelligence. He hold in the last years several role in IEEE Society in particular for Computational Intelligence Society (Chair of Emergent Technologies Technical Committee, IEEE CIS European Representative, Vice-Chair of Intelligent Systems Applications Technical Committee).
Witold Pedrycz (IEEE Fellow, 1998) is Professor and Canada Research Chair (CRC) in Computational Intelligence in the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada. He is also with the Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland. In 2009 Dr. Pedrycz was elected a foreign member of the Polish Academy of Sciences. In 2012 he was elected a Fellow of the Royal Society of Canada. In 2007 he received a prestigious Norbert Wiener award from the IEEE Systems, Man, and Cybernetics Society. He is a recipient of the IEEE Canada Computer Engineering Medal, a Cajastur Prize for Soft Computing from the European Centre for Soft Computing, a Killam Prize, a Fuzzy Pioneer Award from the IEEE Computational Intelligence Society, and 2019 Meritorious Service Award from the IEEE Systems Man and Cybernetics Society. His main research directions involve Computational Intelligence, fuzzy modeling and Granular Computing, knowledge discovery and data science, pattern recognition, data science, knowledge-based neural networks, and control engineering. He has published numerous papers in these areas; the current h-index is 111 (Google Scholar) and 82 on the list top-h scientists for computer science and electronics http://www.guide2research.com/scientists/. He is also an author of 21 research monographs and edited volumes covering various aspects of Computational Intelligence, data mining, and Software Engineering. Dr. Pedrycz is vigorously involved in editorial activities. He is an Editor-in-Chief of Information Sciences, Editor-in-Chief of WIREs Data Mining and Knowledge Discovery (Wiley), and Co-editor-in-Chief of Int. J. of Granular Computing (Springer) and J. of Data Information and Management (Springer). He serves on an Advisory Board of IEEE Transactions on Fuzzy Systems and is a member of a number of editorial boards of international journals.
Stefania Tomasiello, Ph.D. in computer science (University of Salerno, Italy) is currently a lecturer of Artificial Intelligence with the Institute of Computer Science, University of Tartu, Estonia, where she is responsible for the course of Fuzzy Logic and Soft Computing. Formerly, permanent researcher with CO.RI.SA. (Research Consortium on Agent Systems), University of Salerno, Italy and Senior Research Fellow with the Dept. of Management and Innovation Systems (DISA-MIS), University of Salerno. She holds the Italian Scientific Qualification to function as associate professor in computer science and numerical analysis. Workpackage leader in several funded projects and expert evaluator (ex-ante and ex-post) of applied research projects for the Italian Ministry of Economic Development and, formerly, the Italian Ministry of University and Research. She has been adjunct professor of Fundamentals of Computer Science, Human-Computer Interaction, Computational Methods and Finite Element Analysis in the University of Basilicata, Italy. TPC member in many international conferences, here included ACM and IEEE sponsored events. Her research interests lie in scientific and soft computing, AI, fuzzy mathematics, nonlinear dynamics. She authored and co-authored numerous papers in the above mentioned areas. She is managing editor of Evolutionary Intelligence (Springer), associate editor of International Journal of Computer Mathematics (Taylor&Francis), and editorial board member of some journals. ACM, ECMI, EUSFLAT, IEEE and INNS member.