Smoothly distributed fuzzy c-means: a new self-organizing map
Introduction
The well-known self-organizing map (SOM) algorithm [1], [2] is extremely simple in its description and practical implementation. It has been demonstrated to be very successful in producing an orderly mapping of high-dimensional data items onto a regular low-dimensional grid. Specifically, its main property is that it conserves quite consistently the original topological and metric relationships of the items. However, despite the long history and widespread use of the method [3], its theoretical properties are still not fully understood. According to a recent review [4], only the particular SOM case for data and grid in “dimension 1” has been fully worked out. Informally, one can say that “the SOM algorithm usually works well, but we do not thoroughly know why”.
The main theoretical approach towards an understanding of the SOM algorithm in general has been based on efforts to solve the following inverse problem: “find the functional (or equivalently, the cost function) whose numerical optimization corresponds to the SOM algorithm”. For example, energy-type functionals that are clearly related to the algorithm have been proposed and studied by many authors [5], [6], [7], [8], [9]. Although a great deal of insight has been gained, the general problem remains unsolved.
Due to these difficulties, some research groups have developed other procedures different from SOM that attempt to conserve the topological structure of the data. The methodology used for the development of the algorithms is based on the optimization of well-defined cost functions. This approach allows a complete mathematical characterization of the mapping. Although an extensive review is not the aim of this paper, two examples will be mentioned. In one case, Graepel et al. [10], extending the work of Luttrell [11], propose several cost functions that are optimized with a combination of the EM algorithm and a deterministic annealing strategy. In another case, Bishop et al. propose a probabilistic cost function, based on a latent-variables model for the data, which is optimized by means of an EM algorithm [12], [13]. Although these algorithms are usually not quite as simple as SOM, they have the advantage of offering a better understanding and control of the mapping process.
In this study, a different self-organizing map is presented, based on a new cost function and its optimization algorithm. The cost function is derived by introducing two modifications to Bezdek's generalized fuzzy c-means functional, which is widely used in cluster analysis and pattern recognition [14], [15]. First, the code vectors are distributed on a regular low-dimensional grid, as in SOM. Second, a penalization term is added in order to guarantee a smooth distribution for the values of the code vectors on the grid. It should be pointed out that this strategy attempts to conserve the topological structure of the data by explicitly satisfying two conditions: faithfulness to the data (expressed in the original fuzzy c-means functional), and “ordering” (the penalization term).
To the best of our knowledge, few related articles have been previously published. In general, they exhibit only a loose connection to the method presented here. For instance, Lampinen and Oja [9] demonstrated that the SOM algorithm is related to a clustering algorithm. The “batch map” [16], [17], which is a modification of the classical SOM, has been shown by Cheng [18] to be a generalized version of the hard c-means clustering algorithm. In a study by Vuorimaa [19], the SOM algorithm was modified by replacing the neurons with fuzzy rules, allowing an efficient modeling of continuous valued functions. Finally, Chen-Kuo Tsao et al. [20] integrate some aspects of the fuzzy c-means model into the classical SOM framework. The present contribution differs from [20] in that no use is made here of the classical SOM algorithm.
This paper is organized as follows. In Section 2, the fuzzy c-means functional and optimization algorithm is very briefly described. In Section 3, the functional is extended by distributing the code vectors on a low-dimensional grid, and by adding a penalization term that will force them to be “smoothly” distributed. In Section 4, the optimization algorithm is presented. Section 5 presents some mapping examples. Finally, Section 6 contains some concluding remarks about relevant future work that was not included in the aims of the present paper.
Section snippets
Fuzzy c-means: functional and optimization algorithm
A detailed discussion and derivation of the fuzzy c-means problem and algorithm can be found in Ref. [15]. A very brief description follows.
Let , i=1,…,n, denote the data items (represented as n real-valued column vectors of dimension p). Let j=1,…,c, denote the code vectors, with 2⩽c<n. Let denote the partition matrix comprising fuzzy memberships. The elements of U satisfy the following constraints:The fuzzy c-means optimization problem is
The smoothly distributed fuzzy c-means functional
In this case, the code vectors will be distributed on a regular grid. For instance, Fig. 1 shows a 3×3 regular square grid, corresponding to c=9 code vectors. Other topologies can be considered, such as code vectors distributed on a circle, on a 3D cubic grid, on a hexagonal grid, etc. The next ingredient to include in the functional in Eq. (2) is a penalization term that will guarantee smoothness of spatial distribution of the code vectors on the grid. Intuitively, “smoothness”, in the sense
The new self-organizing map algorithm
The basic algorithm is essentially a regularization of the fuzzy c-means algorithm. The solution will be iterative, alternating between Eq. (3) and a solution to Eq. (10). Note that Eq. (10) can be rewritten as:for j=1, …, c, which is now in the convenient form of the Gauss–Seidel iterative algorithm.
The number of computations in Eq. (11) can be made relatively small if the matrix D is sparse. One particular choice for a sparse matrix D is the following
Examples of the new self-organizing map
A first interesting example corresponds to the mapping of 2D data items, consisting of 855 points sampled from within a triangle (Fig. 2a), onto a linear grid formed by 64 code vectors. Mapping back the code vectors in 2D space produces the famous Peano-like curve [16] that tends to fill the triangle (Fig. 2b). In this case, the following parameters were used: ϑ=0.5, with m decreasing from 3 to 1.02 in 500 steps.
The second example corresponds to the mapping of 2D data items, consisting of 62
Discussion
The new cost function introduced in this paper embodies in a straightforward manner the design of an ordered map that attempts to conserve topological structure. Theoretically, this is achieved by complying with two important conditions. The first term in the functional in Eq. (8) demands that the code vectors be faithful to the data, whereas the second term demands that the values of the code vectors change minimally (smoothly) over the grid space, thus ensuring ordering. In practice, as shown
Acknowledgements
We are indebted to an anonymous reviewer for valuable comments. This research was partly supported by the Spanish “Comisión Interministerial de Ciencia y Tecnologia (CICYT)” Grant No. BIO 98-0761, and by the European Union Grant No. QLK2-2000-00634.
References (22)
- et al.
Theoretical aspects of the SOM algorithm
Neurocomputing
(1998) - et al.
Self-organizing mapsgeneralizations and new optimization techniques
Neurocomputing
(1998) - et al.
Developments of the generative topographic mapping
Neurocomputing
(1998) The self-organizing maps
Neurocomputing
(1998)Fuzzy self-organizing map
Fuzzy Sets and Systems
(1994)Self-organized formation of topologically correct feature maps
Biol. Cybernet.
(1982)Analysis of a simple self-organizing process
Biol. Cybernet.
(1982)- S. Kaski, J. Kangas, T. Kohonen, Bibliography of self-organizing map (SOM) papers: 1981–1997, Neural Comput. Surv. 1...
- H. Ritter, K. Schulten, Kohonen's self-organizing maps: exploring their computational capabilities, Proceedings of IEEE...
An analysis of Kohonen's self-organizing maps using a system of energy functionals
Biol. Cybernet.
(1990)
Cited by (52)
A neuro-fuzzy Kohonen network for data stream possibilistic clustering and its online self-learning procedure
2018, Applied Soft Computing JournalCitation Excerpt :Most of the well-known fuzzy clustering methods turn out to be ineffective. Clustering Kohonen neural networks [12] can be used for processing data sequences, however they can solve only crisp clustering tasks and their fuzzy modifications [13,14] work in a batch mode and actually implement the conventional fuzzy C‐means method (FCM) by J. Bezdek [7]. In [15–17], an approach to fast online data streams’ clustering is proposed at the expense of parallelizing data processes.
Incorporating gene ontology into fuzzy relational clustering of microarray gene expression data
2018, BioSystemsCitation Excerpt :In FCM, genes can be assigned to multiple clusters and those genes are involved in different type of coregulation process. To add more advantages to FCM, authors incorporated other ideas to FCM and proposed a hybrid system such as FuzzySOM (Pascual-Marqui et al., 2001) and Fuzzy J-means (Belacel, 2004). Fuzzy J-means considers the local minima problem for cluster solutions and for solving this problem, it used variable neighborhood searching.
An evolving connectionist system for data stream fuzzy clustering and its online learning
2017, NeurocomputingCitation Excerpt :These neural networks have demonstrated their effectiveness solving different tasks with non-overlapping (separable) classes. The necessity of solving the fuzzy clustering tasks in real time (as data streams come to a system) has led to self-learning neuro-fuzzy systems, which are hybrid systems consisting of the Kohonen self-organizing map (SOM) and the fuzzy C-means algorithm by Bezdek (FCM) [15–25]. These hybrid systems possess vast functionality due to the usage of specific tuning algorithms, which are based on optimization procedures of an accepted goal function.
Multi-stage filtering for improving confidence level and determining dominant clusters in clustering algorithms of gene expression data
2013, Computers in Biology and MedicineSelf-Organizing Maps applied to ecological sciences
2011, Ecological InformaticsCitation Excerpt :Further hierarchical data structure can be achieved with the use of network modification. In one study, a secondary SOM (Pascual-Marqui et al., 2001) was used, and the SOM was enhanced when the sets of data associated with a particular weight vector were partitioned into a finer set of clusters. Vesanto and Alhoniemi (2000) applied hierarchical agglomerative and partitive clustering to prototypes that were initially produced by a SOM, and they were subsequently clustered into a second stage.