Spiking neuro-fuzzy clustering system and its memristor crossbar based implementation

Abstract

This study proposes a spiking neuro-fuzzy clustering system based on a novel spike encoding scheme and a compatible learning algorithm. In this system, we utilize an analog to binary encoding scheme that properly maps the concept of “distance” in multi-dimensional analog spaces to the concept of “dissimilarity” of binary bits in the equivalent binary spaces. When this scheme is combined with a novel binary to spike encoding scheme and a proper learning algorithm is applied, a powerful clustering algorithm is produced. This algorithm creates flexible fuzzy clusters in its analog input space and modifies their shapes to different convex shapes during the learning process. This system has plausible biological support due to its spike-based learning mechanism, its Quasi Spike Time Dependent Plasticity learning policy and its brain-like fuzzy clustering performance. Moreover, this neuro-fuzzy system is fully implementable on the hybrid memristor-crossbar/CMOS platform. The resultant circuit was simulated on one clustering task carried out in the binary input space on the Simon Lucas handwritten dataset and another clustering task carried out in the analog input space on Fisher׳s Iris standard dataset. The results show that it attained a higher clustering rate in comparison with other algorithms such as the Self Organizing Map, K-mean and the Spiking Radial Basis Function. The circuit was also successfully simulated on an image segmentation task and some clustering tasks performed in noisy spaces with various cluster sizes. Furthermore, the circuit variability analysis shows that device and signal variations up to 20% had no significant impact on the circuit׳s clustering performance, so the system is sufficiently immune to different variations due to its fuzzy nature.

Introduction

The main goal of research in the field of Artificial Intelligence (AI) is to obtain an artificial system with the far-ranging capacities and high performance of the human brain. Research activities can be broken down into several subfields such as Artificial Neural Networks (ANN) and fuzzy logic focusing on the brain׳s different specifications and capabilities. The concept of fuzzy logic was introduced by Zadeh [1] to represent and manipulate data and information possessing with non-statistical uncertainties. From an arithmetical viewpoint, fuzzy logic is a multi-valued logic that makes it possible to define intermediate values between crisp values like yes/no, true/false, black/white, etc. In other words, linguistic terms are given indistinct boundaries by introducing gradual membership functions. In contrast to the classical set theory, in which an object or a case either is or is not a member of a given set, fuzzy logic makes it possible for an object or a case to belong to a set only to a certain degree.

Fuzzy logic, applied in combination with other theoretical and practical concepts, has resulted in robust systems such as fuzzy controllers [2], [3], [4], neuro-fuzzy networks [5], [6], fuzzy decision makers [7], fuzzy arithmetic algorithms [8], etc. While fuzzy rules generally take the form of easily understood if… then statements, it is sometimes difficult to understand how neural network rules actually perform tasks. For this reason, fuzzy logic and neural networks are considered to be complementary areas of research [9].

One of the basic tasks which the human brain performs proficiently is environmental object clustering. In a clustering task, the network must categorize input samples into appropriate clusters. Clustering applications are problematic mostly because of the abstruse spreading of samples and the varying boundary shapes of the clusters. The complexity and limitations of the previously presented mechanisms are largely attributable to the lack of an effective way to define cluster boundaries. With higher-dimensionality inputs, this problem becomes even more serious.

In neural networks, synapses are the connections between neurons by which signals, known as spikes, are exchanged. Synaptic plasticity is the capability of synapses to tune their transmission strengths, known as weights, to learn a specific function. Some biological experiments [10] support a weight change policy called Spike Time Dependent Plasticity (STDP). Memristor, as two-terminal, nano-scale device possessing a combination of resistive and memory characteristics, is one of the most promising devices to be used as synapse for hardware realization of neural networks. Accordingly, the analog values of synaptic weights are stored in the memristors, and may vary through STDP. Although this so-called missing fourth circuit element was first theoretically predicted in 1971 [12], the Hewlett-Packard (HP) memristor [11] was the first acknowledged physical realization of this thin-film device using titanium dioxide. Potential applications of the memristor include resistive memories (RRAM) [13], programmable analog circuits, variable gain amplifiers and adaptive filters [14], digital logic [15], synapses in Spiking Neural Networks (SNN) [16], [17], [18], [19], [20], [21], [22], [23], [24], fuzzy systems [25] and bio-inspired neuron models [26].

The memristor׳s memory property makes it a suitable device for neuromorphic engineers and AI researchers for use in bio-inspired neural networks with specific applications [19], [20], [21], [22]. Compared to the other options, such as capacitor-based memory cells, the memristor can retain its memristance forever with no input voltage or supplied current. Furthermore, its nano-scale size, low power consumption and fabrication process compatible with conventional CMOS [27] make it an excellent choice. At the architectural level, a crossbar-based structure appears to be the most promising nanotechnology architecture [28]. The major advantages of this architecture are its inherent defect-tolerance capability, its simplicity, its flexibility, its scalability and its ability to provide maximum density.

In this paper, we propose a novel neuro-fuzzy approach, where each neuron creates a flexible convex membership function for its specific cluster in the input space. Input samples are applied to the neurons using spikes with a certain encoding scheme. The “winner neuron”, which displays the highest degree of membership, is trained through modifying its membership functions. Thanks to the adaptive membership functions used in this approach, complex clustering tasks can be performed. In the proposed structure, computing and memory units are integrated with each other in a similar way to how they are integrated in the brain. This structure is easily implementable on the memristor-crossbar/CMOS platform thanks to its spiking nature and its compatible network structure, spike encoding schemes and learning algorithm. In this scheme, we used a Quasi STDP (QSTDP) learning algorithm, which has been used before in robust approaches [21], [22], and the compatible signal shapes proposed by Querlioz et al. [21].

One of the most significant spike-based computational systems is the Spiking Radial Basis Function (Spiking RBF) [29], a system that has achieved a high clustering performance in comparison with conventional neural networks, but is very hard to implement because of its various embedded delay units, its complex learning policy, and its complex data to spike encoder. A powerful neuro-fuzzy approach, fully implementable on memristor-crossbar/CMOS, was described in Ref. [25]. This approach applied an analog input current for computation, causing high sensitivity to variability and noise. Other troublesome issues are how to produce input currents proportional to the membership degrees and how to implement the learning mechanism to change the state of the memristors. One solid study in this direction proposed a memristive neural network [21]. This approach ignored the similarity between zero bits, and used a variable-threshold neuron with a homeostatic-type mechanism, which is difficult to implement. Moreover, it was not capable of working in multi-dimensional analog spaces. Another worthwhile study into spike-based computational systems proposed a classifier on the event-based digital video input [22]. This work proposed a dual Phase-Change Memory (2-PCM) synapse [22] for a memristor-based implementation of the classifier. The 2-PCM synapse can properly mimic synaptic behavior and apply the learning policy, but it has some drawbacks: firstly, it uses 2 memristors and 2 CMOS gates to implement one synapse, which occupies a relatively large area. Secondly, it cannot be implemented on a memristor-only crossbar structure. And thirdly, it requires an ongoing refresh process due to memristance changes in one direction. Another scheme for implementing a memristive synapse, called Bridge Synapse [17], suffered from the first two disadvantages mentioned above and also used external equipment for off-chip learning, thereby increasing the area consumption due to the computer-connecting auxiliary circuits and external computer. Taking into account all the defects of the other, similar works mentioned above, we have tried to develop a high performance, easily implementable, low cost approach that is robust to noise and variations and uses a stand-alone learning mechanism without the need for external equipment.

The paper is organized as follows: Section 2 discusses the encoding scheme in binary space and the learning mechanism, proposes the binary to spike Pair Encoding (PE) scheme and examines its benefits in comparison with a single spike encoding scheme. This section is the basis of the proposed neuro-fuzzy algorithm introduced in the next section. Section 3 illustrates the prerequisite conditions for the analog to binary encoding scheme, studies these conditions on three possible schemes, and proposes the analog to binary Step Encoding (SE) scheme as the most satisfactory choice. This section then provides a neuro-fuzzy interpretation of the final scheme. The memristor crossbar-based circuit of the system proposed in the previous sections is introduced in Section 4. In Section 5 we simulate the circuits step-by-step with standard case studies such as the Simon Lucas handwritten dataset (binary input space), Fisher׳s Iris dataset (analog input space), hierarchical and noisy data samples, and an image segmentation task, and compare the clustering performance of the network with other approaches. We also look at the circuit׳s immunity to device and signal variations.