An adaptive efficient memristive ink drop spread (IDS) computing system

Abstract

Active Learning Method (ALM) is one of the powerful tools in soft computing and it is inspired by the human brain capabilities in approaching complicated problems. ALM, which is in essence an adaptive fuzzy learning algorithm, tries to model a Multi-Input Single-Output system with several single-input single-output subsystems. Each of these subsystems is then modeled by an ink drop spread (IDS) plane. IDS operator, which is the main processing engine of ALM, extracts two kinds of informative features, Narrow Path and Spread, from each IDS plane without complicated computations. These features from all IDS planes are then aggregated in the inference engine. Despite the great performance of ALM in different applications, an efficient hardware implementation has remained a challenge, which is mainly due to considerably high memory requirement of IDS operation. In this paper, in a novel approach to IDS operation, we propose an abstract representation of the IDS planes which minimizes the memory requirement and the computational cost, and consequently, benefits the hardware implementation in terms of area and speed. The proposed approach is fully compatible with memristor-crossbar implementation with an adaptive learning capability. Simpler learning algorithm and higher speed make our proposed algorithm suitable for applications where real-time process, low-cost and small implementation are of high priority. Applications in the classification of real-world datasets and function approximation are provided to confirm the effectiveness of the algorithm. Eventually, the paper concludes that the proposed computing structure provides a synergy between artificial neural networks and fuzzy domains.

Appendix A: Proof of convergence

As it has also been discussed in the body of paper, the training mode in the original ALM algorithm is Batch-Mode, and in our proposed algorithm is Sample-Mode. This modification in training mode is the main reason for the increase in the training speed and the decrease in the required memory space and hardware implementation. Convergence occurs in both modes, but not necessarily equivalent. As the computing space has uncertainty (fuzzy space), this error and different equivalency do not significantly affect the final result. In this computing space, it cannot be said that the output has become better or worse (but the average of convergence of Batch-based algorithm is better than that of Sample-Based). In order to prove the convergence of $c_{\text{NP}}$ vector, first, we show the change in one element of this vector when different training inputs are introduced:

$$c_{{{\text{NP}}K_{I} + 1}}^{I} = c_{{{\text{NP}}0}}^{I} + \alpha_{2} \times \mathop \sum \limits_{t = 0}^{{K_{I} }} {\text{e}}^{{ - \frac{{(I - x_{t} )^{2} }}{{2\sigma^{2} }}}} \times \left( {y_{t} - c_{{{\text{NP}}_{t} }}^{{x_{t} }} } \right)$$

(40)

Where, each training input is shown as pair $(x_{t} ,y_{t} )$, I is the index of the vector’s elements for updating, $K_{I}$ is the total number of training samples in element I, $\alpha_{2}$ is learning rate, $x_{t}$ is the input that vary between one and the maximum of the quantization level of $X$ axis ($k_{Rsnx}$). For each training input associated with each quantization level, following equation holds:

$$k_{1} + k_{2} + \cdots + k_{Rsnx} = K$$

(41)

$K$ is the total number of training samples in an IDS plane (number of iterations). For the benefit of simplicity and less computations, if $x_{t} = I$, the previous equation can be simplified as follows:

$$c_{{{\text{NP}}K_{I} + 1}}^{I} = c_{{{\text{NP}}0}}^{I} + \alpha_{2} \times \sum\limits_{t = 0}^{{K_{I} }} {\left( {y_{t} - c_{{{\text{NP}}_{t} }}^{I} } \right)}$$

(42)

If we expand the summation, following equations can be obtained:

$$\begin{aligned} & k_{I} = 0, \quad c_{{{\text{NP}}1}}^{I} = c_{{{\text{NP}}0}}^{I} + \alpha_{2} \times \left( {y_{0} - c_{{{\text{NP}}_{0} }}^{I} } \right) \\ & k_{I} = 1, \quad c_{{{\text{NP}}2}}^{I} = c_{{{\text{NP}}1}}^{I} + \alpha_{2} \times \left( {y_{1} - c_{{{\text{NP}}_{1} }}^{I} } \right) \\ & k_{I} = 2, \quad c_{{{\text{NP}}3}}^{I} = c_{{{\text{NP}}2}}^{I} + \alpha_{2} \times \left( {y_{2} - c_{{{\text{NP}}_{2} }}^{I} } \right) \\ & \ldots \\ \end{aligned}$$

(43)

Eventually, a recursive equation can be obtained as follows:

$$c_{{{\text{NP}}K_{I} + 1}}^{I} = \left( {1 - \alpha_{2} } \right)^{{k_{I} + 1}} \times c_{{{\text{NP}}0}}^{I} + \alpha_{2} \times \mathop \sum \limits_{i = 0}^{{K_{I} }} (1 - \alpha_{2} )^{{k_{I} - i}}\times y_{i}$$

(44)

If $K_{I} \gg 1,$ in the previous equation, the first term of the equation tends to zero, and by variable replacement as $p = k_{I} - i$ following equation can be obtained:

$$c_{{{\text{NP}}K_{I} + 1}}^{I} = \alpha_{2} \times \mathop \sum \limits_{i = 0}^{p} (1 - \alpha_{2} )^{p} *y_{{k_{I} - p}}$$

(45)

This equation shows that the final output of the algorithm converges to the weighted sum of outputs. With regard to forgetting property introduced in the proposed method, the impact of training samples that were shown later in the training phase is higher than that of training samples that were shown to the algorithm earlier. If $y_{{k_{I} - p}}$ are the same, the previous equation is a geometric series which converges to $y_{{k_{I} - p}}$ eventually. It should be mentioned that because of the fact that ALM and EMALM algorithms partition the inputs domains. This results in lower spread in inputs $y_{{k_{I} - p}}$ and consequently lower range of variation in $y_{{k_{I} - p}}$. Therefore, for the mentioned recursive algorithm, the standard deviation of Narrow Path is small and because the computations are conducted in the space with uncertainty and random choice of $y_{{k_{I} - p}}$, this error and lack of equivalency do not significantly affect the final result.

With regard to assumptions we made about the convergence of describing vectors, the value of spread would be:

$$c_{{{\text{UB}}K_{I} + 1}}^{I} = c_{{{\text{UB}}0}}^{I} + \alpha_{1} \times \left( {y_{{k_{I} }} - c_{{{\text{UB}}K_{I} }}^{I} } \right)$$

(46)

$$c_{{{\text{LB}}K_{I} + 1}}^{I} = c_{{{\text{LB}}0}}^{I} + \alpha_{1} \times \left( {y_{{k_{I} }} - c_{{{\text{LB}}K_{I} }}^{I} } \right)$$

(47)

If $c_{{{\text{UB}}K_{I} + 1}}^{I} - c_{{{\text{LB}}K_{I} + 1}}^{I} = SP_{{K_{I} + 1}}^{I}$ is inserted in the previous equations, we would have:

$$SP_{{K_{I} + 1}}^{I} = SP_{{K_{I} }}^{I} - \alpha_{1} \times SP_{{K_{I} }}^{I} = \left( {1 - \alpha_{1} } \right)\times SP_{{K_{I} }}^{I}$$

(48)

For some values of $k_{I}$, we would have:

$$\begin{aligned} & k_{I} = 0,\quad SP_{1}^{I} = (1 - \alpha_{1} ) \times SP_{0}^{I} \\ & k_{I} = 1,\quad SP_{2}^{I} = (1 - \alpha_{1} ) \times SP_{1}^{I} \\ & k_{I} = 2,\quad SP_{3}^{I} = (1 - \alpha_{1} ) \times SP_{2}^{I} \\ & \ldots \\ \end{aligned}$$

(49)

Eventually, a recursive equation can be obtained as follows:

$$SP_{{K_{I} + 1}}^{I} = (1 - \alpha_{1} )^{{K_{I} + 1}} \times SP_{0}^{I}$$

(50)

This equation shows that as the number of training samples (or the number of iterations) increases, the value of spread decreases, and eventually our belief to the occurrence of that event ($C_{\text{NP}}$ vector) increases.