SOM-Based System for Sequence Chunking and Planning

Abstract

In this paper we present a connectionist architecture called C-block combining several powerful and cognitively relevant features. It can learn sequential dependencies in incoming data and predict probability distributions over possible next inputs, notice repeatedly occurring sequences, automatically detect sequence boundaries (based on surprise in prediction) and represent sequences declaratively as chunks/plans for future execution or replay. It can associate plans with reward, and also with their effects on the system state. It also supports plan inference from an observed sequence of behaviours: it can recognize possible plans, along with their likely intended effects and expected reward, and can revise these inferences as the sequence unfolds. Finally, it implements goal-driven behaviour, by finding and executing a plan that most effectively reduces the difference between the current system state and the agent’s desired state (goal). C-block is based on modified self-organizing maps that allow fast learning, approximate queries and Bayesian inference.

A Bayesian Inference in the SOM

In our version of SOM, the activity $A_i$ of each unit is computed as

$$\begin{aligned} A_i = \frac{a_i}{\sum _{j=1}^N{a_j}}\text{, } \text{ where } a_i = exp\left( -c \cdot d^2\!\left( \mathbf {x},\mathbf {w}_i\right) \right) \cdot m_i\ . \end{aligned}$$

(1)

$d^2\!\left( \mathbf {x},\mathbf {w}_i\right)$ is the squared Euclidean distance between the input $\mathbf {x}$ and the weight vector $\mathbf {w}_i$, $a_i$ is the (unnormalized) activity of the i-th unit, $m_i$ is the activation mask for the i-th unit. Activities $A_i$ are normalized to sum to 1.

Comparing Eq. 1 with the standard Bayes’ rule

$$\begin{aligned} p(h_i|d) = \frac{p(d|h_i)\cdot p(h_i)}{p(d)} =\frac{p(d|h_i)\cdot p(h_i)}{\sum _{j=1}^N{p(d|h_j)\cdot p(h_j)}} \end{aligned}$$

(2)

we can interpret the activity of each unit as the posterior probability $p(h_i|d)$ of the hypothesis that the current SOM input (data) belongs to the class represented by the unit i. The Gaussian⁹ term $\exp \left( -c \cdot d^2\!\left( \mathbf {w}_i,\mathbf {x}\right) \right)$ corresponds to the likelihood $p(d|h_i)$. The mask $m_i$ corresponds to the prior probability of the i-th hypothesis $p(h_i)$. The denominator $\sum _{j=1}^N{a_j}$ in the formula for normalized activities $A_i$ is a total response of the map to the current input and corresponds to $\sum _{j=1}^p(d|h_j)\cdot p(h_j)=p(d)$, which is just the probability of the data itself. A very low total activity in the map indicates strange (or novel) input data.

By specifying coefficients $m_i$, we can choose different prior bias on the SOM, for example relative frequency of how often the i-th neuron became the best-matching unit in the past. All $m_i$ equal to the same value would effectively mean a uniform prior and will have no influence.

Normalized activity of the whole SOM corresponds to the posterior probability distribution over all the hypotheses/neurons given the current input/data. We can reconstruct the most likely input either as the weights of the winner (‘hard’ output) or as an activity-weighted combination of the weights of all the neurons (‘soft’ output): $\mathbf {y}=\sum _{j=1}^N{A_j\cdot \mathbf {w}_j}$, which corresponds to the expected value of the input given the distribution.