Growing Self-Organizing Maps for Nonlinear Time-Varying Function Approximation

Abstract

Function approximation may be described as the task of modeling the input-output relation and therefore yielding an estimation of the real output function value. In many domains, an ideal learning algorithm needs to approximate nonlinear time-varying functions from a high-dimensional input space and avoid problems from irrelevant or redundant input data. Therefore, the method has to meet three requirements, namely, it must: allow incremental learning to deal with changing functions and changing input distributions; keep the computational cost low; and achieve accurate estimations. In this paper, we explore different approaches to perform function approximation based on the Local Adaptive Receptive Fields Self-Organizing Map (LARFSOM). Local models are built by calculating between the output associated with the winning node and the difference vector between the input vector and the weight vector. These models are combined by using a weighted sum to yield the final approximate value. The topology is adapted in a self-organizing way, and the weight vectors are adjusted in a modified unsupervised learning algorithm for supervised problems. Experiments were carried out on synthetic and real-world datasets. Experimental results indicate that the proposed approaches perform competitively against Support Vector Regression (SVR) and can improve function approximation accuracy and computational cost against the locally weighted interpolation (LWI), a state-of-the-art interpolating algorithm for self-organizing maps.

Appendices

Derivatives for Learning Rule of Difference Technique

For the vector $\varvec{k}_{s}$ of the winner $n_{s}$:

$$\begin{aligned} \frac{\partial \widetilde{M}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{s}} = \frac{\partial }{\partial \varvec{k}_{s}}\left( w^{out}_{s} + \varvec{k}_{s} \cdot \varvec{d}_{s} \right) = \varvec{d}_{s} \end{aligned}$$

(22)

Derivatives for Learning Rule of Extended Difference Technique

For the vectors $\varvec{k}_{s,j}$ of the winner $n_{s}$:

$$\begin{aligned} \frac{\partial \widetilde{M}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{s,j}} = \frac{\partial }{\partial \varvec{k}_{s,j}}\left( w^{out}_{s} + \sum _{k = 0}^{ |N_{s}| } \varvec{k}_{s,k} \cdot \varvec{d}_{s,k} \right) = \sum _{k = 0}^{|N_{s}|} \frac{\partial \varvec{k}_{s,k}}{\partial \varvec{k}_{s,j}} \cdot \varvec{d}_{s,k} \end{aligned}$$

(23)

since

$$\begin{aligned} \frac{\partial \varvec{k}_{s,k}}{\partial \varvec{k}_{s,j}} = \left\{ \begin{array}{ll} \varvec{I} &{}\quad k = j \\ \varvec{0} &{}\quad k \ne j \end{array}\right. \end{aligned}$$

(24)

then

$$\begin{aligned} \frac{\partial \widetilde{M}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{s,j}} = \varvec{d}_{s,j} \end{aligned}$$

(25)

Derivatives for Learning Rule of Locally Weighted Extended Difference Technique

To calculate the derivative $\frac{\partial \widetilde{F}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}}$, we need to determine $\frac{\partial \widetilde{M}_{(s,k)}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}}$

$$\begin{aligned} \frac{\partial \widetilde{F}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}} = \frac{\partial }{\partial \varvec{k}_{i}}\left( \sum _{k = 0}^{|N_{s}|} m_{s,k}(\varvec{\xi }^{in})\widetilde{M}_{(s,k)}(\varvec{\xi }^{in}) \right) = \sum _{k = 0}^{|N_{s}|} m_{s,k}(\varvec{\xi }^{in}) \frac{\partial \widetilde{M}_{(s,k)}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}} \end{aligned}$$

(26)

We can distinguish two cases:

• 1. Case: $n_{i} = n_{(s,k)}$

  $$\begin{aligned} \frac{\partial \widetilde{M}_{(s,k)}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}} = \frac{\partial }{\partial \varvec{k}_{(s,k),j}} \left( w^{out}_{(s,k)} + \sum _{l = 0}^{ |N_{(s,k)}| } \varvec{k}_{(s,k),l} \cdot \varvec{d}_{(s,k),l} \right) = \sum _{l = 0}^{ |N_{(s,k)}| } \frac{\partial \varvec{k}_{(s,k),l}}{\partial \varvec{k}_{(s,k),j}} \cdot \varvec{d}_{(s,k),l}\nonumber \\ \end{aligned}$$

  (27)

  since

  $$\begin{aligned} \frac{\partial \varvec{k}_{(s,k),l}}{\partial \varvec{k}_{(s,k),j}} = \left\{ \begin{array}{ll} \varvec{I} &{}\quad l = j \\ \varvec{0} &{}\quad l \ne j \end{array}\right. \end{aligned}$$

  (28)

  then

  $$\begin{aligned} \frac{\partial \widetilde{M}_{(s,k)}(\varvec{\xi }^{in})}{\partial \varvec{k}_{(s,k),j}} = \varvec{d}_{(s,k),j} \end{aligned}$$

  (29)

• 2. Case: $n_{i} \ne n_{(s,k)}$

  $$\begin{aligned} \frac{\partial \widetilde{F}_{s}(\varvec{\xi }^{in})}{\partial \varvec{k}_{i}} = \varvec{0} \end{aligned}$$

  (30)

  since

  $$\begin{aligned} \frac{\partial \varvec{k}_{(s,k),l}}{\partial \varvec{k}_{i}} = \varvec{0} \end{aligned}$$

  (31)