Preserving Spatio-Temporal Information in Machine Learning: A Shift-Invariant k-Means Perspective

Abstract

In conventional machine learning applications, each data attribute is assumed to be orthogonal to others. Namely, every pair of dimension is orthogonal to each other and thus there is no distinction of in-between relations of dimensions. However, this is certainly not the case in real world signals which naturally originate from a spatio-temporal configuration. As a result, the conventional vectorization process disrupts all of the spatio-temporal information about the order/place of data whether it be 1D, 2D, 3D, or 4D. In this paper, the problem of orthogonality is first investigated through conventional k-means of images, where images are to be processed as vectors. As a solution, shift-invariant k-means is proposed in a novel framework with the help of sparse representations. A generalization of shift-invariant k-means, convolutional dictionary learning is then utilized as an unsupervised feature extraction method for classification. Experiments suggest that Gabor feature extraction as a simulation of shallow convolutional neural networks provides a little better performance compared to convolutional dictionary learning. Other alternatives of convolutional-logic are also discussed for spatio-temporal information preservation, including a spatio-temporal hypercomplex encoding scheme.

Appendix

1.1 A Brief Discussion on the Spatio-temporal Information Preservation

Variations on Neural Networks

Convolution with a kernel in the input side of a layer corresponds to a locally connected structure instead of a traditional fully connected one. Neighboring cells now occur in a relation, preserving the original spatial configuration. As an alternative to the convolutional approach then, neighboring cells in the input or the output side of a neural network layer can also be put in relation with direct edges in-between, as another way of preserving the original spatial configuration that the input cells have. Possibility of edges in-between in the same layer might force to think of a neural network as a more general directed graph. In fact, this line of logic leads to an alternative structure known as recurrent neural networks (RNN). In most general sense, RNNs represent directed graphs. Note that it is possible to build upon basic RNN structure through bidirectional logic [34] and long-short term memory concept [35].

On the other hand, empirical evaluation suggests that temporal convolution, or in other words 1D convolutional-logic surpasses the capacity of recurrent architectures in sequence modeling [36]. It is still an open question whether temporal dimension should be regarded as just another spatial dimension or whether a hybrid approach is better. This is rather a deep issue related to properties of space and time. Instead, considering neural networks of any structure as directed and possibly cyclic graphs, or in other words as neural graphs, might pave way to better understanding of the brain. Note that this concept is rather different than graph neural networks which use graphs as inputs [37].

Another generalization for neural networks is possible by considering infinite width neural networks [38]. Recent results suggest that deep neural networks that are allowed to become infinitely wide converge to models called Gaussian processes [39]. However, such studies do not consider the case when there are in-between connections within layers. Considering the existence of these connections, this can further lead to having an infinite but continuous (input or output) layers, which is indeed applicable mathematically and practically. A generalization of neural network layer cases in this sense is depicted in Fig. 9. The third case in this figure is important in that, it leads to the concept of functional machine learning. This alone may not be enough to preserve the spatial configuration of the input layer. Therefore, additional locally connected versions of these structures can also be proposed.

Figure 9

A generalization of neural network layer cases. (From left-to-right) Discrete-discrete (classical), discrete-continuous, continuous-discrete and continuous-continuous input and output layers.

Tensor-based Sparse Representations

The fact is that images are not vectors, thus vectorization breaks the spatial coherency of images which is investigated by [40]. This line of thought is centralized around tensor factorization as a generalization. The study in [40] reports that by treating training images as a 3D cube and performing a non-negative tensor factorization (NTF); higher efficiency, discrimination and representation power can be achieved when compared to non-negative matrix factorization (NMF).

There are two main branches of tensor decomposition. In the first branch, studies are based on canonical polyadic decomposition (CPD), sometimes also referred to as CANDECOMP/PARAFAC [41]. The most relevant example from literature is K-CPD [42], an algorithm of overcomplete dictionary learning for tensor sparse coding based on a multilinear version of OMP and CANDECOMP/PARAFAC decomposition. K-CPD surpasses conventional methods in a series of image denoising experiments. Most recently, a similar framework is also successfully utilized in tensor-based sparse representations for classification of multiphase medical images [43]. The second branch is centered around the Tucker decomposition model instead, which is a more general model than CPD [44]. The study in [45] presents the foundations of the Tucker decomposition model by defining the Tensor-OMP algorithm which computes a block-sparse representation of a tensor with respect to a Kronecker basis. In [44], authors report that a block-sparse structure imposed on a core tensor through subtensors provide significant results. The Tucker model together with block-sparsity restriction may work significantly well, since the higher dimensional block structure is meaningfully applied on the original sparse tensor in the form of subtensors. There are many other studies in literature specifically based on the Tucker model of sparse representations with or without block-sparsity and additionally including dictionary learning [46,47,48].

Certain parallels can be drawn between convolutional dictionary learning and tensor-based sparse representations. As an example, the study in [49] proposes a novel framework for learning convolutional models through tensor decomposition and shows that cumulant tensors have a CPD whose components correspond to convolutional filters and their circulant shifts.

On the other side, tensor-based approaches (both CPD and Tucker models) do not still provide a solution to 1D case. Without loss of generality, let us assume that the signal is in the form of a column vector $\mathbf{s}$. Since the signal is one-dimensional, there will be a single matrix $\mathbf{D}$ for that single dimension in the Tucker model. Therefore, the model attained is $\mathbf{s} = \mathbf{x} \times _{1}{} \mathbf{D}$ in Eq. (5). It is also possible to show that $\mathbf{x}\times _{1}{} \mathbf{D} = \mathbf{D}{} \mathbf{x}$. From the CPD model perspective, there is equivalently $\sum _{i}{x_{i} \mathbf{d}_{i}^{(1)}}$ where $x_{i}$ is the single sparse coefficient associated with $i^{th}$ atom $\mathbf{d}_i$. Hence, one arrives at a standard formulation in Eq. (5), namely Tucker and CPD models are equivalent in one-dimensional case, all corresponding to conventional orthogonal sparse representation.

$$\begin{aligned} \mathbf{s} = \mathbf{x}\times _{1}{} \mathbf{D} = \mathbf{D}{} \mathbf{x} = {\sum _{i}{x_{i} \mathbf{d}_{i}^{(1)}}} \end{aligned}$$

(5)

The above observation brings up an important question onto the table. Although tensor-based approaches provide advantage when the signals are multidimensional, these formulations will not provide an edge for 1D signals. The remedy may come from considering a 1D signal, not as a 1D vector of elements solely. In other words, a 1D complex vector can be formed by coding the cell positions in the imaginary parts to overcome the orthogonality problem in standard 1D vector representation as depicted in Fig. 10. This paves way to performing sparse representations of complex valued data, or even quaternion valued data, to accommodate more information in cases of higher dimensionality. Utmost generalization is achieved through geometric algebra as a generalization of hypercomplex numbers.

Figure 10

An encoding scheme to preserve spatio-temporal information for (top) 1D mono audio and (bottom) 2D grayscale image cases.

Complex, Hypercomplex and Geometric Algebra Based Approaches

Note that quaternion algebra is the first hypercomplex number system to be devised that is similar to real and complex number systems [50]. The study in [51] states that a quaternion-based model can achieve more structured representation when compared to a tensor-based model. Comparisons between quaternion-SVD and tensor-SVD [52] provide their equivalence, but superiority of quaternion-SVD arises when it is combined with the sparse representation model. It is possible to formulate a quaternion-valued sparse representation of color images that surpasses the conventional logic [51].

There are four possible models to represent color images as suggested in [51]. The first one is the monochromatic model, in which each color channel is represented separately. The second one is the concatenation model, where a single vector is formed by concatenating three color channels [53]. The third is the tensor-based model, where the color image is thought of as a 3D cube of values. The last one is the quaternion-based model, where each color channel is assigned to each imaginary value, i.e., r,g,b to i,j,k respectively. Most importantly, all these models are analytically unified.

There is also one more possible model that is subtler. As depicted in Fig. 10, one can encode a mono audio as a vector of complex numbers where imaginary values indicate the timed position, in a similar way one can encode a grayscale image as a quaternion-valued vector where imaginary parts are allocated to indicate the pixel positions. While thinking of a color image as a 3D cube, there is a possible quaternion-based model in which imaginary units encode the position within this cube and the scalar denotes the value of that cell. The same quaternion-based encoding can be applied to any 3D scalar data.

For further machine learning in this proposed scheme, a hypercomplex to real feature extraction layer is required since current mainstream classification algorithms need real-valued data. Another option is to consult classification algorithms that can directly handle hypercomplex values. This line of logic paves way to consider complex/hypercomplex valued neural networks as viable tools [54, 55]. As a future work, comparison of spatio-temporally encoded hypercomplex neural networks with conventional convolutional or recurrent neural networks may lead to deeper understanding of the deep learning concept. As a motivation, a single complex-valued neuron can solve the XOR problem [56]. In addition, the fact that quaternions can be used to implement associative memory in neural networks is promising [57].

Another line of generalization can deal with the case when the data has more than three dimensions. In such a case, a quaternion is not enough to designate the cell position and its value. As an extension, octonion algebra can accommodate up to seven imaginary channels [58, 59]; however, loses the associativity property. The study in [60] reports that all algebras of dimension larger than eight lose important properties, since they contain algebras of smaller dimension as subalgebras. This might be an issue related to physics of space and time, which is out of scope of this study. The important fact is that the domain dealing with generalization of hypercomplex numbers is called “geometric algebra” and is gaining great attention lately [61].