Learning invariance manifolds

doi:10.1016/S0925-2312(99)00011-9

Neurocomputing

Volumes 26–27, June 1999, Pages 925-932

https://doi.org/10.1016/S0925-2312(99)00011-9 Get rights and content

Abstract

A new algorithm for learning invariance manifolds is introduced that allows a neuron to learn a non-linear input–output function to extract invariant or rather slowly varying features from a vectorial input sequence. This is demonstrated by a simple model of learning complex cell responses. The algorithm is generalized to a group of neurons, referred to as a Gibson-clique, to learn slowly varying features that are uncorrelated. Since the input–output functions are non-linear, this technique can be applied iteratively. This is demonstrated by a hierarchical network of Gibson-cliques learning translation invariance.

Introduction

Third […], the process of perception must be described. This is not the processing of sensory inputs, however, but the extracting of invariants from the stimulus flux [4, p. 2].

Learning invariant representations is one of the major problems in neural systems. The approach described in this paper is conceptually most closely related to [1], [2], [8], [9]. The idea is that while an input signal may change quickly due to changes in the sensing conditions, e.g. scale, location, and pose of the object, certain aspects of the input signal change slowly or rarely only, e.g. the presence of a feature or an object. The task of a neural system in learning invariances is therefore the extracting of slow aspects from the input signal.

On an abstract level, the input $x = x (t)$ of a sensor array can be viewed as a trajectory in a high-dimensional input space. Many points in this space can represent the same feature if they only differ in their sensing conditions. One can imagine that these points lie on a manifold (cf. [6]), which may be called invariance manifold. Looking at an object under varying sensing conditions means that the trajectory lies within the invariance manifold. Saccading to a new object, for instance, will cause a jump in the trajectory with a component perpendicular to the manifold. Here, a single manifold is defined by an equipotential surface of a scalar input–output function $g(x)$ in the high-dimensional space. The set of all equipotential surfaces defines a (continuous) family of manifolds. This can be extended to a set of input–output functions $g_{i} (x)$ providing a set of manifold families.

The proposed algorithm differs from [1], [2], [8], [9] in the mathematical formulation, one distinct feature being that input signals are individually combined in a non-linear fashion, which follows the idea that complex non-linear computation can be performed by the dendritic tree [7]. Furthermore, the system is formulated as a learning algorithm rather than an online learning rule, and it is naturally generalized to a group of output neurons, here referred to as a Gibson-clique.

Section snippets

The learning algorithm

Consider a neuron that receives an N-dimensional input signal $x = x (t)$ where t indicates time and $x =[x_{1},…, x_{N}]^{T}$ is a vector. The neuron is able to perform a non-linear transformation on this input defined as a weighted sum over a vector $h =[h_{1},…, h_{M}]^{T}$ of M non-linear functions $h_{m} =h_{m} (x)$ (usually M>N). Here polynomials of order two are used, but other sets of non-linear functions could be used as well. Applying $h$ to the input signal yields the non-linearly expanded signal $s (t)≡ h (x (t))$ . The

Examples

The properties of the learning algorithm are now illustrated by two examples. The first example is about learning complex cell behavior based on simple cell outputs. One Gibson-clique of second-order polynomials is sufficient in this case. A hierarchical network of Gibson-cliques is considered in the second example, which is a model of a visual system learning translation invariance.

Conclusion

A new unsupervised learning algorithm has been presented and tested on two examples. With the algorithm a group of neurons, referred to as a Gibson-clique, can be trained to learn a high-dimensional non-linear input–output function to extract slow components from a vectorial input signal. Since the learned input–output functions are non-linear, the algorithm can be applied iteratively, so that complex input–output functions can be learned in a hierarchical network of Gibson-cliques with limited

Acknowledgements

I am grateful to Terrence Sejnowski for his support and valuable feedback. I have been partially supported by a Fedor-Lynen fellowship by the Alexander von Humboldt-Foundation, Bonn, Germany.

Laurenz Wiskott studied Physics in Göttingen and Osnabrück and received his diploma in 1990. Until 1995 he worked in the group of Christoph von der Malsburg at the Ruhr-University Bochum, Germany, where he received his Ph.D. in Physics. He then was in the group of Terrence Sejnowski at the Salk Institute for Biological Studies, San Diego. Since August 1998 he is at the Institute for Advanced Studies in Berlin. His interests are self-organization and unsupervised learning in the visual system.

References (9)

S. Becker, G.E. Hinton, Spatial coherence as an internal teacher for a neural network, in: Yves Chauvin, David E....
P. Földiák
Learning invariance from transformation sequences
Neural Comput.
(1991)
K. Fukushima, S. Miyake, T. Ito, Neocognitron: a neural network model for a mechanism of visual pattern recognition,...
J.J. Gibson. The Ecological Approach to Visual Perception, Lawrence Erlbaum Associates, London, 1986. Originally...

There are more references available in the full text version of this article.

Cited by (17)

Slow feature subspace: A video representation based on slow feature analysis for action recognition
2023, Machine Learning with Applications
This paper proposes a new video representation for subspace-based action recognition. Traditional subspace-based methods represent a video as a subspace by applying principal component analysis (PCA) to its frames. However, this subspace might lead to an imprecise representation of actions, as PCA loses the temporal information of frames. Therefore, we introduce the slow feature subspace based on the slow feature analysis (SFA). SFA extracts a set of slow features of an input video by projecting the video frames onto the weight vectors that minimize the data variance over time. Motivated by these properties, several methods based on SFA were proposed for action recognition. However, they do not explicitly consider the distribution of the slow features, which represents essential action information. Therefore, our key idea is to capture this distribution through a low-dimensional subspace called slow feature subspace. Our subspace is generated by applying PCA to several weight vectors corresponding to the slowest components obtained by SFA. Our framework replaces the subspaces of several traditional mutual subspace methods with our slow feature subspace to improve their results in action recognition. This approach transforms the comparison between two videos into the comparison between two slow feature subspaces using the canonical angles between them, avoiding vector concatenation and data aggregation. The effectiveness of our framework is demonstrated through extensive experiments with various datasets. Our results show that our slow feature subspace can improve the traditional subspace-based methods and achieve competitive performance compared to different methods, including state-of-the-art neural networks.
An unsupervised domain adaptation approach for change detection and its application to deforestation mapping in tropical biomes
2021, ISPRS Journal of Photogrammetry and Remote Sensing
Citation Excerpt :
They include image difference methods (Bruzzone and Prieto, 2000), image ratio (Afify, 2011) and Change Vector Analysis (CVA) (Liu et al., 2015; Malila, 1980), which has been the basis for more advanced approaches, e.g., (Thonfeld et al., 2016). Transformation-based algorithms (Asokan and Anitha, 2019) involve mapping intensity values into a new feature space through methods such as Principal Component Analysis (PCA)(Sadeghi et al., 2016; Pearson, 1901), Tasselled Cap Transformation (Han et al., 2007; Kauth and Thomas, 1976) and Slow Feature Analysis (SFA) (Wu et al., 2013; Wiskott and Sejnowski, 2002; Wiskott, 1999). More advanced methods include probability graph models (Zhang et al., 2012; Koller and Friedman, 2009), Markov Random Fields (Gu et al., 2017; Li, 2009), Conditional Random Fields (Zhou et al., 2016; Lafferty et al., 2001), Wavelets (Celik and Ma, 2010), among many others.
Changes in environmental conditions, geographical variability and different sensor properties typically make it almost impossible to employ previously trained classifiers for new data without a significant drop in classification accuracy. Domain adaptation (DA) techniques been proven useful to alleviate that problem. In particular, appearance adaptation techniques may be used to adapt images from a specific dataset in such a way that the generated images have a style that is similar to the images from another dataset. Such techniques are, however, prone to creating artifacts that hinder proper classification of the adapted images. In this work we propose an unsupervised DA approach for change detection tasks, which is based on a particular appearance adaptation method: the Cycle-Consistent Generative Adversarial Network (CycleGAN). Specifically, we extend that method by introducing additional constraints in the training phase of the model components, which make it preserve the semantic structure and class transitions in the adapted images. We evaluate the proposed approach on a deforestation detection application, considering different sites in the Amazon rain-forest and in the Brazilian Cerrado (savanna) using Landsat-8 images. In the experiments, each site corresponds to a domain, and the accuracy of a classifier trained with images and references from one (source) domain is measured in the classification of another (target) domain. The results show that the proposed approach is successful in producing artifact-free adapted images, which can be satisfactory classified by the pre-trained source classifiers. On average, the accuracies achieved in the classification of the adapted images outperformed the baselines (when no adaptation was made) by 7.1% in terms of mean average precision, and 9.1% in terms of F1-Score. To the best of our knowledge, the proposed method is the first unsupervised domain adaptation approach devised for change detection.
Kernel approximately harmonic projection
2011, Neurocomputing
Citation Excerpt :
Therefore, this leads us to find out ways to discover the intrinsic low dimensionality of the data in order to solve real learning problems [7–10]. Recently, various researchers [6,11–15] have assumed that those high-dimensional data reside on or close to a submanifold embedded in the ambient space. Two of the representative nonlinear manifold methods are Isomap [13] and Laplacian eigenmap (LE) [6].
Dimensionality reduction is an important preprocessing procedure in computer vision, pattern recognition, information retrieval, and data mining. In this paper we present a kernel method based on approximately harmonic projection (AHP), a recently proposed linear manifold learning method that has an excellent performance in clustering. The kernel matrix implicitly maps the data into a reproducing kernel Hilbert space (RKHS) and makes the structure of data more distinct, which distributes on nonlinear manifold. It retains and extends the advantages of its linear version and keeps the sensitive to the connected components. This makes the method particularly suitable for unsupervised clustering. Besides, this method can cover various classes of nonlinearities with different kernels. We experiment the new method on several well-known data sets to demonstrate its effectiveness. The results show that the new algorithm performs a good job and outperforms other classic algorithms on those data sets.
Enhancing Multi-agent Coordination via Dual-channel Consensus
2024, Machine Intelligence Research
Ha-Bsn: Hardware Acceleration of Bio-Sfa and Bio-Nica, Biological Neural Networks, on Fpga
2023, SSRN
Weakly Supervised Domain Adversarial Neural Network for Deforestation Detection in Tropical Forests
2023, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

View all citing articles on Scopus

¹: http://www.cnl.salk.edu/CNL/

View full text