Abstract
Traditional approaches to sensory coding use code component-oriented objective functions (COCOFs) to evaluate code quality. Previous COCOFs do not take into account the information-theoretic complexity of the code-generating mapping itself. We do: “Low-complexity coding and decoding” (LOCOCODE) generates so-called lococodes that (1) convey information about the input data, (2) can be computed from the data by a low-complexity mapping (LCM), and (3) can be decoded by a LCM. We implement LococoDE by training autoassociators with Flat Minimum Search (FMS), a general method for finding lowcomplexity neural nets. LococoDE extracts optimal codes for difficult versions of the “bars” benchmark problem. As a preprocessor for a vowel recognition benchmark problem it sets the stage for excellent classification performance.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
References
H. B. Barlow, T. P. Kaushal, and G. J. Mitchison. Finding minimum entropy codes. Neural Computation, 1(3):412–423, 1989.
P. Dayan and R. Zemel. Competition and multiple cause models. Neural Computation, 7:565–579, 1995.
B. A. Olshausen; D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
D. J. Field. What is the goal of sensory coding? Neural Computation, 6:559–601, 1994.
T. J. Hastie, R. J. Tibshirani, and A. Buja. Flexible discriminant analysis by optimal scoring. Technical report, AT&T Bell Laboratories, 1993.
G. E. Hinton and Z. Ghahramani. Generative models for discovering sparse distributed representations. Technical report, University of Toronto, Department of Computer Science, Toronto, Ontario, M5S 1A4, Canada, 1997. A modified version to appear in Philosophical Transactions of the Royal Society B.
S. Hochreiter and J. Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.
T. Kohonen. Self-Organization and Associative Memory. Springer, second ed., 1988.
A. J. Robinson. Dynamic Error Propagation Networks. PhD thesis, Trinity Hall and Cambridge University Engineering Department, 1989.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1997 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hochreiter, S., Schmidhuber, J. (1997). Unsupervised coding with lococode. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, JD. (eds) Artificial Neural Networks — ICANN'97. ICANN 1997. Lecture Notes in Computer Science, vol 1327. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0020229
Download citation
DOI: https://doi.org/10.1007/BFb0020229
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-63631-1
Online ISBN: 978-3-540-69620-9
eBook Packages: Springer Book Archive