Skip to main content
SpringerLink
Log in

Markovian Bias of Neural-based Architectures With Feedback Connections

  • Chapter
Perspectives of Neural-Symbolic Integration

Part of the book series: Studies in Computational Intelligence ((SCI,volume 77))

Dynamic neural network architectures can deal naturally with sequential data through recursive processing enabled by feedback connections. We show how such architectures are predisposed for suffix-based Markovian input sequence representations in both supervised and unsupervised learning scenarios. In particular, in the context of such architectural predispositions, we study computational and learning capabilities of typical dynamic neural network architectures. We also show how efficient finite memory models can be readily extracted from untrained networks and argue that such models should be used as baselines when comparing dynamic network performances in a supervised learning task. Finally, potential applications of the Markovian architectural predisposition of dynamic neural networks in bioinformatics are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Christiansen, M., Chater, N.: Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23 (1999) 417-437

    Article  Google Scholar 

  2. Kolen, J.: Recurrent networks: state machines or iterated function systems? In Mozer, M., Smolensky, P., Touretzky, D., Elman, J., Weigend, A., eds.: Pro-ceedings of the 1993 Connectionist Models Summer School. Erlbaum Associates, Hillsdale, NJ (1994) 203-210

    Google Scholar 

  3. Kolen, J.: The origin of clusters in recurrent neural network state space. In: Proceedings from the Sixteenth Annual Conference of the Cognitive Science Society, Hillsdale, NJ: Lawrence Erlbaum Associates (1994) 508-513

    Google Scholar 

  4. Tiňo, P., Čerňanský, M., Beňušková, L.: Markovian architectural bias of recurrent neural networks. IEEE Transactions on Neural Networks 15 (2004) 6-15

    Article  Google Scholar 

  5. Hammer, B., Tino, P.: Recurrent neural networks with small weights implement definite memory machines. Neural Computation 15 (2003) 1897-1929

    Article  MATH  Google Scholar 

  6. Ron, D., Singer, Y., Tishby, N.: The power of amnesia. Machine Learning 25 (1996)

    Google Scholar 

  7. Tiňo, P., Hammer, B.: Architectural bias in recurrent neural networks: Fractal analysis. Neural Computation 15 (2004) 1931-1957

    Google Scholar 

  8. Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78 (1990) 1464-1479

    Article  Google Scholar 

  9. Chappell, G., Taylor, J.: The temporal kohonen map. Neural Networks 6 (1993) 441-445

    Article  Google Scholar 

  10. Koskela, T., znd J. Heikkonen, M.V., Kaski, K.: Recurrent SOM with local linear models in time series prediction. In: 6th European Symposium on Artificial Neural Networks. (1998) 167-172

    Google Scholar 

  11. Horio, K., Yamakawa, T.: Feedback self-organizing map and its application to spatio-temporal pattern classification. International Journal of Computational Intelligence and Applications 1 (2001) 1-18

    Article  Google Scholar 

  12. Voegtlin, T.: Recursive self-organizing maps. Neural Networks 15 (2002) 979-992

    Article  Google Scholar 

  13. Strickert, M., Hammer, B.: Merge SOM for temporal data. Neurocomputing 64 (2005)39-72

    Article  Google Scholar 

  14. Hagenbuchner, M., Sperduti, A., Tsoi, A.: Self-organizing map for adaptive processing of structured data. IEEE Transactions on Neural Networks 14 (2003) 491-505

    Article  Google Scholar 

  15. Hammer, B., Micheli, A., Strickert, M., Sperduti, A.: A general framework for unsupervised processing of structured data. Neurocomputing 57 (2004) 3-35

    Article  Google Scholar 

  16. Tiňo, P., Farkaš, I., van Mourik, J.: Dynamics and topographic organization of recursive self-organizing maps. Neural Computation 18 (2006) 2529-2567

    Article  MATH  MathSciNet  Google Scholar 

  17. Falconer, K.: Fractal Geometry: Mathematical Foundations and Applications. John Wiley and Sons, New York (1990)

    MATH  Google Scholar 

  18. Barnsley, M.: Fractals everywhere. Academic Press, New York (1988)

    MATH  Google Scholar 

  19. Ron, D., Singer, Y., Tishby, N.: The power of amnesia. In: Advances in Neural Information Processing Systems 6, Morgan Kaufmann (1994) 176-183

    Google Scholar 

  20. Guyon, I., Pereira, F.: Design of a linguistic postprocessor using variable memory length markov models. In: International Conference on Document Analysis and Recognition, Monreal, Canada, IEEE Computer Society Press (1995) 454-457

    Chapter  Google Scholar 

  21. Weinberger, M., Rissanen, J., Feder, M.: A universal finite memory source. IEEE Transactions on Information Theory 41 (1995) 643-652

    Article  MATH  Google Scholar 

  22. Buhlmann, P., Wyner, A.: Variable length markov chains. Annals of Statistics 27 (1999) 480-513

    Article  MathSciNet  Google Scholar 

  23. Rissanen, J.: A universal data compression system. IEEE Trans. Inform. Theory 29 (1983) 656-664

    Article  MATH  MathSciNet  Google Scholar 

  24. Giles, C., Omlin, C.: Insertion and refinement of production rules in recurrent neural networks. Connection Science 5 (1993)

    Google Scholar 

  25. Doya, K.: Bifurcations in the learning of recurrent neural networks. In: Proc. of 1992 IEEE Int. Symposium on Circuits and Systems. (1992) 2777-2780

    Google Scholar 

  26. Tiňo, P., Dorffner, G.: Predicting the future of discrete sequences from fractal representations of the past. Machine Learning 45 (2001) 187-218

    Article  MATH  Google Scholar 

  27. Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. Technical Report GMD Report 148, German National Research Center for Information Technology (2001)

    Google Scholar 

  28. Jaeger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304 (2004) 78-80

    Article  Google Scholar 

  29. Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14 (2002) 2531-2560

    Article  MATH  Google Scholar 

  30. Maass, W., Legenstein, R.A., Bertschinger, N.: Methods for estimating the computational power and generalization capability of neural microcircuits. In: Advances in Neural Information Processing Systems. Volume 17., MIT Press (2005)865—872

    Google Scholar 

  31. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks 2 (1989) 359-366

    Article  Google Scholar 

  32. Hammer, B.: Generalization ability of folding networks. IEEE Transactions on Knowledge and Data Engineering 13 (2001) 196-206

    Article  Google Scholar 

  33. Koiran, P., Sontag, E.: Vapnik-chervonenkis dimension of recurrent neural net-works. In: European Conference on Computational Learning Theory. (1997) 223-237

    Google Scholar 

  34. Vapnik, V.: Statistical Learning Theory. Wiley-Interscience (1998)

    Google Scholar 

  35. Shawe-Taylor, J., Bartlett, P.L.: Structural risk minimization over data-dependent hierarchies. IEEE Trans. on Information Theory 44 (1998) 1926-1940

    Article  MATH  MathSciNet  Google Scholar 

  36. Williams, R., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1 (1989) 270-280

    Article  Google Scholar 

  37. Williams, R.: Training recurrent networks using the extended kalman filter. In: Proc. 1992 Int. Joint Conf. Neural Networks. Volume 4. (1992) 241-246

    Article  Google Scholar 

  38. Patel, G., Becker, S., Racine, R.: 2d image modelling as a time-series prediction problem. In Haykin, S., ed.: Kalman filtering applied to neural networks. Wiley (2001)

    Google Scholar 

  39. Manolios, P., Fanelli, R.: First order recurrent neural networks and deterministic finite state automata. Neural Computation 6 (1994) 1155-1173

    Article  Google Scholar 

  40. Tiňo, P., Hammer, B.: Architectural bias in recurrent neural networks - fractal analysis. In Dorronsoro, J., ed.: Artificial Neural Networks - ICANN 2002. Lecture Notes in Computer Science, Springer-Verlag (2002) 1359-1364

    Google Scholar 

  41. de A. Barreto, G., Araújo, A., Kremer, S.: A taxanomy of spatiotemporal connectionist networks revisited: The unsupervised case. Neural Computation 15 (2003) 1255-1320

    Article  MATH  Google Scholar 

  42. Kilian, J., Siegelmann, H.T.: On the power of sigmoid neural networks. Infor-mation and Computation 128 (1996) 48-56

    Article  MATH  MathSciNet  Google Scholar 

  43. Siegelmann, H.T., Sontag, E.D.: Analog computation via neural networks. The-oretical Computer Science 131 (1994) 331-360

    Article  MATH  MathSciNet  Google Scholar 

  44. Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Networks 17 (2004) 1061-1086

    Article  MATH  Google Scholar 

  45. Hammer, B., Neubauer, N.: On the capacity of unsupervised recursive neural networks for symbol processing. In d’Avila Garcez, A., Hitzler, P., Tamburrini, G., eds.: Workshop proceedings of NeSy’06. (2006)

    Google Scholar 

  46. Sonnenburg, S.: New methods for splice site recognition. Master’s thesis, Diplom thesis, Institut für Informatik, Humboldt-Universität Berlin (2002)

    Google Scholar 

  47. Martinetz, T., Berkovich, S., Schulten, K.: ‘neural-gas’ networks for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Networks 4 (1993) 558-569

    Article  Google Scholar 

  48. Baldi, P., Brunak, S.: Bioinformatics: The machine learning approach. MIT Press, Cambridge, Mass (2001)

    MATH  Google Scholar 

  49. Bodén, M., Hawkins, J.: Improved access to sequential motifs: A note on the architectural bias of recurrent networks. IEEE Transactions on Neural Networks 16 (2005) 491-494

    Article  Google Scholar 

  50. Bailey, T.L., Elkan, C.: Fitting a mixture model by expectation maximization to discover motifs in biopolymers. In: Proceedings of ISMB, Stanford, CA (1994) 28-36

    Google Scholar 

  51. Stormo, G.D.: Dna binding sites: representation and discovery. Bioinformatics 16 (2000) 16-23

    Article  Google Scholar 

  52. Sorek, R., Shemesh, R., Cohen, Y., Basechess, O., Ast, G., Shamir, R.: A non-EST-based method for exon-skipping prediction. Genome Research 14 (2004) 1617-1623

    Article  Google Scholar 

  53. Hawkins, J., Bodén, M.: The applicability of recurrent neural networks for bio-logical sequence analysis. IEEE/ACM Transactions on Computational Biology and Bioinformatics 2 (2005) 243-253

    Article  Google Scholar 

  54. Dyrlöv Bendtsen, J., Nielsen, H., von Heijne, G., Brunak, S.: Improved predic-tion of signal peptides: SignalP 3.0. Journal of Molecular Biology 340 (2004) 783-795

    Article  Google Scholar 

  55. Hawkins, J., Bodén, M.: Detecting and sorting targeting papetides with neural networks and support vector machines. Journal of Bioinformatics and Compu-tational Biology 4 (2006) 1-18

    Article  Google Scholar 

  56. Baldi, P., Brunak, S., Frasconi, P., Soda, G., Pollastri, G.: Exploiting the past and the future in protein secondary structure prediction. Bioinformatics 15 (1999) 937-946

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Birmingham, B15 2TT, Edgbaston, Birmingham, UK

    Peter Tiňo

  2. Institute of Informatics, University Clausthal, Julius Albert Strabe 4, 38678, Clausthal-Zellerfeld, Germany

    Dr. Barbara Hammer

  3. School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, Australia

    Mikael Bodén

Authors
  1. Peter Tiňo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dr. Barbara Hammer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mikael Bodén
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University Clausthal, Julius Albert Strabe 4, 38678, Clausthal-Zellerfeld, Germany

    Barbara Hammer

  2. University of Karlsruhe, 76128, Karlsruhe, Germany

    Pascal Hitzler

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Tiňo, P., Hammer, B., Bodén, M. (2007). Markovian Bias of Neural-based Architectures With Feedback Connections. In: Hammer, B., Hitzler, P. (eds) Perspectives of Neural-Symbolic Integration. Studies in Computational Intelligence, vol 77. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73954-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-73954-8_5

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-73953-1

  • Online ISBN: 978-3-540-73954-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation