Abstract
One of the enduring challenges for the Machine Learning community is developing models that can process and learn from very long data sequences. Transformer-based models and Recurrent Neural Networks (RNNs) have excelled in processing long sequences, yet face challenges in transitioning to processing infinite-length sequences online, a crucial step in mimicking human learning over continuous data streams. While Transformer models handle large context windows efficiently, they suffer from quadratic computational costs, motivating research into alternative attention mechanisms. Conversely, RNNs, particularly Deep State-Space Models (SSMs), have shown promise in long sequence tasks, outperforming Transformers in certain benchmarks. However, current approaches are limited to finite-length sequences, which are pre-buffered and randomly shuffled to cope with stochastic gradient descent. This paper addresses the fundamental gap in transitioning from offline-processing of a dataset of sequences to online-processing of possibly infinite-length sequences, a scenario often neglected in existing research. Empirical evidence is presented, demonstrating the performance and limits of existing models. We highlight the challenges and opportunities in learning from a continuous data stream, paving the way for future research in this area.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
From [29]: “an on-line algorithm, designed to be used to train a network while it runs; no manual state resets or segmentations of the training stream is required”. From [8]: LSTMs were introduced with a learning algorithm that unlike BPTT is “local in space and time”, where “there is no need to store activation values observed during sequence processing in a stack with potentially unlimited size”.
- 2.
\(D\ne 0\) introduces a skip connection that was not included in the original Elman network [4].
- 3.
Transformers usually include multiple multi-head attention-based layers interleaved by FFNs. We considered a simple head, with no further projections, to better connect it to the RNN case.
- 4.
For a detailed description of each dataset, see [32], except for Synth, which is a sinusoidal signal at frequency of 2.8 mHz, with 36000 samples.
- 5.
- 6.
In FSNet, Synth data, we considered the second-best sets of parameter values, since the best ones were yielding numerical errors.
References
Betti, A., Casoni, M., Gori, M., Marullo, S., Melacci, S., Tiezzi, M.: Neural time-reversed generalized riccati equation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 7935–7942 (2024)
Betti, A., et al.: Continual learning through Hamilton equations. In: Conference on Lifelong Learning Agents, vol. 199, pp. 201–212. PMLR (2022)
Betti, A., Gori, M., Melacci, S.: Cognitive action laws: the case of visual features. IEEE Trans. Neural Netw. Learn. Syst. 31(3), 938–949 (2020). https://doi.org/10.1109/TNNLS.2019.2911174
Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)
Anil, R.G., et al.: Gemini: a family of highly capable multimodal models (2023)
Gori, M., Melacci, S.: Collectionless artificial intelligence. arXiv 2309.06938 (2023)
Gunasekara, N., Pfahringer, B., Gomes, H.M., Bifet, A.: Survey on online streaming continual learning. In: Proceedings of the IJCAI, pp. 6628–6637 (2023)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are RNNs: fast autoregressive transformers with linear attention. In: International Conference on Machine Learning, pp. 5156–5165 (2020)
Lipton, Z.C., Berkowitz, J., Elkan, C.: A critical review of recurrent neural networks for sequence learning. arXiv:1506.00019 (2015)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Marschall, O., Cho, K., Savin, C.: A unified framework of online learning algorithms for training recurrent neural networks. J. Mach. Learn. Res. 21(1), 5320–5353 (2020)
Marullo, S., Tiezzi, M., Betti, A., Faggi, L., Meloni, E., Melacci, S.: Continual unsupervised learning for optical flow estimation with deep networks. In: Conference on Lifelong Learning Agents, pp. 183–200. PMLR (2022)
Massaroli, S., et al.: Laughing hyena distillery: extracting compact recurrences from convolutions. In: Advances in NeurIPS (2023). https://openreview.net/forum?id=OWELckerm6
Orvieto, A., et al.: Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349 (2023)
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International Conference on ML, pp. 1310–1318 (2013)
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N., Kong, L.: Random feature attention. In: ICLR (2021). https://openreview.net/forum?id=QtTKTdVrFBB
Pham, Q., Liu, C., Sahoo, D., Hoi, S.: Learning fast and slow for online time series forecasting. In: ICLR (2022)
Rumelhart, D.E., et al.: Learning internal representations by error propagation (1985)
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., Valaee, S.: Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078 (2017)
Schäfer, A.M., Zimmermann, H.G.: Recurrent neural networks are universal approximators. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4131, pp. 632–640. Springer, Heidelberg (2006). https://doi.org/10.1007/11840817_66
Smith, J.T., Warrington, A., Linderman, S.W.: Simplified state space layers for sequence modeling. preprint arXiv:2208.04933 (2022)
Tallec, C., Ollivier, Y.: Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043 (2017)
Tay, Y., et al.: Long range arena: a benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 (2020)
Tiezzi, M., Marullo, S., Faggi, L., Meloni, E., Betti, A., Melacci, S.: Stochastic coherence over attention trajectory for continuous learning in video streams. In: Proceedings of IJCAI-22, pp. 3480–3486 (2022)
Tiezzi, M., Melacci, S., Betti, A., Maggini, M., Gori, M.: Focus of attention improves information transfer in visual features. In: Advances in NeurIPS, vol. 33, pp. 22194–22204 (2020)
Vaswani, A., et al.: Attention is all you need. In: Advances in NeurIPS, vol. 30 (2017)
Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)
Williams, R.J., Peng, J.: An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Comput. 2(4), 490–501 (1990)
Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Comput. 1(2), 270–280 (1989)
Yu, Y., Si, X., Hu, C., Zhang, J.: A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput. 31(7), 1235–1270 (2019)
Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI Conference on AI, vol. 35(12), pp. 11106–11115 (2021). https://doi.org/10.1609/aaai.v35i12.17325, https://ojs.aaai.org/index.php/AAAI/article/view/17325
Zucchet, N., Meier, R., Schug, S., Mujika, A., Sacramento, J.: Online learning of long range dependencies. In: NeurIPS (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Casoni, M., Guidi, T., Tiezzi, M., Betti, A., Gori, M., Melacci, S. (2024). Pitfalls in Processing Infinite-Length Sequences with Popular Approaches for Sequential Data. In: Suen, C.Y., Krzyzak, A., Ravanelli, M., Trentin, E., Subakan, C., Nobile, N. (eds) Artificial Neural Networks in Pattern Recognition. ANNPR 2024. Lecture Notes in Computer Science(), vol 15154. Springer, Cham. https://doi.org/10.1007/978-3-031-71602-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-71602-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71601-0
Online ISBN: 978-3-031-71602-7
eBook Packages: Computer ScienceComputer Science (R0)
