Abstract
This study investigates the topological structures of neural network activation graphs, with a focus on detecting higher-order Betti numbers during reinforcement learning. The paper presents visualisations of the neurotopological dynamics of reinforcement learning agents both during and after training, which are useful for different dynamics analyses which we explore in this work. Two applications are considered: frame-by-frame analysis of agent neurotopology and tracking per-neuron presence in cavity boundaries over training steps. The experimental analysis suggests that higher-order Betti numbers found in a neural network’s functional graph can be associated with learning more complex behaviours.
This research was partially supported by the Australian Government through the ARC’s Discovery Projects funding scheme (project DP210103304). The first author was supported by a PhD scholarship from the University of Newcastle.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Birdal, T., Lou, A., Guibas, L.J., Simsekli, U.: Intrinsic dimension, persistent homology and generalization in neural networks. Adv. Neural. Inf. Process. Syst. 34, 6776–6789 (2021)
Botnan, M.B., Hirsch, C.: On the consistency and asymptotic normality of multiparameter persistent betti numbers. J. Appl. Comput. Topol. 1–38 (2022). https://doi.org/10.1007/s41468-022-00110-9
Brockman, G., et al.: Openai gym. CoRR abs/1606.01540 (2016)
Cavanna, N.J., Jahanseir, M., Sheehy, D.R.: A geometric perspective on sparse filtrations. arXiv:1506.03797 (2015)
Chen, C., Ni, X., Bai, Q., Wang, Y.: A topological regularizer for classifiers via persistent homology. In: Chaudhuri, K., Sugiyama, M. (eds.) Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 89, pp. 2573–2582. PMLR (4 2019)
Corneanu, C., Madadi, M., Escalera, S., Martinez, A.: Explainable early stopping for action unit recognition. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 693–699 (2020). https://doi.org/10.1109/FG47880.2020.00080
Corneanu, C.A., Escalera, S., Martinez, A.M.: Computing the testing error without a testing set. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2674–2682 (2020). https://doi.org/10.1109/CVPR42600.2020.00275
Corneanu, C.A., Madadi, M., Escalera, S., Martinez, A.M.: What does it mean to learn in deep networks? and, how does one detect adversarial attacks? In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4752–4761 (2019). https://doi.org/10.1109/CVPR.2019.00489
Edelsbrunner, H., Harer, J.: Computational Topology: An Introduction. American Mathematical Society, Applied Mathematics (2010)
Edelsbrunner, H., Parsa, S.: On the computational complexity of betti numbers: reductions from matrix rank. In: Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 152–160. SIAM (2014)
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)
Gebhart, T., Schrater, P.: Adversary detection in neural networks via persistent homology. arXiv:1711.10056 (2017)
Geirhos, R., et al.: Shortcut learning in deep neural networks. Nature Mach. Intell. 2(11), 665–673 (2020). https://doi.org/10.1038/s42256-020-00257-z
Gromov, M.: Hyperbolic groups. In: Gersten, S.M. (ed.) Essays in Group Theory, pp. 75–263. Springer New York, New York, NY (1987). https://doi.org/10.1007/978-1-4613-9586-7_3
Gutiérrez-Fandiño, A., Fernández, D.P., Armengol-Estapé, J., Villegas, M.: Persistent homology captures the generalization of neural networks without A validation set. arXiv:2106.00012 (2021)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems 28 (2015)
Hatcher, A.: Algebraic Topology. Cambridge University Press, Algebraic Topology (2002)
Hensel, F., Moor, M., Rieck, B.: A survey of topological machine learning methods. Front. Artif. Intell. 4 (2021). https://doi.org/10.3389/frai.2021.681108
Hofer, C., Graf, F., Niethammer, M., Kwitt, R.: Topologically densified distributions. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 4304–4313. PMLR (7 2020)
Hofer, C., Kwitt, R., Niethammer, M., Uhl, A.: Deep learning with topological signatures. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1633–1643. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
Lütgehetmann, D., Govc, D., Smith, J.P., Levi, R.: Computing persistent homology of directed flag complexes. Algorithms 13(1) (2020). https://doi.org/10.3390/a13010019
Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pp. 8024–8035 (2019)
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22(268), 1–8 (2021)
Reimann, M.W., et al.: Cliques of neurons bound into cavities provide a missing link between structure and function. Frontiers in Computational Neuroscience 11 (2017). https://doi.org/10.3389/fncom.2017.00048
Rote, G., Vegter, G.: Computational topology: an introduction. In: Effective Computational Geometry for Curves and Surfaces, pp. 277–312. Springer Heidelberg (2006). https://doi.org/10.1007/978-3-540-33259-6_7
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA (2018). https://doi.org/10.5555/3312046
Todorov, E., Erez, T., Tassa, Y.: Mujoco: A physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033. IEEE (2012). https://doi.org/10.1109/IROS.2012.6386109
Tralie, C., Saul, N., Bar-On, R.: Ripser.py: A lean persistent homology library for python. J. Open Source Softw. 3(29), 925 (9 2018). https://doi.org/10.21105/joss.00925
Vietoris, L.: Über den höheren zusammenhang kompakter räume und eine klasse von zusammenhangstreuen abbildungen. Math. Ann. 97(1), 454–472 (1927)
Watanabe, S., Yamana, H.: Topological measurement of deep neural networks using persistent homology. Ann. Math. Artif. Intell. 90(1), 75–92 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Muller, M., Kroon, S., Chalup, S. (2024). Topological Dynamics of Functional Neural Network Graphs During Reinforcement Learning. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1963. Springer, Singapore. https://doi.org/10.1007/978-981-99-8138-0_16
Download citation
DOI: https://doi.org/10.1007/978-981-99-8138-0_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8137-3
Online ISBN: 978-981-99-8138-0
eBook Packages: Computer ScienceComputer Science (R0)