Neural String Diagrams: A Universal Modelling Language for Categorical Deep Learning

Abstract

Category theory has been successfully applied to compositional modelling of diverse systems, including computational systems, logical systems, and physical systems, paving the way for a new kind of general system science (or rather general process science). It has been particularly successful in quantum computing and natural language processing recently; traditionally, it has played major rôles in compositional semantics of programming languages and symbolic reasoning systems. Building upon them, we propose the mathematical system of neural string diagrams as a universal modelling language for categorical deep learning, which allows us to turn informal neural network architecture pictures into formally explainable, mathematically verifiable, and systematically composable entities in their own right. We give, in particular, a neural string diagram account of CNN and Transformer (which has never been achieved before). Neural string diagrams can be computed with DisCoPy, Quantomatic, and their extensions. Categorically formalised neural networks can be instantiated for both ordinary vector spaces and other monoidal categorical structures, allowing for generalisations of deep learning (e.g., deep learning on relational structures, deep learning on graph and other network structures, etc.). We envisage that the category theory approach to artificial intelligence ultimately contributes to the development of artificial general intelligence, giving a universal modelling language for intelligent systems and agents.

This work was supported by JST (JPMJMS2033; JPMJPR17G9).