Equations of mind: Data science for inferring nonlinear dynamics of socio-cognitive systems
Introduction
Differential equations define the time evolution of a dynamical system. Their precision inspires some to see such mathematical formulation as critical to scientific understanding. This perspective on differential equations found prominent expression in the dynamical systems approach to cognition of the 1990s (Port and Van Gelder, 1995, Van Gelder, 1995), and was the subject of vigorous debate (Bechtel, 1998, Eliasmith, 1996): “Dynamical systems governed by differential equations are a particularly interesting and important subcategory, not least because of their central role in the history of science.” (Van Gelder, 1995, p. 368) Simon (1992) famously expressed an even stronger position, arguing that cognitive explanation is founded on “difference equations” which characterize much cognitive systems research still:
“For systems that change through time, explanation takes the form of laws acting on the current state of the system to produce a new state – endlessly. Such explanations can be formalized with differential or difference equations. A properly programmed computer can be used to explain the behavior of the dynamic system that it simulates. Theories can be stated as computer programs.” (Simon, 1992, p. 160)
Nowadays this mode of mathematical description and explanation permanently inhabits many realms of cognitive science.1 It was well established even before this recent debate. From the firing of single nerve cells (Hodgkin & Huxley, 1952) and the control of an entire physical body (Beek et al., 1992, Kugler et al., 1980) to multi-agent models (Richardson et al., 2016), systems of differential equations have long captured a wide variety of psychological phenomena. When we have a set of differential equations for a system, we can predict its time evolution, understand its controlling variables, and identify how system variables interact. These dynamic equations can also participate with other forms of cognitive explanation, such as mechanistic explanations of how a cognitive architecture is composed of various particular parts and their interactions (Kaplan & Bechtel, 2011).
Despite their power, differential equations are not always easy to identify. Identification of governing equations can involve an interacting cycle of mathematical invention and empirical tinkering. Guided by intuition, a scientist can happen upon a formulation that generates a covering law (Hempel, 1966). Consequences of this covering law can be explored to consider other formulae in other domains of application. The literature on this is deep and colorful, and excellent reviews of the philosophy and history of science abound (Brush, 1974, Hempel, 1966, Hirsch, 1984, Kuhn, 1962).
Cognitive scientists continue to study and model this psychological process of identifying scientific generalizations and natural law (Addis et al., 2016, Klahr and Simon, 1999, Langley, 1987). A complementary approach, made possible by computational tools of the day, is to use data and algorithms together to automatically recover dynamical laws. This is what we consider here in this paper. There is an emerging domain, growing rapidly with the advent of data science and machine learning, to precisely recover differential equations from raw data. This offers considerable potential to researchers interested in the dynamics of socio-cognitive systems. It may be possible to use these tools for new and explicit descriptions of system dynamics, even when the data are noisy, and especially when there are plenty of data to be found (a common circumstance these days: Paxton & Griffiths, 2017).
There has been considerable prior work on equation discovery. Motivated by the same points we raise above, researchers over the past two decades have explored different frameworks for automatic recovery of governing equations. Below we first briefly review this past work through influential examples. After this, we introduce a recent simple and elegant formulation of equation discovery (SINDy; Brunton, Proctor, & Kutz, 2016). Based only on transformation of time series data, and simple sparse regression, a researcher can recover equations for their measured systems. In some simple cases, these equations may reflect a full reconstruction of a system’s underlying dynamics. More complex cases present other challenges, but in these more complex situations SINDy may still be useful. Below, we introduce SINDy and then showcase how it works on a number of example systems. We also outline its key limitations. After this, we summarize a few outstanding issues in these domains, including how SINDy and related methods could be expanded in the future to help recover governing equations of social systems.
There has been considerable prior work on equation discovery. Classic work in cognitive science itself can be found in Langley (1981), who used symbolic cognitive models to infer equations from data. His early model, BACON.3, is meant to capture some important aspects of human scientific activity. More recently, Langley and colleagues (Langley, Sanchez, Todorovski, & Dzeroski, 2002) have also used time series data in an Inductive Process Modeler that can fix certain parameters on population dynamics models. These general approaches fall under the rubric of symbolic machine learning, as a kind of heuristic search. For example, process models of biological systems can include a space of parameters that describe the relationship among variables (Džeroski & Todorovski, 2008). A heuristic search navigates this parameter space under certain constraints to best fit a set of data.
Crutchfield, Shalizi, and others have developed a hidden Markov approach that generates a directed graph that represents a theory of a system from a time series of its behavior (Crutchfield, 1994, Crutchfield, 2011, Shalizi and Crutchfield, 2001, Shalizi and Shalizi, 2004). This framework finds transitions between system states in coarse-grained representation of the time series. The result is a kind of compact theory which can describe the time evolution of the system. It also provides descriptive measures of the system, such as its computational complexity. This modeling framework can be used to simulate the relationship between measurement level and theory, and can be likened to a cognitive agent seeking to explain and model a system’s dynamics (Crutchfield, 1994, Dale and Vinson, 2013).
There are many related techniques, both in cognitive science and in other realms of the physical sciences. An excellent review can be found in Sozou, Lane, Addis, and Gobet (2017). Much work used clever analysis of time series with assumed form of laws to recover particular systems (Bezruchko et al., 2001, Bünner et al., 1997, Crutchfield and McNamara, 1987, Smith, 1992).
With the advent of large matrix libraries, advanced regression methods are now possible. Schmidt and Lipson (2009) use symbolic regression and motion tracking of physical systems to derive various equations of motion. Example systems included chaotic systems, such as double pendula. Their approach involves extraction of motion time series, and then seeking invariances (correlation structure) among the measured variables according to candidate symbolic functions. The symbolic functions are found via a search through a space of candidates, generated randomly and gradually winnowed down based on best fit (see their Fig. 2). This method is closely related to the one we showcase below, with the primary difference that in SINDy candidate functions are defined comprehensively as a search through all possible functions defined by a set of features of interest to the researcher. Modeling more complex systems, Pikovsky has shown how time series of measurements from a neural network can be used to reconstruct the neural network itself (Pikovsky, 2016). Pikovsky’s method can reconstruct a connection matrix using time-difference neuron states through solving for a linear system with singular value decomposition (similar to the regression-based method used here).
In many of the examples reviewed here, researchers estimate derivatives numerically. This differencing is key in these approaches (and the one we illustrate below). Recent research has sought to overcome limitations in differencing raw data. For a given signal (e.g., a noisy time series), one will typically find that differentiation amplifies noise while integration filters noise out. Chen, Shojaie, and Witten (2017) have shown how to learn dynamical systems without using numerical differencing or differentiation. In their work, they use the time-integrated or integral equation form of the dynamical system.
Equation discovery seeks to find a dynamical system that best fits a given data set. Each dynamical system is specified by one or more functions – the space of all such functions is typically infinite-dimensional. As in many other nonparametric problems, this leads to a model selection problem. As we increase the dimensionality of the space over which we search for a best-fitting dynamical system, we will decrease training error. However, this fit to training data comes at the expense of generalization to new data. Using techniques from compressed sensing, non-convex optimization, and the statistics of chaotic systems, recent work has investigated conditions under which equation discovery techniques converge to the correct underlying dynamical system (Tran and Ward, 2017, Zhang and Schaeffer, 2018). A recent approach also seeks to find lower-order models of network dynamics by using Bayesian model comparison (Daniels & Nemenman, 2015). These papers reflect an exciting new direction of this work. They will help refine the selection of models among many that may be formulated for a given set of complex data.
Some recent research in cognitive science is inspired by this data-driven reconstruction of lawful regularities. Using first-principle Newtonian mechanics, a “mental landscape” can also be reconstructed via behavioral data (O’Hora et al., 2013, Zgonnikov et al., 2017). In this approach, researchers collected a series of computer mouse trajectories towards two possible decisions, at the top left or top right of a computer screen. These computer-mouse data are represented as -coordinates, starting from a set of fixed coordinates (). Each time series is a decision, with the mouse moving to one final decision point on the left () or right () on the computer screen. O’Hora et al. (2013) and Zgonnikov et al. (2017) treat these movements as a kind of “descent” into an attractor on an uneven surface. These attractors model a decision as starting from the peak of a hill, and falling into one of two valleys. Assuming a set of equations with the form of Newtonian mechanics, these decision surfaces can be estimated from these time series data.
Many statistical approaches to model and explore complex data are related to these techniques. For example, the large and still growing application of structural equation modeling (SEM) by social scientists is fundamentally about both exploring and confirming theoretical hypotheses from complex response data (Keith, 2005). SEM models tend to be structural, rather than dynamic, in nature. However, many other still common quantitative methods are closely related to our goals. The notion of a model as a scientific explanation of some phenomenon cannot be neatly distinguished from general statistical practices (Stigler, 2016, Chap. 6). In signal processing and statistical modeling, for example, methods such as Kalman filters, time series regression modeling, and other applications of hidden Markov approaches, offer a rich array of choices (for brief reviews see Brockwell, 2014, Rydén, 2015).
It should therefore be emphasized that many statistical modeling techniques have relevance to understanding underlying relationships. What is unique about this recent trend in data science is to (i) find methods that have some relative transparency of output, (ii) relate output to low-dimensional lawful regularities, which express (iii) dynamical equations that govern a system’s behavior. Surely HMMs and other techniques can be placed under this designation. But the synergy among (i)–(iii) reflects a distinct trend. Our brief review of this trend shows a long-standing interest in techniques that have these properties. Recently, rather extensible out-of-the-box methods are now available, and these may increase accessibility to researchers in many areas, such as the social and cognitive sciences. Indeed, with the emergence of machine learning techniques for training models on very large datasets over very large feature sets, it is now possible to fit models with few assumptions about their form. We use a recent example based on this “data science” approach to recovering nonlinear dynamics.
Section snippets
Present study
An emerging approach to model building in the computational and social sciences is to exploit the ready availability of high-density measurements and machine learning algorithms to estimate models. We describe one of these techniques, showcase it on simple and known systems, and then develop ideas for how it might be expanded to raw data. Importantly, we offer full source code in R, along with simulations, that can reconstruct these demonstrations, and serve as a foundation for further methods
Demonstration of SINDy in known systems
We first revisit two simple examples from Brunton et al. (2016). We show that the classic logistic map can be fit with SINDy. By sampling a subset of values of its single control parameter, SINDy recovers a close approximation of the closed-form update equation. We then showcase something similar with the Lorenz system. Sampling its three system variables under parameter settings that generate chaotic behavior, SINDy can accurately recover the differential equations, including nonlinear terms,
Extending related methods, and other issues
But what if we don’t know the governing equations underlying some data? In fact, what if it seems reasonable to suspect that the equations won’t be simple, and perhaps not even stable, over a period of time? In addition, what are the impacts of noise? Brunton et al. (2016) showed that in these model systems, introduction of small amounts of noise does not disrupt equation discovery. In our own explorations of SINDy, we find that noise at higher levels – levels perhaps reasonable for data in
Conclusion
There are many emerging approaches to “computational scientific methods,” using machine learning or other techniques to recover equations from data (Sozou et al., 2017). The model we showcase here has certain desirable properties. It is extremely simple to deploy. It requires only a handful of adjustments to raw data, and the process of finding coefficients does not require high-performance computing resources. With these in hand, a high-dimensional space of features, obtained from raw data
Acknowledgments
HSB acknowledges support from the National Science Foundation award DMS-1723272.
References (77)
The calculi of emergence: computation, dynamics and induction
Physica D: Nonlinear Phenomena
(1994)- et al.
On the spontaneous discovery of a mathematical relation during problem solving
Cognitive Science
(2004) - et al.
The self-organization of cognitive structure
- et al.
Perspective-taking in dialogue as self-organization under social constraints
New Ideas in Psychology
(2014) - et al.
Equation discovery for systems biology: finding the structure and dynamics of biological networks from time course data
Current Opinion in Biotechnology
(2008) - et al.
On the concept of coordinative structures as dissipative structures: I. Theoretical lines of convergence
Data-driven discovery of physical laws
Cognitive Science
(1981)The relative judgment theory of two choice response time
Journal of Mathematical Psychology
(1975)Identification and prediction of low dimensional dynamics
Physica D: Nonlinear Phenomena
(1992)Discovery in complex adaptive systems
Cognitive Systems Research
(2018)
A non-representational approach to imagined action
Cognitive Science
Computational scientific discovery and cognitive science theories
Representations and cognitive explanations: Assessing the dynamicist’s challenge in cognitive science
Cognitive Science
Autonomous and nonautonomous dynamics of coordinated rhythmic movements
Ecological Psychology
Reconstruction of time-delay systems from chaotic time series
Physical Review E
Bayesian inference of stochastic pursuit models from basketball tracking data
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Proceedings of the National Academy of Sciences
Should the History of Science Be Rated X?: The way scientists behave (according to historians) might not be a good model for students
Science
A nonlinear dynamic model of social interaction
Communication Research
Recovery of the time-evolution equation of time-delay systems from time series
Physical Review E
Numerical differentiation of noisy, nonsmooth data
ISRN Applied Mathematics
Radical embodied cognitive science
Network reconstruction from high-dimensional ordinary differential equations
Journal of the American Statistical Association
Between order and chaos
Nature Physics
Equation of motion from a data series
Complex Systems
The observer’s observer’s paradox
Journal of Experimental & Theoretical Artificial Intelligence
Nominal cross recurrence as a generalized lag sequential analysis for behavioral streams
International Journal of Bifurcation and Chaos
Automated adaptive inference of phenomenological dynamical models
Nature Communications
The third contender: A critical examination of the dynamicist theory of cognition
Philosophical Psychology
Radical embodied cognitive neuroscience: addressing grand challenges of the mind sciences
Frontiers in Human Neuroscience
The animal-environment system
Perceptual and Emotional Embodiment: Foundations of Embodied Cognition
Order parameter dynamics of body-scaled hysteresis and mode transitions in grasping behavior
Journal of Biological Physics
A theoretical model of phase transitions in human hand movements
Biological Cybernetics
The dynamical systems approach to differential equations
Bulletin of the American Mathematical Society
Cited by (16)
Multimodal coordination and pragmatic modes in conversation
2023, Language SciencesData-driven automated discovery of variational laws hidden in physical systems
2020, Journal of the Mechanics and Physics of SolidsCitation Excerpt :In this way, the presented data-driven method can be extended to new fields with little prior knowledge. In fact, such new fields may not necessarily be physical fields; they can be fields in the social sciences (Dale and Bhat, 2018), since the concerned motions (e.g., the response of a specific population to an event) are often time-dependent. In summary, the presented data-driven method in variational form possesses two intrinsic advantages, namely, a reduced data requirement and robustness to measurement noise, which enable its applicability to the discovery of governing equations in real time for physical fields or more complicated social sciences, with or without prior knowledge.
Editor's introduction: Innovative dynamical approaches to cognitive systems
2019, Cognitive Systems ResearchCitation Excerpt :In this way, governing equations provide models that allow for prediction and understanding of systems, especially complex dynamic systems. In “Equations of mind: Data science for inferring nonlinear dynamics of socio-cognitive systems,” Dale and Bhat (2018) introduce a recent data science methodology to the investigation of cognitive systems: sparse identification of nonlinear dynamics (SINDy; Brunton, Proctor, & Kutz, 2016). SINDy provides a method for inferring differential equations directly from data.
Automatedly Distilling Canonical Equations From Random State Data
2023, Journal of Applied Mechanics, Transactions ASMEBenchmarking sparse system identification with low-dimensional chaos
2023, Nonlinear Dynamics