Analyzing relationships among ARMA processes based on non-Gaussianity of external influences
Introduction
The problem of finding a graphical representation of data generating systems has received much attention in various fields of research and covers a broad range of applications, such as bioinformatics [1], neuroinformatics [2], [3] and data mining [4].
Under the assumption that data is independent and identically generated, this problem has been discussed as structural learning of graphical models in machine learning and is known to be closely related with causal analysis [5], [6]. On the other hand, the most popular method to learn a structure of a model for time-series data is to estimate a relationship among variables through the identification of parameters in a classical autoregressive model, and the relationship is often called the Granger causality in economics [7]. Because this modeling is usually performed through classical regression methods by using covariance information only, the identifiability of the model structure is not ensured in general, as well as most conventional methods for causal analysis or structure learning based on Bayesian networks. Accordingly, a unique and probable structure cannot be estimated for a set of observed data without prior knowledge about the structure.
It has been recently reported that causal analysis based on the non-Gaussian structure of data overcomes the identifiability problem [8], [9]. If the external influences are non-Gaussian, the graphical structure can be uniquely estimated by using observed data only without any prior knowledge (under an assumption of acyclicity). In this paper, we propose a systematic way to estimate a relationship among variables with identifiability, based on the non-Gaussianity and an autoregressive moving-average (ARMA) model. This approach does not suffer from the identifiability problem as well as the original method using the non-Gaussianity. Our model is the combination of an ARMA model and a classical structural-equation model (SEM), and is estimated through the analysis on the non-Gaussianity of the residual processes that is obtained by estimating an ARMA model alone using data.
Another main motivation of our approach is the analysis of data generating processes involving slow and fast interactions. Since we may set data acquisition conditions with little consideration about the interaction-speed among processes (for example, because of the limitations of experimental facilities or because the data was gathered previously), there would exist both slower and faster interactions than the time-resolution of the measurements. The interaction faster than the time-resolution would be observed as an instantaneous interaction. However, conventional approaches based on neither non-Gaussianity nor time-series information cannot identify the model structure reflecting the both types of interactions simultaneously in principle. In this paper, we demonstrate analyses on the data containing both of the instantaneous causality and the Granger (temporal) causality [7], [10] by using our proposed method where the datasets for the demonstration cover both artificial and real physical systems.
Recently, Hyvärinen et al., have proposed a model taking both instantaneous and lagged effects into account with an AR model [11]. However, although an AR model is useful to estimate apparent effects or to construct a prediction model, in principle, it cannot express relationships among variables directly reflecting background processes. That is, an ARMA model is an exact representation of linear differential equations in discrete time-domain, which a standard tool to analyze the dynamics of a variety kind of physical processes. On the other hand, as is well known, an AR model is an asymptotic expansion of an ARMA model. Thus, in an AR model, it is difficult to estimate a physically valid direct dependency among observed variables because the relationship among independent exogenous variables have been convolved in the one among observed variables.
In physics and engineering domains, the definition of temporal causality among variables has not been clearly defined, however, in economics, the Granger causality [7] has become a basic concept for the discussion.1 In this paper, we address that the estimated structure by our approach become equivalent to the one in the sense of the Granger causality in a special case, and thus our framework leads to a generalization of this concept.
The reminder of this paper is organized as follows. In Section 2, we describe two types of effects, i.e., instantaneous and lagged effects, in data generating processes, make the motivation of this paper clear and derive the model discussed. In Section 3, the specific steps for estimating the model is described. Then, in Section 4, we discuss the relation of our model to the Granger causality. Finally, we show some empirical results using artificial data in Section 5 and real data in Section 6, and conclude this paper in Section 7.
Section snippets
Analyzing processes with instantaneous and lagged effects
Let yi(t) (, ) be multiple observed time series, where i and t are the variable and time indices, respectively. The collection of all the variables will be referred as . In this paper, the process is assumed to be stationary, which is a typical assumption in time-series analysis [12], [13].
In this section, we introduce the model for analyzing a relationship among variables and state the problem addressed in the paper.
Estimation of the ARMA-LiNGAM model
We present our novel part of the steps for estimating the ARMA-LiNGAM model (2) and, at the same time, a relationship among variables in data generating processes. Our method is based on the recently proposed analysis to identify the structure of instantaneous influences by using the non-Gaussianity of external influences (the LiNGAM analysis). In this section, we first briefly review the LiNGAM analysis in Section 3.1 and then describe the estimation procedure in Section 3.2. Finally, the
Relation to the Granger causality
The problem described in this paper has close relation to causal analysis, which has been actively discussed in machine learning [5], [6]. In physics and engineering domains, the definition of causality among variables has not been well formalized. On the other hand, in economics, the Granger causality has become a basic concept for the discussion. However, we should note that the Granger causality is not necessarily a natural extension of the causality based on the counter-factual model. In
Simulations
In this section, we illustrate the presented framework using artificial data. Especially, we focus on showing the difference between modeling with a classical ARMA model, an AR model combining a LiNGAM (AR-LiNGAM) and our model (2) (ARMA-LiNGAM).
For this purpose, in the experiments, we created data in the following manner, partly using the LiNGAM code package7:
- 1.
First, a strictly lower-triangular matrix for the instantaneous effects, i.e., , was
Application to real data
We analyzed the real dataset created from the physical system, the duplex-pendulum system illustrated in Fig. 4, using the proposed method. The data was created by filming the real system with a high-speed video camera and then reading out the position using an image analysis software, and consists of four dimensional signals (angles; , , and angular speeds; , ). See Fig. 4 for the definitions of and . This system could be assumed to have no unobserved confounder
Conclusions
In this paper, we proposed an approach for estimating a relationship among variables in data generating processes, based on the non-Gaussianity of external influences and an autoregressive moving-average (ARMA) model. Our model consists of the parts for modeling instantaneous and lagged effects in processes, i.e., a classical structural-equation model and an ARMA model, and is estimated through the analysis using the non-Gaussianity on the residual processes. We addressed that the estimation
Acknowledgments
We are very grateful to Hiroshi Hasegawa, the college of science, Ibaraki University, for helpful discussion and providing the physical data. This research was supported in part by the JST PRESTO program “Synthesis of Knowledge for Information Oriented Society”, JSPS Global COE program “Computationism as a Foundation for the Sciences” and the Grant-in-Aid (22700147, 21700302, 19200013) from the Ministry of Education, Culture, Sports, Science and Technology.
Yoshinobu Kawahara received his Bachelor's, Master's and PhD degrees in Engineering from the University of Tokyo, Japan, in 2003, 2005 and 2008, respectively. He currently holds an Assistant Professor position at the Institute of Scientific and Industrial Researches (ISIR), Osaka University. He also serves as a Research Fellow of PRESTO (Precursory Research for Embryonic Science and Technology) by JST (Japan Science and Technology Agency). His primary research interests lie in combinatorial
References (23)
- et al.
Mapping directed influence over the brain using Granger causality and fMRI
NeuroImage
(2005) - et al.
Estimation of linear non-Gaussian acyclic models for latent factors
Neurocomputing
(2009) - et al.
From correlation to causation networks: a simple approximate learning algorithm and its application to high-dimensional plant gene expression data
BMC Systems Biology
(2007) - et al.
Unified structural equation modeling approach for the analysis of multisubject multivariate functional MRI data
Human Brain Mapping
(2007) - T. Ide, A.C. Lozano, N. Abe, Y. Liu, Proximity-based anomaly detection using sparse structure learning, in: Proceedings...
- et al.
Causality, Prediction and Search
(1993) Causality: Models, Reasoning and Inference
(2000)Investigating causal relations by econometric models and cross-spectral methods
Econometrica
(1969)- et al.
A linear non-Gaussian acyclic model for causal discovery
Journal of Machine Learning Research
(2006) - S. Shimizu, A. Hyvärinen, Y. Kawahara, T. Washio, A direct method for estimating a causal ordering in a linear...
Testing causality between two vectors in multivariate autoregressive moving average models
Journal of the American Statistical Association
Cited by (17)
A survey of causal discovery based on functional causal model
2024, Engineering Applications of Artificial IntelligenceHouse price forecasting with neural networks
2021, Intelligent Systems with ApplicationsCitation Excerpt :In econometrics, much effort has been devoted to accurate and robust forecasting of economic and financial time series. Basic models, such as the autoregressive, vector autoregressive, and vector error correction approaches, as well as their numerous variations, have been widely employed to produce forecasts for different uses and purposes that are summarized in Fig. 1 (e.g. Cabrera et al., 2011; Hyvärinen et al., 2010; Kawahara et al., 2011; Kim et al., 2007; Kouwenberg and Zwinkels, 2014; Liu and Wu, 2020; Milunovich, 2020; Shimizu et al., 2006; Webb et al., 2016; Wei and Cao, 2017; Xu, 2014a; 2015a; 2017a; 2017b; 2017c; 2018b; 2018e; 2019a; 2019c; 2020; Xu and Thurman, 2015a; Xu and Zhang, 2021b; Yang et al., 2018; Zohrabyan et al., 2008). Recently, machine learning models and algorithms, including the support vector regression, regression tree, random forecast, bagging, boosting, and ensemble learning, are found to be useful and attractive solutions to a wide variety of forecasting problems, including those related to house prices, which are summarized in Fig. 1 (e.g. Chen et al., 2017; Embaye et al., 2021; Fu, 2018; Gu et al., 2011; Ho et al., 2021; Huang, 2019; Li et al., 2009; 2020; Liu and Liu, 2019; Milunovich, 2020; Pai and Wang, 2020; Park and Bae, 2015; Plakandaras et al., 2015; Rafiei and Adeli, 2016; Rico-Juan and de La Paz, 2021; Shahhosseini et al., 2019; Wang et al., 2014; Wu et al., 2009; Xu and Li, 2021; Xu and Zhang, 2021c; Yan and Zong, 2020; Yu et al., 2018).
Office property price index forecasting using neural networks
2024, Journal of Financial Management of Property and ConstructionRent index forecasting through neural networks
2022, Journal of Economic StudiesResidential housing price index forecasting via neural networks
2022, Neural Computing and Applications
Yoshinobu Kawahara received his Bachelor's, Master's and PhD degrees in Engineering from the University of Tokyo, Japan, in 2003, 2005 and 2008, respectively. He currently holds an Assistant Professor position at the Institute of Scientific and Industrial Researches (ISIR), Osaka University. He also serves as a Research Fellow of PRESTO (Precursory Research for Embryonic Science and Technology) by JST (Japan Science and Technology Agency). His primary research interests lie in combinatorial methods for machine learning and statistical modeling for time-series data.
Shohei Shimizu is an Assistant Professor in the Institute of Scientific and Industrial Research, Osaka University, Japan. He received his Bachelor's and Master's in Human Sciences (Psychometrics) and PhD in Engineering (Statistical Science) from Osaka University in 2001, 2003, 2006, respectively. His research interests include statistical methodologies for learning data generating processes such as structural equation modeling and independent component analysis.
Takashi Washio received the PhD degree in nuclear engineering from Tohoku University, Japan, in 1988, on the topic of process plant diagnosis based on qualitative reasoning. He is a professor in the Institute of Scientific and Industrial Research (ISIR), Osaka University. At ISIR, he works on the study of scientific discovery, graph mining, and high-dimensional data mining. He is a member of the Association for the Advancement of Artificial Intelligence (AAAI) and the IEEE Computer Society.