Hybrid change point detection for time series via support vector regression and CUSUM method

doi:10.1016/j.asoc.2020.106101

Applied Soft Computing

Volume 89, April 2020, 106101

https://doi.org/10.1016/j.asoc.2020.106101 Get rights and content

Highlights

•
We consider the change point detection problem for ARMA type time series.
•
To detect a change point, we use a hybrid of the SVR and CUSUM methods.
•
We calculate the forecasting errors based on the SVR method.
•
The SVR errors are used in the construction of the LSCUSUM test.
•
Monte Carlo simulations and a real example is provided for illustration.

Abstract

This study considers the change point testing problem regarding time series based on the location and scale-based cumulative sum (LSCUSUM) test constructed with the residuals obtained from support vector regression (SVR)-autoregressive moving average (ARMA) models. For this, we first estimate the model parameters in SVR–ARMA models from a training time series sample, in which a long AR model is fitted to the data to obtain residuals. We then use these as initial values of the error terms in SVR–ARMA ( $p, q$ ) models and obtain the forecasting values recursively until the updated error terms converge to a certain limit. Finally, we select an optimal order of $p, q$ with the root mean square error (RMSE) and use the forecasting errors from this selected model as the residuals for constructing the LSCUSUM test. Monte Carlo simulations are performed to evaluate the validity of the test. A real data example is provided for illustration.

Introduction

In this study, we consider the change point testing problem for time series based on the location and scale-based cumulative sum (LSCUSUM) test constructed with the support vector regression (SVR)-based autoregressive moving average (ARMA) models. Since [1], the problem of testing for a parameter change has been an important issue in economics, engineering, and medicine, and a multitude of articles have been published in various research areas [2]. Because time series often suffer from structural changes owing to changes in policy and critical social events, the change point test has been viewed as a core issue in this context for several decades. The CUSUM test has been quite popular among many change point tests due to its ease of use and abundant articles exist regarding the CUSUM test for time series. For earlier works, we refer to [3], [4], [5], [6] and the papers cited therein, and further, for recent developments, see [7], [8], [9], [10], [11], [12], [13], [14], [15] who consider several different types of CUSUM tests and compare their performance.

The conventional estimate-based CUSUM test is designed to compare the discrepancy among sequentially obtained estimators [5]. This estimates-based test generally performs well but suffers from severe size distortions and produces low powers on some occasions, particularly when the underlying model is complicated and has many unknown parameters. Therefore, the residual-based CUSUM test for time series models has been advocated as a remedy [6], [7], [8]. However, the residual-based CUSUM test for location-scale models undergoes a severe power loss in location parameter changes. To overcome this drawback, [9] and [10] suggested using the score vector-based CUSUM test for ARMA-generalized autoregressive conditional heteroscedastic (GARCH) models. [11] also proposed a modified residual-based CUSUM test to lessen an effort to deal with the derivatives of the log-likelihood functions in constructing the test of [10] and enhanced the power performance. [12] further improved the test of [11] by introducing a lot handier location and scale-based CUSUM (LSCUSUM) test, demonstrating its validity for ARMA-type models. Because the LSCUSUM test is constructed only with observations and residuals, it has advantages over other CUSUM tests in terms of hybrid capability with other methods that can afford to calculate residuals. Motivated by this, we consider using the SVR–ARMA model in implementing the LSCUSUM test.

In the construction of the LSCUSUM test, an important step is to estimate the residuals accurately. That is, a correct time series prediction is crucial because the residuals are merely the prediction errors. Time series prediction is generally important to forecast the behavior of time series and detect malfunctions or anomalies in statistical process control. In the literature, the most popular time series forecast method is using the classical ARMA model. Conventional linear ARMA models yield an accurate prediction when a time series truly follows them. However, if the time series has significant nonlinear characteristics, the prediction result based on the ARMA models is incorrect and hard to harness for further applications. In this case, practitioners can employ nonparametric prediction methods such as a recurrent neural network (RNN) and support vector regression [16], [17]. The RNN method is well known to outperform the ARMA model in many situations, particularly when time series have, to certain extent, nonlinear and non-stationary features. However, it has some limitations such as the need for a large number of tuning parameters, difficulty in finding a unique global solution owing to a different choice of initial weights, and over-fitting [18]. In contrast, the SVR has flexibility, outstanding forecasting accuracy, and a balance between the training and generalization errors, resulting in better empirical performance than the RNN as well as ARMA models [19], [20]. It is well known that the SVR minimizes the structural risk and meets the Structural Risk Minimization Principle, while the RNN minimizes the empirical risk, namely, the error regarding the in-sample estimating data [21]. Motivated by this, we also adopt the SVR method for time series prediction, and based on the obtained ARMA residuals, we construct the LSCUSUM test to test for change points. See [22] for a reference concerning the SVR–ARMA method.

The rest of this paper is organized as follows. Section 2 introduces the LSCUSUM test in the classical ARMA model and outlines its basic principle. Section 3 proposes a forecasting method based on the SVR–ARMA model and describes how to determine an optimal SVR–ARMA model. The residuals are obtained through an optimal SVR–ARMA model to a given training time series sample, which is split into two subseries. A long AR model is fitted to the first subseries to obtain the initial values of residuals, which are used as the error terms in the SVR–ARMA( $p, q$ ) model and are recursively updated until the obtained residuals converge to a certain limit. This procedure is applied to each $p, q$ less than a predetermined $K$ . Then, for each estimated SVR–ARMA( $p, q$ ) model, we calculate the root mean square errors (RMSEs) based on the second subseries and select an optimal ARMA order with the smallest RMSE. The determined SVR–ARMA( $p, q$ ) model is then applied to obtain the prediction errors or residuals, which are finally used to construct the LSCUSUM test. Section 4 performs Monte Carlo simulations to evaluate the LSCUSUM test’s validity for various time series models. Section 4 provides a real data example for illustration. Section 5 provides concluding remarks.

Section snippets

LSCUSUM test for ARMA models

To develop a CUSUM test in time series models, [11] considered the CUSUM test for the location-scale model of the form $y_{t} = g_{t} (μ) + \sqrt{h_{t} (θ)} η_{t}$ , where $g_{t} (μ)$ and $h_{t} (θ)$ are the conditional mean and variance with parameters $μ$ and $θ = {(μ^{T}, λ^{T})}^{T}$ , and $η_{t}$ are iid error terms with mean zero and unit variance. The location-scale model includes a broad class of autoregressive conditional heteroscedastic (ARCH) time series models, covering ARMA-generalized ARCH (GARCH) models. To implement a change point test, they

SVR model

Support vector regression (SVR) is a functional tool to approximate various types of functions and make accurate predictions for time series. SVR aims to identify a nonlinear function $f$ that approximates the output $y_{t}$ within a forecasting error based on given data ${(x_{t}, y_{t})}_{t = 1}^{n}$ , where $x_{t} \in R^{k}$ is a $k$ -dimensional input vector and $y_{t} \in R$ is a scalar output. More precisely, $f$ has the following form: $f (x_{t}) = w^{T} ϕ (x_{t}) + b,$ where $w$ and $b$ are regression parameter vectors and $ϕ (\cdot)$ is a known nonlinear function.

Prediction based on SVR-ARMA model

Suppose that a training sample $y_{1}, \dots, y_{n}, y_{n + 1}, \dots, y_{n + m}$ is given. Here, $y_{1}, \dots, y_{n}$ and $y_{n + 1}, \dots, y_{n + m}$ are also used as validation samples. We assume that the sample is generated from the ARMA model: $y_{t} = f (y_{t - 1}, \dots, y_{t - p}, ϵ_{t - 1}, \dots, ϵ_{t - q}) + ϵ_{t},$ where $f$ is an unknown function to be estimated, $p, q$ are nonnegative integers that should be properly determined, and $ϵ_{t}$ are iid random variables with zero mean and a finite variance. If the training sample is known to follow an SVR–ARMA( $p, q$ ) model with specific orders $p$ and $q$ ,

Simulation study

In this section, we evaluate the performance of the SVR-based LSCUSUM tests ${\hat{T}}_{n}^{L S}$ and ${\hat{T}}_{n}^{m a x}$ for ARMA, threshold ARMA, and time-varying AR models. Each simulation is conducted with a sample size of 500 at the nominal level of 0.050. The sizes and powers are calculated as the rejection number of the null of no changes out of 500 repetitions. Under alternatives, the change is assumed to occur in the middle of the testing sample. The SVR-based LSCUSUM tests are compared with the ARMA-based LSCUSUM

Real data analysis

In this section, we apply the SVR-based LSCUSUM method to the Nikkei daily 225 data. We analyze 100*log-returns of daily Nikkei225 prices from January 4 2010 to December 31 2018. We split the dataset into the training dataset from January 4 2010 to December 30 2014 and the testing dataset from July 1 2015 to December 31 2018. Fig. 2, Fig. 3 plot daily and weekly datasets and Fig. 4, Fig. 5 plot daily and weekly log-return datasets.

Fig. 6, Fig. 7 show that both autocorrelation function (ACF) and

Concluding remarks

In this study, we consider the SVR-based LSCUSUM test to detect a change point for time series. Our simulation study confirms the validity of our method and shows that the SVR-LSCUSUM test outperforms the ARMA-based LSCUSUM test when the underlying model is nonlinear. For illustration, a data analysis was conducted using a Nikkei225 dataset, which also supports the practicality of the SVR-based LSCUSUM test. We plan to extend our work to time series with high volatility in our future project.

CRediT authorship contribution statement

Sangyeol Lee: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing - original draft. Sangjo Lee: Data curation, Formal analysis, Methodology, Software, Validation, Visualization. Miteum Moon: Methodology, Software, Validation.

Declaration of Competing Interest

No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.asoc.2020.106101.

Acknowledgments

We would like to thank the Editor, an AE, and three anonymous referees for their valuable comments. This research is supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and future Planning (No. 2018R1A2A2A05019433).

References (32)

GombayE.
Change detection in autoregressive time series
J. Multivariate Anal.
(2008)
ChenC.W.S. et al.
Generalized autoregressive Poisson models for time series of counts
Comput. Statist. Data Anal.
(2016)
AtsalakisG.S. et al.
Surveying stock market forecasting techniques Part II: Soft computing methods
Expert Syst. Appl.
(2009)
TayF.E. et al.
Application of support vector machines in financial time series forecasting
Omega
(2001)
CavalcanteR.C. et al.
Computational intelligence and financial markets: A survey and future directions
Expert Syst. Appl.
(2016)
CherkasskyV. et al.
Practical selection of SVM parameters and noise estimation for SVM regression
Neural Netw.
(2004)
Broszkiewicz-SuwajE. et al.
On detecting and modeling periodic correlation in financial data
Physica A
(2004)
TangL. et al.
Nonparallel support vector regression model and its SMO-type solver
Neural Netw.
(2018)
PengX.
TSVR: an efficient twin support vector machine for regression
Neural Netw.
(2010)
PageE.S.
A test for a change in a parameter occurring at an unknown point
Biometrika
(1955)

CsörgőM. et al.

Limit Theorems in Change-Point Analysis

(1997)

InclánC. et al.

Use of cumulative sums of squares for retrospective detection of changes of variance

J. Amer. Statist. Assoc.

(1994)

KimS. et al.

On the CUSUM test for parameter changes in GARCH (1, 1) models

Comm. Statist. Theory Methods

(2000)

LeeS. et al.

On the CUSUM of squares test for variance change in nonstationary and nonparametric time series models

Ann. Inst. Statist. Math.

(2003)

LeeS. et al.

The CUSUM test for parameter change in regression models with ARCH errors

J. Japan Statist. Soc.

(2004)

NaO. et al.

Change point detection in SCOMDY models

AStA Adv. Stat. Anal.

(2013)

Cited by (24)

Comprehensive analysis of change-point dynamics detection in time series data: A review
2024, Expert Systems with Applications
In the ever-evolving field of time series analysis, detecting changes in patterns and dynamics is paramount for accurate forecasting and meaningful insights. This article thoroughly explores several algorithms for detecting and analyzing pattern changes in time series data. The exploration covers a broad spectrum of algorithms, further venturing into their categorization based on functional modalities and the ability to identify complex changes. Recognizing pattern changes in time series data holds pivotal importance as it aids in anticipating future trends, ensuring efficient resource allocation, and mitigating potential challenges. This research goes beyond a basic overview and conducts a thorough comparative analysis, highlighting each algorithm’s strengths, drawbacks, and computational complexities. This comparative approach provides practitioners and researchers with the necessary information to select the most suitable algorithm for their requirements. Additionally, this review provides insight into potential future research directions, proposing possible improvements and breakthroughs in the design and application of algorithms. This review also provides a runtime analysis of various pattern change detection algorithms, presenting an in-depth evaluation of the existing methodologies. It serves as a vital reference for individuals dealing with the dynamic nature of time series data.
A self-supervised contrastive change point detection method for industrial time series
2024, Engineering Applications of Artificial Intelligence
Manufacturing process monitoring is crucial to ensure production quality. This paper formulates the detection problem of abnormal changes in the manufacturing process as the change point detection (CPD) problem for the industrial temporal data. The premise of known data property and sufficient data annotations in existing CPD methods limits their application in the complex manufacturing process. Therefore, a self-supervised and non-parametric CPD method based on temporal trend-seasonal feature decomposition and contrastive learning (CoCPD) is proposed. CoCPD aims to solve CPD problem in an online manner. By bringing the representations of time series segments with similar properties in the feature space closer, our model can sensitively distinguish the change points that do not conform to either historical data distribution or temporal continuity. The proposed CoCPD is validated by a real-world body-in-white production case and compared with 10 state-of-the-art CPD methods. Overall, CoCPD achieves promising results by $P r e c i s i o n$ 70.6%, $R e c a l l$ 68.8%, and the mean absolute error ( $M A E$ ) 8.27. With the ability to rival the best offline baselines, CoCPD outperforms online baseline methods with improvements in $P r e c i s i o n$ , $R e c a l l$ and $M A E$ by 14.90%, 11.93% and 43.93%, respectively. Experiment results demonstrate that CoCPD can detect abnormal changes timely and accurately.
Reduced-dimensional skip-inception feature-aggregated classified proportional-integral-derivative for suppression of mixed-mode oscillations in hydropower units
2023, Electric Power Systems Research
The existing methods cannot effectively distinguish and suppress mixed-mode oscillations in hydro-dominated grid operation caused by different factors. This work proposes a reduced-dimensional skip-inception feature-aggregated classified proportional-integral-derivative to solve the imprecise identification and suppression of multiple oscillation modes. The reduced-dimensional skip-inception feature-aggregated network (RSFN) of the proposed controller classifies mixed-mode oscillations accurately. The RSFN introduces a skip connection on the modified Inception module to solve the problem of model degradation and accuracy reduction caused by increasing network depth. Meanwhile, the dimensionality reduction and feature aggregation of RSFN reduce the computation memory and improve the performance of the network. This work classifies different oscillation modes by the proposed network model and adopts appropriate governor parameters according to the classification results to suppress oscillations. The RSFN performs better than other network models and can accurately distinguish the oscillation modes.
Ensemble learning for demand forecast of After-Market spare parts to empower data-driven value chain and an empirical study
2023, Computers and Industrial Engineering
Demand forecast for spare parts in supply chains is essential for ensuring customer satisfaction while minimizing appropriate inventory. The after-market orders mainly depend on repair and maintenance that makes the present problem for demand forecast challenging owing to high variability in demand sizes and time intervals. It is critical to address market fluctuation for effective demand forecast to reduce the risks of oversupply and shortage for supply chain resilience. Intelligent data-driven technologies should be developed to promote value integration and value co-creation among supply chain partners for digital transformation. The shortening product life cycle and the reducing lot sizes of diverse products have increased the challenges of demand forecast and supply chain management. This study aims to classify the demand patterns and develop the corresponding models via stacking ensemble approach to improve the overall forecasting performance. This study develops an alarm system to monitor the performance of the proposed approach and a systematic mechanism for retraining the model to maintain the decision quality. An empirical study is conducted in a leading automotive after-market component manufacturer for validation in real settings. The results have shown the forecast errors and the total cost can be effectively reduced by the developed solution.
Data-driven quantification of public–private partnership experience levels under uncertainty with Bayesian hierarchical model
2021, Applied Soft Computing
Citation Excerpt :
It has been proved to be consistent in detecting the number of change points in a series of sectors, such as hazard rate and finance [22]. Moreover, different hybrid methods have been proposed by scholars to increase the estimation accuracy, such as a hybrid of support vector regression and CUSUM method [23], as well as a hybrid of support vector machine and fuzzy statistical clustering method [24]. However, in real-world applications, the uncertainty in data is unavoidable [25].
Public–private partnership (PPP) is increasingly encouraged to deliver public services in developing countries. Many studies have been conducted to identify factors that affect PPP contract failure. Although a country’s PPP experience is of great importance in controlling the contract failure rate, most of the current studies are based on a qualitative perspective. This research develops a data-driven approach to quantify countries’ PPP experience levels through the Bayesian hierarchical model with uncertainties considered. First, detailed data exploration and selection have been carried out to clean the data source. Second, the number of change points in the dataset is identified based on the binary segmentation method. Third, the Bayesian hierarchical model is developed to locate the positions of the change points, and different experience levels are divided based on the location of change points. Findings show that: (i) PPP experience level is widely varying depending on PPP sectors. Four experience levels are suggested for the energy sector, while five levels are found for the transportation sector, and water & sewerage sector, (ii) PPP experience level is dispersed around the world, for example, Latin America and Caribbean (LAC) and East Asia and Pacific (EAP) regions have higher PPP experience levels than other regions, (iii) a country may have various experience levels in different sectors, such as India, and (iv) the learning rate will decreases as more PPP projects are initiated. This research can contribute to (a) a novel approach that could detect the change points in PPP project experience, and (b) support investors in the decision making process, such as selecting the most appropriate investment direction, contributing to the development of PPP projects in developing countries.
On robust asymmetric Lagrangian ν-twin support vector regression using pinball loss function
2021, Applied Soft Computing
The main objective of twin support vector regression (TSVR) is to find the optimum regression function based on the $ε$ -insensitive up- and down-bound with equal influences on the regression function where all the data points have a different location above the up-bound points and below the down-bound points. However, the effects of all data points must be distinct based on their distribution in the regression function. Recently, asymmetric $ν$ -twin support vector regression (Asy- $ν$ -TSVR) is encouraged on the same subject but still, the present matrices in the mathematical formulation have faced the problem of semi-definite. In order to handle this problem effectively, a new regressor model named as robust asymmetric Lagrangian $ν$ -twin support vector regression using pinball loss function (URALTSVR) proposes as a pair of the unconstrained minimization problem to handle not only the noise sensitivity and instability of re-sampling but also consist positive definite matrices. Here, we suggest the proposed model URALTSVR in such a way where the pinball loss function is playing a vital role to control the fitting error inside the asymmetric tube. One of the advantages is that unlike TSVR and Asy- $ν$ -TSVR, it considers the concept of structural risk minimization principle through the inclusion of regularization term as well as change the one-norm of the vector of the slack variable by two-norm, which yields the dual problem to be strongly convex, stable and well-posed. Aforementioned, the proposed formulation has a continuous and piecewise quadratic problem that is solved by their gradients based iterative approaches. Specifically, we analyze the three implementations of URALTSVR with the baselines approaches support vector regression (SVR), TSVR and Asy- $ν$ -TSVR, which discard the dependencies to solve a pair of quadratic programming problem (QPP) for obtaining the unique global solution. Overall, SRALTSVR1 based on smooth approximation function performs outstanding for artificial and real-world datasets.

View all citing articles on Scopus

View full text

Hybrid change point detection for time series via support vector regression and CUSUM method

Highlights

Abstract

Introduction

Section snippets

LSCUSUM test for ARMA models

SVR model

Prediction based on SVR-ARMA model

Simulation study

Real data analysis

Concluding remarks

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgments

J. Multivariate Anal.

Comput. Statist. Data Anal.

Expert Syst. Appl.

Omega

Expert Syst. Appl.

Neural Netw.

Physica A

Neural Netw.

Neural Netw.

A test for a change in a parameter occurring at an unknown point

Biometrika

Limit Theorems in Change-Point Analysis

Use of cumulative sums of squares for retrospective detection of changes of variance

J. Amer. Statist. Assoc.

On the CUSUM test for parameter changes in GARCH (1, 1) models

Comm. Statist. Theory Methods

On the CUSUM of squares test for variance change in nonstationary and nonparametric time series models

Ann. Inst. Statist. Math.

The CUSUM test for parameter change in regression models with ARCH errors

J. Japan Statist. Soc.

Change point detection in SCOMDY models

AStA Adv. Stat. Anal.