Big data time series forecasting based on pattern sequence similarity and its application to the electricity demand

Abstract

This work proposes a novel algorithm to forecast big data time series. Based on the well-established Pattern Sequence-based Forecasting algorithm, this new approach has two major contributions to the literature. First, the improvement of the original algorithm with respect to the accuracy of predictions, and second, its transformation into the big data context, having reached meaningful results in terms of scalability. The algorithm uses the Apache Spark distributed computation framework and it is a ready-to-use application with few parameters to adjust. Physical and cloud clusters have been used to carry out the experimentation, which consisted in applying the algorithm to real-world data from Uruguay electricity demand.

Introduction

Time series data available has increased considerably in the last decades, given the recent interest in storing and analyzing huge amount of data nowadays [6], as may be the case of datasets extracted from smart meters for decades, from hundreds of buildings or with a very high measurement frequency [24], [25].

Time series forecasting algorithms can create models based on historical data and make predictions for given target variables of interest [44], [43]. The computation time of these algorithms may increase notably when big data time series are evaluated. Therefore, single-core machine environments are not enough and need to be improved with additional computational resources. For such reason, it becomes necessary to distribute the data and its computation across multiple nodes using a cluster of machines.

The Pattern Sequence based Forecasting algorithm (PSF) [21] is an effective general-purpose multi-output approach particularly designed to deal with time series for prediction horizons of an arbitrary length. This multi-output feature has turned PSF into a flexible tool that has been used in different fields of research. PSF is mainly based on the identification of certain patterns that are searched for throughout the whole dataset. Such patterns are calculated by means of any clustering technique (k-means in the original work) and, later, sequences of clusters are formed to characterize the target time series.

In this work, a new algorithm, hereinafter called bigPSF, is proposed. It is inspired by the pattern search strategy introduced in the original PSF algorithm. But bigPSF has two major contributions to the literature. First, it is scalable, thanks to its distributed computation under the Apache Spark framework. That is, bigPSF is suitable for handling big data time series and mining millions of records reporting reduced execution times in contrast with PSF. Second, some modifications have been carried out in bigPSF with respect to the original PSF improving the results of the predictions and achieving higher accuracy. Hence, the algorithm mainly covers volume and velocity dimensions from the well-established 4-Vs big data paradigm [6].

Although bigPSF is also a general-purpose algorithm, data related to electricity demand have been used to assess its performance. A study case of the original methodology along with its last methodological improvement for predicting electricity demand data can be found in Ref. [19]. Compared to PSF and other five well-known prediction algorithms, bigPSF achieved a higher accuracy than that of each of them and was able to deal with big data, exhibiting a linear behavior in terms of computation time.

The rest of the paper is structured as follows. Section 2 reviews the relevant and related papers to PSF and big data time series forecasting. Section 3 describes the proposed methodology. Section 4 reports all results and discusses the performance in terms of both errors and scalability. Finally, Section 5 summarizes the most significant achievements within the manuscript.