Elsevier

Neurocomputing

Volume 179, 29 February 2016, Pages 246-263
Neurocomputing

A distributed spatial–temporal weighted model on MapReduce for short-term traffic flow forecasting

https://doi.org/10.1016/j.neucom.2015.12.013Get rights and content

Abstract

Accurate and timely traffic flow prediction is crucial to proactive traffic management and control in data-driven intelligent transportation systems (D2ITS), which has attracted great research interest in the last few years. In this paper, we propose a Spatial–Temporal Weighted K-Nearest Neighbor model, named STW-KNN, in a general MapReduce framework of distributed modeling on a Hadoop platform, to enhance the accuracy and efficiency of short-term traffic flow forecasting. More specifically, STW-KNN considers the spatial–temporal correlation and weight of traffic flow with trend adjustment features, to optimize the search mechanisms containing state vector, proximity measure, prediction function, and K selection. Furthermore, STW-KNN is implemented on a widely adopted Hadoop distributed computing platform with the MapReduce parallel processing paradigm, for parallel prediction of traffic flow in real time. Finally, with extensive experiments on real-world big taxi trajectory data, STW-KNN is compared with the state-of-the-art prediction models including conventional K-Nearest Neighbor (KNN), Artificial Neural Networks (ANNs), Naïve Bayes (NB), Random Forest (RF), and C4.5. The results demonstrate that the proposed model is superior to existing models on accuracy by decreasing the mean absolute percentage error (MAPE) value more than 11.59% only in time domain and even achieves 89.71% accuracy improvement with the MAPEs of between 3.34% and 6.00% in both space and time domains, and also significantly improves the efficiency and scalability of short-term traffic flow forecasting over existing approaches.

Introduction

It has witnessed the big data era [1] for transportation coming in recent years, and the prediction of traffic condition (e.g., traffic flow, travel time) has attracted great research interest for various D2ITS [2] applications such as advanced traffic management systems (ATMS), advanced traveler information systems (ATIS) and advanced public transportation systems (APTS). Timely and accurate traffic flow prediction plays an increasingly essential role in regional traffic management and control, which can provide design infrastructures, schedule interventions for government agencies, inform traffic conditions for travelers, and offer mobility services for passengers in real time. Moreover, it also assists road users to anticipate traffic congestion, save energy consumption, reduce environment pollution, and improve traffic operation efficiency [3], [4]. Owing to the heterogeneous and dynamic nature of traffic with nonlinear interactions between drivers and environments [5], traffic flow conditions are extremely uncertain in a transportation network. Furthermore, the traffic state of a specific location is highly affected by its upstream and downstream traffic conditions [6], [7], thus making it difficult to be accurately predicted, particularly for short-term traffic flow forecasting (STFF) [8]. However, improving predictive accuracy of short-term traffic flow would be vitally important in this context.

Clearly, short-term traffic flow prediction is a challenging issue, and thus many models and systems have been proposed for addressing this problem over the past several decades. Prominent short-term prediction approaches utilize different empirical and theoretical techniques [9], which can be broadly classified into three categories: (i) parametric techniques [10], [11], including historical average (HA) model, time-series models (e.g., moving average (MA), autoregressive MA (ARMA), autoregressive integrated MA (ARIMA), and seasonal ARIMA (SARIMA)), state space models (e.g., kalman filter (KF)), dynamic-traffic-assignment-based (DTA) models, smoothing methods, and so forth. (ii) Nonparametric techniques [12], [13], such as neural networks (NNs) [14], artificial neural networks (ANNs) [15], [16], k-nearest neighbor (KNN) [17], [18], support vector regression (SVR) [19], Bayesian [20]. (iii) Hybrid integration techniques [21], [22], combining the parametric technique(s) and the nonparametric technique(s) (e.g., graphical lasso and NNs [23], SARIMA and SVM [24], seasonal SVR with chaotic simulation [25], statistical and NNs models [26]). Certainly, these approaches can also be divided into statistical-based methods and computational intelligence (CI)-based methods, and their similarities and differences have been discussed in [10], [16], [27], [28]. The aforementioned famous studies focus on making forecasts about the likely traffic flows changes in the short-term, typically within 5–10 min ahead [29], heavily depending on historical and real-time traffic flow data which are treated as time series data. The traffic condition data are often captured through presence-type detectors (or a set of on-road sensors) deployed at a single location in consistent intervals, and then the collected data are usually used as inputs to a prediction model for forecasting short-term traffic parameters. Thus, most of existing approaches can only process regularly spaced time series data.

In recent years, given that it is totally impractical to equip each lane with presence-type detectors in a town [30] or to cover the whole urban network for traffic state estimation by traffic camera [31], some alternative ways of generating information on traffic conditions have been developed with emerging floating car data (FCD) technologies, such as floating cars that upload global positioning system (GPS) data. Hence, the data collected based on these approaches are irregularly spaced time series data, which are unfit for most of existing methodologies of time series traffic flow forecasting [30]. Moreover, for big GPS data, the computational and storage demands for traffic flow prediction would be prohibitive using existing approaches, and the determination of the amount of past traffic flow conditions used as input patterns of the models is very much time-consuming based on a trial and error method [29]. In particular, taxicabs are commonly equipped with GPS sensors for dispatching and safety in most modern cities, and thus a large number of GPS trajectories of taxicabs with their present location, geo-position, time stamp and occupancy information are being generated to a data center in a certain frequency per day [32]. For example, we employ large-scale GPS trajectory data used in this paper, to form the road network of Beijing (see Fig. 1) which is nearly consistent with the real traffic map, based on a Hadoop platform with ArcGIS. Fig. 1 shows the density distribution of the GPS points (1,232,048 records) generated by 12,000 taxicabs in Beijing during 1 h (00:14:35–01:14:34) on 1 November 2012. However, with the explosive growth of taxi trajectories, the existing prediction models in a sequential processing framework still have limitations in the field of short-time traffic flow forecasting, such as high memory consumption and I/O cost, low performance, poor reliability and other problems. Intuitively, the serial prediction algorithms based on the stand-alone learning model are not good at processing big taxi trajectory data on a single platform.

As a typical methodology of nonparametric techniques, NNs is widely adopted to predict traffic conditions, but it may not overcome the inherent limitations in empirical risk minimization [33] due to the versatility, which has the low prediction accuracy with the limited samples, and the over-fitting with large sample data. Naturally, KNN can be very competitive with the state-of-the-art classification methods1 [34]. For instance, the results reported in [12], [18], [35] showed that KNN outperformed other comparable models containing HA, ARIMA, SARIMA, and ANNs. In particular, KNN can mine direct information from the historical data and keep the random characteristics of the data change with the remarkable advantage of the robustness to noise; nevertheless, it is very suitable for short-term traffic flow forecasting. If the sample data are too large and complicated, KNN will not be able to meet the needs of real-time applications due to the great computational costs. However, recently the research of KNN for short-term traffic flow prediction mainly focused on improving the accuracy of prediction, and reducing the searching time of nearest neighbors to enhance the efficiency of prediction. To the best of our knowledge, except in [4], only a few works have been done on distributed modeling for short-term traffic flow prediction with big traffic data processed in a MapReduce framework on a Hadoop platform. Particularly, using optimal KNN with large-scale taxi trajectories, this paper may be an early feasible work on forecasting short-term traffic flow in a “Big Data” environment. Furthermore, existing KNN models have not considered the spatial–temporal correlation and weight of traffic flow with trend adjustment features on traffic pattern, to the field of short-term traffic flow prediction. It should be mentioned that the spatial–temporal correlation of traffic flow is the inherent features of the traffic pattern.

This paper focuses on the accuracy and efficiency of traffic flow forecasting for 5 min ahead, and aims to establish a distributed Spatial-Temporal Weighted K-Nearest Neighbor (STW-KNN) model. This work optimizes and extends the classical nonparametric KNN approach to meet the aforementioned challenges in a parallel processing framework, MapReduce, on a distributed computing platform, Hadoop, using large-scale irregularly spaced trajectory data captured from GPS-equipped taxis. Real-world data sets analysis and experiments help us to conclude that it is possible to solve accurate and timely traffic flow prediction problem, with the proposed model having highly accurate prediction capability and reasonable execution time.

The contributions of this work are summarized as follows:

  • A general MapReduce framework of distributed modeling for traffic flow forecasting (MF-TFF) is developed to satisfy the computation and storage requirements of big traffic data for a particular application. More specifically, the developed framework is general enough that can be utilized for other data-driven traffic prediction methods as well.

  • A distributed K-nearest neighbor optimization model (STW-KNN) is presented to improve the accuracy of short-term traffic flow forecasting based on the MF-TFF framework. STW-KNN considers the spatial–temporal correlation and weight in terms of the upstream-downstream and past-future of traffic flow by modifying the state vector, proximity (distance/similarity) measure, prediction function and choice of K. In particular, STW-KNN not only considers the signs of changes but also includes the magnitudes of changes in traffic flow on proximity measure and prediction function, based on the state vector of both space and time domains.

  • A MapReduce-based parallel prediction algorithm (MBSTW-KNN) is put forward to reduce the computational costs of large-scale traffic flow data on a Hadoop platform, which is the parallelism of STW-KNN. MBSTW-KNN implements the parallel prediction of traffic flow through the Mapper, Combiner and Reducer functions, respectively, in big data environment. More importantly, it can offer a practical reference for the parallelism of the same type of algorithms.

  • The above-mentioned approach is applied to forecast short-term traffic flow of Sanlihe E. Rd. in the city of Beijing. Particularly, our extensive performance evaluation indicates that the proposed model outperforms other comparable models with robust accuracy, and its MapReduce implementation significantly improves the efficiency and scalability of short-term traffic flow prediction over conventional serial approaches on a single machine.

The remainder of this paper is organized as follows. In Section 2, we briefly review related work and provide the preliminaries, and then the motivation with solution and the developed MF-TFF framework are shown in Section 3. The proposed STW-KNN model and its MapReduce implementation are described in detail, respectively, in 4 Proposed STW-KNN model, 5 STW-KNN with MapReduce implementation. In Section 6, we give the effectiveness validations of STW-KNN and the performance evaluations of MBSTW-KNN to verify the accuracy, efficiency and scalability of our approach, respectively, and then discuss the experimental results. Section 7 concludes the paper, and then presents the direction of future work.

Section snippets

Related work

In this section, we briefly review related work on the short-term traffic flow forecasting (STFF), and then provide the preliminaries on the MapReduce framework.

Motivation and MF-TFF

In this section, we describe the motivation of this work, then analyze the existing problem in short-term traffic flow forecasting and give the problem definition, and finally provide an overview of our solution and a general MapReduce framework of distributed modeling for traffic flow forecasting (MF-TFF) developed in this paper.

Proposed STW-KNN model

In this section, we propose a spatial–temporal weighted K-nearest neighbor (STW-KNN) model to enhance the accuracy of short-term traffic flow forecasting based on the MF-TFF framework.

In the STW-KNN model, to find the best nearest neighbors, we aim to optimize the search mechanisms of the traditional KNN model, including the state vector, proximity measure, prediction function and the choice of k which are crucial to the accuracy of forecasting. On the one hand, according to the

STW-KNN with MapReduce implementation

In this section, to improve the scalability and efficiency of STW-KNN in “Big Data” environment, we implement it in a MapReduce framework on a Hadoop platform. For convenience, STW-KNN with MapReduce implementation is denoted as MBSTW-KNN (MapReduce-based STW-KNN).

In MBSTW-KNN, we implement the parallelism of STW-KNN on Hadoop with a MapReduce framework by splitting it into multiple MapReduce jobs, for parallel prediction of traffic flow with big traffic flow data. A MapReduce job is performed

Performance evaluation

In this section, we validate the accuracy of the proposed mathematical model (STW-KNN) in comparison with several well-known prediction models including basic K-nearest neighbor (KNN), Artificial Neural Networks (ANNs), Naïve Bayes (NB), Random Forest (RF) and C4.5, and then evaluate the efficiency and scalability of the implemented parallel prediction algorithm (MBSTW-KNN) by a case study.

Conclusions and future work

This paper aims to develop a distributed spatial–temporal weighted model and implement it in a MapReduce framework on a Hadoop platform, for solving accurate and timely traffic flow forecasting problem. Specifically, to enhance the accuracy of traffic flow prediction, based on the developed MF-TFF framework, the spatial–temporal correlation and weight of traffic flow with trend adjustment features are incorporated into the optimal KNN model, STW-KNN. It employs the correlations of both time and

Acknowledgments

The authors would like to thank the associate editor and the anonymous reviewers for their valuable comments and suggestions to improve this paper. This work was supported in part by the National Natural Science Foundation of China (Grant nos. 61403314, 61402380, and 61528206), the Scientific Project of State Ethnic Affairs Commission of the People׳s Republic of China (Grant no. 14GZZ012), the Science and Technology Foundation of Guizhou (Grant no. LH20147386), the Natural Science Foundation of

Dawen Xia is an Associate Professor at the School of Information Engineering, Guizhou Minzu University, Guiyang, China. He is currently working toward the Ph.D. degree in the School of Computer and Information Science & School of Software, Southwest University, Chongqing, China. His research interests include big data analytics, multi-agent systems, parallel and distributed computing, and spatial-temporal data mining.

References (62)

  • E.I. Vlahogianni et al.

    Short-term traffic forecastingwhere we are and where we׳re going

    Transp. Res. Part C: Emerg. Technol.

    (2014)
  • M.-W. Li et al.

    Urban traffic flow forecasting using Gauss-SVR with cat mapping cloud model and PSO hybrid algorithm

    Neurocomputing

    (2013)
  • P.J. García-Laencina et al.

    K nearest neighbours with mutual information for simultaneous classification and missing data imputation

    Neurocomputing

    (2009)
  • H. Chen et al.

    Real-time travel time prediction using particle filtering with a non-explicit state-transition model

    Transp. Res. Part C: Emerg. Technol.

    (2014)
  • H. Li et al.

    Model reduction for interval type-2 Takagi–Sugeno fuzzy systems

    Automatica

    (2015)
  • V. Marx

    The big challenges of big data

    Nature

    (2013)
  • J. Zhang et al.

    Data-driven intelligent transportation systemsa survey

    IEEE Trans. Intell. Transp. Syst.

    (2011)
  • Y. Wang et al.

    A real-time freeway network traffic surveillance tool

    IEEE Trans. Control Syst. Technol.

    (2006)
  • Y. Lv et al.

    Traffic flow prediction with big dataa deep learning approach

    IEEE Trans. Intell. Transp. Syst.

    (2015)
  • S. Chandra et al.

    Predictions of freeway traffic speeds and volumes using vector autoregressive models

    J. Intell. Transp. Syst.

    (2009)
  • M. Schönhof et al.

    Empirical features of congested traffic states and their implications for traffic modeling

    Transp. Sci.

    (2007)
  • Y.S. Jeong et al.

    Supervised weighting-online learning algorithm for short-term traffic flow prediction

    IEEE Trans. Intell. Transp. Syst.

    (2013)
  • T.T. Tchrakian et al.

    Real-time traffic flow forecasting using spectral analysis

    IEEE Trans. Intell. Transp. Syst.

    (2012)
  • E.I. Vlahogianni et al.

    Short-term traffic forecastingoverview of objectives and methods

    Transp. Rev.

    (2004)
  • X. Jiang et al.

    Dynamic wavelet neural network model for traffic flow forecasting

    J. Transp. Eng.

    (2005)
  • W. Zheng et al.

    Short-term freeway traffic flow predictionBayesian combined neural network approach

    J. Transp. Eng.

    (2006)
  • K.Y. Chan et al.

    Neural-network-based models for short-term traffic flow forecasting using a hybrid exponential smoothing and Levenberg–Marquardt algorithm

    IEEE Trans. Intell. Transp. Syst.

    (2012)
  • G.A. Davis et al.

    Nonparametric regression and short-term freeway traffic forecasting

    J. Transp. Eng.

    (1991)
  • H. Su, L. Zhang, S. Yu, Short-term traffic flow prediction based on incremental support vector regression, in:...
  • S. Sun et al.

    A Bayesian network approach to traffic flow forecasting

    IEEE Trans. Intell. Transp. Syst.

    (2006)
  • M.C. Tan et al.

    An aggregation approach to short term traffic flow prediction

    IEEE Trans. Intell. Transp. Syst.

    (2009)
  • Cited by (159)

    • Traffic state estimation of urban road networks by multi-source data fusion: Review and new insights

      2022, Physica A: Statistical Mechanics and its Applications
      Citation Excerpt :

      In traffic prediction based on DIDI trajectories, Fafoutellis et al. [93] proposed a dilated LSTM networks model, which effectively avoided the exploding or disappearing gradients and improved the accuracy. Based on this, Xia et al. [94] proposed a spatio-temporally weighted k-nearest neighbour model in MapReduce framework for distributed modelling on Hadoop platform. Based on the spatio-temporal correlation of DiDi trajectories in urban road networks, Zhang et al. [95] proposed a trip information maximization generation adversarial network model to approximate the trip time distribution in each road segment.

    • Spatiotemporal multi-graph convolutional networks with synthetic data for traffic volume forecasting

      2022, Expert Systems with Applications
      Citation Excerpt :

      With the exponential increase in the amount of available data and computational power, data-driven methods including traditional machine learning methods and deep learning methods have been developed rapidly in various fields, and their usefulness for extracting features was demonstrated in our previous works (Zhang et al. 2019; Zhang et al. 2021) and in other existing works (Ma et al. 2017; Varela-Santos and Melin 2021). In the transportation field, compared to traditional machine learning methods (e.g., support vector regression (SVR) (Castro-Neto et al. 2009) and K-nearest neighbors (Xia et al. 2016)), deep learning methods can handle high-dimensional data and obtain intricate and abstract features more effectively (LeCun et al. 2015; Guo et al. 2019), making them more suitable for solving traffic forecasting problems. For example, both Jo et al. (2018) and Ge et al. (2020) demonstrated through experiments that the forecasting results of deep learning methods were superior to those of traditional machine learning methods (e.g., SVR and KNN), indicating that deep learning methods are more effective in modeling complex traffic data.

    • Estimating cycle-level real-time traffic movements at signalized intersections

      2022, Journal of Intelligent Transportation Systems: Technology, Planning, and Operations
    View all citing articles on Scopus

    Dawen Xia is an Associate Professor at the School of Information Engineering, Guizhou Minzu University, Guiyang, China. He is currently working toward the Ph.D. degree in the School of Computer and Information Science & School of Software, Southwest University, Chongqing, China. His research interests include big data analytics, multi-agent systems, parallel and distributed computing, and spatial-temporal data mining.

    Binfeng Wang is currently working toward the M.Eng. degree in the School of Computer and Information Science & School of Software, Southwest University, Chongqing, China. His research interests include cloud computing, and big data analytics.

    Huaqing Li is an Associate Professor at the School of Electronic and Information Engineering, Southwest University, Chongqing, China. He received the B.S. degree from the College of Mathematics and Physics, Chongqing University of Posts and Telecommunications and the Ph.D. degree from the College of Computer Science and Technology, Chongqing University, Chongqing, China, in 2009 and 2013. His research interests include consensus of multi-agent systems, distributed optimization and big data analytics.

    Yantao Li is an Associate Professor at the School of Computer and Information Science & School of Software, Southwest University, Chongqing, China. He received the Ph.D. degree from the College of Computer Science at Chongqing University, in December 2012. His research interests include wireless communication and networking, sensor networks and ubiquitous computing, and information security.

    Zili Zhang is a Professor at Southwest University, Chongqing, China, and a Senior Lecturer at Deakin University, Australia. He received his B.Sc. from Sichuan University, M.Eng. from Harbin Institute of Technology, and Ph.D. from Deakin University, all in Computing. He authored and co-authored more than 130 refereed papers in International Journals or Conference Proceedings, six monographs or textbooks. His research interests include multi-agent systems, bio-inspired AI, and big data analytics.

    View full text