skip to main content
10.1145/3411764.3445083acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

mTSeer: Interactive Visual Exploration of Models on Multivariate Time-series Forecast

Published: 07 May 2021 Publication History

Abstract

Time-series forecasting contributes crucial information to industrial and institutional decision-making with multivariate time-series input. Although various models have been developed to facilitate the forecasting process, they make inconsistent forecasts. Thus, it is critical to select the model appropriately. The existing selection methods based on the error measures fail to reveal deep insights into the model’s performance, such as the identification of salient features and the impact of temporal factors (e.g., periods). This paper introduces mTSeer, an interactive system for the exploration, explanation, and evaluation of multivariate time-series forecasting models. Our system integrates a set of algorithms to steer the process, and rich interactions and visualization designs to help interpret the differences between models in both model and instance level. We demonstrate the effectiveness of mTSeer through three case studies with two domain experts on real-world data, qualitative interviews with the two experts, and quantitative evaluation of the three case studies.

Supplementary Material

VTT File (3411764.3445083_videofigurecaptions.vtt)
VTT File (3411764.3445083_videopreviewcaptions.vtt)
Supplementary Materials (3411764.3445083_supplementalmaterials.zip)
MP4 File (3411764.3445083_videofigure.mp4)
Supplemental video
MP4 File (3411764.3445083_videopreview.mp4)
Preview video

References

[1]
Nesreen K Ahmed, Amir F Atiya, Neamat El Gayar, and Hisham El-Shishiny. 2010. An empirical comparison of machine learning models for time series forecasting. Econometric Reviews 29, 5-6 (2010), 594–621.
[2]
Wolfgang Aigner, Silvia Miksch, Wolfgang Müller, Heidrun Schumann, and Christian Tominski. 2008. Visual methods for analyzing time-oriented data. IEEE Transactions on Visualization and Computer Graphics 14, 1(2008), 47–60.
[3]
Naomi S Altman. 1992. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46, 3 (1992), 175–185.
[4]
Sylvain Arlot, Alain Celisse, 2010. A survey of cross-validation procedures for model selection. Statistics Surveys 4(2010), 40–79.
[5]
J Scott Armstrong and Fred Collopy. 1992. Error measures for generalizing about forecasting methods: Empirical comparisons. International Journal of Forecasting 8, 1 (1992), 69–80.
[6]
Devon K Barrow, Sven F Crone, and Nikolaos Kourentzes. 2010. An evaluation of neural network ensembles and model selection for time series prediction. In Proceedings of the International Joint Conference on Neural Networks (IJCNN). IEEE, IEEE, Barcelona, Spain, 1–8.
[7]
Christoph Bergmeir and José M Benítez. 2012. On the use of cross-validation for time series predictor evaluation. Information Sciences 191(2012), 192–213.
[8]
Markus Bögl, Wolfgang Aigner, Peter Filzmoser, Tim Lammarsch, Silvia Miksch, and Alexander Rind. 2013. Visual analytics for model selection in time series analysis. IEEE Transactions on Visualization and Computer Graphics 19, 12(2013), 2237–2246.
[9]
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. 2015. Time series analysis: Forecasting and control. John Wiley & Sons, Hoboken, NJ, USA.
[10]
Leo Breiman and Jerome H Friedman. 1997. Predicting multivariate responses in multiple linear regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 59, 1 (1997), 3–54.
[11]
Lee Byron and Martin Wattenberg. 2008. Stacked graphs–geometry & aesthetics. IEEE Transactions on Visualization and Computer Graphics 14, 6(2008), 1245–1252.
[12]
Francesco Camastra and Maurizio Filippone. 2009. A comparative evaluation of nonlinear dynamics methods for time series prediction. Neural Computing and Applications 18, 8 (2009), 1021.
[13]
Patrick S Carmack, William R Schucany, Jeffrey S Spence, Richard F Gunst, Qihua Lin, and Robert W Haley. 2009. Far casting cross-validation. Journal of Computational and Graphical Statistics 18, 4(2009), 879–893.
[14]
Michael P Clements, Philip Hans Franses, and Norman R Swanson. 2004. Forecasting economic and financial time-series with non-linear models. International Journal of Forecasting 20, 2 (2004), 169–183.
[15]
Gustavo Deco, Ralph Neuneier, and Bernd Schümann. 1997. Non-parametric data selection for neural learning in non-stationary time series. Neural Networks 10, 3 (1997), 401–407.
[16]
Naser Ezzati-Jivan and Michel R Dagenais. 2017. Multi-scale navigation of large trace data: A survey. Concurrency and Computation: Practice and Experience 29, 10(2017), e4068.
[17]
Tingting Fang and Risto Lahdelma. 2016. Evaluation of a multiple linear regression model and SARIMA model in forecasting heat demand for district heating system. Applied Energy 179(2016), 544–552.
[18]
Johannes Fuchs, Fabian Fischer, Florian Mansmann, Enrico Bertini, and Petra Isenberg. 2013. Evaluation of alternative glyph designs for time series data in a small multiple setting. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris, France, 3237–3246.
[19]
Steve R Gunn 1998. Support vector machines for classification and regression. ISIS Technical Report 14, 1 (1998), 5–16.
[20]
David Gunning. 2017. Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA), nd Web 2 (2017), 1–8.
[21]
Jeffrey Heer, Nicholas Kong, and Maneesh Agrawala. 2009. Sizing the horizon: The effects of chart size and layering on the graphical perception of time series visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Boston, MA, USA, 1303–1312.
[22]
Jeffrey Heer, Fernanda B Viégas, and Martin Wattenberg. 2007. Voyagers and voyeurs: Supporting asynchronous collaborative information visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, San Jose, CA, USA, 1029–1038.
[23]
Youssef Hmamouche, Piotr Marian Przymus, Hana Alouaoui, Alain Casali, and Lotfi Lakhal. 2019. Large Multivariate Time Series Forecasting: Survey on Methods and Scalability. In Utilizing Big Data Paradigms for Business Intelligence. IGI Global, Hershey, PA, USA, 170–197.
[24]
Sui-Lau Ho, Min Xie, and Thong Ngee Goh. 2002. A comparative study of neural network and Box-Jenkins ARIMA modeling in time series prediction. Computers & Industrial Engineering 42, 2-4 (2002), 371–375.
[25]
Tin Kam Ho. 1995. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 1. IEEE, Montreal, Quebec, Canada, 278–282.
[26]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.
[27]
Rob J Hyndman and Anne B Koehler. 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 4 (2006), 679–688.
[28]
Tobias Isenberg, Petra Isenberg, Jian Chen, Michael Sedlmair, and Torsten Möller. 2013. A systematic review on the practice of evaluating visualization. IEEE Transactions on Visualization and Computer Graphics 19, 12(2013), 2818–2827.
[29]
Søren Johansen. 1995. Likelihood-based inference in cointegrated vector autoregressive models. Oxford University Press on Demand, Oxford, England.
[30]
Michael J Kane, Natalie Price, Matthew Scotch, and Peter Rabinowitz. 2014. Comparison of ARIMA and Random Forest time series models for prediction of avian influenza H5N1 outbreaks. BMC Bioinformatics 15, 1 (2014), 276.
[31]
Few,Stephen. 2008. Time on the Horizon Visual. Last accessed on 12/31/2020 at https://www.perceptualedge.com/articles/visual_business_intelligence/time_on_the_horizon.pdf.
[32]
Hyndman, Rob J, Anne B Koehler, J Keith Ord, and Ralph D Snyder. 2008. Forecasting with exponential smoothing: The state space approach. Springer Science & Business Media, Berlin, Germany.
[33]
Munroe, Randall. 2009. Xkcd# 657: Movie narrative charts.
[34]
Ummul Khair, Hasanul Fahmi, Sarudin Al Hakim, and Robbi Rahim. 2017. Forecasting error calculation with mean absolute deviation and mean absolute percentage error. In Journal of Physics: Conference Series, Vol. 930. IOP Publishing, PA, USA, 012002.
[35]
Josua Krause, Adam Perer, and Enrico Bertini. 2014. INFUSE: Interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics 20, 12(2014), 1614–1623.
[36]
Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, San Jose, CA, USA., 5686–5697.
[37]
Bum Chul Kwon, Min-Je Choi, Joanne Taery Kim, Edward Choi, Young Bin Kim, Soonwook Kwon, Jimeng Sun, and Jaegul Choo. 2018. RetainVis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Transactions on Visualization and Computer Graphics 25, 1(2018), 299–309.
[38]
Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2011. Seven guiding scenarios for information visualization evaluation. HAL 2011, 992 (2011), 04.
[39]
Philip A Legg, David HS Chung, Matthew L Parry, Mark W Jones, Rhys Long, Iwan W Griffiths, and Min Chen. 2012. MatchPad: Interactive glyph-based visualization for real-time sports performance analysis. In Computer Graphics Forum, Vol. 31. Wiley Online Library, Vienna, Austria, 1255–1264.
[40]
Gordon Leitch and J Ernest Tanner. 1991. Economic forecast evaluation: Profits versus the conventional error measures. The American Economic Review 81 (1991), 580–590.
[41]
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems. NIPS, Long Beach, CA, USA, 4765–4774.
[42]
Helmut Lütkepohl. 2013. Vector autoregressive models. In Handbook of Research Methods and Applications in Empirical Macroeconomics. Edward Elgar Publishing, Cheltenham, UK.
[43]
Ganapathy Mahalakshmi and S.Sridevi and Rajaram, Shyamsundar. 2016. A survey on forecasting of time series data. In Proceedings of the International Conference on Computing Technologies and Intelligent Data Engineering (ICCTIDE). IEEE, Kovilpatti, India, 1–8.
[44]
Daniel J McDonald, Cosma Shalizi, and Mark Schervish. 2012. Time series forecasting: model evaluation and selection using nonparametric risk bounds. CoRR abs(2012), 1212.0463.
[45]
José Maria P Menezes Jr and Guilherme A Barreto. 2008. Long-term time series prediction with the NARX network: An empirical evaluation. Neurocomputing 71, 16-18 (2008), 3335–3343.
[46]
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. IEEE, Florence, Italy, 33–38.
[47]
Tarun K Mukherjee and Atsuyuki Naka. 1995. Dynamic relations between macroeconomic variables and the Japanese stock market: An application of a vector error correction model. Journal of Financial Research 18, 2 (1995), 223–237.
[48]
Wolfgang Müller and Heidrun Schumann. 2003. Visualization for modeling and simulation: Visualization methods for time-dependent data-an overview. In Proceedings of the 35th Winter Simulation Conference: Driving Innovation. Winter Simulation Conference, New Orleans, Louisiana, USA, 737–745.
[49]
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. (2016). arXiv:1609.03499
[50]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why should I trust you” Explaining the predictions of any classifier. In Proceedings of the 22nd SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, San Francisco, CA, USA, 1135–1144.
[51]
Takafumi Saito, Hiroko Nakamura Miyamura, Mitsuyoshi Yamamoto, Hiroki Saito, Yuka Hoshiya, and Takumi Kaseda. 2005. Two-tone pseudo coloring: Compact visualization for one-dimensional data. In Symposium on Information Visualization. IEEE, Minneapolis, MN, USA], 173–180.
[52]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. (2017). arXiv:1708.08296
[53]
Ramesh Sharda and Rajendra B Patil. 1992. Connectionist approach to time series prediction: An empirical test. Journal of Intelligent Manufacturing 3, 5 (1992), 317–323.
[54]
Qiaomu Shen, Yanhong Wu, Yuzhe Jiang, Wei Zeng, KH Alexis, Anna Vianova, and Huamin Qu. 2020. Visual Interpretation of Recurrent Neural Network on Multi-dimensional Time-series Forecast. In 2020 IEEE Pacific Visualization Symposium (PacificVis). IEEE, Tianjin, China, 61–70.
[55]
Ben Shneiderman. 2003. The eyes have it: A task by data type taxonomy for information visualizations. In The Craft of Information Visualization. Elsevier, San Francisco, CA, USA, 364–371.
[56]
Tom AB Snijders. 1988. On cross-validation for predictor evaluation in time series. In On Model Uncertainty and Its Statistical Implications. Springer, Berlin, Heidelberg, 56–69.
[57]
Eric Stellwagen, Len Tashman, 2013. ARIMA: The models of Box and Jenkins. Foresight: The International Journal of Applied Forecasting 30 (2013), 28–33.
[58]
Dong Sun, Zezheng Feng, Yuanzhe Chen, Yong Wang, Jia Zeng, Mingxuan Yuan, Ting-Chuen Pong, and Huamin Qu. 2020. DFSeer: A Visual Analytics Approach to Facilitate Model Selection for Demand Forecasting. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA, 1–13.
[59]
E Toth, A Brath, and A Montanari. 2000. Comparison of short-term rainfall prediction models for real-time flood forecasting. Journal of hydrology 239, 1-4 (2000), 132–147.
[60]
Michail Vlachos, Philip Yu, and Vittorio Castelli. 2005. On periodicity detection and structural periodic similarity. In Proceedings of the SIAM International Conference on Data Mining. SIAM, Newport Beach, CA, USA, 449–460.
[61]
Martin Wattenberg and Jesse Kriss. 2006. Designing for social data analysis. IEEE Transactions on Visualization and Computer Graphics 12, 4(2006), 549–557.
[62]
SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems. Curran Associates, Inc, Montreal, Quebec, Canada, 802–810.

Cited By

View all
  • (2024)User‐Centered Evaluation of Explainable Artificial Intelligence (XAI): A Systematic Literature ReviewHuman Behavior and Emerging Technologies10.1155/2024/46288552024:1Online publication date: 15-Jul-2024
  • (2024)EXplainable Artificial Intelligence (XAI)—From Theory to Methods and ApplicationsIEEE Access10.1109/ACCESS.2024.340984312(80799-80846)Online publication date: 2024
  • (2024)The Influence of Social Networking Usage Experience and Activity on Preferences of Explainable Artificial Intelligence (XAI) Representation Methods in a Hate Speech Detection SystemWeb Information Systems Engineering – WISE 202410.1007/978-981-96-0567-5_6(67-77)Online publication date: 2-Dec-2024
  • Show More Cited By

Index Terms

  1. mTSeer: Interactive Visual Exploration of Models on Multivariate Time-series Forecast
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
        May 2021
        10862 pages
        ISBN:9781450380966
        DOI:10.1145/3411764
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 May 2021

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. feature extraction
        2. forecasting
        3. machine learning
        4. model evaluation
        5. multivariate time series
        6. visualization

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        CHI '21
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)483
        • Downloads (Last 6 weeks)91
        Reflects downloads up to 16 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)User‐Centered Evaluation of Explainable Artificial Intelligence (XAI): A Systematic Literature ReviewHuman Behavior and Emerging Technologies10.1155/2024/46288552024:1Online publication date: 15-Jul-2024
        • (2024)EXplainable Artificial Intelligence (XAI)—From Theory to Methods and ApplicationsIEEE Access10.1109/ACCESS.2024.340984312(80799-80846)Online publication date: 2024
        • (2024)The Influence of Social Networking Usage Experience and Activity on Preferences of Explainable Artificial Intelligence (XAI) Representation Methods in a Hate Speech Detection SystemWeb Information Systems Engineering – WISE 202410.1007/978-981-96-0567-5_6(67-77)Online publication date: 2-Dec-2024
        • (2023): Diagnosing Time Representations for Time-Series Forecasting with Counterfactual ExplanationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332738930:1(1183-1193)Online publication date: 26-Oct-2023
        • (2023)Supporting Guided Exploratory Visual Analysis on Time Series Data with Reinforcement LearningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332720030:1(1172-1182)Online publication date: 26-Oct-2023
        • (2023)FSLens: A Visual Analytics Approach to Evaluating and Optimizing the Spatial Layout of Fire StationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332707730:1(847-857)Online publication date: 25-Oct-2023
        • (2023): A Visual Analytics System for Exploring Children's Physical and Mental Health Profiles with Multimodal DataIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332694330:1(1205-1215)Online publication date: 24-Oct-2023
        • (2023)LiveRetro: Visual Analytics for Strategic Retrospect in Livestream E-CommerceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332691130:1(1117-1127)Online publication date: 24-Oct-2023
        • (2023)Tax-Scheduler: An interactive visualization system for staff shifting and scheduling at tax authoritiesVisual Informatics10.1016/j.visinf.2023.02.0017:2(30-40)Online publication date: Jun-2023
        • (2022)PromotionLens: Inspecting Promotion Strategies of Online E-commerce Via Visual AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.3209440(1-11)Online publication date: 2022
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media