Cited By
View all- Fan JQi MLiu LMa H(2024)Diffusion-driven Incomplete Multimodal Learning for Air Quality PredictionACM Transactions on Internet of Things10.1145/37022436:1(1-24)Online publication date: 28-Oct-2024
Num2vec: Pre-Training Numeric Representations for Time Series Forecasting in the Sensing System
Article No.: 94, Pages 1 - 23
Abstract
Time series forecasting in the sensing system aims to predict future values based on historical records that sensors have collected. Previous works, however, usually focus on improving model structure or algorithm for better performance but the perspective of learning proper numeric representations is overlooked. The inappropriate and coarse numeric representations are not expressive enough to capture the intrinsic characteristics of numbers, which will obviously degrade the prediction performance.
In this article, we propose Num2vec, an algorithmic framework to learn numeric representations. Specifically, Num2vec lists three main logic characteristics of numbers: arithmetic, direction, and periodicity. By representing numbers into a transition space, Num2vec can translates numbers agilely to different Internet of Things tasks through selecting the corresponding characteristics. According to such a design, Num2vec enjoys flexible numeric representations to fit different Internet of Things time series tasks. Extensive experiments on four real-world datasets show that the approach achieves the best performance when compared with state-of-the-art baselines.
References
[1]
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 1798–1828.
[2]
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS’13), Vol. 2. 2787–2795.
[3]
Anastasia Borovykh, Sander Bohte, and Cornelis W. Oosterlee. 2017. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv:1703.04691 (2017).
[4]
Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. 2018. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering 30, 9 (2018), 1616–1637.
[5]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
[6]
Shivani Choudhary, Tarun Luthra, Ashima Mittal, and Rajat Singh. 2021. A survey of knowledge graph embedding and their applications. arXiv preprint arXiv:2107.07842 (2021).
[7]
Yuanfei Dai, Shiping Wang, Neal N. Xiong, and Wenzhong Guo. 2020. A survey on knowledge graph embedding: Approaches, applications and benchmarks. Electronics 9, 5 (2020), 750.
[8]
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019).
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[10]
Zulong Diao, Xin Wang, Dafang Zhang, Yingru Liu, Kun Xie, and Shaoyao He. 2019. Dynamic spatial-temporal graph convolutional neural networks for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 890–897.
[11]
Yuyu Ding, Hai Zhou, Zhiping Tan, Ying Chen, and Jie Ding. 2012. The influence of wind direction on short-term wind power prediction: A case study in north China. In Proceedings of IEEE PES Innovative Smart Grid Technologies. IEEE, Los Alamitos, CA, 1–5.
[12]
Josefine Gibson. 2015. Air pollution, climate change, and health. Lancet Oncology 16, 6 (2015), e269.
[13]
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 922–929.
[14]
Mohammad M. Hamed, Hashem R. Al-Masaeid, and Zahi M. Bani Said. 1995. Short-term prediction of traffic volume in urban arterials. Journal of Transportation Engineering 121, 3 (1995), 249–254.
[15]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.
[16]
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321 (2019).
[17]
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. 95–104.
[18]
Xinzhi Lin. 2021. The application of machine learning models in the prediction of PM2. 5/PM10 concentration. In Proceedings of the 2021 4th International Conference on Computers in Management and Business. 94–101.
[19]
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the 29th AAAI Conference on Artificial Intelligence.
[20]
Yan Lin, Huaiyu Wan, Shengnan Guo, and Youfang Lin. 2020. Pre-training context and time aware location embeddings from spatial-temporal trajectories for user next location prediction. In Proceedings of the AAAI Conference on Artificial Intelligence.
[21]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
[22]
Manfred Mudelsee. 2019. Trend analysis of climate time series: A review of methods. Earth-Science Reviews 190 (2019), 310–322.
[23]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 1532–1543.
[24]
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018).
[25]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Preprint.
[26]
Shun-Yao Shih, Fan-Keng Sun, and Hung-Yi Lee. 2019. Temporal pattern attention for multivariate time series forecasting. Machine Learning 108, 8 (2019), 1421–1441.
[27]
Sehyun Tak, Soomin Woo, and Hwasoo Yeo. 2016. Data-driven imputation method for traffic data in sectional units of road links. IEEE Transactions on Intelligent Transportation Systems 17, 6 (2016), 1762–1771.
[28]
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the International Conference on Machine Learning. 2071–2080.
[29]
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering 29, 12 (2017), 2724–2743.
[30]
Shuo Wang, Yanran Li, Jiang Zhang, Qingye Meng, Lingwei Meng, and Fei Gao. 2020. PM2. 5-GNN: A domain knowledge enhanced graph neural network for PM2. 5 forecasting. In Proceedings of the 28th International Conference on Advances in Geographic Information Systems. 163–166.
[31]
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 28.
[32]
Jing Wei, Zhanqing Li, Lin Sun, Wenhao Xue, Zongwei Ma, Lei Liu, Tianyi Fan, and Maureen Cribb. 2021. Extending the EOS long-term PM\(_{2.5}\) data records since 2013 in China: Application to the VIIRS Deep Blue aerosol products. IEEE Transactions on Geoscience and Remote Sensing 60 (2021), Article 4100412.
[33]
Haomin Wen, Youfang Lin, Fan Wu, Huaiyu Wan, Shengnan Guo, Lixia Wu, Chao Song, and Yinghui Xu. 2021. Package pick-up route prediction via modeling couriers’ spatial-temporal behaviors. In Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE’21). IEEE, Los Alamitos, CA, 2141–2146.
[34]
Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. 2020. Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002.12478 (2020).
[35]
Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and Intelligent Laboratory Systems 2, 1-3 (1987), 37–52.
[36]
Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575 (2014).
[37]
Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Proceedings of the 24th International Joint Conference on Artificial Intelligence.
[38]
Eddie Yatiyana, Sumedha Rajakaruna, and Arindam Ghosh. 2017. Wind speed and direction forecasting for wind power generation using ARIMA model. In Proceedings of the 2017 Australasian Universities Power Engineering Conference (AUPEC’17). IEEE, Los Alamitos, CA, 1–6.
[39]
Poorya Zaremoodi, Wray Buntine, and Gholamreza Haffari. 2018. Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 656–661.
[40]
G. Peter Zhang. 2003. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50 (2003), 159–175.
[41]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI’21).
[42]
Francisco Martínez, María Pilar Frías, María Dolores Pérez, and Antonio Jesús Rivera. 2019. A methodology for applying k-nearest neighbor to time series forecasting. Artificial Intelligence Review 52, 3 (2019), 2019–2037.
[43]
Andrew D. Short and Arthur C. Trembanis. 2004. Decadal scale patterns in beach oscillation and rotation Narrabeen Beach, Australia—time series, PCA and wavelet analysis. Journal of Coastal Research 20, 2 (2004), 523–532.
[44]
Kiyoung Yang and Cyrus Shahabi. 2004. A PCA-based similarity measure for multivariate time series. In Proceedings of the 2nd ACM International Workshop on Multimedia Databases, 65–74.
[45]
Yen-Hsien Lee, Chih-Ping Wei, Tsang-Hsiang Cheng, and Ching-Ting Yang. 2012. Nearest-neighbor-based approach to time-series classification. Decision Support Systems 53, 1 (2012), 207–217.
Index Terms
- Num2vec: Pre-Training Numeric Representations for Time Series Forecasting in the Sensing System
Recommendations
Combining seasonal ARIMA models with computational intelligence techniques for time series forecasting
Seasonal autoregressive integrated moving average (SARIMA) models form one of the most popular and widely used seasonal time series models over the past three decades. However, in several researches it has been argued that they have two basic ...
A Novel Hybridization of ARIMA, ANN, and K-Means for Time Series Forecasting
This article aims to propose a novel hybrid forecasting model involving autoregressive integrated moving average ARIMA, artificial neural networks ANNs and k-means clustering. The single models and k-means clustering are used to build the hybrid ...
A hybrid evolutionary decomposition system for time series forecasting
In the field of time series forecasting, the combination of linear and nonlinear models has been explored to improve the accuracy of predictions when compared with the performance of individual models. Traditional forecasting methods such as the ...
Comments
Information & Contributors
Information
Published In
November 2023
622 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Journal Family
Publication History
Published: 10 July 2023
Online AM: 07 June 2023
Accepted: 20 May 2023
Revised: 17 October 2022
Received: 06 April 2022
Published in TOSN Volume 19, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- A3 Foresight Program of NSFC
- Funds for Creative Research Groups of China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 277Total Downloads
- Downloads (Last 12 months)102
- Downloads (Last 6 weeks)8
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
Cited By
View all- Fan JQi MLiu LMa H(2024)Diffusion-driven Incomplete Multimodal Learning for Air Quality PredictionACM Transactions on Internet of Things10.1145/37022436:1(1-24)Online publication date: 28-Oct-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format