Skip to main content

Advertisement

SpringerLink
Log in

Multi-head attention-based model for reconstructing continuous missing time series data

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

A Correction to this article was published on 06 July 2023

This article has been updated

Abstract

Time series data sensed by underwater wireless sensor networks (UWSNs) play a crucial role in prediction and decision-making in marine applications. Unfortunately, equipment and environmental precision and interference problems in UWSNs may lead to a large amount of missing data in a specific time period. In this work, we propose a multi-head attention-based sequence-to-sequence model (MSSM) for reconstructing continuous missing data. It can reduce the negative impact of missing data due to the harsh underwater communication environment. MSSM has a dual encoder architecture that can process known data on both sides of missing values. Multi-head self-attention mechanism and bidirectional gate recurrent unit (Bi-GRU) can thoroughly learn the temporal patterns and the inter-sequence dependencies; moreover, soft thresholding can also reduce noise interference. Datasets are used to test the performance, and experimental results show that metrics are lower than other relevant alternatives, demonstrating that MSSM is an effective model with solid generalization ability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

Authors do not have permission to share the datasets used.

Change history

References

  1. Lin C, Han G, Guizani M, Bi Y, Du J (2019) A scheme for delay-sensitive spatiotemporal routing in SDN-enabled underwater acoustic sensor networks. IEEE Trans Veh Technol 68(9):9280–9292

    Google Scholar 

  2. Liang W, Huang W, Long J, Zhang K, Li K-C, Zhang D (2020) Deep reinforcement learning for resource protection and real-time detection in IoT environment. IEEE Internet Things J 7(7):6392–6401

    Google Scholar 

  3. Mei X, Han D, Saeed N, Wu H, Ma T, Xian J (2022) Range difference-based target localization under stratification effect and NLOS bias in UWSNs. IEEE Wirel Commun Lett 11(10):2080–2084

    Google Scholar 

  4. Wang C, Shen X, Wang H, Mei H (2022) Energy-efficient collection scheme based on compressive sensing in underwater wireless sensor networks for environment monitoring over fading channels. Digit Signal Process 127:103530

    Google Scholar 

  5. Liang W, Huang Y, Xu J, Xie S (2017) A distributed data secure transmission scheme in wireless sensor network. Int J Distrib Sens Netw 13(4):1550147717705552

    Google Scholar 

  6. Kao C-C, Lin Y-S, Wu G-D, Huang C-J (2017) A comprehensive study on the internet of underwater things: applications, challenges, and channel models. Sensors 17(7):1477

    Google Scholar 

  7. Pompili D, Melodia T, Akyildiz IF (2009) A CDMA-based medium access control for underwater acoustic sensor networks. IEEE Trans Wirel Commun 8(4):1899–1909

    Google Scholar 

  8. Xu Z, Liang W, Li K-C, Xu J, Jin H (2021) A blockchain-based roadside unit-assisted authentication and key agreement protocol for internet of vehicles. J Parallel Distrib Comput 149:29–39

    Google Scholar 

  9. Zhang T, Han D, Marino MD, Wang L, Li K-C (2021) An evolutionary-based approach for low-complexity intrusion detection in wireless sensor networks. Wirel Pers Commun 126:1–24

    Google Scholar 

  10. Liu X, Li J, Dong Z, Xiong F (2017) Joint design of energy-efficient clustering and data recovery for wireless sensor networks. IEEE Access 5:3646–3656

    Google Scholar 

  11. Qiao G, Muhammad A, Muzzammil M, Shoaib Khan M, Tariq MO, Khan MS (2022) Addressing the directionality challenge through RSSI-based multilateration technique, to localize nodes in underwater WSNs by using magneto-inductive communication. J Mar Sci Eng 10(4):530

    Google Scholar 

  12. Zhang R, Zhang J, Wang Q, Zhang H (2023) DOIDS: an intrusion detection scheme based on DBSCAN for opportunistic routing in underwater wireless sensor networks. Sensors 23(4):2096

    Google Scholar 

  13. Debastiani VJ, Bastazini VA, Pillar VD (2021) Using phylogenetic information to impute missing functional trait values in ecological databases. Eco Inform 63:101315

    Google Scholar 

  14. Yogeshwary B, Shivaprakasha K, Yashwanth N (2022) Node localization techniques in underwater sensor networks. In: 2022 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS). IEEE, pp 1042–1050

  15. Gou Y, Zhang T, Yang T, Liu J, Song S, Cui J-H (2022) A deep MARL-based power management strategy for improving the fair reuse of UWSNs. IEEE Internet Things J 10(7), 6507–6522

    Google Scholar 

  16. Mei X, Han D, Chen Y, Wu H, Ma T (2023) Target localization using information fusion in WSNs-based marine search and rescue. Alex Eng J 68:227–238

    Google Scholar 

  17. Bello O, Zeadally S (2022) Internet of underwater things communication: architecture, technologies, research challenges and future opportunities. Ad Hoc Netw 135:102933

    Google Scholar 

  18. Wei X, Liu Y, Gao S, Wang X, Yue H (2019) An RNN-based delay-guaranteed monitoring framework in underwater wireless sensor networks. IEEE access 7:25959–25971

    Google Scholar 

  19. Zhang Y-F, Thorburn PJ, Xiang W, Fitch P (2019) SSIM-a deep learning approach for recovering missing time series sensor data. IEEE Internet Things J 6(4):6618–6628

    Google Scholar 

  20. Mohsan SAH, Li Y, Sadiq M, Liang J, Khan MA (2023) Recent advances, future trends, applications and challenges of internet of underwater things (iout): a comprehensive review. J Mar Sci Eng 11(1):124

    Google Scholar 

  21. Shih S-Y, Sun F-K, Lee H (2019) Temporal pattern attention for multivariate time series forecasting. Mach Learn 108:1421–1441

    MathSciNet  MATH  Google Scholar 

  22. Diao C, Zhang D, Liang W, Li K-C, Hong Y, Gaudiot J-L (2023) A novel spatial-temporal multi-scale alignment graph neural network security model for vehicles prediction. IEEE Trans Intell Transp Syst 24(1):904–914

    Google Scholar 

  23. Caillault ÉP, Lefebvre A, Bigand A et al (2020) Dynamic time warping-based imputation for univariate time series data. Pattern Recogn Lett 139:139–147

    Google Scholar 

  24. Liang W, Li Y, Xie K, Zhang D, Li K-C, Souri A, Li K (2022) Spatial-temporal aware inductive graph neural network for C-ITS data recovery. IEEE Trans Intell Transp Syst (forthcoming). https://doi.org/10.1109/TITS.2022.3156266

    Article  Google Scholar 

  25. Zhang Y, Thorburn PJ (2021) A dual-head attention model for time series data imputation. Comput Electron Agric 189:106377

    Google Scholar 

  26. Mitiche I, McGrail T, Boreham P, Nesbitt A, Morison G (2021) Data-driven anomaly detection in high-voltage transformer bushings with LSTM auto-encoder. Sensors 21(21):7426

    Google Scholar 

  27. Xiao Z, Fu X, Zhang L, Zhang W, Liu RW, Liu Z, Goh RSM (2020) Big data driven vessel trajectory and navigating state prediction with adaptive learning, motion modeling and particle filtering techniques. IEEE Trans Intell Transp Syst 23(4):3696–3709

    Google Scholar 

  28. Habiba M, Pearlmutter BA (2020) Neural odes for informative missingess in multivariate time series. In: 2020 31st Irish Signals and Systems Conference (ISSC). IEEE, pp 1–6

  29. Nishanth KJ, Ravi V (2016) Probabilistic neural network based categorical data imputation. Neurocomputing 218:17–25

    Google Scholar 

  30. Sun K, Qian T, Chen X, Zhong M (2021) Context-aware seq2seq translation model for sequential recommendation. Inf Sci 581:60–72

    MathSciNet  Google Scholar 

  31. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in Neural Information Processing Systems, vol 30

  32. Beretta L, Santaniello A (2016) Nearest neighbor imputation algorithms: a critical evaluation. BMC Med Inform Decis Mak 16(3):197–208

    Google Scholar 

  33. Ghomrawi HM, Mandl LA, Rutledge J, Alexiades MM, Mazumdar M (2011) Is there a role for expectation maximization imputation in addressing missing data in research using WOMAC questionnaire? comparison to the standard mean approach and a tutorial. BMC Musculoskelet Disord 12(1):1–7

    Google Scholar 

  34. Azur MJ, Stuart EA, Frangakis C, Leaf PJ (2011) Multiple imputation by chained equations: what is it and how does it work? Int J Methods Psychiatr Res 20(1):40–49

    Google Scholar 

  35. Shao J, Zhong B (2003) Last observation carry-forward and last observation analysis. Stat Med 22(15):2429–2441

    Google Scholar 

  36. Mei X, Han D, Saeed N, Wu H, Chang C-C, Han B, Ma T, Xian J (2022) Trajectory optimization of autonomous surface vehicles with outliers for underwater target localization. Remote Sens 14(17):4343

    Google Scholar 

  37. Andayani F, Theng LB, Tsun MT, Chua C (2022) Hybrid LSTM-transformer model for emotion recognition from speech audio files. IEEE Access 10:36018–36027

    Google Scholar 

  38. Shi T, Keneshloo Y, Ramakrishnan N, Reddy CK (2021) Neural abstractive text summarization with sequence-to-sequence models. ACM Trans Data Sci 2(1):1–37

    Google Scholar 

  39. Llugsi R, El Yacoubi S, Fontaine A, Lupera P (2021) A novel encoder–decoder structure for time series analysis based on Bayesian uncertainty reduction. In: 2021 IEEE Latin American Conference on Computational Intelligence (LA-CCI). IEEE, pp 1–6

  40. Cho K, Van Merriënboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder–decoder approaches. arXiv preprint arXiv:1409.1259

  41. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  42. Van Zoest V, Liu X, Ngai E (2021) Data quality evaluation, outlier detection and missing data imputation methods for IoT in smart cities. In: Machine Intelligence and Data Analytics for Sustainable Future Smart Cities. Springer, pp 1–18

  43. Liu H, Han D, Li D (2020) Fabric-IoT: a blockchain-based access control system in IoT. IEEE Access 8:18207–18218

    Google Scholar 

  44. Li K-C, Gupta BB, Agrawal DP (2020) Recent advances in security, privacy, and trust for internet of things (IoT) and cyber-physical systems (CPS)

  45. Li D, Han D, Zheng Z, Weng T-H, Li H, Liu H, Castiglione A, Li K-C (2022) MOOCschain: a blockchain-based secure storage and sharing scheme for MOOCs learning. Comput Stand Interfaces 81:103597

    Google Scholar 

  46. Liang W, Yang Y, Yang C, Hu Y, Xie S, Li K-C, Cao J (2023) PDPchain: a consortium blockchain-based privacy protection scheme for personal data. IEEE Trans Reliab 72(2):586–598

    Google Scholar 

  47. Liu H, Han D, Cui M, Li K-C, Souri A, Shojafar M (2023) IdenMultiSig: identity-based decentralized multi-signature in internet of things. IEEE Trans Comput Soc Syst (forthcoming). https://doi.org/10.1109/TCSS.2022.3232173

    Article  Google Scholar 

  48. Lei X, Fan Y, Li K-C, Castiglione A, Hu Q (2021) High-precision linearized interpretation for fully connected neural network. Appl Soft Comput 109:107572

    Google Scholar 

  49. Xu J, Lin J, Liang W, Li K-C (2022) Privacy preserving personalized blockchain reliability prediction via federated learning in IoT environments. Clust Comput 25(4):2515–2526

    Google Scholar 

  50. Li D, Han D, Crespi N, Minerva R, Li K-C (2023) A blockchain-based secure storage and access control scheme for supply chain finance. J Supercomput 79(1):109–138

    Google Scholar 

Download references

Acknowledgements

This work acknowledges the CARINA and CALCOFI data support from the National Marine Data Center, and National Science and Technology Resource Sharing Service Platform of China; for Jiang Hongfeng, Zheng Chongwei, Chen Fei, et al. (2020). Sea surface meteorological observation and statistical product dataset of Northwest Pacific. V1. Science Data Bank; for the Volume data from the MN Department of Transportation Weather data from Open Weather Map; for the Intel Data presented by Peter Bodik, Wei Hong, Carlos Guestrin, Sam Madden, Mark Paskin, and Romain Thibaux in the Intel Berkeley Research laboratory.

Funding

This work was supported in part by the National Key Research and Development Program (Grant No. 2021YFC2801002), in part by the National Natural Science Foundation of China (Grant Nos. 52071200, 52201401, 52201403, and 52102397), in part by the Shanghai Committee of Science and Technology, China (Grant No. 23010502000), in part by the China Postdoctoral Science Foundation (Grant Nos. 2022M712027), in part by the Shanghai Post-doctoral Excellence Program (Grant No. 2022767), in part by the Top-Notch Innovative Program for Postgraduates of Shanghai Maritime University under Grant 2022YBR012, and in part by the Natural Science Foundation of Fujian Province under Grant 2022J01131710.

Author information

Authors and Affiliations

  1. Merchant Marine College, Shanghai Maritime University, Shanghai, 201306, China

    Huafeng Wu, Yuxuan Zhang & Linian Liang

  2. College of Information Engineering, Shanghai Maritime University, Shanghai, 201306, China

    Xiaojun Mei & Dezhi Han

  3. Shanghai Ship and Shipping Research Institute Co., Ltd., Shanghai, 200135, China

    Bing Han

  4. Department of Computer Science and Information Engineering, Providence University, Taichung, 43301, Taiwan

    Tien-Hsiung Weng & Kuan-Ching Li

Authors
  1. Huafeng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linian Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaojun Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dezhi Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bing Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tien-Hsiung Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kuan-Ching Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HW involved in methodology, writing—original draft preparation, and software; YZ involved in conceptualization, methodology, writing—original draft preparation, resources, and software; LL involved in methodology, data curation, and resources; XM involved in resources, software, and validation; DH involved in Methodology and supervision; K-CL involved in Methodology, data curation, validation, visualization, supervision, and writing—reviewing and editing; T-HW involved in methodology, visualization, and writing—reviewing and editing; BH involved in writing—reviewing and editing.

Corresponding author

Correspondence to Tien-Hsiung Weng.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: The Funding information section was missing

Appendix

Appendix

Table 6 shows the parameters and explanations in all formulas in this paper. They are listed in the form of a table to help readers better understanding.

Table 6 Parameters and annotations in formulas
Full size table

And the following content is about parameter settings for other methods. Input dimension, output dimension, and length of the time window for other methods are the same as those for MSSM. And for Dual-SSIM, encoder, and decoder, GRU input size is 50, learning rate adjustment strategy is Constantwarmup, and it does not have parameters about multi-head self-attention layer. Other parameter settings for MSSM and Dual-SSIM are the same. The value of k in KNN is 10. The value of loops in EM is 100. Other parameter settings for KNN, EM, MICE, and LOCF are all equal. The size of input data equals [Length of the Time Window, Input Dimension], and the size of output data equals length of missing values which is 6 in this paper.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Zhang, Y., Liang, L. et al. Multi-head attention-based model for reconstructing continuous missing time series data. J Supercomput 79, 20684–20711 (2023). https://doi.org/10.1007/s11227-023-05465-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05465-z

Keywords

Search

Navigation