Abstract
In end-edge-cloud collaborative frameworks, deploying long-term sequence prediction models near edge devices reduces reliance on network resources while enhancing service quality and response speed. However, a key challenge lies in the limited computational capacity of edge devices, necessitating a compromise between model parameter size and prediction accuracy. Additionally, the physical isolation of edge devices, coupled with data processing at the edge, leads to the logical isolation of deployed models, complicating knowledge sharing across devices. We propose the Multi-stage Catalytic Distillation (MCD) framework to address this challenge. MCD utilizes knowledge distillation to share knowledge across models of different scales, thus achieving logical interconnectivity despite physical isolation. Furthermore, the framework introduces an innovative catalytic purification method designed to expedite model convergence and enhance prediction accuracy in edge devices. This approach mitigates the trade-offs between model size and precision, facilitating efficient knowledge transfer in edge computing environments. Within the MCD framework, models across different levels exhibit an enhancement in prediction accuracy while maintaining the parameter count essentially constant. The overall improvement rate amounts to 17.6%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ajit, A., Acharya, K., Samanta, A.: A review of convolutional neural networks. In: 2020 international conference on emerging trends in information technology and engineering (ic-ETITE). pp. 1–5. IEEE (2020)
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
Cai, S., Shu, Y., Wang, W.: Dynamic routing networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3588–3597 (2021)
Chen, H., Zeng, L., Yu, S., Chen, X.: Knowledge distillation for mobile edge computation offloading. arXiv preprint arXiv:2004.04366 (2020)
Dai, Z., Liu, H., Le, Q.V., Tan, M.: Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems 34, 3965–3977 (2021)
Deng, L., Li, G., Han, S., Shi, L., Xie, Y.: Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE 108(4), 485–532 (2020)
Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. International Journal of Computer Vision 129, 1789–1819 (2021)
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)
Queralta, J.P., Gia, T.N., Tenhunen, H., Westerlund, T.: Edge-AI in lora-based health monitoring: Fall detection system with fog computing and LSTM recurrent neural networks. In: 2019 42nd international conference on telecommunications and signal processing (TSP). pp. 601–604. IEEE (2019)
Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: Vision and challenges. IEEE Internet of Things Journal 3(5), 637–646 (2016)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
WANG, Y., YU, L., TENG, F., SONG, J., YUAN, Y.: Resource load prediction model based on long-short time series feature fusion. Journal of Computer Applications 42(5), 1508 (2022)
Wu, D., Xu, H., Jiang, Z., Yu, W., Wei, X., Lu, J.: EdgeLSTM: Towards deep and sequential edge computing for iot applications. IEEE/ACM Transactions on Networking 29(4), 1895–1908 (2021)
Yu, R., Li, P.: Toward resource-efficient federated learning in mobile edge computing. IEEE Network 35(1), 148–155 (2021)
Acknowledgement
This work is supported in part by China NSFC (Youth) through Grant No. 62306208 and No. 62002260, in part by China NSFC through Grant No. 62072332, in part by Tianjin Natural Science Foundation (Youth) Project through Grant No.23JCQNJC00920, in part by Tianjin Natural Science Foundation General Project No. 23JCYBJC00780, in part by the Tianjin Xinchuang Haihe Lab of ITAI under Grant No. 22HHXCJC00002.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ma, R., Zhang, C., Kang, Y., Wang, X., Qiu, C. (2024). MCD: Multi-stage Catalytic Distillation for Time Series Forecasting. In: Onizuka, M., et al. Database Systems for Advanced Applications. DASFAA 2024. Lecture Notes in Computer Science, vol 14854. Springer, Singapore. https://doi.org/10.1007/978-981-97-5569-1_34
Download citation
DOI: https://doi.org/10.1007/978-981-97-5569-1_34
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5568-4
Online ISBN: 978-981-97-5569-1
eBook Packages: Computer ScienceComputer Science (R0)