Liang Yuan
ABSTRACT
Powered by the massive data generated by the blossom of mobile and Web-of-Things (WoT) devices, Deep Neural Networks (DNNs) have developed both in accuracy and size in recent years. Conventional cloud-based DNN training incurs rapidly-increasing data and model transmission overheads as well as privacy issues. Mobile edge computing (MEC) provides a promising solution by facilitating DNN model training on edge servers at the network edge. However, edge servers often suffer from constrained resources and need to collaborate on DNN training. Unfortunately, managed by different telecoms, edge servers cannot properly collaborate with each other without incentives and trust. In this paper, we introduce PipeEdge, a scheme that promotes collaborative edge training between edge servers by introducing incentives and trust based on blockchain. Under the PipeEdge scheme, edge servers can hire trustworthy workers for pipelined DNN training tasks based on model parallelism. We implement PipeEdge and evaluate it comprehensively with four different DNN models. The results show that it outperforms state-of-the-art schemes by up to 173.98% with negligible overheads.
Supplemental Material
- Soumith Chintala Adam Paszke, Sam Gross and Gregory Chanan. 2022. PyTorch. https://github.com/pytorch/pytorch.git.Google Scholar
- Sarah Azouvi, Patrick McCorry, and Sarah Meiklejohn. 2018. Betting on blockchain consensus with fantomette. arXiv preprint arXiv:1805.06786 (2018).Google Scholar
- Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, and Ion Stoica. 2022. Ekya: Continuous learning of video analytics models on edge compute servers. In 19th USENIX Symposium on Networked Systems Design and Implementation. 119–135.Google Scholar
- Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In 19th International Conference on Computational Statistics. 177–186.Google Scholar
- Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, 2020. Online fast adaptation and knowledge accumulation (OSAKA): A new approach to continual learning. 34th International Conference on Neural Information Processing Systems 33 (2020), 16532–16545.Google Scholar
- Miguel Castro and Barbara Liskov. 1999. Practical Byzantine Fault Tolerance. In OSDI, Vol. 99. 173–186.Google Scholar
- Lixing Chen, Sheng Zhou, and Jie Xu. 2018. Computation peer offloading for energy-constrained mobile edge computing in small-cell networks. IEEE/ACM Transactions on Networking 26, 4 (2018), 1619–1632.Google Scholar
- Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. 2015. Compressing neural networks with the hashing trick. In International Conference on Machine Learning. PMLR, 2285–2294.Google Scholar
- Xuhui Chen, Jinlong Ji, Changqing Luo, Weixian Liao, and Pan Li. 2018. When machine learning meets blockchain: A decentralized, privacy-preserving and secure design. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 1178–1187.Google Scholar
- Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. Association for Computing Machinery, 143–154. https://doi.org/10.1145/1807128.1807152Google Scholar
- Jeffrey Dean, Greg S Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V Le, Mark Z Mao, Marc’Aurelio Ranzato, Andrew Senior, and Paul Tucker. 2012. Large scale distributed deep networks. In 25th International Conference on Neural Information Processing Systems. 1223–1231.Google Scholar
- Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition. 4690–4699.Google Scholar
- Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. 2021. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things 2, 3 (2021), 1–49.Google Scholar
- Yossi Gilad, Rotem Hemo, Silvio Micali, Georgios Vlachos, and Nickolai Zeldovich. 2017. Algorand: Scaling byzantine agreements for cryptocurrencies. In Proceedings of the 26th Symposium on Operating Systems Principles. 51–68.Google Scholar
- Jialiang Han, Yun Ma, Qiaozhu Mei, and Xuanzhe Liu. 2021. Deeprec: On-device deep learning for privacy-preserving sequential recommendation in mobile commerce. In Proceedings of the Web Conference 2021. 900–911.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 770–778.Google Scholar
- Qiang He, Guangming Cui, Xuyun Zhang, Feifei Chen, Shuiguang Deng, Hai Jin, Yanhui Li, and Yun Yang. 2019. A game-theoretical approach for user allocation in edge computing environment. IEEE Transactions on Parallel and Distributed Systems 31, 3 (2019), 515–529.Google Scholar
- Qiang He, Cheng Wang, Guangming Cui, Bo Li, Rui Zhou, Qingguo Zhou, Yang Xiang, Hai Jin, and Yun Yang. 2021. A Game-Theoretical Approach for Mitigating Edge DDoS Attack. IEEE Transactions on Dependable and Secure Computing (2021). https://doi.org/10.1109/TDSC.2021.3055559Google Scholar
- Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, 2019. GPipe: Efficient training of giant neural networks using pipeline parallelism. In 33rd International Conference on Neural Information Processing Systems. 103–112.Google Scholar
- IBM. 2021. Hyperledger Fabric. https://github.com/hyperledger/fabric.git.Google Scholar
- Shiqi Jiang, Zhiqi Lin, Yuanchun Li, Yuanchao Shu, and Yunxin Liu. 2021. Flexible high-resolution object detection on edge devices with tunable latency. In 27th Annual International Conference on Mobile Computing and Networking. 559–572.Google Scholar
- Sepandar D Kamvar, Mario T Schlosser, and Hector Garcia-Molina. 2003. The eigentrust algorithm for reputation management in P2P networks. In 12th International Conference on World Wide Web. 640–651.Google Scholar
- Aggelos Kiayias, Alexander Russell, Bernardo David, and Roman Oliynykov. 2017. Ouroboros: A provably secure proof-of-stake blockchain protocol. In Annual International Cryptology Conference. Springer, 357–388.Google Scholar
- Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2019. Blockchained on-device federated learning. IEEE Communications Letters 24, 6 (2019), 1279–1283.Google Scholar
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Eleftherios Kokoris-Kogias, Philipp Jovanovic, Linus Gasser, Nicolas Gailly, Ewa Syta, and Bryan Ford. 2018. Omniledger: A secure, scale-out, decentralized ledger via sharding. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 583–598.Google Scholar
- Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. CIFAR-10 dataset. https://www.cs.toronto.edu/ kriz/cifar.html.Google Scholar
- Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A Gibson, and Eric P Xing. 2014. On model parallelization and scheduling strategies for distributed machine learning. In 27th International Conference on Neural Information Processing Systems. 2834–2842.Google Scholar
- Bo Li, Qiang He, Feifei Chen, Haipeng Dai, Hai Jin, Yang Xiang, and Yun Yang. 2021. Cooperative Assurance of Cache Data Integrity for Mobile Edge Computing. IEEE Transactions on Information Forensics and Security 16 (2021), 4648–4662. https://doi.org/10.1109/TIFS.2021.3111747Google Scholar
- Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation). 583–598.Google Scholar
- Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. 2018. Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training. In 32nd International Conference on Neural Information Processing Systems. 8056–8067.Google Scholar
- Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. 2020. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials 22, 3 (2020), 2031–2063.Google Scholar
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In IEEE International Conference on Computer Vision. 2736–2744.Google Scholar
- Yunlong Mao, Shanhe Yi, Qun Li, Jinghao Feng, Fengyuan Xu, and Sheng Zhong. 2018. A privacy-preserving deep learning approach for face recognition with edge computing. In Proc. USENIX Workshop Hot Topics Edge Comput.(HotEdge). 1–6.Google Scholar
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182 (2017).Google Scholar
- Silvio Micali, Michael Rabin, and Salil Vadhan. 1999. Verifiable random functions. In 40th annual Symposium on Foundations of Computer Science (cat. No. 99CB37039). IEEE, 120–130.Google Scholar
- Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model.. In Interspeech, Vol. 2. Makuhari, 1045–1048.Google Scholar
- Satoshi Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system. Decentralized Business Review (2008), 21260.Google Scholar
- Arvind Narayanan, Eman Ramadan, Jason Carpenter, Qingxu Liu, Yu Liu, Feng Qian, and Zhi-Li Zhang. 2020. A first look at commercial 5G performance on smartphones. In The Web Conference. 894–905.Google Scholar
- Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In 27th ACM Symposium on Operating Systems Principles. 1–15.Google Scholar
- Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. 2021. Memory-efficient pipeline-parallel DNN training. In International Conference on Machine Learning. PMLR, 7937–7947.Google Scholar
- Mohanad Odema, Nafiul Rashid, Berken Utku Demirel, and Mohammad Abdullah Al Faruque. 2021. LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies. In 58th ACM/IEEE Design Automation Conference. IEEE, 403–408.Google Scholar
- Jihong Park, Sumudu Samarakoon, Mehdi Bennis, and Mérouane Debbah. 2019. Wireless network intelligence at the edge. Proc. IEEE 107, 11 (2019), 2204–2239.Google Scholar
- Jay H Park, Gyeongchan Yun, M Yi Chang, Nguyen T Nguyen, Seungmin Lee, Jaesik Choi, Sam H Noh, and Young-ri Choi. 2020. Hetpipe: Enabling large DNN training on (whimpy) heterogeneous GPU clusters through integration of pipelined model parallelism and data parallelism. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). 307–321.Google Scholar
- Yuxin Ren, Guyue Liu, Vlad Nitu, Wenyuan Shao, Riley Kennedy, Gabriel Parmer, Timothy Wood, and Alain Tchana. 2020. Fine-grained isolation for scalable, dynamic, multi-tenant edge clouds. In USENIX Annual Technical Conference. 927–942.Google Scholar
- Yuvraj Sahni, Jiannong Cao, Lei Yang, and Yusheng Ji. 2020. Multi-Hop Multi-Task Partial Computation Offloading in Collaborative Edge Computing. IEEE Transactions on Parallel and Distributed Systems 32, 5 (2020), 1133–1145.Google Scholar
- Sambhav Satija, Apurv Mehra, Sudheesh Singanamalla, Karan Grover, Muthian Sivathanu, Nishanth Chandran, Divya Gupta, and Satya Lokam. 2020. Blockene: A High-throughput Blockchain Over Mobile Devices. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 567–582.Google Scholar
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. arXiv preprint arXiv:1606.02891 (2016).Google Scholar
- Muhammad Shayan, Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. 2020. Biscotti: A blockchain system for private and secure federated learning. IEEE Transactions on Parallel and Distributed Systems 32, 7 (2020), 1513–1525.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, 2021. Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). 495–514.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).Google Scholar
- Zhiyuan Wang, Hongli Xu, Jianchun Liu, He Huang, Chunming Qiao, and Yangming Zhao. 2021. Resource-Efficient Federated Learning with Hierarchical Aggregation in Edge Computing. In IEEE Conference on Computer Communications. IEEE, 1–10.Google Scholar
- Xiaoyu Xia, Feifei Chen, Qiang He, John Grundy, Mohamed Abdelrazek, and Hai Jin. 2020. Online Collaborative Data Caching in Edge Computing. IEEE Transactions on Parallel and Distributed Systems 32, 2 (2020), 281–294.Google Scholar
- Li Xiong and Ling Liu. 2004. Peertrust: Supporting reputation-based trust for peer-to-peer electronic communities. IEEE Transactions on Knowledge and Data Engineering 16, 7 (2004), 843–857.Google Scholar
- Dixi Yao, Liyao Xiang, Zifan Wang, Jiayu Xu, Chao Li, and Xinbing Wang. 2021. Context-Aware Compilation of DNN Training Pipelines across Edge and Cloud. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1–27.Google Scholar
- Maofan Yin, Dahlia Malkhi, Michael K Reiter, Guy Golan Gueta, and Ittai Abraham. 2019. Hotstuff: BFT consensus with linearity and responsiveness. In 2019 ACM Symposium on Principles of Distributed Computing. 347–356.Google Scholar
- Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. 2021. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning. PMLR, 12073–12086.Google Scholar
- Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, 2021. MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge. 35th International Conference on Neural Information Processing Systems 34 (2021).Google Scholar
- Jie Zhang, Zhihao Qu, Chenxi Chen, Haozhao Wang, Yufeng Zhan, Baoliu Ye, and Song Guo. 2021. Edge learning: The enabling technology for distributed big data analytics in the edge. Comput. Surveys 54, 7 (2021), 1–36.Google Scholar
- Letian Zhang, Lixing Chen, and Jie Xu. 2021. Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning. In Proceedings of the Web Conference 2021. 3111–3123.Google Scholar
- Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. 2022. Personalized Federated Learning via Variational Bayesian Inference. In International Conference on Machine Learning. PMLR, 26293–26310.Google Scholar
- Qihua Zhou, Song Guo, Zhihao Qu, Jingcai Guo, Zhenda Xu, Jiewei Zhang, Tao Guo, Boyuan Luo, and Jingren Zhou. 2021. Octo: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny On-device Learning.. In USENIX Annual Technical Conference. 177–191.Google Scholar
- Runfang Zhou and Kai Hwang. 2007. Powertrust: A robust and scalable reputation system for trusted peer-to-peer computing. IEEE Transactions on Parallel and Distributed Systems 18, 4 (2007), 460–473.Google Scholar
Index Terms
- PipeEdge: A Trusted Pipelining Collaborative Edge Training based on Blockchain
Recommendations
EdgeMove: Pipelining Device-Edge Model Training for Mobile Intelligence
WWW '23: Proceedings of the ACM Web Conference 2023Training machine learning (ML) models on mobile and Web-of-Things (WoT) has been widely acknowledged and employed as a promising solution to privacy-preserving ML. However, these end-devices often suffer from constrained resources and fail to accommodate ...
CoopEdge: A Decentralized Blockchain-based Platform for Cooperative Edge Computing
WWW '21: Proceedings of the Web Conference 2021Edge computing (EC) has recently emerged as a novel computing paradigm that offers users low-latency services. Suffering from constrained computing resources due to their limited physical sizes, edge servers cannot always handle all the incoming ...
Blockchain-based edge computing for deep neural network applications
INTESA '18: Proceedings of the Workshop on INTelligent Embedded Systems Architectures and ApplicationsDeep neural network (DNN) applications require heavy computations, so an embedded device with limited hardware such as an IoT device cannot run the apps by itself. One solution is to offload DNN computations from the client device to nearby edge servers ...
Comments