skip to main content
10.1145/3357223.3362708acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
research-article

Cartel: A System for Collaborative Transfer Learning at the Edge

Published: 20 November 2019 Publication History

Abstract

As Multi-access Edge Computing (MEC) and 5G technologies evolve, new applications are emerging with unprecedented capacity and real-time requirements. At the core of such applications there is a need for machine learning (ML) to create value from the data at the edge. Current ML systems transfer data from geo-distributed streams to a central datacenter for modeling. The model is then moved to the edge and used for inference or classification. These systems can be ineffective because they introduce significant demand for data movement and model transfer in the critical path of learning. Furthermore, a full model may not be needed at each edge location. An alternative is to train and update the models online at each edge with local data, in isolation from other edges. Still, this approach can worsen the accuracy of models due to reduced data availability, especially in the presence of local data shifts.
In this paper we propose Cartel, a system for collaborative learning in edge clouds, that creates a model-sharing environment in which tailored models at each edge can quickly adapt to changes, and can be as robust and accurate as centralized models. Results show that Cartel adapts to workload changes 4 to 8x faster than isolated learning, and reduces model size, training time and total data transfer by 3x, 5.7x and ~1500x, respectively, when compared to centralized learning.

References

[1]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).
[2]
Amazon. 2018. Amazon Machine Learning. System Limits. https://docs.aws.amazon.com/machine-learning/latest/dg/system-limits.html.
[3]
B. Amento, B. Balasubramanian, R. J. Hall, K. Joshi, G. Jung, and K. H. Purdy. 2016. FocusStack: Orchestrating Edge Clouds Using Location-Based Focus of Attention. In ACM Symposium on Edge Computing (SEC'16).
[4]
Shabab Bazrafkan and Peter M Corcoran. 2018. Pushing the AI envelope: merging deep networks to accelerate edge artificial intelligence in consumer electronics devices and systems. IEEE Consumer Electronics Magazine 7, 2 (2018), 55--61.
[5]
Rudolf Beran et al. 1977. Minimum Hellinger distance estimates for parametric models. The annals of Statistics 5, 3 (1977), 445--463.
[6]
Ketan Bhardwaj, Ming-Wei Shih, Pragya Agarwal, Ada Gavrilovska, Taesoo Kim, and Karsten Schwan. 2016. Fast, Scalable and Secure Onloading of Edge Functions using AirBox. In Proceedings of the 1st IEEE/ACM Symposium on Edge Computing (SEC'16).
[7]
Albert Bifet and Ricard Gavalda. 2007. Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference on data mining. SIAM, 443--448.
[8]
Léon Bottou. 1998. Online Algorithms and Stochastic Approximations. (1998). http://leon.bottou.org/papers/bottou-98x revised, oct 2012.
[9]
Leo Breiman. 1996. Bagging predictors. Machine Learning 24, 2 (01 Aug 1996), 123--140. https://doi.org/10.1007/BF00058655
[10]
Ignacio Cano, Markus Weimer, Dhruv Mahajan, Carlo Curino, Giovanni Matteo Fumarola, and Arvind Krishnamurthy. 2017. Towards Geo-Distributed Machine Learning. IEEE Data Eng. Bull. 40, 4 (2017), 41--59. http://sites.computer.org/debull/A17dec/p41.pdf
[11]
Kai-Wei Chang, Cho-Jui Hsieh, and Chih-Jen Lin. 2008. Coordinate Descent Method for Large-scale L2-loss Linear Support Vector Machines. Journal of Machine Learning Research 9 (2008), 1369--1398. https://dl.acm.org/citation.cfm?id=1442778
[12]
Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In OSDI, Vol. 14. 571--582.
[13]
Radha Chitta, Rong Jin, and Anil K Jain. 2012. Efficient kernel clustering using random fourier features. In 2012 IEEE 12th International Conference on Data Mining. IEEE, 161--170.
[14]
Yi-Min Chou, Yi-Ming Chan, Jia-Hong Lee, Chih-Yi Chiu, and Chu-Song Chen. 2018. Unifying and Merging Well-trained Deep Neural Networks for Inference Stage. CoRR abs/1805.04980 (2018). arXiv:1805.04980 http://arxiv.org/abs/1805.04980
[15]
Cisco. 2017. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2016--2021 White Paper. https://cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.pdf
[16]
Stephane Daeuble. [n.d.]. Small cells and Mobile Edge Computing cover all the bases for Taiwan baseball fans. https://www.nokia.com/blog/small-cells-mobile-edge-computing-cover-bases-taiwan-baseball-fans/.
[17]
Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. 2007. Boosting for Transfer Learning. (2007), 193--200. https://doi.org/10.1145/1273496.1273521
[18]
Facebook. [n.d.]. Applying machine learning science to Facebook products. https://research.fb.com/category/machine-learning/.
[19]
João Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation. ACM Comput. Surv. 46, 4 (2014), 44:1--44:37. https://doi.org/10.1145/2523813
[20]
Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S. Sathiya Keerthi, and S. Sundararajan. 2008. A dual coordinate descent method for large-scale linear SVM. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 (ACM International Conference Proceeding Series), William W. Cohen, Andrew McCallum, and Sam T. Roweis (Eds.), Vol. 307. ACM, 408--415. https://doi.org/10.1145/1390156.1390208
[21]
Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R Ganger, Phillip B Gibbons, and Onur Mutlu. 2017. Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds. In NSDI. 629--647.
[22]
Ke-Jou Carol Hsu, Ketan Bhardwaj, and Ada Gavrilovska. 2019. Couper: DNN Model Slicing for Visual Analytics Containers at the Edge. In 4th ACM/IEEE Symposium on Edge Computing (SEC'19).
[23]
Yun Chao Hu, Milan Patel, Dario Sabella, Nurit Sprecher, and Valerie Young. 2015. Mobile edge computing---A key technology towards 5G. ETSI white paper 11, 11 (2015), 1--16.
[24]
Geoff Hulten, Laurie Spencer, and Pedro Domingos. 2001. Mining time-changing data streams. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 97--106.
[25]
Tetsuya Ishikawa. 2019. Random Fourier Features. https://github.com/tiskw/RandomFourierFeatures
[26]
Hyuk-Jin Jeong, Hyeon-Jae Lee, Chang Hyun Shin, and Soo-Mook Moon. 2018. IONN: Incremental Offloading of Neural Network Computations from Mobile Devices to Edge Servers. In Proceedings of the ACM Symposium on Cloud Computing. ACM, 401--411.
[27]
Ruoming Jin and Gagan Agrawal. 2003. Efficient decision tree construction on streaming data. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 571--576.
[28]
James M Joyce. 2011. Kullback-leibler divergence. In International encyclopedia of statistical science. Springer, 720--722.
[29]
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGPLAN Notices 52, 4 (2017), 615--629.
[30]
Sebastian Kauschke and Johannes Fürnkranz. 2018. Batchwise Patching of Classifiers. In Thirty-Second AAAI Conference on Artificial Intelligence.
[31]
Daniel Kifer, Shai Ben-David, and Johannes Gehrke. 2004. Detecting change in data streams. In Proceedings of the Thirtieth international conference on Very large data bases-Volume 30. VLDB Endowment, 180--191.
[32]
Juyong Kim, Yookoon Park, Gunhee Kim, and Sung Ju Hwang. 2017. SplitNet: Learning to semantically split deep networks for parameter reduction and model parallelization. In International Conference on Machine Learning. 1866--1874.
[33]
Jakub Konečny, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016).
[34]
Balaji Lakshminarayanan, Daniel M Roy, and Yee Whye Teh. 2014. Mondrian forests: Efficient online random forests. (2014), 3140--3148.
[35]
Yann LeCun. 2010. The MNIST database of handwritten digits. http://yann. lecun.com/exdb/mnist/ (2010).
[36]
Dawei Li, Theodoros Salonidis, Nirmit V Desai, and Mooi Choo Chuah. 2016. Deepcham: Collaborative edge-mediated adaptive deep learning for mobile object recognition. (2016), 64--76.
[37]
Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. In OSDI, Vol. 14. 583--598.
[38]
Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. 14 (2014), 583--598.
[39]
Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. In OSDI, Vol. 14. 583--598.
[40]
Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Trans. Information Theory 37, 1 (1991), 145--151. https://doi.org/10.1109/18.61115
[41]
Dirk Lindemeier. 2015. Nokia: EE and Mobile Edge Computing ready to rock Wembley stadium. https://www.nokia.com/blog/ee-mobile-edge-computing-ready-rock-wembley-stadium/.
[42]
Viktor Losing, Barbara Hammer, and Heiko Wersing. 2018. Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing 275 (2018), 1261--1274. https://doi.org/10.1016/j.neucom.2017.06.084
[43]
Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M Hellerstein. 2012. Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment (2012), 716--727.
[44]
Frank J Massey Jr. 1951. The Kolmogorov-Smirnov test for goodness of fit. Journal of the American statistical Association 46, 253 (1951), 68--78.
[45]
Jose G. Moreno-Torres, Troy Raeder, Rocío Alaíz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera. 2012. A unifying view on dataset shift in classification. Pattern Recognition 45, 1 (2012), 521--530. https://doi.org/10.1016/j.patcog.2011.06.019
[46]
IoT Now and Sheetal Kumbhar. 2017. Intel, RIFT.io, Vasona Networks and Xaptum to Demo IoT Multi-Access Edge Computing. https://tinyurl.com/intel-riftio-Vasona-Xaptum.
[47]
Opinov8. 2019. How Do Amazon, Facebook, Apple and Google Use AI? https://opinov8.com/how-do-amazon-facebook-apple-and-google-use-ai/.
[48]
Anand Padmanabha Iyer, Li Erran Li, Mosharaf Chowdhury, and Ion Stoica. 2018. Mitigating the Latency-Accuracy Trade-off in Mobile Data Analytics Systems. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 513--528.
[49]
Sinno Jialin Pan, Qiang Yang, et al. 2010. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2010), 1345--1359.
[50]
Manohar Parakh. 2018. How Companies Use Machine Learning. https://dzone.com/articles/how-companies-use-machine-learning.
[51]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830. http://dl.acm.org/citation.cfm?id=2078195
[52]
Michele Polese, Rittwik Jana, Velin Kounev, Ke Zhang, Supratim Deb, and Michele Zorzi. 2018. Machine Learning at the Edge: A Data-Driven Architecture with Applications to 5G Cellular Networks. CoRR abs/1808.07647 (2018). arXiv:1808.07647 http://arxiv.org/abs/1808.07647
[53]
Ali Rahimi and Benjamin Recht. 2008. Random features for large-scale kernel machines. In Advances in neural information processing systems. 1177--1184.
[54]
Jon NK Rao and Alastair J Scott. 1981. The analysis of categorical data from complex sample surveys: chi-squared tests for goodness of fit and independence in two-way tables. Journal of the American statistical association 76, 374 (1981), 221--230.
[55]
Pablo Rodriguez. 2017. The Edge: Evolution or Revolution. In ACM/IEEE Symposium on Edge Computing (SEC'17).
[56]
Amir Saffari, Christian Leistner, Jakob Santner, Martin Godec, and Horst Bischof. 2009. On-line random forests. In Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on. IEEE, 1393--1400.
[57]
Mahadev Satyanarayanan. 2017. The emergence of edge computing. Computer 50, 1 (2017), 30--39.
[58]
Mahadev Satyanarayanan, Zhuo Chen, Kiryong Ha, Wenlu Hu, Wolfgang Richter, and Padmanabhan Pillai. 2014. Cloudlets: at the leading edge of mobile-cloud convergence. In 2014 6th International Conference on Mobile Computing, Applications and Services (MobiCASE). IEEE, 1--9.
[59]
Iman Sharafaldin, Arash Habibi Lashkari, and Ali A Ghorbani. 2018. Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization. In ICISSP. 108--116.
[60]
Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. 2016. Edge computing: Vision and challenges. IEEE Internet of Things Journal 3, 5 (2016), 637--646.
[61]
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. 2017. Federated multi-task learning. In Advances in Neural Information Processing Systems. 4424--4434.
[62]
Heng Wang and Zubin Abraham. 2015. Concept drift detection for streaming data. In 2015 International Joint Conference on Neural Networks, IJCNN 2015, Killarney, Ireland, July 12-17, 2015. 1--9. https://doi.org/10.1109/IJCNN.2015.7280398
[63]
Shuo Wang, Leandro L. Minku, Davide Ghezzi, Daniele Caltabiano, Peter Tiño, and Xin Yao. 2013. Concept drift detection for online class imbalance learning. In The 2013 International Joint Conference on Neural Networks, IJCNN 2013, Dallas, TX, USA, August 4-9, 2013. 1--10. https://doi.org/10.1109/IJCNN.2013.6706768
[64]
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. 2018. When edge meets learning: Adaptive control for resource-constrained distributed machine learning. (2018), 63--71.
[65]
Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. HotCloud 10, 10-10 (2010), 95.
[66]
Wuyang Zhang, Jiachen Chen, Yanyong Zhang, and Dipankar Raychaudhuri. 2017. Towards efficient edge cloud augmentation for virtual reality MMOGs. In Proceedings of the Second ACM/IEEE Symposium on Edge Computing. ACM, 8.
[67]
Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, and Wenjun Zeng. 2016. Deep convolutional neural networks with merge-and-run mappings. arXiv preprint arXiv:1611.07718 (2016).

Cited By

View all
  • (2025)Knowledge Routing in Decentralized LearningIEEE Intelligent Systems10.1109/MIS.2024.350554340:1(38-44)Online publication date: Jan-2025
  • (2024)Cross-Network Embeddings Transfer for Traffic AnalysisIEEE Transactions on Network and Service Management10.1109/TNSM.2023.332944221:3(2686-2699)Online publication date: Jun-2024
  • (2024)An Accurate and Efficient Clustered Federated Learning for Mobile Edge Devices2024 IEEE/ACM Symposium on Edge Computing (SEC)10.1109/SEC62691.2024.00017(110-122)Online publication date: 4-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SoCC '19: Proceedings of the ACM Symposium on Cloud Computing
November 2019
503 pages
ISBN:9781450369732
DOI:10.1145/3357223
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 November 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Mobile-access Edge Computing (MEC)
  2. collaborative learning
  3. distributed machine learning
  4. transfer learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SoCC '19
Sponsor:
SoCC '19: ACM Symposium on Cloud Computing
November 20 - 23, 2019
CA, Santa Cruz, USA

Acceptance Rates

SoCC '19 Paper Acceptance Rate 39 of 157 submissions, 25%;
Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)43
  • Downloads (Last 6 weeks)1
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Knowledge Routing in Decentralized LearningIEEE Intelligent Systems10.1109/MIS.2024.350554340:1(38-44)Online publication date: Jan-2025
  • (2024)Cross-Network Embeddings Transfer for Traffic AnalysisIEEE Transactions on Network and Service Management10.1109/TNSM.2023.332944221:3(2686-2699)Online publication date: Jun-2024
  • (2024)An Accurate and Efficient Clustered Federated Learning for Mobile Edge Devices2024 IEEE/ACM Symposium on Edge Computing (SEC)10.1109/SEC62691.2024.00017(110-122)Online publication date: 4-Dec-2024
  • (2024)High-Performance Federated Learning Techniques for Deep Learning on Medical DatasetICT for Intelligent Systems10.1007/978-981-97-5799-2_33(387-399)Online publication date: 27-Sep-2024
  • (2023)Edge Intelligence with Distributed Processing of DNNs: A SurveyComputer Modeling in Engineering & Sciences10.32604/cmes.2023.023684136:1(5-42)Online publication date: 2023
  • (2023)Multi-mode LearningProceedings of the 2nd ACM Workshop on Data Privacy and Federated Learning Technologies for Mobile Edge Network10.1145/3615593.3615722(113-118)Online publication date: 6-Oct-2023
  • (2023)CLUE: Systems Support for Knowledge Transfer in Collaborative Learning With Neural NetsIEEE Transactions on Cloud Computing10.1109/TCC.2023.329449011:4(3541-3554)Online publication date: Oct-2023
  • (2023)Anomaly Detection and Resolution on the Edge: Solutions and Future Directions2023 IEEE International Conference on Service-Oriented System Engineering (SOSE)10.1109/SOSE58276.2023.00034(227-238)Online publication date: Jul-2023
  • (2023)Learning to Be Green: Carbon-Aware Online Control for Edge Intelligence with Colocated Learning and Inference2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS57875.2023.00033(567-578)Online publication date: Jul-2023
  • (2023)AI-based fog and edge computing: A systematic review, taxonomy and future directionsInternet of Things10.1016/j.iot.2022.10067421(100674)Online publication date: Apr-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media