skip to main content
10.1145/3301418.3313946acmconferencesArticle/Chapter ViewAbstractPublication PageseurosysConference Proceedingsconference-collections
research-article

A Reality Check on Inference at Mobile Networks Edge

Published: 25 March 2019 Publication History

Abstract

Edge computing is considered a key enabler to deploy Artificial Intelligence platforms to provide real-time applications such as AR/VR or cognitive assistance. Previous works show computing capabilities deployed very close to the user can actually reduce the end-to-end latency of such interactive applications. Nonetheless, the main performance bottleneck remains in the machine learning inference operation. In this paper, we question some assumptions of these works, as the network location where edge computing is deployed, and considered software architectures within the framework of a couple of popular machine learning tasks. Our experimental evaluation shows that after performance tuning that leverages recent advances in deep learning algorithms and hardware, network latency is now the main bottleneck on end-to-end application performance. We also report that deploying computing capabilities at the first network node still provides latency reduction but, overall, it is not required by all applications. Based on our findings, we overview the requirements and sketch the design of an adaptive architecture for general machine learning inference across edge locations.

References

[1]
3GPP. TS23.501, V15.3.0 (2018-09), Technical Specification Group Services and System Aspects; Study on Architecture for the 5G System; Stage 2, Sept. 2018.
[2]
I. E. Akkus, R. Chen, I. Rimac, M. Stein, et al. SAND: Towards high-performance serverless computing. In Proceedings of the USENIX ATC, 2018.
[3]
T. Alumäe. Full-duplex Speech-to-text System for Estonian. In Baltic HLT 2014, Kaunas, Lithuania, 2014.
[4]
C. E. Andrade, S. D. Byers, V. Gopalakrishnan, E. Halepovic, D. J. Poole, L. K. Tran, and C. T. Volinsky. Connected cars in cellular network: A measurement study. In Proceedings of the 2017 IMC, pages 235--241. ACM, 2017.
[5]
I. Baldini, P. Castro, K. Chang, P. Cheng, et al. Serverless Computing: Current Trends and Open Problems, pages 1--20. Springer Singapore, Singapore, 2017.
[6]
Z. Chen, W. Hu, J. Wang, S. Zhao, B. Amos, et al. An Empirical Study of Latency in an Emerging Class of Edge Computing Applications for Wearable Cognitive Assistance. In Proceedings of the SEC 2017, New York, NY, USA. ACM.
[7]
J. Cho, K. Sundaresan, R. Mahindra, et al. ACACIA: Context-aware Edge Computing for Continuous Interactive Applications over Mobile Networks. In Proceedings of the CoNEXT '16, pages 375--389, New York, NY, USA, 2016. ACM.
[8]
Cloud Harmony. Transparency for the cloud. https://cloudharmony.com/.
[9]
Daniyal Shahrokhian. Tensorflow implementation of You Only Look Once. https://github.com/dshahrokhian/YOLO_tensorflow.
[10]
DjiNN and Tonic. DNN as a service. http://djinn.claritylab.org.
[11]
Docker, Inc. Docker: Enterprise container platform. https://www.docker.com.
[12]
M. Everingham, L. Van Gool, C. K. I. Williams, et al. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
[13]
K. Ha, Z. Chen, W. Hu, W. Richter, et al. Towards Wearable Cognitive Assistance. In Proceedings of the MobiSys '14, pages 68--81, New York, NY, USA, 2014. ACM.
[14]
J. Hauswald, Y. Kang, M. A. Laurenzano, et al. DjiNN and Tonic: DNN As a Service and Its Implications for Future Warehouse Scale Computers. SIGARCH Comput. Archit. News, 43(3):27--40, June 2015.
[15]
J. Hauswald, M. A. Laurenzano, Y. Zhang, C. Li, et al. Sirius: An Open End-to-End Voice and Vision Personal Assistant and Its Implications for Future Warehouse Scale Computers. SIGPLAN Not., 50(4):223--238, Mar. 2015.
[16]
T. Holtgraves, S. Ross, C. Weywadt, and T. Han. Perceiving artificial social agents. Computers in Human Behavior, 23(5):2163--2174, 2007.
[17]
J. Huang, V. Rathod, C. Sun, M. Zhu, et al. Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors. In Proceedings of the CVPR 2017, July 2017.
[18]
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, et al. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093, 2014.
[19]
K. Kaljurand. Kõnele (v1.6.78). http://kaljurand.github.io/K6nele, 2018.
[20]
Y. Kang, J. Hauswald, C. Gao, A. Rovinski, et al. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. In Proceedings of the ASPLOS '17, pages 615--629, New York, NY, USA, 2017. ACM.
[21]
M. Kocour, M. Gabonay, K. Mihalíková, and T. Juřica. Newvision (v1.0). https://gitlab.com/xkocou08/newvision, 2018.
[22]
N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao, et al. DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices. In Proceedings of the IPSN 2016, pages 1--12, April 2016.
[23]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, et al. Microsoft coco: Common objects in context. In Computer Vision -- ECCV. Springer International Publishing, 2014.
[24]
Lucida. Speech and vision based intelligent personal assistant. https://github.com/claritylab/lucida.
[25]
J. Luque, C. Segura, A. SÃąnchez, M. Umbert, and L. A. Galindo. The role of linguistic and prosodic cues on the prediction of self-reported satisfaction in contact centre phone calls. In Proc. Interspeech 2017, pages 2346--2350, 2017.
[26]
V. Peddinti, D. Povey, and S. Khudanpur. A time delay neural network architecture for efficient modeling of long temporal contexts. In Proc. Interspeech, 2015.
[27]
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlíček, Y. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý. The Kaldi Speech Recognition Toolkit. In Proc. of IEEE ASRU, 2011.
[28]
J. Redmon and A. Farhadi. YOLO9000: Better, Faster, Stronger. arXiv preprint arXiv:1612.08242, 2016.
[29]
J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85--117, 2015.
[30]
A. Singhvi, S. Banerjee, Y. Harchol, A. Akella, et al. Granular Computing and Network Intensive Applications: Friends or Foes? In Proceedings of the Workshop HotNets-XVI, pages 157--163, New York, NY, USA, 2017. ACM.
[31]
S. Thorpe, D. Fize, and C. Marlot. Speed of processing in the human visual system. nature, 381(6582):520, 1996.
[32]
D. Vasisht, Z. Kapetanovic, J. Won, X. Jin, R. Chandra, S. N. Sinha, A. Kapoor, M. Sudarshan, and S. Stratman. FarmBeats: An IoT Platform for Data-Driven Agriculture. In NSDI, pages 515--529, 2017.
[33]
K. Veselý, C. Segura, I. Szöke, J. Luque, and J. Cernocký. Lightly supervised vs. semi-supervised training of acoustic model on luxembourgish for low-resource automatic speech recognition. In Proc. Interspeech, 2018.
[34]
VOIP. Qos. https://www.voip-info.org/qos/, 2018.
[35]
W. Zhang, B. Han, and P. Hui. On the Networking Challenges of Mobile Augmented Reality. In Proceedings of the Workshop on VR/AR Network '17, pages 24--29, New York, NY, USA, 2017. ACM.

Cited By

View all
  • (2024)RAPNet: Resolution-Adaptive and Predictive Early Exit Network for Efficient Image RecognitionIEEE Internet of Things Journal10.1109/JIOT.2024.342855411:20(33492-33507)Online publication date: 15-Oct-2024
  • (2024)Flow Control Solution to Avoid Bottlenecks in Edge Computing for Video Analytics2024 9th International Conference on Fog and Mobile Edge Computing (FMEC)10.1109/FMEC62297.2024.10710217(74-81)Online publication date: 2-Sep-2024
  • (2024)Split DNN Inference for Exploiting Near-Edge Accelerators2024 IEEE International Conference on Edge Computing and Communications (EDGE)10.1109/EDGE62653.2024.00020(84-91)Online publication date: 7-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
EdgeSys '19: Proceedings of the 2nd International Workshop on Edge Systems, Analytics and Networking
March 2019
71 pages
ISBN:9781450362757
DOI:10.1145/3301418
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 March 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Artificial Intelligence
  2. Edge computing

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

EuroSys '19
Sponsor:

Acceptance Rates

Overall Acceptance Rate 10 of 23 submissions, 43%

Upcoming Conference

EuroSys '25
Twentieth European Conference on Computer Systems
March 30 - April 3, 2025
Rotterdam , Netherlands

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)23
  • Downloads (Last 6 weeks)2
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)RAPNet: Resolution-Adaptive and Predictive Early Exit Network for Efficient Image RecognitionIEEE Internet of Things Journal10.1109/JIOT.2024.342855411:20(33492-33507)Online publication date: 15-Oct-2024
  • (2024)Flow Control Solution to Avoid Bottlenecks in Edge Computing for Video Analytics2024 9th International Conference on Fog and Mobile Edge Computing (FMEC)10.1109/FMEC62297.2024.10710217(74-81)Online publication date: 2-Sep-2024
  • (2024)Split DNN Inference for Exploiting Near-Edge Accelerators2024 IEEE International Conference on Edge Computing and Communications (EDGE)10.1109/EDGE62653.2024.00020(84-91)Online publication date: 7-Jul-2024
  • (2024)The computing continuum: From IoT to the cloudInternet of Things10.1016/j.iot.2024.10127227(101272)Online publication date: Oct-2024
  • (2023)STI: Turbocharge NLP Inference at the Edge via Elastic PipeliningProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3575693.3575698(791-803)Online publication date: 27-Jan-2023
  • (2023)Secure Edge Computing-Assisted Video Reporting Service in 5G-Enabled Vehicular NetworksIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328773118(3774-3786)Online publication date: 2023
  • (2023)Edge AI Inference in Heterogeneous Constrained Computing: Feasibility and Opportunities2023 IEEE 28th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD)10.1109/CAMAD59638.2023.10478416(225-232)Online publication date: 6-Nov-2023
  • (2022)DynO: Dynamic Onloading of Deep Neural Networks from Cloud to DeviceACM Transactions on Embedded Computing Systems10.1145/351083121:6(1-24)Online publication date: 18-Oct-2022
  • (2022)Edge-Cloud Collaboration for Human Activity Recognition on Multiple Subjects2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM)10.1109/WoWMoM54355.2022.00030(80-89)Online publication date: Jun-2022
  • (2022)Speeding up Machine Learning Inference on Edge Devices by Improving Memory Access Patterns using Coroutines2022 IEEE 25th International Conference on Computational Science and Engineering (CSE)10.1109/CSE57773.2022.00011(9-16)Online publication date: Dec-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media