Zhehao Jiang
Zhenyu Yan
ABSTRACT
Recent years have witnessed the emergence of a new class of cooperative edge systems in which a large number of edge nodes can collaborate through local peer-to-peer connectivity. In this paper, we propose CoEdge, a novel cooperative edge system that can support concurrent data/compute-intensive deep learning (DL) models for distributed real-time applications such as city-scale traffic monitoring and autonomous driving. First, CoEdge includes a hierarchical DL task scheduling framework that dispatches DL tasks to edge nodes based on their computational profiles, communication overhead, and real-time requirements. Second, CoEdge can dramatically increase the execution efficiency of DL models by batching sensor data and aggregating the inferences of the same model. Finally, we propose a new edge containerization approach that enables an edge node to execute concurrent DL tasks by partitioning the CPU and GPU workloads into different containers. We extensively evaluate CoEdge on a self-deployed smart lamppost testbed on a university campus. Our results show that CoEdge can achieve up to reduction on deadline missing rate compared to baselines.
- Advantech. 2022. MIC-720AI - AI Inference System based on NVIDIA® Jetson Tegra X2. https://www.advantech.com/en-eu/products/9140b94e-bcfa-4aa4-8df2-1145026ad613/mic-720ai/mod_19d7f198-a3f3-4975-ac87-e8facd1045b3.Google Scholar
- Mohammed AA Al-qaness, Aaqif Afzaal Abbasi, Hong Fan, Rehab Ali Ibrahim, Saeed H Alsamhi, and Ammar Hawbani. 2021. An improved YOLO-based road traffic monitoring system. Computing 103, 2 (2021), 211–230.Google ScholarDigital Library
- Junjie Bai, Fang Lu, Ke Zhang, 2019. Onnx: Open neural network exchange. GitHub repository (2019), 54. Online; accessed 4-March-2023.Google Scholar
- Johan Barthélemy, Nicolas Verstaevel, Hugh Forehead, and Pascal Perez. 2019. Edge-computing video analytics for real-time traffic monitoring in a smart city. Sensors 19, 9 (2019), 2048.Google ScholarCross Ref
- Soroush Bateni, Husheng Zhou, Yuankun Zhu, and Cong Liu. 2018. Predjoule: A timing-predictable energy optimization framework for deep neural networks. In 2018 IEEE Real-Time Systems Symposium (RTSS). IEEE, 107–118.Google ScholarCross Ref
- Baotong Chen, Jiafu Wan, Lei Shu, Peng Li, Mithun Mukherjee, and Boxing Yin. 2017. Smart factory of industry 4.0: Key technologies, application case, and challenges. Ieee Access 6 (2017), 6505–6519.Google ScholarCross Ref
- Long Chen, Jigang Wu, Xin Long, and Zikai Zhang. 2017. ENGINE: Cost effective offloading in mobile edge computing with fog-cloud cooperation. arXiv preprint arXiv:1711.01683 (2017).Google Scholar
- OpenFog Consortium 1934. IEEE standard for adoption of OpenFog reference architecture for fog computing. IEEE Std 2018, 2018 (1934), 1–176.Google Scholar
- Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. 2021. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 13733–13742.Google ScholarCross Ref
- Biyi Fang, Xiao Zeng, and Mi Zhang. 2018. NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (New Delhi, India) (MobiCom ’18). Association for Computing Machinery, New York, NY, USA, 115–127. https://doi.org/10.1145/3241539.3241559Google ScholarDigital Library
- Teledyne FLIR. 2022. FREE Teledyne FLIR Thermal Dataset for Algorithm Training. https://www.flir.asia/oem/adas/adas-dataset-form/.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.Google ScholarCross Ref
- Yuze He, Li Ma, Zhehao Jiang, Yi Tang, and Guoliang Xing. 2021. VI-Eye: Semantic-Based 3D Point Cloud Registration for Infrastructure-Assisted Autonomous Driving. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (New Orleans, Louisiana) (MobiCom ’21). Association for Computing Machinery, New York, NY, USA, 573–586. https://doi.org/10.1145/3447993.3483276Google ScholarDigital Library
- Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, and Yonghye Kwon. 2022. ultralytics/yolov5: V6. 1-TensorRT TensorFlow edge TPU and OpenVINO export and inference. Zenodo 2 (2022), 2.Google Scholar
- Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News 45, 1 (2017), 615–629.Google ScholarDigital Library
- Alex Krizhevsky, Geoffrey Hinton, 2009. Learning multiple layers of features from tiny images. (2009).Google Scholar
- Stefanos Laskaridis, Stylianos I Venieris, Mario Almeida, Ilias Leontiadis, and Nicholas D Lane. 2020. SPINN: synergistic progressive inference of neural networks over device and cloud. In Proceedings of the 26th annual international conference on mobile computing and networking. 1–15.Google ScholarDigital Library
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755.Google ScholarCross Ref
- Neiwen Ling, Xuan Huang, Zhihe Zhao, Nan Guan, Zhenyu Yan, and Guoliang Xing. 2022. BlastNet: Exploiting Duo-Blocks for Cross-Processor Real-Time DNN Inference. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems. 91–105.Google ScholarDigital Library
- Neiwen Ling, Kai Wang, Yuze He, Guoliang Xing, and Daqi Xie. 2021. Rt-mdl: Supporting real-time mixed deep learning tasks on edge platforms. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 1–14.Google ScholarDigital Library
- Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).Google Scholar
- Shaoshan Liu, Liangkai Liu, Jie Tang, Bo Yu, Yifan Wang, and Weisong Shi. 2019. Edge Computing for Autonomous Driving: Opportunities and Challenges. Proc. IEEE 107, 8 (2019), 1697–1716. https://doi.org/10.1109/JPROC.2019.2915983Google ScholarCross Ref
- Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. 2022. Robot Operating System 2: Design, architecture, and uses in the wild. Science Robotics 7, 66 (2022), eabm6074. https://doi.org/10.1126/scirobotics.abm6074Google ScholarCross Ref
- Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled B. Letaief. 2017. A Survey on Mobile Edge Computing: The Communication Perspective. IEEE Communications Surveys & Tutorials 19, 4 (2017), 2322–2358. https://doi.org/10.1109/COMST.2017.2745201Google ScholarCross Ref
- Christian Meurisch Max Mühlhäuser. 2020. Street lamps as a platform. https://cacm.acm.org/magazines/2020/6/245163-street-lamps-as-a-platform/abstractGoogle Scholar
- Lifan Mei, Runchen Hu, Houwei Cao, Yong Liu, Zifa Han, Feng Li, and Jin Li. 2019. Realtime mobile bandwidth prediction using lstm neural network. In Passive and Active Measurement: 20th International Conference, PAM 2019, Puerto Varas, Chile, March 27–29, 2019, Proceedings 20. Springer, 34–47.Google ScholarDigital Library
- Jiaying Meng, Haisheng Tan, Xiang-Yang Li, Zhenhua Han, and Bojie Li. 2019. Online deadline-aware task dispatching and scheduling in edge computing. IEEE Transactions on Parallel and Distributed Systems 31, 6 (2019), 1270–1286.Google ScholarDigital Library
- Jiaying Meng, Haisheng Tan, Chao Xu, Wanli Cao, Liuyan Liu, and Bojie Li. 2019. Dedas: Online task dispatching and scheduling with bandwidth constraint in edge computing. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2287–2295.Google ScholarDigital Library
- Roberto Morabito. 2017. Virtualization on Internet of Things Edge Devices With Container Technologies: A Performance Evaluation. IEEE Access 5 (2017), 8835–8850. https://doi.org/10.1109/ACCESS.2017.2704444Google ScholarCross Ref
- Zhaolong Ning, Peiran Dong, Xiangjie Kong, and Feng Xia. 2018. A cooperative partial computation offloading scheme for mobile edge computing enabled Internet of Things. IEEE Internet of Things Journal 6, 3 (2018), 4804–4814.Google ScholarCross Ref
- NVIDIA. 2022. Nvidia TENSORRT. https://developer.nvidia.com/tensorrt.Google Scholar
- NVIDIA. 2023. Jetson TX2 Module. https://developer.nvidia.com/embedded/jetson-tx2.Google Scholar
- Jisun Oh, Seoyoung Kim, and Yoonhee Kim. 2018. Toward an adaptive fair GPU sharing scheme in container-based clusters. In 2018 IEEE 3rd International Workshops on Foundations and Applications of Self* Systems (FAS* W). IEEE, 79–85.Google ScholarCross Ref
- Misun Park, Ketan Bhardwaj, and Ada Gavrilovska. 2020. Toward Lighter Containers for the Edge.. In HotEdge.Google Scholar
- Lihua Ruan, Maluge Pubuduni Imali Dias, and Elaine Wong. 2019. Machine learning-based bandwidth prediction for low-latency H2M applications. IEEE Internet of Things Journal 6, 2 (2019), 3743–3752.Google ScholarCross Ref
- Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang. 2022. VIPS: Real-Time Perception Fusion for Infrastructure-Assisted Autonomous Driving. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking (Sydney, NSW, Australia) (MobiCom ’22). Association for Computing Machinery, New York, NY, USA, 133–146. https://doi.org/10.1145/3495243.3560539Google ScholarDigital Library
- Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. 2016. Edge Computing: Vision and Challenges. IEEE Internet of Things Journal 3, 5 (2016), 637–646. https://doi.org/10.1109/JIOT.2016.2579198Google ScholarCross Ref
- Zhan Shi, Yongping Xie, Wei Xue, Yong Chen, Liuliu Fu, and Xiaobo Xu. 2020. Smart factory in Industry 4.0. Systems Research and Behavioral Science 37, 4 (2020), 607–617.Google ScholarCross Ref
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- Jakub Sochor, Roman Juránek, Jakub Špaňhel, Lukáš Maršík, Adam Širokỳ, Adam Herout, and Pavel Zemčík. 2018. Comprehensive data set for automatic single camera visual speed measurement. IEEE Transactions on Intelligent Transportation Systems 20, 5 (2018), 1633–1643.Google ScholarCross Ref
- Petroc Taylor. 2022. User experience data rates of 4G, 5G and 6G technology. https://www.statista.com/statistics/1183674/mobile-broadband-user-data-rates/. Accessed: 2022.Google Scholar
- Yanan Wang, Nicolas Coudray, Yun Zhao, Fuyi Li, Changyuan Hu, Yao-Zhong Zhang, Seiya Imoto, Aristotelis Tsirigos, Geoffrey I Webb, Roger J Daly, 2021. HEAL: an automated deep learning framework for cancer histopathology image analysis. Bioinformatics 37, 22 (2021), 4291–4295.Google ScholarCross Ref
- Wikipedia. 2023. 4G. https://en.wikipedia.org/wiki/4G. Accessed: 2023.Google Scholar
- Hongyue Wu, Shuiguang Deng, Wei Li, Samee U Khan, Jianwei Yin, and Albert Y Zomaya. 2018. Request dispatching for minimizing service response time in edge cloud systems. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1–9.Google ScholarCross Ref
- Xinglong Wu, Shangbin Chen, Jin Huang, Anan Li, Rong Xiao, and Xinwu Cui. 2020. DDeep3M: Docker-powered deep learning for biomedical image segmentation. Journal of Neuroscience Methods 342 (2020), 108804.Google ScholarCross Ref
- Yecheng Xiang and Hyoseung Kim. 2019. Pipelined Data-Parallel CPU/GPU Scheduling for Multi-DNN Real-Time Inference. In 2019 IEEE Real-Time Systems Symposium (RTSS). IEEE, 392–405.Google Scholar
- Ying Xiong, Yulin Sun, Li Xing, and Ying Huang. 2018. Extend cloud to edge with KubeEdge. In 2018 IEEE/ACM Symposium on Edge Computing (SEC). IEEE, 373–377.Google ScholarCross Ref
- Dianlei Xu, Tong Li, Yong Li, Xiang Su, Sasu Tarkoma, Tao Jiang, Jon Crowcroft, and Pan Hui. 2021. Edge Intelligence: Empowering Intelligence to the Edge of Network. Proc. IEEE 109, 11 (2021), 1778–1837. https://doi.org/10.1109/JPROC.2021.3119950Google ScholarCross Ref
- Shenghao Yang and Raymond W Yeung. 2017. BATS Codes: Theory and practice. Synthesis Lectures on Communication Networks 10, 2 (2017), 1–226.Google ScholarCross Ref
- Shuochao Yao, Yiran Zhao, Huajie Shao, ShengZhong Liu, Dongxin Liu, Lu Su, and Tarek Abdelzaher. 2018. FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices(SenSys ’18). Association for Computing Machinery, New York, NY, USA, 278–291. https://doi.org/10.1145/3274783.3274840Google ScholarDigital Library
- S. Yi, Z. Hao, Z. Qin, and Q. Li. 2015. Fog Computing: Platform and Applications. In 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb). 73–78. https://doi.org/10.1109/HotWeb.2015.22Google ScholarDigital Library
- Li Lyna Zhang, Shihao Han, Jianyu Wei, Ningxin Zheng, Ting Cao, Yuqing Yang, and Yunxin Liu. 2021. Nn-Meter: Towards accurate latency prediction of deep-learning model inference on diverse edge devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 81–93.Google ScholarDigital Library
- Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. 2016. How far are we from solving pedestrian detection?. In Proceedings of the iEEE conference on computer vision and pattern recognition. 1259–1267.Google ScholarCross Ref
- Zhihe Zhao, Zhehao Jiang, Neiwen Ling, Xian Shuai, and Guoliang Xing. 2018. ECRT: An edge computing system for real-time image-based object tracking. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 394–395.Google ScholarDigital Library
- Zhihe Zhao, Kai Wang, Neiwen Ling, and Guoliang Xing. 2021. EdgeML: An AutoML framework for real-time deep learning on the edge. In Proceedings of the International Conference on Internet-of-Things Design and Implementation. 133–144.Google ScholarDigital Library
Index Terms
- CoEdge: A Cooperative Edge System for Distributed Real-Time Deep Learning Tasks
Recommendations
Analysis on quantum-based fixed priority scheduling of real-time tasks
ICUIMC '09: Proceedings of the 3rd International Conference on Ubiquitous Information Management and CommunicationFixed priority schedulers are widely used for real-time systems, and there were efforts to improve the schedulability. Preemption threshold scheduling is one of such efforts with a dual priority scheme. It increases the schedulability by introducing ...
The limited-preemptive feasibility of real-time tasks on uniprocessors
The preemptive scheduling paradigm is known to strictly dominate the non-preemptive scheduling paradigm with respect to feasibility. On the other hand, preemptively scheduling real-time tasks on uniprocessors, unlike non-preemptive scheduling, may lead ...
Edge scheduling framework for real-time and non real-time tasks
SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied ComputingThis paper presents a two-stage edge scheduling framework that maps the tasks of a real-time artificial intelligence (AI) application across a collection of edge computing resources. The first stage is global and it creates schedules with execution ...
Comments