An Empirical Study on the Challenges of eBPF Application Development
Authors: 
Mugdha Deokar, Jingyang Men, Lucas Castanheira, Ayush Bhardwaj, Theophilus A. BensonAuthors Info & Claims
Mugdha Deokar, Jingyang Men, Lucas Castanheira, Ayush Bhardwaj, Theophilus A. BensonAuthors Info & ClaimsPages 1 - 8
Abstract
eBPF has become a crucial tool for the development of specialized and customized network functions, observability frameworks, and security tools. To support these growing use cases, the eBPF ecosystem (i.e., tool chains, set of language primitives, and kernel interfaces) has evolved at an extremely fast pace. Despite its rapid evolution, as a community, we understand very little about the challenges faced by developers in designing eBPF programs or the issues that hamper operators in managing them. This study aims to shed light on these challenges by analyzing eBPF issues on Stack Overflow along several eBPF-specific dimensions. We make several interesting observations that call attention to under-explored areas of the eBPF ecosystem, as well as highlight interesting research directions.
References
[1]
A. Alourani, A. D. Kshemkalyani, and M. Grechanik. 2020. T-BASIR: Finding Shutdown Bugs for Cloud-Based Applications in Cloud Spot Markets. in IEEE Transactions on Parallel and Distributed Systems, vol 31, 8 (2020), 1912--1924.
[2]
Ahmed Alquraan, Hatem Takruri, Mohammed Alfatafta, and Samer Al-Kiswany. 2018. An analysis of network-partitioning failures in cloud systems. In Proceedings of the 13th USENIX conference on Operating Systems Design and Implementation (OSDI'18). USENIX Association, USA, 51--68USENIX Association, USA, 51--68.
[3]
Omar Faroque Anik. 2022. Error fixing up map structure, incompatible struct bpf_elf_map used? https://stackoverflow.com/questions/71202769/error-fixing-up-map-structure-incompatible-struct-bpf-elf-map-used
[4]
Ido Ben-Yair, Pavel Rogovoy, and Nezer Zaidenberg. 2019. AI & EBPF Based Performance Anomaly Detection System. In Proceedings of the 12th ACM International Conference on Systems and Storage (Haifa, Israel) (SYSTOR '19). Association for Computing Machinery, New York, NY, USA, 180. https://doi.org/10.1145/3319647.3325842
[5]
Gilberto Bertin. 2017. AXDP in practice: integrating XDP in our DDoS mitigation pipeline. https://legacy.netdevconf.info/2.1/session.html?bertin
[6]
Ayush Bhardwaj. 2023. Navigating the Complexities of the Cloud-Native World: A Study of Developer Perspectives. In 2023 6th International Conference on Advanced Communication Technologies and Networking (CommNet). 1--6. https://doi.org/10.1109/CommNet60167.2023.10365297
[7]
Sanjit Bhat and Hovav Shacham. 2022. Formal Verification of the Linux Kernel eBPF Verifier Range Analysis.
[8]
bpfd 2024. bpfman: An eBPF Manager. https://bpfman.io/main/.
[9]
Haicheng Chen, Wensheng Dou, Yanyan Jiang, and Feng Qin. 2019. Understanding Exception-Related Bugs in Large-Scale Cloud Systems. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, San Diego, CA, USA, 339--351. https://doi.org/10.1109/ASE.2019.00040
[10]
Xin Chen, Dongjin Yu, Xulin Fan, Lin Wang, and Jie Chen. 2022. Multiclass Classification for Self-Admitted Technical Debt Based on XGBoost. IEEE Transactions on Reliability 71, 3 (2022), 1309--1324. https://doi.org/10.1109/TR.2021.3087864
[11]
Cloudflare. 2020. Unimog - Cloudflare's edge load balancer. Unimog-CloudflareâĂŹsedgeloadbalancer
[12]
Domenico Cotroneo, Luigi De Simone, Pietro Liguori, Roberto Natella, and Nematollah Bidokhti. 2019. How bad can a bug get? an empirical analysis of software failures in the Open-Stack cloud computing platform. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). Association for Computing Machinery, NY, USA, 200--211. https://doi.org/10.1145/3338906.3338916Association for Computing Machinery, New York, NY, USA, 200--211. https://doi.org/10.1145/3338906.3338916, New York.
[13]
Domenico Cotroneo, Luigi De Simone, Pietro Liguori, Roberto Natella, and Nematollah Bidokhti. 2019. How bad can a bug get? an empirical analysis of software failures in the Open-Stack cloud computing platform. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). Association for Computing Machinery, NY, USA, 200--211. https://doi.org/10.1145/3338906.3338916Association for Computing Machinery, New York, NY, USA, 200--211. https://doi.org/10.1145/3338906.3338916, New York.
[14]
T. Dai, J. He, X. Gu, and S. Lu. 2018. Understanding Real-World Timeout Problems in Cloud Server Systems. IEEE International Conference on Cloud Engineering (IC 10, 1109 (2018), 1--11.
[15]
Datadog. 2021. eBPF Manager. https://opensource.datadoghq.com/projects/ebpf-manager/
[16]
Luca Deri, Samuele Sabella, and Simone Mainardi. 2019. Combining System Visibility and Security Using eBPF. In Italian Conference on Cybersecurity. https://api.semanticscholar.org/CorpusID:59616648
[17]
Sanjeev Rampal Donald Hunter. 2023. eBPF application development: Beyond the basics. https://developers.redhat.com/articles/2023/10/19/ebpf-application-development-beyond-basics
[18]
Facebook. 2018. A High performance layer 4 load-balancer. https://github.com/facebookincubator/katran
[19]
Washington Garcia and Theophilus Benson. 2016. A First Look at Bugs in OpenStack. In Proceedings of the 2016 ACM Workshop on Cloud-Assisted Networking (CAN '16). Association for Computing Machinery, NY, USA, 67--72. https://doi.org/10.1145/3010079.3010086Association for Computing Machinery, New York, NY, USA, 67--72. https://doi.org/10.1145/3010079.3010086, New York.
[20]
Elazar Gershuni, Nadav Amit, Arie Gurfinkel, Nina Narodytska, Jorge A. Navas, Noam Rinetzky, Leonid Ryzhyk, and Mooly Sagiv. 2019. Simple and Precise Static Analysis of Untrusted Linux Kernel Extensions. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (Phoenix, AZ, USA) (PLDI 2019). Association for Computing Machinery, New York, NY, USA, 1069--1084. https://doi.org/10.1145/3314221.3314590
[21]
Brendan Gregg. 2020. BPF binaries: BTF, CO-RE, and the future of BPF perf tools. https://www.brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbpf.html
[22]
Haryadi S. Gunawi, Mingzhe Hao, Tanakorn Leesatapornwongsa, Tiratat Patana-anake, Thanh Do, Jeffry Adityatama, Kurnia J. Eliazar, Agung Laksono, Jeffrey F. Lukman, Vincentius Martin, and Anang D. Satria. 2014. What Bugs Live in the Cloud? //doi.org/10.1145/2670979.2670986, A Study of 3000+ Issues in Cloud Systems. In Proceedings of the ACM Symposium on Cloud Computing (SOCC '14). Association for Computing Machinery, New York, NY, USA, 1--14. https.
[23]
Haryadi S. Gunawi, Mingzhe Hao, Riza O. Suminto, Agung Laksono, Anang D. Satria, Jeffry Adityatama, and Kurnia J. Eliazar. 2016. Why Does the Cloud Stop Computing? //doi.org/10.1145/2987550.2987583, Lessons from Hundreds of Service Outages. In Proceedings of the Seventh ACM Symposium on Cloud Computing (SoCC '16). Association for Computing Machinery, New York, NY, USA, 1--16. https.
[24]
Hsin-Wei Hung and Ardalan Amiri Sani. 2023. BRF: eBPF Runtime Fuzzer. arXiv:2305.08782 [cs.CR]
[25]
Mathieu Jadin, Quentin De Coninck, Louis Navarre, Michael Schapira, and Olivier Bonaventure. 2022. Leveraging eBPF to make TCP path-aware. IEEE Transactions on Network and Service Management 19, 3 (2022), 2827--2838.
[26]
Bill Mulligan Jean Yang. 2022. Why eBPF for All Means People Don't Need to Care about eBPF. https://ebpf.io/blog/ebpf-for-all/
[27]
Jinghao Jia, Raj Sahu, Adam Oswald, Dan Williams, Michael V. Le, and Tianyin Xu. 2023. Kernel Extension Verification is Untenable. In Proceedings of the 19th Workshop on Hot Topics in Operating Systems (Providence, RI, USA) (HOTOS '23). Association for Computing Machinery, New York, NY, USA, 150--157. https://doi.org/10.1145/3593856.3595892
[28]
libbpf. 2023. libbpf v1.3.0. https://github.com/libbpf/libbpf/releases/tag/v1.3.0
[29]
libbpfgo. 2021. libbpfgo. https://pkg.go.dev/github.com/aquasecurity/tracee/libbpfgo
[30]
Haopeng Liu, Shan Lu, Madan Musuvathi, and Suman Nath. 2019. What bugs cause production cloud incidents? In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS '19). Association for Computing Machinery, NY, USA, 155--162. https://doi.org/10.1145/3317550.3321438Association for Computing Machinery, New York, NY, USA, 155--162. https://doi.org/10.1145/3317550.3321438, New York.
[31]
Mark. 2018. who creates map in BPF. https://stackoverflow.com/questions/48067163/who-creates-map-in-bpf#:~:text=Your%20bpftool%20program%20is%20the,is%20creating%20the%20eBPF%20map
[32]
Andrii Nakryiko. 2019. Bringing BPF developer experience to the next level. https://lpc.events/event/4/contributions/448/
[33]
Nexus. 2023. Use tc and xdp programs to broker services in docker containers. https://stackoverflow.com/questions/75718679/use-tc-and-xdp-programs-to-broker-services-in-docker-containers
[34]
The University of British Columbia. 2018. A Comparison of LDA and NMF for Topic Modeling on Literary Themes. https://wiki.ubc.ca/Course:CPSC522/A_Comparison_of_LDA_and_NMF_for_Topic_Modeling_on_Literary_Themes.
[35]
Muriz Serifovic. 2018. Image-to-Recipe Translation with Deep Convolutional Neural Networks. https://towardsdatascience.com/this-ai-is-hungry-b2a8655528be.
[36]
StackOverflow. 2024. Questions tagged [ebpf]. https://stackoverflow.com/questions/tagged/ebpf
[37]
Viet-Hoang Tran and Olivier Bonaventure. 2019. Beyond socket options: making the Linux TCP stack truly extensible. In 2019 IFIP Networking Conference (IFIP Networking). IEEE, 1--9.
[38]
Viet-Hoang Tran and Olivier Bonaventure. 2019. Making the Linux TCP stack more extensible with eBPF. In Proc. of the Netdev 0x13, Technical Conference on Linux Networking.
[39]
Dave Tucker, Andrew Stoycos, Andre Fredette, Brent Salisbury, and Mauro Ezequiel Moltrasio. 2024. bpfman: An eBPF Manager. https://bpfman.io/v0.4.1/
[40]
Harishankar Vishwanathan, Matan Shachnai, Srinivas Narayana, and Santosh Nagarakatte. 2023. Verifying The Verifier: eBPF Range Analysis Verification. In Computer Aided Verification: 35th International Conference, CAV 2023, Paris, France, July 17-22, 2023, Proceedings, Part III (Paris, France). Springer-Verlag, Berlin, Heidelberg, 226--251. https://doi.org/10.1007/978-3-031-37709-9_12
[41]
Walmart. 2021. L3AF. https://l3af.io//
[42]
Erci Xu, Mai Zheng, Feng Qin, Yikang Xu, and Jiesheng Wu. 2019. Lessons and actions: what we learned from 10K SSD-related storage system failures. In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC '19). USENIX Association, USA, 961--975USENIX Association, USA, 961--975.
[43]
Pravin Singh Yadav, Rajwant Singh Rao, and Alok Mishra. 2024. An Evaluation of Multi-Label Classification Approaches for Method-Level Code Smells Detection. IEEE Access 12 (2024), 53664--53676. https://doi.org/10.1109/ACCESS.2024.3387856
[44]
Sepehr Abbasi Zadeh, Ali Munir, Mahmoud Mohamed Bahnasy, Shiva Ketabi, and Yashar Ganjali. 2023. On Augmenting TCP/IP Stack via eBPF. In Proceedings of the 1st Workshop on EBPF and Kernel Extensions (New York, NY, USA) (eBPF '23). Association for Computing Machinery, New York, NY, USA, 15--20. https://doi.org/10.1145/3609021.3609300
[45]
Wei Zheng, Chen Feng, Tingting Yu, Xibing Yang, and Xiaoxue Wu. 2019. Towards understanding bugs in an open source cloud management stack: An empirical study of OpenStack software bugs. Journal of Systems and Software 151 (2019), 210--223. https://doi.org/10.1016/j.jss.2019.02.025
Index Terms
- An Empirical Study on the Challenges of eBPF Application Development
Recommendations
Understanding Performance of eBPF Maps
eBPF '24: Proceedings of the ACM SIGCOMM 2024 Workshop on eBPF and Kernel ExtensionsThe Linux community has witnessed the rapid development of eBPF technology that allows users to load custom programs into the Linux kernel to extend its capabilities. A key feature that makes eBPF powerful is eBPF maps, which provide data storage and ...
On Integrating eBPF into Pluginized Protocols
eBPF is a popular technology originating from the Linux kernel that enables safely running user-provided programs in a kernel-context. This technology opened the door for efficient programming in the operating system, especially in its network stack. ...
Verifying the Verifier: eBPF Range Analysis Verification
Computer Aided VerificationAbstractThis paper proposes an automated method to check the correctness of range analysis used in the Linux kernel ’s eBPF verifier. We provide the specification of soundness for range analysis performed by the eBPF verifier. We automatically generate ...
Comments
Information & Contributors
Information
Published In
August 2024
77 pages
ISBN:9798400707124
DOI:10.1145/3672197
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ACM SIGCOMM '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 12 of 21 submissions, 57%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 681Total Downloads
- Downloads (Last 12 months)681
- Downloads (Last 6 weeks)75
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in