Skip to main content
Springer Nature Link
Log in

Fixing Latent Unsound Abstract Operators in the eBPF Verifier of the Linux Kernel

  • Conference paper
  • First Online:
Static Analysis (SAS 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14995))

Included in the following conference series:

Abstract

This paper describes our experience deploying automated verification techniques for proving the correctness of value tracking components of the eBPF verifier in the Linux Kernel over the last four years. The eBPF verifier uses abstract interpretation with multiple abstract domains for value tracking. The eBPF verifier uses non-standard approaches for refining the results from multiple abstract domains, which necessitated us to design new techniques to show their correctness. During this process, we also discovered that some of the abstract operators are unsound in isolation. The unsoundness of these operators are eventually corrected by a shared refinement operator. The presence of intermediate “latent” unsound abstract operators makes the task of verification harder. We describe our patches to the Linux kernel, which have been upstreamed, that fix these latent errors and make the abstract operators correct in isolation, which enables faster automated verification.

M. Shachnai and H. Vishwanathan—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://go.rutgers.edu/90ueywub.

  2. 2.

    In SMT-LIB is a bitvector of length 1 that is equal to the value 1.

  3. 3.

    In SMT-LIB is a bitvector of length 64 equal to the value 0.

References

  1. Agni’s verification of kernel 6.4 takes weeks. https://github.com/bpfverif/agni/issues/13

  2. bpf: fix incorrect sign extension in check_alu_op(). https://github.com/torvalds/linux/commit/95a762e2c8c942780948091f8f2a4f32fce1ac6f

  3. bpf, Harden and/or/xor value tracking in verifier. https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=1f586614f3ff

  4. bpf, Register bounds logic and testing improvements. https://elixir.bootlin.com/linux/v6.8-rc1/source/kernel/bpf/verifier.c

  5. bpf, x32: Fix bug with ALU64 LSH, RSH, ARSH BPF_X shift by 0. https://github.com/torvalds/linux/commit/68a8357ec15bdce55266e9fba8b8b3b8143fa7d2

  6. Cilium API-aware networking and security. https://cilium.io/

  7. CVE-2017-16996 Mishandling of register truncation. https://nvd.nist.gov/vuln/detail/CVE-2017-16996

  8. CVE-2017-17852 Mishandling of 32-bit ALU ops. https://nvd.nist.gov/vuln/detail/CVE-2017-17852

  9. CVE-2017-17853 Mishandling of 32-bit ALU ops. https://nvd.nist.gov/vuln/detail/CVE-2017-17853

  10. CVE-2017-17864 Mishandled comparison between pointer and unknown data types. https://nvd.nist.gov/vuln/detail/CVE-2017-17864

  11. CVE-2018-18445 Mishandling of 32-bit RSH op. https://nvd.nist.gov/vuln/detail/CVE-2018-18445

  12. CVE-2020-8835 Mishandling of bounds tracking for 32-bit JMPs. https://nvd.nist.gov/vuln/detail/CVE-2020-8835

  13. CVE-2021-3490 The eBPF ALU32 bounds tracking for bitwise ops (AND, OR and XOR) in the Linux kernel did not properly update 32-bit bounds. CVE-2021-3490

    Google Scholar 

  14. Facebook’s Katran load balancer: Kernel XDP program. https://github.com/facebookincubator/katran/blob/master/katran/lib/bpf/balancer_kern.c

  15. Netconf 2018 day 1. https://lwn.net/Articles/757201/

  16. Suricata: ebpf and xdp. https://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html

  17. workflows: CI for the verification step . https://github.com/bpfverif/agni/commit/18b7513facda0d6f57e69c293f9a494ede683be7

  18. workflows: Initial CI for the LLVM-to-SMT step . https://github.com/bpfverif/agni/commit/a30260d2b8c4c14f356f9501c4a9dac28f768f5d

  19. bpf, tnums: Provably sound, faster, and more precise algorithm for tnum_mul (2021). https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=05924717ac70. Accessed 19 Oct 2022

  20. New GKE Dataplane V2 increases security and visibility for containers (2021). https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine

  21. LLVM’s MemorySSA (2023). https://llvm.org/docs/MemorySSA.html

  22. Verifying the Verifier: eBPF Range Analysis Verification (2023). https://doi.org/10.5281/zenodo.7931901

  23. Bhat, S., Shacham, H.: Formal verification of the linux kernel ebpf verifier range analysis (2022). https://sanjit-bhat.github.io/assets/pdf/ebpf-verifier-range-analysis22.pdf

  24. Borkmann, D.: bpf: Fix scalar32_min_max_or bounds tracking (2020). https://github.com/torvalds/linux/commit/5b9fbeb75b6a98955f628e205ac26689bcb1383e

  25. Borkmann, D.: bpf: undo incorrect __reg_bound_offset32 handling (2020). https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=f2d67fec0b43edce8c416101cdc52e71145b5fef

  26. Borkmann, D.: bpf: fix alu32 const subreg bound tracking on bitwise operations (2021). https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=049c4e13714ecbca567b4d5f6d563f05d431c80e

  27. Borkmann, D.: bpf: Fix signed_sub,add32_overflows type handling (2021). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bc895e8b2a64e502fbba72748d59618272052a8b

  28. Cortesi, A., Costantini, G., Ferrara, P.: A survey on product operators in abstract interpretation. Electron. Proc. Theor. Comput. Sci. 129, 325–336 (2013). https://doi.org/10.4204/eptcs.129.19

    Article  MathSciNet  MATH  Google Scholar 

  29. Cousot, P., Cousot, R.: Higher-order abstract interpretation (and application to comportment analysis generalizing strictness, termination, projection and per analysis of functional languages). In: Proceedings of 1994 IEEE International Conference on Computer Languages (ICCL 1994), pp. 95–112 (1994). https://doi.org/10.1109/ICCL.1994.288389

  30. Cousot, P.: Abstract interpretation based formal methods and future challenges. In: Wilhelm, R. (ed.) Informatics. LNCS, vol. 2000, pp. 138–156. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44577-3_10

    Chapter  MATH  Google Scholar 

  31. Cousot, P.: Lecture 13 notes: Mit 16.399, abstract interpretation (2005). http://web.mit.edu/afs/athena.mit.edu/course/16/16.399/www/lecture_13-abstraction1/Cousot_MIT_2005_Course_13_4-1.pdf

  32. Cousot, P., Cousot, R.: Static determination of dynamic properties of programs. In: Proceedings of the 2nd International Symposium on Programming, Paris, France, pp. 106–130. Dunod (1976)

    Google Scholar 

  33. Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages. POPL 1977, pp. 238–252. Association for Computing Machinery, New York (1977). https://doi.org/10.1145/512950.512973

  34. Cousot, P., Cousot, R.: Systematic design of program analysis frameworks. In: Proceedings of the 6th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages. POPL 1979, pp. 269–282. Association for Computing Machinery, New York (1979). https://doi.org/10.1145/567752.567778

  35. Cree, E.: bpf/verifier: fix bounds calculation on BPF_RSH (2017). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4374f256ce8182019353c0c639bb8d0695b4c941

  36. Duplyakin, D., et al.: The design and operation of cloudlab. In: Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference. USENIX ATC 2019, pp. 1–14. USENIX Association, USA (2019)

    Google Scholar 

  37. Fabre, A.: L4drop: Xdp ddos mitigations. https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/

  38. Gershuni, E., et al.: Simple and precise static analysis of untrusted linux kernel extensions. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. PLDI 2019, pp. 1069–1084 Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3314221.3314590

  39. Gregg, B.: Bpf performance analysis at netflix. https://www.slideshare.net/brendangregg/reinvent-2019-bpf-performance-analysis-at-netflix

  40. Horn, J.: Arbitrary read+write via incorrect range tracking in ebpf. https://bugs.chromium.org/p/project-zero/issues/detail?id=1454

  41. Horn, J.: bpf: fix 32-bit ALU op verification (2017). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=468f6eafa6c44cb2c5d8aad35e12f06c240a812a

  42. Horn, J.: bpf: 32-bit RSH verification must truncate input before the ALU op (2018). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b799207e1e1816b09e7a5920fbb2d5fcf6edd681

  43. Leong, L.: ZDI-20-1440: an incorrect calculation bug in the linux kernel eBPF verifier. https://www.zerodayinitiative.com/blog/2021/1/18/zdi-20-1440-an-incorrect-calculation-bug-in-the-linux-kernel-ebpf-verifier

  44. Paul, M.: CVE-2020-8835: Linux kernel privilege escalation via improper eBPF program verification. https://www.zerodayinitiative.com/blog/2020/4/8/cve-2020-8835-linux-kernel-privilege-escalation-via-improper-ebpf-program-verification

  45. Miné, A.: Abstract domains for bit-level machine integer and floating-point operations. In: WING 2012 - 4th International Workshop on invariant Generation. p. 16. Manchester, United Kingdom (2012). https://hal.science/hal-00748094

  46. Miné, A.: Tutorial on static inference of numeric invariants by abstract interpretation. Found. Trends® Program. Lang. 4(3-4), 120–372 (2017). https://doi.org/10.1561/2500000034

  47. Monniaux, D.: Verification of device drivers and intelligent controllers: a case study. In: Proceedings of the 7th ACM & IEEE International Conference on Embedded Software, pp. 30–36 (2007). https://doi.org/10.1145/1289927.1289937

  48. Nakryiko, A.: BPF register bounds logic and testing improvements (2023). https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=cd9c127069c0

  49. Nelson, L., Bornholt, J., Gu, R., Baumann, A., Torlak, E., Wang, X.: Scaling symbolic evaluation for automated verification of systems code with serval. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles. SOSP 2019, pp. 225–242 Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3341301.3359641

  50. Nelson, L., Van Geffen, J., Torlak, E., Wang, X.: Specification and verification in the field: applying formal methods to BPF just-in-time compilers in the linux kernel. In: Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. OSDI 2020, USENIX Association, USA (2020)

    Google Scholar 

  51. Onderka, J., Ratschan, S.: Fast three-valued abstract bit-vector arithmetic. In: Finkbeiner, B., Wies, T. (eds.) VMCAI 2022. LNCS, vol. 13182, pp. 242–262. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-94583-1_12

    Chapter  MATH  Google Scholar 

  52. Palmiotti, V.: Kernel pwning with eBPF: a love story. https://www.graplsecurity.com/post/kernel-pwning-with-ebpf-a-love-story

  53. Regehr, J., Duongsaa, U.: Deriving abstract transfer functions for analyzing embedded software. In: Proceedings of the 2006 ACM SIGPLAN/SIGBED Conference on Language, Compilers, and Tool Support for Embedded Systems. LCTES 2006, pp. 34–43 Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1134650.1134657

  54. Larabee, R.: eBPF and Analysis of the get-rekt-linux-hardened.c Exploit for CVE-2017-16995. https://ricklarabee.blogspot.com/2018/07/ebpf-and-analysis-of-get-rekt-linux.html

  55. Shirokov, N.V.: XDP: 1.5 years in production. Evolution and lessons learned. In: Linux Plumbers Conference (2018)

    Google Scholar 

  56. Singh, G., Püschel, M., Vechev, M.: Fast polyhedra abstract domain. In: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages. POPL 2017, pp. 46–59 Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3009837.3009885

  57. Starovoitov, A.: Bpf at facebook. https://kernel-recipes.org/en/2019/talks/bpf-at-facebook/

  58. Sun, H., Xu, Y., Liu, J., Shen, Y., Guan, N., Jiang, Y.: Finding correctness bugs in ebpf verifier with structured and sanitized program. In: Proceedings of the Nineteenth European Conference on Computer Systems. EuroSys 2024, pp. 689–703. Association for Computing Machinery, New York (2024). https://doi.org/10.1145/3627703.3629562

  59. Van Geffen, J., Nelson, L., Dillig, I., Wang, X., Torlak, E.: Synthesizing JIT compilers for in-kernel DSLs. In: Lahiri, S.K., Wang, C. (eds.) CAV 2020. LNCS, vol. 12225, pp. 564–586. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53291-8_29

    Chapter  Google Scholar 

  60. Venet, A.: Abstract cofibered domains: application to the alias analysis of untyped programs. In: Cousot, R., Schmidt, D.A. (eds.) SAS 1996. LNCS, vol. 1145, pp. 366–382. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-61739-6_53

    Chapter  MATH  Google Scholar 

  61. Vishwanathan, H., Shachnai, M., Narayana, S., Nagarakatte, S.: Sound, precise, and fast abstract interpretation with tristate numbers. In: Proceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization. CGO 2022, pp. 254–265. IEEE Press (2022). https://doi.org/10.1109/CGO53902.2022.9741267

  62. Vishwanathan, H., Shachnai, M., Narayana, S., Nagarakatte, S.: Agni: verifying the Verifier (eBPF Range Analysis Verification) (2023). https://github.com/bpfverif/ebpf-range-analysis-verification-cav23

  63. Vishwanathan, H., Shachnai, M., Narayana, S., Nagarakatte, S.: Verifying the verifier: eBPF range analysis verification. In: Enea, C., Lal, A. (eds.) CAV 2023. LNCS, vol. 13966, pp. 226–251. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-37709-9_12

    Chapter  Google Scholar 

  64. Wang, X., Lazar, D., Zeldovich, N., Chlipala, A., Tatlock, Z.: Jitk: a trustworthy in-kernel interpreter infrastructure. In: Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation. OSDI 2014, pp. 33–47. USENIX Association, USA (2014)

    Google Scholar 

  65. Xu, Q., Wong, M.D., Wagle, T., Narayana, S., Sivaraman, A.: Synthesizing safe and efficient kernel extensions for packet processing. In: Proceedings of the 2021 ACM SIGCOMM 2021 Conference. SIGCOMM 2021, pp. 50–64. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3452296.3472929

Download references

Acknowledgements

This paper is based upon work supported in part by the National Science Foundation under FMITF-Track I Grant No. 2019302 and the Facebook Systems and Networking Award. We thank the SAS reviewers and our shepherd, Sébastien Bardin, for their valuable feedback and guidance. We thank Paul Chaignon for his invaluable inputs and efforts to extend Agni. We also thank the kernel developers Andrii Nakryiko, Eduard Zingerman, Daniel Borkmann, and Alexei Starovoitov for their feedback to our patches. We also thank CloudLab for providing the research testbed for our experiments.

Author information

Authors and Affiliations

  1. Rutgers University, New Brunswick, USA

    Matan Shachnai, Harishankar Vishwanathan, Srinivas Narayana & Santosh Nagarakatte

Authors
  1. Matan Shachnai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harishankar Vishwanathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Srinivas Narayana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Santosh Nagarakatte
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matan Shachnai .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Arizona, Tucson, AZ, USA

    Roberto Giacobazzi

  2. IMDEA Software Institute, Pozuelo de Alarcon, Madrid, Spain

    Alessandra Gorla

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shachnai, M., Vishwanathan, H., Narayana, S., Nagarakatte, S. (2025). Fixing Latent Unsound Abstract Operators in the eBPF Verifier of the Linux Kernel. In: Giacobazzi, R., Gorla, A. (eds) Static Analysis. SAS 2024. Lecture Notes in Computer Science, vol 14995. Springer, Cham. https://doi.org/10.1007/978-3-031-74776-2_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-74776-2_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-74775-5

  • Online ISBN: 978-3-031-74776-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics