skip to main content
10.1145/3422392.3422409acmotherconferencesArticle/Chapter ViewAbstractPublication PagessbesConference Proceedingsconference-collections
research-article

An Empirical Study on Configuration-Related Code Weaknesses

Published:21 December 2020Publication History

ABSTRACT

Developers often use the C preprocessor to handle variability and portability. However, many researchers and practitioners criticize the use of preprocessor directives because of their negative effect on code understanding, maintainability, and error proneness. This negative effect may lead to configuration-related code weaknesses, which appear only when we enable or disable certain configuration options. A weakness is a type of mistake in software that, in proper conditions, could contribute to the introduction of vulnerabilities within that software. Configuration-related code weaknesses may be harder to detect and fix than weaknesses that appear in all configurations, because variability increases complexity. To address this problem, we propose a sampling-based white-box technique to detect configuration-related weaknesses in configurable systems. To evaluate our technique, we performed an empirical study with 24 popular highly configurable systems that make heavy use of the C preprocessor, such as Apache Httpd and Libssh. Using our technique, we detected 57 configuration-related weaknesses in 16 systems. In total, we found occurrences of the following five kinds of weaknesses: 30 memory leaks, 10 uninitialized variables, 9 null pointer dereferences, 6 resource leaks, and 2 buffer overflows. The corpus of these weaknesses is a valuable source to better support further research on configuration-related code weaknesses.

References

  1. 2020. Cppcheck Design. http://cppcheck.sourceforge.net/.Google ScholarGoogle Scholar
  2. Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 Variability Bugs in the Linux Kernel: A Qualitative Analysis. In Proceedings of the International Conference on Automated Software Engineering. 421--432.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Iago Abal, Jean Melo, Stefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wasowski. 2018. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. Transactions on Software Engineering and Methodology 26, 3 (2018), 10:1--10:34.Google ScholarGoogle Scholar
  4. Ira D. Baxter. 1992. Design maintenance systems. Commun. ACM 35, 4 (1992), 73--89.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Ira D. Baxter and Michael Mehlich. 2001. Preprocessor conditional removal by simple partial evaluation. In Proceedings of the Working Conference on Reverse Engineering. IEEE, Germany, 281--290.Google ScholarGoogle Scholar
  6. Michael D. Bond and Kathryn S McKinley. 2008. Tolerating memory leaks. In Proceedings of the Object-Oriented Programming Systems Languages and Applications. 109--126.Google ScholarGoogle Scholar
  7. Larissa Braz, Rohit Gheyi, Melina Mongiovi, Márcio Ribeiro, Flávio Medeiros, and Leopoldo Teixeira. 2016. A Change-centric Approach to Compile Configurable Systems with #Ifdefs. In Proceedings of the 15th International Conference on Generative Programming: Concepts & Experiences. 109--119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Larissa Braz, Rohit Gheyi, Melina Mongiovi, Márcio Ribeiro, Flávio Medeiros, Leopoldo Teixeira, and Sabrina Souto. 2018. A change-aware per-file analysis to compile configurable systems with #ifdefs. Computer Languages, Systems & Structures 54 (2018), 427--450.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Renée Bryce and Charles Colbourn. 2006. Prioritized interaction testing for pairwise coverage with seeding and constraints. Information and Software Technology 48, 10 (2006), 960--970.Google ScholarGoogle ScholarCross RefCross Ref
  10. Al Danial. 2020. CLOC. http://cloc.sourceforge.net/.Google ScholarGoogle Scholar
  11. Christian Dietrich, Reinhard Tartler, Wolfgang Schroder-Preikschat, and Daniel Lohmann. 2012. A robust approach for variability extraction from the Linux build system. In Proceedings of the Software Product-Line Conference. 21--30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Michael Ernst, Greg Badros, and David Notkin. 2002. An Empirical Analysis of C Preprocessor Use. Transactions on Software Engineering 28, 12 (2002), 1146--1170.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Gabriel Ferreira, Momin Malik, Christian Kästner, Jürgen Pfeffer, and Sven Apel. 2016. Do #ifdefs influence the occurrence of vulnerabilities? An empirical study of the Linux kernel. In Proceedings of the International Systems and Software Product Line Conference. 65--73.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Matthew Finifter, Devdatta Akhawe, and David Wagner. 2013. An empirical study of vulnerability rewards programs. In Proceedings of the USENIX Conference on Security. 273--288.Google ScholarGoogle Scholar
  15. Stefan Frei, Dominik Schatzmann, Bernhard Plattner, and Brian Trammell. 2010. Modeling the security ecosystem - the dynamics of (In)security. Springer US, 79--106.Google ScholarGoogle Scholar
  16. Alejandra Garrido and Ralph Johnson. 2002. Challenges of Refactoring C Programs. In Proceedings of the International Workshop on Principles of Software Evolution. 6--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Alejandra Garrido and Ralph Johnson. 2003. Refactoring C with Conditional Compilation. In Proceedings of the International Conference on Automated Software Engineering. 323--326.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Alejandra Garrido and Ralph Johnson. 2005. Analyzing Multiple Configurations of a C Program. In Proceedings of the International Conference on Software Maintenance. 379--388.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Brady Garvin and Myra Cohen. 2011. Feature Interaction Faults Revisited: An Exploratory Study. In Proceedings of the International Symposium on Software Reliability Engineering. 90--99.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Brady Garvin, Myra Cohen, and Matthew Dwyer. 2011. Using Feature Locality: Can We Leverage History to Avoid Failures During Reconfiguration?. In Proceedings of the Workshop on Assurances for Self-adaptive Systems.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Paul Gazzillo and Robert Grimm. 2012. SuperC: parsing all of C by taming the preprocessor. In Proceedings of the Programming Language Design and Implementation. 323--334.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2017. Test them all, is it worth it? A ground truth comparison of configuration sampling strategies. arXiv preprint arXiv:1710.07980 (2017).Google ScholarGoogle Scholar
  23. Kyo Kang, Sholom Cohen, James Hess, William Novak, and Spencer Peterson. 1990. Feature-Oriented Domain Analysis Feasibility Study. Technical Report. Carnegie Mellon University.Google ScholarGoogle Scholar
  24. Christian Kastner and Sven Apel. 2009. Virtual Separation of Concerns -A Second Chance for Preprocessors. Journal of Object Technology 8, 6 (2009), 59--78.Google ScholarGoogle ScholarCross RefCross Ref
  25. Christian Kastner, Paolo Giarrusso, Tillmann Rendel, Sebastian Erdweg, Klaus Ostermann, and Thorsten Berger. 2011. Variability-Aware Parsing in the Presence of Lexical Macros and Conditional Compilation. In Proceedings of the Object-Oriented Programming Systems Languages and Applications. 805--824.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Jorg Liebig, Sven Apel, Christian Lengauer, Christian Kastner, and Michael Schulze. 2010. An analysis of the variability in forty preprocessor-based software product lines. In Proceedings of the International Conference on Software Engineering. 105--114.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jorg Liebig, Christian Kastner, and Sven Apel. 2011. Analyzing the discipline of preprocessor annotations in 30 million lines of C code. In Proceedings of the International Conference on Aspect-Oriented Software Development. 191--202.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jorg Liebig, Alexander von Rhein, Christian Kastner, Sven Apel, Jens Dorre, and Christian Lengauer. 2013. Scalable Analysis of Variable Software. In Proceedings of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. 81--91.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Flávio Medeiros, Christian Kastner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proceedings of the International Conference on Software Engineering. 643--654.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Flávio Medeiros, Christian Kastner, Márcio Ribeiro, Sarah Nadi, and Rohit Gheyi. 2015. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In Proceedings of the European Conference on Object-Oriented Programming. 999--1022.Google ScholarGoogle Scholar
  31. Flávio Medeiros, Márcio Ribeiro, and Rohit Gheyi. 2013. Investigating Preprocessor-Based Syntax Errors. In Proceedings of the International Conference on Generative Programming: Concepts & Experiences. 75--84.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Flávio Medeiros, Iran Rodrigues, Márcio Ribeiro, Leopoldo Teixeira, and Rohit Gheyi. 2015. An Empirical Study on Configuration-Related Issues: Investigating Undeclared and Unused Identifiers. In Proceedings of the International Conference on Generative Programming: Concepts & Experiences. 35--44.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Mitre. 2019. Top 25 Most Dangerous Software Errors. http://cwe.mitre.org/top25/.Google ScholarGoogle Scholar
  34. Mitre. 2020. Uninitialized Variable. https://cwe.mitre.org/data/definitions/457.html.Google ScholarGoogle Scholar
  35. Mitre. 2020. Weaknesses. https://cwe.mitre.org/documents/glossary/index.html#Weakness.Google ScholarGoogle Scholar
  36. Austin Mordahl, Jeho Oh, Ugur Koc, Shiyi Wei, and Paul Gazzillo. 2019. An empirical study of real-world variability bugs detected by variability-oblivious tools. In Proceedings of the Foundations of Software Engineering. 50--61.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Raphael Muniz, Larissa Braz, Rohit Gheyi, Wilkerson Andrade, Baldoino Fonseca, and Márcio Ribeiro. 2018. A Qualitative Analysis of Variability Weaknesses in Configurable Systems with #Ifdefs. In Proceedings of the International Workshop on Variability Modelling of Software-Intensive Systems. 51--58.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Sarah Nadi and Richard Holt. 2014. The Linux kernel: A case study of build system variability. Journal of Software: Evolution and Process 26, 8 (2014), 730--746.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Changhai Nie and Hareton Leung. 2011. A Survey of Combinatorial Testing. Computing Surveys 43, 2 (2011), 11:1--11:29.Google ScholarGoogle Scholar
  40. Sebastian Oster, Florian Markert, and Philipp Ritter. 2010. Automated Incremental Pairwise Testing of Software Product Lines. In Software Product Lines: Going Beyond, Jan Bosch and Jaejoon Lee (Eds.). Lecture Notes in Computer Science, Vol. 6287. 196--210.Google ScholarGoogle Scholar
  41. OWASP. 2020. Buffer Overflow. https://owasp.org/www-community/vulnerabilities/Buffer_Overflow.Google ScholarGoogle Scholar
  42. OWASP. 2020. Memory Leak. https://owasp.org/www-community/vulnerabilities/Memory_leak.Google ScholarGoogle Scholar
  43. OWASP. 2020. Null Pointer Dereference. https://owasp.org/www-community/vulnerabilities/Null_Dereference.Google ScholarGoogle Scholar
  44. OWASP. 2020. Resource Leak. https://owasp.org/www-community/vulnerabilities/Unreleased_Resource.Google ScholarGoogle Scholar
  45. Nicolas Palix, Gael Thomas, Suman Saha, Christophe Calves, Julia Lawall, and Gilles Muller. 2011. Faults in Linux: Ten Years Later. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems. 305--318.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Leonardo Passos, Jianmei Guo, Leopoldo Teixeira, Krzysztof Czarnecki, Andrzej Wasowski, and Paulo Borba. 2013. Coevolution of Variability Models and Related Artifacts: A Case Study from the Linux Kernel. In Proceedings of the International Software Product Line Conference. 91--100.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Gilles Perrouin, Sagar Sen, and Jacques Klein. 2010. Automated and Scalable T-wise Test Case Generation Strategies for Product Lines. In Proceeding of the International Conference on Software Testing, Verification and Validation. 459--468.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Sabrina Souto, Marcelo d'Amorim, and Rohit Gheyi. 2017. Balancing Soundness and Efficiency for Practical Testing of Configurable Systems. In Proceedings of the International Conference on Software Engineering. 632--642.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Henry Spencer and Geoff Collyer. 1992. Ifdef Considered Harmful, or Portability Experience with C News. In Proceendings of the USENIX Annual Technical Conference. USENIX Association.Google ScholarGoogle Scholar
  50. Reinhard Tartler, Christian Dietrich, Julio Sincero, Wolfgang Schroder-Preikschat, and Daniel Lohmann. 2014. Static Analysis of Variability in System Software: The 90,000 #ifdefs Issue. In USENIX Annual Technical Conference. 421--432.Google ScholarGoogle Scholar
  51. Our Team. 2020. Supplementary website. https://sbesweaknesses.github.io/.Google ScholarGoogle Scholar
  52. David Wheeler. 2020. FlawFinder. https://www.dwheeler.com/flawfinder/.Google ScholarGoogle Scholar

Index Terms

  1. An Empirical Study on Configuration-Related Code Weaknesses
            Index terms have been assigned to the content through auto-classification.

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in
            • Published in

              cover image ACM Other conferences
              SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering
              October 2020
              901 pages
              ISBN:9781450387538
              DOI:10.1145/3422392

              Copyright © 2020 ACM

              Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 21 December 2020

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed limited

              Acceptance Rates

              Overall Acceptance Rate147of427submissions,34%

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader