skip to main content
10.1145/3465481.3469187acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaresConference Proceedingsconference-collections
research-article

Do Security Reports Meet Usability?: Lessons Learned from Using Actionable Mitigations for Patching TLS Misconfigurations

Published: 17 August 2021 Publication History

Abstract

Several automated tools have been proposed to detect vulnerabilities. These tools are mainly evaluated in terms of their accuracy in detecting vulnerabilities, but the evaluation of their usability is a commonly neglected topic. Usability of automated security tools is particularly crucial when dealing with problems of cryptographic protocols for which even small—apparently insignificant—changes in their configuration can result in vulnerabilities that, if exploited, pave the way to attacks with dramatic consequences for the confidentiality and integrity of exchanged messages. This becomes even more acute when considering such ubiquitous protocols as the one for Transport Layer Security (TLS for short). In this paper, we present the design and the lessons learned of a user study, meant to compare two different approaches when reporting misconfigurations. Results reveal that including contextualized actionable mitigations in security reports significantly impact the accuracy and the time needed to patch TLS vulnerabilities. Along with the lessons learned, we share the experimental material that can be used during cybersecurity labs to let students configure and patch TLS first-hand.

References

[1]
2021. Replication Package: Do Security Reports Meet Usability? Lessons Learned from Using Actionable Mitigations for Patching TLS Misconfigurations. https://st.fbk.eu/complementary/ETACS2021.
[2]
Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L. Mazurek, and Christian Stransky. 2016. You Get Where You’re Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP). IEEE, San Jose, 289–305.
[3]
Luca Allodi, Silvio Biagioni, Bruno Crispo, Katsiaryna Labunets, Fabio Massacci, and Wagner Santos. 2017. Estimating the Assessment Difficulty of CVSS Environmental Metrics: An Experiment. In Future Data and Security Engineering, Tran Khanh Dang, Roland Wagner, Josef Küng, Nam Thoai, Makoto Takizawa, and Erich J. Neuhold(Eds.). Springer International Publishing, Cham, 23–39.
[4]
Luca Allodi, Marco Cremonini, Fabio Massacci, and Woohyun Shim. 2020. Measuring the accuracy of software vulnerability assessments: experiments with students and professionals. Empirical Software Engineering 25 (01 2020). https://doi.org/10.1007/s10664-019-09797-4
[5]
Matthew Bernhard, Jonathan Sharman, Claudia Ziegler Acemyan, Philip Kortum, Dan S. Wallach, and J. Alex Halderman. 2019. On the Usability of HTTPS Deployment https://doi.org/10.1145/3290605.3300540. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300540
[6]
Karthikeyan Bhargavan and Gaëtan Leurent. 2016. On the Practical (In-)Security of 64-bit Block Ciphers: Collision Attacks on HTTP over TLS and OpenVPN. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016. https://doi.org/10.1145/2976749.2978423
[7]
Michelle Cartwright. 2001. Book Review: Experimentation in Software Engineering: An Introduction. By Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén. Kluwer Academic Publishers, 1999, ISBN 0-7923-8682-5. Software Testing, Verification and Reliability 11, 3(2001), 198–199. https://doi.org/10.1002/stvr.230
[8]
Mariano Ceccato, Massimiliano Di Penta, Paolo Falcarin, Filippo Ricca, Marco Torchiano, and Paolo Tonella. 2014. A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empirical Software Engineering 19, 4 (2014), 1040–1074.
[9]
Mariano Ceccato, Massimiliano Di Penta, Paolo Falcarin, Filippo Ricca, Marco Torchiano, and Paolo Tonella. 2014. A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empirical Software Engineering 19, 4 (2014), 1040–1074.
[10]
Mariano Ceccato, Massimiliano Di Penta, Jasvir Nagra, Paolo Falcarin, Filippo Ricca, Marco Torchiano, and Paolo Tonella. 2009. The effectiveness of source code obfuscation: An experimental assessment. In 2009 IEEE 17th International Conference on Program Comprehension. IEEE. https://doi.org/10.1109/icpc.2009.5090041
[11]
Mariano Ceccato, Alessandro Marchetto, Leonardo Mariani, Cu D Nguyen, and Paolo Tonella. 2015. Do automatically generated test cases make debugging easier? an experimental assessment of debugging effectiveness and efficiency. ACM Transactions on Software Engineering and Methodology (TOSEM) 25, 1(2015), 1–38.
[12]
Mariano Ceccato and Riccardo Scandariato. 2016. Static analysis and penetration testing from the perspective of maintenance teams. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery, New York, NY, USA, 1–6.
[13]
J. Cohen. 1988. Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates, Hillsdale, NJ.
[14]
Datanyze. 2021. OpenSSL Market Share and Competitor Report https://www.datanyze.com/market-share/other-it-infrastructure-software.
[15]
Datanyze. 2021. Web and Application Servers Market Share Report https://www.datanyze.com/market-share/web-and-application-servers.
[16]
Jay L. Devore. 2007. Probability and Statistics for Engineering and the Sciences. Duxbury Press; 7 edition.
[17]
Thomas Dierks and Eric Rescorla. 2008. The Transport Layer Security (TLS) Protocol Version 1.2 http://www.rfc-editor.org/rfc/rfc5246.txt. Internet Requests for Comments.
[18]
Alban Diquet. 2021. Github: sslyze https://github.com/nabla-c0d3/sslyze.
[19]
Association for Computing Machinery. 2018. ACM Code of Ethics and Professional Conduct https://www.acm.org/binaries/content/assets/about/acm-code-of-ethics-booklet.pdf.
[20]
Martin Georgiev, Subodh Iyengar, Suman Jana, Rishita Anubhai, Dan Boneh, and Vitaly Shmatikov. 2012. The most dangerous code in the world. In Proceedings of the 2012 ACM conference on Computer and communications security - CCS ’12. ACM Press. https://doi.org/10.1145/2382196.2382204
[21]
B. G. Glaser and A. L. Strauss. 1967. The Discovery of Grounded Theory. Aldine, Chicago.
[22]
Y. Gluck, N. Harris, and A. Prado. 2012. BREACH: reviving the CRIME attack http://breachattack.com/.
[23]
Peter Leo Gorski, Luigi Lo Iacono, Dominik Wermke, Christian Stransky, Sebastian Möller, Yasemin Acar, and Sascha Fahl. 2018. Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse https://www.usenix.org/conference/soups2018/presentation/gorski. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018). USENIX Association, Baltimore, MD, 265–281.
[24]
Matthew Green. 2011. A diversion: BEAST Attack on TLS/SSL Encryption https://blog.cryptographyengineering.com/2011/09/21/brief-diversion-beast-attack-on-tlsssl/.
[25]
Robert J. Grissom and John J. Kim. 2005. Effect sizes for research: A broad practical approach (2nd edition ed.). Lawrence Earlbaum Associates.
[26]
Norman Hänsch, Andrea Schankin, Mykolai Protsenko, Felix Freiling, and Zinaida Benenson. 2018. Programming Experience Might Not Help in Comprehending Obfuscated Source Code Efficiently https://www.usenix.org/conference/soups2018/presentation/hansch. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018). USENIX Association, Baltimore, MD, 341–356.
[27]
Scott Hollenbeck. 2004. Transport Layer Security Protocol Compression Methods http://www.rfc-editor.org/rfc/rfc3749.txt. Internet Requests for Comments.
[28]
Martin Höst, Björn Regnell, and Claes Wohlin. 2000. Using students as subjects—a comparative study of students and professionals in lead-time impact assessment. Empirical Software Engineering 5, 3 (2000), 201–214.
[29]
ISO 9241-11 2018. ISO 9241. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts.
[30]
Jiming Jiang. 2007. Linear and generalized linear mixed models and their applications. Springer Science & Business Media.
[31]
Stephen Farrell Kathleen Moriarty, CIS. 2021. Deprecating TLSv1.0 and TLSv1.1 https://tools.ietf.org/html/rfc8996.
[32]
Mak Kolybabi and Gabriel Lawrence. 2020. ssl-enum-ciphers https://nmap.org/nsedoc/scripts/ssl-enum-ciphers.html.
[33]
Katharina Krombholz, Karoline Busse, Katharina Pfeffer, Matthew Smith, and Emanuel von Zezschwitz. 2019. ”If HTTPS Were Secure, I Wouldn’t Need 2FA” - End User and Administrator Mental Models of HTTPS. In 2019 IEEE Symposium on Security and Privacy (SP). 246–263.
[34]
Katharina Krombholz, Wilfried Mayer, Martin Schmiedecker, and Edgar Weippl. 2017. ”I Have No Idea What I’m Doing” - On the Usability of Deploying HTTPS https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/krombholz. In 26th USENIX Security Symposium (USENIX Security 17). USENIX Association, Vancouver, BC, 1339–1356.
[35]
Anton Kühberger, Astrid Fritz, Eva Lermer, and Thomas Scherndl. 2015. The significance fallacy in inferential statistics. BMC research notes 8, 1 (2015), 84.
[36]
Katsiaryna Labunets, Fabio Massacci, Federica Paci, Sabrina Marczak, and Flávio Moreira de Oliveira. 2017. Model comprehension for security risk assessment: an empirical comparison of tabular vs. graphical representations. Empirical Software Engineering 22, 6 (Feb 2017), 3017–3056. https://doi.org/10.1007/s10664-017-9502-8
[37]
Katsiaryna Labunets, Fabio Massacci, Federica Paci, and Le Minh Sang Tran. 2013. An Experimental Comparison of Two Risk-Based Security Methods. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 163–172. https://doi.org/10.1109/esem.2013.29
[38]
Frank Li, Lisa Rogers, Arunesh Mathur, Nathan Malkin, and Marshini Chetty. 2019. Keepers of the Machines: Examining How System Administrators Manage Software Updates For Multiple Machines. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019). USENIX Association, Santa Clara, CA. https://www.usenix.org/conference/soups2019/presentation/li
[39]
Microsoft-Inria. 2014. Triple Handshakes Considered Harmful: Breaking and Fixing Authentication over TLS https://www.mitls.org/pages/attacks/3SHAKE.
[40]
Bodo Möller, Thai Duong, and Krzysztof Kotowicz. 2014. This POODLE Bites: Exploiting The SSL 3.0 Fallback https://www.openssl.org/~bodo/ssl-poodle.pdf.
[41]
Mozilla Security. 2018. Web Security Cheat Sheet https://infosec.mozilla.org/guidelines/web_security.
[42]
Alena Naiakshina, Anastasia Danilova, Eva Gerlitz, Emanuel von Zezschwitz, and Matthew Smith. 2019. ”If You Want, I Can Store the Encrypted Password”: A Password-Storage Field Study with Freelance Developers https://doi.org/10.1145/3290605.3300370. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300370
[43]
Alena Naiakshina, Anastasia Danilova, Christian Tiefenau, Marco Herzog, Sergej Dechand, and Matthew Smith. 2017. Why Do Developers Get Password Storage Wrong? A Qualitative Usability Study https://doi.org/10.1145/3133956.3134082. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, New York, NY, USA, 311–328. https://doi.org/10.1145/3133956.3134082
[44]
NIST. 2012. CVE-2012-4929 https://nvd.nist.gov/vuln/detail/CVE-2012-4929.
[45]
NowSecure. 2017. Fully Validate SSL/TLS https://books.nowsecure.com/secure-mobile-development/en/sensitive-data/fully-validate-ssl-tls.html.
[46]
Juan C. Perez. 2016. SSL: Deceptively Simple, Yet Hard to Implement https://blog.qualys.com/product-tech/2016/12/12/ssl-deceptively-simple-yet-hard-to-implement.
[47]
Qualys. 2021. SSL Pulse https://www.ssllabs.com/ssl-pulse/.
[48]
Qualys. 2021. SSL Server Test https://www.ssllabs.com/ssltest/.
[49]
rbsec. 2017. sslscan https://github.com/rbsec/sslscan/releases/tag/1.11.11-rbsec.
[50]
Benjamin Saefken, Thomas Kneib, Clara-Sophie van Waveren, Sonja Greven, 2014. A unifying approach to the estimation of the conditional Akaike information in generalized linear mixed models. Electronic Journal of Statistics 8, 1 (2014), 201–225.
[51]
Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015. Are Students Representatives of Professionals in Software Engineering Experiments?(ICSE ’15). IEEE Press, Florence, Italy, 666–676.
[52]
Riccardo Scandariato, James Walden, and Wouter Joosen. 2013. Static analysis versus penetration testing: A controlled experiment. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE. https://doi.org/10.1109/issre.2013.6698898
[53]
Bruce Schneier. 1999. Attack Trees https://www.schneier.com/academic/archives/1999/12/attack_trees.html.
[54]
Security & Trust Research Unit. [n.d.]. TLSAssistant https://github.com/stfbk/tlsassistant.
[55]
Amazon Web Services. 2021. Alexa Top Sites https://aws.amazon.com/alexa-top-sites/.
[56]
Yaron Sheffer, Ralph Holz, and Peter Saint-Andre. 2015. Summarizing Known Attacks on Transport Layer Security (TLS) and Datagram TLS (DTLS) http://www.rfc-editor.org/rfc/rfc7457.txt. Internet Requests for Comments.
[57]
David J. Sheskin. 2007. Handbook of Parametric and Nonparametric Statistical Procedures (4th Ed.). Chapman & All.
[58]
Raul Siles. 2013. TLSSLed v1.3 http://blog.taddong.com/2013/02/tlssled-v13.html.
[59]
Janet M. Six and Ritch Macefield. 2016. How to determine the right number of participants for usability studies https://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-participants-for-usability-studies.php.
[60]
A. Strauss and J. Corbin. 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage, London.
[61]
Mikael Svahnberg, Aybüke Aurum, and Claes Wohlin. 2008. Using Students as Subjects - an Empirical Evaluation https://doi.org/10.1145/1414004.1414055. In Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement(Kaiserslautern, Germany) (ESEM ’08). Association for Computing Machinery, New York, NY, USA, 288–290. https://doi.org/10.1145/1414004.1414055
[62]
Christian Tiefenau, Maximilian Häring, Katharina Krombholz, and Emanuel von Zezschwitz. 2020. Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators. In Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020). USENIX Association, 239–258. https://www.usenix.org/conference/soups2020/presentation/tiefenau
[63]
Alessio Viticchié, Leonardo Regano, Cataldo Basile, Marco Torchiano, Mariano Ceccato, and Paolo Tonella. 2020. Empirical assessment of the effort needed to attack programs protected with client/server code splitting. Empirical Software Engineering 25, 1 (2020), 1–48.
[64]
Alessio Viticchie, Leonardo Regano, Marco Torchiano, Cataldo Basile, Mariano Ceccato, Paolo Tonella, and Roberto Tiella. 2016. Assessment of Source Code Obfuscation Techniques. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, Los Alamitos, CA, USA, 11–20. https://doi.org/10.1109/scam.2016.17
[65]
Dirk Wetter. 2021. /bin/bash based SSL/TLS tester: testssl.sh https://testssl.sh.

Cited By

View all
  • (2022)Demo: TLSAssistant v2: A Modular and Extensible Framework for Securing TLSProceedings of the 27th ACM on Symposium on Access Control Models and Technologies10.1145/3532105.3535042(271-272)Online publication date: 7-Jun-2022
  • (2022)A Modular and Extensible Framework for Securing TLSProceedings of the Twelfth ACM Conference on Data and Application Security and Privacy10.1145/3508398.3511505(119-124)Online publication date: 14-Apr-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ARES '21: Proceedings of the 16th International Conference on Availability, Reliability and Security
August 2021
1447 pages
ISBN:9781450390514
DOI:10.1145/3465481
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. TLS misconfiguration
  2. actionable mitigations
  3. security reports
  4. usability study
  5. vulnerability detection

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ARES 2021

Acceptance Rates

Overall Acceptance Rate 228 of 451 submissions, 51%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Demo: TLSAssistant v2: A Modular and Extensible Framework for Securing TLSProceedings of the 27th ACM on Symposium on Access Control Models and Technologies10.1145/3532105.3535042(271-272)Online publication date: 7-Jun-2022
  • (2022)A Modular and Extensible Framework for Securing TLSProceedings of the Twelfth ACM Conference on Data and Application Security and Privacy10.1145/3508398.3511505(119-124)Online publication date: 14-Apr-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media