Abstract
The TLS protocol is the standard for secure Internet communication between two parties. Unfortunately, there have been recently successful attacks like DROWN, ROBOT, or BREACH that indicate the necessity for thoroughly testing TLS implementations. In our research work, we focus on automated test case generation and execution for the TLS security protocol, where the aim is to combine planning with combinatorial methods for providing test cases that ideally also reveal previously unknown attacks. This is made feasible by creating appropriate input parameter models for different messages that can appear in a TLS message sequence. In this paper, we present the resulting test case generation and execution framework together with the corresponding test oracle. Furthermore, we discuss in detail empirical results obtained via testing different TLS implementations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
AlFardan, N., & Paterson, K.G. (2012). Plaintext-recovery attacks against datagram tls. In Network and distributed system security symposium (NDSS 2012).
Berbecaru, D., & Lioy, A. (2007). On the robustness of applications based on the ssl and tls security protocol. In European Public key infrastructure workshop (pp. 248–264). Springer.
Beurdouche, B., Bhargavan, K., Delignat-Lavaud, A., Fournet, C., Kohlweiss, M., Pironti, A., Strub, P.Y., Zinzindohoue, J.K. (2015). A messy state of the union: taming the composite state machines of tls. In Proceedings of the 36th IEEE symposium on security and privacy.
Beurdouche, B., Delignat-Lavaud, A., Kobeissi, N., Pironti, A., Bhargavan, K. (2015). Flextls: a tool for testing tls implementations. In 9th USENIX Workshop on offensive technologies (WOOT’15).
Bhargavan, K., & Leurent, G. (2016). Transcript collision attacks: breaking authentication in tls, ike, and ssh. In Network and distributed system security symposium–NDSS 2016.
Bhargavan, K., Lavaud, A.D., Fournet, C., Pironti, A., Strub, P.Y. (2014). Triple handshakes and cookie cutters: breaking and fixing authentication over tls. In 2014 IEEE Symposium on security and privacy (SP) (pp. 98–113). IEEE.
Blum, A., & Furst, M. (1995). Fast planning through planning graph analysis. In IJCAI95 (pp. 1636–1642).
Bozic, J., & Wotawa, F. (2014). Plan it! automated security testing based on planning. In Proceedings of the 26th IFIP WG 6.1 international conference (ICTSS’14) (pp. 48–62).
Bozic, J., & Wotawa, F. (2015). Purity: a planning-based security testing tool. In 2015 IEEE International conference on software quality, reliability and security-companion (QRS-C) (pp. 46–55).
Bozic, J., Kleine, K., Simos, D.E., Wotawa, F. (2017). Planning-based security testing of the SSL/TLS protocol. In Proceedings of the IEEE international conference on software testing, verification and validation workshops (ICSTW).
Brubaker, C., Jana, S., Ray, B., Khurshid, S., Shmatikov, V. (2014). Using frankencerts for automated adversarial testing of certificate validation in ssl/tls implementations. In Proceedings of the 2014 IEEE symposium on security and privacy.
de Ruiter, J., & Poll, E. (2015). Protocol state fuzzing of tls implementations. In 24th USENIX Security Symposium (USENIX Security 15) (pp. 193–206).
Dierks, T., & Rescorla, E. (2008). Rfc 5246: the transport layer security (tls) protocol. The Internet Engineering Task Force.
Dowling, B., Fischlin, M., Günther, F., Stebila, D. (2015). A cryptographic analysis of the tls 1.3 handshake protocol candidates. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1197–1210). ACM.
Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R. (2017). Optimizing ipog’s vertical growth with constraints based on hypergraph coloring. In 2017 IEEE International Conference on software testing, verification and validation workshops (ICSTW) (pp. 181–188). IEEE.
Galler, S.J., Zehentner, C., Wotawa, F. (2010). Aiana: an ai planning system for test data generation. In 1st Workshop on testing object-oriented software systems (pp. 30–37).
Hollenbeck, S. (2004). Transport layer security protocol compression methods.
Jager, T., Schwenk, J., Somorovsky, J. (2015). Practical invalid curve attacks on tls-ecdh. In European Symposium on research in computer security (pp. 407–425). Springer.
Kleine, K., & Simos, D.E. (2017). Coveringcerts: combinatorial methods for x.509 certificate testing. In 2017 IEEE International conference on software testing, verification and validation (ICST) (pp. 69–79).
Krawczyk, H., Paterson, K.G., Wee, H. (2013). On the security of the tls protocol: a systematic analysis. In CRYPTO.
Kuhn, R., Lei, Y., Kacker, R. (2008). Practical combinatorial testing: beyond pairwise. It Professional 10(3).
Kuhn, D.R., Bryce, R., Duan, F., Ghandehari, L.S., Lei, Y., Kacker, R.N. (2015). Chapter one-combinatorial testing: theory and practice. Advances in Computers, 99, 1–66.
Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J. (2008). Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability, 18(3), 125–148.
Leitner, A., & Bloem, R. (2005). Automatic testing through planning. Tech. rep. Technische Universität Graz, Institute for Software Technology.
Mavrogiannopoulos, N., Vercauteren, F., Velichkov, V., Preneel, B. (2012). A cross-protocol attack on the tls protocol. In ACM CCS 12: 19th Conference on computer and communications security.
McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D. (1998). Pddl - the planning domain definition language. In The AIPS-98 planning competition comitee.
Memon, A.M., Pollack, M.E., Soffa, M.L. (2000). A planning-based approach to gui testing. In Proceedings of the 13th international software / internet quality week (QW’00).
Meyer, C., Somorovsky, J., Weiss, E., Schwenk, J., Schinzel, S., Tews, E. (2014). Revisiting ssl/tls implementations: new bleichenbacher side channels and attacks. In USENIX Security (Vol. 14, pp. 733–748).
Morais, A., Martins, E., Cavalli, A., Jimenez, W. (2009). Security protocol testing using attack trees. In CSE (2), IEEE Computer Society (2009) (pp. 690–697).
Raffelt, H., Steffen, B., Berg, T. (2005). Learnlib: a library for automata learning and experimentation. In Proceedings of the 10th international workshop on formal methods for industrial critical systems (FMICS’05) (pp. 62–71).
Russell, S.J., & Norvig, P. (1995). Artificial intelligence: a modern approach. Prentic Hall.
Shmaryahu, D., Shani, G., Hoffmann, J., Steinmetz, M. (2017). Partially observable contingent planning for penetration testing. In: Proceedings of first international workshop on AI in security (IWAIse).
Simos, D.E., Kuhn, R., Voyiatzis, A.G., Kacker, R. (2016). Combinatorial methods in security testing. IEEE Computer, 49, 40–43.
Simos, D.E., Bozic, J., Duan, F., Garn, B., Kleine, K., Lei, Y., Wotawa, F. (2017). Testing tls using combinatorial methods and execution framework. In Proceedings of the IFIP international conference on testing software and systems (ICTSS’17).
Somorovsky, J. (2016). Systematic fuzzing and testing of tls libraries. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security (CCS’16).
Walz, A., & Sikora, A. (2017). Exploiting dissent: towards fuzzing-based differential black box testing of tls implementations. IEEE Transactions on Dependable and Secure Computing, 99, 1–1.
Yadav, N., Thangarajah, J., Sardina, S. (2017). Agent design consistency checking via planning. In Proceedings of the twenty-sixth international joint conference on artificial intelligence (IJCAI-17).
Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R. (2013). Acts: a combinatorial test generation tool. In 2013 IEEE Sixth International Conference on software testing, verification and validation (ICST) (pp. 370–375). IEEE.
Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R.N., Kuhn, D.R. (2013). An efficient algorithm for constraint handling in combinatorial test generation. In 2013 IEEE Sixth International Conference on software testing, verification and validation (ICST) (pp. 242–251). IEEE.
Acknowledgements
The research presented in the paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grant 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and the Austrian COMET Program (FFG).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Simos, D.E., Bozic, J., Garn, B. et al. Testing TLS using planning-based combinatorial methods and execution framework. Software Qual J 27, 703–729 (2019). https://doi.org/10.1007/s11219-018-9412-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-018-9412-z