skip to main content
10.1145/3548659.3561307acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

KUBO: a framework for automated efficacy testing of anti-virus behavioral detection with procedure-based malware emulation

Published: 09 November 2022 Publication History

Abstract

Traditional testing of Anti-Virus (AV) products is usually performed on a curated set of malware samples. While this approach can evaluate an AV's overall performance on known threats, it fails to provide details on the coverage of exact attack techniques used by adversaries and malware. Such coverage information is crucial in helping users understand potential attack paths formed using new code and combinations of known attack techniques.
This paper describes KUBO, a framework for systematic large-scale testing of behavioral coverage of AV software. KUBO uses a novel malware behavior emulation method to generate a large number of attacks from combinations of adversarial procedures and runs them against a set of AVs. Contrary to other emulators, our attacks are coordinated by the adversarial procedures themselves, rendering the emulated malware independent of agents and semantically coherent.
We perform an evaluation of KUBO on 7 major commercial AVs utilizing tens of distinct attack procedures and thousands of their combinations. The results demonstrate that our approach is feasible, leads to automatic large-scale evaluation, and is able to unveil a multitude of open attack paths. We show how the results can be used to assess general behavioral efficacy and efficacy with respect to individual adversarial procedures.

References

[1]
Rawan Al-Shaer, Jonathan M. Spring, and Eliana Christou. 2020. Learning the Associations of MITRE ATT&CK Adversarial Techniques. arxiv:2005.01654.
[2]
AMTSO. 2019. Testing Protocol Standard. https://www.amtso.org/wp-content/uploads/2019/12/AMTSO-Testing-Protocol-Standard-for-the-Testing-of-Anti-Malware-Solutions-v1.3.pdf
[3]
Anonymous. 2018. Invoke-Adversary. https://github.com/CyberMonitor/Invoke-Adversary
[4]
Andy Applebaum, Doug Miller, Blake Strom, Chris Korban, and Ross Wolf. 2016. Intelligent, Automated Red Team Emulation. In Proceedings of the 32nd Annual Conference on Computer Security Applications. 363–373.
[5]
AV-Comparatives. 1999. AV-Comparatives. https://www.av-comparatives.org/
[6]
AV-Test. 2004. AV-Test. https://www.av-test.org
[7]
Zahra Bazrafshan, Hashem Hashemi, Seyed Mehdi Hazrati Fard, and Ali Hamzeh. 2013. A survey on heuristic malware detection techniques. In The 5th Conference on Information and Knowledge Technology. 113–120.
[8]
Fabrice Bellard. 2005. QEMU, a fast and portable dynamic translator. In USENIX annual technical conference, FREENIX Track. 41, 46.
[9]
Red Canary. 2017? Atomic Red Team. https://atomicredteam.io/
[10]
Red Canary. 2020. ChainReactor. https://redcanary.com/blog/chain-reactor-framework-for-linux/
[11]
Seungoh Choi, Jongwon Choi, Jeong-Han Yun, Byung-Gil Min, and HyoungChun Kim. 2020. Expansion of ICS Testbed for Security Validation based on MITRE ATT&CK Techniques. In 13th USENIX Workshop on Cyber Security Experimentation and Test, CSET 2020, August 10, 2020, Tamara Denning and Tyler Moore (Eds.). USENIX Association. https://www.usenix.org/conference/cset20/presentation/choi
[12]
Mihai Christodorescu and Somesh Jha. 2004. Testing malware detectors. ACM SIGSOFT Software Engineering Notes, 29, 4 (2004), 34–44.
[13]
Anusha Damodaran, Fabio Di Troia, Corrado Aaron Visaggio, Thomas H Austin, and Mark Stamp. 2017. A comparison of static, dynamic, and hybrid analysis for malware detection. Journal of Computer Virology and Hacking Techniques, 13, 1 (2017), 1–12.
[14]
Joseph Demarco. 2017. Invoke-Keylogger. https://gist.github.com/D3F4LT99/65d15c3c48da960b5e946a4f10e639df
[15]
H. M. Farooq and N. M. Otaibi. 2018. Optimal Machine Learning Algorithms for Cyber Threat Detection. In 2018 UKSim-AMSS 20th International Conference on Computer Modelling and Simulation (UKSim). 32–37. https://doi.org/10.1109/UKSim.2018.00018
[16]
Eric Filiol, Grégoire Jacob, and Mickaël Le Liard. 2007. Evaluation methodology and theoretical model for antiviral behavioural detection strategies. Journal in Computer Virology, 3, 1 (2007), 01 Apr, 23–37. issn:1772-9904 https://doi.org/10.1007/s11416-006-0026-9
[17]
Jameel Haffejee and Barry Irwin. 2014. Testing antivirus engines to determine their effectiveness as a security layer. In 2014 Information Security for South Africa. 1–6.
[18]
Nwokedi Idika and Aditya P Mathur. 2007. A survey of malware detection techniques. Purdue University, 48 (2007), 2007–2.
[19]
Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. KVM: the Linux virtual machine monitor. In Proceedings of the Linux symposium. 225–230.
[20]
Joxean Koret and Elias Bachaalany. 2015. The Antivirus Hacker’s Handbook (1st ed.). Wiley Publishing. isbn:1119028752
[21]
Guardicore Labs. 2021? Infection Monkey. https://www.guardicore.com/infectionmonkey/
[22]
ICSA Labs. 1989. ICSA Labs. https://www.icsalabs.com/
[23]
SE Labs. 2015. SE Labs. https://selabs.uk
[24]
West Coast Labs. 2015. West Coast Labs. http://www.westcoastlabs.com/
[25]
Valentine Legoy, Marco Caselli, Christin Seifert, and Andreas Peter. 2020. Automated Retrieval of ATT&CK Tactics and Techniques for Cyber Threat Reports. arxiv:2004.14322.
[26]
Guozhu Meng, Yinxing Xue, Chandramohan Mahinthan, Annamalai Narayanan, Yang Liu, Jie Zhang, and Tieming Chen. 2016. Mystique: Evolving android malware for auditing anti-malware tools. In Proceedings of the 11th ACM on Asia conference on computer and communications security. 365–376.
[27]
Mitre. 2020. Event Triggered Execution: Accessibility Features. https://attack.mitre.org/techniques/T1546/008/
[28]
MRG-Effitas. 2009. MRG-Effitas. https://www.mrg-effitas.com/
[29]
Elastic NV. 2018. Red Team Automation. https://github.com/endgameinc/RTA
[30]
Kris Oosthoek and Christian Doerr. 2019. SoK: ATT&CK Techniques and Trends in Windows Malware. In International Conference on Security and Privacy in Communication Systems. 406–425.
[31]
Martin Rosso, Michele Campobasso, Ganduulga Gankhuyag, and Luca Allodi. 2020. SAIBERSOC: Synthetic Attack Injection to Benchmark and Evaluate the Performance of Security Operation Centers. Annual Computer Security Applications Conference, Dec, isbn:9781450388580 https://doi.org/10.1145/3427228.3427233
[32]
Alireza Souri and Rahil Hosseini. 2018. A state-of-the-art survey of malware detection approaches using data mining techniques. Human-centric Computing and Information Sciences, 8, 1 (2018), 1–22.
[33]
Mark Stamp and S. Venkatachalam. 2011. Detecting Undetectable Metamorphic Viruses. In International Conference on Security & Management.
[34]
Blake E. Strom, Andy Applebaum, Douglas P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. 2018. MITRE ATT&CK™ : Design and Philosophy. Mitre.
[35]
Blake E. Strom, Joseph A. Battaglia, Michael S. Kemmerer, William Kupersanin, Douglas P. Miller, Craig Wampler, Sean M. Whitley, and Ross D. Wolf. 2017. Finding Cyber Threats with ATT&CK-Based Analytics. Mitre.
[36]
Orathai Sukwong, Hyong Kim, and James Hoe. 2011. Commercial antivirus software effectiveness: an empirical study. IEEE Computer Architecture Letters, 44, 03 (2011), 63–70.
[37]
Yusuke Takahashi, Shigeyoshi Shima, Rui Tanabe, and Katsunari Yoshioka. 2020. APTGen: An Approach towards Generating Practical Dataset Labelled with Targeted Attack Sequences. In 13th $USENIX$ Workshop on Cyber Security Experimentation and Test ($CSET$ 20).
[38]
Uber. 2017. Metta. https://github.com/uber-common/metta
[39]
Mauricio Velazco. 2020. PurpSharp. https://www.purplesharp.com/en/latest/
[40]
P Vinod, R Jaipur, V Laxmi, and M Gaur. 2009. Survey on malware detection methods. In Proceedings of the 3rd Hackers’ Workshop on computer and internet security (IITKHACK’09). 74–79.
[41]
VirusTotal. 2004. VirusTotal. https://www.virustotal.com
[42]
Winscripting. 2017. First entry: Welcome and fileless UAC bypass. https://winscripting.blog/2017/05/12/first-entry-welcome-and-uac-bypass/
[43]
Yinxing Xue, Guozhu Meng, Yang Liu, Tian Huat Tan, Hongxu Chen, Jun Sun, and Jie Zhang. 2017. Auditing anti-malware tools by evolving android malware and dynamic loading technique. IEEE Transactions on Information Forensics and Security, 12, 7 (2017), 1529–1544.
[44]
Ilsun You and Kangbin Yim. 2010. Malware obfuscation techniques: A brief survey. In 2010 International conference on broadband, wireless computing, communication and applications. 297–300.
[45]
Polina Zilberman, Rami Puzis, Sunders Bruskin, Shai Shwarz, and Yuval Elovici. 2020. SoK: A Survey of Open-Source Threat Emulators. arXiv preprint arXiv:2003.01518.

Cited By

View all
  • (2023)13th Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2022) co-located with ESEC/FSE conferenceACM SIGSOFT Software Engineering Notes10.1145/3573074.357309348:1(76-78)Online publication date: 17-Jan-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
A-TEST 2022: Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation
November 2022
63 pages
ISBN:9781450394529
DOI:10.1145/3548659
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. anti-virus testing
  2. attack emulation
  3. malware

Qualifiers

  • Research-article

Funding Sources

  • RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative

Conference

A-TEST '22
Sponsor:

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)23
  • Downloads (Last 6 weeks)5
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)13th Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2022) co-located with ESEC/FSE conferenceACM SIGSOFT Software Engineering Notes10.1145/3573074.357309348:1(76-78)Online publication date: 17-Jan-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media