skip to main content
research-article

Maat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective Malware Detection

Published: 19 July 2021 Publication History

Abstract

The malware analysis and detection research community relies on the online platform VirusTotal to label Android apps based on the scan results of around 60 antiviral scanners. Unfortunately, there are no standards on how to best interpret the scan results acquired from VirusTotal, which leads to the utilization of different threshold-based labeling strategies (e.g., if 10 or more scanners deem an app malicious, it is considered malicious). While some of the utilized thresholds may be able to accurately approximate the ground truths of apps, the fact that VirusTotal changes the set and versions of the scanners it uses makes such thresholds unsustainable over time. We implemented a method, Maat, that tackles these issues of standardization and sustainability by automatically generating a Machine Learning (ML)-based labeling scheme, which outperforms threshold-based labeling strategies. Using the VirusTotal scan reports of 53K Android apps that span 1 year, we evaluated the applicability of Maat’s Machine Learning (ML)-based labeling strategies by comparing their performance against threshold-based strategies. We found that such ML-based strategies (a) can accurately and consistently label apps based on their VirusTotal scan reports, and (b) contribute to training ML-based detection methods that are more effective at classifying out-of-sample apps than their threshold-based counterparts.

References

[1]
2019. K-9 Mail—Advanced Email for Android. Retrieved on June 2019 from http://tiny.cc/1fbb7y.
[2]
Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016. Androzoo: Collecting millions of android apps for the research community. In Proceedings of the 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR’16). IEEE, 468–471.
[3]
Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, and Konrad Rieck. 2014. DREBIN: Effective and explainable detection of android malware in your pocket. In NDSS.
[4]
Saba Arshad, Munam Ali Shah, Abid Khan, and Mansoor Ahmed. 2016. Android malware detection and protection: A survey. International Journal of Advanced Computer Science and Applications 7 (2016), 463–475.
[5]
Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 1 (2020), 6.
[6]
Ken Dunham, Shane Hartman, Manu Quintans, Jose Andre Morales, and Tim Strazzere. 2014. Android Malware and Analysis. Auerbach Publications.
[7]
Roberto Jordaney Johannes Kinder Feargus Pendlebury, Fabio Pierazzi and Lorenzo Cavallaro. 2019. TESSERACT: Eliminating experimental bias in malware classification across space and time. In 28th USENIX Security Symposium. USENIX Association, Santa Clara, CA.
[8]
Ali Feizollah, Nor Badrul Anuar, Rosli Salleh, and Ainuddin Wahid Abdul Wahab. 2015. A review on feature selection in mobile malware detection. Digital Investigation 13 (2015), 22–37.
[9]
Médéric Hurier, Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016. On the lack of consensus in anti-virus decisions: Metrics and insights on building ground truths of android malware. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 142–162.
[10]
Médéric Hurier, Guillermo Suarez-Tangil, Santanu Kumar Dash, Tegawendé F. Bissyandé, Yves Le Traon, Jacques Klein, and Lorenzo Cavallaro. 2017. Euphony: Harmonious unification of cacophonous anti-virus vendor labels for android malware. In Proceedings of the 14th International Conference on Mining Software Repositories. IEEE Press, 425–435.
[11]
AV-Test: The Independent IT-Security Institute. 2019. The best antivirus software for Android. Retrieved on June 2019 from http://tiny.cc/1k66az.
[12]
Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D. Joseph, and J. Doug Tygar. 2015. Better malware ground truth: Techniques for weighting anti-virus vendor labels. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security. ACM, 45–56.
[13]
Tom Kelchner. 2010. The (in) consistent naming of malcode. Computer Fraud & Security 2010, 2 (2010), 5–7.
[14]
Li Li, Daoyuan Li, Tegawendé F. Bissyandé, Jacques Klein, Yves Le Traon, David Lo, and Lorenzo Cavallaro. 2017. Understanding android app piggybacking: A systematic study of malicious code grafting. IEEE Transactions on Information Forensics and Security 12, 6 (2017), 1269–1284.
[15]
Symphony Luo and Peter Yan. 2014. Fake Apps: Feigning Legitimacy. Retrieved on April 2019 from https://goo.gl/hzZBiH.
[16]
PC Magazine. 2008. BitDefender Free Edition. Retrieved on June 2019 from http://tiny.cc/8vx9az.
[17]
Federico Maggi, Andrea Bellini, Guido Salvaneschi, and Stefano Zanero. 2011. Finding non-trivial malware naming inconsistencies. In International Conference on Information Systems Security. Springer, 144–159.
[18]
Brad Miller, Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Rekha Bachwani, Riyaz Faizullabhoy, Ling Huang, Vaishaal Shankar, Tony Wu, George Yiu, et al. 2016. Reviewer integration and performance measurement for malware detection. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 122–141.
[19]
Aziz Mohaisen and Omar Alrawi. 2014. AV-meter: An evaluation of antivirus scans and labels. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 112–131.
[20]
Aziz Mohaisen, Omar Alrawi, Matt Larson, and Danny McPherson. 2013. Towards a methodical evaluation of antivirus scans and labels. In International Workshop on Information Security Applications. Springer, 231–241.
[21]
Jon Oberheide and Charlie Miller. 2012. Dissecting the android bouncer. SummerCon2012, New York.
[22]
Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2018. Enabling fair ML evaluations for security. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2264–2266.
[23]
Peng Peng, Limin Yang, Linhai Song, and Gang Wang. 2019. Opening the blackbox of virustotal: Analyzing online phishing scan engines. In Proceedings of the Internet Measurement Conference. ACM, 478–485.
[24]
Roberto Perdisci et al. 2012. VAMO: Towards a fully automated malware clustering validity analysis. In Proceedings of the 28th Annual Computer Security Applications Conference. ACM, 329–338.
[25]
Neil J. Rubenking. 2015. False Positives Sink Antivirus Ratings. Retrieved on June 2019 from http://tiny.cc/eyh2uz.
[26]
Shefali Sachdeva, Romuald Jolivot, and Worawat Choensawat. 2018. Android malware classification based on mobile security framework. IAENG International Journal of Computer Science 45, 4 (2018).
[27]
Aleieldin Salem and Alexander Pretschner. 2018. Poking the bear: Lessons learned from probing three android malware datasets. In Proceedings of the 1st International Workshop on Advances in Mobile App Analysis. ACM, 19–24.
[28]
Hillary Sanders and Joshua Saxe. 2017. Garbage in, garbage out: How purportedly great ML models can be screwed up by bad data. Technical Report.
[29]
Borja Sanz, Igor Santos, Carlos Laorden, Xabier Ugarte-Pedrero, and Pablo Garcia Bringas. 2012. On the automatic categorisation of android applications. In Proceedings of the 2012 IEEE Consumer Communications and Networking Conference (CCNC’12). IEEE, 149–153.
[30]
Borja Sanz, Igor Santos, Carlos Laorden, Xabier Ugarte-Pedrero, Pablo Garcia Bringas, and Gonzalo Álvarez. 2013. Puma: Permission usage to detect malware in android. In International Joint Conference CISIS’12-ICEUTE’12-SOCO’12 Special Sessions. Springer, 289–298.
[31]
Ryo Sato, Daiki Chiba, and Shigeki Goto. 2013. Detecting android malware by analyzing manifest files. Proceedings of the Asia-Pacific Advanced Network 36 (2013), 23–31.
[32]
scikit learn. 2019. Feature Selection. Retrieved on October 2019 from http://tiny.cc/2d997y.
[33]
scikit learn. 2019. GridSearchCV. Retrieved on October 2019 from http://tiny.cc/mquy9y.
[34]
Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. 2016. AVclass: A tool for massive malware labeling. In International Symposium on Research in Attacks, Intrusions, and Defenses. Springer, 230–253.
[35]
Tim Stazzere. 2016. Detecting pirated and malicious android apps with apkid. Retrieved from https://goo.gl/Na2ZAX.
[36]
Guillermo Suarez-Tangil, Santanu Kumar Dash, Mansour Ahmadi, Johannes Kinder, Giorgio Giacinto, and Lorenzo Cavallaro. 2017. DroidSieve: Fast and accurate classification of obfuscated android malware. In Proceedings of the 7th ACM Conference on Data and Application Security and Privacy. ACM, 309–320.
[37]
Kimberly Tam, Ali Feizollah, Nor Badrul Anuar, Rosli Salleh, and Lorenzo Cavallaro. 2017. The evolution of android malware and android analysis techniques. ACM Computing Surveys (CSUR) 49, 4 (2017), 76.
[38]
VirusTotal. 2019. VirusTotal. Retrieved on September 2019 from http://tiny.cc/xjbb7y.
[39]
Haoyu Wang, Zhe Liu, Jingyue Liang, Narseo Vallina-Rodriguez, Yao Guo, Li Li, Juan Tapiador, Jingcun Cao, and Guoai Xu. 2018. Beyond Google Play: A large-scale comparative study of Chinese android app markets. In Proceedings of the Internet Measurement Conference 2018. ACM, 293–307.
[40]
Ting Wang, Shicong Meng, Wei Gao, and Xin Hu. 2014. Rebuilding the tower of babel: Towards cross-system malware information sharing. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 1239–1248.
[41]
Fengguo Wei, Yuping Li, Sankardas Roy, Xinming Ou, and Wu Zhou. 2017. Deep ground truth analysis of current android malware. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 252–276.
[42]
Dong-Jie Wu, Ching-Hao Mao, Te-En Wei, Hahn-Ming Lee, and Kuo-Ping Wu. 2012. Droidmat: Android malware detection through manifest and api calls tracing. In Proceedings of the 2012 7th Asia Joint Conference on Information Security (Asia JCIS’12). IEEE, 62–69.
[43]
Wei Yang, Deguang Kong, Tao Xie, and Carl A. Gunter. 2017. Malware detection in adversarial settings: Exploiting feature evolutions and confusions in android apps. In Proceedings of the 33rd Annual Computer Security Applications Conference. ACM, 288–302.
[44]
Wu Zhou, Yajin Zhou, Michael Grace, Xuxian Jiang, and Shihong Zou. 2013. Fast, scalable detection of piggybacked mobile applications. In Proceedings of the 3rd ACM Conference on Data and Application Security and Privacy. ACM, 185–196.
[45]
Yajin Zhou and Xuxian Jiang. 2012. Dissecting android malware: Characterization and evolution. In 2012 IEEE Symposium on Security and Privacy (SP’12). IEEE, 95–109.
[46]
Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, and Gang Wang. 2020. Measuring and modeling the label dynamics of online anti-malware engines. In 29th USENIX Security Symposium (USENIX Security 20).

Cited By

View all
  • (2025)Analysis of Malicious Files Gathering via Honeypot Trap System and Benchmark of Anti-Virus SoftwareBalkan Journal of Electrical and Computer Engineering10.17694/bajece.150655412:4(337-348)Online publication date: 13-Jan-2025
  • (2024)Optimal Weighted Voting-Based Collaborated Malware Detection for Zero-Day Malware: A Case Study on VirusTotal and MalwareBazaarFuture Internet10.3390/fi1608025916:8(259)Online publication date: 23-Jul-2024
  • (2024)MLPhishChain: a machine learning-based blockchain framework for reducing phishing threatsFrontiers in Blockchain10.3389/fbloc.2024.14848947Online publication date: 12-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Privacy and Security
ACM Transactions on Privacy and Security  Volume 24, Issue 4
November 2021
295 pages
ISSN:2471-2566
EISSN:2471-2574
DOI:10.1145/3476876
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2021
Accepted: 01 May 2021
Revised: 01 April 2021
Received: 01 July 2020
Published in TOPS Volume 24, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Android security
  2. machine learning
  3. malware detection

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)113
  • Downloads (Last 6 weeks)10
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Analysis of Malicious Files Gathering via Honeypot Trap System and Benchmark of Anti-Virus SoftwareBalkan Journal of Electrical and Computer Engineering10.17694/bajece.150655412:4(337-348)Online publication date: 13-Jan-2025
  • (2024)Optimal Weighted Voting-Based Collaborated Malware Detection for Zero-Day Malware: A Case Study on VirusTotal and MalwareBazaarFuture Internet10.3390/fi1608025916:8(259)Online publication date: 23-Jul-2024
  • (2024)MLPhishChain: a machine learning-based blockchain framework for reducing phishing threatsFrontiers in Blockchain10.3389/fbloc.2024.14848947Online publication date: 12-Dec-2024
  • (2024)A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLsACM SIGMETRICS Performance Evaluation Review10.1145/3673660.365504252:1(55-56)Online publication date: 13-Jun-2024
  • (2024)A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLsAbstracts of the 2024 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3652963.3655042(55-56)Online publication date: 10-Jun-2024
  • (2024)CySecBERT: A Domain-Adapted Language Model for the Cybersecurity DomainACM Transactions on Privacy and Security10.1145/365259427:2(1-20)Online publication date: 8-Apr-2024
  • (2024)Uncovering and Mitigating the Impact of Code Obfuscation on Dataset Annotation with Antivirus EnginesProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680302(553-565)Online publication date: 11-Sep-2024
  • (2024)Comparative Analysis of YARA and VirusTotal Integrations with Wazuh for Enhanced Malware Detection2024 5th International Conference on Advanced Information Technologies (ICAIT)10.1109/ICAIT65209.2024.10754931(1-6)Online publication date: 7-Nov-2024
  • (2024)Software installation threat detection based on attention mechanism and improved convolutional neural network in IOT platformEngineering Research Express10.1088/2631-8695/ad612d6:3(035210)Online publication date: 18-Jul-2024
  • (2024)Evolving techniques in cyber threat huntingJournal of Network and Computer Applications10.1016/j.jnca.2024.104004232:COnline publication date: 1-Dec-2024
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media