Skip to main content
SpringerLink
Log in

A graph-based model for malware detection and classification using system-call groups

  • Original Paper
  • Published:
Journal of Computer Virology and Hacking Techniques Aims and scope Submit manuscript

Abstract

In this paper we present a graph-based model that, utilizing relations between groups of System-calls, detects whether an unknown software sample is malicious or benign, and classifies a malicious software to one of a set of known malware families. More precisely, we utilize the System-call Dependency Graphs (or, for short, ScD-graphs), obtained by traces captured through dynamic taint analysis. We design our model to be resistant against strong mutations applying our detection and classification techniques on a weighted directed graph, namely Group Relation Graph, or Gr-graph for short, resulting from ScD-graph after grouping disjoint subsets of its vertices. For the detection process, we propose the \(\Delta \)-similarity metric, and for the process of classification, we propose the SaMe-similarity and NP-similarity metrics consisting the SaMe-NP similarity. Finally, we evaluate our model for malware detection and classification showing its potentials against malicious software measuring its detection rates and classification accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Alazab, M., Layton, R., Venkataraman, S., Watters, P.: Malware detection based on structural and behavioural features of API calls. In: Proceedings of the 1st Int’l Conference on Cyber Resilience (CR’10), pp. 1–10 (2010)

  2. Babic, D., Reynaud, D., Song, D.: Malware analysis with tree automata inference. In: Proceedings of the 23rd Int’l Conference on Computer Aided Verification (CAV’11), pp. 116–131 (2011)

  3. Bayer, U., Comparetti, P.M., Hlauschek, C., Kruegel, C., Kirda, E.: Scalable behavior-based malware clustering. In: Proceedings of the 16th Annual Network and Distributed System Security Symposium (NDSS’09), pp. 8–11 (2009)

  4. Bayer, U., Moser, A.C., Kruegel, C., Kirda, E.: Dynamic analysis of malicious code. J. Comput. Virol. 2, 67–77 (2006)

    Article  Google Scholar 

  5. Canfora, G., Mercaldo, F., Visaggio, C.A., Di Notte, P.: Metamorphic malware detection using code metrics. Inf. Secur. J. 23, 57–67 (2014)

    Google Scholar 

  6. Canfora, G., Iannaccone, A.N., Visaggio, C.A.: Static analysis for the detection of metamorphic computer viruses using repeated-instructions counting heuristics. J. Comput. Virol. Hacking Tech. 10, 11–27 (2014)

    Article  Google Scholar 

  7. Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: Proceedings of the 1st ACM India Software Engineering Conference (ISEC’08), pp. 5–14 (2008)

  8. Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: Proceedings of the 25th IEEE Symposium on Security and Privacy (SP’05), pp. 32–46 (2005)

  9. Fredrikson, M., Jha, S., Christodorescu, M., Sailer, R., Yan, X.: Synthesizing near-optimal malware specifications from suspicious behaviors. In: Proceedings of the 30th IEEE Symposium on Security and Privacy (SP’10), pp. 45–60 (2010)

  10. Islam, R., Tian, R., Batten, L., and Versteeg S.: Classification of malware based on string and function feature selection. In: Proceedings of the Cybercrime and Trustworthy Computing and Workshop (CTC’10), pp. 9–17 (2010)

  11. Kolbitsch, C., Comparetti, C.P.M., Kruegel, C., Kirda, E., Zhou, X.Y., Wang, X.: Effective and efficient malware detection at the end host. In: Proceedings of the 18th USENIX Security Symposium (USENIX Security’09), pp. 351–366 (2009)

  12. Kong, D., Yan, G.: Discriminant malware distance learning on structural information for automated malware classification. In Proceedings of the 19th ACM SIGKDD Int’l Conference on Knowledge Discovery and Data Mining (KDD’13), pp. 1357–1365 (2013)

  13. Luh, R., Tavolato, P.: Behavior-based malware recognition. Technical Report, St. Polten University of Applied Sciences, TR-79-84 (2012)

  14. Mathur, K., Hiranwal, S.: A survey on techniques in detection and analyzing malware executables. J. Adv. Res. Comput. Sci. Softw. Eng. 3, 22–428 (2013)

    Google Scholar 

  15. Mohaisen, A., Alrawi, O.: Unveiling zeus: automated classification of malware samples. In: Proceedings of the 22nd Int’l Conference on World Wide Web Companion (WWW’13), pp. 829–832 (2013)

  16. Mungale, M., Mark, S.: Software similarity and metamorphic detection. In: Proceedings of the 11th Intl Conference on Security and Management (SAM12) (2012)

  17. Nataraj, L., Karthikeyan, S., Jacob, G., Manjunath, B.S.: Malware images: visualization and automatic classification. In: Proceedings of the 8th Int’l Symposium on Visualization for Cyber Security (VizSec’11), pp. 4–11 (2011)

  18. Nataraj, L., Karthikeyan, S., Jacob, G., Manjunath, B. S.: A comparative assessment of malware classification using binary texture analysis and dynamic analysis. In: Proceedings of the 4th ACM workshop on Security and Artificial Intelligence, pp. 21–30 (2011)

  19. Newsome, J., Song, D.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS05) (2005)

  20. Nikolopoulos, S.D., Polenakis, I.: Detecting malicious code by exploiting dependencies of system-call groups, Cornell University Library. arXiv:1412.8712v1 (2014)

  21. Park, Y., Reeves, D., Mulukutla, V., Sundaravel, B.: Fast malware classification by automated behavioral graph matching. In: Proceedings of the 6th ACM Annual Workshop on Cyber Security and Information Intelligence Research (CSIIRW’10), pp. 45–49 (2010)

  22. Rad, B.B., Maslin, M., Suhaimi, I.: Camouflage in malware: from encryption to metamorphism. J. Comput. Sci. Netw. Secur. 12, 74–83 (2012)

    Google Scholar 

  23. Rieck, K., Thorsten, H., Carsten, W., Patrick, D., Laskov, P.: Learning and classification of malware behavior. In: Proceedings of the 5th Conference on Detection of Intrusions and Malware and Vulnerability Assessment (DIMVA’08), pp. 108–125 (2008)

  24. Sikorski, M., Honig, A.: Practical malware analysis: the hands-on guide to dissecting malicious software. No Starch Press, USA (2012)

  25. Szor, P., Ferrie, P.: Hunting for metamorphic. In: Virus Bulletin Conference (VB’01) (2001)

  26. Tian, R., Batten, L.M., Versteeg, S.C.: Function length as a tool for malware classification. In: Proceedings of the 3rd Int’l Conference on Malicious and Unwanted Software (MALWARE’08), pp. 69–76 (2008)

  27. Hu, X., Chiueh, T., Shin, K. G.: Large-scale malware indexing using function-call graphs. In: Proceedings of the 16th ACM Conference on Computer and Communications Security (CCS’09), pp. 611–620 (2009)

  28. Yan, G., Brown, N., Kong, D.: Exploring discriminatory features for automated malware classification. In: Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA’13), pp. 41–61(2013)

  29. Ye, Y., Dingding, W., Tao, L. Dongyi, Y.: IMDS: Intelligent malware detection system. In: Proceedings of the 13th ACM Int’l Conference on Knowledge Discovery and Data Mining (SIGKDD’07), pp. 1043–1047 (2007)

  30. You, I., Yim, K.: Malware obfuscation techniques: a brief survey. In: Proceedings of the 5th Int’l Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA’10), pp. 297–300 (2010)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Ioannina, 45110, Ioannina, Greece

    Stavros D. Nikolopoulos & Iosif Polenakis

Authors
  1. Stavros D. Nikolopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iosif Polenakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stavros D. Nikolopoulos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nikolopoulos, S.D., Polenakis, I. A graph-based model for malware detection and classification using system-call groups. J Comput Virol Hack Tech 13, 29–46 (2017). https://doi.org/10.1007/s11416-016-0267-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-016-0267-1

Keywords

Search

Navigation