Abstract
This article presents new contributions for Remote Access Trojan (RAT) analysis using symbolic execution techniques. The first part of the article identifies the challenges in the application of such an analysis, as well as the procedures put in place to address these challenges. The second part of the article presents a practical analysis of samples from known RAT families with the help of the SEMA toolchain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abuse.ch: Malwarebazaar (2023). https://bazaar.abuse.ch/
Afianian, A., Niksefat, S., Sadeghiyan, B., Baptiste, D.: Malware dynamic analysis evasion techniques: a survey. ACM Comput. Surv. 52(6), 1–28 (2019)
Aghakhani, H., et al.: When malware is packin’ heat; limits of machine learning classifiers based on static analysis features. In: Network and Distributed Systems Security (NDSS) Symposium 2020 (2020)
Amer, E., Zelinka, I.: A dynamic windows malware detection and prediction method based on contextual understanding of API call sequence. Comput. Secur. 92, 101760 (2020)
Bertrand Van Ouytsel, C.-H., Crochet, C., Legay, A., Lucca, S.: SEMA-ToolChain. GitHub. GitHub repository. https://github.com/csvl/SEMA-ToolChain
Avllazagaj, E., Zhu, Z., Bilge, L., Balzarotti, D., Dumitras, T.: When malware changed its mind: an empirical study of variable program behaviors in the real world. In: USENIX Security Symposium, pp. 3487–3504 (2021)
Baldoni, R., Coppa, E., D’Elia, D.C., Demetrescu, C.: Assisting malware analysis with symbolic execution: a case study. In: Dolev, S., Lodha, S. (eds.) Cyber Security Cryptography and Machine Learning. CSCML 2017. LNCS, vol. 10332, pp. 171–188. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60080-2_12
Bertrand Van Ouytsel, C.-H., Crochet, C., Dam, K.H.T., Legay, A.: Tool paper - SEMA: symbolic execution toolchain for malware analysis. In: Kallel, S., Jmaiel, M., Zulkernine, M., Hadj Kacem, A., Cuppens, F., Cuppens, N. (eds.) Risks and Security of Internet and Systems, CRiSIS 2022, pp. 62–68. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-31108-6_5
Bertrand Van Ouytsel, C., Legay, A.: Malware analysis with symbolic execution and graph kernel. In: Reiser, H.P., Kyas, M. (eds.) Secure IT Systems - 27th Nordic Conference, NordSec 2022. LNCS, vol. 13700, pp. 292–310. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-22295-5_16
Biondi, F., Given-Wilson, T., Legay, A., Puodzius, C., Quilbeuf, J.: Tutorial: an overview of malware detection and evasion techniques. In: Margaria, T., Steffen, B. (eds.) Leveraging Applications of Formal Methods, Verification and Validation. Modeling - 8th International Symposium, ISoLA 2018. LNCS, vol. 11244, pp. 565–586. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03418-4_34
Blokhin, K., Saxe, J., Mentis, D.: Malware similarity identification using call graph based system call subsequence features. In: 2013 IEEE 33rd International Conference on Distributed Computing Systems Workshops, pp. 6–10. IEEE (2013)
Borzacchiello, L., Coppa, E., D’Elia, D.C., Demetrescu, C.: Reconstructing C2 servers for remote access Trojans with symbolic execution. In: Dolev, S., Hendler, D., Lodha, S., Yung, M. (eds.) Cyber Security Cryptography and Machine Learning. CSCML 2019. LNCS, vol. 11527, pp. 121–140. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20951-3_12
Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., Yin, H.: Automatically identifying trigger-based behavior in malware. In: Lee, W., Wang, C., Dagon, D. (eds.) Botnet Detection, pp. 65–88. Springer, Boston (2008). https://doi.org/10.1007/978-0-387-68768-1_4
Calleja, A., Tapiador, J., Caballero, J.: The malsource dataset: quantifying complexity and code reuse in malware development. IEEE Trans. Inf. Forens. Secur. 14(12), 3175–3190 (2018)
Chen, J., et al.: \(\{\)SYMSAN\(\}\): time and space efficient concolic execution via dynamic data-flow analysis. In: 31st USENIX Security Symposium (USENIX Security 22), pp. 2531–2548 (2022)
Dataprot: A Not-So-Common Cold: Malware Statistics in 2022 (2023). https://dataprot.net/statistics/malware-statistics/
Godefroid, P.: Test generation using symbolic execution. In: D’Souza, D., Kavitha, T., Radhakrishnan, J. (eds.) IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS 2012. LIPIcs, vol. 18, pp. 24–33. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2012)
Gorecki, C., Freiling, F.C., Kührer, M., Holz, T.: TrumanBox: improving dynamic malware analysis by emulating the internet. In: Défago, X., Petit, F., Villain, V. (eds.) Stabilization, Safety, and Security of Distributed Systems, pp. 208–222. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24550-3_17
HackTricks. Common API used in malware (2023). https://book.hacktricks.xyz/reversing-and-exploiting/common-api-used-in-malware
Massarelli, L., Di Luna, G.A., Petroni, F., Querzoni, L., Baldoni, R.: Function representations for binary similarity. IEEE Trans. Depend. Secure Comput. 19(4), 2259–2273 (2021)
Microsoft: Programming reference for the win32 API (2023). https://learn.microsoft.com/en-us/windows/win32/api/
Namani, N., Khan, A.: Symbolic execution based feature extraction for detection of malware. In: 2020 5th International Conference on Computing, Communication and Security (ICCCS), pp. 1–6. IEEE (2020)
NSA. Ghidra (2023). https://ghidra-sre.org/
Obdržálek, J., Trtík, M.: Efficient loop navigation for symbolic execution. In: Bultan, T., Hsiung, PA. (eds.) Automated Technology for Verification and Analysis. ATVA 2011. LNCS, vol. 6996, pp. 453–462. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24372-1_34
Park, K., et al.: Identifying behavior dispatchers for malware analysis. In: Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, pp. 759–773 (2021)
Said, N.B., et al.: Detection of Mirai by syntactic and behavioral analysis. In: Ghosh, S., Natella, R., Cukic, B., Poston, R.S., Laranjeiro, N. (eds.) 29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018, Memphis, 15–18 October 2018, pp. 224–235. IEEE Computer Society (2018)
Schrittwieser, S., Katzenbeisser, S.: Code obfuscation against static and dynamic reverse engineering. In: Filler, T., Pevný, T., Craver, S., Ker, A. (eds.) Information Hiding, pp. 270–284. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24178-9_19
Schrittwieser, S., Kochberger, P., Pucher, M., Lawitschka, C., König, P., Weippl, E.R.: Obfuscation-resilient semantic functionality identification through program simulation. In: Reiser, H.P., Kyas, M. (eds) Secure IT Systems. NordSec 2022. LNCS, vol. 13700, pp. 273–291. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-22295-5_15
Schwartz, E.J., Avgerinos, T., Brumley, D.: All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In: 2010 IEEE Symposium on Security and Privacy, pp. 317–331. IEEE (2010)
Sebastio, S., et al.: Optimizing symbolic execution for malware behavior classification. Comput. Secur. 93, 101775 (2020)
Shoshitaishvili, Y., et al.: Sok:(state of) the art of war: offensive techniques in binary analysis. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 138–157. IEEE (2016)
Talos, C.: Magicrat: Lazarus’ Latest Gateway into Victim Networks (2022). https://blog.talosintelligence.com/lazarus-magicrat/
Team, Y.: Yararules (2023). https://github.com/Yara-Rules/rules
TrendMicro: Indicators of Compromise (2023). https://www.trendmicro.com/vinfo/us/security/definition/indicators-of-compromise
Valeros, V., Garcia, S.: Growth and commoditization of remote access trojans. In: 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW), pp. 454–462. IEEE (2020)
Vasilescu, M., Gheorghe, L., Tapus, N.: Practical malware analysis based on sandboxing. In: 2014 RoEduNet Conference 13th Edition: Networking in Education and Research Joint Event RENAM 8th Conference, pp. 1–6. IEEE (2014)
Yan, X., Han, J.: gspan: Graph-based substructure pattern mining. In: 2002 IEEE International Conference on Data Mining, 2002, pp. 721–724. IEEE (2002)
Yun, I., Lee, S., Xu, M., Jang, Y., Kim, T.: \(\{\)QSYM\(\}\): a practical concolic execution engine tailored for hybrid fuzzing. In: 27th \(\{\)USENIX\(\}\) Security Symposium (\(\{\)USENIX\(\}\) Security 18), pp. 745–761 (2018)
Acknowledgments
This research is supported by the Walloon region’s CyberExcellence program (Grant #2110186).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lucca, S., Crochet, C., Bertrand Van Ouytsel, CH., Legay, A. (2024). On Exploiting Symbolic Execution to Improve the Analysis of RAT Samples with angr. In: Mosbah, M., Sèdes, F., Tawbi, N., Ahmed, T., Boulahia-Cuppens, N., Garcia-Alfaro, J. (eds) Foundations and Practice of Security. FPS 2023. Lecture Notes in Computer Science, vol 14551. Springer, Cham. https://doi.org/10.1007/978-3-031-57537-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-57537-2_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-57536-5
Online ISBN: 978-3-031-57537-2
eBook Packages: Computer ScienceComputer Science (R0)