Abstract
Encryption is increasingly used in network communications, especially by malicious software (malware) to hide its malicious activities and protect itself from being detected or analyzed. Understanding malware’s encryption schemes helps researchers better analyze its network protocol, and then derive the internal structure of the malware. However, current techniques of encrypted protocol analysis have a lot of limitations. For example, they usually require the encryption part being separated from message processing which is hardly satisfied in today’s malware, and they cannot provide detailed information about the encryption parameter such as the algorithm used and its secret key. Therefore, these techniques cannot fulfill the needs of today’s malware analysis.
In this paper, we propose a novel and enhanced approach to automatically detect and analyze encryption and encoding functions within network applications. Utilizing dynamic taint analysis and data pattern analysis, we are able to detect encryption, encoding and checksum routines within the normal processing of protocol messages without prior knowledge of the protocol, and provide detailed information about its encryption scheme, including the algorithms used, secret keys, ciphertext and plaintext. We can also detect private or custom encryption routines made by malware authors, which can be used as signature of the malware. We evaluate our method with several malware samples to demonstrate its effectiveness.
Supported by the National Science and Technology Major Projects (Grant No.: 2012ZX03002011-002), the National Key Technology R&D Program (2012BAH426B02) and NSFC under Grant No. 61103040.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
PIN - a dynamic binary instrumentation tool. http://software.intel.com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool
QEMU open source processor emulator. http://wiki.qemu.org/Main_Page
Caballero, J., Poosankam, P., Kreibich, C., Song, D.: Dispatcher: enabling active botnet infiltration using automatic protocol reverse-engineering. In: Proceedings of the 16th ACM Conference on Computer and Communications Security, pp. 621–634. ACM (2009)
Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, pp. 317–329. ACM (2007)
Calvet, J., Fernandez, J.M., Marion, J.Y.: Aligot: cryptographic function identification in obfuscated binary programs. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security, pp. 169–182. ACM (2012)
Cho, C.Y., Shin, E.C.R., Song, D., et al.: Inference and analysis of formal models of botnet command and control protocols. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, pp. 426–439. ACM (2010)
Comparetti, P.M., Wondracek, G., Kruegel, C., Kirda, E.: Prospex: protocol specification extraction. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 110–125. IEEE (2009)
Cui, W., Kannan, J., Wang, H.J.: Discoverer: automatic protocol reverse engineering from network traces. In: Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium, pp. 1–14 (2007)
Cui, W., Peinado, M., Chen, K., Wang, H.J., Irun-Briz, L.: Tupni: automatic reverse engineering of input formats. In: Proceedings of the 15th ACM Conference on Computer and Communications Security, pp. 391–402. ACM (2008)
Elisan, C.: The XOR bypass (2012). https://blog.damballa.com/archives/tag/malware-dropper
Gröbert, F.: Automatic identification of cryptographic primitives in software. Diploma thesis, Ruhr-University Bochum, Germany (2010)
Lee, C.P.: Framework for botnet emulation and analysis. ProQuest (2009)
Li, X., Wang, X., Chang, W.: CipherXRay: exposing cryptographic operations and transient secrets from monitored binary execution (2012)
Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic protocol format reverse engineering through context-aware monitored execution. In: NDSS, vol. 8, pp. 1–15 (2008)
Lutz, N.: Towards revealing attackers intent by automatically decrypting network traffic. Master’s thesis, ETH, Zürich, Switzerland, July 2008
Newsome, J., Song, D.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: NDSS (2005)
Rossow, C., Dietrich, C.J.: ProVeX: detecting botnets with encrypted command and control channels. In: DIMVA (2013)
Wang, Y., Zhang, Z., Yao, D.D., Qu, B., Guo, L.: Inferring protocol state machine from network traces: a probabilistic approach. In: Lopez, J., Tsudik, G. (eds.) ACNS 2011. LNCS, vol. 6715, pp. 1–18. Springer, Heidelberg (2011)
Wang, Z., Jiang, X., Cui, W., Wang, X., Grace, M.: ReFormat: automatic reverse engineering of encrypted messages. In: Backes, M., Ning, P. (eds.) ESORICS 2009. LNCS, vol. 5789, pp. 200–215. Springer, Heidelberg (2009)
Wondracek, G., Comparetti, P.M., Kruegel, C., Kirda, E., Anna, S.S.S.: Automatic network protocol analysis. In: NDSS, vol. 8, pp. 1–14 (2008)
Zhao, R.: Lochsemu process emulator for windows x86. https://github.com/zhaoruoxu/lochsemu
Zhao, R., Gu, D., Li, J., Liu, H.: Detecting encryption functions via process emulation and IL-based program analysis. In: Chim, T.W., Yuen, T.H. (eds.) ICICS 2012. LNCS, vol. 7618, pp. 252–263. Springer, Heidelberg (2012)
Zhao, R., Gu, D., Li, J., Yu, R.: Detection and analysis of cryptographic data inside software. In: Lai, X., Zhou, J., Li, H. (eds.) ISC 2011. LNCS, vol. 7001, pp. 182–196. Springer, Heidelberg (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A ZeuS Botnet Message Format
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Zhao, R., Gu, D., Li, J., Zhang, Y. (2014). Automatic Detection and Analysis of Encrypted Messages in Malware. In: Lin, D., Xu, S., Yung, M. (eds) Information Security and Cryptology. Inscrypt 2013. Lecture Notes in Computer Science(), vol 8567. Springer, Cham. https://doi.org/10.1007/978-3-319-12087-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-12087-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-12086-7
Online ISBN: 978-3-319-12087-4
eBook Packages: Computer ScienceComputer Science (R0)