Abstract
Context
Code hardening is meant to fight malicious tampering with sensitive code executed on client hosts. Code splitting is a hardening technique that moves selected chunks of code from client to server. Although widely adopted, the effective benefits of code splitting are not fully understood and thoroughly assessed.
Objective
The objective of this work is to compare non protected code vs. code splitting protected code, considering two levels of the chunk size parameter, in order to assess the effectiveness of the protection - in terms of both attack time and success rate - and to understand the attack strategy and process used to overcome the protection.
Method
We conducted an experiment with master students performing attack tasks on a small application hardened with different levels of protection. Students carried out their task working at the source code level.
Results
We observed a statistically significant effect of code splitting on the attack success rate that, on the average, was reduced from 89% with unprotected clear code to 52% with the most effective protection. The protection variant that moved some small-sized code chunks turned out to be more effective than the alternative moving fewer but larger chunks. Different strategies were identified yielding different success rates. Moreover we discovered that successful attacks exhibited different process w.r.t. failed ones.
Conclusions
We found empirical evidence of the effect of code splitting, assessed the relative magnitude, and evaluated the influence of the chunk size parameter. Moreover we extracted the process used to overcome such obfuscation technique.


















Similar content being viewed by others
Notes
SpaceGame can be obtained from SourceForge at https://sourceforge.net/projects/game/.
Interested readers can find details about these functions in the file gamespace.c.
GrammaTech CodeSurfer https://www.grammatech.com/products/codesurfer
Two examples of excellent reverse engineering and exploitation tools are IDA-Pro https://www.hex-rays.com/products/ida/ and radare2 https://rada.re/
The syllabus reports the following course that “Computer and System Security” students have followed: Computer Science (introduction to C programming), Algorithms and Programming, Object oriented programming, Operating systems (scripting languages), at Bachelor level, and System programming, Distributed Programming I and for the Master together with the optional course named Distributed Programming II.
In the Italian University system, grades are assigned on a 30 values scale.
References
Anckaert B., Madou M., De Sutter B., De Bus B., De Bosschere K., Preneel B. (2007) Program obfuscation: a quantitative approach. In: Proceedings of ACM workshop on quality of protection, pp 15–20
Barak B., Goldreich O., Impagliazzo R., Rudich S., Sahai A., Vadhan S., Yang K. (2001) On the (im) possibility of obfuscating programs. Lect Notes Comput Sci 2139:19–23
Canavese D., Regano L., Basile C., Viticchié A (2017) Estimating software obfuscation potency with artificial neural networks. In: International workshop on security and trust management, pp 193–202. Springer
Ceccato M., Capiluppi A., Falcarin P., Boldyreff C. (2015) A large study on the effect of code obfuscation on the quality of java code. Empir Softw Eng 20 (6):1486–1524
Ceccato M., Dalla Preda M., Nagra J., Collberg C., Tonella P. (2009) Trading-off security and performance in barrier slicing for remote software entrusting. Autom Softw Eng 16(2):235–261
Ceccato M., Dalla Preda M., Nagra J., Collberg C. S., Tonella P. (2007) Barrier slicing for remote software trusting. In: SCAM, pp 27–36
Ceccato M., Di Penta M., Falcarin P., Ricca F., Torchiano M., Tonella P. (2014) A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empir Softw Eng 19(4):1040–1074
Ceccato M., Di Penta M., Nagra J., Falcarin P., Ricca F., Torchiano M., Tonella P. (2009) The effectiveness of source code obfuscation: An experimental assessment. In: IEEE 17th international conference on program comprehension (ICPC), pp 178–187. https://doi.org/10.1109/ICPC.2009.5090041
Ceccato M., Tonella P., Basile C., Coppens B., De Sutter B., Falcarin P., Torchiano M. (2017) How professional hackers understand protected code while performing attack tasks. In: Proceedings of the 25th IEEE international conference on program comprehension (ICPC)
Ceccato M., Tonella P., Basile C., Falcarin P., Torchiano M., Coppens B., De Sutter B. (2018) Understanding the behaviour of hackers while performing attack tasks in a professional setting and in a public challenge. Empirical Software Engineering (EMSE), pp 1–47
Collberg C., Thomborson C., Low D. (1997) A taxonomy of obfuscating transformations. Technical report 148, Dept. of Computer Science, The Univ. of Auckland
Collberg C. S., Thomborson C. (2002) Watermarking, tamper-proofing, and obfuscation - tools for software protection. IEEE Trans. Softw. Eng. 28(8):735–746. https://doi.org/10.1109/TSE.2002.1027797
Falcarin P., Collberg C., Atallah M., Jakubowski M. (2011) Guest editors’ introduction: Software protection. IEEE Softw 28(2):24–27. 10.1109/MS.2011.34
Feldt R., Zimmermann T., Bergersen G. R., Falessi D., Jedlitschka A., Juristo N., Münch J., Oivo M., Runeson P., Shepperd M., Sjøberg DIK, Turhan B (2018) Four commentaries on the use of students and professionals in empirical software engineering experiments. Empir Softw Eng 23(6):3801–3820
Futcher L., von Solms R. (2008) Guidelines for secure software development. In: Proceedings of SAICSIT. ACM, New York, pp 56–65
Goto H., Mambo M., Matsumura K., Shizuya H. (2000) An approach to the objective and quantitative evaluation of tamper-resistant software. In: Third Int. workshop on information security, pp 82–96. Springer
Hänsch N., Schankin A., Protsenko M., Freiling F., Benenson Z. (2018) Programming experience might not help in comprehending obfuscated source code efficiently. In: 14th symposium on usable privacy and security ({SOUPS} 2018), pp 341–356
Krinke J. (2003) Barrier slicing and chopping. In: SCAM, pp 81–87
Linn C., Debray S. (2003) Obfuscation of executable code to improve resistance to static disassembly. In: Proc. ACM conf. computer and communications security, pp 290–299
Oppenheim A. N. (1992) Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London
R Core Team (2016) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
Sutherland I., Kalb G. E., Blyth A., Mulley G. (2006) An empirical examination of the reverse engineering process for binary files. Comput Secur 25(3):221–228
Visaggio C. A., Pagin G. A., Canfora G. (2013) An empirical study of metric-based methods to detect obfuscated code. International Journal of Security & Its Applications 7(2)
Viticchié A., Basile C., Avancini A., Ceccato M., Abrath B., Coppens B. (2016) Reactive attestation: Automatic detection and reaction to software tampering attacks. In: Proceedings of the 2016 ACM workshop on software PROtection, pp 73–84. ACM
Viticchié A., Regano L., Torchiano M., Basile C., Ceccato M., Tonella P., Tiella R. (2016) Assessment of source code obfuscation techniques. In: 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM), pp 11–20. IEEE
Weiser M. (1981) Program slicing. In: Proceedings of the 5th international conference on software engineering, ICSE ’81. http://dl.acm.org/citation.cfm?id=800078.802557. IEEE Press, Piscataway, pp 439–449
Wheeler B, Torchiano M (2016) lmPerm: Permutation tests for linear models. http://CRAN.R-project.org/package=lmPerm. R package version 2.1.0
Wohlin C., Runeson P., Hst M., Ohlsson M. C., Regnell B., Wessln A. (2012) Experimentation in software engineering, Springer Publishing Company, Incorporated
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Saurabh Sinha
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Additional Analyses
Two additional analyses are reported for the sake of completeness, though they are not central for the main findings of this work. The first analysis concerns the distribution of the time spent to complete the task and the second reports the correlation between attack task correctness and the self-claimed experience with the C language, optionally expressed by a subset of the subjects.
1.1 A.1 Elapsed Time
The distribution of elapsed time required to complete the task is reported in Fig. 18 for both successful and failed attacks. By looking at Fig. 18, we can see that the tasks with a wrong solution are all very high and compressed against the 120 min time limitation. Such phenomenon is probably due to the participant trying to use all the allotted time to attack the code without succeeding.
1.2 A.2 Experience of Participants with the C Language
Since the questionnaire about personal experience was not mandatory for the participants, we were able to collect information concerning only 35 students (out of 87).
Figure 19 reports the average correctness (with the relative 95% confidence interval) for the subjects reporting different levels of experience with the C language. We cannot observe any clear and statistically significant trend.
A more detailed view can be observed by looking at the success rate when controlling for the treatment. Figure 20 reports the average correctness by experience and treatment. Also in this case the effect of experience is difficult to identify.
Figure 21 reports the effect of experience on the time to complete the attack task, divided by treatment. We observe a general reduction in time as experience increases though such effect is not statistically significant (too few data points and too much variability).
Finally, we report the relationship between the experience and the strategy selected, divided by treatment. This is reported as a heat map in Fig. 22. Also in this case we do not observe any clear effect of experience on the strategy selection.
Appendix B: Call Graph of the SpaceGame Application
We report in Fig. 23 for the sake of completeness, the call graph of the object of our experiment, the SpaceGame program, extracted with an ad hoc java program based on antlrFootnote 15 and JGraphT.Footnote 16
Rights and permissions
About this article
Cite this article
Viticchié, A., Regano, L., Basile, C. et al. Empirical assessment of the effort needed to attack programs protected with client/server code splitting. Empir Software Eng 25, 1–48 (2020). https://doi.org/10.1007/s10664-019-09738-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-019-09738-1