1 Introduction

Nowadays, a software product is a large engineering project that consists of a large number of interconnected components and libraries, with millions of lines of code in each of them. The development of this type of product leads to the inclusion of faults or bugs that can represent serious security risks where the software is deployed.

A buffer overflow is one of the most dangerous and, at the same time, most frequent vulnerabilities in software development. It occurs during the execution of a program when an application writes data outside the limits established for it. This data overwrites adjacent memory locations and, depending on how it is done, may affect the behavior of the program. A buffer overflow can produce a denial-of-service (DoS) attack if the program overwrites data necessary for its correct operation. In addition, a buffer overflow can cause an unintentional remote code execution (RCE) if, when writing (or when injecting code), it steals the execution flow and execute a shellcode [18]. These attacks are increasingly sophisticated and are capable of performing very diverse tasks that can range from the obtaining of a shell, the theft of sensitive information in the compromised system, the elevation of privileges, to the downloading of malware and its subsequent propagation within the network. This situation is aggravated when this software is implemented in critical infrastructures.

The lack of operations that check the limits of the buffers allows this type of errors. Applications written in the C or C++ languages are commonly associated with these types of vulnerabilities, since they allow overwriting part of the memory without checking whether the written data occurs in non-reserved positions. There are two well-differentiated types of buffer overflow vulnerabilities: those of the stack type, and those based on the heap or arising during dynamic memory allocation [24]. We will see more details about these types in Sect. 2. The scientific community has made a great effort to mitigate and neutralize these types of vulnerabilities [36]. The different techniques devised include manual and automatic analyses [38]. Analysts can choose to debug the program step by step by entering breakpoints, and looking for these types of vulnerabilities. This work is inefficient and tedious, and even applications can include techniques to detect their presence [20, 32]. On the other hand, automatic techniques can look for this type of vulnerabilities in the source code [21], although this is not always available. Another protection solution is to enable different options in the compiler. For example, the No eXecute protection (NX) in AMD and Intel eXecute Disable (XD) [14] are aimed at avoiding the execution of code in memory areas destined for data, such as the stack. Another technique is the so-called Address Space Layout Randomization (ASLR) [19], whose function is to randomize the virtual address space of a process in order to avoid exploits that use static addresses, thus hindering the execution of exploitation techniques such as return oriented programming (ROP) [29]. Finally, canary cookies [16] involve adding a value in the stack at the beginning of each of the functions that contain variables that are susceptible to an overflow; it is checked whether that random value is unchanged. If it has been modified, the program will stop its execution since it has been overwritten intentionally.

In this paper, a scalable and transparent algorithm is proposed that detects buffer overflow type vulnerabilities. Based on the algorithm’s approach, we develop a tool built on top of PIN [8], a Dynamic Binary Instrumentation (DBI) Framework [27]. Through the instrumentation of the code, we detect the variables and their size in the functions that are executed, as well as the overflows that occur between them by checking for consecutive memory writes in these variables. The experiments carried out confirm the viability of the proposed algorithm, which detects the variables of each stack frame and the writings that occur in them. The results obtained have been satisfactory, detecting memory overflows for the applications selected for our experiments.

The rest of the paper is organized as follows. Section 2 includes the theoretical concepts related to buffer overflow vulnerabilities and concepts about computer architecture. Section 3 presents the state-of-the-art methods for tackling the problem. The proposed algorithm is described in detail in Sect. 4.1, and it is evaluated in Sect. 5. Finally, the corresponding conclusions are given in Sect. 6.

2 Technical background

We will discuss concepts related to the architecture of computers in Sect. 2.1, the different types of buffer overflow in Sect. 2.2 and finally, Sect. 2.3 will describe how dynamic binary instrumentation works.

2.1 x86 architecture

Initially, when a program is executed, the loader of the operating system reads the necessary information from the headers of the executable file. Once loaded, four sections appear: data, instructions, heap and stack. We will emphasize these last two: the heap is used to dynamically allocate blocks of memory at run time, while the stack is used in the storage of local variables and for passing parameters to functions that are invoked by the program. Physically, the stack is an area that has been defined for this purpose, and its difference from any other data is only a logical difference. Internally, the stack is handled through the Last In, First Out (LIFO) method and its memory is assigned in a descending way. The return address is stored in the stack by obtaining the current value of the Instruction Pointer Register (EIP). This value is important since most attacks try to overwrite their memory location, with the aim of redirecting the execution flow towards memory areas controlled by the malicious user or parts of the program not visible to the end user [12].

2.2 Memory corruption vulnerabilities

Buffer overflow is a security flaw that involves writing data beyond the size that was reserved for a variable in memory. They occur due to the lack of, or incorrect, checking of the buffer limits and can enable an attacker to gain control of an affected system. Depending on which memory area the buffer overflow occurs in, it is either called stack buffer overflow or heap buffer overflow. As for the heap, there may also be memory corruption problems associated with memory allocation and release.

2.2.1 Stack-based buffer overflow

This occurs in the memory area reserved for the stack and is the easiest type of buffer overflow to exploit. The exploitation of this security failure can overwrite variables stored in the stack or the return address of the affected function, which can cause a change in the execution flow of a program.

2.2.2 Heap-based buffer overflow

Unlike the stack, in which the programmer is abstracted from the assignment and release of memory (each function has its prolog where it will create its memory space on the stack to store its local variables and the size of these cannot be managed by the user), in the heap it is necessary for the programmer to make calls to functions of the operating system to assign or release said memory space [25]. When a memory request is made, the heap manager will allocate a block of memory in the heap area and return a pointer to that portion of memory that is now available. Therefore, there must be a data structure for memory management and an allocation algorithm. Therefore, the exploitation of this type of vulnerabilities is not as trivial as it was in the stack. A heap overflow could allow an attacker to modify the data from other allocated chunks of memory.

2.2.3 Memory assignment and release failures

Among these types of failures is the Use After Free (UaF) error, which occurs when a pointer that has already been released is used. They tend to be errors in the logic of the program and, therefore, are difficult to detect by static analysis. Once the memory has been assigned, the functions responsible for releasing it are also responsible for unassigning the corresponding memory chunks. After the release, what actually happens is that those chunks are marked as available for later use, but the pointer still exists and points to that memory area. In addition, there are failures of the Double Free (DF) type, which occur when you try to release a pointer that has been previously released and, although in this type of errors it is very difficult to achieve the execution of arbitrary code, they can cause the process to finish its execution unexpectedly.

2.3 Binary instrumentation

Binary instrumentation consists in inserting code in an application to analyze its behavior. It can be entered statically at compile time, in which case it is necessary to have the source code of the application. It is also possible to perform the instrumentation dynamically, by inserting code at run time. The main advantage of dynamic instrumentation is that it allows controlling the execution of a program and it is not necessary to have its source code. This aspect is very important, for example, in the analysis of malware and the search for vulnerabilities since, in most cases, only the executable part of the application is available [35].

There are different dynamic executable instrumentation systems (DBI) that offer an application programming interface to create your own tools, such as PIN [8], Valgrind [10], Frida [6] or DynamoRIO [3].

3 Related works

Given that DBI allows the transparent follow-up of calls to libraries and system functions, is nowadays applied in the security industry. In addition, it can also be used to unpack malware automatically by detecting write accesses in memory regions that are executed later [15]. In the work [31] the use of DBI is proposed to prevent the protection measures incorporated by the malware to avoid its analysis and execution in controlled environments.

Other proposals for the use DBI focus on the detection of attacks, as in [17], in which a tool is designed to detect ROP attacks at runtime by creating an alternative stack so that each time a return instruction is executed it can be checked whether it matches the one stored in the alternative stack. Otherwise, the return address in the stack has been overwritten, so an attack has occurred.

In [30] a tool is described that uses DBI for the automatic analysis of injected code. The proposal is based on the detection of code execution in memory zones that are different from the modules that have been loaded in memory, that is, if, for example, the memory address belongs to an area of the stack or the heap. As in the previous work, it also makes use of an alternative stack to detect ROP-type attacks.

In [28] a technique is implemented to avoid the use of brute force in canary cookies in programs that use the fork function. When a child process is created, it inherits all the address space of the parent process, also inheriting the value of the canary cookie and, therefore, an attacker could use brute force to avoid this protection mechanism. Their proposal is based on a pintool that modifies the value of the canary cookie of the child process and of all of the stack frames that it inherits from the parent process. For the search for vulnerabilities, [37] proposes the use of DBI to perform a flow analysis of the data of a program in order to determine which variables are contaminated. Then it uses this information to direct the symbolic execution process and find possible vulnerabilities in the application.

In [26] a tool is proposed that is divided into three retro-fed parts. The first is responsible for monitoring and storing the context of the instructions executed by the program. The second is responsible for using that file of frames and reproducing the execution to analyze the contaminated variables, creating an index file for quick queries. Finally, the last stage analyzes the result in search of security flaws in a series of predefined rules to detect writing errors and memory allocation.

In [39] a DBI-based tool is described that identifies overflows in the stack frames of a program. In this study, the authors present two ways of identifying buffer overflows. The first consists in verifying the accesses to the stack against the limits of the local variables of that function. The second focuses on detecting when the return address has been overwritten. Finally, its implementation is based on this second way because of the difficulty of determining the start and end limits of the variables due to the lack of types and other semantics at the machine code level.

For the detection of errors in the heap area, [35] and [12] insert instrumentation code in the functions related to the management of the heap, adding marks before and after each assigned memory block to check whether any reading or writing of memory is performed in those memory zones. The main difference between them is that [35] is completely transparent to the instrumented application, since it creates an alternative memory map to map the blocks of memory allocated in the heap.

In [33] a tool for the detection of memory corruption errors in both the heap and the stack is described. The disadvantage of this tool is that it uses compilation instrumentation, which means that it is necessary to have the source code, so it cannot be used to implement closed source applications.

DECAF [22] is a binary analysis platform based on QEMU which allows a complete vision of the system and its information flow. One of its main characteristics is the dynamic taint analysis at system level that allows to track the information propagated through system calls.

Memcheck [34] is a tool built using the Valgrind binary instrumentation framework. It is capable of detecting memory errors such as the use of uninitialized values, memory leaks, incorrect memory releases, among others. It is the most used tool to analyze memory problems in unix-based operating systems.

In general, there are several fields of computer security where the use of dynamic instrumentation is applied. Regarding the detection of memory vulnerabilities, most of them focus on detecting bugs in the heap area and attacks using ROP or injection of code. These attacks are the result of the exploitation of programming errors found in the program. Unlike the main related work solutions based on DBI, the proposal described in this paper detects the variables and their size in the stack frame of each function that is being executed, allowing to detect memory overflows among the variables of each stack frame and not just those buffer overflows that overwrite the return address.

4 Proposal for stack overflow detection

This section presents the proposal, where a motivation of the research and the technical challenges are provided. In addition, the design of the tool built is described to face the problems associated with buffer overflow in the stack area.

4.1 Research motivation

Although buffer overflow can occur in a similar way both in the heap and in the stack, that is, by a bad or non-existent check on the limits of a buffer, the techniques to diagnose it are different. In the heap, it is possible to detect the vulnerability between the different memory blocks assigned using DBI, since the address and size of the allocated memory blocks can be obtained at runtime by instrumentation of the functions that are used for dynamic memory management, and, therefore, store the blocks that are being used and those that have been released. Finally, when a memory write occurs, it can be checked whether it occurs within the limits of the assigned block. If it is outside the area, it is a heap overflow problem. Also, when storing the blocks of memory that have been released, it is possible to diagnose errors of the UaF type if there is a read or write access in an already released block, and of type DF if a block is released which had previously been released.

As for the detection of buffer overflow in the stack, it is more complicated to detect since there may be several variables and, since the stack is managed at compilation time, it is necessary to isolate the functions of a program to store the detected variables and release them when the function finishes. Although as we saw in Sect. 3 there are other approaches in the literature where the detection of this type of errors is based on checking whether the return address stored in the stack has been overwritten. This type of approach would not detect the problem when there is an overflow between variables of the same stack frame that does not change the return address but it changes some other variable of a function, causing certain unwanted behavior in the application. Therefore, the objective is to detect the variables of a function and its size at runtime as well as consecutive writes in memory with the aim to detect when a buffer overflow is occurred between them.

4.2 Detecting variables from stack frames

To solve the problem associated with the detection of stack overflow, we need to detect the variables and obtain their memory address in the stack while the program is running. Normally, the variables of a stack frame are referenced through the EBP or ESP register and, therefore, we can obtain their memory address each time an effective load instruction (LEA) or a memory access instruction is produced in which one of its operands is referenced by one of these two registers. Also, we detect those variables whose access is determined by any register plus a fixed value but that match the pattern that the register points to a variable already detected. This allows detecting variables of a structure whose starting address is moved to a register and for example passed as a parameter to a function. In addition, we detect as variables those values pushed to the stack. Although these are not real variables declared in the source code, we think that it is interesting to detect whether any of these “variables” are overwritten. Therefore, to be able to check later whether any writing causes an overflow, it is necessary to store each of the detected variables.

Fig. 1
figure 1

Detection of variables and mapping in the shadow memory

Full size image

For the storage of the local variables of a program we have used a shadow memory so that we can directly access a memory position through an index and thus avoid the use of more complex data structures such as linked lists, which would degrade the performance when having to go through them to find the stack frame of a function. Being an x86 architecture, we assume that the memory of the program is aligned, so we mapped a word (four bytes) of the stack memory of a program in one byte in our shadow memory. In addition, we use a flag to indicate the initial memory position of each variable (the size of a variable will be indicated by the distance from that memory position to the next one).

In this way, we can directly access the memory position in our shadow memory by dividing the memory address of the application by four and using this value as an index of the shadow memory. Otherwise, we can obtain the memory address of the variable in the stack frame by multiplying by four.

Since a stack frame is created every time a call is made to a function, and its variables are “destroyed” each time the return of a function occurs, it is necessary to clear those variables from the shadow memory. To solve this problem, since each execution thread created in the program has its own stack, we obtain for each thread the start and end of the stack and upon the return of a function we clean all the values that are below the memory address pointed to by the ESP register (in the x86 architecture the stack grows towards lower memory addresses).

The Fig. 1 shows a schema of how we obtain the variables and their storage in the shadow memory. It is divided into 4 steps according to the instruction that is being executed. The instruction that is executed, the value of the ESP register and the storage or deletion of addresses in the shadow memory are displayed. The sub-pictures represent the following:

  • Image 1.a saves the value 0x9C on the stack. Although it is not exactly a variable, we store the memory addresses where the values pushed to the stack are stored as variables to detect if an overflow occurs in any of them (e.g. saved ebp, arguments to functions, etc.). Before the value is pushed to the stack, ESP has the value 0x00142CA4. This address points to the top of the stack and will be decremented by four once the push instruction is executed. The index where we will store that a variable exists in that address is calculated by moving 2 bits to the right (division by 4) the memory address where it will be written (0x00142CA4). The result of this operation is 0x50B29, which will be the index of the shadow memory where we will mark that a variable exists there.

  • In Image 1.b, the value of the ESP register is already updated (the stack grows towards lower memory addresses). The memory instruction to be executed will store the value found in the ESI register in the address pointed to by the ESP register plus 12. As it is an address pointed to by the ESP or EBP register, we consider that it is of a variable and, therefore, we mark that address in the shadow memory. The calculation of the index occurs in the same way as in Image 1.a.

  • Image 1.c is a call to a function. Within this function, the variables that are detected in the shadow memory will be mapped and when it finishes executing, it will return to execute the following instruction. When the function ends, it is responsible for releasing the stack frame of the function as well as the arguments with which it was called. Therefore, the variables stored in the shadow stack and the arguments (the address where 0x9C is stored in Image 1.a will also be removed.

  • Finally, the image 1.d shows the ESP register acted with the same value that it had initially. The instruction will store a zero in the memory address pointed to by the content of ESP register plus 16 (It is a variable of the pointer type). Therefore, we will calculate the shadow memory index and mark the ESP address plus 16 as a variable.

4.3 Detecting consecutive writings in stack area

To detect memory overflows in the stack area, it is necessary to obtain the size written in bytes and check whether it exceeds the limit for that variable. In this case, we assume that a buffer overflow is given by a consecutive writing in memory and we need to check every writing instruction executed.

When a consecutive write of data is made, normally, the memory address where it is written is increased in each iteration and the instruction that is responsible for writing in memory is the same within the loop. Therefore, to detect these consecutive writes, it is necessary to instrument all the writings that are produced in the stack and store the instruction that performs the writing, the memory address where the writing begins as well as the size written up to that time.

Each time a write occurs, it is checked if that instruction has already been written in memory and if so, it is checked if the destination address is consecutive to the last written address (initial memory address plus size written up to that time). If so, the size to be written is added to the written size. Otherwise, the starting address is modified with the new address and the size written with the new size to be written.

Finally, it is checked if this writing produces an overflow by checking in the shadow memory if there is a variable between the writing start address and the last written address of that instruction. The Algorithm 1 represents the pseudo-code of the function responsible for detecting consecutive writes in memory.

figure a

4.4 Details of the development of the tool

To verify the functioning of our proposal, a tool has been developed on top of the PIN DBI framework. PIN [8] is a tool created by Intel that allows you to insert code transparently into the application at runtime. Figure 2 shows an outline of its architecture [9]. It can be seen that it is composed mainly of two components: a virtual machine that contains the JIT compiler and the emulation unit, and a code cache that is used to store the already instrumented code, thus reducing the overhead [13].

Fig. 2
figure 2

Scheme of the PIN architecture

Full size image

Although it supports different types of granularity when carrying out the instrumentation, in our case it is carried out at the level of routine and machine code. Tools created with PINs are called pintools and allow you to configure the behavior of the tool. The implemented pintool receives a text file with each of the functions to be instrumented outside the main module. In our case, we used a text file with the name of functions whose use is not recommended by the Microsoft Developer Network (MSDN) [23] because they are potentially vulnerable functions. These functions are associated with the input and copy of data and have more secure versions which are therefore less prone to errors. Our pintool uses these function names that have been received by text file to instrument the writing memory that is executed outside the main program module. For this, we use the granularity offered by PIN at the routine level, checking whether the name of the routine to be executed is outside the main module and whether it matches the functions mentioned above. If this is true, we insert instrumentation code in those instructions that perform some memory writing within that function. Regarding the main executable module, we insert instrumentation code to detect the variables that belong to the stack area and in all the instructions that perform writings in memory in order to detect whether there is an overflow based on the stack.

So, although the pintool designed supports the complete instrumentation of any module (detection of local variables and all writings), in the experiments carried out it has only been considered to instrument the main executable module in this way and only functions considered vulnerable by the MSDN, with the objective of reducing the overhead by avoiding the instrumentation of all the writes in memory that can be performed by system and shared Dynamic-Link Libraries (DLLs).

Before carrying out the tests on the pintool built, a small example of its use will be presented. In Listing 1 we can observe the code of a vulnerable function. Perhaps, the function does not make much sense but it represents well the functioning of the pintool. The function initializes two variables, prints its memory addresses and the value they contain before and after the call to “strcpy”, which will produce a buffer overflow if the variable “data” is larger than the size reserved for “buffer”.

figure b

On the other hand, in Listing 2 the first instructions of the vulnerable function in assembly code are represented, where we can see the access to [ebp-4] and [ebp-8]. When those instructions are executed, they will be marked as variables in the shadow memory. The same happens with the buffer variable, it is marked as a variable the first time the memory address is used to show it by “printf”. Therefore, if the data size is 264 bytes (263 + null character), it would overwrite the function’s variables without reaching the return address, causing an erroneous output of the values of “num1” and “num2” after the call to “strcpy”.

figure c

In Fig. 3 we can observe the output of the program when the size of the variable is exceeded and the log provided by our pintool. We can see the memory address of the different variables and that the value of “num1” and “num2” has changed due to the overflow of the buffer. If we observe the generated log, the buffer overflow was detected in the memory address “0x18fd28”, the total bytes written was 264 and the variables that have been overflowed. In addition, we can see the direction of the instruction that causes the buffer overflow. Looking at the address of this instruction, we can see that it is among the range of memory addresses where the module “MSVCR120.dll” is loaded in memory. Therefore, this address is part of the “strcpy” function that generated the overflow in the program buffer.

Fig. 3
figure 3

Output application and log generated by the pintool

Full size image

5 Effectiveness and performance

In this section, we present an experimental evaluation of the proposal described in Sect. 4.1. For the tests carried out, a virtual machine equipped with Windows 7 SP1 x86, 3 GB of RAM and an Intel Core i5-7267U at 3.10 GHz was implemented.

5.1 Evaluation of applications

For the first experiment carried out, applications that have a stack overflow vulnerability and whose exploits are publicly available in exploit-db [7] were selected.

The applications that were tested with the pintool developed were the following:

  • Easy File Sharing Web Server. A web server for sharing files that can be exploited through a malicious request to the web server, causing an overflow of memory in the stack area.

  • VUPlayer. A multimedia player that can be exploited via a very long URL in .pls files, a format used in playlists.

  • Notepad++ CCompletion Plugin. A text editor that supports numerous programming languages. The vulnerability occurs when you select text and this is sent to the CCompletion plugin that copies the text, producing a buffer overflow.

  • RM-MP3 Converter. Tool that converts RM files to MP3 format. It allows the execution of code through ULRs in files of multimedia playback lists.

  • Jnes. An emulator of the Nintendo Entertainment System (NES) video game console. The vulnerability occurs when inserting data in one of the options of the emulator for the games, allowing an attacker to take control of the application.

  • UltraMiniHTTPd. Consists of an HTTP server that can be attacked remotely through a POST request.

  • CoreFTPServer. Using the TYPE command, this FTP server can be exploited, causing a denial of service on the server. The overflow overwrites a pointer to an object, causing a crash in the application when trying to access that memory area.

  • ASX to MP3 Converter. An application that converts files in ASX format to MP3 format and can be exploited locally.

  • Alloc Video Joiner. A tool that enables you to join video files. It presents a vulnerability in the form for registering the application, which allows the execution of arbitrary code.

  • PCMAN FTP Server. A FTP server that can be exploited remotely through the mkd command. The server does not correctly handle the received data, causing a stack-based buffer overflow.

  • Easy Chat Server. A chat server that can be exploited through the user parameter causing a crash on the server.

  • Easy File Management Web Server. A is a tool to manage files and folders through a web server. It presents a vulnerability that allows you to take control of an application through a UserID that is too long in the cookie.

The selected applications were executed with the pintool and, subsequently, the exploit that triggered the vulnerability was executed. Table 1 shows the results obtained by our pintool. It can be seen that the results obtained by our proposal were satisfactory, showing the write instruction that caused the overflow, the address of the variable where the writing started, as well as the total size in bytes that were written in the overflow variable.

Table 1 Details of the results obtained
Full size table

To make sure that the experimental results were not biased, that is, the application is proned to detect vulnerabilities, a study was carried out to analyze if the detections made by the application were always true.

5.1.1 False positives

Since no false positives were found during the previous experiments, we decided to test the tool with different common applications that, a priori, do not contain any vulnerabilities such as Google Chrome, Python, Firefox, Internet Explorer and HxD in its latest versions. During the tests, we detected false positives in some of the applications. Specifically, false positives occurred in two of them among more than ten that have been instrumented.

We went one step further and analyzed the false positives using reverse engineering techniques to figure out what caused these false positives. Mainly, false positives were given in some structures whose fields were referenced within the function to which this variable belonged through the EBP or ESP registers. Once these memory addresses were accessed and were considered variables, a consecutive copy was produced in the structure (for example, a copy of a structure of the same type) in another part of the program. As a result, it detected a buffer overflow because it started writing to a variable and continued writing “beyond” its size; however, it was not a buffer overflow.

5.2 Comparison with other related-work tool

With the purpose of evaluating our solution with other tools, we compared our results with DStack [39], since, to the best of our knowledge, DStack is the work most similar to ours. The authors of DStack carried out their evaluation with the following applications:

  • CVE-2009-1328 [1] is a vulnerability based on the stack buffer overflow in version 3.0.0.7 of Mini-stream RM-MP3 Converter. It is a vulnerability that can be exploited locally through a long URI in a playlist file (the exploit can be found in exploit-db [7]).

  • CVE-2010-2883 [2] is a stack-based overflow in the library “CoolType.dll” of Adobe Reader. This vulnerability affects versions 9.x prior to 9.4 and versions 8.x prior to 8.2.5 and can be exploited through a pdf file with a long field in a Smart INdependent Glyphlets (SING) table in a TTF font. The exploit is public and is integrated into metasploit [5].

In their experiments, DStack detected in both cases that there was a buffer overflow, upon detecting that the return address was modified since it was not located after a call type instruction.

In our tests, we instrumented both applications with our pintool. Since the [2] vulnerability is in an Adobe library, we configured the pintool so that, in addition to instrumentation of the main module and the functions of the MSDN, it also checked the shared libraries. In both cases, the results were satisfactory and a buffer overflow was detected. The logs generated by pintool are shown in Fig. 4, where it can be seen that the overflow was detected and that it had overwritten several variables in the area of the stack. In addition, it can be seen that in CVE-2009-1328 two overflows were detected. To understand what was happening, we applied reverse engineering to the application. We figured out that the first buffer overflow was found in the “MSRMfilter03.dll” library. This function copied the URI to the stack variable addressed in “0xf04AC”, overflowing some variables that had previously been detected. The second overflow occurred in the function to which the previous variable belonged. In this function, it was copied from that address (0xF05AC) to “0xEE2A4” overlapping itself and, therefore, was detected as a buffer overflow since it exceeded the limits of the destination variable.

Fig. 4
figure 4

Logs generated by pintool

Full size image
Fig. 5
figure 5

Overload at runtime of our pintool compared to the execution through the PIN framework without any type of instrumentation

Full size image

Although in both cases the vulnerability was detected, DStack only detects overflows that change the execution flow, that is, it modifies the return address while our tool also detects overflows between variables that are detected in the stack frames. For example, in the example shown in the Sect. 4.4, DStack would not detect any overflow since the return address was not modified however, the values of the variables were modified because of the overflow.

5.3 Overload analysis

To measure the overload caused by dynamic instrumentation, the same set of previously tested applications were used. Since these are applications with a graphical interface in which the intervention of the user or of an event is necessary to continue, they were modified so that the execution time was not affected by this interaction. To do this, the applications were patched to complete their execution by calling the ExitProcess function once the program’s GUI was displayed or when was waiting for an incoming connection in the case of server-type applications. In addition, to measure the overload caused by the pintool, the load that entails executing via the PIN framework without instrumentation was also measured. To obtain the PIN overload, we have created a pintool that is only in charge of executing the application without any type of instrumentation code. Figure 5 shows the results obtained in the tests carried out to measure the overload of our pintool.

Based on the results achieved, it is observed that it introduces an overload between 1.89x and 5.18x with an average of 2.61x. This overload is caused by the instrumentation code inserted to detect the variables of the stack frame and by the instrumentation of all the write accesses in memory that occur in the main module and in the functions of the DLLs cataloged as dangerous by the MSDN, all of which are necessary parts for the detection of buffer overflow type errors.

As one can see, this type of solution is not suitable for applications that will be executed in real time, but for those that are in pre-production or within the secure cycle of software developments and it can be used to automate the search for memory errors before putting software into production. DBI has many advantages when it comes to analyzing software, since it allows to analyze any program that can run and obtain memory or registers values, among other features, in execution time, but its main disadvantage is that the more code you insert in the instrumentation, the slower the execution of the instrumented application will be.

6 Conclusions

In this paper an original algorithm for the detection of variables and errors caused by buffer overflow in the area of the stack was presented. To demonstrate the feasibility of the proposal, a dynamic instrumentation tool was developed as a proof of concept to detect this type of error. The system designed allows you to analyze applications in search of this type of flaws and it can be integrated in the testing phases of the software development cycle.

The results obtained in the experiments indicate that it is feasible to detect variables and overflows between them at runtime. Although DBI increases the overload of the application, this tool can be combined with a fuzzer and used for the automation of searching for errors of this type in memory before putting a piece of software into production.

As mentioned above in the experiments carried out, by default, only the main module of the program and some functions of the DLLs whose use is considered dangerous are instrumented, but the pintool can be configured to instrument all shared or system DLLs. The problem of instrumenting these DLLs is the overload that this implies for the application, since it needs to detect each variable of the stack frames of the different functions that are executed in each of the libraries included in the analysis, as well as the Instrumentation each of the writing instructions in the memory.

A possible improvement to reduce the system overload in case of wanting to instrument all the modules used by the application is to take advantage of the power of some static analysis tools such as IDA[4] or Radare2[11], and instrument only those functions that are likely to have a buffer overflow. This could be achieved by instrumenting those functions that contain loops and buffers inside them as well as the functions that call or are called from said functions.