A Practical Approach for Dynamic Taint Tracking with Control-flow Relationships

3.1 Binding Scope

Overall, Conflux aims to identify conditional branch executions that are not strictly information-preserving and avoid propagating along the resulting control flows. To achieve this, Conflux uses an alternative control flow scope definition that distinguishes between statements that can execute only as a result of a single, specific condition and statements that can execute under multiple conditions. The traditional definition for control flows’ scopes is highly conservative; it considers every basic block between a branch in the flow of control and where that branch rejoins to be within the scope of the branch. Unlike traditional control flow scopes, which are defined with respect to nodes in the control flow graph, “binding” scopes are defined with respect to edges. In particular, binding scopes are defined with respect to the edges that are traversed when a conditional branching statement is “taken” or “not taken” (or for \(\text {switch}\) statements the edges associated with the cases of the \(\text {switch}\)). The binding scope of a branch edge includes instructions that only execute if the edge is traversed, i.e., an instruction is included if every path from the distinguished entry node of the control flow graph to that instruction contains the branch edge. Since Conflux only considers branch edges corresponding to equality checks (e.g., the branch taken side of an equality check or the branch not taken side of an inequality check), the execution of an instruction within the scope of a branch edge occurs only if some value was equal or “bound” to another value.

Binding scopes can be calculated for the branch edges in a method by constructing its control flow graph, \(G = [V, E]\), with designated entry and exit nodes denoted by \(v_{entry} \in V\) and \(v_{exit} \in V\), respectively. Each node in \(V\) other than \(v_{entry}\) and \(v_{exit}\) represents a basic block and consists of a sequence of instructions. The successors of a node \(u \in V\) is defined as \(succ(u) = \lbrace v \mid (u, v) \in E\rbrace\). A node \(v \in V\) is said to be a branch node if it has more than one successor. The last instruction of a branch node is its conditional branch instruction. We refer to the set of outgoing edges of a branch node as its branch edges. Each branch edge is associated with a set of conditions under which the edge is traversed. For example, an \(\text {if}\) statement will have two branch edges: one edge corresponding to the branch being taken with a singleton condition set corresponding to the statement’s predicate evaluating to true and one edge corresponding to the branch not being taken with a singleton condition set corresponding to the statement’s predicate evaluating to false. Whereas, a \(\text {switch}\) statement’s branch edges’ condition sets will partition the cases of the \(\text {switch}\) statement.

An instruction \(i\) is within the binding scope of a branch edge \(e\) if \(i\) can only execute if \(e\) has been traversed. By definition [2], all of the instructions within a basic block execute if and only if the first instruction in the basic block executes. Therefore, we can define binding scopes with respect to basic blocks instead of individual instructions. Thus, the binding scope of a branch edge \(e\) is the set of all basic blocks \(v\) such that all paths in \(G\) from \(v_{entry}\) to \(v\) contain \(e\).

To calculate the binding scope of branch edges, we first construct a modified control flow graph, \(G^{\prime }\), from \(G\) by replacing each branch edge \((u, v) \in E\) with a new node \(b_{u, v}\) and a pair of edges \((u, b_{u, v})\) and \((b_{u, v}, v)\). Given this construction, the binding scope of a branch edge \(e\) is the set of all \(v \in V\) such that \(b_e\) dominates \(v\) in \(G^{\prime }\). Proof of the correctness of this calculation is as follows:

The binding scope of a branch edge is “scope-like”; all nodes within the scope lie on paths between the edge and its dominance frontier. Thus, the taint tag of a branch’s predicate need only be pushed onto a taint stack once at the start of the branch edge’s scope and can be safely popped at the ends of its scope. Another desirable property of binding scopes is that the set of basic blocks within a branch edge’s binding scope is a subset of the basic blocks within its branch’s standard (post-dominator based) scope. The immediate post-dominator of a branch node is reachable from all of its branch edges and therefore either part of or beyond the dominance frontier of its node replacement in the modified graph.

We can now apply the binding scope definition to the \(\text {spaceDecode}\) method in Figure 1(a) using its control flow graph shown in Figure 1(b). There are two branch edges corresponding to equality checks: the one associated with \(case ^{\prime }+^{\prime }\) and the one associated with \(case ^{\prime \prime }\). The \(case ^{\prime }+^{\prime }\)’s edge’s binding scope contains only the basic block with the instruction \(result[i] = ^{\prime \prime }\). The \(case ^{\prime \prime }\)’s edge’s binding scope contains only the basic block with the instruction \(throw new\) \(RuntimeException()\). In this case, using binding scopes allows the relationship between a plus sign in the input and a space in the output to be reflected without introducing over-tainting.

3.2 Loop-relative Stability Heuristic

When propagating along control flows, taint tags often accumulate on program data during the execution of a loop leading to an “explosion” of taint tags. For example, regardless of whether taint tags are applied in standard control flow scopes or only in binding scopes, during the execution of the loop in the \(\text {percentDecode}\) method (Figure 2(a)), taint tags build up on the looping variable, \(\text {i}\), resulting in progressively larger label sets for each successive output. Conflux mitigates this accumulation of taint tags by making special considerations when determining whether to propagate taint tags between the branch of a control flow and a statement within the control flow’s scope. This process is guided by the novel “loop-relative stability” heuristic.

The underlying idea for loop-relative stability is that loops introduce alternative paths to statements within the scope of a control flow. For instance, within a single call to the \(\text {percentDecode}\) method (Figure 2(a)), the instruction on line 7, \(i += 2\), can execute even if the branch on line 5 is not taken on a particular iteration of the loop. That is, on subsequent iterations of the loop, the branch on line 5 could evaluate to true, causing the statement \(i += 2\) to execute, thereby producing the same effect that would have happened had the branch been taken on the earlier iteration. This weakens the relationship between the value of \(\text {i}\) and the element of the \(\text {input}\) array that caused the branch on line 5 to be taken. However, when a value is stored to \(result[size++]\) on line 6, the storage location to which that value is stored is different on each iteration of loop. Like before, on subsequent iterations of the loop, the branch on line 5 could evaluate to true causing the statement on line 6 to execute. However, unlike the statement on line 7 that updates the same local variable \(\text {i}\) on different iterations, the statement on line 6 updates a different element in the \(\text {result}\) array on different iterations. This produces a stronger relationship between the value of \(input[i]\) on line 5 and the value written on line 6 to \(result[size++]\) than the relationship between the value of \(input[i]\) on line 5 and the value written on line 7 to \(\text {i}\).

The loop-relative stability heuristic aims to identify these cases in which a loop introduces multiple conditions under which the same location could be assigned the same value. For this to occur, there must be a conditional branching statement that is contained within a loop and an assignment statement within the scope of that branch’s control flow. If the values used by the branch change on different iterations of the loop but the values used by the assignment statement stay the same, then on each iteration of the loop a different condition could cause the same effect. Since the conditional branching statement might occur in a different method than the loop and assignment statement, the loop-relative stability heuristic is defined in terms of the dynamic execution of statements. To capture these ideas, we define an execution of a program statement as being “stable” relative to an executing loop if the values used by that statement are the same on every iteration of that loop. The loop-relative stability heuristic determines whether to propagate along a particular control flow based on the stabilities of statement executions. More specifically, let \(b\) be an execution of conditional branching statement and \(a\) be an execution of an assignment statement that is within the scope of \(b\)’s control flow. The loop-relative stability heuristic propagates along the control flow from \(b\) to \(a\) only if \(b\) is stable relative to every loop to which \(a\) is relatively stable.

The loop-relative stability heuristic considers only “natural” loops as defined by Reference [2]. In particular, a natural loop is a single-point-of-entry cycle in the control flow graph defined with respect to a back edge, i.e., an edge whose target dominates its source. The natural loop of some back edge \((u, v)\) consists of all nodes \(x\) such that \(v\) dominates \(x\) and there exists a path from \(x\) to \(u\) not containing \(v\). The node \(v\) is said to be the header of the natural loop defined by the back edge \((u, v)\). Any two natural loops with the same header are combined and treated as a single loop. This definition of loops ensures that any two loops are either disjoint or nested, i.e., one loop is fully contained within the other. The use of arbitrary \(\text {GOTO}\) statements may in some cases produce control flow graphs that contain cycles that do not correspond to natural loops. However, most structured programming languages do not allow programmers to produce control flow graphs that contain cycles that do not correspond to natural loops. Thus, we feel that it is appropriate for the loop-relative stability heuristic to only consider natural loops.

3.2.1 Instability Levels.

Conceptually, the loop-relative stability heuristic could be expressed in terms of “stability sets”; a stability set is the set of loops with respect to which a statement execution is relatively stable. Let \(S_b\) be the stability set of an execution of conditional branching statement \(b\) and \(S_a\) be the stability set of an execution of an assignment statement within the scope of \(b\)’s control flow. The loop-relative stability heuristic propagates along the control flow from \(b\) to \(a\) only if \(S_b\) is a superset of \(S_a\). However, instead of tracking these stability sets, it is possible to express the same concept using a single number, an “instability level.” An instability level is a numeric value between zero and the number of loops containing the statement currently executing. This number represents the “depth” of the innermost loop relative to which a statement is not stable. The loop-relative stability heuristic propagates along the control flow from an execution of a conditional branching statement \(b\) to an execution of an assignment statement \(a\) within the scope of \(b\)’s control flow only if \(b\)’s instability level is less than or equal to \(a\)’s instability level. An explanation of why this simplification is possible is provided below.

Given a statement execution contained within two nested loops, if the values used by the statement change over the duration of the inner loop, then they must also change over the duration of the outer loop, since the inner loop is fully contained within the outer loop. Thus, the stability set of a statement execution is defined by the innermost loop relative to which it is not stable. If an execution of a program statement occurs outside of a loop, then that execution must be stable relative to that loop, since it is not possible for the values used by the statement to change over the duration of the loop. As a result, when calculating the loop-relative stability for the statement currently executing, only the set of loops containing that statement needs to be considered in the calculation.

However, this does not by itself guarantee that all comparisons for the loop-relative stability heuristic made at runtime only need to consider the set of loops containing the statement currently executing. The stability set of the execution of a conditional branching statement needs to be used before the execution of any assignment statement within the scope of the branch’s control flow. This means that the stability set of an execution of a conditional branching statement may be considered by the loop-relative stability heuristic after the innermost loop relative to which the execution was not stable was exited. However, once the innermost loop relative to which an execution of a conditional branching statement is not stable is exited, all subsequent statements will execute outside of that loop and therefore be stable relative to it. Since these executions are stable relative to a loop that the branch execution was not, the loop-relative stability heuristic will prevent the branch’s predicate from propagating to these statements as a result of the control flow. Thus, Conflux stops all propagation along the control flow from the execution of a conditional branching statement as soon as the innermost loop relative to which that branch execution is not stable is exited. As a result, all comparisons for the loop-relative stability heuristic made at runtime only need to consider the set of loops containing the statement currently executing. Given this and the fact that the stability set of a statement execution can be defined by the innermost loop relative to which it is not stable, loop-relative stabilities can be specified at runtime as a single number, an instability level, between zero and the number of loops containing the statement currently executing.

These instability levels are calculated at runtime as a function of both static information, the stability “classifier” of a statement (Section 3.2.2), and dynamic information, the context of the statement’s execution (Section 3.2.3). A purely dynamic approach could only consider instructions that have already executed. This is problematic because it is possible for the predicate of a conditional branching statement to be the same on all but the last iteration of a loop. By the time that last iteration occurs the loop-relative stability heuristic may have already been applied, causing Conflux to incorrectly propagate along a control flow from that branch. Furthermore, the loop-relative stability is concerned with the presence of alternative conditions that could have produced the same outcome. It does not matter whether those conditions were met on a particular execution. For example, consider the \(\text {any}\) method on line 3 of Listing 2. The for-loop on lines 4–8 introduces multiple conditions under which the value of \(\text {x}\) is set to \(\text {true}\), specifically if any of the elements of the array \(\text {z}\) is \(\text {true}\). If \(\text {any}\) is passed an array |new boolean[]false, false, false, true, then there happened to be one condition, \(z[3] == true\) that was satisfied and caused \(\text {x}\) to be assigned the value \(\text {true}\). However, there were still possible alternative conditions under which that assignment could have occurred. Therefore, propagation should not occur from the branch on line 5 to the assignment statement on line 6 according to the loop-relative stability heuristic. To handle this, Conflux uses static information to reason about possible executions and assign stability classifiers to program elements.

Listing 2.

View Figure

Conflux also relies upon dynamic information about calling contexts. Consider the \(any\_\) method on line 15 of Listing 2. The \(any\_\) method is functionally equivalent to the \(\text {any}\) method on line 3 of Listing 2, but this functionality is split across two methods, \(any\_\) and \(\text {setX}\). Because \(any\_\) is functionally equivalent to \(\text {any}\), taint tag propagation for the two methods should ideally be the same, meaning that propagation should not occur during the execution of the assignment statement on line 12 of the \(\text {setX}\) method. However, \(\text {setX}\) is also called by the method \(\text {checkFirst}\) on line 25 of Listing 2. The conditional branching statement on line 24 of \(\text {checkFirst}\) does not necessarily occur within a loop. Therefore, propagation may need to occur during the execution of the assignment statement on line 12 of the \(\text {setX}\) method when called from the \(\text {checkFirst}\) method. Thus, it is not possible to determine whether propagation should occur during the execution of the assignment statement on line 12 without considering the dynamic execution context of the call to \(\text {setX}\). Thus, Conflux constructs and passes between methods information about execution contexts at runtime. These execution contexts are used to calculate instability levels.

3.2.2 Stability Classifiers.

Conflux statically assigns stability classifiers to program elements; these classifiers encode information about possible program executions. There are three types of stability classifiers: stable, dependent, and unstable. The stable classifier type indicates that all executions of a program element are stable relative to all loops regardless of the dynamic execution context. We use Stable to denote a stable-type classifier. The dependent classifier type indicates that the instability level of an execution of a program element depends on the instability level of one or more arguments passed to the method call that contains the execution. We refer to the set of parameters corresponding to these arguments as the dependency set of the classifier. We use \({\rm D}{\rm\small{EPENDENT}}\langle d \rangle\) to denote a dependent-type classifier with a dependency set \(d\). In addition to dependencies on arguments, a dependent-type classifier can also indicate a dependency on the execution context’s return value location. We use \(\alpha\) to denote a dependency on the execution context’s return value location.

The unstable classifier type indicates that the instability level of an execution of a program element depends on the loops that contain the method call that contains the execution. In this case, the element is not stable with respect to all loops containing the method call and possibly additional loops within the method. We use \({\rm U}{\rm\small{NSTABLE}}\langle l \rangle\) to denote an unstable-type classifier for a program element that is not stable relative to the elements of \(l\), a set of loops within the method.

At runtime, Conflux determines instability levels based on stability classifiers and execution contexts. The instability level of a stable-type classifier is always zero. The instability level of a dependent-type classifier is dynamically calculated as the maximum instability level of the arguments corresponding to the parameters in the program element’s dependency set. The instability level of an unstable-type classifier, \({\rm U}{\rm\small{NSTABLE}}\langle l \rangle\) , is dynamically calculated as the number of loops containing the method call plus \(\vert l \vert\). We describe this more precisely in Section 3.2.4 below.

To calculate stability classifiers for a method, Conflux first converts the method into an intermediate representation in static single assignment (SSA) form. By converting the method into SSA form, Conflux can easily identify reaching definitions, since there will be exactly one definition reaching each use. Next, Conflux creates a control flow graph for the method and uses this graph to identify which natural loops within the method contain each instruction. We use \(\operatorname{loops}(i)\) to denote the set of natural loops containing an instruction \(i\). Finally, Conflux calculates the stability classifier of each conditional branching statement, assignment statement, non-void return statement, method call receiver, method call argument, and method call return value location. The stability classifiers of conditional branching statements, assignment statements, and non-void return statements (which Conflux treats like interprocedural assignment statements) are directly used to determine whether to propagate along a control flow in accordance with the loop-relative stability heuristic. To construct an execution context for a method call, Conflux uses the stability classifier of the method call’s receiver, arguments, and return value location. We discuss this in detail in Section 3.2.3.

For the sake of computing stability classifiers, we define a function \(\operatorname{merge}(c_1, c_2)\) in Algorithm 1 for combining two classifiers, \(c_1\) and \(c_2\) to produce a classifier \(c_3\) such that for any possible execution context the instability level of \(c_3\) will be greater than or equal to the the instability level for \(c_1\) and \(c_2\). Using this \(\operatorname{merge}\) function, Conflux can then calculate stability classifiers for value expressions (program entities that are evaluated to produce a value) and storage locations (program entities that represent a place for storing a value, i.e., the left-hand side of an assignment statement) as described in Algorithms 2 and 3, respectively.

Conflux directly uses the \(\text {valueClassifier}\) function (Algorithm 2) to calculate stability classifiers for method call receivers and arguments, since these program elements are value expressions. The \(\text {valueClassifier}\) function (Algorithm 2) is also used to calculate the stability classifier of each conditional branching statement by applying the function to the predicate expression of the branching statement. To calculate the stability classifier of a non-void return statement, Conflux first calculates the stability classifier of the expression returned by the statement and then combines this classifier with a dependent-type classifier with a single dependency on the execution context’s return value location, More formally, the stability classifier of a non-void return statement is \({merge({\rm D}{\rm\small{EPENDENT}}\langle \lbrace \alpha \rbrace \rangle , c_v)}\), where \(c_v\) is the stability classifier of the expression returned by the statement. The stability classifier of a method call return value location is determined by applying the \(\text {locationClassifier}\) (Algorithm 3) function to the storage location assigned the value of the result of the method call. If the return value of a method call is unused, then the stability classifier of the return value location for that method call is Stable.

Conflux uses both the \(\text {valueClassifier}\) and \(\text {locationClassifier}\) functions to calculate the stability classifier of an assignment statement. However, Conflux makes a special consideration when calculating the stability classifiers of assignment statements to address situations in which the value to be stored by an assignment was produced from an expression containing a term matching the current value at the destination storage location of the assignment statement. We refer to this as an “update” assignment and exclude any update terms, terms that match the current value at the storage location, from the stability calculation for the value. Thus, before Conflux can determine the stability classifier of an assignment statement, it must identify any portions of the right-hand side of the assignment statement that correspond to excluded update terms. If the destination storage location of the assignment statement is a local variable, then any uses of that definition of that local variable that reach the assignment statement are considered to be update terms. If the destination storage location is an array, then Conflux considers any values that are loaded from the same array at the same position to be candidate update terms. These values may represent update terms, but it is also possible that the value at that storage location was written between the instruction that loaded the candidate update term and the assignment statement. Conflux ignores the potential for concurrent writes to that storage location and accepts a candidate as being a true update term if there does not exist an execution path containing an array store or a method call between the instruction that loaded the candidate and the assignment statement. In practice, checking all possible paths between the two instructions may be impractical. Thus, for small control flow graphs (those containing less than 10 basic blocks), Conflux checks all possible paths between the two instructions. For larger control flow graphs, Conflux assumes that such a path exists if the two instructions are not in the same basic block. If the destination storage location of the assignment statement is a field, then Conflux considers any values loaded from the same field and, for instance fields, from the same instance to be candidate update terms. Once again, Conflux checks all paths between the instruction that loads the candidate update term and the assignment statement. However, in this case, the paths are checked for method calls and field stores.

Once Conflux has identified any portions of the right-hand side of the assignment statement that are considered to be excluded update terms, it calculates the stability classifier of the assignment statement based on the remaining portions of the right-hand side of the assignment statement and the left-hand side of the assignment statement (i.e., its destination storage location). Algorithm 4 shows how Conflux performs this calculation.

3.2.4 Applying the Loop-relative Stability Heuristic.

At runtime, Conflux combines the static stability classifier (Section 3.2.2) and the dynamic execution context (Section 3.2.3) of an executing program element to compute its instability level. This computation is detailed in Algorithm 5. Computed instability levels are used to construct execution contexts as discussed in Section 3.2.3 and, ultimately, to apply the loop-relative stability heuristic. In particular, let \(b\) be an execution of a conditional branching statement and \(a\) be an execution of an assignment statement that is within the scope of \(b\)’s control flow. The loop-relative stability heuristic propagates along the control flow from \(b\) to \(a\) only if \(b\)’s instability level is less than or equal to \(a\)’s instability level.

For example, consider the programs shown in Listings 3 and 4 in Figure 3. These programs are nearly identical; they differ only in the names of methods and the value passed to the call to the method \(\text {set}\) on line 13. However, this slight, semantic difference is enough to the impact instability levels in a way that produces different taint tag propagation.

Fig. 3.

View Figure

Consider two program executions: one starting from the \(\text {main}\) method in Listing 3 and the other starting from the \(\text {main}\) method in Listing 4. Both executions proceed as follows: The \(\text {main}\) method calls an intermediate method, \(\text {indexOf}\) in one case and \(\text {contains}\) in the other. This intermediate method contains one natural loop \(L\) (the for-loop on lines 9–15). In both executions, there are two iterations of this loop. On the first iteration, when \(\text {i}\) is \(0\), the predicate of the branch on line 12 evaluates to true, causing a call to be made to the \(\text {set}\) method. The \(\text {set}\) method assigns the value 0 to a static field \(\text {x}\) and then returns. On the second iteration of \(L\), the predicate of the branch on line 12 evaluates to false, and no call is made to the \(\text {set}\) method. After the loop \(L\) ends, the intermediate method returns. Then, finally, the method \(\text {main}\) returns.

In both executions, when the branch on line 12 of the intermediate method was taken, the value of the operand \(a[0]\) of the predicate of that branch was tainted (as depicted in the middle frame of Figures 3(a) and 3(b)). Thus, when the assignment statement on line 4 of \(\text {set}\) executed, it wrote a value within the scope of a control flow introduced by a tainted predicate. Figure 3(a) shows the stack trace for the moment just before the assignment statement on line 4 of the \(\text {set}\) method executed for the execution of the program in Listing 3. Similarly, Figure 3(b) shows the stack trace for the same moment for the execution of the program in Listing 4. To determine whether to propagate along these control flows, Conflux must know the static stability classifiers for the methods called in these two executions and the dynamic execution contexts of the calls made at runtime to these methods.

We annotated Listings 3 and 4 to indicate the stability classifiers of program elements. These stability classifiers are statically determined by applying the rules described in Section 3.2.2. The assignment statement \(x = y\) on line 4 of both programs has a stability classifier of \({\rm D}{\rm\small{EPENDENT}}\langle \lbrace y\rbrace \rangle\) . The storage location \(\text {x}\) always refers to the same location regardless of the context in which it executes. However, whether the value represented by the expression \(\text {y}\) changes over the iterations of some theoretical loop containing a call to \(\text {set}\) depends on whether the argument \(\text {y}\) changes over the iterations of that loop. Thus, this assignment statement has a stability classifier of \({\rm D}{\rm\small{EPENDENT}}\langle \lbrace y\rbrace \rangle\) . The assignment statement \(i = 0\) on line 9 of both programs has a stability classifier of Stable because, regardless of the context in which it executes, it always assigns the same value, \(0\), to the same location, the local variable \(\text {i}\). The statement \(i++\) on line 11 of both programs is an augmented assignment equivalent to \(i = i + 1\). The term \(\text {i}\) in the assignment \(i = i + 1\) is an excluded update term, and the remaining term \(1\) is constant. Therefore, the statement \(i++\) on line 11 of both programs has a stability classifier of Stable. The stability classifiers of the branch on line 10 of both programs (\(i \lt a.length\)) and the branch on line 12 of both programs (\(a[i] == ^{\prime }h^{\prime }\)) are both \({\rm U}{\rm\small{NSTABLE}}\langle \lbrace L\rbrace \rangle\) , because both statements use the value of \(\text {i}\), which changes over the iterations of \(L\). The argument \(\text {i}\) passed to the call to \(\text {set}\) on line 13 of Listing 3 has a stability classifier of \({\rm U}{\rm\small{NSTABLE}}\langle \lbrace L\rbrace \rangle\) , because the value of \(\text {i}\) changes over the iterations of \(L\). By contrast, the argument \(0\) passed to the call to \(\text {set}\) on line 13 of Listing 4 has a stability classifier of Stable because it is a literal. The assignment statement on lines 19–22 of both programs contains a storage allocation, thus its stability classifier is \({\rm U}{\rm\small{NSTABLE}}\langle \emptyset \rangle\) . The argument \(\text {a}\) passed to the method call on line 23 of both programs has a stability classifier \({\rm U}{\rm\small{NSTABLE}}\langle \emptyset \rangle\) , because the value of the reaching definition of \(\text {a}\) from lines 19–22 contains a storage allocation.

In addition to these stability classifiers, Conflux must also calculate the dynamic execution contexts of method calls according to the rules described in Section 3.2.3. Let \(e_{main}\), \(e_{indexOf}\), and \(e_{set}\) denote the execution contexts for the calls made to \(\text {main}\), \(\text {indexOf}\), and \(\text {set}\), respectively, in the program execution for Listing 3. Let \(e_{main^{\prime }}\), \(e_{contains}\), and \(e_{set^{\prime }}\) denote the execution contexts for the calls made to \(\text {main}\), \(\text {contains}\), and \(\text {set}\), respectively, in the program execution for Listing 4.

Since \(\text {main}\) is the program entry point, \(e_{main}\) and \(e_{main^{\prime }}\) are initial execution contexts. Thus, \({\rm depth}(e_{main})\) and \({\rm depth}(e_{main^{\prime }})\) are 0, and \({\rm level}(e_{main},\) \(\text {args}\)\()\) and \({\rm level}(e_{main^{\prime }},\) \(\text {args}\)\()\) are also 0. The bottom frame of Figure 3(a) (labeled “main: line 23”) depicts \(e_{main}\), and the bottom frame of Figure 3(b) (labeled “main: line 23”) depicts \(e_{main^{\prime }}\).

Conflux constructs \(e_{indexOf}\) using \(e_{main}\) and \(e_{contains}\) using \(e_{main^{\prime }}\). Since there are no natural loops inside \(\text {main}\) that contain the call to \(\text {indexOf}\), \({\rm depth}(e_{indexOf})\) is \({\rm depth}(e_{main})\) + \(0 = 0\). The argument passed to the call to \(\text {indexOf}\) has a stability classifier of \({\rm U}{\rm\small{NSTABLE}}\langle \emptyset \rangle\) , thus \({\rm level}(e_{indexOf},\) \(\text {a}\)\()\) is \({\rm depth}(e_{main})\) + \(\vert \emptyset \vert = 0\). The middle frame of Figure 3(a) (labeled “indexOf: line 13”) depicts \(e_{indexOf}\). Since the \(\text {main}\) methods for the two programs are identical and \(e_{main} = e_{main^{\prime }}\), \(e_{contains}\) is identical to \(e_{indexOf}\). The middle frame of Figure 3(b) (labeled “contains: line 13”) depicts \(e_{contains}\).

Next, Conflux constructs \(e_{set}\) using \(e_{indexOf}\) and \(e_{set^{\prime }}\) using \(e_{contains}\). Since the loop \(L\) contains the call to \(\text {set}\) in \(\text {indexOf}\), \({\rm depth}(e_{set})\) is \({\rm depth}(e_{indexOf})\) + \(1 = 1\). For the same reason, \({\rm depth}(e_{set^{\prime }})\) is \({\rm depth}(e_{contains})\) + \(1 = 1\) The argument passed to the call to \(\text {set}\) in \(\text {indexOf}\) has a stability classifier of \({\rm U}{\rm\small{NSTABLE}}\langle \lbrace L\rbrace \rangle\) . Therefore, \({\rm level}(e_{set},\) \(\text {y}\)\()\) is \({\rm depth}(e_{indexOf})\) + \(\vert \lbrace L\rbrace \vert = 1\). However, the argument passed to the call to \(\text {set}\) in \(\text {contains}\) has a stability classifier of Stable. Therefore, \({\rm level}(e_{set^{\prime }},\) \(\text {y}\)\()\) is 0. The top frame of Figure 3(a) (labeled “set: line 4”) depicts \(e_{set}\). The top frame of Figure 3(b) (labeled “set: line 4”) depicts \(e_{set^{\prime }}\).

Finally, these execution contexts and stability classifiers can be used to calculate instability levels for the execution of the program in Listing 3. The stability classifier of the branch on line 12 of \(\text {indexOf}\) is \({\rm U}{\rm\small{NSTABLE}}\langle \lbrace L\rbrace \rangle\) , and the execution of this branch occurred within the runtime execution context \(e_{indexOf}\). Therefore, the instability level of the execution of the conditional branching statement is \({\rm depth}(e_{indexOf})\) + \(\vert \lbrace L\rbrace \vert = 1\). The stability classifier of the assignment statement on line 4 of \(\text {set}\) is \({\rm D}{\rm\small{EPENDENT}}\langle \lbrace y\rbrace \rangle\) . The execution of the assignment statement occurred within the runtime execution context \(e_{set}\). Therefore, the instability level of the execution of the assignment statement is \(\max _{a \in \lbrace y\rbrace } \operatorname{level}(e_{set}, a) = 1\). Overall, the instability level of the execution of the conditional branching statement was less than or equal to that of the execution of the assignment statement. Thus, following the loop-relative stability heuristic, Conflux would propagate along the control flow between the two statement executions.

As was done for the execution of the program in Listing 3, the execution contexts and stability classifiers can be used to calculate instability levels for the execution of the program in Listing 4. The stability classifier of the branch on line 12 of \(\text {contains}\) is \({\rm U}{\rm\small{NSTABLE}}\langle \lbrace L\rbrace \rangle\) , and the execution of this branch occurred within the runtime execution context \(e_{contains}\). Therefore, the instability level of the execution of the conditional branching statement is \({\rm depth}(e_{contains})\) + \(\vert \lbrace L\rbrace \vert = 1\). The stability classifier of the assignment statement on line 4 of \(\text {set}\) is \({\rm D}{\rm\small{EPENDENT}}\langle \lbrace y\rbrace \rangle\) . The execution of the assignment statement occurred within the runtime execution context \(e_{set^{\prime }}\). So, the instability level of the execution of the assignment statement is \(\max _{a \in \lbrace y\rbrace } \operatorname{level}(e_{set^{\prime }}, a) = 0\). In this case, the instability level of the execution of the conditional branching statement was greater than that of the execution of the assignment statement. Therefore, unlike in the other execution, Conflux would not propagate along the control flow between the two statement executions.

4.1 Annotation

Before a method is instrumented, Conflux statically annotates its instructions with control flow information. These annotations mark static features of the method (e.g., the scopes of control flows) as well as properties of those features (e.g., stability classifiers). When the method is instrumented, these features and properties are used to determine what code to generate.

Conflux starts by adding annotations to delineate the binding scopes of control flows. The control flow graph of the method being annotated is created to identify branch edges that introduce control flows. Each branch edge that performs an equality check is marked as introducing a control flow that should be propagated at runtime. A branch edge is considered to perform an equality check if one of the following is true:

• it is the edge between a branching instruction conditioned on a non-null equality check and its branch target;

• it is the edge between a branching instruction conditioned on a non-null inequality check and the instruction following it;

• it is the edge between a \(\text {switch}\) instruction and the branch target associated with one of its non-default cases and no other case for that \(\text {switch}\) instruction has the same branch target;

• it is either of the edges out of a branching instruction conditioned on a Boolean comparison.

The last condition, which deals with Boolean comparisons, is a special case. Since there are only two possible values for a Boolean, both edges out of the branch are traversed for only a single value. Thus, both edges correspond to equality checks. The Java Virtual Machine does not provide dedicated Boolean instructions and instead uses instructions that operate on integer values [44]. As a result, Conflux will mark both edges of an integer conditional branch instruction for propagation if it statically determines that the instruction likely performs a Boolean comparison.

Once branch edges have been marked for propagation, Conflux annotates the start of the scope of the control flow associated with each of the edges. Then, it constructs the modified control flow graph described in Section 3.1 and uses Cooper et al. [23]’s algorithm to calculate the dominance frontier of each replacement node (\(b_{e}\)) associated with each marked branch edge (\(e\)). An annotation is added at the beginning of each basic block in the dominance frontier of a replacement node to indicate the end of the scope of the control flow associated with the node.

Next, Conflux annotates program elements with their stability classifiers. The method being annotated is converted from Java bytecode into a register-based intermediate representation and then placed into SSA form using Cytron et al. [25]’s algorithm. Conflux maintains a direct correspondence between this intermediate representation and the original Java bytecode so properties calculated on the intermediate representation can be used to annotate instructions in the original bytecode without having to convert out of the intermediate form. This intermediate representation is used to calculate the stability classifiers of program elements in the method being analyzed (as described in Section 3.2.2). Conflux labels each conditional branching statement, assignment statement, non-void return statement, method call receiver, method call argument, and method call return value location with its calculated stability classifier.

Finally, Conflux uses the control flow graph for the method to record loop information. Each method call is annotated with the number of natural loops that contains it within the method being annotated. Additionally, Conflux records the exit points (i.e., the first instruction of a basic block not contained within a particular loop whose predecessor is contained within that loop) of any loops within the method. These features of the method are used at runtime to calculate execution contexts and instability levels.

4.2 Instrumentation and Runtime

During the instrumentation of the method, the annotations added by Conflux are used to generate method calls to the \(\text {ControlFlowStack}\), the structure Phosphor uses to store the taint tags of branches and propagate control flows [11]. Phosphor’s default behavior adds method calls to push the taint tag associated with each control flow’s predicate at the start of its scope and pop any taint tags pushed for each control flow at the end of its scope, as described in Section 2. Conflux modifies this behavior to support the loop-relative stability heuristic.

Before making a method call, the \(\text {ControlFlowStack}\) prepares an execution context for the call. The instability levels of the receiver, arguments, and return value location of the method call are calculated based on their stability classifiers and the current method call’s execution context. These instability levels are recorded by the \(\text {ControlFlowStack}\). \(\text {ControlFlowStack}\) also calculates the number of loops containing the method call based on the number of loops within the current method containing the method call and the current method call’s execution context.

At the start of the callee method, the \(\text {ControlFlowStack}\) initializes an execution context based on the values prepared by the caller method. Then, the \(\text {ControlFlowStack}\) pushes this new execution context onto a stack of execution contexts. Before a method exits, the current method call’s execution context is popped from this stack. When the start of a control flow’s scope is hit, the \(\text {ControlFlowStack}\) calculates the instability level of the control flow based on the conditional branching statement’s stability classifier and the current method call’s execution context. Then, the \(\text {ControlFlowStack}\) records the taint tag of the predicate of the branch with the flow’s instability level. At the end of the scope of a control flow, any taint tags recorded for the flow are cleared. As discussed in Section 3.2.1, to use instability levels, all control flows from a branch’s predicate must be terminated as soon as the innermost loop relative to which that branch is not stable is exited. Thus, at the exit point of a loop any taint tags recorded at the instability level associated with that loop are removed from the \(\text {ControlFlowStack}\).

When a non-void return statement or assignment statement executes, the \(\text {ControlFlowStack}\) calculates its instability level based on its stability classifier and the current method call’s execution context. The taint tags of any control flow recorded at an instability level less than or equal the instability level of the statement propagate to the program data being assigned a value by the statement.

At instrumentation boundaries, such as native code calls, the callee method will not receive a prepared execution context from the caller. In these cases, Conflux assumes that no loops contain the method call. If there are no loops that contain the method call, then there are no loops with respect to which the method call’s receiver, arguments, and return value location could be non-stable. Thus, Conflux uses an execution context \(e\) such that \(depth(e) = 0\) and \(\operatorname{level}(e, a) = 0\) for all method elements \(a\).

6.1 Experimental Setup

We evaluated Conflux in comparison to three other propagation policies: data-only, basic-control, and SCD, an implementation of the control flow semantics presented by Bao et al. [8]. Each of these policies was implemented using Phosphor. All four of the policies propagate taint tags along data flows in a similar fashion to what was described by Bell and Kaiser [10]. Additionally, all of the policies propagate taint tags from an index used to access an element of an array to the element accessed; this applies to both read and write accesses. Only the data-only and basic-control policies propagate taint tags through instanceof operations, as that instruction does not match the semantics targeted by Conflux and SCD. Conflux, basic-control, and SCD all propagate taint tags through pointer dereferencing. Each of the policies uses different semantics for propagating control flows. data-only does not propagate along control flows. basic-control propagates along all control flows using the standard scoping semantics described in Section 2. SCD uses the standard scoping semantics, but only propagates along edges corresponding to equality checks. These edges are determined using the same rules described in Section 4.1.

All of our experiments were conducted on OpenJDK version 1.8.0_222. Tests for each propagation policy were run in a JVM that was instrumented according to the policy. Before a test started, any taint tags on values in the JVM were cleared to ensure that taint tags from one test could not impact the results of another test.

6.2 Benchmark

The benchmark we created to answer RQ1 and RQ2 consists of methods for encoding and decoding text drawn from the OpenJDK Java Class Library (version 1.8.0_222) [54] and seven different real-world Java projects:

• Apache Commons Text (version 1.8) [4]

• Apache Commons Codec (version 1.14) [3]

• Bouncy Castle Provider (version 1.46) [43]

• Guava (version 28.2-jre) [29]

• jsoup (version 1.11.3) [39]

• Spring Web (version 5.2.5) [57]

• Tomcat Embed Core (version 9.0.19) [5],

To the best of our knowledge, DroidBench [6] is the only existing Java taint tracking benchmark that contains tests that consider control flows. However, DroidBench only contains four tests involving control flows and was designed for evaluating static analyzers. Thus, we choose not to use it in our evaluation.

The programs featured in our benchmark for encoding and decoding text are all information-preserving. Each of these programs transforms one representation of a sequence of abstract entities into another representation of the same sequence of abstract entities. An abstract entity is an atomic unit of information. For example, when escaping text for inclusion in HTML, the character <, the character entity reference <, and the numeric character reference &#60; all represent the same abstract entity. The information-preserving nature of the programs included in our benchmark provides a clear ground truth: The expected set of taint tags for a program output contains the taint tags of the program inputs that represent the same abstract entity that the output represents. For example, a program might perform percent encoding on the string :@ resulting in the output string %3A%40. The first percent-encoded octet, %3A, represents the same abstract entity represented by the input character :. The second percent-encoded octet, %40, represents the same abstract entity represented by the input character @. Thus, the expected label set for each of the characters in the first octet would contain the unique label assigned to :, and the expected label set for each of the characters in the second octet would contain the unique label assigned to @.

Each method selected for the benchmark we created had to meet several criteria. First, the method had to be deterministic. For valid input, the output of the method had to represent the same sequence of abstract entities as the input. A taint tracking policy that does not propagate control flows should not produce false positives when tracking taint tags through the method. Likewise, a taint tracking policy that propagates along every control flow using the standard semantics described in Section 2 should not produce false negatives when tracking taint tags through the method. This limits the scope of the benchmark to situations in which observed under-tainting or over-tainting is likely related to control flow propagation, as opposed to other sources of imprecision such as caches and bit-packing.

Every test in the benchmark uses one of the selected methods and follows the same general format. The test starts by creating an input representing a sequence of abstract entities appropriate for the method. Each character (or byte in the case of hex encoding) in the input is assigned a unique taint label. The test then transforms the tainted input using the target test method. The taint label set that is propagated to each character (or byte in the case of hex decoding) of the method’s output is compared to the set of labels expected for that element. Labels in both the propagated and expected sets are counted as true positives. Labels in the expected set, but not the propagated set, are counted as false negatives. Labels in the propagated set, but not the expected set, are counted as false positives.

In some cases, a single method is used in multiple tests (e.g., the method \(PercentCodec.encode()\) from Apache Commons Codec is used in tests in the groups unicode-percent-decode and reserved-percent-decode). In these instances, we choose to separate different transformations performed by a single method into multiple tests so their results could be more directly compared to a different implementation of the same transformation.

6.3 RQ1: Accuracy

To compare the accuracy of data-only, basic-control, SCD, and Conflux, we applied each of the propagation policies to the tests in our benchmarks. For each test, we created an input sequence of eight abstract entities (RQ2 considers different length inputs). We recorded the number of false negatives, false positives, and true positives that were reported by each propagation policy on each test.

Table 1 presents our findings for the different propagation policies on the benchmarks. Overall, Conflux had the highest F1 score on a majority of the tests, 43 out of the 48 total tests. data-only had the highest F1 score on 23 tests. SCD had the highest F1 score on 21 tests. And finally, basic-control had the highest F1 score on only 3 tests.

Table 1.

View Table

Table 1. Comparison of data-only, basic-control, SCD, and Conflux on Our Benchmark

Due to the selection criteria for the methods used in the benchmark, data-only reports no false positives and basic-control reports no false negatives. In some of the tests, data-only fails to propagate any tags to the output, because there are no data flows between the input and the output. For example, in the html-escape test using jsoup’s \(\text {Entities}\) class, input values flow into a \(\text {switch}\) statement that selects a constant string to append to the output; all of the information that flows between the input and the output is transferred through the \(\text {switch}\) statement. In other tests, data-only does report some or all of the possible true positives, because data flows are able to fully or partially capture the relationship between the input and the output. For instance, for tests in the reserved-percent-encode group, data-only reports two true positives for every false negative. This occurs because tests in this group take a sequence of URI-reserved characters and encode them using percent-encoded octets (e.g., the character @ would end up encoded as %40). There is a data flow between the value of the input character and the two hex digits in the octet, but there is only a control flow between the input character and the percent sign. Thus, data-only correctly tracks the relationship between an input character and the two hex digits of the output octet. However, it misses the relationship between an input character and the percent sign of the output octet. By contrast, basic-control never missed a relationship between an input and an output, but reported a relatively large number of false positives on all but three tests. In many cases, basic-control marked all of the inputs as being related to all of the outputs.

SCD reported relatively few false negatives, and 102 out of the 132 of these false negatives occurred in tests from the unicode-percent-encode group. Tests in the unicode-percent-encode group take a sequence of non-ASCII characters and transform them into UTF-8, percent-encoded octets. Typical implementations of this transformation determine whether an input character needs to be encoded either by checking if it falls into some value range, is outside some value range, or is not present in some set of values that do not need to be encoded. These checks are generally not equality checks, so SCD does not propagate along the branches associated with them, resulting in under-tainting. Conflux also under-taints in these tests for the same reason.

Conflux only reported false positives in two tests, as opposed to the 28 tests in which SCD reported false positives and the 45 tests in which basic-control reported false positives. Both of these two tests have the same problematic flow. A simplified version of this flow is shown in Listing 5. Conflux marks both the branch on line 6 and the statement on line 12 as unstable with respect to all loops that contain them. Thus, Conflux propagates taint tags from the predicate of the branch on line 6 to the assigned value on line 12, \(\text {c}\), resulting in over-tainting. In this case, the loop header (\(c == ^{\prime }\%^{\prime }\) Had the loop header instead been written as \(input[inputPosition] == ^{\prime }\%^{\prime }\)

Listing 5.

View Figure

6.4 RQ2: Accuracy versus Input Size

Taint tags often accumulate on program data over loop iterations leading to progressively more over-tainting on each iteration. Many common applications of taint tracking (e.g., fuzzing guidance) tend to use relatively large inputs that often trigger a large number of loop iterations. In these cases, it may be impractical to use the standard semantics for control flow propagation due to the “explosion” of taint tags resulting from their accumulation in loops. To evaluate whether Conflux could be used to address this issue, we applied the propagation policies (data-only, basic-control, SCD, and Conflux) to our benchmark with inputs of various lengths (8, 16, 32, 64, 128, 256, 512, and 1,024 abstract entities). We then recorded the F1 score for each policy on each test for each of the lengths and produced a series of plots. We examined these plots to determine how the F1 score for each policy changed as the size of the input scaled. Figure 5 shows four of these plots that represent common cases seen across many of the plots.

Fig. 5.

View Figure

In all of the tests the F1 score for data-only was constant as the size of the input increased. The behavior of the F1 scores for the other three policies either stayed constant, decreased to some non-zero value, or decreased to zero as the size of the input increased. We divided tests into categories based on how the F1 score reported for the different policies changed as the size of the input increased.

There were 3 tests where the F1 scores for all of the policies remained constant as the input size increased. In 17 tests, the F1 scores for data-only, SCD, and Conflux remained constant, but the F1 scores for basic-control decreased to zero. A plot for one of these tests is depicted in Figure 5(a). In 6 tests, the F1 scores for data-only and Conflux remained constant, the F1 scores for basic-control decreased to zero, and the F1 scores for SCD decreased to some non-zero value. A plot for one of these tests is depicted in Figure 5(b). In 20 of the tests, the F1 scores for data-only and Conflux remained constant, but the F1 scores for basic-control and SCD decreased to zero. A plot for one of these tests is depicted in Figure 5(c). There were only 2 tests in which the F1 scores for data-only remained constant, and the F1 scores for basic-control, SCD, and Conflux decreased to zero. A plot for one of these tests is depicted in Figure 5(d).

Overall, Conflux’s F1 score stayed constant in all but 2 tests, similar to data-only. By contrast, the F1 score of basic-control and SCD typically degraded as input sizes scaled. In this respect, Conflux’s control flow tracking behaved more similarly to data flow tracking than the control flow tracking performed by basic-control and SCD.

6.5 RQ3: Impact on a Concrete Application of Taint Tracking

To examine the effect of Conflux on a practical application of taint tracking, we implemented a prototype of Clause and Orso [19]’s approach for identifying which failure-inducing inputs (i.e., inputs that produce a failure) are failure-relevant (i.e., useful for analyzing the failure). We used this prototype to perform a case study exploring the impact of Conflux on Clause and Orso [19]’s approach. We conducted an experiment similar to the one originally performed by Clause and Orso [19]. In this experiment, we provide a qualitative assessment of the failure-inducing inputs that were marked as relevant by the different propagation policies, data-only, basic-control, SCD, and Conflux, on five failures from popular, open-source Java projects. Our assessment is limited to the impact of the propagation policies on the values marked as failure-relevant; it does not examine the usefulness of Clause and Orso [19]’s approach in general, since that was explored in the original work.

Table 2 details the five failures that we included in our case study. Each of the these failures was chosen from an issue reported in a project’s issue tracker that described a system failure. Only issues in which the system accepted a human-interpretable input or inputs were considered, since it would otherwise be difficult to apply Clause and Orso [19]’s approach. For the sake of simplicity, we only selected issues in which the described failure could be consistently reproduced. Furthermore, we only selected issues that had already been fixed in a single, associated commit to facilitate analysis of the failure. A similar criterion was used by Just et al. [40] in the selection of bugs for Defects4J, a widely used database of real Java defects. We used these fixing commits along with the issues filed in the issue trackers to construct appropriate failure-inducing inputs for each failure.

Table 2.

View Table

Table 2. Evaluation Subjects Used in the RQ3 Case Study

Additionally, for each of the failures, we identified developer comments made in either the fixing commit or the issue that discussed the conditions that produced the failure. These comments reflect the developers’ understanding of the failure and the conditions under which the failure manifests. However, mapping these conditions to specific portions of the input is subjective, and the developers’ understanding of the failure may be flawed. Thus, these developer-identified failure conditions cannot be used as a ground truth for the failure-relevant portions of an input. For each of the failures, we also created a simplified, failure-inducing input using a combination of random trials of reduced inputs and Zeller and Hildebrandt [73]’s ddmin algorithm. The ddmin algorithm produces a failure-inducing input that is guaranteed to be “1-minimal,” i.e., removing any single element would cause the input to no longer induce the failure; however, the simplified input is not guaranteed to be minimal [73]. Even though a simplified, failure-inducing input can be useful in understanding a failure, it does not necessarily correspond to the failure-relevant portions of the original, failure-inducing input. For example, even a minimized, failure-inducing input can contain portions of the input that are necessary to pass a validation step, but not necessarily useful for investigating the failure. While neither developer-identified failure conditions nor simplified failure-inducing inputs can be used as a ground truth for the failure-relevant portions of an input, they both provide valuable insights into the nature of a failure. Therefore, we used these insights to guide our qualitative assessment of the failure-inducing inputs that were marked as relevant by the different propagation policies.

Our prototype implementation applies a unique taint tag to each character input presented to the applications. These taint tags are propagated in accordance with a particular propagation policy, as described in Section 6.1. In addition to this policy-based propagation, our prototype employs special propagation logic for exceptions. If a Java exception is thrown by the execution of an instruction (as detailed in the Java Virtual Machine Specification [44]), then our prototype propagates the taint tags of the operands of that instruction to the exception. Any input values associated with taint tags that propagate to the exception that produces the studied system failure are marked as failure-relevant.

For each of the bugs that we analyzed, we display the entire program input with annotations that specify which portions of the input were marked as failure-relevant by each policy. These visualizations demonstrate the impact of propagation policies on a concrete software engineering application of taint tracking. Across all five examples, data-only marks only a single character of input as failure-relevant, demonstrating the need to propagate taint tags along control flows in this application. Meanwhile, basic-control marks almost every input character as failure-relevant, underscoring the consequences of control-flow-related over-tainting. We discuss in detail the input values marked as failure-relevant by each policy for each of the studied failures below.

View Figure

Checkstyle. Checkstyle is a static analysis tool that finds and reports violations of coding standards in Java code [16]. We studied the failure reported in Checkstyle issue #8934 [15]. A Checkstyle developer described this failure by saying, “FinalLocalVariable throws a NPE on Switch expression in assignment” [15]. However, in the fixing commit for the failure, a different Checkstyle developer noted that, “assigning to [the] \(\text {switch}\) is not the problem ...wrapping [the] \(\text {switch}\) inside a function foo() makes the problem disappear” [14]. These developer comments indicate that the failure reported in Checkstyle issue #8934 occurs when the “FinalLocalVariable” rule is applied to a Java source code class containing a \(\text {switch}\) expression that is not contained within a method-level or block-level scope [14, 15].

The failure-inducing input that we used to reproduce the failure reported in Checkstyle issue #8934 was a Java source code class based on the Java source code class included in issue #8934 [15]. Figure 6 shows the input Java source code class in its entirety along with the portions of the input identified as failure-relevant by each propagation policy. Listing 11 depicts the simplified, failure-inducing input produced for the failure-inducing input in Figure 6. This simplified, failure-inducing input consists of a Java class declaration containing a field declaration that initializes the field to be the value of a \(\text {switch}\) expression. Both the developer-identified failure conditions and the simplified, failure-inducing input suggest that the \(\text {switch}\) expression that appears on lines 27 through 35 of Figure 6 is the cause of the failure.

Fig. 6.

View Figure

As shown in Figure 6, data-only did not mark any portions of the input as failure-relevant. By contrast, basic-control reported the entire input text except the final closing bracket as failure-relevant. Both SCD and Conflux report large portions of the \(\text {switch}\) expression that triggered the failure including the \(\text {switch}\) keyword. However, both of these policies also report other regions of the input that are not likely to be helpful to developers trying to debug the failure. Conflux reports fewer of these regions than SCD.

Google Closure Compiler. The Google Closure Compiler is a tool for compiling and optimizing JavaScript code [22]. The failure we selected from the Google Closure Compiler was reported in issue #652 [21]. In the fix for this failure, a Closure developer noted that Closure should “report a parse error if there is a throw followed by a semicolon or newline” [20]. This developer continued by noting that according to grammar for JavaScript, “‘throw;’ or ‘thrown expr;’ are illegal” [20]. These developer comments indicate that the failure manifests when Closure compiles code containing a malformed throw statement where the \(\text {throw}\) keyword is followed by a semicolon or newline and not an expression.

We reproduced the failure reported in issue #652 using JavaScript source code based on the JavaScript source code provided in the original issue. Figure 7(a) shows the input JavaScript source code in its entirety along with the portions of the input identified as failure-relevant by each propagation policy. Listing 6 depicts the simplified, failure-inducing input produced for the failure-inducing input in Figure 7(a). The simplified, failure-inducing input consists of just the keyword \(\text {throw}\). Both the developer-identified failure conditions and the simplified, failure-inducing input indicate that the failure described in issue #652 is triggered by the malformed throw statement on line 9 of Figure 7(a).

Fig. 7.

View Figure

As in the Checkstyle failure, data-only did not mark any portions of the input as failure-relevant. basic-control, SCD, and Conflux all marked the malformed throw statement as failure-relevant. However, basic-control also marked almost every other input character as failure-relevant. SCD and Conflux report some additional regions of the input that may obfuscate the cause of the failure. Once again, Conflux reports fewer of these regions than SCD.

Mozilla Rhino. Mozilla Rhino is an implementation of JavaScript written in Java [48]. Rhino includes a compiler for translating JavaScript source code into Java class files. Issue #539 [47] describes the failure that we examined. In the fix for this failure, a Rhino developer noted that the failure “cropped up when generators were used in a function that had a try..catch..finally block and a yield after the finally” [46]. This comment indicates that the failure manifests when Rhino tries to compile JavaScript source code that contains a generator function or legacy generator function in which a yield expression is present after a \(\text {finally}\) block.

We reproduced the failure reported in issue #539 using JavaScript source code based on a test case that was added in the commit that fixed the failure. Figure 7(c) shows the input JavaScript source code in its entirety along with the portions of the input identified as failure-relevant by each propagation policy. Listing 9 depicts the simplified, failure-inducing input produced for the failure-inducing input in Figure 7(c). The simplified, failure-inducing input contains a legacy generator function with a yield expression following a \(\text {try}\)-\(\text {finally}\) statement. Both the developer-identified failure conditions and the simplified, failure-inducing input indicate that the failure described in issue #539 is related to the \(\text {finally}\) on line 6 and the yield on line 11 of Figure 7(c).

As shown in Figure 6, data-only did not mark any portions of the input as failure-relevant, and basic-control marked almost every input character as failure-relevant. Unexpectedly, SCD and Conflux often marked only part of a keyword as failure-relevant, for example, both policies marked only the “i” in \(\text {finally}\) as failure-relevant. This was due to the structure of the code that Rhino uses for lexing, converting characters sequences into tokens, JavaScript inputs [48]. Part of this code is displayed in Listing 12. In the case of the keyword \(\text {finally}\), there is not control flow from all of the characters of the input string “finally” to the token produced from it; there is only a flow from the character “i.”

Listing 12.

View Figure

OpenRefine. OpenRefine is a tool for manipulating, managing, and visualizing data [53]. We studied the OpenRefine failure reported in issue #2584 [52]. A developer for OpenRefine described this failure as occurring “when both trim and autodetect are enabled in tabular parser” [51]. In this and other comments, the developer notes that if both the “trimStrings” and “guessCellValueTypes” configuration options are enabled, OpenRefine will crash when importing a numeric value using an importer that subclasses \(\text {TabularImportingParserBase}\) [51, 52].

Two separate inputs were needed to reproduce the failure reported in issue #2584 [52] The first input was a comma separated values (CSV) file containing input data to be imported that was provided in issue #2584 [52]. The other input was a JavaScript object notation (JSON) configuration file that was based on a test case that was added in the commit that fixed the failure. Figure 7(b) shows the CSV input in its entirety, and Figure 7(d) shows the JSON input in its entirety. Both figures show which portions of the input were identified as failure-relevant by each propagation policy. Listing 7 depicts the simplified, failure-inducing input produced for the failure-inducing, CSV input in Figure 7(b). The simplified, failure-inducing input contains two rows: an empty header row and a row containing a single numeric value. Listing 8 depicts the simplified, failure-inducing input produced for the failure-inducing, JSON input in Figure 7(d). This input is a JSON object with two properties: “guessCellValueTypes” and “trimStrings.” The value of both of these properties is the Boolean \(\text {true}\). Both the developer-identified failure conditions and the simplified, failure-inducing input suggest that the JSON properties “guessCellValueTypes” and “trimStrings” and their values are relevant to the failure. Additionally, both the developer-identified failure conditions and the simplified, failure-inducing input indicate that a numeric value in the CSV input is relevant to the failure. The fixing commit suggests that first numeric value in the CSV input that is not in a header row (the number “1” on line 2 of Figure 7(b)) triggers the failure [51].

As depicted in Figure 7(b), all of the propagation policies, even data-only, marked the numeric value that triggered the failure (the number “1” on line 2 of Figure 7(b)) as failure-relevant. This is the only input value ever marked as failure-relevant by data-only in any of the failures we examined. basic-control marked additional portions of the CSV input as failure-relevant, likely obfuscating the true cause of the failure. For the JSON input depicted in Figure 7(d), data-only did not mark any portions of the input as failure-relevant, and basic-control marked almost every input character as failure-relevant. SCD and Conflux reported the properties “guessCellValueTypes” and “trimStrings,” and their values as failure-relevant. However, they both reported additional regions of the JSON input that are unlikely to be helpful to a developer trying to debug the failure. Conflux reports fewer of these regions than SCD.

H2 Database Engine. H2 is a Relational Database Management System (RDBMS) implemented in Java that supports a subset of Structured Query Language (SQL) [32]. H2 issue #2550 [31] details the failure that we selected. In the fixing commit for this failure, an H2 developer described the failure as a “NullPointerException with MERGE containing unknown column in AND condition of WHEN” [30]. This comment indicates that the failure occurs when H2 tries to execute a \(MERGE\) statement containing a \(WHEN NOT MATCHED\) clause with an \(AND\) expression that refers to an unknown column [30, 31].

We reproduced the failure reported in issue #2550 using the SQL statements provided in issue #2550 [31]. Figure 7(e) shows the input SQL statements in their entirety along with the portions of the input identified as failure-relevant by each propagation policy. Listing 10 depicts the simplified, failure-inducing input produced for the failure-inducing input in Figure 7(e). The simplified, failure-inducing input contains a \(CREATE TABLE\) statement and a \(MERGE\) statement. Both the developer-identified failure conditions and the simplified, failure-inducing input indicate that the command \(MERGE\), the clause \(WHEN NOT MATCHED\), the keyword \(AND\), and the unknown reference, “s.b,” which appears on line 8 of Figure 7(e), are relevant to the failure.

As depicted in Figure 7(e), data-only did not mark any portions of the input as failure-relevant. Conversely, basic-control reported the entire input as failure-relevant. SCD and Conflux only marked portions of the \(MERGE\) statement as failure-relevant. SCD marked almost the entire \(MERGE\) statement as failure-relevant. Conflux only marked small portions of the \(MERGE\) statement as relevant; it is unclear whether these smaller portions better illuminate the cause of the failure.

Summary. When considering the input values marked as failure-relevant by each propagation policy, it is clear that data-only and basic-control are unlikely to be useful to a developer attempting to debug a failure. This underscores the need for alternative taint tag propagation semantics. The portions of each input marked as failure-relevant by SCD and Conflux appear to be more useful for analyzing the failures. SCD tended to mark more characters as failure-relevant than Conflux and, in some cases, marked most of the input as failure-relevant.

6.6 Threats to Validity

One threat to the validity of our experiments stems from our selection of evaluation subjects for the benchmark. Our benchmark tests a limited number of methods from only a handful of projects. As discussed in Section 6, it is challenging to determine which taint tags should propagate to which values for an arbitrary, real-world program. Thus, only methods that met certain criteria (detailed in Section 6.2) could be included in the benchmark. As a result, the benchmark is not necessarily representative of all Java programs. However, we selected these methods based on a search for popular Java libraries.

Additionally, the ground truth expected label sets we used for the benchmark may not be appropriate for every application of taint tracking. For example, in some applications it could be desirable for propagated labels to reflect looser or stronger relationships than those reflected in our ground truth. Nonetheless, we feel that our ground truth selection follows best practices of state-of-the-art taint tracking evaluations [45, 56].

Unlike the benchmark, the case study evaluation that we performed did not require a manually specified ground truth. However, the case study evaluation was limited to a single application of taint tracking and examined a limited number of subjects and failures. Therefore, it may not generalize to other applications of taint tracking or other subjects.