4.3 CARROT Adversarial Attacker
The goal of adversarial attack is to find the code snippet that solves the optimization problem in Equation (
10). We propose CARROT
A to generate valid adversarial examples by sharing the similar spirit of hill climbing, which is an iterative strategy seeking for the potential optima. Compared with other simulation approaches such as
Metropolis-Hastings (M-H) sampling, the overhead of hill climbing is often small, making it feasible in practice.
Objective transformation. Performing hill climbing to search for optimal
\(\hat{x}\) of Equation (
10) is equivalent to minimizing the predicted probability of
y by
C, as follows:
Equation (
10) finds
\(\hat{x}\) that causes
y and
\(C(\hat{x})\) to be the most different. Maximizing
J, while minimizing Prob(
y;
C) achieves the same goal. In other words, Equations (
10) and (
13) are equivalent. One benefit and purpose that we eventually choose to work on optimizing Equation (
13) is that it reduces the computational cost, since we no longer need to compute the loss
\(L(y,C(\hat{x}))\).
CARROTA algorithm. CARROT
A takes the DL model
C and code snippet as inputs and outputs an adversarial example at its best effort. In Algorithm 1 (also see Figure
1), at
ith iteration, the mutators (see Table
2) are used to randomly generate a set of equivalent code variants from the current code snippet
xi − 1 as candidates, denoted as
\(\mathcal {T}=\lbrace x_1^*,\ldots ,x_n^*\rbrace \subset T(x_{i-1})\), among which all variants are able to pass the compilation and are equivalent to
xi − 1 (Line 3). Then, CARROT
A tests all candidates against
C to determine whether an adversarial example is found (Lines 4–8). If an adversarial example is not found, then the candidate with the lowest probability on
y is selected, denoted as
\(x^*_{idx}\). When the probability decreases,
\(x^*_{idx}\) is passed to the next iteration as
xi; otherwise,
xi remains the same as
xi − 1 (Lines 9–14). The generation process continues until the allocated budget (i.e., iteration size) exhausts or an adversarial example is found.
CARROTA takes both effectiveness and efficiency into consideration during the design. Hill climbing is similar to gradient descent, which is widely adopted in DL, as it is guaranteed to find the local optima for non-convex objectives and global optima for convex objectives. The computational efforts of CARROTA mainly consist of two parts—mutation operations and DL model probing. We consider the invocation of DL models to estimate the computational cost, including forward prediction and backward gradient propagation, because other arithmetic or logic operations in Algorithm 1 are much less time-consuming than one DL model invocation. Mutators in CARROTA at most require one invocation of the model to obtain the gradient information, and in other situations where the gradient is not employed, this invocation is not even required. Therefore, The computational cost of mutators is O(1). However, at Line 9 in Algorithm 1, the probability evaluation requires to invoke the DL model for n times (candidate size), and the computational cost of this part is O(n). Therefore, the computational cost of CARROTA is O(n), and the overhead of the mutators can be neglected. CARROTA seeks every opportunity to make the objective decrease, leading to fewer invocations of C with higher efficiency. We further incorporate gradient information into the mutation operation whenever available, guiding the searching process with even higher efficiency. Although retrieving the gradients may cost much more time during the mutation operations, it can effectively reduce the searching iterations.
4.4 Mutators for Candidate Code Generation
Mutators play an important role in CARROT
A, which could potentially influence the diversity and effectiveness of the attack. We design mutators to follow rule-based transformations that satisfy the constraints in Equation (
7). We also take the mutator set extensibility into consideration, where different mutators could be easily integrated. As an early step in adversarial attack for source code, in this article, we cover basic while efficient mutators at different levels for attacking integration (please refer to “I-Mutator with gradient guidance” and “S-Mutator” in this section for detailed discussion about the efficiency). In particular, we include six common mutators at two levels, i.e., token-level (I-Mutator) and statement-level (S-Mutator) (see Table
2).
The token-level mutators are inherited from the previous work for adversarial attack against DL for source code processing [
93]. They manipulate tokens in the tokenized source code sequences, such as variable (global/local) renaming, data type renaming, function renaming (please refer to “Token” in Table
2). The statement-level mutation performs higher-level transformations, which involves dead statement insertion and deletion, including empty statement (e.g., “;” in C/C++) insertion/deletion, no-entry branch (e.g., “if(false);”) insertion/deletion and no-entry loop (e.g., “while(false);”) insertion/deletion (please refer to “Stmt” in Table
2). The statement-level operations are inspired by EMI [
50], which generates semantically equivalent code snippets for compiler testing by inserting or deleting dead code. In particular, no-entry branch and loop insertion and deletion in Table
2 are absorbed from Hermes [
73], which is an extension of EMI.
I-Mutator. I-CARROTA is CARROTA equipped with the I-Mutator, which performs multiple renaming operations. I-Mutator requires to gather all renamable candidate identifiers (e.g., variables, structures, unions, enumerates, functions) that form the set \(\mathcal {S}\). \(\mathcal {S}\) can be obtained by a fast scan during pre-processing, and it is updated during the attack iterations. The target identifiers are drawn from a set \(\mathcal {T}(x,s)={\rm ID}(\mathcal {V}_C)-\mathcal {S}\), where x is the code snippet, \(s\in \mathcal {S}\) is the identifier to be renamed, and \({\rm ID}(\mathcal {V}_C)\) are legal identifiers in the vocabulary of C. This definition ensures the validity of the renaming operation and avoids duplicated names.
There may be some concerns that identifier renaming can be too naive and trivial. Ideally, the DL models for SE should be capable to resist against semantic equivalent transformations, such as identifier renaming, for better generalization ability. However, as an early stage to study the robustness of DL for source code processing, this article takes one step further towards the ultimate goal. In addition, our experimental results reveal that the classic or even the current SOTA DL models for source code processing cannot handle the renaming perturbations very well (please refer to Section
6.2). The subject models are obtained following instructions from the original papers, and they produce comparable performance upon training, validation, and test sets for each subject task. In short, our findings reveal that the DL models for source code processing contain severe potential risks of non-robustness, even against simple perturbations such as identifier renaming. This article aims to better understand how well the current DL models for SE perform against adversary, and we also would like to learn the challenges and the opportunities along this research direction.
Simple random I-Mutator. A simple implementation of I-Mutator is to randomly draw
\(t\in \mathcal {T}(x,s)\), generating one single candidate by renaming
s in
x to
t. CARROT
A equipped with random I-Mutator, denoted as I-RW,
6 can be viewed as a degeneration of the previously proposed MHM [
93]. I-RW first generates a random renaming proposal and then accepts it if the probability on the ground-truth class decreases, otherwise, I-RW rejects it. I-RW also serves as the random baseline in this article.
I-Mutator with gradient guidance. The random sampling approach can be useful but inefficient, since the attack space is quite large and simple random walking without gradient guidance would miss the optimal candidate in many cases, as illustrated in Figure
2(a). Inspired by gradient-based attack algorithms (e.g., FGSM [
34] and BIM [
49]) that optimize Equation (
4) by jumping according to the gradient direction, we incorporate gradient information, which can be produced by most DL models for source code processing, into I-Mutator. In particular, I-Mutator generates
\(s\in \mathcal {S}\), which change towards similar directions to the gradient in the embedding space, as candidates. The similarity is measured by a cosine-similarity-like scoring function, and I-Mutator chooses the top-
n identifiers. The candidate size is a hyper-parameter of I-Mutator. I-Mutator renames code snippet
x on identifier
s based on the scoring function defined as:
where
\(t\in \mathcal {T}(x,s)\) is the possible target identifier,
e(
s) and
e(
t) are embedding vectors of
s and
t, and
\(\frac{\partial L(y,C(x))}{\partial e(s)}\) is the embedding-level gradient vector of
s. Equation (
14), which is proportional to the cosine similarity, measures the similarity between the changing direction of renaming
s to
t and the gradient direction. By selecting the top-
n identifiers that cause the similar changes as the gradient, we introduce the gradient into I-CARROT
A. This technique picks the code snippets lying within a thin cone with the gradient as its axis (Figure
2(b)), and therefore includes the desirable candidate identifiers, which leads to great increasing in
L(
y,
C(
x)). We select
\(t_1,\ldots ,t_n=\arg n\max S(t,s|x)\) and rename
s in
x to
t1, …,
tn to generate the equivalent candidates
\(x_1^*,x_2^*,\ldots ,x_n^*\) (see Line 3 in Algorithm 1).
Unlike the completely random strategy in MHM, the gradient information brings two impacts towards I-CARROT
A: ❶ The computational cost does not increase due to the gradient operation. I-Mutator is relevant to the DL model
C for sure, since it needs to invoke
C to obtain
\(\frac{\partial L(y,C(x))}{\partial e(s)}\). However, compared to the testing step in CARROT
A (Lines 4–8 in Algorithm 1 invoke
C for
n times), the overhead of this single invocation can be ignored. ❷ The efficiency of the iterative process can be improved greatly with the gradient guidance. MHM randomly samples identifiers, forming the candidate set, as shown in Figure
2(a). Such strategy does not consider the desirable changing direction, instead, it mainly picks the “bad” candidates, which cannot increase the loss. However, candidates from I-CARROT
A (Figure
2(b)) are purposeful. These candidates are in a thin cone with the gradient as its axis, and they are guided to perturb the code towards the direction of increasing the loss. As I-CARROT
A generates more effective substitution candidates than MHM, I-CARROT
A would overall show better effectiveness and efficiency.
Validity of I-Mutator. We have taken compilation and semantically equavalent issue during transformation into consideration during design. By design, I-Mutator makes no influence to the compilation of the perturbed code, because we rename only the user-defined identifiers, which have no dependencies with the external environment. Meanwhile, identifier substitutions by I-Mutator do not alter the execution as well, because the renaming towards a certain identifier results in identical executable file after compilation as the original. Therefore, \(\mathcal {X}(x^{\prime },x)=1\) is guaranteed in I-Mutator. To be more specific, I-Mutator collects all renamable identifiers, forming \(\mathcal {S}\), from the given compilable and executable code. For instance, given a single source code file, which includes the whole program, \(\mathcal {S}\) is collected by traversal searching for definition or declaration nodes in the syntax tree. In a more complex scenario, where the project consists of multiple files with dependency, I-Mutator compiles the whole project to obtain the syntax tree and still traverses the tree to collect \(\mathcal {S}\). I-Mutator does not substitute those identifiers that may have dependencies from other files that are not provided. Identifiers in \(\mathcal {S}\) obtained in such approach exists in and only within the given code itself. During perturbation, I-Mutator renames identifiers in the given code files and all downstream files to ensure validity, i.e., \(\mathcal {X}(x^{\prime },x)=1\).
S-Mutator. S-CARROTA represents CARROTA equipped with the S-Mutator. S-Mutator inserts (or deletes) three kinds of statements into (or from) the snippets, including ❶ empty statements, e.g., “;” and “printf(””);” in C/C++, ❷ no-entry branches, e.g., “if(false);”, and ❸ no-entry loops, e.g., “while(false);” and “for(;false;);”. For S-Mutator, we define an insertable statement set (denoted as \(\mathcal {S}\)) and collect all positions (denoted as \(\mathcal {P}\)) in the code snippet, where statement insertion is possible, by one pass scanning during pre-processing. Also, a set \(\mathcal {I}\) is needed to record all deletable statements.
At each iteration, S-Mutator first randomly decides whether to insert or delete a statement. For insertion, S-Mutator randomly samples \(s\in \mathcal {S}\) and \(p\in \mathcal {P}\), and inserts s into x at position p, to generate a candidate. Otherwise, S-Mutator samples \(d\in \mathcal {I}\) and deletes d in x. At the end of an iteration, S-Mutator updates \(\mathcal {P}\) and \(\mathcal {I}\) according to the decision of S-CARROTA. In addition, the probability of insertion is also adjusted at each iteration to avoid too many insertions. The probability of insertion is defined as \(r=1-\frac{n_{ins}}{n_{max}}\), where nins is the number of currently inserted statements and nmax is the max insertion threshold.
In S-Mutator, which is in fact black-box, gradient information is not incorporated due to two major challenges. ❶ Most DL models for source code cannot produce statement-level gradient. Although it can be tackled by aggregating (e.g., averaging) gradient vectors of all tokens in the perturbed statement, such technique still cannot deal with the second challenge. ❷ S-Mutator performs statement insertion and deletion, which are discrete and underivable operations. Gradient for operations such as insertion or deletion is undefined and nonexistent. Therefore, we adopt random searching in S-Mutator.
Still, S-CARROT
A meets the same convergence with or without the gradient guidance. Hill climbing in S-CARROT
A can be viewed as a special case of
Metropolis-Hastings (M-H) sampling [
22,
38,
61]. As long as the transition proposal in M-H is aperiodic and ergodic, given enough iterations, the algorithm converges to the same stationary distribution. In S-CARROT
A, the transition proposal includes insertion and deletion of three types of statements that are aperiodic and ergodic, hence, S-CARROT
A converges to the same stationary distribution.
Simple random S-Mutator. Similar to I-RW, we also implement a simple random baseline for S-CARROTA, denoted as S-RW. S-RW generates one single candidate by inserting a random statement \(s\in \mathcal {S}\) into a random position \(p\in \mathcal {P}\) or deleting a random statement \(d\in \mathcal {I}\) from the snippet. To be clear, the difference between S-Mutator and S-RW is the candidate size. S-Mutator generates multiple (n) candidates of insertion or deletion, while S-RW produces only one candidate. Therefore, S-RW is based on randomness, which creates a scenario similar to random walking, as the name suggests.
Validity of S-Mutator. By design, S-Mutator does not alter the compilation and execution validity of the code after perturbation as well, because the inserted or deleted statements are dead code. Such perturbations do not change the execution trace in data flow or control flow. The insertable position \(\mathcal {P}\) is collected from the complete syntax tree, where the statements can be easily split. The inserted statements \(\mathcal {S}\) are all in fact dead code, which does not change the execution results. However, from the perspective of compilation process, the inserted dead code may be eliminated due to some optimizations. For instance, GCC eliminates the dead code when the “-fdce” optimization option is on. Some pre-processing with dead code elimination is very possible to invalidate S-CARROTA. Nevertheless, as an early step, we notice that most DL models for SE do not contain such pre-processing, making S-CARROTA still a potential threat to them.