Guided Feature Identification and Removal for Resource-constrained Firmware

2.1 Communication Protocols

Smart, embedded appliances such as cameras, locks, thermostats, and a variety of sensors, are oftentimes equipped with the ability to connect to other devices and the Internet. For example, temperature sensors may be connected in a wireless network to collect temperature information in a large building automatically. Designers use a variety of low-overhead, application-level network protocols for communication between embedded devices; examples include MQTT, AMQP, and DDS. Many of these protocols have robust and well-maintained open-source implementations, for example, Mosquitto [1] for MQTT and OpenDDS [9] for DDS. We also include Quiche, a userspace implementation of the QUIC transport protocol and HTTP/3, due to the current industry interest in applying QUIC to the IoT space [25].

Features. Table 1, lines 2–9, summarizes a set of popular protocol implementations in the IoT space. For each, we manually inspected source code and determined the number of optional features (column 3). Table 1 corroborates our insight that IoT-related software comes equipped with a significant amount of non-core functionality. For example, Mosquitto implements support for Websockets, which simplifies communication between a web browser and a server. If an IoT installation does not include a browser UI , then the feature—whose implementation consists of 800+ LOCs—can be deactivated.

2.2 Multimedia Encoders/Decoders

Many IoT sensors incorporate media capturing and streaming capabilities. Audio/video codecs are extremely complex pieces of software and reuse of standardized implementations is common; we therefore consider leading open-source implementations of multimedia codecs and streaming logic as a relevant case study. We include three popular and actively maintained codebases: FFmpeg [14], an extensive, multiplatform collection of audio/video codecs and related tools; and two implementations of the industry-standard AV1 video codec: rav1e [15] (written in Rust), and libaom [11] (C/C++).

Features. Based on lines 10–12 of Table 1, the three programs in this category include, in aggregate, 66 discrete build-exposed features. This is not surprising, as media tools tend to implement support for optional optimizations, algorithms, and subcomponents (e.g., video post-processing routines in FFmpeg). This result underlines the importance of customizing these codebases for embedded applications.

2.3 Motivation of Our Work

The abundance of optional features, and the difficulty of fully removing them, are undesirable. First, when building firmware for highly resource-constrained IoT devices, it is essential to avoid unnecessary use of storage space for useless functionalities. Feature-level removal goes beyond dead or unreachable code and is focused on reachable code that is not used or needed. It is worth pointing out that leftover code may also offer attackers additional avenues (larger attack surface) to mount attacks, such as ROP and code reuse [42].

Second, optional features present opportunities for vulnerabilities to lurk in a codebase. To validate this, we performed a survey of CVE reports [13] related to a subset of programs in Table 1 (Mosquitto, azure-uamqp-c, OpenDDS, and FFmpeg). We sampled approximately 100 results and eliminated those that did not apply to feature-related functionality based on description. For the remaining ones, we identified the lines of code where the vulnerability lies, and confirmed that these lines are removed by PRAT when disabling the relevant feature. This analysis resulted in eight CVEs: CVE-2018-12546 in Mosquitto, CVE-2019-1535 in OpenDDS, and CVEs 2018-13303, 2017-16840, 2019-9721, 2019-9718, 2019-11339, 2018-1999015 in FFmpeg. Five of those are rated “medium,” two “high,” and one “critical,” for an average CVSS score of 7.3. While these features can be manually deactivated, PRAT simplifies the process of discovering and disabling them, encouraging and streamlining their removal.

Code removal (or debloating) at the feature-level directly answers the need for methods to keep IoT firmware small in code size and minimize attack surface. It also facilitates static and dynamic analyses. Refining the code by removing all unnecessary features makes analysis scope smaller, complexity lower, and precision higher. Compared with feature-unaware debloating, feature-level debloating is more user-friendly, explainable, and tractable. It is easier for users to understand what is being removed and why; it guides and takes inputs from the analyst through the debloating process.

We believe that an easy-to-use IoT feature-based software debloating tool is an important—yet missing—building block towards optimized, reliable IoT firmware. Our core goals are therefore to: (i) identify available features and their ramifications within a codebase, (ii) ensure that deactivation of a feature at build time results in removal of all code solely associated with that feature, and (iii) ensure the process results in functionally correct binaries.

3.1 Mosquitto Overview

Mosquitto is a popular implementation of the MQTT publisher/subscriber communication protocol. It is a mature codebase that has incorporated several features over time, summarized in Figure 1(b). In general, not all such features are necessary for a given deployment. In our scenario, we assume an analyst uses our tool to prepare a Mosquitto build for a Linux-based sensor network. Devices have limited storage that must suffice for the OS and additional custom code. The network exists in a controlled power generation building, and it is not connected to the Internet. The function is non-critical (environmental monitoring rather than process control) but access to the network nodes is difficult (most of the monitoring network exists within a reactor containment building) so software reliability is important. The process of feature selection is carried in five steps, described in the following and depicted in Figure 1(a).

Fig. 1.

View Figure

3.2 Step 1: Feature Identification

First, PRAT identifies all features available within the source code. Feature information is not readily available or always documented, and therefore PRAT infers features by analyzing and filtering the build system options. We implement parsers to analyze the build system of the project and return a list of available features.

We term the remaining set of build options as the candidate feature set—in other words, those are options that are likely to control activation/deactivation of optional protocol features. As part of the feature identification step, we also have a process that filters out certain features from being displayed to the analyst. The default behavior—and that used in our running example—is to display all features to the analyst so they can determine which are important or not. However, if we want to simplify the process for the analyst further, then we filter out features that would otherwise inundate the analyst with options. The feature identification step is described in greater detail in Section 4. In our example, the tool would identify the set of features in Figure 1(b).

3.3 Step 2: Feature-to-code Mapping

In the second step, PRAT identifies lines of code exclusively associated with each feature (i.e., lines of code whose purpose is to implement a given feature but are not associated with/necessary for any other functionality). This is necessary, since, as outlined in Section 1, simply deactivating a build option associated with a feature does not guarantee full removal of all associated code. The process builds a set of binaries, each with a different build option—among those identified in step 1—turned off. All binaries are instrumented to collect code coverage data and executed on a set \(T\) of symbolically generated tests. Differential analysis of code coverage allows PRAT to determine which code is associated exclusively with each feature. Because all the binaries are instrumented to track coverage information when we run a test against the binary, all code that gets executed is marked in corresponding coverage files. Seeing code being executed with one feature activated but not another means that the code block pertains to the feature being tested. Results of these tests are gathered and used to determine the lines of code associated with each feature (the process is described in detail in Section 5. The feature-to-code mapping is then converted to a graphical representation we call a feature graph. For a given source tree, a feature graph lists high-level features and associates each with source files and individual LOCs. Figure 2 shows a simplified view of the feature graph for Mosquitto’s Websockets feature.

Fig. 2.

View Figure

3.4 Step 3: Feature Selection

Feature graphs—described in detail in Section 6—constitute an easy-to-use decision-support tool for feature selection. Using a feature graph, the analyst: (i) identifies the set of available optional features; (ii) assesses the amount of code associated with each; and (iii) delves into the specifics of each feature’s implementation. As the output of this step, the system receives back a set \(R\) of features that the analyst wishes to be fully removed from the code. In our example, the analyst identifies Websockets and possibly TLS, TLS_PSK as superfluous (browser connectivity is not required, and the analyst decides that, given the controlled setting, encryption may be unnecessary and even hamper debugging).

3.5 Steps 4–5: Feature Removal and Testing

PRAT removes from the source tree all lines of code associated with the features in \(R\). The modified source is then compiled into the final binary. The three identified features in our example still have code artifacts remaining in the source code when deactivated via build options. By using our tool, on average, we remove an extra 46,396 LOCs, which amounts to an additional 2.9 MB reduction in the size of the compiled binary. The program binary is once again run through the set \(T\) of test cases as well as rounds of fuzzing to ensure its functional correctness. We note that the extensive use of dynamic analysis, both in feature identification and testing, comes with some limitations. We further discuss these limitations, and motivate why we deem them acceptable, in Section 10.

5.1 Per-feature Builds

Given the set of features \(F\), PRAT builds \(n+1\) separate versions of the target program binary. First, our system builds a version of the binary with all features enabled (line 4 of Algorithm 1). Next, it builds additional \(n\) versions where the \(i\)th build has all features enabled except feature \(f_i\) (lines 6–8 in Algorithm 1). This is typically the most time-consuming step in the feature removal process, as it requires re-compiling a potentially large codebase multiple times. Overall, the entire PRAT analysis pipeline takes on average 12.8 minutes to complete on the codebases evaluated in Section 8. We consider this acceptable, as manually identifying all features in an unknown codebase, and tracking all the locations in the code where each feature is implemented, is likely to take significantly longer. Additionally, tools like ccache [3] could be applied to further speed up the process via memoization. At this stage, we also discard build options that result in a failed compilation. This step outputs a set of \(n+1\) binaries \(B_{all}, B_1,\ldots , B_n\), where \(B_{all}\) has all features enabled, and each binary \(B_i\) has all features enabled except \(f_i\).

5.2 Code Coverage Analysis

The next step (lines 5 and 9–11 in Algorithm 1) leverages code coverage analysis to identify lines of code associated with each feature via an exclusion process. Code coverage instruments compiled binaries to determine which lines in the source code correspond to binary instructions that were executed in a given run. Consider the set \(L_i\) of lines of code returned by the tool when running \(B_i\), i.e., the binary with feature \(f_i\) disabled. We posit that lines in \(D_i = L_{all} \setminus L_i\) (where \(L_{all}\) is the set of lines of code returned with all features enabled) represent the logical implementation of feature \(f_i\).

5.3 Symbolic Test Case Generation

Code coverage, to be successful, requires the availability of tests that make use of as many features as possible. We rely on a set of test cases \(T\) to provide the highest-fidelity results.

First, this set includes the set of unit tests \(U\) shipped with the code, if any. For example, both azure-uamqp-c and OpenDDS ship with a comprehensive suite of unit tests, which we used without modifications. However, other software tools (e.g., Mosquitto) may not ship with sufficiently comprehensive tests.

To ensure that the analysis can proceed regardless of the availability of \(U\), we leverage symbolic execution to generate an additional set \(S\) of high-code-coverage tests.

It should be noted that, in general, symbolic execution is not the only approach that can be used to carry such task, however, we found it to be the most effective one in our domain. Beyond symbolic execution, we investigated using state-of-the-art testing tools, namely, the concolic executors QSYM [50] and Driller [46]. However, both are based on AFL [17], which is not designed for fuzzing network protocols (network protocol implementations constitute some of the most relevant targets for our analysis in the IoT space). Integrating AFL file-based interface with network code requires either modifying the target program in non-trivial ways or engineering complex test harnesses [27]; while network-specific forks of AFL have been developed [16], they are not integrated with concolic engines. Furthermore, we want the executor to generate a single test case for each new path explored in the codebase, which we can then simply replay against any variants of the same implementation.

Based on the considerations above, we leverage the KLEE symbolic execution engine [21] to generate high-quality tests. KLEE is a symbolic virtual machine built on top of the LLVM compiler infrastructure, which aptly suits our needs. In particular, we use its capability to generate a test case for every new path that it explores. By running KLEE against a target, we can generate a suite of high-coverage test cases, \(S\), even in the absence of bundled unit tests. Overall, the final set \(T\) of tests consist of \(T = U \cup S\).

The test generation process is summarized in lines 2–3 of Algorithm 1. In particular, the SymbolicTestGeneration procedure in line 2 consists of the following steps:

During code coverage analysis, KLEE-generated tests can be replayed against the actual protocol binary by using the klee-replay utility. This approach requires no modification of source code; the analyst can set the number (and corresponding size) of the symbolic variables and files that KLEE uses. To balance our tradeoff between runtime for test generation and code coverage achieved through our tests, we set parameters for max-fail and max-time. We run KLEE against our target protocols for 60 minutes and, on average, generate 4,369 tests.

5.4 KLEE Configuration and Alternatives

Running KLEE requires setting arguments for the symbolic environment. However, PRAT strives to automate the entirety of the testing process so it is opaque to the analyst. Through testing various permutations of candidate parameters to KLEE, we settled on a default parametrization (given in Table 2) that provided generally comprehensive results across different projects. If the operator desires to maximize code removal and has sufficient domain knowledge, then it is possible to refine the set of symbolic arguments and files passed to the LLVM executable.

Given our domain knowledge of the protocols under test, we can run the LLVM executables with a targeted set of symbolic arguments and files. For example, if we want to exercise the functionality of starting a broker using a configuration file instead of command-line arguments, then we can pass –sym-files <NUM> <N> where NUM is the number of files of size N. Using knowledge of the system under test, we can target KLEE much more precisely; however, since this knowledge cannot be assumed, we run KLEE using a standard set of parameters that tend to result in generally comprehensive coverage. In all cases, our targets were tested using the same set of parameters, shown in Table 2.

We also explored the option of augmenting the base unit tests in \(U\) by making some of their inputs symbolic and “amplifying” their coverage. This requires much more manual effort, as the analyst needs to understand how the code base and unit tests work to properly flag variables in the source as symbolic. Results in Section 8.3 suggest that this additional effort does not result in improved code coverage. Therefore, we do not consider this approach further.

Instead, to avoid requiring the user to manually modify the source code for symbolic execution purposes, we dismissed the idea of making individual variables symbolic for testing and instead left the target codebase untouched. This means that we run KLEE against our target binary with the aforementioned set of parameters without needing to manually insert any klee_make_symbolic calls. Utilizing KLEE in this way circumvents the need for manual instrumentation of the target codebase and simplifies the overall workflow.

5.5 Correctness

By design, Algorithm 1 removes code conservatively, removing only those LOCs that are executed when the relevant feature is active, but not when the same feature is disabled. The algorithm does not remove LOCs that are never executed regardless of whether the feature is active or not. Thus, in the event that some feature-specific LOCs are not covered by the tests, they will not be removed through this process. This design purposely strives for soundness over completeness. It contains the effects of incomplete coverage, which thus results in the algorithm failing to remove all unused LOCs, but not in the removal of non-feature-related LOCs.

Overall, feature-to-code mapping outputs the sets \(D_1,\ldots ,D_n\) of lines of code associated with each feature \(i=1,\ldots ,n\); and a feature graph (described in the next section).

8.1 Implementation and Dataset

Tooling. To evaluate our approach, we created a prototype implementing the PRAT pipeline described in Sections 3–7. Internally, the prototype leverages gcov (for C/C++) and kcov (for Rust) to generate code coverage reports. The prototype has been publicly released (refer to Section 14). To validate the correctness of feature removal on the Mosquitto broker codebase (Section 8.7), we built a fuzzing MQTT client based on the Boofuzz fuzzer [6]. We provide details on the fuzzer in Section 8.7.

Program Dataset. Table 3 describes the program codebases used for the experiments (large codebase sizes are due to the inclusion of tests and other resources). We selected these seven codebases after an extensive review of open-source middleware protocols and multimedia tools. Ultimately, we converged on codebases that represent stable, popular implementations of mature functionality relevant to the IoT space.

8.2 Feature Analysis

Methodology. In this section, we evaluate the effectiveness of PRAT in identifying optional features embedded in a codebase. We identify false positives and negatives by manually analyzing each codebase and collecting available features and comparing the result with the list returned by PRAT. Manual identification of features was carried by two of the authors using the process detailed below. On average, each person spent 30 minutes per codebase. The union of these two sets of features was used as a reference for evaluating false positives and negatives. To collect features, we first obtained the list of build system flags, retaining those matching the definition of a feature described in Section 4. Further, to ensure that no hidden features (i.e., features not accessible to the build system) exist, we parsed and collected #define directives within the source code of all programs, manually comparing such variables with the identified features. This analysis did not reveal any features existing in the code but not accessible to the build system. Overall, our target codebases include 115 optional features. We were unable to support a total of 7 features across the targets; 2 of them were Windows-specific and thus incompatible with our framework, while 5 of them triggered various compilation bugs, which we determined to be unrelated to our infrastructure. The most interesting case was a bug we uncovered in azure-uamqp-c, where if no SSL implementation is specified to use, the code defaults to using OpenSSL. However, in the checks for the defined SSL implementation, in the unspecified, default case, the appropriate headers for OpenSSL are not imported. This results in compile-time errors, which were found through our feature extraction process. Following this discovery, we reported the bug to developers.

Results. In principle, feature identification could generate false negatives in two cases: (i) features configurable via the build system are not recognized; and (ii) features exist that are not exported to the build system. Mosquitto’s build system defines 20 build-time options, azure-uamqp-c 32, OpenDDS 10, Quiche 5, FFmpeg 34, rav1e 16, and libaom 26. Of those, respectively, 19, 16, 9, 4, 33, 14, 20 are features according to our definition. Our feature analysis (refer to Section 4) identifies all 115 true features.

False positives can be generated if feature analysis confuses non-feature build options for features. In our analysis, this only happens in 1 case. The lone misidentified feature is the option to build with or without a shared library. However, this build option does not affect the program binary and has no source code associated with it; therefore, including it in the analysis does not affect the final result.

8.3 Symbolic Test Generation

Before evaluating feature removal, we briefly discuss PRAT’s approach to generate test cases using symbolic execution. As in any dynamic analysis-based system, we rely on exercising as much code as possible. Comprehensive tests allow our approach to identify feature-related code accurately.

In our case, symbolic execution can be implemented in one of the two variants discussed in Section 5.3. First, KLEE can be directly run against our program binaries. Second, existing unit tests (if any) can be manually modified to turn some of the concrete inputs into symbolic ones. We evaluated both approaches. Running KLEE directly against the binary results, on average, in 80.2% LOC coverage, while symbolizing existing tests results in 39.6% LOC coverage. Direct binary execution via KLEE achieves superior coverage; therefore, we do not further consider the symbolization of existing unit tests.

8.4 Feature Removal: PRAT vs. Manual Removal

Methodology. In this experiment, we compare the effectiveness of PRAT in removing unwanted feature code to manual removal. For each codebase in our set, we do the following: First, for reference, we compile each codebase to a binary with all features activated. Then, for manual feature deactivation, we conservatively assume that the user can manually identify and turn off all optional features (except those resulting in compilation bugs, as discussed in Section 8.2). After doing so, we compute both the #LOCs that actually get compiled and the size of the compiled binary. Finally, we run PRAT on the codebase and configure it to remove all identified optional features, measuring both the resulting #LOCs and compiled binary size.

Results. Table 4 describes, for each program in our evaluation set, source code size and compiled binary size for a version of the program with all features enabled (columns 2–3). It then compares code reduction obtained by manual feature deactivation (columns 4–5) and our approach (columns 6–7). Figure 3 provides a more in-depth look into the impact of our approach on the removal of individual features (for clarity, we randomly select 25 features across codebases). Note the log-scale y-axis; missing bars represent #LOC values of 0. In most cases, our approach results in a significant reduction in the size of the program source compared to manual feature deactivation. Overall, PRAT resulted in up to 29% more LOCs being removed compared to manual deactivation of feature-related options.

Fig. 3.

View Figure

8.5 Characteristics of Code Removed by PRAT

The code that is additionally removed by PRAT tends to look like what might typically be contained within a preprocessor conditional group. In other words, it is code whose execution is predicated upon the availability of a feature. Determining why such code was not marked by conditional preprocessor directives (e.g., #ifdefs) is beyond the scope of our work. However, we suspect that this may be related to the presence of numerous developers on the same project, which creates ambiguity w.r.t. the purpose of specific blocks code. In other words, developers may not single out those blocks as belonging to a feature out of concern that core functionality may depend on it. Incidentally, this observation highlights the potential applications of PRAT as a code understanding tool, as PRAT can quickly infer and display the ramifications of a given feature with the codebase. An example of code removed by PRAT is shown in Figure 4.

Fig. 4.

View Figure

8.6 Feature Removal: PRAT vs. Piece-wise

Methodology. In this section, we compare PRAT to a state-of-art debloating algorithm, Piece-wise, by Quach et al. [38]. Differently from PRAT, Piece-wise is a binary debloating technique and focuses on unguided unused code removal. It works by generating a fine-grained program dependency graph and only loading code that is truly needed at runtime. To evaluate Piece-wise, we manually de-activate all optional feature, compile the code, and run Piece-wise on the resulting binary for additional code reduction. We execute PRAT using the same procedure as in Section 8.4.

It should be noted that the above is an imperfect comparison. Much like in the manual removal case, when using Piece-wise, the burden of manually discovering and disabling optional features is placed on the user. Conversely, PRAT performs this function with high accuracy and automatically, as elucidated in Section 8.2. It is also important to point out that PRAT can complement a binary debloating tool such as Piece-wise, rather than replacing it. Feature removal at the source code and binary level work at different levels of abstraction. Therefore, each approach is able to identify “bloat” that is not visible to the other one. In our last experiment, we examine the advantages of the combined approach by performing the removal of all features via PRAT and further debloating the resulting binary with Piece-wise.

Results. Table 5 compares the size of binaries generated by PRAT (column 3) with those debloated using Piece-wise (column 4). We omit rav1e and Quiche, as they are written in Rust, and Piece-wise does not currently support this language. Binaries generated by PRAT are up to 17% smaller than those generated by Piece-wise, which validates the principle of feature-aware code removal as an effective debloating technique. Column 5 in Table 5 summarizes final binary size for each program in our dataset, when using both PRAT and Piece-wise. Results show that using both tools in combination results in a modest improvement over PRAT alone, suggesting that it may be useful to combine source code- and binary-based debloating.

8.7 Correctness

Methodology. In this section, we evaluate the correctness of PRAT’s output using Mosquitto as a case study.

First, when mapping LOCs to a certain feature, PRAT must avoid false positives and minimize false negatives. To verify this, we parsed the output of code coverage analysis to generate code comparison reports that display, side-by-side, the original code and the code post-debloating, highlighting feature-relevant code. We then manually inspected these reports to identify inconsistencies.

Second, even when LOCs are correctly mapped to features and removed, it is important to ensure that the final compiled binary still works correctly. We evaluated the correctness of the codebase using a testing-based approach. First, we run available tests against the debloated codebase, which did not evidence any newly introduced bug. Second, we carry further analysis on Mosquitto by complementing existing tests with fuzzing. In particular, we built a custom MQTT fuzzing engines based on the popular Boofuzz fuzzer [6]. Fuzzing as a testing strategy was not considered or employed during the design and implementation of PRAT; therefore, we consider it a useful “sanity check” with potential to uncover issues not otherwise identified. To keep execution times practical, we did not test all possible combinations of features; rather, we generated 8 variants of Mosquitto. We started with a binary that includes all features, which represent variant 0. This also gives us a baseline for #crashes present in the source prior to feature removal. We generated variants 1 to 7 in this way: variant \(i\) is obtained from variant \(i-1\) by selecting and deactivating a feature that was active in variant \(i\).

Finally, in principle, it is possible for features to be dependent on each other in such a way that removing a feature may break another one. Throughout the analyses of all the target programs, there have not been instances of multiple features utilizing or overlapping the same code blocks. They are outside of the core components of the project and are fairly compartmentalized in that fully removing a given feature will not break the build. Should this situation arise, we expect that removing a feature without removing dependent features would result in breaking the build, which would still prevent an incorrect implementation from being generated.

Results. Review of code comparison reports did not provide evidence of any incorrectly identified or missing lines of code; we, therefore, conclude that our approach successfully identified all code related to the target features in Mosquitto.

Table 6 shows code coverage results obtained via fuzzing, including the time for which each fuzzing session was run. Diminishing execution times are due to the fact that removing a feature entails that related code paths no longer exist, resulting in a shorter fuzzing session. For all but one variant, fuzzing covered \(\gt\)60% of LOCs and \(\gt\)80% of functions, without identifying any bug that was not originally present prior to feature removal. While dynamic testing does not guarantee functional correctness or the complete absence of bugs, supplementing unit tests with fuzzing provides extra assurance that our feature removal did not inadvertently break another component. These results give us confidence that our approach results in functionally correct source code.

9.1 Overall Study Design

The primary goal of the study is to understand whether software developers are likely to find PRAT useful and easy to use. As such, each participant in the study was asked to complete two tasks: (1) identifying code pertaining to a given codebase feature via manual analysis; and (2) completing the same task using PRAT. Tasks were conducted using a think-aloud protocol. Through observing the participant during the task, we could follow their intuition for defining what a feature is and determining where a given feature is implemented in a code base. Further, after completing the tasks, each participant was involved in a semi-structured interview in which we asked the following four questions:

• Was there a discrepancy between your definition of a feature and PRAT’s? Defining what a feature is can vary from person-to-person. It is important that PRAT’s definition of a feature is commonly held.

• How did PRAT impact your workflow for feature identification and removal? When evaluating the usability of PRAT, we need to guarantee that the benefits of using PRAT are worth learning a new toolchain.

• Is PRAT’s notion of a feature graph/web page output intuitive? The artifacts that PRAT outputs are important for guiding a user through feature removal. Thus, the reports need to be intuitive to users.

• What are some pros and cons of using PRAT over manual identification and removal? An open question for potential improvements to PRAT is useful to see if there are common improvements users would like to see.

During the experiments, we also measured human performance as number of lines of code correctly identified (discussed below).

9.2 Participant Recruiting

Initial participants were recruited via email and word-of-mouth advertisement among the CS graduate student population at Northeastern University and Worcester Polytechnic Institute; the rest were recruited through snowball sampling. We consider this user population (predominantly young graduate students) appropriate to test, given our goals; as it ensures that participants are generally familiar with software development, but unlikely to have prior experience with the codebases under examination (which could skew results). Prior to the actual study, we conducted pilot testing with two subjects. For the actual experiment, we recruited 10 participants, consisting of 3 women and 7 men aged 24–35. One participant worked in industry, while the remaining were graduate students. According to best practices, this sample size is appropriate for a semi-structured interview design like ours [33].

The recruiting materials indicated that the study would take 30–60 minutes and focus on the usability of a new tool used for program debloating. Participants were asked for consent to being recorded and compensated with a $15 gift card.

9.3 Study Protocol

At the beginning of each session, we explained the task and the participant’s role at a high level. We asked that the participants talk aloud to explain their thought process for each of the subsequent tasks. We presented the study to the participants as a task, asking them to define what would constitute a feature agnostic of code base, and once they defined a feature, how would they go about finding all the implementation locations within a given code base.

We then gave participants the Mosquitto [1] code base for experimenting with. They first started by proposing a definition of a feature as it relates to Mosquitto. Next, we asked them to explain how they would find all the implementation locations for one of the features they selected. Finally, we asked them to go about manually removing a given feature using the methodology they outlined.

After we gave participants the task of manually removing a feature from Mosquitto, we introduced them to PRAT. Once we explained PRAT to the participants, we asked them to use the tool to remove the same feature they had previously removed manually. Next, we asked the participants to examine the feature graph that PRAT generated and explain whether the artifacts were intuitive or otherwise useful to the process. We visualized feature graphs as hierarchical HTML documents. The reports listed each files containing relevant lines; clicking on a filename would display a colorized representation of the lines located by PRAT.

While our experiments are within-subjects, we did not compensate for learning effects, as we postulate that such effects do not arise. In our design, we asked each participant to carry manual analysis first, and then use PRAT. Because the set of lines identified by PRAT is independent of the user understanding of a given feature, the experience gained during manual analysis is irrelevant to the output and performance of PRAT. Conversely, a different design involving using PRAT before manual analysis would risk incurring in such effects.

9.4 Data Collection

Due to the impact of the COVID pandemic, seven sessions were conducted in person, and three were conducted remotely. The procedure was identical in both cases: During interviews, notes of user responses to our questions were taken; the subsequent analysis and results were generated based on these notes. We also recorded task completion time and number of lines correctly identified for each user and task.

9.5 Interview Results

We now present the results from our study’s four questions.

Feature definition. Of the 10 participants, 3 defined the notion of a feature differently than the definition used by PRAT. One participant defined a feature as any component used by the codebase. This did not imply that the user had control over its configuration. Another participant said a feature was any formatting properties of a codebase such as ordering of fields in a packet. A third participant stated a feature to be the overview of an application. This would be the application’s UI, name, or contents returned by running man against the target. The remaining seven participants defined a feature in the same way as PRAT.

Impact on workflow. When asked how using PRAT would impact a participant’s workflow, six participants responded by saying it speeds up the process of feature removal significantly. Three other users did not explicitly comment on workflow speedup, and one qualified that PRAT could speed up the workflow on large codebases, but smaller codebases could be analyzed more quickly by hand. Two participants stated that PRAT is helpful for removal of useless code beyond what compiler optimizations can typically capture. According to three participants, PRAT may produce output that could be difficult for non-technical users to understand, and in instances of small codebases it may be easier to manually grep through the project to find code to remove. Another two participants stated that PRAT makes code understanding much simpler, as the tool shows what the code to remove looks like, as opposed to just showing how many LOCs to remove. Seeing the context of the code block is a helpful aid for feature removal.

Feature graph and reports. All 10 of our participants responded that they thought the HTML-based reports PRAT generates are useful for guided feature removal. Of those, 3 participants stated that the feature graph can be unnecessary, especially if the user is unfamiliar with code. Another 4 of the participants said that the HTML reports give a simple and effective overview of the targeted feature in the context of its source files. One participant preferred PRAT’s stdout output, but followed by saying the HTML report is good for someone less familiar with the code, as it is easier to understand. Three participants also suggested that the HTML reports be made to look a little more like code commit diffs on GitHub, since our reports are similar to that already.

Pros and cons. Three participants responded that the automation is one of the biggest advantages of PRAT, with one participant specifically mentioning that an automated and complete workflow provided by PRAT makes the results more reliable than trusting a user’s manual work. All of participants said that PRAT is fast and can save users a lot of time. But two participants also mentioned that large codebases may increase PRAT’s running time because of multiple compilations PRAT needs when it analyzes programs to generate coverage diffs. Two of the participants stated that PRAT is easy to use, and another two participants also suggested that it will be better if PRAT can provide a UI window for users to not only view results, but to interact with it during the removal process. Although one participant responded that they think PRAT’s results are reliable, three participants said that PRAT still has the risk of introducing new bugs into programs, like breaking other features by removing incorrect code. Furthermore, the three users that provided a definition of feature different from PRAT’s consistently stated that they would consider PRAT useful as a code understanding tool, but not as a debloating tool. One further qualified that they would prefer performing the actual debloating by hand, relying on their own skills.

9.6 Performance Results

We measured time for manual and PRAT-based feature identification; to avoid ambiguities, all times were rounded to the minute. However, we did not use these measures to quantify performance for two reasons: (i) Due to the think-aloud protocol, it is likely that actual times would be lower in a non-lab setting; and (ii) performance of PRAT-based identification is dominated by compilation time and thus codebase-dependent. We report the measures for completeness: The mean manual tasks completion is 12 minutes (standard deviation is 4 minutes); all PRAT-based tasks completed in 2 minutes.

A more interesting question concerns accuracy, in terms of number of lines of code identified. On average, participants identified 2% of relevant lines of code (standard deviation 2%). All PRAT-based tasks identified all relevant lines of code, as verified in our correctness evaluation (refer to Section 8.7).

9.7 Take-aways

Review of the semi-structured interviews evidences that PRAT’s definition of feature aligns with most user’s intuitive understanding. They also suggest that PRAT can simplify code removal and understanding and save developers’ time, and feature graphs are a useful output format for the relevant information. However, a small number of participants also expressed concerns about clarity of the output and the possibility of introducing bugs. This suggests that the most useful direction for future work may be providing further context to increase user confidence in the correctness of PRAT’s determinations.

In terms of performance, PRAT-aided removal appears to significantly improve human performance over manual analysis alone. One may argue that a user willing to invest time and practice into code understanding may eventually achieve the same result as PRAT; however, one of the benefits of PRAT is precisely to avoid the need for investing time in such activities.

Overall, we found the results of the study encouraging, as they show that PRAT has the potential to significantly simplify feature identification and removal.