Observation-based approximate dependency modeling and its use for program slicing

Highlights

• Approximation of program dependence modeling using observations of behavior.

• Evaluation of the approximated dependence model via program slices.

• Comparisons to slices generated by the well known static slicer, CodeSurfer.

• Quantitative and qualitative analysis of generated slices.

Abstract

While dependency analysis is foundational to much program analysis, many techniques have limited scalability and handle only monolingual systems. We present a novel dependency analysis technique that aims to approximate program dependency from a relatively small number of perturbed executions. Our technique, MOAD (Modeling Observation-based Approximate Dependency), reformulates program dependency as the likelihood that one program element is dependent on another (instead of a Boolean relationship). MOAD generates program variants by deleting parts of the source code and executing them while observing the impact. MOAD thus infers a model of program dependency that captures the relationship between the modification and observation points. We evaluate MOAD using program slices obtained from the resulting probabilistic dependency models. Compared to the existing observation-based backward slicing technique, ORBS, MOAD requires only 18.6% of the observations, while the resulting slices are only 12% larger on average. Furthermore, we introduce the notion of the observation-based forward slices. Unlike ORBS, which inherently computes backward slices, MOAD’s model’s dependences can be traversed in either direction allowing us to easily compute forward slices. In comparison to the static forward slice, MOAD only misses deleting 0–6 lines (median 0), while excessively deleting 0–37 lines (median 8) from the slice.

Introduction

Understanding dependency between program elements is a fundamental task in software engineering (Livadas and Roy, 1992, Baah et al., 2010). It provides a basis for many software engineering tasks including program comprehension (Zhifeng Yu and Rajlich, 2001), software testing (Binkley, 1997), maintenance (Gallagher, 1989, Hajnal and Forgács, 2012), refactoring (Ettinger and Verbaere, 2004), security (Karim et al., 2019), and debugging (Jiang et al., 2017). The traditional static approach based on dependence graphs (Horwitz et al., 1990) has been widely adopted but suffers from issues such as its inability to easily handle multi-lingual systems (combining analyses for multiple languages can be quite complex) and limited scalability (partial analysis of a large system is not viable using static approaches that require whole-program analyses).

Observation Based Slicing (ORBS) (Binkley et al., 2014, Binkley et al., 2015, Gold et al., 2017, Binkley et al., 2019) was designed to overcome these issues. ORBS applies speculative deletions iteratively to the program under analysis, and observes whether the latest applied deletion is viable (i.e., the code compiles after deletion) and is unrelated to the slicing criteria (i.e., the variable of interest shows the same behavior after deletion with respect to a test suite). When deletions are made at the line-of-text level, ORBS is entirely language agnostic (Binkley et al., 2014, Gold et al., 2017, Binkley et al., 2019, Lee et al., 2020) and can analyze files for which the grammar is unavailable/unknown, and analyze languages with unconventional semantics such as Picture Description Languages (PDLs) (Yoo et al., 2017). Despite its benefits along with a lightweight implementation it needs, ORBS has one clear drawback: the cost of analysis. Being a purely dynamic approach, it iteratively attempts to validate its speculative deletion of every program element via compilations and test executions. As such, it can incur significant cost.

This paper investigates the feasibility of approximate dependency analysis at a greatly reduced cost. A precise dependency analysis aims to report whether program element A depends on program element B or not: the outcome is Boolean. An approximate dependency analysis instead reports the likelihood that A depends on B: the outcome is a real number. While probabilistic program dependency analysis techniques have been proposed before (Baah et al., 2010) they require an initial static analysis which is then extended with probabilistic information based on test executions. We conjecture that a more general analysis, based solely on dynamic observations can still be useful in many program analysis contexts while being significantly less costly.

The approximate nature of our approach stems from the fact that it infers a stochastic model of program dependences. Unlike ORBS which performs iterative deletions to analyze program dependency with respect to a single program element (i.e., the slicing criterion), our technique, MOAD (Modeling Observation-based Approximate Dependency), learns an approximate model of program dependence from significantly fewer dynamic observations. Intuitively, ORBS makes a single slice increasingly more accurate by iterative deletion. In contrast, MOAD employs a set of deletions that can be treated independently. For each, it observes multiple program elements and can thus, ultimately, learn more from each execution. This approach introduces the following benefits:

• MOAD requires many fewer observations than ORBS, as it infers the relationships between individual deletions and thus the dependency, instead of uncovering dependence information by iteratively deleting until it arrives at a one-minimal slice (Binkley et al., 2014).

• The output of MOAD can be used to construct multiple backward and forward slices, whereas a single ORBS run produces a single backward slice.

• Moreover, since the observations required by MOAD are independent from each other, MOAD is inherently parallel.

To evaluate MOAD, we have implemented it and performed dependency analysis against a benchmark suite of programs that have been widely used in the slicing literature. We evaluate the viability and the accuracy of MOAD by producing slices based on the MOAD produced model: program element A is in the backward slice of program element B iff the reported likelihood of B depending on A is greater than a threshold value. A comparison to a baseline random slicing technique shows that MOAD is indeed learning program dependences; a comparison to ORBS slices shows that MOAD can produce slices that are only 16% larger than ORBS slices, while using only 18.7% of the observations. We also investigate various ways to construct the observation sets, as well as the impact of different inference models.

In earlier work (Lee et al., 2019) we presented the basic technique and empirically evaluated its use for producing backward slices. This paper extends our previous work in three main ways. First, we provide a more extensive comparison of backward slices produced by MOAD and those produced by ORBS. Secondly, we compare MOAD-based forward slices to those of the CodeSurfer static analysis tool from Grammatech Inc. (2002). Finally, we also describe our framework in more depth and provide more extensive experiments to show the relative merits of different deletion schemes as well as inference models when inferring MOAD dependency models. Taken together this extends the use of the underlying modeling technique and provides a richer explanation both of its benefits as well as clarifying its limitations. As hinted at in Section 2, our study of slicing only scratches the surface. Our probabilistic dependency modeling is far more general. It can be applied to a wide range of problems such as Fault Diagnosis (Baah et al., 2010). However, we leave these additional use cases to future work and focus here on slicing as a representative example.

To summarize, the technical contributions of this paper are as follows:

• We introduce the concept of learning approximate dependency analysis, which transforms the dependency relationship from Boolean to probabilistic.

• We present MOAD, a technique that models approximate program dependency, and describe its essential steps: how to generate observations and how to infer models from the observations.

• We conduct an empirical evaluation of MOAD via backward program slices instantiated from the learned models.

• We show how to instantiate forward program slices from the models learned by MOAD and empirically compare these slices to those of a static analysis tool.

The rest of this paper is organized as follows. Section 2 introduces the concept of approximate dependency analysis, and explains how it relates to the existing slicing technique ORBS. Section 3 introduces MOAD, a technique that aims to model approximate dependency, and how we can use it for slicing by instantiating program slices from the learned dependency models. Section 4 presents the set-up of empirical evaluation, the results of which are reported in Section 5. Section 6 contains discussions of our findings and potential future work. Section 7 presents the related work, and Section 8 concludes.