Supporting concept location through identifier parsing and ontology extraction

Abstract

Identifier names play a key role in program understanding and in particular in concept location. Programmers can easily “parse” identifiers and understand the intended meaning. This, however, is not trivial for tools that try to exploit the information in the identifiers to support program understanding. To address this problem, we resort to natural language analyzers, which parse tokenized identifier names and provide the syntactic relationships (dependencies) among the terms composing the identifiers. Such relationships are then mapped to semantic relationships.

In this study, we have evaluated the use of off-the-shelf and trained natural language analyzers to parse identifier names, extract an ontology and use it to support concept location. In the evaluation, we assessed whether the concepts taken from the ontology can be used to improve the efficiency of queries used in concept location. We have also investigated if the use of different natural language analyzers has an impact on the ontology extracted and the support it provides to concept location. Results show that using the concepts from the ontology significantly improves the efficiency of concept location queries (e.g., in some cases, an improvement of 127% is observed). The results also indicate that the efficiency of concept location queries is not affected by the differences in the ontologies produced by different analyzers.

Introduction

During program understanding, source code exploration in search for a specific concept is a typical activity. One of the key source code elements which affects this activity is identifiers. Identifiers serve as a link between the intention of a concept and its extension in the source code (Rajlich, 2009). Different approaches which take advantage of this fact have been proposed by various authors to improve code search and support program understanding (see Marcus et al., 2004, Hill et al., 2009, Gay et al., 2009, Abebe and Tonella, 2010, Ratiu et al., 2008, Shepherd et al., 2007).

The intention of a concept in an identifier is reflected in the terms chosen, their relative order and the relationships among them. For example, an identifier name constructed from the terms record and event, might convey two different meanings to the reader when the terms are used in a different sequence: recordEvent, which might mean logging an event, and eventRecord, a type of record. While this difference is easy to grasp for a human, it is not obvious for tools that are designed to support program understanding. To address this problem, we resort to natural language dependency analyzers, which can be used to extract the semantic relationships among the identifier terms.

Natural language dependency analyzers present the syntactic relationship between the identifier terms through natural language dependencies (e.g., noun-specifier or direct-object) among them. Though these relations are syntactic, very often the terms they connect are semantically related. Of course this is not always the case, and, for example, the relation between a determiner and the corresponding noun is not semantically relevant. However, the relation between a verb and its object usually has a semantic nature. Therefore, we focus on some categories of the dependencies extracted from the analyzer in order to obtain relevant semantic relations for our aims.

In our previous work (Abebe and Tonella, 2010), we have exploited such information to extract an ontology from identifiers and support program understanding. The extraction of the ontology is conducted by parsing sentences constructed from identifiers. In this work, in addition to the natural language dependency analyzer used in Abebe and Tonella (2010), we have considered analyzers which are trained on a training set that closely resembles the structure of identifiers. The training set is constructed from sentences and phrases automatically extracted from the textual documentation of the system which comes with the source code and is available online. The documentation we considered consists of source code comments, user manuals, system documentation, and FAQs describing howtos, when available. The extracted sentences and phrases are automatically converted to identifiers that resemble true code identifiers. This is achieved by applying a predefined set of rules for identifier construction from natural language sentences. For these artificially constructed identifiers we know the correct parse trees, obtained from the parsing of the original sentences or phrases. Hence, we can use them as a training set.

The natural language dependency trees generated by the analyzers are used to build ontologies, which can support program understanding. Based on the level of formality, an ontology can vary from a simple taxonomy with almost no formalization, to one which uses a rigorously formalized theory (see Uschold and Gruninger, 2004). Ontology in our context is a “lightweight ontology” which is in between these two extremes and does not include axioms supporting formal reasoning, but only considers concepts and relations connecting the concepts. A lightweight ontology which is built using only concepts and relations connecting the concepts without any formalization is sometimes referred to as “concept map”. In this paper we refer to such lightweight ontology simply as ontology.

In this study, we have assessed the benefits of using ontologies extracted from identifiers in several subject applications and we have investigated the impact of using different analyzers to generate the ontologies. The assessment was conducted in the context of a program understanding task, namely concept location, which uses queries to narrow down the search space and identify the parts of a program that implement a concept of interest. The study we conducted, in particular, answers three research questions: (RQ1) Do the extracted ontology concepts contribute to increasing the effectiveness of basic queries formulated for concept location?, (RQ2) Do the ontologies produced by different analyzers differ between each other?; and (RQ3) Does the choice of the analyzer impact the effectiveness of concept location?. The obtained results indicate that exploiting the concepts taken from the ontologies improves the quality of queries used in concept location and that this is independent of the analyzer employed, despite the fact that different analyzers generate quite different ontologies.

The main contributions of this paper as compared to our previous work (Abebe and Tonella, 2010) are:

• An approach to train natural language analyzers for use with identifiers.

• A through empirical assessment of the impact of ontologies on concept location using two additional systems (FileZilla, JEdit) and computing additional metrics (e.g., average delta percentage, net improvements, mean reciprocal rank, etc.).

• An analysis of the impact of using ontologies on latent semantic indexing based concept location.

• A comparison among the different natural language analyzers investigated in this work in terms of differences between the extracted ontologies (measuring Jaccard index and ratio of unique concepts), in addition to their support to concept location.

Section 2 presents the related works that use identifiers as their main source of information to minimize the search space and facilitate program understanding. In Section 3, we describe the different types of natural language analyzers used in this study. The mapping of natural language dependencies and ontological relations are presented in Section 4 while the steps involved in the concept location task are described in Section 5. Section 6 presents the case study, including procedure, results and discussion. In the last section (Section 7), we provide our concluding remarks.