Using clickstream data to enhance reverse engineering of Web applications

Abstract

Due to advances in Web technologies, existing Web applications are rewritten or replaced by new ones. As a result of either ad hoc or agile development, many of them lack proper technical documentation. Nevertheless, the domain knowledge built into these applications is valuable, which is why the reverse engineering, an activity aimed at detecting software components and their interrelationships to provide multiple views of software systems at a higher level of abstraction, of existing Web applications is becoming an important issue.

Apart from the static reverse engineering based on examining the system's source code, analyzing the dynamic aspect of Web applications often proves worthwhile. One important source of data the dynamic analysis of a Web application can be based on are HTTP server log files. User sessions, results of clickstream analysis, and session reconstruction in particular, can be used as the basis for the first automatic step of reverse engineering, employed in order to gain a quick insight into Web application's source code.

It is shown how clickstream data can be used to reveal not only the intensity of connections between individual Web application's source code artifacts but also the overall structure of a Web application. The extracted structure, based either on code artifacts names or their usage, is presented visually as an ATG, with code artifacts belonging to the same application module grouped together. Because session reconstruction is an inherently probabilistic process and thus in general produces noisy data, clustering the code artifacts becomes a challenging task. It is shown that multidimensional scaling and even a simple graph drawing approaches yield better representation of the application transition graph than hierarchical clustering.

The method was tested against the results obtained by an expert (the author of the Web application used as a test case). Additionally, the method can also be used for verifying the structure obtained by manual reverse engineering of the application's source code.

Introduction

With the advance of Web technologies, many existing Web applications are rewritten or replaced by new ones, which means that they are totally reengineered. In many cases, the source code is available, but reliable and comprehensive documentation is missing.

There are various reasons why the documentation might be missing:

• Some Web applications were written in a hurry without using any proper software development methodology which would ensure proper documentation.

• The existing documentation might be insufficient as the application was produced using agile software development methodology [1] which gives priority to working software over comprehensive documentation [2].

• In an application's lifetime the source code was modified either during maintenance or when new functionalities were added, but quite often many modifications are insufficiently documented or not documented at all.

It happens all too often that in such cases, reverse engineering must be performed to extract business logic from the application's source code.

Reverse engineering is a software reengineering activity aimed at detecting software components and their interrelationships to provide multiple views of software systems at a higher level of abstraction [3]. It could be considered a postcomprehension method if compared to simulation, which is a pre-comprehension method [4]. The other two major activities are comprehension and enhancement of software systems. The former is about finding out what the software components detected during reverse engineering actually do and the latter is about modifying and improving software systems.

Reverse engineering of a Web application seeks to understand its structure and functionality, and in the end it should produce (visual) representation of the application's structure. Visual representation is preferred as it has been shown that graph-based comprehension tools offer a more efficient basis for comprehension than, for example, relational database-based tools [5].

In many cases the number of components, i.e., individual Web pages and/or behind the scene codes for generating Web pages, might be relatively large. Therefore, simply displaying connections between components is not enough. One should, if at all possible, find which components form a certain part of the Web application, e.g., modules implementing individual functionalities, so that later comprehension is easier.

The structure of the article is as follows. After the description of reverse engineering topics specific to Web applications in Sections 2 and 3 gives an overview of different approaches to reverse engineering. As this article aims at extracting the structure of a Web application using clickstreams, Section 4 introduces application transition graphs (ATG), while Section 5 describes preprocessing of clickstreams and user sessions. The metrics used for clustering and visualization of ATGs are defined in Section 6, and Section 7 presents methods used for clustering and final visualization of ATGs.