Discovering better navigation sequences for the session construction problem

Abstract

In this paper, we propose a novel page view based session model and session construction method to address the Web Usage Mining (WUM) problem. Unlike the simple session models, where sessions are sequences of web pages requested from the server (or served from a browser/proxy cache) and viewed in the browser (which may not guarantee a direct relationship between subsequent web pages in the session), we define a more realistic session model in which a session is a set of paths traversed in the web graph that corresponds to a user navigation performed by following links on web pages. We define the session construction process from raw server logs as a new graph problem and present a novel algorithm, Smart-SRA (Smart Session Reconstruction Algorithm), to solve this problem efficiently. An experimental evaluation based on data collected from real web access scenarios showed that Smart-SRA produces more accurate user sessions than the session construction methods found in the literature.

Introduction

In the last fifteen years, the World Wide Web (WWW) has become one of the largest information sources, with amazing growth in the number of web sites, the number of web pages, and the multimedia content provided (e.g., pictures, music and videos). The term “Web” usually refers to hypertext information transmitted via html pages, PDF files and other such documents. As in classical data mining, web mining aims to discover significant patterns from the WWW. The data available on the WWW can be mined mainly in three different dimensions: web content mining, web structure mining and web usage mining.

This work investigates the third dimension of web mining, namely, web usage mining (WUM) [8], [15], [26], [29], [31], which deals with extracting interesting knowledge from web log data produced by web servers. Web usage mining (WUM) has various application areas, such as predicting future requests [21], [22], web personalization [2], navigation sequence clustering [25], and providing guidance learned from user access behaviors [19]. Several e-commerce web sites use these applications to enhance purchasing experiences and customer satisfaction and thus increase their profits.

The success of the WUM applications mentioned above significantly depends on session construction because the quality (i.e., accuracy) of the constructed sessions affects later phases of WUM, such as pattern discovery. If page-view requests are not grouped correctly by session construction methods (in the first phase of web usage mining), the performance of these WUM applications suffers. To address these issues, we focus on the session construction problem alone.

The typical raw data for WUM are obtained from the access logs of web servers. In any web server, when a user agent (Internet Explorer, Mozilla, Safari, etc.) clicks a URL in a web server's domain, the information related to that request is recorded in the web server's access log file. Most access log files keep their data in the Common Log Format (CLF), where each page view request is recorded as a line in the access log file. Each CLF record is a tuple containing the following attributes:

• Client machine's IP address

• Access date and time

• Request method (GET or POST)

• URL of the page accessed

• HTTP protocol version (HTTP 1.0, HTTP 1.1)

• Success or return code

• Number of bytes transmitted

For session construction, the IP address, request time, and requested URL are the only data needed from the user web access log file to obtain the users' navigation paths. Most of the session construction methods consider only these three fields when processing web server logs. Session construction heuristics differ because, during session processing, some use time information and others use the user's navigation information [30], [32].

Producing accurate user sessions and navigation patterns is not an easy task since the HTTP protocol is stateless and connectionless. Additionally, in reactive session construction [9], [10], where it is impossible to know (or generate) client data (e.g., cookies) to identify individual users, all users behind a proxy server have the same IP number and will thus be seen as a single client on the server side. These problems can be handled by proactive strategies [16], [27], e.g., using cookies or adding client-specific information into each page request using dynamic server page codes. However, to employ proactive strategies, the internal structure and content of web pages must be changed, either by inserting JavaScript codes (called page tagging) or with dynamic server page codes. Several web sites use session tracking systems (Web Analytics Tools) provided by external services, usually by including third-party JavaScript codes in their web pages. In this case, usage data are forwarded to the third-party's servers and processed there. However, some site owners may prefer to avoid the use of proactive approaches because of security concerns or resisting modifications to a web site's internal structure; they instead process only the raw server logs containing access requests. We therefore focus this work on reactive approaches and propose a new session construction method to meet these demands. The contributions of this paper are listed below:

• We categorize previous session construction methods and explain the drawbacks of their session models. We then propose a new session model, called the “link-based session model”, and introduce its formal properties.

• We propose a new session construction method, called Smart-SRA (Smart Session Reconstruction Algorithm). It generates link-based web user sessions by inserting missing link information (pages served from the client/proxy cache). We also prove that the sessions produced by Smart-SRA satisfy the properties defined in the link-based session model.

• We perform extensive experiments on a real data set to determine the accuracy and quality of sessions constructed by Smart-SRA. Our experiments show that Smart-SRA produces at least 30% more accurate sessions than the best-known reactive session construction methods. We also conclude that link-based sessions significantly improve page prediction performance.

This paper is organized as follows. In Section 2, we summarize the session construction heuristics studied in the literature and describe the drawbacks of these methods. Section 3 introduces the link-based session model and provides the motivation for this work. Section 4 introduces the Smart-SRA algorithm and gives a detailed description. We present the experimental results in Section 5. Finally, our conclusions are discussed in Section 6.