Hancock: A Language for Analyzing Transactional Data Streams

Abstract

Massive transaction streams present a number of opportunities for data mining techniques. Transactions might represent calls on a telephone network, commercial credit card purchases, stock market trades, or HTTP requests to a web server. While historically such data have been collected for billing or security purposes, they are now being used to discover how the transactors, e.g., credit-card numbers or IP addresses, use the associated services. For over six years, we have computed evolving profiles (called signatures) of the transactors in several large data streams. The signature for each transactor captures the salient features of his or her transactions through time. Programs for processing signatures must be highly optimized because of the size of the data stream (several gigabytes per day) and the number of signatures to maintain (hundreds of millions). Originally, we wrote such programs directly in C, but because signature programs often sacrificed readability for performance, they were difficult to verify and maintain. Hancock is a domain-specific language created to express computationally efficient signature programs cleanly. In this chapter, we describe the obstacles to computing signatures from massive streams and explain how Hancock addresses these problems. For expository purposes, we present Hancock using a running example from the telecommunications industry; however, the language itself is general and applies equally well to other data sources.

C. Cortes, K. Fisher, D. Pregibon, A. Rogers and F. Smith, ACM Transactions on Programming Languages and Systems (TOPLAS), Volume 26 Issue 2, March 2004, Pages 301–338. DOI: 10.1145/973097.973100, © 2004 ACM, Reprinted with permission.