Open Execution Engines of Stream Analysis Operations

Abstract

In this paper we describe our massively parallel and elastic stream analysis platform; it is capable of supporting the graph-structured dataflow process with each logical operator executed by multiple physical instances running in parallel over distributed server nodes. We propose the canonical dataflow operator framework to provide automated and systematic support for executing, parallelizing and granulizing the continuous operations.

We focus on the following issues: first, how to categorize the meta-properties of stream operators such as the I/O, blocking, data grouping characteristics, for providing unified and automated system support; next, how to elastically and correctly parallelize a stateful operator that is history-sensitive, relying on the prior state and data processing results; and further, how to analyze unbounded stream granularly to ensure sound semantics (e.g. aggregation). These issues are not properly abstracted and systematically addressed in the current generation of stream processing systems, but left to user programs which can result in fragile code, disappointing performance and incorrect results.

We tackle these issues by introducing the open-executors. An open executor supports the streaming operations with specific characteristics and running pattern, but is open for the application logic to be plugged-in. We illustrate the power of this approach by showing the system support in parallelizing and granulizing dataflow operations safely and correctly. The proposed canonical operation framework can be generalized to allow us to standardize various operational patterns of stream operators, and have these patterns supported systematically and automatically. We have built this platform; our experience reveals its value in real-time, continuous, elastic data-parallel and topological stream analysis process.