Scheduling Strategies and Their Evaluation in a Data Stream Management System

Abstract

MavStream, a Data Stream Management System (DSMS), has been developed for processing stream data from applications such as network monitoring, sensor monitoring and traffic management systems that require near-real time results and have to process unbounded streams of data. In order to be useful, a result produced by MavStream has to meet certain Quality of Service (QoS) requirements on tuple latency, memory usage, and throughput. Strategies used for scheduling the operators of continuous query (CQ) significantly affect the QoS metrics and hence are of interest. This paper discusses scheduling strategies used in MavStream, their design, implementation, and evaluation. Scheduling is done in MavStream at the operator level. The scheduler maintains a ready queue of operators and decides on the operators to be scheduled based on the scheduling strategy. We first introduce the path capacity scheduling strategy with the goal of minimizing tuple latency by scheduling operator paths with maximum processing capacity. Later we discuss segment-scheduling strategy that aims at minimization of total memory requirement by scheduling operator segments with maximum memory release capacity. We then discuss simplified segment strategy, which splits operator path into just two segments providing better tuple latency performance than segment scheduling strategy and lower memory utilization than path capacity scheduling strategy. Extensive set of experiments have been designed and performed to evaluate the proposed scheduling strategies by simulating real time streams. The performance metrics of average tuple latency, memory utilization and throughput are compared with each other for different strategies and with round robin strategy to validate the analytical conclusions.

This work was supported, in part, by the following NSF grants: IIS-0326505, EIA-0216500, MRI 0421282, and IIS 0534611.