Lunatory: A Real-Time Distributed Trajectory Clustering Framework for Web Big Data

Abstract

Web big data contains a wealth of valuable information, which can be extracted through web mining and knowledge extraction. Among them, the real-time location information of web can provide a richer calculation basis for existing applications, such as real-time monitoring systems and recommendation systems based on real-time trajectory clustering. However, as a trajectory is a sequence of user positions in the time dimension, the correlation calculation of the trajectories will inevitably incur a massive computational cost. In addition, such trajectory data is usually time-sensitive, that is, once the trajectory data has been generated and changed, the corresponding clustering results need to be output with low latency. Although the offline trajectory clustering has been well studied, extending such work to an online environment directly tends to incur (1) expensive network cost, (2) high processing latency, and (3) low accuracy results. To enable a real-time clustering on trajectory stream, we propose a distributed cLustering framework for hexagonal-based streaming trajectory (Lunatory). Lunatory covers three key components, that are: (1) Simplifier: to solve the problem of extensive network transmission in a distributed trajectory streaming system, a pivot trajectory data structure is introduced to simplify trajectories by reducing the number of samples and extracting key features; (2) Partitioner: to enhance the local computational efficiency of subsequent clustering, a hexagonal-based indexing strategy is proposed to index the pivot trajectories; (3) Executor extends DBSCAN to pivot trajectories and implements real-time trajectory clustering based on Flink. Empirical studies on real-world data validate the usefulness of our proposal and prove the huge advantage of our approach over available solutions in the literature.