SVIS: Large Scale Video Data Ingestion into Big Data Platform

Abstract

Utilizing big data processing platform to analyze and extract insights from unstructured video streams becomes emerging trend in video surveillance area. As the first step, how to efficiently ingest video sources into big data platform is most demanding but challenging problem. However, existing data loading or ingesting tools either lack of video ingestion capability or cannot handle such huge volume of video data. In this paper, we present SVIS, a highly scalable and extendable video data ingestion system which can fast ingest different kinds of video source into centralized big data stores. SVIS embeds rich video content processing functionalities, e.g. video transcoding and object detection. As a result, the ingested data will have desired formats (i.e. structured data, well-encoded video sequence files) and hence can be analyzed directly. With a highly scalable architecture and an intelligent schedule engine, SVIS can be dynamically scaled out to handle large scale online camera streams and intensive ingestion jobs. SVIS is also highly extendable. It defines various interfaces to enable embedding user-defined modules to support new types of video source and data sink. Experimental results show that SVIS system has high efficiency and good scalability.