VESS: Variable Event Stream Structure for Event-based Instance Segmentation Benchmark

Comparing with traditional frame-based camera, event camera (also known as dynamic vision sensor) has received increasing attention due to various outstanding advantages. Inspired by biology, the camera naturally captures the dynamics of a scene with low latency, filtering out redundant information with low power consumption. Deep learning based instance segmentation, which are influential research in visual recognition tasks, could potentially take advantage of the benefits of event camera, but the event based application combined with deep learning still faces some challenges. In this work, we propose to develop event-based instance segmentation that unlocks the potential of the event data by combining event camera and deep learning. To make the best out of the event data, we propose a novel event representation method - variable event stream structure (VESS) for event-based instance segmentation. However, event-based datasets are rare, and none of them contains instance segmentation labels, we produce the accurate label specialized for instance segmentation on event camera. The proposed method before is verified on the dataset, and our work can reach an average Intersection over Union (IOU) of 55.75% in real-time and work properly under challenging environment like motion blur and extreme lighting condition.