Semi-Supervised Multi-modal Sensor Fusion Framework for In-Vehicle Networks

With the rapid development of technologies such as autonomous driving, vehicle-to-everything communication, and edge computing, an increasing number of vehicles are equipped with multiple sensors to perceive the surroundings. As a result, the amount of sensing data has exploded, and the communication pressure on the in-vehicle network becomes severe. In-sensor or near-sensor computation is considered an effective method to address these issues. However, current multi-modal fusion frameworks are challenging to be modularised and trained in a distributed manner across multiple devices. In this paper, we propose a variational autoencoder (VAE) based multi-modal fusion solution with its theoretical analysis framework. Notably, we design two auxiliary tasks to utilize data from a single modality to discover the joint distribution of multiple modalities. Compared to traditional algorithms, the proposed solution is able to use unlabeled data for self-supervised learning and has the added advantage of modularity, which helps to reduce the communication overhead in in-vehicle networks. Experiments show that, compared to single-modality algorithms, our multi-modal fusion framework increases average precision by over 10% on the KITTI dataset.