FeDZIO: Decentralized Federated Knowledge Distillation on Edge Devices

Abstract

In recent years, the proliferation of edge devices and distributed sensors has fueled the need for training sophisticated deep learning models directly on resource-constrained nodes, in order to guarantee data locality and prevent the transmission of private information to centralized training infrastructures. However, executing large-scale models on edge devices poses significant challenges due to limited computational power, memory constraints and energy consumption limitation. Federated Learning (FL) has emerged as a promising approach to partially address these issues, enabling decentralized model training across multiple devices without the need to exchange local data. At the same time, Knowledge Distillation (KD) has demonstrated its efficacy in compressing complex models by transferring knowledge from a larger teacher model to a smaller student model.

This paper presents a novel framework combining Federated Learning with Knowledge Distillation, specifically tailored for accelerating training on edge devices. The proposed approach leverages the collaborative learning capabilities of federated learning to perform knowledge distillation in a privacy-preserving and efficient manner. Instead of relying on a central server for aggregation, edge devices with localized data collaboratively exchange knowledge with each other, enabling transmission of minimal quantities of data without compromising data privacy and model performance. The distributed nature of this approach allows edge devices to leverage collective intelligence while avoiding the need for sharing raw data across the network.

We conduct extensive experiments on diverse edge device scenarios using state-of-the-art deep learning architectures. The results demonstrate that our approach achieves substantial model compression while maintaining competitive performance compared to traditional knowledge distillation methods. Additionally, the federated nature of our approach ensures scalability and robustness, even in dynamic edge device environments.