ASAP: Asynchronous Split Inference for Accelerated DNN Execution

With the increasing demand for real-time video processing in various applications, optimizing the deployment of deep learning models becomes crucial for efficient execution on resource-constrained devices. This paper investigates the problem of determining the optimal split point for video inference models to achieve enhanced throughput in a client-server setup. The split point refers to the layer number at which the model is divided between the client and server. In this study, we propose an asynchronous execution approach, where the client handles the initial portion of the video inference, while the server takes over the subsequent stages by processing the asynchronous execution requests sent by the client. By doing so, we use the available computational resources effectively and enhance overall throughput. The primary factors considered to identify the optimal split point are the Floating-Point Operations (FLOPs) of the client model and the data size transmitted to the server. We explore various split points within the model architecture and analyze their impact on performance in terms of computation and communication overhead. Furthermore, we quantify the benefits of the asynchronous approach by comparing it against traditional synchronous execution. Our experimental results indicate that asynchronous running achieves up to 32% higher throughput compared to synchronous execution across many state-of-the-art DNNs, demonstrating its potential for real-time video processing tasks.