With the advancement of intelligent edge computing, the deployment of deep learning models (e.g., Convolutional Neural Networks) on edge devices is becoming increasingly popular. However, the limited storage and computing capabilities of these devices present a significant bottleneck for the deployment and execution of large models. In this paper, based on Tucker decomposition, we propose a joint compression scheme (J-Tucker) to compress the CNN models in a multi-task scenario. In J-Tucker, we propose to utilize the core tensor to represent the unique information in each task while utilizing the shared factor matrices to represent the correlations and common features among multi-tasks. Moreover, we propose several novel techniques, such as a shared rank selection strategy to select the ranks to maximally reduce the redundancy in the kernels while reducing the impact on model’s accuracy. We have done extensive experiments on two neural networks (AlexNet and VGG-16) with three real datasets (CIFAR-10, CIFAR-100, and STL-10) to evaluate the effectiveness of the proposed algorithms. The experimental results demonstrate that compared with existing compression algorithms that compress each task individually, our joint compression scheme J-Tucker can achieve significantly better performance with a much higher compression ratio.