Edge-cloud collaborative systems are becoming essential for neural networks in edge applications. However, while applying pruning on the edge model is a typical process, previous works have simply removed the networks' sparsity with a global strategy, neglecting the filters' traits, environmental constraints, and edge-cloud joint optimization. This may lead to sub-optimal performance. In this paper, we propose a trainable pruning method considering system optimization. We introduce trainable gates to achieve filter-wise optimization regarding computation-aware and bandwidth-aware training to achieve a better performance. In addition, we adopt a performance predictor and a two-stage training strategy to estimate the optimal constraints-accuracy trade-off to the entire system. Finally, we validate the proposed method on the CIFAR-100 and the Tiny-ImageNet-200 dataset, and the results show that our approach significantly reduces the computation complexity and bandwidth requirement by approximately $75\%$ and $70\%$ compared to the prior work.