Convolutional neural networks (CNNs) have achieved remarkable achievements in the fields of artificial intelligence (AI). A large volume of weights hinder the deployment of CNNs on embedded devices. The compression of CNNs can reduce the volume of weights and facilitate the hardware acceleration. In this paper, a hybrid CNN compression approach for hardware acceleration is proposed. The approach is divided into pruning phase and quantization phase. In the first phase of pruning, we use filter pruning for the convolutional (Conv) layers with complicated computing and weight pruning for the fully connected (FC) layers with the large volume of weights. In the second phase of quantization, weights bit-widths are set to 16-bit for the Conv layers and 8-bit for the FC layers. Based on experimental verification, for VGG16, 93.05% of weights are pruned and the model is compressed by 37.88 × with 1.17% accuracy loss. For ResNet18, the model is compressed by 7.27× with 0.47% accuracy loss. Experimental results show that the proposed approach can effectively reduce computational complexity and memory overhead for the implementation of CNN hardware accelerators.