Abstract
Zero-shot quantization (ZSQ) is an effective way to compress neural networks, especially when real training sets are inaccessible because of privacy and security issues. Most existing synthetic-data-driven zero-shot quantization methods introduce diversity enhancement to simulate the distribution of real samples. However, the adaptivity between the enhancement degree and network is neglected, i.e., whether the enhancement degree benefits different network layers and different classes, and whether it reaches the best match between the inter-class distance and intra-class diversity. Due to the absence of the metric for class-wise and layer-wise diversity, maladaptive enhancement degree run the vulnerability of mode collapse of the inter-class inseparability. To address this issue, we propose a novel zero-shot quantization method, ADEQ. For layer-wise and class-wise adaptivity, the enhancement degree of different layers is adaptively initialized with a diversity coefficient. For inter-class adaptivity, an incremental diversity enhancement strategy is proposed to achieve the trade-off between inter-class distance and intra-class diversity. Extensive experiments on the CIFAR-100 and ImageNet show that our ADEQ is observed to have advanced performance at low bit-width quantization. For example, when ResNet-18 is quantized to 3 bits, we improve top-1 accuracy by 17.78% on ImageNet compared to the advanced ARC. Code at https://github.com/dangsingrue/ADEQ.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704–2713 (2018)
Nagel, M., Fournarakis, M., Amjad, R.A., Bondarenko, Y., Baalen, M.V., Blankevoort, T.: A white paper on neural network quantization (2021). arXiv preprint arXiv:2106.08295
Krishnamoorthi, R.: Quantizing deep convolutional networks for efficient inference: A whitepaper (2018). arXiv preprint arXiv:1806.08342
Kim, D., Lee, J., Ham, B.: Distance-aware quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5271–5280 (2021)
Nahshan, Y., et al.: Loss aware post-training quantization. Mach. Learn. 110(11–12), 3245–3262 (2021)
Bai, H., Cao, M., Huang, P., Shan, J.: BatchQuant: quantized-for-all architecture search with robust quantizer. Adv. Neural. Inf. Process. Syst. 34, 1074–1085 (2021)
Zhao, S., Yue, T., Hu, X.: Distribution-aware adaptive multi-bit quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9281–9290 (2021)
Fang, J., Shafiee, A., Abdel-Aziz, H., Thorsley, D., Georgiadis, G., Hassoun, J.H.: Post-training piecewise linear quantization for deep neural networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 69–86. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_5
Jeon, Y., Lee, C., Cho, E., Ro, Y.: Mr. BiQ: post-training non-uniform quantization based on minimizing the reconstruction error. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12329–12338 (2022)
Banner, R., Nahshan, Y., Soudry, D.: Post training 4-bit quantization of convolutional networks for rapid-deployment. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Finkelstein, A., Fuchs, E., Tal, I., Grobman, M., Vosco, N., Meller, E.: QFT: post-training quantization via fast joint finetuning of all degrees of freedom. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds.) Computer Vision – ECCV 2022 Workshops. ECCV 2022. LNCS, vol. 13807. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-25082-8_8
Zhang, X., et al.: Diversifying sample generation for accurate data-free quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15658–15667 (2021)
Zhong, Y., et al.: IntraQ: learning synthetic images with intra-class heterogeneity for zero-shot network quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12339–12348 (2022)
Gao, Y., Zhang, Z., Hong, R., Zhang, H., Fan, J., Yan, S.: Towards feature distribution alignment and diversity enhancement for data-free quantization. In: 2022 IEEE International Conference on Data Mining (ICDM), pp. 141–150. IEEE (2022)
Nagel, M., Baalen van, M., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1325–1334 (2019)
Yvinec, E., Dapogny, A., Cord, M., Bailly, K.: SPIQ: data-free per-channel static input quantization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3869–3878 (2023)
Guo, C., et al.: SQuant: On-the-fly data-free quantization via diagonal hessian approximation (2022). arXiv preprint arXiv:2202.07471
Meller, E., Finkelstein, A., Almog, U., Grobman, M.: Same, same but different: recovering neural network quantization error through weight factorization. In: International Conference on Machine Learning, pp. 4486–4495. PMLR (2019)
Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M.W., Keutzer, K.: ZeroQ: a novel zero shot quantization framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13169–13178 (2020)
Xu, S., et al.: Generative low-bitwidth data free quantization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 1–17. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_1
Zhu, B., Hofstee, P., Peltenburg, J., Lee, J., Alars, Z.: AutoReCon: neural architecture search-based reconstruction for data-free. In: International Joint Conference on Artificial Intelligence (2021)
Choi, K., Hong, D., Park, N., Kim, Y., Lee, J.: Qimera: data-free quantization with synthetic boundary supporting samples. Adv. Neural. Inf. Process. Syst. 34, 14835–14847 (2021)
Liu, Y., Zhang, W., Wang, J.: Zero-shot adversarial quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1512–1521 (2021)
Choi, K., et al.: It’s all in the teacher: zero-shot quantization brought closer to the teacher. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8311–8321 (2022)
Qian, B., Wang, Y., Hong, R., Wang, M.: Adaptive data-free quantization. arXiv e-prints, pages arXiv-2303 (2023)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115, 211–252 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Zhong, Y., et al.: Fine-grained data distribution alignment for post-training quantization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13671. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20083-0_5
Acknowledgements
This work is jointly supported by the National Natural Science Foundation of China (NSFC) (61975089), the Science and Technology Research Program of Shenzhen City (KCXFZ20201221173207022, WDZC2020200821141349001), and the Jilin Fuyuan Guan Food Group Co., Ltd.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, X. et al. (2024). ADEQ: Adaptive Diversity Enhancement for Zero-Shot Quantization. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14447. Springer, Singapore. https://doi.org/10.1007/978-981-99-8079-6_5
Download citation
DOI: https://doi.org/10.1007/978-981-99-8079-6_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8078-9
Online ISBN: 978-981-99-8079-6
eBook Packages: Computer ScienceComputer Science (R0)