Abstract
Dataset distillation aims to reduce the dataset size by capturing important information from original dataset. It can significantly improve the feature extraction effectiveness, storage efficiency and training robustness. Furthermore, we study the features from the data distillation and found unique discriminative properties that can be exploited. Therefore, based on Potential Energy Minimization, we propose a generalized and explainable dataset distillation algorithm, called Potential Energy Minimization Dataset Distillation (PEMDD). The motivation is that when the distribution for each class is regular (that is, almost a compact high-dimensional ball in the feature space) and has minimal potential energy in its location, the mixed-distributions of all classes should be stable. In this stable state, Unscented Transform (UT) can be implemented to distill the data and reconstruct the stable distribution using these distilled data. Moreover, a simple but efficient framework of using the distilled data to fuse different datasets is proposed, where only a lightweight finetune is required. To demonstrate the superior performance over other works, we first visualize the classification results in terms of storage cost and performance. We then report quantitative improvement by comparing our proposed method with other state-of-the-art methods on several datasets. Finally, we conduct experiments on few-shot learning, and show the efficiency of our proposed methods with significant improvement in terms of the storage size requirement.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ba, L.J., Caruana, R.: Do deep nets really need to be deep? arXiv preprint arXiv:1312.6184 (2013)
Bachem, O., Lucic, M., Krause, A.: Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476 (2017)
Bezdek, J.C., Kuncheva, L.I.: Nearest prototype classifier designs: an experimental study. Int. J. Intell. Syst. 16(12), 1445–1473 (2001)
Jain, J.M.C.: Textbook of Engineering Physics. Prentice-Hall of India Pvt. Limited, Indore (2009). https://books.google.com/books?id=DqZlU3RJTywC
Chen, W.Y., Liu, Y.C., Kira, Z., Wang, Y.C.F., Huang, J.B.: A closer look at few-shot classification. arXiv preprint arXiv:1904.04232 (2019)
Chen, Z., Fu, Y., Zhang, Y., Jiang, Y.G., Xue, X., Sigal, L.: Multi-level semantic feature augmentation for one-shot learning. IEEE Trans. Image Process. 28(9), 4594–4605 (2019)
Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. J. Artif. Intell. Res. 4, 129–145 (1996)
Dina, Z.E.A.: Force and potential energy, June 2019. https://chem.libretexts.org/@go/page/2063
Hariharan, B., Girshick, R.B.: Low-shot visual object recognition. CoRR abs/1606.02819 (2016). http://arxiv.org/abs/1606.02819
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logistics Q. 2(1–2), 83–97 (1955)
Larrañaga, P., Lozano, J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation, vol. 2. Springer, Cham (2001). https://doi.org/10.1007/978-1-4615-1539-5
Liu, B., et al.: Negative margin matters: understanding margin in few-shot classification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 438–455. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_26
Liu, Y., Schiele, B., Sun, Q.: An ensemble of epoch-wise empirical Bayes for few-shot learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 404–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_24
Van der Maaten, L., Hinton, G.: Visualizing data using T-SNE. J. Mach. Learn. Res. 9(11), 2579–2605 (2008)
Mangla, P., Kumari, N., Sinha, A., Singh, M., Krishnamurthy, B., Balasubramanian, V.N.: Charting the right manifold: manifold mixup for few-shot learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2218–2227 (2020)
McCall, R.: Physics of the Human Body. Johns Hopkins University Press, Baltimore (2010). https://books.google.com/books?id=LSyC41h6CG8C
Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.R.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. no. 98th8468), pp. 41–48. IEEE (1999)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Peterson, L.E.: K-nearest neighbor. Scholarpedia 4(2), 1883 (2009)
Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
Rueda, M., Martínez, S., Martínez, H., Arcos, A.: Estimation of the distribution function with calibration methods. J. Stat. Plann. Infer. 137(2), 435–448 (2007)
Rusu, A.A., et al.: Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 (2018)
Salakhutdinov, R., Tenenbaum, J., Torralba, A.: One-shot learning with a hierarchical nonparametric Bayesian model. In: Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pp. 195–206. JMLR Workshop and Conference Proceedings (2012)
Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. arXiv preprint arXiv:1708.00489 (2017)
Song, H., Diethe, T., Kull, M., Flach, P.: Distribution calibration for regression. In: International Conference on Machine Learning, pp. 5897–5906. PMLR (2019)
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP (2019)
Sucholutsky, I., Schonlau, M.: Improving dataset distillation. CoRR abs/1910.02551 (2019). http://arxiv.org/abs/1910.02551
Sucholutsky, I., Schonlau, M.: ‘Less than one’-shot learning: learning n classes from m \(<\) n samples (2020)
Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2(Nov), 45–66 (2001)
Tsang, I.W., Kwok, J.T., Cheung, P.M., Cristianini, N.: Core vector machines: fast SVM training on very large data sets. J. Mach. Learn. Res. 6(4), 363–392 (2005)
Tukey, J.W., et al.: Exploratory Data Analysis, vol. 2. Addison-Wesley, Reading (1977)
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. arXiv preprint arXiv:1606.04080 (2016)
Wan, E.A., Van Der Merwe, R.: The unscented Kalman filter for nonlinear estimation. In: Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), pp. 153–158. IEEE (2000)
Wan, E.A., Van Der Merwe, R., Haykin, S.: The unscented Kalman filter. Kalman filtering and neural networks 5(2007), 221–280 (2001)
Wang, T., Zhu, J., Torralba, A., Efros, A.A.: Dataset distillation. CoRR abs/1811.10959 (2018). http://arxiv.org/abs/1811.10959
Wang, T., Zhu, J.Y., Torralba, A., Efros, A.A.: Dataset distillation. arXiv preprint arXiv:1811.10959 (2018)
Welinder, P., et al.: Caltech-UCSD birds 200 (2010)
Xian, Y., Lorenz, T., Schiele, B., Akata, Z.: Feature generating networks for zero-shot learning. CoRR abs/1712.00981 (2017). http://arxiv.org/abs/1712.00981
Yang, S., Liu, L., Xu, M.: Free lunch for few-shot learning: distribution calibration (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Z., Yang, W., Liu, Z., Chen, Q., Ni, J., Jia, Z. (2023). Gift from Nature: Potential Energy Minimization for Explainable Dataset Distillation. In: Zheng, Y., Keleş, H.Y., Koniusz, P. (eds) Computer Vision – ACCV 2022 Workshops. ACCV 2022. Lecture Notes in Computer Science, vol 13848. Springer, Cham. https://doi.org/10.1007/978-3-031-27066-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-27066-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27065-9
Online ISBN: 978-3-031-27066-6
eBook Packages: Computer ScienceComputer Science (R0)