Skip to main content

Gift from Nature: Potential Energy Minimization for Explainable Dataset Distillation

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 Workshops (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13848))

Included in the following conference series:

  • 320 Accesses

Abstract

Dataset distillation aims to reduce the dataset size by capturing important information from original dataset. It can significantly improve the feature extraction effectiveness, storage efficiency and training robustness. Furthermore, we study the features from the data distillation and found unique discriminative properties that can be exploited. Therefore, based on Potential Energy Minimization, we propose a generalized and explainable dataset distillation algorithm, called Potential Energy Minimization Dataset Distillation (PEMDD). The motivation is that when the distribution for each class is regular (that is, almost a compact high-dimensional ball in the feature space) and has minimal potential energy in its location, the mixed-distributions of all classes should be stable. In this stable state, Unscented Transform (UT) can be implemented to distill the data and reconstruct the stable distribution using these distilled data. Moreover, a simple but efficient framework of using the distilled data to fuse different datasets is proposed, where only a lightweight finetune is required. To demonstrate the superior performance over other works, we first visualize the classification results in terms of storage cost and performance. We then report quantitative improvement by comparing our proposed method with other state-of-the-art methods on several datasets. Finally, we conduct experiments on few-shot learning, and show the efficiency of our proposed methods with significant improvement in terms of the storage size requirement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ba, L.J., Caruana, R.: Do deep nets really need to be deep? arXiv preprint arXiv:1312.6184 (2013)

  2. Bachem, O., Lucic, M., Krause, A.: Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476 (2017)

  3. Bezdek, J.C., Kuncheva, L.I.: Nearest prototype classifier designs: an experimental study. Int. J. Intell. Syst. 16(12), 1445–1473 (2001)

    Article  MATH  Google Scholar 

  4. Jain, J.M.C.: Textbook of Engineering Physics. Prentice-Hall of India Pvt. Limited, Indore (2009). https://books.google.com/books?id=DqZlU3RJTywC

  5. Chen, W.Y., Liu, Y.C., Kira, Z., Wang, Y.C.F., Huang, J.B.: A closer look at few-shot classification. arXiv preprint arXiv:1904.04232 (2019)

  6. Chen, Z., Fu, Y., Zhang, Y., Jiang, Y.G., Xue, X., Sigal, L.: Multi-level semantic feature augmentation for one-shot learning. IEEE Trans. Image Process. 28(9), 4594–4605 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  7. Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. J. Artif. Intell. Res. 4, 129–145 (1996)

    Article  MATH  Google Scholar 

  8. Dina, Z.E.A.: Force and potential energy, June 2019. https://chem.libretexts.org/@go/page/2063

  9. Hariharan, B., Girshick, R.B.: Low-shot visual object recognition. CoRR abs/1606.02819 (2016). http://arxiv.org/abs/1606.02819

  10. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)

    Google Scholar 

  11. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  12. Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logistics Q. 2(1–2), 83–97 (1955)

    Article  MathSciNet  MATH  Google Scholar 

  13. Larrañaga, P., Lozano, J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation, vol. 2. Springer, Cham (2001). https://doi.org/10.1007/978-1-4615-1539-5

  14. Liu, B., et al.: Negative margin matters: understanding margin in few-shot classification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 438–455. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_26

    Chapter  Google Scholar 

  15. Liu, Y., Schiele, B., Sun, Q.: An ensemble of epoch-wise empirical Bayes for few-shot learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 404–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_24

    Chapter  Google Scholar 

  16. Van der Maaten, L., Hinton, G.: Visualizing data using T-SNE. J. Mach. Learn. Res. 9(11), 2579–2605 (2008)

    MATH  Google Scholar 

  17. Mangla, P., Kumari, N., Sinha, A., Singh, M., Krishnamurthy, B., Balasubramanian, V.N.: Charting the right manifold: manifold mixup for few-shot learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2218–2227 (2020)

    Google Scholar 

  18. McCall, R.: Physics of the Human Body. Johns Hopkins University Press, Baltimore (2010). https://books.google.com/books?id=LSyC41h6CG8C

  19. Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.R.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. no. 98th8468), pp. 41–48. IEEE (1999)

    Google Scholar 

  20. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  21. Peterson, L.E.: K-nearest neighbor. Scholarpedia 4(2), 1883 (2009)

    Article  Google Scholar 

  22. Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)

    Google Scholar 

  23. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)

  24. Rueda, M., Martínez, S., Martínez, H., Arcos, A.: Estimation of the distribution function with calibration methods. J. Stat. Plann. Infer. 137(2), 435–448 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  25. Rusu, A.A., et al.: Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 (2018)

  26. Salakhutdinov, R., Tenenbaum, J., Torralba, A.: One-shot learning with a hierarchical nonparametric Bayesian model. In: Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pp. 195–206. JMLR Workshop and Conference Proceedings (2012)

    Google Scholar 

  27. Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. arXiv preprint arXiv:1708.00489 (2017)

  28. Song, H., Diethe, T., Kull, M., Flach, P.: Distribution calibration for regression. In: International Conference on Machine Learning, pp. 5897–5906. PMLR (2019)

    Google Scholar 

  29. Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP (2019)

    Google Scholar 

  30. Sucholutsky, I., Schonlau, M.: Improving dataset distillation. CoRR abs/1910.02551 (2019). http://arxiv.org/abs/1910.02551

  31. Sucholutsky, I., Schonlau, M.: ‘Less than one’-shot learning: learning n classes from m \(<\) n samples (2020)

    Google Scholar 

  32. Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2(Nov), 45–66 (2001)

    Google Scholar 

  33. Tsang, I.W., Kwok, J.T., Cheung, P.M., Cristianini, N.: Core vector machines: fast SVM training on very large data sets. J. Mach. Learn. Res. 6(4), 363–392 (2005)

    MathSciNet  MATH  Google Scholar 

  34. Tukey, J.W., et al.: Exploratory Data Analysis, vol. 2. Addison-Wesley, Reading (1977)

    MATH  Google Scholar 

  35. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. arXiv preprint arXiv:1606.04080 (2016)

  36. Wan, E.A., Van Der Merwe, R.: The unscented Kalman filter for nonlinear estimation. In: Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), pp. 153–158. IEEE (2000)

    Google Scholar 

  37. Wan, E.A., Van Der Merwe, R., Haykin, S.: The unscented Kalman filter. Kalman filtering and neural networks 5(2007), 221–280 (2001)

    Article  Google Scholar 

  38. Wang, T., Zhu, J., Torralba, A., Efros, A.A.: Dataset distillation. CoRR abs/1811.10959 (2018). http://arxiv.org/abs/1811.10959

  39. Wang, T., Zhu, J.Y., Torralba, A., Efros, A.A.: Dataset distillation. arXiv preprint arXiv:1811.10959 (2018)

  40. Welinder, P., et al.: Caltech-UCSD birds 200 (2010)

    Google Scholar 

  41. Xian, Y., Lorenz, T., Schiele, B., Akata, Z.: Feature generating networks for zero-shot learning. CoRR abs/1712.00981 (2017). http://arxiv.org/abs/1712.00981

  42. Yang, S., Liu, L., Xu, M.: Free lunch for few-shot learning: distribution calibration (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zijia Wang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Yang, W., Liu, Z., Chen, Q., Ni, J., Jia, Z. (2023). Gift from Nature: Potential Energy Minimization for Explainable Dataset Distillation. In: Zheng, Y., Keleş, H.Y., Koniusz, P. (eds) Computer Vision – ACCV 2022 Workshops. ACCV 2022. Lecture Notes in Computer Science, vol 13848. Springer, Cham. https://doi.org/10.1007/978-3-031-27066-6_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-27066-6_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-27065-9

  • Online ISBN: 978-3-031-27066-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics