Nadiya Shvai
ABSTRACT
The development of machine learning solutions often relies on training using large labeled datasets. This raises challenges in terms of data storage, data privacy protection, and longer model training time. One of the possible solutions to overcome these problems is called dataset distillation - a process of creating a smaller dataset while maximizing the preservation of its task-related information. In this paper, a new dataset distillation algorithm is proposed, called DEvS, which uses an evolutionary strategy approach to condense the training samples initially available for an image classification task, while minimizing the loss of classification accuracy. Experiments on CIFAR-10 demonstrate the competitiveness of the proposed approach. Also, contrary to recent trends, DEvS is derivative-free image generation, and therefore has greater scalability on larger input image sizes.
- Pietro Barbiero, Giovanni Squillero, and Alberto Tonda. 2020. Uncovering Core-sets for Classification With Multi-Objective Evolutionary Algorithms. arXiv preprint arXiv:2002.08645 (2020).Google Scholar
- Trevor Campbell and Tamara Broderick. 2018. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning. PMLR, 698--706.Google Scholar
- Kenneth L Clarkson. 2010. Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Transactions on Algorithms (TALG) 6, 4 (2010), 1--30.Google ScholarDigital Library
- Brendan Collins, Jia Deng, Kai Li, and Li Fei-Fei. 2008. Towards scalable dataset construction: An active learning approach. In European conference on computer vision. Springer, 86--98.Google ScholarDigital Library
- Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. 2004. Least angle regression. The Annals of statistics 32, 2 (2004), 407--499.Google ScholarCross Ref
- MA Efroymson. 1960. Multiple regression analysis. Mathematical methods for digital computers (1960), 191--203.Google Scholar
- Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence 28, 4 (2006), 594--611.Google ScholarDigital Library
- Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).Google Scholar
- Xin Li and Yuhong Guo. 2013. Adaptive active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 859--866.Google ScholarDigital Library
- Stephane G Mallat and Zhifeng Zhang. 1993. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing 41, 12 (1993), 3397--3415.Google ScholarDigital Library
- Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. 2020. Dataset Meta-Learning from Kernel Ridge-Regression. arXiv preprint arXiv:2011.00050 (2020).Google Scholar
- Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. 2021. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems 34 (2021).Google Scholar
- Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krishnaprasad. 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers. IEEE, 40--44.Google ScholarCross Ref
- Jeff M Phillips. 2016. Coresets and sketches. arXiv preprint arXiv:1601.00617 (2016).Google Scholar
- Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001--2010.Google ScholarCross Ref
- David Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web. 1177--1178.Google ScholarDigital Library
- Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017).Google Scholar
- Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017).Google Scholar
- Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1199--1208.Google ScholarCross Ref
- Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 (2018).Google Scholar
- Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems 29 (2016).Google Scholar
- Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. 2018. Dataset distillation. arXiv preprint arXiv:1811.10959 (2018).Google Scholar
- Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR) 53, 3 (2020), 1--34.Google ScholarDigital Library
- Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. Journal of Big data 3, 1 (2016), 1--40.Google ScholarCross Ref
- Gert W Wolf. 2011. Facility location: concepts, models, algorithms and case studies. Series: Contributions to Management Science: edited by Zanjirani Farahani, Reza and Hekmatfar, Masoud, Heidelberg, Germany, Physica-Verlag, 2009, 549 pp.,€ 171.15, $219.00, £ 144.00, ISBN 978-3-7908-2150-5 (hardprint), 978-3-7908-2151-2 (electronic).Google Scholar
- Donggeun Yoo and In So Kweon. 2019. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 93--102.Google ScholarCross Ref
- Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017).Google Scholar
- Bo Zhao and Hakan Bilen. 2021. Dataset Condensation with Differentiate Siamese Augmentation. arXiv preprint arXiv:2102.08259 (2021).Google Scholar
- Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2020. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 (2020).Google Scholar
- Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2020. A comprehensive survey on transfer learning. Proc. IEEE 109, 1 (2020), 43--76.Google ScholarCross Ref
Index Terms
- DEvS: data distillation algorithm based on evolution strategy
Recommendations
Multi-instance multi-label image classification: A neural approach
In this paper, a multi-instance multi-label algorithm based on neural networks is proposed for image classification. The proposed algorithm, termed multi-instance multi-label neural network (MIMLNN), consists of two stages of MultiLayer Perceptrons (MLP)...
Multiple-instance learning based decision neural networks for image retrieval and classification
The revolutionary Internet and digital technologies have spawned a need for technology that can organize abundantly available digital images for easy categorization and retrieval. Hence, content-based image retrieval (CBIR) has become one of the most ...
Transductive Multilabel Learning via Label Set Propagation
The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image ...
Comments