Abstract
Learning features from raw data is an important topic in machine learning. This paper presents Genetic Program Feature Learner (GPFL), a novel generative GP feature learner for 2D images. GPFL executes multiple GP runs, each run generates a model that focuses on a particular high-level feature of the training images. Then, it combines the models generated by each run into a function that reconstructs the observed images. As a sanity check, we evaluated GPFL on the popular MNIST dataset of handwritten digits, and compared it with the convolutional neural network LeNet5. Our evaluation results show that when considering smaller training sets, GPFL achieves comparable/slightly-better classification accuracy than LeNet5. However, GPFL drastically outperforms LeNet5 when considering noisy images as test sets.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We presented a preliminary version of this work in a poster paper [40].
- 2.
The cost of computing the linear scaling coefficients is \(\mathcal {O}(\mid \mathbb {\hat{Y}}\mid \cdot \mid \mathcal {P} \mid )\).
- 3.
When comparing the classification accuracy of GPFL and LeNet5, we computed the p-values with the non-parametric pairwise Wilcoxon rank-sum test [15].
References
Albukhanajer, W.A., Briffa, J.A., Jin, Y.: Evolutionary multiobjective image feature extraction in the presence of noise. IEEE Trans. Cybern. 45(9), 1757–1768 (2015). https://doi.org/10.1109/TCYB.2014.2360074
Alvear-Sandoval, R.F., Sancho-Gómez, J.L., Figueiras-Vidal, A.R.: On improving CNNs performance: the case of MNIST. Inf. Fusion 52, 106–109 (2019)
Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283, 38–52 (2018). https://doi.org/10.1016/j.neucom.2017.12.049
Baldominos, A., Saez, Y., Isasi, P.: Model selection in committees of evolved convolutional neural networks using genetic algorithms. In: Intelligent Data Engineering and Automated Learning, IDEAL 2018, pp. 364–373 (2018). https://doi.org/10.1007/978-3-030-03493-1_39
Baldominos, A., Saez, Y., Isasi, P.: Hybridizing evolutionary computation and deep neural networks: an approach to handwriting recognition using committees and transfer learning. Complexity (2019). https://doi.org/10.1155/2019/2952304
Baldominos, A., Saez, Y., Isasi, P.: A survey of handwritten character recognition with MNIST and EMNIST. Appl. Sci. 9(15), 3169 (2019)
Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32
Bochinski, E., Senst, T., Sikora, T.: Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In: Proceedings International Conference on Image Processing, ICIP 2017, pp. 3924–3928 (2017). https://doi.org/10.1109/ICIP.2017.8297018
Brameier, M.F., Banzhaf, W.: Linear Genetic Programming. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-31030-5_1
Butterworth, J., Savani, R., Tuyls, K.: Evolving indoor navigational strategies using gated recurrent units in NEAT. In: Proceedings of the Companion of Genetic and Evolutionary Computation Conference, GECCO 2019, pp. 111–112 (2019). https://doi.org/10.1145/3319619.3321995
Davison, J.: DEvol: automated deep neural network design via genetic programming (2020). https://github.com/joeddav/devol
Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 594–611 (2006)
George, D., et al.: A generative vision model that trains with high data efficiency and breaks text-based captchas. Science 358(6368), 2612 (2017)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
Haynes, W.: Wilcoxon rank sum test. In: Encyclopedia of Systems Biology, pp. 2354–2355 (2013)
Impedovo, S., Mangini, F.: A novel technique for handwritten digit classification using genetic clustering. In: Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2012, pp. 236–240 (2012). https://doi.org/10.1109/ICFHR.2012.167
Keijzer, M.: Improving symbolic regression with interval arithmetic and linear scaling. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 70–82. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36599-0_7
Kramer, M.A.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 37(2), 233–243 (1991)
LeCun, Y.: Lenet-5, convolutional neural networks (2020). http://yann.lecun.com/exdb/lenet
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition, vol. 86, pp. 2278–2324. IEEE (1998)
Legge, G.E., Foley, J.M.: Contrast masking in human vision. Josa 70(12), 1458–1471 (1980)
Lensen, A., Al-Sahaf, H., Zhang, M., Xue, B.: Genetic programming for region detection, feature extraction, feature construction and classification in image data. In: Heywood, M.I., McDermott, J., Castelli, M., Costa, E., Sim, K. (eds.) EuroGP 2016. LNCS, vol. 9594, pp. 51–67. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30668-1_4
Lensen, A., Xue, B., Zhang, M.: Can genetic programming do manifold learning too? In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds.) EuroGP 2019. LNCS, vol. 11451, pp. 114–130. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16670-0_8
Lensen, A., Zhang, M., Xue, B.: Multi-objective genetic programming for manifold learning: balancing quality and dimensionality. Genet. Program Evolvable Mach. 21(3), 399–431 (2020). https://doi.org/10.1007/s10710-020-09375-4
Liu, L., Shao, L., Li, X.: Evolutionary compact embedding for large-scale image classification. Inf. Sci. 316, 567–581 (2015). https://doi.org/10.1016/j.ins.2014.06.030
López, U., Trujillo, L., Martinez, Y., Legrand, P., Naredo, E., Silva, S.: RANSAC-GP: dealing with outliers in symbolic regression with genetic programming. In: Proceedings of the European Conference on Genetic Programming, EuroGP 2017, pp. 114–130 (2017)
Makhzani, A., Frey, B.J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)
McDermott, J.: Why is auto-encoding difficult for genetic programming? In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds.) EuroGP 2019. LNCS, vol. 11451, pp. 131–145. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16670-0_9
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.: Reading digits in natural images with unsupervised feature learning. Google technical report (2011)
Neumann, L., Matas, J.: Real-time scene text localization and recognition. In: Proceedings of Conference on Computer Vision and Pattern Recognition, CVPR 2012, pp. 3538–3545 (2012)
Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36 (2006)
Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now?: a large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 1183–1190 (2018). https://doi.org/10.1145/3205455.3205539
Papavasileiou, E., Jansen, B.: An investigation of topological choices in FS-NEAT and FD-NEAT on XOR-based problems of increased complexity. In: Proceedings of the Companion of Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 1431–1434 (2017). https://doi.org/10.1145/3067695.3082497
Peng, Y., Chen, G., Singh, H., Zhang, M.: NEAT for large-scale reinforcement learning through evolutionary feature learning and policy gradient search. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 490–497 (2018). https://doi.org/10.1145/3205455.3205536
Perez, C.B., Olague, G.: Genetic programming as strategy for learning image descriptor operators. Intell. Data Anal. 17(4), 561–583 (2013). https://doi.org/10.3233/IDA-130594
Rodriguez-Coayahuitl, L., Morales-Reyes, A., Escalante, H.J.: Structurally layered representation learning: towards deep learning through genetic programming. In: Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., García-Sánchez, P. (eds.) EuroGP 2018. LNCS, vol. 10781, pp. 271–288. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77553-1_17
Rodriguez-Coayahuitl, L., Morales-Reyes, A., Escalante, H.J.: Evolving autoencoding structures through genetic programming. Genet. Program Evolvable Mach. 20(3), 413–440 (2019). https://doi.org/10.1007/s10710-019-09354-4
Ruberto, S., Terragni, V., Moore, J.H.: GPFL replication package. experimental data of GPFL and source code of Lenet5, April 2020. https://doi.org/10.5281/zenodo.3899891
Ruberto, S., Terragni, V., Moore, J.H.: Image feature learning with a genetic programming autoencoder. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2020 (2020)
Ruberto, S., Terragni, V., Moore, J.H.: SGP-DT: semantic genetic programming based on dynamic targets. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds.) EuroGP 2020. LNCS, vol. 12101, pp. 167–183. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44094-7_11
Ruberto, S., Terragni, V., Moore, J.H.: SGP-DT: towards effective symbolic regression with a semantic GP approach based on dynamic targets. In: Proceedings of the Genetic and Evolutionary Computation Conference (Hot Off the Press track), GECCO 2020 (2020)
Ruberto, S., Vanneschi, L., Castelli, M.: Genetic programming with semantic equivalence classes. Swarm Evol. Comput. 44, 453–469 (2019). https://doi.org/10.1016/j.swevo.2018.06.001
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy (2018)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the International Conference on Machine Learning, ICML 2008, pp. 1096–1103 (2008)
Wang, J., Zhang, Z., Zha, H.: Adaptive manifold learning. In: Advances in Neural Information Processing Systems, NIPS 2005 (2005)
Yadav, C., Bottou, L.: Cold case: the lost MNIST digits. In: Advances in Neural Information Processing Systems, NIPS 2019, pp. 13443–13452 (2019)
Yann LeCun, C.C., Burges, C.: MNIST handwritten digit database (2020)
Zheng, A., Casari, A.: Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O’Reilly Media Inc., Sebastopol (2018)
Zhou, H., Yuan, Y., Shi, C.: Object tracking using sift features and mean shift. Comput. Vis. Image Underst. 113(3), 345–352 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ruberto, S., Terragni, V., Moore, J.H. (2020). Image Feature Learning with Genetic Programming. In: Bäck, T., et al. Parallel Problem Solving from Nature – PPSN XVI. PPSN 2020. Lecture Notes in Computer Science(), vol 12270. Springer, Cham. https://doi.org/10.1007/978-3-030-58115-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-58115-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58114-5
Online ISBN: 978-3-030-58115-2
eBook Packages: Computer ScienceComputer Science (R0)