Abstract
Many Deep Neural Networks (DNNs) are implemented with the single objective to achieve high classification scores. However, there can be additional objectives like the minimization of computational costs. This is especially important in the field of mobile computing where not only the computational power itself is a limiting factor but also each computation consumes energy affecting the battery life. Unfortunately, the determination of minimal structures is not straightforward.
In our paper, we present a new approach to determine DNNs employing reduced structures. The networks are determined by an Evolutionary Algorithm (EA). After the DNN is trained, the EA starts to remove neurons from the network. Thereby, the fitness function of the EA is depending on the accuracy of the DNN. Thus, the EA is able to control the influence of each individual neuron. We introduce our new approach in detail. Thereby, we employ motion data recorded by accelerometer and gyroscope sensors of a mobile device. The data are recorded while drawing Japanese characters in the air in a learning context. The experimental results show that our approach is capable to determine reduced networks with similar performance to the original ones. Additionally, we show that the reduction can improve the accuracy of a network. We analyze the reduction in detail. Further, we present arising structures of the reduced networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Asher, J.J.: The total physical response approach to second language learning*. Mod. Lang. J. 53(1), 3–17 (1969)
Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Heidelberg (2007)
Bolger, D.J., Perfetti, C.A., Schneider, W.: Cross-cultural effect on the brain revisited: universal structures plus writing system variation. Hum. Brain Mapp. 25(1), 92–104 (2005)
Cun, Y.L., Denker, J.S., Solla, S.A.: Advances in neural information processing systems. In: Optimal Brain Damage, vol. 2, pp. 598–605. Morgan Kaufmann Publishers Inc., San Francisco (1990)
Deng, L., Yu, D.: Deep learning: methods and applications. Found. Trends Signal Process. 7, 197–387 (2014)
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N.: Predicting parameters in deep learning. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. NIPS 2013, vol. 2, pp. 2148–2156. Curran Associates Inc., New York (2013)
Denton, E., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. NIPS 2014, vol. 1, pp. 1269–1277. MIT Press, Cambridge (2014)
Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-662-05094-1
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org
Haarmann, H.: Symbolic Values of Foreign Language Use: From the Japanese Case to a General Sociolinguistic Perspective, Contributions to the Sociology of Language, vol. 51. Mouton de Gruyter, Berlin, New York (1989)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural network with pruning, trained quantization and Huffman coding. CoRR abs/1510.00149 (2015)
Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems. NIPS 2015, vol. 1, pp. 1135–1143. MIT Press, Cambridge (2015)
Hansen, N.: The CMA evolution strategy: a comparing review. In: Lozano, J., Larranaga, P., Inza, I., Bengoetxea, E. (eds.) Towards a New Evolutionary Computation: Advances on Estimation of Distribution Algorithms, pp. 75–102. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-32494-1_4
Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with back-propagation. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems, vol. 1, pp. 177–185. Morgan-Kaufmann, San Mateo (1989)
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, Upper Saddle River (1999)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)
Bergmann, K., Macedonia, M.: A virtual agent as vocabulary trainer: iconic gestures help to improve learners’ memory performance. In: Aylett, R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.) IVA 2013. LNCS (LNAI), vol. 8108, pp. 139–148. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40415-3_12
Kramer, O.: Evolution of convolutional highway networks. In: Sim, K., Kaufmann, P. (eds.) EvoApplications 2018. LNCS, vol. 10784, pp. 395–404. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77538-8_27
Loshchilov, I., Hutter, F.: CMA-ES for hyperparameter optimization of deep neural networks. CoRR abs/1604.07269 (2016)
Lückehe, D., Kramer, O.: Alternating optimization of unsupervised regression with evolutionary embeddings. In: Mora, A.M., Squillero, G. (eds.) EvoApplications 2015. LNCS, vol. 9028, pp. 471–480. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16549-3_38
Lückehe, D., Wagner, M., Kramer, O.: Constrained evolutionary wind turbine placement with penalty functions. In: IEEE Congress on Evolutionary Computation. CEC, pp. 4903–4910 (2016)
Macedonia, M., Mueller, K.: Exploring the neural representation of novel words learned through enactment in a word recognition task. Front. Psychol. 7, 953 (2016)
Mandischer, M.: Representation and evolution of neural networks. In: Albrecht, R.F., Reeves, C.R., Steele, N.C. (eds.) Artificial Neural Nets and Genetic Algorithms, pp. 643–649. Springer, Vienna (1993). https://doi.org/10.1007/978-3-7091-7533-0_93
Manessi, F., Rozza, A., Bianco, S., Napoletano, P., Schettini, R.: Automated pruning for deep neural network compression. CoRR abs/1712.01721 (2017)
Morse, G., Stanley, K.O.: Simple evolutionary optimization can rival stochastic gradient descent in neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016. GECCO 2016, pp. 477–484. ACM, New York (2016)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML2010, pp. 807–814. Omnipress, Madison (2010)
Olson, D., Delen, D.: Advanced Data Mining Techniques. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-76917-0
Saito, H., Masuda, H., Kawakami, M.: Form and sound similarity effects in kanji recognition. In: Leong, C.K., Tamaoka, K. (eds.) Cognitive Processing of the Chinese and the Japanese languages. Neuropsychology and Cognition, vol. 14, pp. 169–203. Springer, Dordrecht and London (1998). https://doi.org/10.1007/978-94-015-9161-4_9
Schmidhuber, J.: Deep learning in neural networks: an overview. CoRR abs/1404.7828 (2014)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)
Szegedy, C., et al.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015)
van Aacken, S.: What motivates l2 learners in acquisition of kanji using call: a case study. Comput. Assist. Lang. Learn. 12(2), 113–136 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Lückehe, D., Veith, S., von Voigt, G. (2018). Evolutionary Structure Minimization of Deep Neural Networks for Motion Sensor Data. In: Trollmann, F., Turhan, AY. (eds) KI 2018: Advances in Artificial Intelligence. KI 2018. Lecture Notes in Computer Science(), vol 11117. Springer, Cham. https://doi.org/10.1007/978-3-030-00111-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-00111-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00110-0
Online ISBN: 978-3-030-00111-7
eBook Packages: Computer ScienceComputer Science (R0)