Abstract
Climate change is considered to be one of the most important issues we are facing right now as a specie and existent metrics and benchmarks used to evaluate the performance of different Deep Learning (DL) models and systems are currently focused mainly on their accuracy and speed, without also considering their energy consumption and cost. In this paper, we introduce four novel DL metrics, two regarding inference called Accuracy Per Consumption (APC) and Accuracy Per Energy Cost (APEC) and two regarding training called Time To Closest APC (TTCAPC) and Time To Closest APEC (TTCAPEC), which take into account not only a DL model’s accuracy but also its energy consumption, energy cost and the time it takes to train it up to that point. Experimental results prove that all four DL metrics are promising, encouraging future DL researchers to make use of models and platforms that require low power consumption as well as of green energy when powering their DL-based systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mattson, P., et al.: MLPerf Training Benchmark. arXiv:191001500 (2019)
Reddi, V.J., et al.: MLPerf Inference Benchmark. arXiv:191102549 (2019)
Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green AI. arXiv:1907.10597v3 (2019)
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. arXiv:1906.02243 (2019)
Ben-Nun, T., et al.: A modular benchmarking infrastructure for high-performance and reproducible deep learning. arXiv:190110183 (2019)
Zhu, H., et al.: TBD: benchmarking and analyzing deep neural network training. arXiv:180306905 (2018)
Jurj, S.L., Rotar, R., Opritoiu, F., Vladutiu, M.: Efficient implementation of a self-sufficient solar-powered real-time deep learning-based system. In: Iliadis, L., Angelov, P.P., Jayne, C., Pimenidis, E. (eds.) EANN 2020. PINNS, vol. 2, pp. 99–118. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-48791-1_7
Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications. arXiv:160507678 (2017)
Bianco, S., Cadene, R., Celona, L., Napoletano, P.: Benchmark analysis of representative deep neural network architectures. IEEE Access 6, 64270–64277 (2018). https://doi.org/10.1109/ACCESS.2018.2877890
García-Martín, E., et al.: Estimation of energy consumption in machine learning. J. Parallel Distrib. Comput. 134, 75–88 (2019). https://doi.org/10.1016/j.jpdc.2019.07.007
Verma, S., et al.: Metrics for machine learning workload benchmarking. In: International Workshop on Performance Analysis of Machine Learning Systems (FastPath) in Conjunction with ISPASS, Texas, The University of Texas (2019)
Strom-Report. https://1-stromvergleich.com/electricity-prices-europe/. Accessed 25 Apr 2020
Jurj, S.L., Opritoiu, F., Vladutiu, M.: Real-time identification of animals found in domestic areas of Europe. In: Proceedings of SPIE 11433, Twelfth International Conference on Machine Vision (ICMV 2019), p. 1143313 (2020). https://doi.org/10.1117/12.2556376
Convenient Power Measurement Script on the Jetson TX2/Tegra X2. https://embeddeddl.wordpress.com/2018/04/25/convenient-power-measurements-on-the-jetson-tx2-tegra-x2-board/. Accessed 03 May 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jurj, S.L., Opritoiu, F., Vladutiu, M. (2020). Environmentally-Friendly Metrics for Evaluating the Performance of Deep Learning Models and Systems. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12534. Springer, Cham. https://doi.org/10.1007/978-3-030-63836-8_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-63836-8_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63835-1
Online ISBN: 978-3-030-63836-8
eBook Packages: Computer ScienceComputer Science (R0)