skip to main content
10.1145/3530019.3530035acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

Towards a Catalog of Energy Patterns in Deep Learning Development

Published: 13 June 2022 Publication History

Abstract

The exponential rise of deep learning, aided by the availability of several frameworks and specialized hardware, has led to its application in a wide variety of domains. The availability of GPUs has made it easier to train networks with a huge number of parameters. However, this rise has come at the expense of ever-increasing energy requirements and carbon footprint. While the existing work tries to combat this issue by proposing optimizations in the hardware and the neural network architectures, there is an absence of general energy efficiency guidelines for deep learning developers. In this paper, we propose an initial catalog of 8 energy patterns for developing deep learning applications by analyzing 1361 posts from Stack Overflow. Our hope is that these energy patterns may help the developers adopt energy efficient practices in their deep learning projects. A survey with 14 deep learning developers showed us that the developers are largely in agreement with the usefulness of the catalog from an energy efficiency perspective. A detailed description of the catalog, along with the posts related to each energy pattern, is available at the following link: https://rishalab.github.io/dl_energy_patterns/

References

[1]
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051(2020).
[2]
Amin Banitalebi-Dehkordi. 2021. Knowledge distillation for low-power object detection: A simple technique and its extensions for training compact models using unlabeled data. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 769–778.
[3]
Shabab Bazrafkan, Tudor Nedelcu, Pawel Filipczuk, and Peter Corcoran. 2017. Deep learning for facial expression recognition: A step closer to a smartphone that knows your moods. In 2017 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 217–220.
[4]
Nazanin Beheshti and Lennart Johnsson. 2020. Squeeze U-net: A memory and energy efficient image segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 364–365.
[5]
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165(2020).
[6]
Ruichu Cai, Boyan Xu, Xiaoyan Yang, Zhenjie Zhang, Zijian Li, and Zhihao Liang. 2017. An encoder-decoder framework translating natural language to database queries. arXiv preprint arXiv:1711.06061(2017).
[7]
Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. 2016. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE journal of solid-state circuits 52, 1 (2016), 127–138.
[8]
Zhenpeng Chen, Huihan Yao, Yiling Lou, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, and Xuanzhe Liu. 2021. An empirical study on deployment faults of deep learning based mobile applications. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 674–685.
[9]
Shaiful Alam Chowdhury and Abram Hindle. 2016. Characterizing energy-aware software projects: Are they different?. In Proceedings of the 13th International Conference on Mining Software Repositories. 508–511.
[10]
Luis Cruz and Rui Abreu. 2019. Catalog of energy patterns for mobile applications. Empirical Software Engineering 24, 4 (2019), 2209–2235.
[11]
Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas. 2013. Predicting parameters in deep learning. In Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2. 2148–2156.
[12]
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 2018. Gradient Descent Provably Optimizes Over-parameterized Neural Networks. In International Conference on Learning Representations.
[13]
Xiaoting Du, Guanping Xiao, and Yulei Sui. 2020. Fault triggers in the tensorflow framework: An experience report. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 1–12.
[14]
Ottmar Edenhofer. 2015. Climate change 2014: mitigation of climate change. Vol. 3. Cambridge University Press.
[15]
Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods 5, 1 (2006), 80–92.
[16]
Junxiao Han, Emad Shihab, Zhiyuan Wan, Shuiguang Deng, and Xin Xia. 2020. What do programmers discuss about deep learning frameworks. Empirical Software Engineering 25, 4 (2020), 2694–2747.
[17]
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567(2014).
[18]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
[19]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531(2015).
[20]
Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1110–1121.
[21]
Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 510–520.
[22]
Md Johirul Islam, Rangeet Pan, Giang Nguyen, and Hridesh Rajan. 2020. Repairing deep neural networks: Fix patterns and challenges. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 1135–1146.
[23]
Xun Jiao, Vahideh Akhlaghi, Yu Jiang, and Rajesh K Gupta. 2018. Energy-efficient neural networks using approximate computation reuse. In 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 1223–1228.
[24]
Gokcen Kestor, Roberto Gioiosa, Darren J Kerbyson, and Adolfy Hoisie. 2013. Quantifying the energy cost of data movement in scientific applications. In 2013 IEEE international symposium on workload characterization (IISWC). IEEE, 56–65.
[25]
Jong Hwan Ko, Burhan Mudassar, Taesik Na, and Saibal Mukhopadhyay. 2017. Design of an energy-efficient accelerator for training of convolutional neural networks using frequency-domain computation. In 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC). IEEE, 1–6.
[26]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097–1105.
[27]
Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. Deepear: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 283–294.
[28]
Edward H Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S Simon Wong. 2017. Lognet: Energy-efficient neural networks using logarithmic computation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5900–5904.
[29]
Sun-Ro Lee, Min-Jae Heo, Chan-Gun Lee, Milhan Kim, and Gaeul Jeong. 2017. Applying deep learning based automatic bug triager to industrial projects. In Proceedings of the 2017 11th Joint Meeting on foundations of software engineering. 926–931.
[30]
Shengwen Liang, Ying Wang, Cheng Liu, Lei He, LI Huawei, Dawen Xu, and Xiaowei Li. 2020. Engn: A high-throughput and energy-efficient accelerator for large graph neural networks. IEEE Trans. Comput. (2020).
[31]
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. 2016. Fixed point quantization of deep convolutional networks. In International conference on machine learning. PMLR, 2849–2858.
[32]
Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, and Shanping Li. 2021. An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks. Empirical Software Engineering 26, 2 (2021), 1–36.
[33]
Haroon Malik, Peng Zhao, and Michael Godfrey. 2015. Going green: An exploratory analysis of energy-related questions. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 418–421.
[34]
Ian McGraw, Rohit Prabhavalkar, Raziel Alvarez, Montse Gonzalez Arenas, Kanishka Rao, David Rybach, Ouais Alsharif, Haşim Sak, Alexander Gruenstein, Françoise Beaufays, 2016. Personalized speech recognition on mobile devices. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5955–5959.
[35]
Gaurav Mittal, Kaushal B Yagnik, Mohit Garg, and Narayanan C Krishnan. 2016. Spotgarbage: smartphone app to detect garbage using deep learning. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 940–945.
[36]
Bert Moons, Bert De Brabandere, Luc Van Gool, and Marian Verhelst. 2016. Energy-efficient convnets through approximate computing. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1–8.
[37]
Bert Moons, Koen Goetschalckx, Nick Van Berckelaer, and Marian Verhelst. 2017. Minimum energy quantized neural networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers. IEEE, 1921–1925.
[38]
Irineu Moura, Gustavo Pinto, Felipe Ebert, and Fernando Castor. 2015. Mining energy-aware commits. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 56–67.
[39]
Bogdan Nicolae, Jiali Li, Justin M Wozniak, George Bosilca, Matthieu Dorier, and Franck Cappello. 2020. Deepfreeze: Towards scalable asynchronous checkpointing of deep learning models. In 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). IEEE, 172–181.
[40]
Rangeet Pan. 2020. Does fixing bug increase robustness in deep learning?. In 2020 IEEE/ACM 42nd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 146–148.
[41]
Candy Pang, Abram Hindle, Bram Adams, and Ahmed E Hassan. 2015. What do programmers know about software energy consumption?IEEE Software 33, 3 (2015), 83–89.
[42]
Eunhyeok Park, Dongyoung Kim, and Sungjoo Yoo. 2018. Energy-efficient neural network accelerator based on outlier-aware low-precision computation. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 688–698.
[43]
Abhinav Pathak, Y Charlie Hu, and Ming Zhang. 2011. Bootstrapping energy debugging on smartphones: a first look at energy bugs in mobile devices. In Proceedings of the 10th ACM Workshop on Hot Topics in Networks. 1–6.
[44]
Gustavo Pinto, Fernando Castor, and Yu David Liu. 2014. Mining questions about software energy consumption. In Proceedings of the 11th Working Conference on Mining Software Repositories. 22–31.
[45]
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084(2019).
[46]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252.
[47]
Hojjat Salehinejad and Shahrokh Valaee. 2021. Edropout: Energy-based dropout and pruning of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems (2021).
[48]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108(2019).
[49]
Syed Shakib Sarwar, Swagath Venkataramani, Aayush Ankit, Anand Raghunathan, and Kaushik Roy. 2018. Energy-efficient neural computing with approximate multipliers. ACM Journal on Emerging Technologies in Computing Systems (JETC) 14, 2(2018), 1–23.
[50]
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. 2018. A survey on deep transfer learning. In International conference on artificial neural networks. Springer, 270–279.
[51]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144(2016).
[52]
Mingyuan Xia, Wenbo He, Xue Liu, and Jie Liu. 2013. Why application errors drain battery easily? A study of memory leaks in smartphone apps. In Proceedings of the Workshop on Power-Aware Computing and Systems. 1–5.
[53]
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. 2017. Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 6071–6079.
[54]
Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 (2020), 1095–1135.
[55]
Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. 2019. An empirical study of common challenges in developing deep learning applications. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 104–115.
[56]
Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 129–140.

Cited By

View all
  • (2025)Classification and challenges of non-functional requirements in ML-enabled systems: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2025.107678181(107678)Online publication date: May-2025
  • (2023)Video Game Bad Smells: What They Are and How Developers Perceive ThemACM Transactions on Software Engineering and Methodology10.1145/356321432:4(1-35)Online publication date: 26-May-2023
  • (2023)An Exploratory Study on Energy Consumption of Dataframe Processing Libraries2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00048(284-295)Online publication date: May-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering
June 2022
466 pages
ISBN:9781450396134
DOI:10.1145/3530019
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep learning
  2. energy efficiency
  3. neural networks
  4. patterns

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

EASE 2022

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)61
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Classification and challenges of non-functional requirements in ML-enabled systems: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2025.107678181(107678)Online publication date: May-2025
  • (2023)Video Game Bad Smells: What They Are and How Developers Perceive ThemACM Transactions on Software Engineering and Methodology10.1145/356321432:4(1-35)Online publication date: 26-May-2023
  • (2023)An Exploratory Study on Energy Consumption of Dataframe Processing Libraries2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00048(284-295)Online publication date: May-2023
  • (2023)A catalogue of game-specific anti-patterns based on GitHub and Game Development Stack ExchangeJournal of Systems and Software10.1016/j.jss.2023.111789204:COnline publication date: 1-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media