Abstract
Over the last decades a considerable share of software engineering research has been dedicated to the area of software testing. Still, however, testing often fails or causes major problems in practice. In this paper we present insights and experiences from a large project in the energy sector. The obligatory switch from analog energy meters to smart metering technology poses a big challenge for many energy providers. Apart from technical issues concerning meters and transmission technology, the adaption of the internal business processes together with the development of backend software can turn out to be more difficult than expected. The criticality, size and complexity of the analyzed project are reflected in software and system testing, where the underestimated effort, mistakes, and wrong decisions caused serious difficulties. In our work we describe the observed testing problems and the underlying causes. Subsequently, we compare the identified problems with a catalogue of commonly known testing pitfalls and anti-patterns. The results show that the majority of the observed problems are not new or specific to the studied project. Furthermore, additional candidates for extending the list of common pitfalls are identified. Besides recommendations on how to mitigate the problems in the studied project, we conclude with the general insight that there is a great potential to improve software testing practice by developing measures for early recognition, communication, and avoiding of common mistakes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Tricentis: Software Fail Watch, 5th Ed. White paper, 27 February 2018, Tricentis (2018). https://www.tricentis.com/software-fail-watch/. Accessed 25 Aug 2018
ISO/IEC/IEEE: International Standard 29119-1 Software and systems engineering - Software testing - Part 1: Concepts and definitions. Institute of Electrical and Electronics Engineers (2013)
Jones, C., Bonsignour, O.: The Economics of Software Quality. Addison-Wesley Professional, Upper Saddle River (2011)
Bertolino, A.: Software testing research: achievements, challenges, dreams. In: Future of Software Engineering. IEEE Computer Society (2007)
Orso, A., Rothermel, G.: Software testing: a research travelogue (2000–2014). In: Proceedings of the on Future of Software Engineering. ACM (2014)
Scargle, J.D.: Publication bias: the “File-Drawer” problem in scientific inference. J. Sci. Explor. 14(1), 91–106 (2000)
Tassey, G.: The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, RTI Project, 7007-011 (2002)
Martin, D., Rooksby, J., Rouncefield, M., Sommerville, I.: ‘Good’ organisational reasons for ‘Bad’ software testing: an ethnographic study of testing in a small software company. In: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society (2007)
Kasurinen, J., Taipale, O., Smolander, K.: Analysis of problems in testing practices. In: Proceedings of the 2009 Asia-Pacific Software Engineering Conference, APSEC 2009. IEEE (2009)
Firesmith, D.: Common System and Software Testing Pitfalls: How to Prevent and Mitigate Them: Descriptions, Symptoms, Consequences, Causes, and Recommendations. Addison Wesley Professional, Upper Saddle River (2013)
European Commission: Mandate M/441 - Standardisation mandate to CEN, CENELEC and ETSI in the field of measuring instruments for the development of an open architecture for utility meters involving communication protocols enabling interoperability. European Commission, Enterprise and Industry Directorate-General, M/441 EN (2009)
Elsberg, M.: Blackout: Tomorrow Will Be Too Late. Penguin Books, London (2017)
Firesmith, D.G.: Common system and software testing pitfalls. In: Proceedings of the Team Software Process Symposium 2014 (TSP-2014), Pittsburgh, Pennsylvania. SEI (2014). https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=423692
Rooksby, J., Rouncefield, M., Sommerville, I.: Testing in the wild: the social and organisational dimensions of real world practice. Comput. Support. Coop. Work (CSCW) 18(5–6), 559–580 (2009)
Taipale, O., Smolander, K.: Improving software testing by observing practice. In: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering. ACM (2006)
Mohacsi, S., Felderer, M., Beer, A.: Estimating the cost and benefit of model-based testing: a decision support procedure for the application of model-based testing in industry. In: Proceedings of the Euromicro SEAA, Madeira, Portugal (2015)
Acknowledgments
This research has been supported by the Austrian Research Pro-motion Agency, the Austrian Ministry for Transport, Innovation and Technology, the Federal Ministry of Science, Research and Economy, and the Province of Upper Austria in the frame of the COMET center SCCH (FFG 844597).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Mohacsi, S., Ramler, R. (2019). Why Software Testing Fails: Common Pitfalls Observed in a Critical Smart Metering Project. In: Winkler, D., Biffl, S., Bergsmann, J. (eds) Software Quality: The Complexity and Challenges of Software Engineering and Software Quality in the Cloud. SWQD 2019. Lecture Notes in Business Information Processing, vol 338. Springer, Cham. https://doi.org/10.1007/978-3-030-05767-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-05767-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05766-4
Online ISBN: 978-3-030-05767-1
eBook Packages: Computer ScienceComputer Science (R0)