research-article

Unit Testing Performance in Java Projects: Are We There Yet?

Authors:

Petr TumaAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 401 - 412

https://doi.org/10.1145/3030207.3030226

Published: 17 April 2017 Publication History

Abstract

Although methods and tools for unit testing of performance exist for over a decade, anecdotal evidence suggests unit testing of performance is not nearly as common as unit testing of functionality. We examine this situation in a study of GitHub projects written in Java, looking for occurrences of performance evaluation code in common performance testing frameworks. We quantify the use of such frameworks, identifying the most relevant performance testing approaches, and describe how we adjust the design of our SPL performance testing framework to follow these conclusions.

References

[1]

S. Balsamo, A. D. Marco, P. Inverardi, and M. Simeoni. Model-based performance prediction in software development: a survey. IEEE Transactions on Software Engineering, 30(5), 2004.

Digital Library

[2]

V. Bergmann. ContiPerf. http://databene.org/contiperf, 2012.

[3]

C. Bezemer, E. Milon, A. Zaidman, and J. Pouwelse. Detecting and analyzing I/O performance regressions. Journal of Software: Evolution and Process, 26(12), 2014.

Digital Library

[4]

P. Brebner. Thoughts on the ABS Census website crash on census night (9 August 2016). http://www.performance-assurance.com.au/thoughts-on-the-abs-census-website-crash-on-census-night-9-august-2016/, 2016.

[5]

L. Bulej et al. Capturing Performance Assumptions Using Stochastic Performance Logic. In Proc. of ICPE '12. ACM, 2012.

Digital Library

[6]

L. Bulej et al. Unit testing performance with Stochastic Performance Logic. Accepted for Automated Software Engineering, 2016.

[7]

Clarkware Consulting, Inc. JUnitPerf. http://clarkware.com/software/JUnitPerf.html, 2009.

[8]

E. Daka and G. Fraser. A Survey on Unit Testing Practices and Problems. In Proc. of ISSRE '14. IEEE Computer Society, 2014.

Digital Library

[9]

A. B. de Oliveira, J.-C. Petkovich, T. Reidemeister, and S. Fischmeister. DataMill: Rigorous Performance Evaluation Made Easy. In Proce. of ICPE '13. ACM, 2013.

Digital Library

[10]

E. Engström and P. Runeson. A Qualitative Survey of Regression Testing Practices. In Product-Focused Software Process Improvement. Springer, Berlin, Heidelberg, 2010.

Digital Library

[11]

D. G. Feitelson, E. Frachtenberg, and K. L. Beck. Development and Deployment at Facebook. IEEE Internet Computing, 17(4), 2013.

Digital Library

[12]

B. George and L. Williams. An Initial Investigation of Test Driven Development in Industry. In Proc. of SAC '03. ACM, 2003.

Digital Library

[13]

W. Gottesheim. Challenges, Benefits and Best Practices of Performance Focused DevOps. In Proc. of LT '15. ACM, 2015.

Digital Library

[14]

M. Greiler, A. v. Deursen, and M.-A. Storey. Test Confessions: A Study of Testing Practices for Plug-in Systems. In Proc. of ICSE '12. IEEE Press, 2012.

[15]

C. Heger, J. Happe, and R. Farahbod. Automated Root Cause Isolation of Performance Regressions During Software Development. In Proc. of ICPE '13. ACM, 2013.

Digital Library

[16]

V. Horký et al. Performance Regression Unit Testing: A Case Study. In Computer Performance Engineering, 8168 in LNCS. Springer Berlin Heidelberg, 2013.

[17]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and Detecting Real-world Performance Bugs. In Proc. of PLDI '12. ACM, 2012.

Digital Library

[18]

JUnit. JUnit. http://junit.org/, 2006.

[19]

G. Kick, C. Decker, P. Duffin, et al. Caliper. https://github.com/google/caliper, 2015.

[20]

P. S. Kochhar, T. F. Bissyandé, D. Lo, and L. Jiang. Adoption of Software Testing in Open Source Projects--A Preliminary Study on 50,000 Projects. In Proc. of CSMR '13, 2013.

Digital Library

[21]

J. Kroß, F. Willnecker, T. Zwickl, and H. Krcmar. PET: Continuous Performance Evaluation Tool. In Proc. of QUDOS '16. ACM, 2016.

Digital Library

[22]

M. Linares-Vásquez, C. Vendome, Q. Luo, and D. Poshyvanyk. How developers detect and fix performance bottlenecks in Android apps. In Proc. of ICSME '15, 2015.

Digital Library

[23]

Y. Liu, C. Xu, and S.-C. Cheung. Characterizing and Detecting Performance Bugs for Smartphone Applications. In Proc. of ICSE '14. ACM, 2014.

Digital Library

[24]

J. D. McGregor. Test early, test often. The Journal of Object Technology, 6(4), 2007.

[25]

N. Nagappan, E. M. Maximilien, T. Bhat, and L. Williams. Realizing quality improvement through test driven development: results and experiences of four industrial teams. Empirical Software Engineering, 13(3), 2008.

Digital Library

[26]

A. Nistor, T. Jiang, and L. Tan. Discovering, Reporting, and Fixing Performance Bugs. In Proc. of MSR '13. IEEE Press, 2013.

[27]

Oracle Corporation. Java Microbenchmarking Harness (JMH). http://openjdk.java.net/projects/code-tools/jmh/, 2016.

[28]

Oracle Corporation, Project Kenai, and Cognisync. Japex Micro-benchmark Framework. https://japex.java.net/, 2014.

[29]

P. Runeson. A Survey of Unit Testing Practices. IEEE Softw., 23(4), 2006.

Digital Library

[30]

P. Stefan, V. Horký, L. Bulej, and P. Tůma. Supplementary Material. http://d3s.mff.cuni.cz/resources/icpe2017, 2017.

[31]

TestNG. TestNG. http://testng.org/, 2006.

[32]

A. van Hoorn, J. Waller, and W. Hasselbring. Kieker: A Framework for Application Performance Monitoring and Dynamic Software Analysis. In Proc. of ICPE '12. ACM, 2012.

Digital Library

[33]

J. Waller, N. C. Ehmke, and W. Hasselbring. Including Performance Benchmarks into Continuous Integration to Enable DevOps. SIGSOFT Softw. Eng. Notes, 40(2), 2015.

[34]

J. Walter, A. van Hoorn, H. Koziolek, D. Okanovic, and S. Kounev. Asking "What"?, Automating the "How"?: The Vision of Declarative Performance Engineering. In Proc. of ICPE '16. ACM, 2016.

Digital Library

[35]

E. J. Weyuker and F. I. Vokolos. Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study. IEEE Trans. Softw. Eng., 26(12), 2000.

Digital Library

[36]

M. Woodside, G. Franks, and D. C. Petriu. The Future of Software Performance Engineering. In Proc. of FOSE '07. IEEE Computer Society, 2007.

Digital Library

Cited By

Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Cao JHuang HLiu FZhang QHao Z(2024)Multi-task collaborative method based on manifold optimization for automated test case generation based on path coverageExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.123932251:COnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.123932
Jangali MTang YAlexandersson NLeitner PYang JShang W(2023)Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit TestsIEEE Transactions on Software Engineering10.1109/TSE.2022.318800549:4(1704-1725)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3188005
Show More Cited By

Index Terms

Unit Testing Performance in Java Projects: Are We There Yet?
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

The usage of open source (OS) software is wide-spread across many industries. While the functional quality of OS projects is considered to be similar to closed-source software, much is unknown about the quality in terms of performance. One challenge for ...
An Empirical Study on Code Coverage of Performance Testing
EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

Performance testing aims to ensure the operational efficiency of software systems. However, many factors influencing the efficacy and adoption of performance tests in practice are not yet fully understood. For instance, while code coverage is widely ...
Unit testing performance with Stochastic Performance Logic

Unit testing is an attractive quality management tool in the software development process, however, practical obstacles make it difficult to use unit tests for performance testing. We present Stochastic Performance Logic, a formalism for expressing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
652
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Cao JHuang HLiu FZhang QHao Z(2024)Multi-task collaborative method based on manifold optimization for automated test case generation based on path coverageExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.123932251:COnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.123932
Jangali MTang YAlexandersson NLeitner PYang JShang W(2023)Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit TestsIEEE Transactions on Software Engineering10.1109/TSE.2022.318800549:4(1704-1725)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3188005
Zhang ZXing ZXia XXu XZhu LLu Q(2023)Faster or Slower? Performance Mystery of Python Idioms Unveiled with Empirical Evidence2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00130(1495-1507)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00130
Traini LCortellessa VDi Pompeo DTucci M(2023)Towards effective assessment of steady state performance in Java software: are we there yet?Empirical Software Engineering10.1007/s10664-022-10247-x28:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10247-x
Abdullah MBulej LBures THnetynka PHorky VTuma P(2022)Reducing Experiment Costs in Automated Software Performance Regression Detection2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA56994.2022.00017(56-59)Online publication date: Aug-2022
https://doi.org/10.1109/SEAA56994.2022.00017
Reichelt DKühne SHasselbring W(2022)Automated Identification of Performance Changes at Code Level2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00096(916-925)Online publication date: Dec-2022
https://doi.org/10.1109/QRS57517.2022.00096
Markusse FLeitner PSerebrenik A(2022)Using Benchmarking Bots for Continuous Performance AssessmentIEEE Software10.1109/MS.2022.318443039:5(50-55)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1109/MS.2022.3184430
Javed OSingh PReger GToor S(2022)To test, or not to test: A proactive approach for deciding complete performance test initiation2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020543(4758-4767)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020543
Cazzola WCesarini FTansini L(2022)PerformERL: a performance testing framework for erlangDistributed Computing10.1007/s00446-022-00429-735:5(439-454)Online publication date: 1-Aug-2022
https://doi.org/10.1007/s00446-022-00429-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten