skip to main content
10.1145/1808266.1808268acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Automating performance testing of interactive Java applications

Published: 03 May 2010 Publication History

Abstract

Interactive applications with graphical user interfaces are prevalent in today's environment: Everybody with access to any kind of computer constantly uses them. A significant body of prior work has devised approaches for automating the functional testing of such applications. However, no such work exists for automatically testing their performance. Performance testing imposes additional requirements upon GUI test automation tools: the tools have to be able to replay complex interactive sessions, and they have to avoid perturbing the application's performance. We study the feasibility of using five Java GUI capture & replay tools for GUI performance test automation. Besides confirming the severity of the previously known GUI element identification problem, we also identify a related problem, the temporal synchronization problem, which is of increasing importance for GUI applications that use timer-driven activity. We find that most of the tools we study have severe limitations when used for recording and replaying realistic sessions of real-world Java applications, and that all of them suffer from the temporal synchronization problem. However, we find that the most reliable tool, Pounder, causes only limited perturbation, and thus can be used to automate performance testing. Besides the significance of our findings to GUI performance testing, the results are also relevant to capture & replay-based functional GUI test automation approaches.

References

[1]
I. Alsmadi. The utilization of user sessions in testing. In ICIS '08: Proceedings of the Seventh IEEE/ACIS International Conference on Computer and Information Science (icis 2008), pages 581--585, Washington, DC, USA, 2008. IEEE Computer Society.
[2]
P. A. Brooks and A. M. Memon. Automated gui testing guided by usage profiles. In ASE '07: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pages 333--342, New York, NY, USA, 2007. ACM.
[3]
D. A. S. de Oliveira, J. R. Crandall, G. Wassermann, S. F. Wu, Z. Su, and F. T. Chong. Execrecorder: Vm-based full-system replay for attack analysis and system recovery. In ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability, pages 66--71, New York, NY, USA, 2006. ACM.
[4]
M. Grechanik, Q. Xie, and C. Fu. Maintaining and evolving gui-directed test scripts. In ICSE '09: Proceedings of the 2009 IEEE 31st International Conference on Software Engineering, pages 408--418, Washington, DC, USA, 2009. IEEE Computer Society.
[5]
M. Jovic and M. Hauswirth. Measuring the performance of interactive applications with listener latency profiling. In PPPJ '08: Proceedings of the 6th international symposium on Principles and practice of programming in Java, pages 137--146, New York, NY, USA, 2008. ACM.
[6]
H. Liu, H. Jin, X. Liao, L. Hu, and C. Yu. Live migration of virtual machine based on full system trace and replay. In HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing, pages 101--110, New York, NY, USA, 2009. ACM.
[7]
S. McMaster and A. M. Memon. An extensible heuristic-based framework for gui test case maintenance. In TESTBEDS '09: Proceedings of the First International Workshop on TESTing Techniques & Experimentation Benchmarks for Event-Driven Software, 2009.
[8]
A. M. Memon. Automatically repairing event sequence-based gui test suites for regression testing. ACM Trans. Softw. Eng. Methodol., 18(2):1--36, 2008.
[9]
S. Narayanasamy, G. Pokam, and B. Calder. Bugnet: Continuously recording program execution for deterministic replay debugging. In ISCA '05: Proceedings of the 32nd annual international symposium on Computer Architecture, pages 284--295, Washington, DC, USA, 2005. IEEE Computer Society.
[10]
M. Ronsse and K. De Bosschere. Recplay: a fully integrated practical record/replay system. ACM Trans. Comput. Syst., 17(2):133--152, 1999.
[11]
M. Ronsse, K. De Bosschere, M. Christiaens, J. C. de Kergommeaux, and D. Kranzlmüller. Record/replay for nondeterministic program executions. Commun. ACM, 46(9):62--67, 2003.
[12]
J. Steven, P. Chandra, B. Fleck, and A. Podgurski. jRapture: A Capture/Replay tool for observation-based testing. SIGSOFT Softw. Eng. Notes, 25(5):158--167, 2000.

Cited By

View all
  • (2020)A Survey of Context Simulation for Testing Mobile Context-Aware ApplicationsACM Computing Surveys10.1145/337278853:1(1-39)Online publication date: 6-Feb-2020
  • (2020)Performance Evaluation of Graphical User Interfaces in Java and C#2020 International Conference on Computer Science and Software Engineering (CSASE)10.1109/CSASE48920.2020.9142075(290-295)Online publication date: Apr-2020
  • (2020)Robust UI Automation Using Deep Learning and Optical Character Recognition (OCR)Proceedings of International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications10.1007/978-981-15-7234-0_4(33-44)Online publication date: 18-Oct-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AST '10: Proceedings of the 5th Workshop on Automation of Software Test
May 2010
154 pages
ISBN:9781605589701
DOI:10.1145/1808266
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 May 2010

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Conference

ICSE '10
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)2
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)A Survey of Context Simulation for Testing Mobile Context-Aware ApplicationsACM Computing Surveys10.1145/337278853:1(1-39)Online publication date: 6-Feb-2020
  • (2020)Performance Evaluation of Graphical User Interfaces in Java and C#2020 International Conference on Computer Science and Software Engineering (CSASE)10.1109/CSASE48920.2020.9142075(290-295)Online publication date: Apr-2020
  • (2020)Robust UI Automation Using Deep Learning and Optical Character Recognition (OCR)Proceedings of International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications10.1007/978-981-15-7234-0_4(33-44)Online publication date: 18-Oct-2020
  • (2019)Sara: self-replay augmented record and replay for Android in industrial casesProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330557(90-100)Online publication date: 10-Jul-2019
  • (2019)Automating Detection of Security-Related Software Engineering Failures2019 IEEE AUTOTESTCON10.1109/AUTOTESTCON43700.2019.8961892(1-7)Online publication date: Aug-2019
  • (2018)A Perception of the Practice of Software Security and Performance Verification2018 25th Australasian Software Engineering Conference (ASWEC)10.1109/ASWEC.2018.00018(71-80)Online publication date: Nov-2018
  • (2018)Balancing parallelization and asynchronization in event‐driven programs with OpenMPConcurrency and Computation: Practice and Experience10.1002/cpe.495931:4Online publication date: 21-Sep-2018
  • (2017)ERP prototype with built-in task and process supportEuropean Journal of Information Systems10.1057/s41303-017-0060-327:2(189-206)Online publication date: 4-Dec-2017
  • (2016)MobiPlayProceedings of the 38th International Conference on Software Engineering10.1145/2884781.2884854(571-582)Online publication date: 14-May-2016
  • (2013)RERAN: timing- and touch-sensitive record and replay for AndroidProceedings of the 2013 International Conference on Software Engineering10.5555/2486788.2486799(72-81)Online publication date: 18-May-2013
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media