skip to main content
10.1145/1134760.1134776acmconferencesArticle/Chapter ViewAbstractPublication PagesveeConference Proceedingsconference-collections
Article

Relative factors in performance analysis of Java virtual machines

Published: 14 June 2006 Publication History

Abstract

Many new Java runtime optimizations report relatively small, single-digit performance improvements. On modern virtual and actual hardware, however, the performance impact of an optimization can be influenced by a variety of factors in the underlying systems. Using a case study of a new garbage collection optimization in two different Java virtual machines, we show the relative effects of issues that must be taken into consideration when claiming an improvement. We examine the specific and overall performance changes due to our optimization and show how unintended side-effects can contribute to, and distort the final assessment. Our experience shows that VM and hardware concerns can generate variances of up to 9.5% in whole program execution time. Consideration of these confounding effects is critical to a good, objective understanding of Java performance and optimization.

References

[1]
B. Alpern, C. R. Attanasio, A. Cocchi, D. Lieber, S. Smith, T. Ngo, J. J. Barton, S. F. Hummel, J. C. Sheperd, and M. Mergen. Implementing Jalapeño in Java. In OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 314--324, Oct. 1999. ISBN1-58113-238-7.]]
[2]
J. M. Anderson, L. M. Berc, J. Dean, S. Ghemawat, M. R. Henzinger, S.-T. A. Leung, R. L. Sites, M. T. Vandevoorde, C. A. Waldspurger, and W. E. Weihl. Continuous profiling: where have all the cycles gone? ACM Trans. Comput. Syst., 15(4):357--390, Nov. 1997. ISSN0734-2071.]]
[3]
R. Berrendorf, H. Ziegler, and B. Mohr. PCL-the performance counter library. http://www.fz-juelich.de/zam/PCL/.]]
[4]
S. M. Blackburn, P. Cheng, and K. S. McKinley. Myths and realities: The performance impact of garbage collection. In Proceedings of the ACM SIGMETRICS Conference on Measurement & Modeling Computer Systems, pages 25--36, June 2004.]]
[5]
S. M. Blackburn, R. Jones, K. S. McKinley, and J. E. B. Moss. Beltway: getting around garbage collection gridlock. SIGPLAN Not., 37(5):153--164, June 2002. ISSN0362-1340.]]
[6]
H.-J. Boehm. Reducing garbage collector cache misses. In ISMM '00: Proceedings of the 2nd international symposium on Memory management, pages 59--64, Oct. 2000. ISBN1-58113-263-8.]]
[7]
S. Brown, J. Dongarra, N. Garner, K. London, and P. Mucci. PAPI. http://icl.cs.utk.edu/papi.]]
[8]
C.-Y. Cher, A. L. Hosking, and T. N. Vijaykumar. Software prefetching for mark-sweep garbage collection: hardware analysis and software redesign. In ASPLOS-XI: Proceedings of the 11th international conference on Architectural support for programming languages and operating systems, pages 199--210, Oct. 2004. ISBN1-58113-804-0.]]
[9]
E. Duesterwald, C. Cascaval, and S. Dwarkadas. Characterizing and predicting program behavior and its variability. In PACT '03: Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques, page 220. IEEE Computer Society, Sep. 2003. ISBN0-7695-2021-9.]]
[10]
B. Dufour, K. Driesen, L. Hendren, and C. Verbrugge. Dynamic metrics for Java. In Proceedings of the ACM SIGPLAN 2003 Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 149--168, Oct. 2003. ISBN1-58113-712-5.]]
[11]
L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In Proceedings of the 18th ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications, pages 169--186, Oct. 2003. ISBN1-58113-712-5.]]
[12]
E. M. Gagnon. SableVM. http://www.sablevm.org/.]]
[13]
E. M. Gagnon. A Portable Research Framework for the Execution of Java Bytecode. PhD thesis, McGill University, 2002.]]
[14]
E. M. Gagnon and L. J. Hendren. SableVM:A Research Framework for the Efficient Execution of Java Bytecode. In Proceedings of the Java Virtual Machine Research and Technology Symposium (JVM '01), pages 27--40. USENIX Association, Apr. 2001.]]
[15]
D. Gu, C. Verbrugge, and E. Gagnon. Assessing the impact of optimization in Java virtual machines. Technical Report SABLE-TR-2005-4, Sable Research Group, McGill University, Oct. 2005.]]
[16]
D. Gu, C. Verbrugge, and E. Gagnon. Code layout as a source of noise in JVM performance. Studia Informatica Universalis, 4(1):83--99, March 2005. ISBN2-912590-31-0.]]
[17]
S. Z. Guyer and K. S. McKinley. Finding your cronies: static analysis for dynamic object colocation. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 237--250, Oct. 2004. ISBN1-58113-831-9.]]
[18]
K. Hammond, G. L. Burn, and D. B. Howe. Spiking your caches. In J. T. O. Donnell and K. Hammond, editors, GLA, pages 58--68. Springer-Verlag, July 1993.]]
[19]
M. Hauswirth, P. F. Sweeney, A. Diwan, and M. Hind. Vertical profiling: understanding the behavior of object-priented applications. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 251--269, Oct. 2004. ISBN1-58113-831-9.]]
[20]
X. Huang, S. M. Blackburn, K. S. McKinley, J. E. B. Moss, Z. Wang, and P. Cheng. The garbage collection advantage: improving program locality. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 69--80, Oct. 2004. ISBN1-58113-831-9.]]
[21]
R. Jones and R. Lins. Garbage Collection: Algorithms for Automatic Dynamic Memory Management. John Wiley and Sons, Ltd, 1996.]]
[22]
J. Lau, J. Sampson, E. Perelman, G. Hamerly, and B. Calder. The strong correlation between code signatures and performance. In ISPASS '05: Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software, page 220. IEEE Computer Society, March 2005.]]
[23]
D. Project. The DaCapo benchmark suite (beta050224). http://www-ali.cs.umass.edu/DaCapo/index.html, 2003.]]
[24]
F. Qian and L. Hendren. An adaptive, region-based allocator for java. In ISMM '02: Proceedings of the 3rd international symposium on Memory management, pages 127--138, June 2002. ISBN1-58113-539-4.]]
[25]
R. M. Rabbah, H. Sandanagobalane, M. Ekpanyapong, and W.-F. Wong. Compiler orchestrated prefetching via speculation and predication. In ASPLOS-XI: Proceedings of the 11th international conference on Architectural support for programming languages and operating systems, pages 189--198, Oct. 2004. ISBN1-58113-804-0.]]
[26]
F. Schneider and T. R. Gross. Using platform-specific performance counters for dynamic compilation. In Proceedings of the 18th International Workshop on Languages and Compilers for Parallel Computing (LCPC'05), Oct 2005. to appear.]]
[27]
S. Soman, C. Krintz, and D. F. Bacon. Dynamic selection of application-specific garbage collectors. In ISMM '04: Proceedings of the 4th international symposium on Memory management, pages 49--60, Oct 2004. ISBN1-58113-945-4.]]
[28]
Standard Performance Evaluation Corporation. SPECjvm98 benchmarks. http://www.spec.org/osg/jvm98.]]
[29]
D. Stefanović, M. Hertz, S. M. Blackburn, K. S. McKinley, and J. E. B. Moss. Older-first garbage collection in practice: evaluation in a java virtual machine. In MSP '02: Proceedings of the 2002 workshop on Memory system performance, pages 25--36, June 2002.]]
[30]
D. Stefanović, K. S. McKinley, and J. E. B. Moss. Age-based garbage collection. In OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 370--381, Oct. 1999. ISBN1-58113-238-7.]]
[31]
P. F. Sweeney, M. Hauswirth, B. Cahoon, P. Cheng, A. Diwan, D. Grove, and M. Hind. Using hardware performance monitors to understand the behavior of Java applications. In VM'04:Proceedings of the 3rd Virtual Machine Research and Technology Symposium, May 2004.]]
[32]
D. Ungar. Generation scavenging: A non-disruptive high performance storage reclamation algorithm. In SDE 1: Proceedings of the first ACM SIGSOFT/SIGPLAN software engineering symposium on Practical software development environments, pages 157--167, Apr. 1984. ISBN0-89791-131-8.]]
[33]
X. Vera and J. Xue. Let's study whole program cache behavior analytical. In International Symposium on High-Performance Computer Architecture (HPCA 8) (IEEE), pages 175--186, Feb. 2002.]]

Cited By

View all
  • (2021)A Systematic Hardening of Java's Information HidingProceedings of the 2021 International Symposium on Advanced Security on Software and Systems10.1145/3457340.3458300(11-22)Online publication date: 7-Jun-2021
  • (2020)Profiling JVM for AI Applications Using Deep Learning LibrariesMachine Learning for Predictive Analysis10.1007/978-981-15-7106-0_39(395-404)Online publication date: 23-Oct-2020
  • (2018)A hardware accelerator for tracing garbage collectionProceedings of the 45th Annual International Symposium on Computer Architecture10.1109/ISCA.2018.00022(138-151)Online publication date: 2-Jun-2018
  • Show More Cited By

Index Terms

  1. Relative factors in performance analysis of Java virtual machines

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        VEE '06: Proceedings of the 2nd international conference on Virtual execution environments
        June 2006
        194 pages
        ISBN:1595933328
        DOI:10.1145/1134760
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 14 June 2006

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Java
        2. caches
        3. garbage collection
        4. hardware counters
        5. performance analysis

        Qualifiers

        • Article

        Conference

        VEE06

        Acceptance Rates

        Overall Acceptance Rate 80 of 235 submissions, 34%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)8
        • Downloads (Last 6 weeks)1
        Reflects downloads up to 17 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2021)A Systematic Hardening of Java's Information HidingProceedings of the 2021 International Symposium on Advanced Security on Software and Systems10.1145/3457340.3458300(11-22)Online publication date: 7-Jun-2021
        • (2020)Profiling JVM for AI Applications Using Deep Learning LibrariesMachine Learning for Predictive Analysis10.1007/978-981-15-7106-0_39(395-404)Online publication date: 23-Oct-2020
        • (2018)A hardware accelerator for tracing garbage collectionProceedings of the 45th Annual International Symposium on Computer Architecture10.1109/ISCA.2018.00022(138-151)Online publication date: 2-Jun-2018
        • (2018)Performance Assessment Of The Integration Between Industrial Agents And Low-Level Automation Functions2018 IEEE 16th International Conference on Industrial Informatics (INDIN)10.1109/INDIN.2018.8471927(121-126)Online publication date: Jul-2018
        • (2017)Hardening Java’s Access Control by Abolishing Implicit Privilege Elevation2017 IEEE Symposium on Security and Privacy (SP)10.1109/SP.2017.16(1027-1040)Online publication date: May-2017
        • (2015)DOs and DON'Ts of Conducting Performance Measurements in JavaProceedings of the 6th ACM/SPEC International Conference on Performance Engineering10.1145/2668930.2688820(337-340)Online publication date: 28-Jan-2015
        • (2015)Approaches to interpreter compositionComputer Languages, Systems and Structures10.1016/j.cl.2015.03.00144:PC(199-217)Online publication date: 1-Dec-2015
        • (2011)A comprehensive evaluation of object scanning techniquesACM SIGPLAN Notices10.1145/2076022.199348446:11(33-42)Online publication date: 4-Jun-2011
        • (2011)A comprehensive evaluation of object scanning techniquesProceedings of the international symposium on Memory management10.1145/1993478.1993484(33-42)Online publication date: 4-Jun-2011
        • (2009)VM performance evaluation with functional modelsProceedings of the Third Workshop on Virtual Machines and Intermediate Languages10.1145/1711506.1711512(1-2)Online publication date: 25-Oct-2009
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media