research-article

Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine

Authors:

Aibek Sarimbekov,

Walter BinderAuthors Info & Claims

OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications

Pages 657 - 676

https://doi.org/10.1145/2048066.2048118

Published: 22 October 2011 Publication History

Abstract

Originally conceived as the target platform for Java alone, the Java Virtual Machine (JVM) has since been targeted by other languages, one of which is Scala. This trend, however, is not yet reflected by the benchmark suites commonly used in JVM research. In this paper, we thus present the design and analysis of the first full-fledged benchmark suite for Scala. We furthermore compare the benchmarks contained therein with those from the well-known DaCapo 9.12 benchmark suite and show where the differences are between Scala and Java code---and where not.

References

[1]

B. Alpern, A. Cocchi, S. Fink, and D. Grove. Efficient implementation of Java interfaces: invoke interface considered harmless. In Proceedings of the 16th Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2001.

Digital Library

[2]

G. Ammons, T. Ball, and J. R. Larus. Exploiting hardware performance counters with flow and context sensitive profiling. In Proceedings of the 10th Conference on Programming Language Design and Implementation (PLDI), 1997.

Digital Library

[3]

M. Bebenita, M. Chang, G. Wagner, A. Gal, C. Wimmer, and M. Franz. Trace-based compilation in execution environments without interpreters. In Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2010.

Digital Library

[4]

W. Binder. A portable and customizable profiling framework for Java based on bytecode instruction counting. In Proceedings of the 3rd Asian Symposium on Programming Languages and Systems (APLAS), 2005.

Digital Library

[5]

W. Binder, J. Hulaas, and P. Moret. A quantitative evaluation of the contribution of native code to Java workloads. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC), 2006.

[6]

W. Binder, J. Hulaas, P. Moret, and A. Villazo ?n. Platform-independent profiling in a virtual execution environment. Software: Practice and Experience, 39(1):47--79, 2009.

Digital Library

[7]

S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J.E.B.Moss,B.Moss,A.Phansalkar,D.Stefanovic, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proceedings of the 21st Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2006.

Digital Library

[8]

E. Bodden, A. Sewe, J. Sinschek, M. Mezini, and H. Oueslati. Taming reflection: Aiding static analysis in the presence of reflection and custom class loaders. In Proceedings of the 33rd International Conference on Software Engineering (ICSE), 2011.

Digital Library

[9]

M. Braux and J. Noye. Towards partially evaluating reflection in Java. In Proceedings of the Workshop on Partial Evaluation and Semantics-Based Program Manipulation (PEPM), 1999.

Digital Library

[10]

J. M. Bull, L. A. Smith, M. D. Westhead, D. S. Henty, and R. A. Davey. A methodology for benchmarking Java Grande applications. In Proceedings of the ACM 1999 Conference on Java Grande, 1999.

Digital Library

[11]

Y. Chiba. Redundant boxing elimination by a dynamic compiler for Java. In Proceedings of the 5th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2007.

Digital Library

[12]

S. R. Chidamber and C. F. Kemerer. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6):476--493, June 1994. ISSN 0098-5589.

Digital Library

[13]

C. Collberg, G. Myles, and M. Stepp. An empirical study of Java bytecode programs. Software: Practice and Experience, 37(6):581--641, May 2007. ISSN 0038-0644.

Digital Library

[14]

C. Daly, J. Horgan, J. Power, and J. Waldron. Platform independent dynamic Java virtual machine analysis: the Java Grande Forum benchmark suite. In Proceedings of the 2001 joint ACM-ISCOPE Conference on Java Grande, 2001.

Digital Library

[15]

D. Detlefs and O. Agesen. Inlining of virtual methods. In Proceedings of the 13th European Conference on Object-oriented Programming (ECOOP), 1999.

Digital Library

[16]

S. Dieckmann and U. Holzle. A study of the allocation behavior of the SPECjvm98 java benchmarks. In Proceedings of the 13th European Conference on Object-oriented Programming (ECOOP), 1999.

Digital Library

[17]

I. Dragos and M. Odersky. Compiling generics through user-directed type specialization. In Proceedings of the 4th Workshop on the Implementation, Compilation, Optimization of Object-oriented Languages and Programming Systems (ICOOOLPS), 2009.

Digital Library

[18]

G. Dubochet and M. Odersky. Compiling structural types on the JVM: a comparison of reflective and generative techniques from Scalas perspective. In Proceedings of the 4th Workshop on the Implementation, Compilation, Optimization of Object-oriented Languages and Programming Systems (ICOOOLPS), 2009.

Digital Library

[19]

B. Dufour, K. Driesen, L. Hendren, and C. Verbrugge. Dynamic metrics for Java. In Proceedings of the 18th Conference on Object-oriented Programing, Systems, Languages, and Applications (OOPSLA), 2003.

Digital Library

[20]

Common Language Infrastructure (CLI). ECMA International, 5th edition, December 2010.

[21]

L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In Proceedings of the 18th Conference on Object-oriented Programing, Systems, Languages, and Applications (OOPSLA), 2003.

Digital Library

[22]

D. Gregg, J. Power, and J. Waldron. A method-level comparison of the Java Grande and SPEC JVM98 benchmark suites. Concurrency and Computation: Practice and Experience, 17: 757--773, 2005.

Digital Library

[23]

N. M. Hanish and W. E. Cohen. Hardware support for profiling Java programs. In Proceedings of the Workshop on Hardware Support for Objects and Microarchitectures for Java, 1999.

[24]

R. E. Hank, W.-M. W. Hwu, and B. R. Rau. Region-based compilation: an introduction and motivation. In Proceedings of the 28th International Symposium on Microarchitecture (MICRO), 1995.

Digital Library

[25]

U. Holzle, C. Chambers, and D. Ungar. Optimizing dynamically-typed object-oriented languages with polymorphic inline caches. In Proceedings of the 5th European Conference on Object-oriented Programming (ECOOP), 1991.

Digital Library

[26]

K. Hoste and L. Eeckhout. Microarchitecture-independent workload characterization. IEEE Micro, 27:63--72, 2007.

Digital Library

[27]

I. Jibaja, S. Blackburn, M. Haghighat, and K. McKinley. Deferred gratification: Engineering for high performance garbage collection from the get go. In Proceedings of the Workshop on Memory Systems Performance and Correctness (MSPC), 2011.

Digital Library

[28]

R. E. Jones and C. Ryder. A study of Java object demographics. In Proceedings of the 7th International Symposium on Memory Management (ISMM), 2008.

Digital Library

[29]

R. K. Karmani, A. Shali, and G. Agha. Actor frameworks for the JVM platform: a comparative analysis. In Proceedings of the 7th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2009.

Digital Library

[30]

G. Lashari and S. Srinivas. Characterizing Java application performance. In Proceedings of the 17th International Parallel and Distributed Processing Symposium (IPDPS), 2003.

Digital Library

[31]

T. Lindholm and F. Yellin. The Java Virtual Machine Specification. Addison-Wesley, 2nd edition, 1999.

Digital Library

[32]

A. McCallum, K. Schultz, and S. Singh. FACTORIE: Probabilistic programming via imperatively defined factor graphs. Advances on Neural Information Processing Systems, 2009.

[33]

P. Moret, W. Binder, and A. Villazon. CCCP: Complete calling context profiling in virtual execution environments. In Proceedings of the Workshop on Partial Evaluation and Program Manipulation (PEPM), 2009.

Digital Library

[34]

M. Odersky, L. Spoon, and B. Venners. Programming in Scala. Artima Press, 2nd edition, 2010.

[35]

K. Ogata, D. Mikurube, K. Kawachiya, S. Trent, and T. Onodera. A study of Javas non-Java memory. In Proceedings of the 25th Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2010.

Digital Library

[36]

K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2:559--572, 1901.

[37]

D. Ramage, E. Rosen, J. Chuang, C. D. Manning, and D. A. McFarland. Topic modeling for the social sciences. In NIPS Workshop on Applications for Topic Models: Text and Beyond, 2009.

[38]

P. Ratanaworabhan, B. Livshits, and B. G. Zorn. JSMeter: comparing the behavior of JavaScript benchmarks with real Web applications. In Proceedings of the 2010 USENIX Conference on Web Application Development, 2010.

Digital Library

[39]

G. Richards, S. Lebresne, B. Burg, and J. Vitek. An analysis of the dynamic behavior of JavaScript programs. In Proceedings of the Conference on Programming Language Design and Implementation (PLDI), 2010.

Digital Library

[40]

I. Rogers, J. Zhao, and I. Watson. Approaches to reflective method invocation. In Proceedings of the 3rd Workshop on Implementation, Compilation, Optimization of Object-oriented Languages, Programs and Systems (ICOOOLPS), 2008.

[41]

J. R. Rose. Bytecodes meet combinators: invokedynamic on the JVM. In Proceedings of the 3rd Workshop on Virtual Machines and Intermediate Languages (VMIL), 2009.

Digital Library

[42]

A. Sarimbekov, P. Moret, W. Binder, A. Sewe, and M. Mezini. Complete and platform-independent calling context profiling for the Java virtual machine. In Proceedings of the 6th Workshop on Bytecode Semantics, Verification, Analysis and Transformation (BYTECODE), 2011.

Digital Library

[43]

A. Sarimbekov, A. Sewe, W. Binder, P. Moret, M. Schoberl, and M. Mezini. Portable and accurate collection of calling-context-sensitive bytecode metrics for the java virtual machine. In Proceedings of the 9th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2011.

Digital Library

[44]

M. Schinz. Compiling Scala for the Java Virtual Machine. PhD thesis, EPFL, Switzerland, September 2005.

[45]

A. Sewe. Scala ð Java mod JVM. In Proceedings of the Work-in-Progress Session at the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), volume 692 of CEUR Workshop Proceedings, 2010.

[46]

K. Shiv, K. Chow, Y. Wang, and D. Petrochenko. SPECjvm2008 performance characterization. In Proceedings of the SPEC Benchmark Workshop, 2009.

Digital Library

[47]

A. M. Sloane. Experiences with domain-specific language embedding in Scala. In Proceedings of the 2nd International Workshop on Domain-Specific Program Development (DSPD), 2008.

[48]

T. Suganuma, T. Yasue, and T. Nakatani. A region-based compilation technique for a Java just-in-time compiler. In Proceedings of the 16th Conference on Programming Language Design and Implementation (PLDI), 2003.

Digital Library

[49]

C. Thalinger and J. Rose. Optimizing invokedynamic. In Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2010.

Digital Library

Cited By

Milikic LCugurovic MJovanovic VDoerfert JGrosser TLeather HSadayappan P(2025)GraalNN: Context-Sensitive Static Profiling with Graph Neural NetworksProceedings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimization10.1145/3696443.3708958(123-136)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1145/3696443.3708958
Blackburn SCai ZChen RYang XZhang JZigman JEeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Rethinking Java Performance AnalysisProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707217(940-954)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707217
Bouzenia IKrishan BPradel M(2024)DyPyBench: A Benchmark of Executable Python SoftwareProceedings of the ACM on Software Engineering10.1145/36437421:FSE(338-358)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643742
Show More Cited By

Index Terms

Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine

Recommendations

A Comprehensive Java Benchmark Study on Memory and Garbage Collection Behavior of DaCapo, DaCapo Scala, and SPECjvm2008
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Benchmark suites are an indispensable part of scientific research to compare different approaches against each another. The diversity of benchmarks is an important asset to evaluate novel approaches for effectiveness and weaknesses. In this paper, we ...
Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine
OOPSLA '11

Originally conceived as the target platform for Java alone, the Java Virtual Machine (JVM) has since been targeted by other languages, one of which is Scala. This trend, however, is not yet reflected by the benchmark suites commonly used in JVM ...
Performance and Energy-Efficiency of Scala on Mobile Devices
NGMAST '13: Proceedings of the 2013 Seventh International Conference on Next Generation Mobile Apps, Services and Technologies

The objective of this study was to compare how new languages like Scala perform on mobile environments compared to the classical languages. Since Scala also runs on the Java Virtual Machine (JVM), it is possible to run the code on Android devices and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications

October 2011

1104 pages

ISBN:9781450309400

DOI:10.1145/2048066

General Chair:
Cristina Videira Lopes
University of California, Irvine, USA
,
Program Chair:
Kathleen Fisher
Tufts University, USA

ACM SIGPLAN Notices Volume 46, Issue 10
OOPSLA '11
October 2011
1063 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2076021
Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SPLASH '11

Sponsor:

SIGPLAN

SPLASH '11: Conference on Systems, Programming, and Applications: Software for Humanity

October 22 - 27, 2011

Oregon, Portland, USA

Acceptance Rates

Overall Acceptance Rate 243 of 1,119 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

96
Total Citations
View Citations
705
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)5

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Milikic LCugurovic MJovanovic VDoerfert JGrosser TLeather HSadayappan P(2025)GraalNN: Context-Sensitive Static Profiling with Graph Neural NetworksProceedings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimization10.1145/3696443.3708958(123-136)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1145/3696443.3708958
Blackburn SCai ZChen RYang XZhang JZigman JEeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Rethinking Java Performance AnalysisProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707217(940-954)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707217
Bouzenia IKrishan BPradel M(2024)DyPyBench: A Benchmark of Executable Python SoftwareProceedings of the ACM on Software Engineering10.1145/36437421:FSE(338-358)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643742
Löff JSchiavio FRosà ABasso MBinder WBalsamo SKnottenbelt WAbad CShang W(2024)Vectorized Intrinsics Can Be Replaced with Pure Java Code without Impairing Steady-State PerformanceProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645051(14-24)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645051
Kaur HPirvu MKent K(2024)An Improved Template-Based JIT Compiler for Java2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON)10.1109/CASCON62161.2024.10838016(1-10)Online publication date: 11-Nov-2024
https://doi.org/10.1109/CASCON62161.2024.10838016
Čugurović MVujošević Janičić MJovanović VWürthinger T(2024)GraalSP: Polyglot, efficient, and robust machine learning-based static profilerJournal of Systems and Software10.1016/j.jss.2024.112058213(112058)Online publication date: Jul-2024
https://doi.org/10.1016/j.jss.2024.112058
Greenman B(2023)GTP Benchmarks for Gradual Typing PerformanceProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600034(102-114)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3589806.3600034
Bulej LHorký VTucci MTuma PFarquet FLeopoldseder DProkopec AVieira MCardellini VDi Marco ATuma P(2023)GraalVM Compiler Benchmark Results Dataset (Data Artifact)Companion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3585025(65-69)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3585025
Tavakolisomeh SBruno RFerreira P(2023)BestGC: An Automatic GC SelectorIEEE Access10.1109/ACCESS.2023.329439811(72357-72373)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3294398
Weingarten MTheodoridis TProkopec AKotselidis CProkopec A(2022)Inlining-Benefit Prediction with Interprocedural Partial Escape AnalysisProceedings of the 14th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages10.1145/3563838.3567677(13-24)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563838.3567677
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten