Varun Gohil
ABSTRACT
Performance evaluation is an integral part of computer architecture research. Rigorous performance evaluation is crucial in order to evaluate novel architectures, and is often carried out using benchmark suites. Each suite has a number of workloads with varying behavior and characteristics. Most analysis is done by analyzing the novel architecture across all workloads of a single benchmark suite. However, computer architects studying optimizations of specific microarchitectural components, require evaluation of their proposals on workloads that stress the component being optimized across multiple benchmark suites.
In this paper, we present the design and implementation of FAB - a framework built with Pin and Python based workflow. FAB allows user-driven analysis of benchmarks across multiple axes like instruction distributions, types of instructions etc. through an interactive Python interface to check for desired characteristics, across multiple benchmark suites. FAB aims to provide a toolkit that would allow computer architects to 1) select workloads with desired, user-specified behavior, and 2) create synthetic workloads with desired behavior that have a grounding in real benchmarks.
- {n. d.}. OpenBLAS: An optimized BLAS library. http://www.openblas.net/Google Scholar
- {n. d.}. SPEC CPU 2017 Documentation. https://www.spec.org/cpu2017/Google Scholar
- 2016. Intel 64 and IA-32 Architectures Software Developers Manual Volume 2A:Instruction Set Reference, A-L.Google Scholar
- D. H. Bailey et almbox. 1991. The NAS Parallel Benchmarks & Mdash;Summary and Preliminary Results. In Proceedings of Supercomputing. Google ScholarDigital Library
- Christian Bienia et almbox. 2008a. The PARSEC Benchmark Suite: Characterization and Architectural Implications. Technical Report TR-811-08. Princeton University.Google Scholar
- C. Bienia et almbox. 2008b. PARSEC vs. SPLASH-2: A quantitative comparison of two multithreaded benchmark suites on Chip-Multiprocessors. In Proceedings of IISWC.Google ScholarCross Ref
- Jack J. Dongarra et almbox. 2002. The LINPACK benchmark: Past, present, and future.Google Scholar
- Lieven Eeckhout. 2010. Computer Architecture Performance Evaluation Methods 1st ed.). Morgan & Claypool Publishers. Google ScholarDigital Library
- L. Eeckhout et almbox. 2000. Performance analysis through synthetic trace generation. In Proceedings of ISPASS. Google ScholarDigital Library
- C. Sakalis et al. 2016. Splash-3: A properly synchronized benchmark suite for contemporary research.Google Scholar
- Chi-Keung Luk et almbox. 2005. Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation. In Proceedings PLDI. Google ScholarDigital Library
- John D. McCalpin. 1995. Memory Bandwidth and Machine Balance in Current High Performance Computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter (Dec. 1995), 19--25.Google Scholar
- D. A. Menasce. 2002. TPC-W: a benchmark for e-commerce. IEEE Internet Computing, Vol. 6, 3 (2002). Google ScholarDigital Library
- R. Panda et almbox. 2018. Wait of a Decade: Did SPEC CPU 2017 Broaden the Performance Horizon?. In Proceedings of HPCA.Google ScholarCross Ref
- Timothy Sherwood et almbox. 2002. Automatically Characterizing Large Scale Program Behavior. SIGOPS Oper. Syst. Rev., Vol. 36, 5 (2002). Google ScholarDigital Library
Index Terms
- FAB: Framework for Analyzing Benchmarks
Recommendations
A Preliminary Workload Analysis of SPECjvm2008
ICCET '09: Proceedings of the 2009 International Conference on Computer Engineering and Technology - Volume 02SPECjvm2008 is a new benchmark program suite for measuring client-side Java runtime environment. It replaces JVM98, which has been used for the same purpose for more than ten years. It consists of 38 benchmark programs grouped into eleven categories and ...
A Comparative Study of JVM Implementations with SPECjvm2008
ICCEA '10: Proceedings of the 2010 Second International Conference on Computer Engineering and Applications - Volume 01SPECjvm2008 is a new benchmark program suite for measuring client-side Java runtime environments. It replaces JVM98, which has been used for the same purpose for more than ten years. It consists of 38 benchmark programs grouped into eleven categories ...
Selecting representative benchmark inputs for exploring microprocessor design spaces
The design process of a microprocessor requires representative workloads to steer the search process toward an optimum design point for the target application domain. However, considering a broad set of workloads to cover the large space of potential ...
Comments