Elsevier

Advances in Computers

Volume 41, 1995, Pages 231-253
Advances in Computers

Advances in Benchmarking Techniques: New Standards and Quantitative Metrics

https://doi.org/10.1016/S0065-2458(08)60235-1Get rights and content

Abstract

Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.

References (27)

  • A measure of transaction processing power.

    Datamation

    (1985)
  • B. Case

    Updated SPEC benchmarks released.

    Microprocessor Report

    (1992)
  • W.Y. Chen et al.

    The effect of code expanding optimizations on instruction cache design.

    IEEE Trans on Comput.

    (1993)
  • Conte, T. M. (1992). Systematic computer architecture prototyping. Ph.D. Thesis, Department of Electrical and Computer...
  • T.M. Conte et al.

    Benchmark characterization.

    IEEE Comput.

    (1991)
  • T.M. Conte et al.

    Determining cost-effective multiple issue processor designs.

    “Proceedings of the International Conference on Computer Design, Cambridge, MA.”

    (1993)
  • T.M. Conte et al.

    The effects of traditional compiler optimizations on superscalar architectural design.

  • H.J. Cumow et al.

    A synthetic benchmark.

    Comput. J.

    (1976)
  • G. Cybenko et al.

    Supercomputer performance evaluation and the Perfect Club.

  • K.M. Dixit

    CINT92 and CFP92 benchmark descriptions.

    SPEC Newsl.

    (1991)
  • Golden, M. L. (1991). Issues in trace collection through program instrumentation. Master's Thesis, Department of...
  • W.W. Hwu et al.

    Achieving high instruction cache performance with an optimizing compiler.

  • J. Keatts et al.

    Matrix300 performance concerns.

    SPEC Newsl.

    (1991)
  • Cited by (0)

    View full text