skip to main content
research-article

A Roadmap and Plan of Action for Community-Supported Empirical Evaluation in Computer Architecture

Published: 20 January 2015 Publication History

Abstract

A framework of open interoperable simulators for computer architecture is long overdue. Today there are many separate, uncoordinated efforts to develop simulation and modeling artifacts (tools) for computer architecture research. The artifacts are used to empirically evaluate new computer architecture innovations and compare them with the state of the art. The artifacts are usually developed by individual groups, often for a specific purpose, and may not be publicly released. Consequently, it is difficult to leverage investment in artifact development and to repeat or reproduce experiments. In this position paper, we present recommendations and a roadmap for sharing and building open-source, interoperable simulation and modeling artifacts. The recommendations are the outcome of a community workshop involving industry, government and academia to determine how to coordinate effort, share tools and improve methodology.

References

[1]
T. M. Aaamodt and A. Boktor. GPGPU-Sim 3.x: A performance simulator for many-core accelerator research. International Symposium on Computer Architecture (ISCA), http://www.gpgpu-sim.org/isca2012-tutorial/, 2012. {Online; accessed 30-Sep-2012}.
[2]
T. M. Aaamodt andW.W. Fung. GPGPU-Sim 3.x: A performance simulator for manycore accelerator research. International Conference on Parallel Architectures and Compilation Techniques (PACT), http://www.gpgpu-sim.org/pact2012-tutorial/, 2012. {Online; accessed 30-Sep-2012}.
[3]
ArtistDesign NoE -- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http://www.artist-embedded.org/, 2012. {Online; accessed 21-Sep-2012}.
[4]
A. Bakhoda, G. L. Yuan, W. W. L. Fung, H. Wong, and T. M. Aamodt. Analyzing cuda workloads using a detailed GPU simulator. In 2012 IEEE International Symposium on Performance Analysis of Systems & Software (ISPASS), pages 163--174, 2009.
[5]
B. Beckmann, N. Binkert, A. Saidi, J. Hestness, G. Black, K. Sewell, and D. Hower. gem5: A multiple-ISA full system simulator with detailed memory modeling. Tutorial at the International Symposium on Computer Architecture (ISCA),http://www.m5sim.org/. {Online; accessed 12-Jul-2012}.
[6]
F. Bellard. QEMU, a fast and portable dynamic translator. In Proceedings of the annual conference on USENIX Annual Technical Conference, ATEC '05, pages 41--41, Berkeley, CA, USA, 2005. USENIX Association.
[7]
N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood. The gem5 simulator. SIGARCH Comput. Archit. News, 39(2):1-7, Aug. 2011.
[8]
R. Campbell, I. Gupta, M. Heath, S. Y. Ko, M. Kozuch, M. Kunze, T. Kwan, K. Lai, H. Y. Lee, M. Lyons, D. Milojicic, D. O'Hallaron, and Y. C. Soh. Open Cirrus(TM) cloud computing testbed: federated data centers for open source systems and services research. In Proceedings of the 2009 conference on Hot topics in cloud computing, HotCloud'09, Berkeley, CA, USA, 2009. USENIX Association.
[9]
J. Cappos, I. Beschastnikh, A. Krishnamurthy, and T. Anderson. Seattle: a platform for educational cloud computing. In Proceedings of the 40th ACM technical symposium on Computer science education, SIGCSE'09, pages 111--115, New York, NY, USA, 2009. ACM.
[10]
T. E. Carlson, W. Heirman, and L. Eeckhout. Sniper: Exploring the level of abstraction for scalable and accurate parallel multi-core simulations. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Nov. 2011.
[11]
Chameleon. http://www.chameleoncloud.org/, 2014. {Online; accessed 13-Nov-2014}.
[12]
D. Chiou, D. Sunwoo, J. Kim, N. A. Patil, W. Reinhart, D. E. Johnson, J. Keefe, and H. Angepat. Fpga-accelerated simulation technologies (fast): Fast, full-system, cycle-accurate simulators. In Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 40, pages 249--261, Washington, DC, USA, 2007. IEEE Computer Society.
[13]
B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak, and M. Bowman. Planetlab: an overlay testbed for broad-coverage services. SIGCOMM Comput. Commun. Rev., 33(3):3--12, July 2003.
[14]
E. S. Chung, E. Nurvitadhi, J. C. Hoe, B. Falsafi, and K. Mai. A complexity-effective architecture for accelerating full-system multiprocessor simulations using fpgas. In Proceedings of the 16th international ACM/SIGDA symposium on Field programmable gate,marrays, FPGA '08, pages 77--86, New York, NY, USA, 2008. ACM.
[15]
CloudLab. http://www.cloudlab.us/, 2014. {Online; accessed 13-Nov-2014}.
[16]
CMU Artificial Intelligence Repository. http://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/0.html, 2012. {Online; accessed 02-Jan-2012}.
[17]
L. Eeckhout, W. Heirman, T. E. Carlson, and I. Hur. Sniper: Fast, accurate and scalable multicore simulation. Tutorial at the International Symposium on Computer Architecture (ISCA),//http://snipersim.org/w/Tutorial:ISCA_2012. {Online; accessed 30-Sep-2012}.
[18]
Emulab --- total network testbed. http://www.emulab.net, 2012. {Online; accessed 18-Sep-2012}.
[19]
FPGA-Accelerator Research Infrastructure Cloud (FAbRIC). http://users.ece.utexas.edu/~derek/FABRIC.html. {Online; accessed 29-Sep-2012}.
[20]
The gem5 simulator system. http://www.gem5.org/. {Online; accessed 12-Jul-2012}.
[21]
D. Genbrugge, S. Eyerman, and L. Eeckhout. Interval simulation: Raising the level of abstraction in architectural simulation. In 16th International Conference on High-Performance Computer Architecture (HPCA), pages 1--12, 2010.
[22]
K. Ghose, A. Patel, F. Afram, H. Zheng, J. Tringali, and J. Greensky. Marssx86 tutorial. Tutorial at Conference on Parallel Architecture and Compilation Techniques (PACT), http://marss86.org/~marss86/index.php/PACT_Tutorial_2012. {Online; accessed24-Sep-2012}.
[23]
M. Hibler, R. Ricci, L. Stoller, J. Duerig, S. Guruprasad, T. Stack, K. Webb, and J. Lepreau. Large-scale virtualization in the emulab network testbed. In USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC'08, pages 113--128, Berkeley, CA, USA, 2008. USENIX Association.
[24]
HiPEAC Compilation Architecture --- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http://www.hipeac.net/, 2012. {Online; accessed 27-Sep-2012}.
[25]
Intel Many-core Applications Research Community (MARC). http://communities.intel.com/community/marc, 2012. {Online; accessed 17-Mar-2012}.
[26]
C. L. Janssen, H. Adalsteinsson, and J. P. Kenny. Using simulation to design extremescale applications and architectures: programming model exploration. SIGMETRICS Perform. Eval. Rev., 38(4):4--8, Mar. 2011.
[27]
A. Kerr, G. Diamos, and S. Yalamanchili. A characterization and analysis of PTX kernels. In International Symposium on Workload Characterization (IISWC), 2009.
[28]
A. Kerr, G. Diamos, and S. Yalamanchili. GPU application development, debugging, and performance tuning with GPU Ocelot. In Wen-mei Hwu, editor, GPU Computing GEMS, vol. 2. Morgan Kaufmann, 2011.
[29]
H. Kim, S. Yalamanchili, J. Lee, N. Lakshminarayana, A. Kerr, A. Rodrigues, and G. Hsieh. Tutorial on Ocelot and SST-MacSim Simulator. International Symposium on Computer Architecture, http://comparch.gatech.edu/hparch/isca12_gt.html, 2012. {Online; accessed 15-Sep-2012}.
[30]
H. Kim, S. Yalamanchili, J. Lee, N. Lakshminarayana, A. Kerr, A. Rodrigues, and G. Hsieh. Tutorial on Ocelot and SST-MacSim Simulator. Conference on High-Performance Computer Architecture, http://comparch.gatech.edu/hparch/OcelotMacSim_tutorial.html, 2012. {Online; accessed 15-Sep-2012}.
[31]
P. S. Magnusson, M. Christensson, J. Eskilson, D. Forsgren, G. H?allberg, J. Högberg, F. Larsson, A. Moestedt, and B. Werner. Simics: A full system simulation platform. Computer, 35(2):50--58, Feb. 2002.
[32]
Manifold parallel discrete event simulation. http://manifold.gatech.edu. {Online; accessed 29-Sep-2012}.
[33]
J. Miller, H. Kasture, G. Kurian, C. Gruenwald, N. Beckmann, C. Celio, J. Eastep, and A. Agarwal. Graphite: A distributed parallel simulator for multicores. In High Performance Computer Architecture (HPCA), 2010 IEEE 16th International Symposium on, pages 1--12, jan. 2010.
[34]
nanoHUB.org -- Online Simulation and More for Nanotechnology. nanohub.org. {Online; accessed 20-Sep-2012}.
[35]
OpenCirrus(TM) the HP/Intel/Yahoo! Open Cloud Computing Research Testbed. http://opencirrus.org, 2012. {Online; accessed 15-Sep-2012}.
[36]
M. Pellauer, M. Vijayaraghavan, M. Adler, Arvind, and J. Emer. A-ports: an efficient abstraction for cycle-accurate performance models on fpgas. In Proceedings of the 16th international ACM/SIGDA symposium on Field programmable gate arrays, FPGA'08, pages 87--96, New York, NY, USA, 2008. ACM.
[37]
D. A. Penry, D. Fay, D. Hodgdon, R. Wells, G. Schelle, D. I. August, and D. Connors. Exploiting parallelism and structure to accelerate the simulation of chip multi-processors. In 12th International Symposium on High-Performance Computer Architecture HPCA, pages 29--40, 2006.
[38]
L. Peterson, A. Bavier, M. E. Fiuczynski, and S. Muir. Experiences building planetlab. In Proceedings of the 7th symposium on Operating systems design and implementation, OSDI '06, pages 351--366, Berkeley, CA, USA, 2006. USENIX Association.
[39]
PlanetLab -- An Open Platform for Developing, Deploying and Accessing Planetary-Scale Services. http://www.planet-lab.org/, 2012. {Online; accessed 27-Sep-2012}.
[40]
P. Ren, M. Lis, M. H. Cho, K. S. Shim, C. Fletcher, O. Khan, N. Zheng, and S. Devadas. Hornet: A cycle-level multicore simulator. Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions on, 31(6):890--903, June 2012.
[41]
A. F. Rodrigues, K. S. Hemmert, B. W. Barrett, C. Kersey, R. Oldfield, M. Weston, R. Risen, J. Cook, P. Rosenfeld, E. CooperBalls, and B. Jacob. The structural simulation toolkit (SST). SIGMETRICS Perform. Eval. Rev., 38(4):37--42, Mar. 2011.
[42]
Seattle -- Open Peer-to-Peer Computing. http://seattle.cs.washington.edu, 2012. {Online; accessed 18-Sep-2012}.
[43]
SST: The Structural Simulation Toolkit. http://sst.sandia.gov, 2012. {Online; accessed 12-Aug-2012}.
[44]
R. Ubal, B. Jang, P. Mistry, D. Schaa, and D. Kaeli. Multi2Sim: A Simulation Framework for CPU-GPU Computing . In Proc. of the 21st International Conference on Parallel Architectures and Compilation Techniques, Sep. 2012.
[45]
XSEDE: Extreme Science and Engineering Discovery Environment. http://www.xsede.org, 2012. {Online; accessed 15-Sep-2012}.

Cited By

View all
  • (2021)The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction researchScientific Data10.1038/s41597-021-00839-58:1Online publication date: 11-Feb-2021
  • (2016)VirtualSoCACM Transactions on Embedded Computing Systems10.1145/293066516:1(1-27)Online publication date: 13-Oct-2016

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGOPS Operating Systems Review
ACM SIGOPS Operating Systems Review  Volume 49, Issue 1
Special Issue on Repeatability and Sharing of Experimental Artifacts
January 2015
155 pages
ISSN:0163-5980
DOI:10.1145/2723872
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 January 2015
Published in SIGOPS Volume 49, Issue 1

Check for updates

Author Tags

  1. Computer Architecture
  2. Repeatability
  3. Repository

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2021)The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction researchScientific Data10.1038/s41597-021-00839-58:1Online publication date: 11-Feb-2021
  • (2016)VirtualSoCACM Transactions on Embedded Computing Systems10.1145/293066516:1(1-27)Online publication date: 13-Oct-2016

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media