Skip to main content
SpringerLink
Log in

Using Dynamic Runtime Testing for Rapid Development of Architectural Simulators

  • Published:
International Journal of Parallel Programming Aims and scope Submit manuscript

Abstract

Architectural simulator platforms are particularly complex and error-prone programs that aim to simulate all hardware details of a given target architecture. Development of a stable cycle-accurate architectural simulator can easily take several man-years. Discovering and fixing all visible errors in a simulator often requires significant effort, much higher than for writing the simulator code in the first place. In addition, there are no guarantees that all programming errors will be eliminated, no matter how much effort is put into testing and debugging. This paper presents dynamic runtime testing, a methodology for rapid development and accurate detection of functional bugs in architectural cycle-accurate simulators. Dynamic runtime testing consists of comparing an execution of a cycle-accurate simulator with an execution of a simple and functionally equivalent emulator. Dynamic runtime testing detects a possible functional error if there is a mismatch between the execution in the simulator and the emulator. Dynamic runtime testing provides a reliable and accurate verification of a simulator, during its entire development cycle, with very acceptable performance impact, and without requiring complex setup for the simulator execution. Based on our experience, dynamic testing reduced the simulator modification time from 12–18 person-months to only 3–4 person-months, while it only modestly reduced the simulator performance (in our case under 20 %).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Becker, D., Singh, R.K., Tell, S.G.: Readings in hardware/software co-design. In: An Engineering Environment for Hardware/Software Co-simulation, pp. 550–555. Kluwer, Norwell (2002). http://dl.acm.org/citation.cfm?id=567003.567052

  2. Bellard, F.: Qemu, a fast and portable dynamic translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005)

  3. Binkert N., Dreslinski R., Hsu L., Lim K., Saidi A., Reinhardt S.: The M5 simulator: Modeling networked systems. IEEE Micro 26(4), 52–60 (2006). doi:10.1109/MM.2006.82

    Article  Google Scholar 

  4. Cao Minh, C., Chung, J., Kozyrakis, C., Olukotun, K.: STAMP: Stanford transactional applications for multi-processing. In: IISWC’08: Proceedings of the IEEE International Symposium on Workload Characterization (2008). doi:10.1109/IISWC.2008.4636089

  5. Hammond L., Carlstrom B.D., Wong V., Chen M., Kozyrakis C., Olukotun K.: Transactional coherence and consistency: simplifying parallel hardware and software. IEEE Micro 24(6), 92–103 (2004). doi:10.1109/MM.2004.91

    Article  Google Scholar 

  6. Harris T., Larus J., Rajwar R.: Transactional Memory, 2nd edn. Morgan and Claypool Publishers, San Francisco (2010)

    Google Scholar 

  7. Harrold, M.J.: Testing: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering, ICSE’00, pp. 61–72. ACM, New York (2000). http://doi.acm.org/10.1145/336512.336532

  8. Hoare C.: An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)

    Article  MATH  Google Scholar 

  9. Jindal, R., Jain, K.: Verification of transaction-level systemc models using rtl testbenches. In: 1st ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2003 (MEMOCODE’03), Proceedings, pp. 199–203. IEEE (2003)

  10. Kaner C., Bach J., Pettichord B.: Lessons Learned in Software Testing: A Context-Driven Approach. Wiley, New York (2002)

    Google Scholar 

  11. Moore, K.E., Bobba, J., Moravan, M.J., Hill, M.D., Wood, D.A.: LogTM: Log-based transactional memory. In: In proceedings of the HPCA-12, pp. 254–265 (2006)

  12. Runeson, P.: A survey of unit testing practices. IEEE Softw. 23, 22–29 (2006). doi:10.1109/MS.2006.91. http://portal.acm.org/citation.cfm?id=1159169.1159387

    Google Scholar 

  13. Stallman, R., Pesch, R.: The gnu source-level debugger. User Manual, Edition 4.12, for GDB version 4

  14. Vouk M.: Back-to-back testing. Inf. Softw. Technol. 32(1), 34–45 (1990)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Barcelona Supercomputing Center (BSC), Barcelona, Spain

    Saša Tomić, Adrián Cristal, Osman Unsal & Mateo Valero

  2. Artificial Intelligence Research Institute (IIIA), National Research Council (CSIC), Barcelona, Spain

    Adrián Cristal

  3. Universitat Politècnica de Catalunya (UPC), Barcelona, Spain

    Saša Tomić & Mateo Valero

Authors
  1. Saša Tomić
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrián Cristal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Osman Unsal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mateo Valero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saša Tomić.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tomić, S., Cristal, A., Unsal, O. et al. Using Dynamic Runtime Testing for Rapid Development of Architectural Simulators. Int J Parallel Prog 42, 119–139 (2014). https://doi.org/10.1007/s10766-012-0208-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10766-012-0208-7

Keywords

Search

Navigation