Skip to main content
SpringerLink
Log in

Automatic performance prediction of multithreaded programs: a simulation approach

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

The performance of multithreaded programs is often difficult to understand and predict. Multiple threads engage in synchronization operations and use hardware simultaneously. This results in a complex non-linear dependency between the configuration of a program and its performance. To better understand this dependency a performance prediction model is used. Such a model predicts the performance of a system for different configurations. Configurations reflect variations in the workload, different program options such as the number of threads, and characteristics of the hardware. Performance models are complex and require a solid understanding of the program’s behavior. As a result, building models of large applications manually is extremely time-consuming and error-prone. In this paper we present an approach for building performance models of multithreaded programs automatically. We employ hierarchical discrete-event models. Different tiers of the model simulate different factors that affect performance of the program, while interaction between the model tiers simulates mutual influence of these factors on performance. Our framework uses a combination of static and dynamic analyses of a single representative run of a system to collect information required for building the performance model. This includes information about the structure of the program, the semantics of interaction between the program’s threads, and resource demands of individual program’s components. In our experiments we demonstrate that models accurately predict the performance of various multithreaded programs, including complex industrial applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Notes

  1. http://tomcat.apache.org/.

  2. http://sunflow.sourceforge.net/.

  3. http://docs.oracle.com/javase/6/docs/technotes/guides/concurrency/overview.html.

  4. http://linux.die.net/man/8/btrace.

  5. https://sourceforge.net/projects/persik/.

  6. http://www.omnetpp.org/.

  7. http://asm.ow2.org/.

  8. http://tomcat.apache.org/.

  9. http://itextpdf.com/.

  10. http://www.inet.omnetpp.org/.

  11. http://www.nsnam.org/.

References

  • Akdere, M., etintemel, U., Riondato, M., Upfal, E., Zdonik, S.B.: Learning-based query performance modeling and prediction. In: Kementsietsidis, A., Salles, M.A.V. (eds.) ICDE, pp. 390–401. IEEE (2012)

  • Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the April 18–20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring), pp. 483–485, New York, NY. ACM (1967)

  • Anderson, E.: Hpl–ssp–2001–4: simple table-based modeling of storage devices (2001)

  • Barham, P., Donnelly, A., Isaacs, R., Mortier, R.: Using magpie for request extraction and workload modelling. In: Proceedings of the Symposium on Opearting Systems Design and Implementation, pp. 18–18, Berkeley, CA, USA. USENIX Association (2004)

  • Barnes, J., Hut, P.: A hierarchical o(n log n) force-calculation algorithm. Nature 324, 446–449 (1986)

    Article  Google Scholar 

  • Bause, F.: Queueing petri nets—a formalism for the combined qualitative and quantitative analysis of systems. In: Proceedings of the 5th International Workshop on Petri nets and Performance Models. IEEE, pp. 14–23. IEEE (1993)

  • Bennani, M., Menasce, D.: Resource allocation for autonomic data centers using analytic performance models. In: Proceedings of International Conference on Automatic Computing, pp. 229–240, Washington, DC, USA. IEEE (2005)

  • Brosig, F., Huber, N., Kounev, S.: Automated extraction of architecture-level performance models of distributed component-based systems. In: Proceedings of International Conference on Automated Software Engineering, ASE ’11, pp. 183–192, Washington, DC, USA. IEEE (2011)

  • Brosig, F., Kounev, S., Krogmann, K.: Automated extraction of palladio component models from running enterprise java applications. In: Proceedings of the 1st International Workshop on Run-Time Models for Self-managing Systems and Applications, ROSSA’09. ACM, New York, NY, USA (2009)

  • Brünink, M., Rosenblum, D.S.: Mining performance specifications. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pp. 39–49, New York, NY, USA. ACM (2016)

  • Bull, J.M., Smith, L.A., Westhead, M.D., Henty, D.S., Davey, R.A.: A benchmark suite for high performance java. In: Proceedings of ACM Java Grande Conference, pp. 81–88. ACM (1999)

  • Chen, F., Grundy, J., Schneider, J.-G., Yang, Y., He, Q.: Automated analysis of performance and energy consumption for cloud applications. In: Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering, ICPE ’14, pp. 39–50, New York, NY, USA. ACM (2014)

  • Chen, J., John, L.K., Kaseridis, D.: Modeling program resource demand using inherent program characteristics. SIGMETRICS Perform. Eval. Rev. 39(1), 1–12 (2011)

    Article  Google Scholar 

  • Chen, S., Liu, Y., Gorton, I., Liu, A.: Performance prediction of component-based applications. J. Syst. Softw. 74(1), 35–43 (2005)

    Article  Google Scholar 

  • Cheung, L., Golubchik, L., Sha, F.: A study of web services performance prediction: a client’s perspective. In: Proceedings of the 19th Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS ’11, pp. 75–84, Washington, DC, USA. IEEE (2011)

  • Chun, B.-G., Huang, L., Lee, S., Maniatis, P., Naik, M.: Mantis: predicting system performance through program analysis and modeling. CoRR, abs/1010.0019 (2010)

  • Coppa, E., Demetrescu, C., Finocchi, I.: Input-sensitive profiling. In: Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI’12, pp. 89–98, New York, NY, USA. ACM (2012)

  • Curtsinger, C., Berger, E.D.: Coz: Finding code that counts with causal profiling. In: Proceedings of the 25th Symposium on Operating Systems Principles, SOSP ’15, pp. 184–197, New York, NY, USA. ACM (2015)

  • Duggan, J., Cetintemel, U., Papaemmanouil, O., Upfal, E.: Performance prediction for concurrent database workloads. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, SIGMOD ’11, pp. 337–348, New York, NY, USA. ACM (2011)

  • Feng, W., Zhang, Y.: A birth-death model for web cache systems: numerical solutions and simulation. In: Proceedings of International Conference on Hybrid Systems and Applications, pp. 272–284 (2008)

  • Ferrari, D., Serazzi, G., Zeigner, A.: Measurement and Tuning of Computer Systems. Prentice-Hall, Upper Saddle River (1983)

    Google Scholar 

  • Franks, G., Woodside, M.: Performance of multi-level client-server systems with parallel service operations. In: Proceedings of the 1st International Workshop on Software and Performance, WOSP ’98, pp. 120–130, New York, NY, USA. ACM (1998)

  • Ganapathi, A., Chen, Y., Fox, A., Katz, R., Patterson, D.: Statistics-driven workload modeling for the cloud. In: Proceedings of International Conference on Data Engineering Workshops, pp. 87–92 (2010)

  • Ganapathi, A., Kuno, H., Dayal, U., Wiener, J. L., Fox, A., Jordan, M., Patterson, D.: Predicting multiple metrics for queries: better decisions enabled by machine learning. In: Proceedings of International Conference on Data Engineering, pp. 592–603, Washington, DC, USA. IEEE (2009)

  • Goldsmith, S.F., Aiken, A.S., Wilkerson, D.S.: Measuring empirical computational complexity. In: Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering, ESEC-FSE ’07, pp. 395–404, New York, NY, USA. ACM (2007)

  • Gulur, N., Mehendale, M., Manikantan, R., Govindarajan, R.: Anatomy: an analytical model of memory system performance. In: The 2014 ACM International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’14, pp. 505–517, New York, NY, USA. ACM (2014)

  • Happe, J., Westermann, D., Sachs, K., Kapov, L.: Statistical inference of software performance models for parametric performance completions. In: QoSA’10, pp. 20–35 (2010)

  • Herodotou, H., Babu, S.: Profiling, what-if analysis, and cost-based optimization of mapreduce programs. PVLDB 4(11), 1111–1122 (2011)

    Google Scholar 

  • Hoare, C.A.R.: Monitors: an operating system structuring concept. Commun. ACM 17(10), 549–557 (1974)

    Article  MATH  Google Scholar 

  • Hoecke, S.V., Verdickt, T., Dhoedt, B., Gielen, F., Demeester, P.: Modelling the performance of the web service platform using layered queuing networks. In: Proceedings of International Conference on Software Engineering Research and Practice (2005)

  • Huang, H., Li, S., Szalay, A., Terzis, A.: Performance modeling and analysis of flash-based storage devices. In: Symposium on Mass Storage Systems and Technologies (MSST), pp. 1 –11 (2011)

  • Israr, T.A., Lau, D.H., Franks, G., Woodside, M.: Automatic generation of layered queuing software performance models from commonly available traces. In: Proceedings of International Workshop on Software and Performance, WOSP ’05, pp. 147–158, New York, NY, USA. ACM (2005)

  • Jarvis, S.A., Foley, B.P., Isitt, P.J., Spooner, D.P., Rueckert, D., Nudd, G.R.: Performance prediction for a code with data-dependent runtimes. Concurr. Comput. Pract. Exp. 20, 195–206 (2008)

    Article  Google Scholar 

  • Kounev, S., Spinner, S., Meier, P.: Introduction to queueing petri nets: modeling formalism, tool support and case studies. In: Proceedings of the Third Joint WOSP/SIPEW International Conference on Performance Engineering, ICPE ’12, pp. 9–18, New York, NY, USA. ACM (2012)

  • Koziolek, H., Firus, V.: Parametric performance contracts: non-Markovian loop modelling and an experimental evaluation. In: Kuester-Filipe, J., Poernomo, I.H., Reussner, R.H. (eds.) Proceedings of the 5th International Workshop on Formal Foundations of Embedded Software and Component-Based Software Architectures (FESCA’06), volume 176(2) of ENTCS, pp. 69–87. Elsevier (2006)

  • Kristensen, L.M., Christensen, S., Jensen, K.: The practitioner’s guide to coloured petri nets. Int. J. Softw. Tools Technol. Transfer 2, 98–132 (1998)

    Article  MATH  Google Scholar 

  • Law, A.M., Kelton, W.D.: Simulation Modeling and Analysis, 2nd edn. McGraw-Hill Higher Education, New York (1997)

    MATH  Google Scholar 

  • Lazowska, E.D., Zahorjan, J., Graham, G.S., Sevcik, K.C.: Quantitative System Performance. Computer System Analysis Using Queuing Network Models. Prentice Hall, Upper Saddle River (1984)

    Google Scholar 

  • Lee, B.C., Brooks, D.M., de Supinski, B.R., Schulz, M., Singh, K., McKee, S.A.: Methods of inference and learning for performance modeling of parallel applications. In: Proceedings of SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’07, pp. 249–258, New York, NY, USA. ACM (2007)

  • Liu, Y., Fekete, A., Gorton, I.: Design-level performance prediction of component-based applications. IEEE Trans. Softw. Eng. 31(11), 928–941 (2005)

    Article  Google Scholar 

  • Menasce, D.A., Bennani, M.N.: Analytic performance models for single class and multiple class multithreaded software servers. In: International CMG Conference (2006)

  • Moseley, T., Connors, D.A., Grunwald, D., Peri, R.: Identifying potential parallelism via loop-centric profiling. In: Proceedings of the 4th International Conference on Computing Frontiers, CF ’07, pp. 143–152, New York, NY, USA. ACM (2007)

  • Narayanan, D., Thereska, E., Ailamaki, A.: Continuous resource monitoring for self-predicting DBMS. In: Proceedings of International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 239–248, Washington, DC, USA. IEEE (2005)

  • Nethercote, N., Walsh, R., Fitzhardinge, J.: Building workload characterization tools with valgrind. Invited tutorial, October (2006)

  • Nguyen, H.Q., Apon, A.: Hierarchical performance measurement and modeling of the linux file system. In: Proceedings of International Conference on Performance Engineering, ICPE ’11, pp. 73–84, New York, NY, USA. ACM (2011)

  • Nguyen, H.Q., Apon, A.: Parallel file system measurement and modeling using colored petri nets. In: Proceedings of International Conference on Performance Engineering, ICPE ’12, pp. 229–240, New York, NY, USA. ACM (2012)

  • Ohmann, T., Herzberg, M., Fiss, S., Halbert, A., Palyart, M., Beschastnikh, I., Brun, Y.: Behavioral resource-aware model inference. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pp. 19–30, New York, NY, USA. ACM (2014)

  • Peierls, T., Goetz, B., Bloch, J., Bowbeer, J., Lea, D., Holmes, D.: Java Concurrency in Practice. Addison-Wesley Professional, Boston (2005)

    Google Scholar 

  • Reiss, S., Tarvo, A.: Automatic categorization and visualization of lock behavior. In: Proceedings of the First IEEE Working Conference on Software Visualization, VISSOFT ’13. IEEE (2013)

  • Reiss, S.P.: Chet: A system for checking dynamic specifications. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering, ASE ’04, pages 302–305, Washington, DC, USA, 2004. IEEE Computer Society

  • Reiss, S.P., Renieris, M.: Encoding program executions. In: Proceedings of the 23rd International Conference on Software Engineering, ICSE ’01, pp. 221–230, Washington, DC, USA. IEEE (2001)

  • Rolia, J., Casale, G., Krishnamurthy, D., Dawson, S., Kraft, S.: Predictive modelling of SAP ERP applications: challenges and solutions. In: Proceedings of the Fourth International ICST Conference on Performance Evaluation Methodologies and Tools, VALUETOOLS ’09, pp. 9:1–9:9, ICST, Brussels, Belgium. ICST (2009)

  • Roy, N., Dabholkar, A., Hamm, N., Dowdy, L.W., Schmidt, D.C.: Modeling software contention using colored petri nets. In: Miller, E.L., Williamson, C.L. (eds.) MASCOTS, pp. 317–324. IEEE (2008)

  • Stein, J.Y.: Digital Signal Processing: A Computer Science Perspective. Wiley, New York (2000)

    Book  Google Scholar 

  • Strebelow, R., Tribastone, M., Prehofer, C.: Performance modeling of design patterns for distributed computation. In: International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS ’12, pp. 251–258, Washington, DC, USA. IEEE (2012)

  • Tallent, N.R., Mellor-Crummey, J.M.: Effective performance measurement and analysis of multithreaded applications. In: Proceedings of the 14th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’09, pp. 229–240, New York, NY, USA. ACM (2009)

  • Tarvo, A., Reiss, S.P.: Using computer simulation to predict the performance of multithreaded programs. In: Proceedings of the Third Joint WOSP/SIPEW International Conference on Performance Engineering, ICPE ’12, pp. 217–228, New York, NY, USA. ACM (2012)

  • Tarvo, A., Reiss, S.P.: Automated analysis of multithreaded programs for performance modeling. In: Proceedings of the 29th International Conference on Automated Software Engineering, ASE ’14, pp. 7–18, New York, NY, USA. ACM (2014)

  • Teng, Q.M., Wang, H.C., Xiao, Z., Sweeney, P.F., Duesterwald, E.: Thor: a performance analysis tool for java applications running on multicore systems. IBM J. Res. Dev. 54(5), 456–472 (2010)

    Article  Google Scholar 

  • Thakkar, D., Hassan, A.E., Hamann, G., Flora, P.: A framework for measurement based performance modeling. In: Proceedings of International Workshop on Software and Performance, WOSP ’08, pp. 55–66, New York, NY, USA. ACM (2008)

  • Thereska, E., Doebel, B., Zheng, A.X., Nobel, P.: Practical performance models for complex, popular applications. In: Proceedings of International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, SIGMETRICS ’10, pp. 1–12, New York, NY, USA. ACM (2010)

  • Thereska, E., Ganger, G.R.: Ironmodel: robust performance models in the wild. In: Proceedings of International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’08, pp. 253–264, New York, NY, USA. ACM (2008)

  • van der Mei, R., Hariharan, R., Reeser, P.: Web server performance modeling. Telecommun. Syst. 16, 361–378 (2001). doi:10.1023/A:1016667027983

    Article  MATH  Google Scholar 

  • Varga, A., Hornig, R.: An overview of the OMNeT++ simulation environment. In: Proceedings of the 1st International Conference on Simulation Tools and Techniques for Communications, Networks and Systems and Workshops, Simutools ’08, pp. 60:1–60:10, ICST, Brussels, Belgium, Belgium. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering) (2008)

  • Wang, M., Au, K., Ailamaki, A., Brockwell, A., Faloutsos, C., Ganger, G.R.: Storage device performance prediction with cart models. In: Proceedings of International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS ’04, pp. 588–595, Washington, DC, USA. IEEE (2004)

  • Westermann, D., Happe, J., Krebs, R., Farahbod, R.: Automated inference of goal-oriented performance prediction functions. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pp. 190–199, New York, NY, USA. ACM (2012)

  • Woodside, C.M., Hrischuk, C.E., Selic, B., Bayarov, S.: Automated performance modeling of software generated by a design environment. Perform. Eval. 45(2–3), 107–123 (2001)

    Article  MATH  Google Scholar 

  • Woodside, C.M., Neilson, J.E., Petriu, D.C., Majumdar, S.: The stochastic rendezvous network model for performance of synchronous client-server-like distributed software. IEEE Trans. Comput. 44(1), 20–34 (1995)

  • Xu, J., Oufimtsev, A., Woodside, M., Murphy, L.: Performance modeling and prediction of enterprise javabeans with layered queuing network templates. In: Proceedings of Conference on Specification and Verification of Component-based Systems, SAVCBS ’05, New York, NY, USA. ACM (2005)

  • Zaparanuks, D., Hauswirth, M.: Algorithmic profiling. ACM SIGPLAN Not. 47(6), 67–76 (2012)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Dr. Eno Thereska for his insightful comments and feedback on the paper. This work is supported by the National Science Foundation through Grant CCF1130822.

Author information

Authors and Affiliations

  1. Google Inc, Kirkland, WA, USA

    Alexander Tarvo

  2. Brown University, Providence, RI, USA

    Steven P. Reiss

Authors
  1. Alexander Tarvo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven P. Reiss
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Tarvo.

Additional information

This work was carried out when Alexander Tarvo was at Brown University.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tarvo, A., Reiss, S.P. Automatic performance prediction of multithreaded programs: a simulation approach. Autom Softw Eng 25, 101–155 (2018). https://doi.org/10.1007/s10515-017-0214-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-017-0214-5

Keywords

Search

Navigation