Online root-cause performance analysis of parallel applications

Highlights

• We present a technique for automated application performance modeling and analysis.

• A parallel application is modeled by communication and computational activities.

• The analysis discovers causal dependencies between problems and infers root causes.

• It compares concurrent execution flows correlating problems in causal relationship.

• Results of root-cause analysis for different MPI applications are encouraging.

Abstract

The evolution of hardware is improving at an incredible rate. However, the advances in parallel software have been hampered for many reasons. Developing an efficient parallel application is still not an easy task. Applications rarely achieve good performances immediately and, therefore, careful performance analysis and optimization are crucial. These tasks are difficult and require a thorough understanding of the programs behavior. In this paper, we propose a systematic approach to online root-cause performance analysis. The automated analysis uses an online model to quickly identify the most important performance problems, and correlates them with application source code. Our technique is able to discover causal dependencies among the problems, infer their root causes and explain them to developers. In all of the scenarios we performed, this online modelling and analysis approach allowed us to understand the behavior of the applications, evaluate the performance and locate problem causes without specific knowledge of application internals.

Introduction

Although the evolution of hardware is improving at an incredible rate, the advances in parallel software have been hampered for many reasons. The main reason is that developing efficient parallel applications with existing programming models is a complex task. Moreover, the parallel applications rarely achieve good performances immediately and, hence, careful performance analysis and optimization are crucial. These tasks are difficult and costly and, in practice, developers must understand both the application and the environmental behavior. They must often focus more on resource usage, communication or synchronization, than on the actual problem being solved by their programs.

There are tools that automate the identification of performance bottlenecks and their locations in the source code [1], [2], [3], [4], [5], [6]. They help a developer in understanding what happens, where, and when [7], [8], [9], but they do not automate the inference process to find the causes of the performance problems. Detecting a bottleneck somewhere does not indicate why it happens and is often misleading. Only when the root-causes of a performance problem are correctly identified, is it possible to provide effective solutions. An overhead originating at a certain point in a task can causally propagate through the task flow and then through the message flow to another task and cause further inefficiencies at other points. It is necessary to provide tools that better assist developers in understanding the behavior of their programs by not only automating the search for performance problems, but also for their root causes. Such tools could be valuable for both non-experienced and expert users and, could ease and shorten the performance optimization process. In spite of this, the automation of root-cause analysis is still an open field of research. Many performance problems can be quickly located and explained with automated techniques that work on unmodified parallel applications during their execution [10], [11].

To address all these challenges, we have developed a new two-step approach for dynamic and automated application performance modelling and analysis. In this approach, an MPI application is automatically modelled and diagnosed during its execution. First, an online performance modelling technique enables the automated discovery of causal execution paths through communication and computational activities in message-passing parallel programs. Second, the automated analysis uses the online model to quickly identify the most important performance problems, and correlates them with an application source code (tasks, modules or functions). The performance analysis is performed at run-time and is based on the continuously updated model. Therefore, performance problems can be identified significantly faster than in a post-mortem approach. The analysis techniques investigate not only the performance problems, but also their causal relationships, and infer root causes in certain scenarios. In this context, developers and non-expert users are exempted from some of the performance-related duties.

Using this approach, it is possible to discover causal dependencies among the problems, infer their root causes during the application execution and explain them to developers. The online application model is based on the previous work presented in [12], while the methodology for the root-cause analysis is the main contribution of this paper.

The remainder of this paper is organized as follows. Section 2 introduces our approach for online performance analysis that can be deployed on arbitrary MPI applications running in large-scale parallel systems. Section 3 briefly describes an online performance modelling technique that we have proposed for understanding the behavior of parallel applications. Section 4 presents the automated root-cause analysis of performance problems using the online modelling technique. Section 5 presents examples of parallel applications that we were able to analyze online with our automated techniques. Section 6 surveys related work, and, finally, Section 7 concludes our work and suggests direction for future research.