Performance evaluation of an application-level checkpointing solution on grids

Abstract

In recent years there has been a significant effort to develop middleware that facilitates the execution of applications on Grid infrastructures. However, support for fault-tolerant execution continues to be scarce. The CPPC-G framework is a service-based architecture designed to provide efficient fault-tolerant mechanisms for the execution of sequential and parallel applications on grids. Applications to be managed by CPPC-G are expected to be preprocessed with CPPC (ComPiler for Portable Checkpointing), a tool for automatically inserting portable checkpoint instrumentation into the code of parallel applications. Built on top of existing Globus services, CPPC-G services are in charge of submitting and monitoring CPPC applications, managing generated checkpoint files, detecting failures and automatically restarting failed executions. In this paper the feasibility of this approach is assessed by measuring the performance of CPPC-G, quantitatively addressing its impact on application performance. Results show that the increase in overall throughput and availability comes with minor performance degradation.

Introduction

The emergence of Grid infrastructures has allowed users to remotely execute high-demanding computational applications on powerful nodes of the Grid or distribute them over several nodes. But grids are highly dynamic systems formed by thousands of resources connected to each other. Reliability of individual resources and communications cannot be guaranteed. Furthermore, as parallel platforms connected to the Grid increase their complexity, so does their overall failure rate. This is specially troublesome for long-running applications, whose mean time to complete execution amply exceeds the mean time to failure (MTTF) of the underlying hardware. In such a context, ensuring that not all of the computation done is lost in the event of failure is a must.

Checkpointing is a widely used technique to obtain fault tolerance on such environments. It periodically saves the computation state to stable storage, so that the application execution can be resumed by restoring such state. A number of solutions and techniques have been proposed [1], each having its own pros and cons. But because Grid environments are highly heterogeneous, successful application of checkpointing requires an evolution from traditional non-portable state saving techniques towards really portable tools that allow a computation to be resumed on a wide range of different machines. Portability in this context means to provide the following fundamental features:

• OS-independence: checkpointing strategies must be compatible with any given operating system. This means having at least a basic modular structure to allow for the substitution of certain critical sections of code (e.g. filesystem access) depending on the underlying OS.

• Support for parallel applications with communication protocol independence: the checkpointing framework should not make any assumption about the communication interface or implementation being used. Even recognizing the role of MPI as the message-passing de-facto standard, computational grids include machines belonging to independent entities which cannot be forced to provide a certain version of the MPI interface. The checkpointing technique cannot be theoretically tied to a specific MPI communication interface in order to provide a truly portable, reusable approach.

• Reduced checkpoint file sizes: the tool should optimize the amount of data being saved, avoiding dumping state which will not be necessary upon application restart. This improves performance, which depends heavily on state file sizes. This is specially true on Grid environments if state files have to be moved from one site to another when migrating an application in case of failure.

• Portable data recovery: the state of an application can be seen as a structure containing different types of data. The checkpointing tool must be able to recover all these data in a portable way. This includes recovery of opaque state, such as MPI communicators, as well as of OS-dependent state, such as the file table or the execution stack.

CPPC¹ (ComPiler for Portable Checkpointing) [2] provides all these features which are key issues for fault tolerance support on heterogeneous systems. The CPPC compiler automatically instruments a parallel code, transforming it into a fault-tolerant version. CPPC is aimed at being used with message-passing² parallel codes but it could be used with sequential codes as well. The analyses and transformations performed by the compiler completely automate the instrumentation process.

But to provide users with a really transparent fault-tolerant execution service on Grid environments the availability of a portable checkpointing tool as CPPC is not sufficient. Services are needed for managing an execution on behalf of the user, monitoring its state, making backups of generated state files and automatically detecting faults and taking the necessary corrective actions. CPPC-G [3] is a set of new Grid services³ implemented on top of Globus 4 [4] which provides such functionalities for CPPC-instrumented fault-tolerant applications (CPPC applications from now on). CPPC-G services will be in charge of submitting and monitoring the execution, as well as of managing and replicating the generated state files. Upon detecting a failure, CPPC-G services will restart the application from the most recent consistent state, in a completely transparent way. A large number of experiments have been performed to assess the impact of CPPC-G in the execution of an application. Results show that the overhead is negligible, specially if compared with typical execution times of long-running applications.

The rest of the paper is structured as follows. Section 2 gives an overview of the CPPC tool. Section 3 introduces CPPC-G. Section 4 details the performance evaluation of CPPC-G. Section 5 addresses the framework behavior with real large-scale applications. Section 6 describes related work and, finally, Section 7 concludes the paper.