Autonomic fault mitigation in embedded systems

Abstract

Autonomy, particularly from a maintenance and fault-management perspective, is an increasingly desirable feature in embedded (and non-embedded) computer systems. The driving factors are several—including increasing pervasiveness of computer systems, cost of failures which could potentially be catastrophic in a wide variety of critical systems, and increasing cost and strain on resources in maintaining systems. A trigger system employed in real-time filtering of particle-collision data is a particularly challenging example of a class of large-scale real-time embedded systems that demand a high degree of fault resilience, due to the large cost of operating the facilities and the potential for loss of irreplaceable data. Traditional redundancy-based approaches are not available due to the limited fault-tolerance budget above the system cost. This paper presents an approach based on model integrated computing that provides a set of tools for the system developer to specify, simulate, and synthesize autonomous fault-mitigative behaviors. A hierarchical, role-based organization of fault managers cleanly delineates the data-processing interactions in the system from the fault-mitigative control interactions. The fault-mitigative behaviors, analogous to autonomous biological systems, are characterized as (1) reflex actions—highly autonomous, localized, and uncoordinated response emanating from a single fault manager at any level of hierarchy, and (2) healing actions—highly coordinated behavior implemented with a sequence of interactions between multiple fault managers. The strength of the approach lies in the specification of these behaviors as coordinated interacting hierarchical concurrent finite-state machines, which makes these behaviors formally analyzable.

Introduction

The increasing pervasiveness, scale, and complexity of embedded (and non-embedded) computer systems, and the resultant increased cost in tuning and maintaining these systems has driven the attention of industrial and academic researchers towards novel approaches in self-management of computer systems. Autonomic computing is one such initiative that is being aggressively pursued by several industry leaders, including IBM, HP, Sun, and Microsoft. Autonomic computing has been defined as a self-managing computing model named after, and patterned on, the human body's autonomic nervous system. According to this definition, an autonomic computing system controls the functioning of computer applications and systems with minimal human intervention, in the same way that the autonomic nervous system regulates body systems without conscious input from the individual.

The research presented in this paper is motivated by the fault-tolerance requirements of a class of large-scale real-time embedded (LSRTE) systems, such as those employed in the high-energy physics online trigger application. A defining characteristic of this class of systems is their sheer scale (thousands of processors), which is comparable to grid computing systems. However, what sets these systems apart from grid systems is the tight timing constraints (hard real-time in milliseconds to microseconds), physical co-location of the processing elements (motivated by the timing requirements), and embedding within a large physical system (trigger is embedded in a particle accelerator/collider system). The sheer scale of these (and similar grid) systems makes the traditional redundancy-based approaches to fault tolerance infeasible, due to budgetary, power, and size constraints. Employing a triple-modular redundancy solution will increase the cost of the system significantly higher than three times a non-tolerant system. Another crucial observation in this class of system is that degraded performance is a viable mode of operation compared to a complete and catastrophic failure of the system in response to component failures.

Autonomic computing offers some interesting opportunities in addressing the fault-tolerance requirements of this class of systems. Consider, for example, the characteristics of an autonomic computing system as defined by IBM described in IBM (2004):

• it must maintain comprehensive and specific knowledge about all its components;

• it must have the ability to self-configure to suit varying and possibly unpredictable conditions;

• it must constantly monitor itself for optimal functioning; it must be self-healing and able to find alternate ways to function when it encounters problems;

• it must be able to detect threats and protect itself from them; it must be able to adapt to environmental conditions;

• it must be based on open standards rather than proprietary technologies; and

• it must anticipate demand while remaining transparent to the user.

Clearly, some of the capabilities listed above are desired in a system that can withstand unpredictable failures. A solution that offers fault tolerance by way of actively and adaptively reconfiguring the system (referred to as fault mitigation)—with a significantly lower overhead compared to the redundancy factor of typical fault-tolerance solution—is viable in the LSRTE class of systems. Unfortunately, the characteristics listed above are notional and till date there are no off-the-shelf solutions that could be quickly customized and applied to the trigger system.

Moreover, while the benefits of an autonomic computing approach are significant, they come at the expense of enhanced complexity and cost in developing an autonomic system. Developing a general purpose infrastructure that supports all the capabilities listed above, in addition to meeting the real-time requirements of the systems of interest would be a challenging undertaking. It would require significant efforts in designing and programming autonomic responses at multiple levels of the software infrastructure, including the application, the middleware, and the operating system. Furthermore, the autonomic responses must be coordinated across the different levels of the software infrastructures and across processor boundaries in an essentially large-scale distributed system.

Therefore, automated tools are required to assist the system developers in managing the complexity in designing an autonomic response system. These tools should offer the following: (1) higher-level abstractions for designing adaptive behaviors, which are easier to manipulate and maintain; (2) ability to analyze and simulate the autonomic responses to assess the systems ability to adapt with respect to different failure scenarios; and (3) ability to synthesize low-level programming artifacts from the higher-level abstractions.

This paper describes one such tool suite that is based on the principles of model integrated computing (MIC) (Sztipanovits, 1998; Nordstrom, 1999; Bapty et al., 2000). The key elements of this approach are a graphical modeling environment (GME) as demonstrated in Ledeczi et al. (2000) that is instantiating a domain-specific modeling language (DSME) and a suite of translators that assist in the transformation of domain models to simulation and low-level programming artifacts.

The rest of the paper is organized as follows: Section 2 describes the core concepts of the autonomic fault-mitigation approach. Section 3 provides details of the modeling environment, simulation of the domain models, and the automatic synthesis of low-level programming artifacts from design models. Section 4 presents a case study in application of the tools to a sub-scale prototype of the BTeV trigger system. An overview of related research activities is presented in Section 5, and finally Section 6 concludes the paper with an evaluation of this research and proposals for future work.