Fault tolerance in embedded real-time systems: Importance and treatment of common mode failures

Abstract

Dependable computer architectures used in critical embedded real-time applications have successfully employed Byzantine resilience techniques to tolerate physical, internal, operational faults. The dominant cause of failure of a correctly designed Byzantine resilient computer today is the common-mode failure, i.e., the nearly simultaneously failure of multiple redundant copies, generally due to a single cause. Unlike independent hardware faults, for which theoretically rigorous fault tolerance solutions have been implemented, the sources of common-mode failures are so diverse that numerous disparate techniques are required to predict, avoid, remove, and tolerate them.

This paper describes the technical approach that is being used to reduce the probability of common-mode failure in the Draper Fault Tolerant Parallel Processor which has been designed for critical embedded real-time applications. It begins with placing common-mode failures in the context of overall impairments to dependability to clarify their relative importance with respect to other failure sources. The FTPP's approach to tolerating independent hardware faults is briefly motivated and described. The overall strategy for common-mode failure reduction comprises three major areas: common-mode failure avoidance, removal, and tolerance. For fault avoidance, a novel integrated formal methods and VHDL design methodology has been developed and applied. Common-mode fault tolerance techniques include a combination of on-line checking of timing and functional behavior of operating system and application tasks, use of a formally verified system diagnosis processor to diagnose overall system health, and system-wide recovery actions. Techniques for the reduction of common-mode failure probability due to performance timing faults are also discussed.

This work was supported by NASA Langley Research Center under contract NAS1-18565.