Learning to improve reliability during system development

Abstract

Research, conducted in collaboration with a leading aerospace manufacturer, aimed to facilitate learning in order to improve the reliability of engineering systems during their development phase. In particular, the processes and mathematical models used during reliability growth testing were investigated to assess how they might be better used to support this improvement. This required both soft and hard OR approaches to be adopted. For example, information flows were mapped and reengineered in order to provide a basis for more effective data collection and feed-back to decision-makers. A new mathematical model that combines failure data with engineering judgement was developed to estimate reliability growth. The paper presents a case study describing the problem, the modelling conducted, the recommendations made and the actions implemented. The ways in which the researchers and the manufacturer learnt to improve both the modelling and the reliability growth testing process are reflected upon.

Introduction

Reliability growth testing is a planned test, analyse and fix process, in which engineering systems are tested under simulated and accelerated environments, not outside design limits, for the purpose of disclosing design deficiencies and providing an indication of in-service reliability (Leitch, 1995). It is a standard element of the reliability programme (BS5760) for the safety-critical electronic systems designed and produced by a major aerospace manufacturer. Prior to this project the popular Duane model (Duane, 1964) was used to estimate the growth in reliability of the systems. However, it was perceived to be too simplistic to capture the complexities of the reliability growth testing process. In addition, it provided poor estimates of both the level and the rate of growth in reliability of the systems. For example, the manufacturer's records revealed that reliability performance of the systems in the field was always significantly better than that estimated and, in some instances, the estimates of growth derived during testing were counterintuitive to the engineers understanding of the failure mechanisms that redesigns of the systems were intended to overcome. These concerns are not restricted to this manufacturer. Indeed the shortcomings of Duane modelling are well documented (O'Connor, 1991). Consequently, there is a need to improve reliability growth modelling so that more accurate estimates are generated from models of the reliability growth testing process actually experienced in practice. Further, there is a need to return to the original premise of reliability growth modelling and, that is, to support learning during system development. This is particularly important if manufacturers are to develop engineering systems both effectively and efficiently. In the aerospace sector, with which we are concerned, accurate estimates of reliability growth and a sound understanding of the failure behaviour demonstrated during testing will always be very important from a safety perspective. However, the competitive nature of the industry implies that learning how to achieve high reliability faster and more cheaply is increasingly important.

Other authors have made attempts to address these issues. However, the existing literature either tends to consider high level management issues surrounding reliability improvement (O'Connor, 1994) or to introduce technical improvements relating to aspects of theoretical reliability growth models (Ebrahimi, 1996). Our aim is to bridge the gap by introducing a sound theoretical model that embraces the practical elements of reliability growth testing and addresses the issues surrounding its implementation from data collection to communication of the results back to the decision makers. Thus our focus is upon providing a modelling framework that will encourage everyone involved to think about the implications of the process and the modelling of reliability growth tests. Consequently, we believe that this will put us in a stronger position to learn how to improve the system design more effectively and efficiently.

The paper begins by describing the nature of reliability growth testing for the family of electronic systems designed by our client manufacturer. The sequence of decisions to be made are identified and the idiosyncrasies of the data usually available to support them are discussed. Based on this review we suggest new ways of managing hard failure data from the test so that its information content is fully exploited. By the very nature of their functionality, the type of systems considered tend to be intrinsically reliable so that there is in fact limited hard failure data. However, since their design follows an evolutionary process there is a wealth of expertise and understanding about the failure mechanisms of the systems, or at least aspects of them. Therefore, we describe procedures used to elicit soft data from engineers who are experts about some aspect of the system. This should instigate further thinking about reliability issues and provide more information that can be combined with the hard test data through modelling to provide measures of reliability performance that are in turn fed back to the experts as well as the decision-makers. Both the elicitation process and the mathematical model are given a brief overview, but here we concentrate upon illustrating their application through a case analysis. To conclude, we reflect upon both the theoretical and practical issues addressed during the project.