Systematic approaches to understanding and evaluating design trade-offs

https://doi.org/10.1016/j.jss.2007.10.032Get rights and content

Abstract

The use of trade-off analysis as part of optimising designs has been an emerging technique for a number of years. However, only recently has much work been done with respect to systematically deriving the understanding of the system problem to be optimised and using this information as part of the design process. As systems have become larger and more complex then a need has arisen for suitable approaches. The system problem consists of design choices, measures for individual values related to quality attributes and weights to balance the relative importance of each individual quality attribute. In this paper, a method is presented for establishing an understanding of a system problem using the goal structuring notation (GSN). The motivation for this work is borne out of experience working on embedded systems in the context of critical systems where the cost of change can be large and the impact of design errors potentially catastrophic. A particular focus is deriving an understanding of the problem so that different solutions can be assessed quantitatively, which allows more definitive choices to be made. A secondary benefit is it also enables design using heuristic search approaches which is another area of our research. The overall approach is demonstrated through a case study which is a task allocation problem.

Introduction

For a number of years research has been performed research into two important areas in isolation from one another. These are: understanding design choices, trade-offs and evaluation criteria during the design of systems; and the application of search-based techniques to optimise designs. These two areas are by their very nature linked with the former effectively being an input to the latter, i.e., the first stage derives an understanding of the problems and the second stage finds the solution. However, the majority of work in either one of these areas has been done quite independently of the other. This has lead to a loss of traceability and rationale between the derivation of the problem and its solution.

The aim of this work is to present a technique for understanding the trade-offs. This information can then be used as part of either a manual or automated design process. The motivation for the work is borne out of experience working on embedded systems in the context of critical systems where the cost of design changes can be large and the impact of design errors potentially catastrophic. During our previous work (Bate and Burns, 2003), adapting existing scheduling and timing analysis for use in ‘real’ critical systems, a number of important design decisions (e.g., the type of timing watchdog used to identify timing failures) were faced whose impact had far reaching consequences across the system’s design. However, a lack of suitable techniques was found for considering the trades-off involved. The key deficiencies with these techniques are a lack of a systematic method and poor support for capturing rationale and assumptions. Both of these meant a lack of support for maintenance and the results obtained were of questionable integrity. Therefore, the research into suitable methods presented in this paper was instigated.

The method used for deriving the trade-off analysis problem is based on the goal structuring notation (GSN) (Kelly, 1999). GSN was originally developed for constructing safety arguments for systems and has since achieved widespread use. In the context of this work, it is considered to have some advantages including stronger traceability, better support for capturing rationale and is easily mapped onto traditional optimisation algorithms. However, the key contribution in this paper, with respect to GSN, is how GSN is used as part of designing systems rather than for constructing safety cases. It is noted that other notations offering similar ways of decomposing objectives and capturing assumptions etc could be used within the technique proposed.

The process of establishing the trade-off analysis problem begins with using GSN to decompose the top-level objectives (often referred to as quality attributes) of the system in a hierarchical tree-like fashion. The decomposition is continued until the objectives reach a suitably low-level that they measure how well the specific individual objectives are met. An example of this is a higher-level objective of meeting the requirements could be decomposed through an objective of meeting the timing requirements to a lower-level objective that tasks’ response times must always be less than or equal to their deadline. This last objective can be assessed, and appropriate evidence gathered, using timing analysis (Audsley et al., 1995). Using the hierarchy information given by the tree, individual results can later be combined to give results for higher-level objectives. For instance, a weighted sum of results for lower-level objectives may be used to provide a single higher-level result. Combining the results of individual objective functions into a single overall objective. This can then be often used in any form of cost benefit analysis including as part of a fitness functions within a search algorithms. As such the approach represents a logical approach to systematically building knowledge of how to optimise designs featuring arbitrarily complex trade-offs.

At the same time as performing this decomposition, design choices (e.g., choice of computational model between static scheduling and fixed priority scheduling) and assumptions (e.g., the tasks have predictable and bounded execution times) are captured.

This is a distinctly different approach from techniques such as the Architecture trade-off analysis method (ATAM) (Bass et al., 2003) as their method relies heavily on the information being provided by experts or reusing information derived during previous applications of the technique (e.g., from an associated handbook). There are also a wide variety of methods of performing trade-offs including automated search techniques such as simulated annealing and genetic algorithms (Rayward-Smith et al., 1996). Of these, to the best of our knowledge none of the work has addressed how the problems should be derived by systematic means. This is a key benefit of the work presented here.

The combination of deriving an understanding of the design problem with the mechanism for making the design decisions in a more traceable manner at the same time as capturing the rationale has a number of significant advantages for practitioners in different domains. For all domains, the rationale will provide better support for the change process as the reasons behind the original design and links between parts of the design will allow the risk and impact of change to be considered more thoroughly (Bass et al., 2003). For safety-related domains, the traceability of design through to objectives (including the decomposition of high-level objectives to low-level objectives), mapping the objectives to assessment criteria and mechanisms and then to evidence the objectives are met is the basis for most standards (Herrmann, 2000, United Kingdom Ministry of Defence, 1996, RTCA Inc, 1992, CENELEC, 2001, United Kingdom Ministry of Defence, 2004). In particular, some standards are now moving away from traditional process-oriented approaches (United Kingdom Ministry of Defence, 1996) to product-based (in other words evidence-based) standards (United Kingdom Ministry of Defence, 2004). The reason is the process-oriented standards tend to lead to a tick box mentality rather than fundamentally questioning the needs of the project regarding safety (McDermid, 2001).

Therefore, the key contributions of this paper are: the understanding of the contextual information upon which the design trade-offs are made in order to support reuse and maintenance of the system’s design; the well-defined and systematically derived knowledge of the design problem and solution aids certification which could also be reused on similar systems; and then the mapping of these onto quantitative measures, where possible, raising the possibility of using these measures as part of an automated search strategy. A significant part of the contribution is the means of turning difficult to assess criteria into quantitative measures using a variety of means, e.g., through the use of scenarios.

The structure of the paper is as follows. Section 2 contains a literature survey that considers what related work exists and as such helps establish the contribution in this paper. Background on the GSN and its application in the critical systems domain is given in Section 3. A detailed description of the method is given in Section 4. A case study is then used to demonstrate the method in Section 5. Finally, Section 6 provides a summary of the work, concluding remarks and suggests areas for future work.

Section snippets

Related work

The purpose of this section is to consider the existing work on design and trade-off analysis that is related to this paper. A comprehensive survey of the subject can be found at Dobrica and Niemela (2002).

ATAM, a technique for evaluating architectures for their support of architectural qualities and trade-offs in achieving those qualities, has been developed by the Software Engineering Institute (Bass et al., 2003, Kazman et al., 1999). The approach is largely based on deriving quality

Overview of GSN

As stated earlier, the problem derivation is based on GSN (Kelly, 1999). GSN was originally derived for use in the production of safety cases as part of the certification of systems. During the establishment of the safety argument’s claims (often referred to as goals), context, assumptions and justifications are captured which has a number of uses including managing change. The GSN (Kelly, 1999) – a graphical argumentation notation – explicitly represents the individual elements of any safety

Method for deriving an understanding of system trade-offs

This section is concerned with presenting the method that has been derived along with the reasons behind it.

Background on the problem

The problem considered in this paper is that of task allocation. The main aim of the task allocation problem is to ensure the system’s timing requirements are met where the requirements feature both independent tasks and relationships (i.e., dependencies) between tasks. The task allocation problem has two main parts. First assigning tasks to specific processors (when there are more than one). Second choosing attributes (e.g., priority, ordering, etc.) for tasks. Where there is more than one

Conclusions and summary

In this paper, we have motivated the need for more systematic approaches to understanding the trade-offs within the design of systems. The specific contributions of the paper are to show how our approach provides the degree of rigour suitable for critical systems applications at the same time as capturing the rationale and justifications to help support maintainability of systems in general. The approach presented is based on a well-established approach to safety argumentation that has been

Acknowledgements

I would like to thanks a number of colleagues who have cooperated with earlier work in this area including Neil Audsley, Paul Emberson, Tim Kelly and Peter Nightingale.

References (40)

  • I. Bate et al.

    Architectural considerations in the certification of modular systems

    Reliability Engineering and System Safety

    (2003)
  • I. Bate et al.

    Establishing timing requirements for control loops in real-time systems

    Microprocessors and Microsystems

    (2003)
  • L. Tahvildari et al.

    Quality-driven software re-engineering

    Journal of Systems and Software

    (2003)
  • Anton, A., 1996. Goal-based requirements analysis. In: Proceedings of the 2nd International Conference on Requirements...
  • Anton, A., Potts, C., 1998. The use of goals to surface requirements for evolving systems. In: Proceedings of the 20th...
  • N. Audsley et al.

    Fixed priority pre-emptive scheduling: an historical perspective

    Real-Time Systems

    (1995)
  • Axelsson, J., 1996. Three search strategies for architecture synthesis and partitioning of real-time systems. Technical...
  • L. Bass et al.

    Software Architecture in Practice

    (2003)
  • Bate, I., Audsley, N., 2004. Flexible design of complex high-integrity systems using trade offs. In: Proceedings of the...
  • I. Bate et al.

    An integrated approach to scheduling in safety-critical embedded control systems

    Real-Time Systems Journal

    (2003)
  • Bate, I., Emberson, P., 2005. Design for flexible and scalable avionics systems. In: Proceedings of the IEEE Aerospace...
  • Bate, I., Emberson, P., 2006. Incorporating scenarios and heuristics to improve flexibility in real-time embedded...
  • Bate, I., Kelly, T., 2002. Architectural considerations in the certification of modular systems. In: Proceedings of the...
  • Bate, I., Cervin, A., Nightingale, P., 2003. Establishing timing requirements and control attributes for control loops...
  • Brooks, D., Tiwari, V., Martonosi, M., 2000. Wattch: a framework for architectural-level power analysis and...
  • A. Burns et al.

    Feasibility analysis of fault-tolerant real-time task sets

    Euromicro Real-Time Systems Workshop

    (1996)
  • CENELEC. IEC 61508 Functional Safety of electrical/electronic/programmable electronic safety-related systems,...
  • L. Chung et al.

    Non-functional Requirements in Software Engineering

    (1999)
  • C. Coello Coello

    A comprehensive survey of evolutionary-based multiobjective optimization techniques

    Knowledge and Information Systems

    (1999)
  • L. Dobrica et al.

    A survey on software architecture analysis methods

    IEEE Transactions on Software Engineering

    (2002)
  • Cited by (0)

    View full text