Assessing practical usefulness and performance of the PREDIQT method: An industrial case study

https://doi.org/10.1016/j.infsof.2012.07.006Get rights and content

Abstract

Context

When adapting a system to new usage patterns, processes or technologies, it is necessary to foresee the implications of the architectural design changes on system quality. Examination of quality outcomes through implementation of the different architectural design alternatives is often unfeasible. We have developed a method called PREDIQT with the aim to facilitate model-based prediction of impacts of architectural design changes on system quality. A recent case study indicated feasibility of the PREDIQT method when applied on a real-life industrial system. The promising results encouraged further and more structured evaluation of PREDIQT.

Objective

This paper reports on the experiences from applying the PREDIQT method in a second and more recent case study – on a real-life industrial system from another domain and with different system characteristics, as compared to the previous case study. The objective was to evaluate the method in a fully realistic setting and with respect to carefully defined criteria.

Method

The case study conducted the first two phases of PREDIQT in their entirety, while the last (third) phase was partially covered. In addition, the method was assessed through a thought experiment-based evaluation of predictions and a postmortem review. All prediction models were developed during the analysis and the entire target system was analyzed in a fully realistic setting.

Results

The evaluation argues that the prediction models are sufficiently expressive and comprehensible. It is furthermore argued that PREDIQT: facilitates predictions such that informed decisions can be made; is cost-effective; and facilitates knowledge management.

Conclusion

The experiences and results obtained indicate that the PREDIQT method can be carried out with limited resources, on a real-life system, and result in useful predictions. Furthermore, the observations indicate that the method, particularly its process, facilitates understanding of the system architecture and its quality characteristics, and contributes to structured knowledge management.

Introduction

When adapting a system to new usage patterns, processes or technologies, it is necessary to foresee the implications that the architectural design changes have on system quality. Predictability with respect to non-functional requirements is one of the necessary conditions for the trustworthiness of a system. Examination of quality outcomes through implementation of the different architecture design alternatives is often unfeasible. A model-based approach is then an alternative. We have developed a method called PREDIQT with the aim to facilitate model-based prediction of impacts of architectural design changes on system quality. Examples of quality characteristics include availability, scalability, security and reliability.

A recent case study [1] indicated feasibility of the PREDIQT method when applied on a real-life industrial system. The promising empirical results and experiences from the previous case study encouraged further and more structured evaluation of the PREDIQT method. This paper addresses experiences from applying PREDIQT on another real-life industrial system from a different domain and with different system characteristics (lifetime, purpose, technology the system is implemented on, number of users and kind of users), compared to the previous case study.

The target system analyzed serves as a semantic model and a repository for representation of the system owner’s core working processes and rules, and as a knowledge database. It is a business-critical and complex expert system used for management and support of numerous working processes, involving hundreds of professional users every day. The system is subject to frequent architectural design changes of varying type and extent. The system owner, who was also the client commissioning the analysis, required full confidentiality with respect to the kind of system targeted, the models obtained, the personnel involved and the name of the organization. This paper reports solely on the experiences obtained by the participants of the real-life case, describes the process undergone, the evaluation results, the observations and the properties of the artifacts. The reported experiences and results have provided valuable insight into the strengths and weaknesses of the method.

The case study was conducted in the year 2010. The first overall two phases of the PREDIQT method were conducted in their entirety, while the last phase was partially covered. In addition, the method was assessed through a thought experiment-based evaluation of predictions and a postmortem review. All prediction models were developed during the analysis and the entire target system (within the predefined scope) was analyzed. The analysis was performed in the form of five workshops and six intermediate meetings in a fully realistic setting in terms of the scope, the objectives, the process, the prediction models and the participants.

This paper is a slightly revised and restructured version of a full technical report [2]. The latter is an extended version of a paper [3] originally presented at and published in the proceedings of the SSIRI’11 conference. With respect to the SSIRI’11 conference paper, this paper is extended with deduction of the success criteria, a detailed presentation of the setup and data collection during the PREDIQT-based analysis, a presentation of the outcomes of the PREDIQT-based analysis, a detailed presentation of the research method, specification of the template used for acquiring the written feedback, and a summary the feedback (on the PREDIQT-based analysis) received from the respondents through the evaluation template. Moreover, the extensions include a discussion of threats to validity and reliability, and a brief summary of the related work.

The rest of the paper is structured as follows. We briefly present the PREDIQT method in Section 2. The research method is summarized in Section 3. Section 4 presents five success criteria, which cover the needs of three identified stakeholder groups, and which both the case study and the contents of this paper have primarily been driven by. The process undergone during the PREDIQT-based analysis is presented in Section 5. Setup and data collection during the PREDIQT-based analysis are outlined in Section 6. The outcomes of the process, in terms of artifacts, evaluation results and observations, are reported in Section 7. Results of evaluation and a postmortem review are summarized in Section 8. Section 9 provides an evaluation of the experiences and results with respect to the five pre-defined success criteria. Threats to validity and reliability are discussed in Section 10. Section 11 summarizes some of the related work, before concluding in Section 12. A more thorough presentation of the research method is provided in Appendix A. Appendix B presents the design of the evaluation template used in relation to the postmortem review. A summary of the feedback received through the evaluation template is provided in Appendix C.

Section snippets

Overview of the PREDIQT method

The PREDIQT method produces and applies a multi-layer model structure, called prediction models, which represent system relevant quality concepts (through “Quality Models”), architectural design (through “Design Models”), and the dependencies between architectural design and quality (through “Dependency Views”). The Design Models are used to specify the target system and the changes whose effects on quality are to be predicted. The Quality Models are used to formalize the quality notions and

Research method

The research method is motivated by the guidelines for case study research provided by Yin [9]. A deductive approach is undertaken, where the already defined PREDIQT method is exposed to an empirical trial in the form of a case study. For more details on the research method, see Appendix A.

The main stages of the research method are depicted in Fig. 3. The case study design included characterization of research question, the units of analysis and the success criteria as the main outcomes.

The

Success criteria

Many concerns are relevant in the evaluation of a method like PREDIQT. In order to efficiently cover most prevailing concerns, we start by identifying the stakeholder groups involved: the customers, the domain experts and the analyst. Success criteria (SC) are then deduced from the point of view of each stakeholder group. Note that the degree of relevance of each success criterion may vary between the stakeholder groups.

The customers are the ones needing, requesting and paying for the

Overview of the process undergone during the PREDIQT-based analysis

This section focuses on the process of the PREDIQT-based analysis (that is, second main phase on Fig. 3). We chronologically outline the relevant events and meetings in terms of their contents, participants, preparation and the time spent.

The analysis took place during the year 2010. Two preliminary meetings were held between the customer representatives and the analyst. The preliminary meetings were spent for motivating the analysis and identifying the challenges which the analysis should

Setup and data collection during the PREDIQT-based analysis

The analyst had more than nine years of relevant professional experience in software engineering. The customer management representatives and the domain experts had between 15 and 33 years of relevant professional experience each. Totally, three managers (including the coordinator who is referred to as manager in Table 1, Table 2, Table 3) and four domain experts have been involved at the different stages of the analysis. In addition to following up the progress, internally coordinating the

Outcomes of the PREDIQT-based analysis

This section reports on the main outcomes of the process presented in the previous section. We focus particularly on the final prediction models, and the result of their validation and application carried out during the respective stages of the process.

Assessment

This section reports on the assessment part of the research method, the main stages of which are depicted in Fig. 3. The evaluation of the predictions based on a thought experiment is presented first. Second, the written feedback provided by the analysis participants from the customer organization upon completion of the aforementioned evaluation, is summarized. The written feedback is also referred to as a postmortem review. The third subsection reports on the verbal feedback provided, during

Evaluation with respect to the success criteria

In this section we evaluate the performance of the PREDIQT method in this case study with respect to the success criteria presented in Section 4. Thus, this section addresses the last stage of the research method depicted by Fig. 3.

SC1: The PREDIQT-based analysis facilitates predictions providing sufficient understanding of the impacts of architectural design changes on system quality characteristics, so that informed decisions can be made.

The ability of simulating a realistic change during

Threats to validity and reliability

The validity of the findings with respect to (1) the performance of the PREDIQT method and (2) the results of the evaluation of the predictions based on the thought experiment and the overall assessment, depends to a large extent on how well the threats have been handled. In addition to reliability threats, four types of validity threats, presented in [12], [13], are addressed: construct validity, conclusion validity, internal validity and external validity.

Reliability is concerned with

Related work

The PREDIQT method for model-based prediction of impact of architectural design changes on system quality characteristics makes use of models that capture the system design, the system quality notions and the interplay between system architecture and quality characteristics, respectively. The predictions result in propagation paths and the modified values of the parameters which express the quality characteristic fulfillment at the different abstraction levels. The PREDIQT method aims at

Conclusions

The PREDIQT method makes use of models that capture the system design, the system quality notions and the interplay between system architecture and quality characteristics, respectively. The predictions result in propagation paths and the modified values of the parameters which express the quality characteristic fulfillment at the different abstraction levels. PREDIQT aims at establishing the right balance between the practical usability of the models and usefulness of the predictions. We are

Acknowledgments

This work has been conducted as a part of the DIGIT (180052/S10) project funded by the Research Council of Norway, as well as a part of the SecureChange project and the NESSoS network of excellence both funded by the European Commission within the 7th Framework Programme.

References (27)

  • A. Omerovic, A. Andresen, H. Grindheim, P. Myrseth, A. Refsdal, K. Stølen, J. Ølnes, A Feasibility Study in Model Based...
  • A. Omerovic, B. Solhaug, K. Stølen, Evaluation of Experiences from Applying the PREDIQT Method in an Industrial Case...
  • A. Omerovic et al.

    Evaluation of experiences from applying the PREDIQT method in an industrial case study

  • A. Omerovic et al.

    A feasibility study in model based prediction of impact of changes on system quality

  • A. Omerovic et al.

    A practical approach to uncertainty handling and estimate acquisition in model-based prediction of system quality

    International Journal on Advances in Systems and Measurements

    (2011)
  • A. Omerovic et al.

    Traceability handling in model-based prediction of system quality

  • A. Omerovic, PREDIQT: A Method for Model-based Prediction of Impacts of Architectural Design Changes on System Quality,...
  • International Organisation for Standardisation: ISO/IEC 9126 – Software engineering – Product quality,...
  • R.K. Yin

    Case Study Research: Design and Methods

    (2002)
  • J. Rumbaugh et al.

    Unified Modeling Language Reference Manual

    (2004)
  • A. Omerovic et al.
    (2010)
  • T.D. Cook et al.

    Quasi-Experimentation: Design and Analysis Issues for Field Settings

    (1979)
  • W.R. Shadish et al.

    Experimental and Quasi-Experimental Designs for Generalized Causal Inference

    (2001)
  • Cited by (3)

    View full text