Usage-based reading—an experiment to guide reviewers with use cases

https://doi.org/10.1016/S0950-5849(01)00201-4Get rights and content

Abstract

Reading methods for software inspections are used for aiding reviewers to focus on special aspects in a software artefact. Many experiments were conducted for checklist-based reading and scenario-based reading concluding that the focus is important for software reviewers. This paper describes and evaluates a reading technique called usage-based reading (UBR). UBR utilises prioritised use cases to guide reviewers through an inspection. More importantly, UBR drives the reviewers to focus on the software parts that are most important for a user. An experiment was conducted on 27 third year Bachelor's software engineering students, where one group used use cases sorted in a prioritised order and the control group used randomly ordered use cases. The main result is that reviewers in the group with prioritised use cases are significantly more efficient and effective in detecting the most critical faults from a user's point of view. Consequently, UBR has the potential to become an important reading technique. Future extensions to the reading technique are suggested and experiences gained from the experiment to support replications are provided.

Introduction

Inspections have been in software engineering for more than 25 years [5] and are considered as important contributors to improve software quality and information spreading within a software development team [4], [7]. Further, inspections have been extensively investigated through experiments and case studies. However, there is no common understanding in what contexts reading techniques work best, the most suitable team size, gain of inspection meetings etc. Hence, there is still a need for further experimentation [2], [3], [31]. Two important attributes to investigate are the efficiency of the reading technique, i.e. how many faults are found per unit time spent, and the effectiveness, i.e. how large share of the existing faults are found. Another important attribute of inspections is lead-time. Lead-time reduction can be achieved, for example, by not using the inspection meeting [12], [32].

Different inspection methods provide different techniques and guidelines to the reviewers. Fagan inspections are based on checklists, which guide the reviewers in their task of finding faults in a software artefact [5], [6]. Other reading techniques have been proposed in which the reviewers actively produce some results. Parnas' active design review [21] and Linger's and Mills' reading by stepwise abstraction [17] are early examples of active reading techniques. A more recent technique is scenario-based reading [22] where certain scenarios or instructions are followed to produce an artefact, and thereby identify faults. Perspective-based reading (PBR) is an example of scenario-based reading technique for reading of requirements documents [1], [29], in which three different perspectives, user, designer and tester, produce different artefacts; the artefacts produced are use cases, data flow diagrams and test cases, respectively. PBR has also been adapted and empirically evaluated for design inspections [15], code inspections [14] and usability inspections [35].

Usage-based reading (UBR) is a reading technique based on an inspection method presented by Olofsson and Wennberg [20]. Its goal is to drive the reviewers to focus on crucial parts in a software artefact from a user's point of view. Hence, UBR is based on the assumption that different faults are of different importance. The aim of the method is to find the faults that have the most negative impact on the users' perception of system quality. The principles of the method are that the reviewers are given the use cases in a prioritised order; they trace the use cases through the software artefact in that order, and thereby identify faults. Furthermore, in a software development environment where use cases are utilised in the requirements engineering, see for example Ref. [11], it is of little value to derive new use cases during inspections in addition to those already in the requirements specification, as pointed out by Regnell et al. [23]. Instead, the use cases can be utilised during inspection, which is another benefit of the UBR technique presented and initially evaluated in this paper. UBR applies the principles of usage-based testing [25] and operational profile testing [19] for inspections.

In order to initially investigate the effect of UBR, an experiment was set up with 27 students in their third year of the software engineering Bachelor's program at Lund University [27]. Guided by use cases, the students inspected a high-level design of a taxi management system. First, it was investigated whether the reviewers are controllable by the use case order, i.e. if different orders of scenarios lead to identification of different faults. Second, the effectiveness and efficiency of the reviewers using UBR are investigated, again by using prioritised and random order use cases, respectively.

It is concluded from the experiment that the reviewers find different faults using prioritised use cases compared to random order use cases. Further, UBR with prioritised use cases is significantly more effective and efficient than random order use cases in finding faults of high importance for a user. These are initial results indicating that the UBR technique works as intended. Further experimentation is planned to compare the UBR to a baseline of an established reading technique. An experiment with the purpose to compare UBR with checklist-based reading was carried out and would be analysed.

The paper is structured as follows. In Section 2, the background and the principles of UBR are presented. In the following sections, the experiment is presented; subjects and objects in Section 3, experiment planning in Section 4 and experiment operation in Section 5. In Section 6, the analysis of experiment data is presented and in Section 7, the results are discussed. Finally, a summary is presented in Section 8.

Section snippets

Usage-based reading

Many reading techniques focus on finding as many faults as possible, regardless of their importance. The inspection effectiveness is often measured in terms of number of faults found, without taking into account that some faults in the inspected object are likely to affect the final system quality more than the other faults.

The principal idea behind UBR is to focus the reading effort on the most critical faults in the object of inspection. Hence, faults are not assumed to be of equal

Experimental subjects and object

This section discusses the subjects and the object used in the UBR evaluation. Subjects refer to the reviewers and object to the software artefact inspected. The section also describes the use case notation used and the criterion and classification used for prioritising the use cases and faults.

Experiment planning

This section describes the planning of the inspection experiment. The planning includes the roles and activities, the definition of variables, hypotheses, design and analysis of threats to the validity of the experiment.

Experiment operation

The experiment was run during the autumn 2000 in an academic environment as a part of a verification and validation course. All students had participated in a previous requirements engineering course, where they developed a requirements specification for the same type, but extended system. The students were all familiar with the inspection protocol, since the same protocol was used during inspections in the requirements course. The experiment was a part of an exercise. From an educational view,

Analysis

This section discusses the data collected during the experiment and the statistical tests utilised. Two types of data were collected, time and fault data. Non-parametric tests are used for testing the hypotheses of the experiment.

Discussions

This section interprets the result of the hypotheses stated in Section 4.3. Furthermore, a discussion to support replications is provided, which also consists of experiences from this experiment. Since UBR has shown promising result, future work in the area is discussed.

Summary

The experiment presented in this paper is focused on investigating a reading technique named UBR. The experiment was run with 27 third year Bachelor's students in software engineering, where both groups inspected a software design document following the same 24 use cases but in different orders. The use cases for one group were prioritised from a user's point of view and the use cases for the control group were randomly ordered.

The main results from the analysis are that if the use cases are

Acknowledgements

The authors would like to thank the students for participating in the investigation and Thomas Olsson at the Department of Communication Systems at Lund University for developing the taxi management system. Thanks also to Johan Natt och Dag at the Department of Communication Systems for prioritising the use cases. This work was partly funded by The Swedish National Board for Industrial and Technical Development (NUTEK), under grant for Centre for Applied Software Research at Lund University

References (35)

  • O Laitenberger et al.

    Perspective-based reading of code documents at Robert Bosch GmbH

    Information and Software Technology

    (1997)
  • O Laitenberger et al.

    An experimental comparison of reading techniques for defect detection in UML design documents

    Journal of Systems and Software

    (2000)
  • V.R Basili et al.

    The empirical investigation of perspective-based reading

    Empirical Software Engineering: An International Journal

    (1996)
  • V.R. Basili, The role of experimentation in software engineering: past, current, and future, Proceedings of the 18th...
  • V.R Basili et al.

    Building knowledge through families of experiments

    IEEE Transactions on Software Engineering

    (1999)
  • R.G Ebenau et al.

    Software Inspection Process

    (1994)
  • M.E Fagan

    Design and code inspections to reduce errors in program development

    IBM System Journal

    (1976)
  • M.E Fagan

    Advances in software inspections

    IEEE Transactions on Software Engineering

    (1986)
  • T Gilb et al.

    Software Inspections

    (1993)
  • M Höst et al.

    Using students as subjects—a comparative study of students and professionals in lead-time impact assessment

    Empirical Software Engineering: An International Journal

    (2000)
  • ITU-T Z. 100 1993 Z100 Specification and Description Language, SDL,ITU-T Recommendation Z.100,...
  • ITU-T Z. 120 Message Sequence Charts, MSC,ITU-T, Recommendation Z.120,...
  • I Jacobson et al.

    Object-Oriented Software Engineering: A Use Case Driven Approach

    (1992)
  • P.M Johnson et al.

    Does every inspection really need a meeting?

    Empirical Software Engineering: An International Journal

    (1998)
  • J Karlsson et al.

    A cost-value approach for prioritizing requirements

    IEEE Software

    (1997)
  • S. Lauesen, Software Requirements—Styles and Techniques, Samfundslitteratur, Denmark,...
  • R.C Linger et al.

    Structured Programming—Theory and Practice

    (1979)
  • Cited by (52)

    • Effectiveness of Explaining a Program to Others in Finding Its Bugs

      2021, Proceedings - 22nd IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2021-Fall
    View all citing articles on Scopus
    View full text