Usage-based reading—an experiment to guide reviewers with use cases
Introduction
Inspections have been in software engineering for more than 25 years [5] and are considered as important contributors to improve software quality and information spreading within a software development team [4], [7]. Further, inspections have been extensively investigated through experiments and case studies. However, there is no common understanding in what contexts reading techniques work best, the most suitable team size, gain of inspection meetings etc. Hence, there is still a need for further experimentation [2], [3], [31]. Two important attributes to investigate are the efficiency of the reading technique, i.e. how many faults are found per unit time spent, and the effectiveness, i.e. how large share of the existing faults are found. Another important attribute of inspections is lead-time. Lead-time reduction can be achieved, for example, by not using the inspection meeting [12], [32].
Different inspection methods provide different techniques and guidelines to the reviewers. Fagan inspections are based on checklists, which guide the reviewers in their task of finding faults in a software artefact [5], [6]. Other reading techniques have been proposed in which the reviewers actively produce some results. Parnas' active design review [21] and Linger's and Mills' reading by stepwise abstraction [17] are early examples of active reading techniques. A more recent technique is scenario-based reading [22] where certain scenarios or instructions are followed to produce an artefact, and thereby identify faults. Perspective-based reading (PBR) is an example of scenario-based reading technique for reading of requirements documents [1], [29], in which three different perspectives, user, designer and tester, produce different artefacts; the artefacts produced are use cases, data flow diagrams and test cases, respectively. PBR has also been adapted and empirically evaluated for design inspections [15], code inspections [14] and usability inspections [35].
Usage-based reading (UBR) is a reading technique based on an inspection method presented by Olofsson and Wennberg [20]. Its goal is to drive the reviewers to focus on crucial parts in a software artefact from a user's point of view. Hence, UBR is based on the assumption that different faults are of different importance. The aim of the method is to find the faults that have the most negative impact on the users' perception of system quality. The principles of the method are that the reviewers are given the use cases in a prioritised order; they trace the use cases through the software artefact in that order, and thereby identify faults. Furthermore, in a software development environment where use cases are utilised in the requirements engineering, see for example Ref. [11], it is of little value to derive new use cases during inspections in addition to those already in the requirements specification, as pointed out by Regnell et al. [23]. Instead, the use cases can be utilised during inspection, which is another benefit of the UBR technique presented and initially evaluated in this paper. UBR applies the principles of usage-based testing [25] and operational profile testing [19] for inspections.
In order to initially investigate the effect of UBR, an experiment was set up with 27 students in their third year of the software engineering Bachelor's program at Lund University [27]. Guided by use cases, the students inspected a high-level design of a taxi management system. First, it was investigated whether the reviewers are controllable by the use case order, i.e. if different orders of scenarios lead to identification of different faults. Second, the effectiveness and efficiency of the reviewers using UBR are investigated, again by using prioritised and random order use cases, respectively.
It is concluded from the experiment that the reviewers find different faults using prioritised use cases compared to random order use cases. Further, UBR with prioritised use cases is significantly more effective and efficient than random order use cases in finding faults of high importance for a user. These are initial results indicating that the UBR technique works as intended. Further experimentation is planned to compare the UBR to a baseline of an established reading technique. An experiment with the purpose to compare UBR with checklist-based reading was carried out and would be analysed.
The paper is structured as follows. In Section 2, the background and the principles of UBR are presented. In the following sections, the experiment is presented; subjects and objects in Section 3, experiment planning in Section 4 and experiment operation in Section 5. In Section 6, the analysis of experiment data is presented and in Section 7, the results are discussed. Finally, a summary is presented in Section 8.
Section snippets
Usage-based reading
Many reading techniques focus on finding as many faults as possible, regardless of their importance. The inspection effectiveness is often measured in terms of number of faults found, without taking into account that some faults in the inspected object are likely to affect the final system quality more than the other faults.
The principal idea behind UBR is to focus the reading effort on the most critical faults in the object of inspection. Hence, faults are not assumed to be of equal
Experimental subjects and object
This section discusses the subjects and the object used in the UBR evaluation. Subjects refer to the reviewers and object to the software artefact inspected. The section also describes the use case notation used and the criterion and classification used for prioritising the use cases and faults.
Experiment planning
This section describes the planning of the inspection experiment. The planning includes the roles and activities, the definition of variables, hypotheses, design and analysis of threats to the validity of the experiment.
Experiment operation
The experiment was run during the autumn 2000 in an academic environment as a part of a verification and validation course. All students had participated in a previous requirements engineering course, where they developed a requirements specification for the same type, but extended system. The students were all familiar with the inspection protocol, since the same protocol was used during inspections in the requirements course. The experiment was a part of an exercise. From an educational view,
Analysis
This section discusses the data collected during the experiment and the statistical tests utilised. Two types of data were collected, time and fault data. Non-parametric tests are used for testing the hypotheses of the experiment.
Discussions
This section interprets the result of the hypotheses stated in Section 4.3. Furthermore, a discussion to support replications is provided, which also consists of experiences from this experiment. Since UBR has shown promising result, future work in the area is discussed.
Summary
The experiment presented in this paper is focused on investigating a reading technique named UBR. The experiment was run with 27 third year Bachelor's students in software engineering, where both groups inspected a software design document following the same 24 use cases but in different orders. The use cases for one group were prioritised from a user's point of view and the use cases for the control group were randomly ordered.
The main results from the analysis are that if the use cases are
Acknowledgements
The authors would like to thank the students for participating in the investigation and Thomas Olsson at the Department of Communication Systems at Lund University for developing the taxi management system. Thanks also to Johan Natt och Dag at the Department of Communication Systems for prioritising the use cases. This work was partly funded by The Swedish National Board for Industrial and Technical Development (NUTEK), under grant for Centre for Applied Software Research at Lund University
References (35)
- et al.
Perspective-based reading of code documents at Robert Bosch GmbH
Information and Software Technology
(1997) - et al.
An experimental comparison of reading techniques for defect detection in UML design documents
Journal of Systems and Software
(2000) - et al.
The empirical investigation of perspective-based reading
Empirical Software Engineering: An International Journal
(1996) - V.R. Basili, The role of experimentation in software engineering: past, current, and future, Proceedings of the 18th...
- et al.
Building knowledge through families of experiments
IEEE Transactions on Software Engineering
(1999) - et al.
Software Inspection Process
(1994) Design and code inspections to reduce errors in program development
IBM System Journal
(1976)Advances in software inspections
IEEE Transactions on Software Engineering
(1986)- et al.
Software Inspections
(1993) - et al.
Using students as subjects—a comparative study of students and professionals in lead-time impact assessment
Empirical Software Engineering: An International Journal
(2000)
Object-Oriented Software Engineering: A Use Case Driven Approach
Does every inspection really need a meeting?
Empirical Software Engineering: An International Journal
A cost-value approach for prioritizing requirements
IEEE Software
Structured Programming—Theory and Practice
Cited by (52)
On the representation and aggregation of evidence in software engineering: A theory and belief-based perspective
2013, Electronic Notes in Theoretical Computer ScienceGoal-Oriented Software Design Reviews
2022, IEEE AccessEffectiveness of Explaining a Program to Others in Finding Its Bugs
2021, Proceedings - 22nd IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2021-Fall