Ahab’s legs in scenario-based requirements validation: An experiment to study communication mistakes

Highlights

• The Ahab’s leg (AL) is a known problem of conversion between media.

• The validation of requirements may be subject to the problem of AL.

• This work is an empirical study on participatory validation of requirements.

• The goal is to evaluate the impact of AL due to the translation into scenarios.

Abstract

The correct identification of requirements is a crucial step for the implementation of a satisfactory software system. In the validation of requirements with scenarios, a straightforward communication is central to obtain a good participation from stakeholders. Technical specifications are translated into scenarios to make them concrete and easy to understand for non-technical users, and contextual details are added to encourage user engagement.

However, additional contextual details (Ahab’s legs) could generate a negative impact on the requirements’ validation by leading to proliferating comments that are not pertinent to session objective. The objective of this study is to evaluate the impact of Ahab’s leg to scenario-based requirement validation sessions. We conducted a controlled experiment with human participants and measured the pertinence of the comments formulated by participants when discussing the requirements. The results of our experiment suggest that the potentially negative impact of Ahab’s leg can be effectively controlled by the analyst.

Introduction

The current software development practice includes the involvement of stakeholders into the design process in order to close the gap between the users’ needs and the product to deliver. Good systems cannot be built by designers who proceed with only limited input from users. In particular, a continuous communication with the stakeholders is crucial for the requirements engineering phase, during which the analysts define functional and non-functional assets of the system to be designed. However, techniques for involving non-technical users into a collaborative definition of the product are still challenging and require further research.

Current research suggests that the use of appropriate tools may help in establishing a constructive dialogue and in optimizing the communication effort. For instance, the conjoint use of scenarios and personas (Aoyama, 2007) is quite established in the literature: it provides means for engaging the stakeholders, and for focusing their attention to desired aspects of the design that the analysts need to discuss. In particular narrative scenarios with personas are largely used in focus groups (Merton, 1990), where the aim is to illustrate some functionalities of the system. The idea is that the meaning and the effectiveness of the functions of the system cannot be evaluated in isolation from the context in which the system will run. The strength of scenarios/personas is to instantiate the requirements $-$ too abstract to be communicated directly $-$ into a very concrete spatial-temporal context, easy to illustrate and effecting in easing users participation. However, involving the stakeholders into the design generally increases the cost of the analysis, because, so far, stakeholders do not have a real active role in the process. Analysts must accurately prepare meetings, facilitate the discussions to generate useful contributions, analyse all the user feedback and, when possible, integrate such feedback into the system design.

In this paper we consider what may happen during the requirements validation. In a participatory design, before freezing the set of requirements of the system-to-be, it is important to go back to the stakeholders for validating them, i.e. to check that analysts and stakeholders share the same vision of the system features.

Typically in the participatory design, the analyst may call a validation focus group (Merton, 1990) either for discussing partial progress in the requirements elicitation, or for freezing the whole requirements set and proceed with the other phases. Given the specific objective of these kind of meetings, the discussion should target the supposed topic. Despite all comments are potentially important and should be annotated as possible indication of a revision (Finkelstein, 1995), a measure of the success of the communication with the stakeholder is the number of comments that are pertinent with the objective of the meeting. A skilled facilitator is able to guide the stakeholders into talking about the desired topic, and to discourage them from going off-topic. In scenario-based requirements the role of the facilitator is therefore tricky, as stakeholders are asked to comment about requirements, even thought what they actually see is their translation into a scenario.

Scenarios are however rich in ancillary details that are only added to make the story more concrete and engaging, but which have no relationship with the original requirements. These new extra contextual details – we call them Ahab’s legs (Leonardi et al., 2010) – make scenarios more prone to have communications mistakes. Indeed wrong communications may cause the attention of the stakeholders to shift from the requirement to be validated, to an unimportant detail that is not part of the requirements. Furthermore, comments raised on Ahab’s legs can make the facilitator lose control over the real objective of the discussion and waste valuable time during the session, therefore derailing the objective of the validation.

In this paper, we report on a controlled experiment we conducted to empirically evaluate the role of Ahab’s legs in focus groups. Our objective is to measure the influence of Ahab’s leg on the pertinence of comments formulated by stakeholders during requirements validation sessions, and assess if the impact of these extra details can be effectively controlled by the analyst.

The rest of this paper is organized as follows. After introducing the notion of Ahab’s leg in Section 2, we present the design of the experiment in Section 3. We report on the achieved results in Sections 4 and 5 we discuss the findings, lessons learned, recommendations to analysts and threats to validity. Finally, after comparing with related work in 6 Related works, 7 Conclusion concludes the paper.