Empirical comparisons of animation and narration in requirements validation

Abstract

Increasingly powerful computing technology suggests an expansion of multimedia features in requirements engineering methods. It is not obvious, however, that techniques such as animation and narration would improve the effectiveness of communicating domain information to stakeholders for validation. Three principles from the cognitive theory of multimedia learning (CTML) are used to compare validation methods using animation and narration with more traditional methods using static diagrams and text. Results suggest animation and narration can have significant positive impact on the level of domain understanding attained by participants. In particular, narration yielded strongly significant results. While these results should be viewed as preliminary, they indicate a potential advantage in the use of narration and possibly animation in requirements validation. The results provide justification for further research on the integration of multimedia techniques in developing and validating requirements.

Experiment materials

1.1 Text descriptions

1.1.1 Description for Conference case

An IFIP Working Conference is an international conference centered on a topic of specific interest to one or more IFIP working groups. Participation in the conference is by invitation only. Two objectives of the conference organizers are to ensure that members of the involved working group(s) and technical committee(s) are invited and that attendance is sufficient for financial break-even without exceeding the capacity of the facilities available.

Two committees are involved in organizing an IFIP working conference: a program committee and an organizing committee. The program committee deals with the technical content of the conference, and the organizing committee handles financial and local arrangements, along with invitations and publicity. These committees work together closely and have a need for common information.

When an IFIP conference is to be held, the program committee is responsible for several activities. These include preparing a mailing list of potential authors and sending a call for papers to each of these individuals. If the potential authors reply with a letter of intent indicating they will submit a paper to the conference, the program committee registers the participant’s intent to participate. When the paper is received, the paper is also registered.

After receiving the papers, the program committee assigns a set of reviewers for each paper and then sends the papers to the respective reviewers. Reviewers create a review report and send their reports to the committee. The committee then groups accepted papers into sessions and assigns session chairs. The list of accepted authors is forwarded to the organizing committee. This session information makes up the itinerary for the conference.

The organizing committee begins by preparing a list of people to invite to the conference. The committee issues priority invitations to national representatives and to members of related working groups. The organizing committee also ensures that each of the contributing authors receives an invitation. Individuals who receive an invitation and who intend to come to the conference must indicate their intent to participate by sending an acceptance of invitation to the organizing committee. The final list of participants is then generated by the committee, who makes an effort to avoid sending duplicate invitations to any individual.

1.2 Description for Event case

A company that owns a large sports and entertainment complex holds a variety of events in their building, including professional sports, concerts and conventions. The program of events is planned several months in advance. While no more than one event can take place at any given time, it is possible for more than one event to occur on the same day.

To book an event, a promoter contacts the marketing department for the complex and explains the type of event being promoted, the expected attendance, and a requested date and time for the event. After reviewing the promoter’s application, the marketing department either accepts the application and places the event in the program of events, or rejects the application. Once an event application is accepted, the promoter is responsible for sending a seating plan for the event to the operations department of the complex.

The operations department reviews all seating plans and decides on the appropriate number of security personnel, cleaners and customer service representatives for the event. These estimates for staffing requirements are reviewed with the promoter before they are finalized. Once the staffing requirements are finalized, a deployment sheet is created by the operations department. The deployment sheet lists the start time, end time, location and skill requirements for every shift that will be required for the event. The completed deployment sheets are then passed to the human resource department, who is responsible for scheduling staff to work the event.

The human resource department creates a schedule two weeks in advance for all of the events to be held in the upcoming two-week period. The deployment sheets from all of the events in the two-week period are used to assign shifts to the employees. The scheduler works through an entire deployment sheet one shift at a time. Employees are required to fill out an employee availability card indicating their availability and skills. No employee can work over 30 h a week. In order for the scheduler to assign a shift to an employee, the employee must have the right skills and be available for the event. When the scheduler is finished this process, the final schedule is posted for the employees.

1.3 Diagrams for cases

1.3.1 OOA for Conference case

 

1.3.2 OOA for Event case

 

1.4 Pretest questions

Prior use of analysis methods

• Have you ever used object-oriented diagrams to model a business organization? Y/N

Familiarity with analysis methods

• How familiar are you with object-oriented diagrams? 0–7

Competence with analysis methods

• How competent are you in reading from object-oriented diagrams? 0–7

Confidence in analysis methods

• How confident are you in reading from object-oriented diagrams? 0–7

1.4.1 Pretest, knowledge of domain: Conference and Event case

1.4.1.1 Conference case

Please indicate your level of knowledge of the following businesses:

• -Organizing an academic conference: 0–7

Please indicate which of the activities listed below you have done:

• Attended an academic conference: Y/N

• Helped to organize a conference: Y/N

• Reviewed a paper for a conference: Y/N

• Submitted a paper for a conference: Y/N

• Created a mailing list for a large group: Y/N

1.4.1.2 Event case

Please indicate your level of knowledge of the following businesses:

• Organizing an entertainment event: 0–7

Please indicate which of the activities listed below you have done:

• Helped to organize an entertainment event: Y/N

• Worked at a large entertainment event: Y/N

• Promoted an entertainment event: Y/N

• Delegated work for a group of employees: Y/N

• Arranged security for an entertainment event: Y/N

1.5 Post-test questions

1. What did you like/dislike about the diagrams shown to you in this experiment?

2. Which did you find more useful, the diagrams or narration (written or verbal)?

3. Please provide any general comments you have in the space provided below.

1.5.1 Comprehension questions and answers

Comprehension questions and answers are provided here:

Comprehension questions			Comprehension questions
Conference case			Event case
No.	Question	Ans.	No.	Question	Ans.
1	Is the organizing committee responsible for preparing the final list of participants?	Y	1	Is the operations department responsible for deciding when events are to be held at the complex?	N
2	Does an author know if the program committee has accepted their paper?	Y	2	Does a promoter know that their event application has been accepted and placed into the program of events?	Y
3	Is the organizing committee responsible for developing the conference itinerary?	N	3	Is the human resource department responsible for creating the deployment sheets?	N
4	Does a participant have to be an author to be placed on the final list of participants?	N	4	Can an employee be scheduled on days they are not available?	N
5	Is the program committee responsible for maintaining a list of accepted authors?	Y	5	Is the promoter the person who determines the number of employees who should work an event?	N
6	Does the program committee offer invitations to participants?	N	6	Is the operations department responsible for creating the employee schedule?	N
7	Do the program committee and organizing committee have to interact with each other?	Y	7	Do the marketing department and the operations department share similar information?	Y
8	Does the program committee keep track of all the papers that are submitted to the conference?	Y	8	Is the marketing department responsible for keeping track of all of the events that will be held in the upcoming months?	Y
9	Are all of the papers that are submitted to the program committee accepted by the program committee?	N	9	Are all of the seating plans provided by the promoter accepted by the operations department?	U
10	Can an author also be a part of the organizing committee?	U	10	Do all employees work every event?	N
11	Does the program committee have access to the final list of participants?	N	11	Is the operations department responsible for creating a seating plan for every event?	N
12	Is a paper the first thing that an author sends to the program committee?	N	12	Is the seating plan directly used in the development of the employee schedule?	U

1.5.2 Problem-solving questions and example accepted answers

1.5.2.1 Conference case

Question 1

Is it possible for the same individual to receive two invitations to an IFIP conference? If you answer yes, indicate how this duplication might occur. (Provide as many suggestions as you can think of.) If you answered no, indicate why it cannot happen. (Provide as many suggestions as you can think of.)

Acceptable answers

• 1. Yes. An individual could be both in the potential participation list and the list of accepted authors.

• 2. Yes. Duplication in the potential participation list.

• 3. Yes. Duplication in potential authors participation list.

• 4. Yes. An author may have more than one paper.

• 5. No. If invitation is sent out at end of process (after list of accepted authors), then no duplication.

• 6. Individual was a special representative and an author.

• 7. Individual was a national representative and part of a related working group.

Question 2

Suppose that a person was invited to submit a paper but has not yet received an invitation. What could have happened? (Provide as many possible suggestions as you can think of.)

Acceptable answers

• 1. Person’s paper was rejected.

• 2. Invitation was supposed to be sent but there was a delay in processing.

• 3. Person did not submit an ‘intent to submit paper’ (no registration occurred).

• 4. Author has moved and conference mailing list has not been changed.

• 5. Reviewer did not return the paper.

• 6. The person never submitted a paper.

• 7. Program committee has not sent his name to the organizing committee.

• 8. The paper is still under review.

• 9. Paper was lost during review.

• 10. Person sent a note indicating they could not attend.

• 11. Submitted paper after deadline.

• 12. Two authors with the same name.

Question 3

Neither the program committee nor the organizing committee is directly responsible for maintaining the mailing list of potential authors and participants. What organizational problems do you think could arise from this situation? (Provide as many possible suggestions as you can think of.)

Acceptable answers

• 1. Inconsistencies between two mailing lists (organizing and Program committee lists).

• 2. Invitations sent to incorrect addresses. This may lead to poor turnout.

• 3. Redundancy (duplication) across mailing lists. Likely to invite same people twice.

• 4. Miss potential participants by assuming they are present on the other mailing list.

• 5. Receive information twice.

Question 4

Suppose that a very highly respected author indicated that she would like to present a paper in the conference but she has given the program committee only two weeks notice. What problems would be faced by the committees in accepting the author’s wish? (Provide as many possible suggestions as you can think of.)

Acceptable answers

• 1. Not enough time for review process.

• 2. Not enough time to generate invitation from organizing committee.

• 3. Requires change in the conference itinerary (which have been printed).

• 4. No time left in the conference itinerary.

• 5. Notices of change will have to be sent.

• 6. Sets a dangerous precedent. Other authors may want to do the same.

• 7. There is no process in place to support this (no exception handling).

• 8. Modify the list of participants (printed material must be changed).

• 9. No time to promote the author’s presence at the conference.

Question 5

Suppose that instead of inviting specific authors to submit papers and inviting only selected participants to come to the conference, the two committees agreed to open participation to any individuals and all interested authors. What problems would arise if the current system was used to organize the conference? (Provide as many suggestions as you can think of.)

Acceptable answers

• 1. Keep track of number of participants (by registering those interested).

• 2. Reviewers may be overwhelmed with papers. Screening papers may be necessary.

• 3. Would have to advertise for authors.

• 4. Would have to advertise for participants.

• 5. Would no longer need to send invitations or compile invitation list.

• 6. Larger event to organize.

• 7. Ticketing arrangement would be needed.

• 8. Two committees may not be necessary.

• 9. Keep track of accepted authors.

• 10. Budgeting for conference (how many?).

• 11. Loss of prestige.

1.5.2.2 Event case (questions only)

Question 1

Suppose that an employee was available to work on a Saturday but they were not scheduled to work. How could this have occurred? (Provide as many possible suggestions as you can think of.)

Question 2

Suppose that it is a very busy two weeks for the sport complex, with events scheduled for every day of the week. What problems might be encountered due to this busy schedule? (Provide as many possible suggestions as you can think of.)

Question 3

Suppose that a promoter underestimated the number of people who wanted to attend an event. The promoter now wants to increase the number of seats available for an event. Given that there are some seats that could be opened, what problems would be created by opening more seats? (Provide as many possible suggestions as you can think of.)

Question 4

Some events take a long time and a large number of people to set up and take down (this is referred to as conversion time). What organizational problems can be caused by events with a long conversion time? (Provide as many possible suggestions as you can think of.)

Question 5

Assume that the marketing department, operations department and human resources department act independently of each other. What problems might occur if the departments do not share their information quickly or work closely with the other departments? (Provide as many possible suggestions as you can think of.)