Determining the Usability Effect of Pedagogical Interface Agents on Adult Computer Literacy Training

Abstract

A large part of the population in developing countries is technologically ignorant. Pedagogical interface agents are pieces of educational software with human characteristics that facilitate social learning. The aim of this research was an attempt to evaluate the extent to which a variety of pedagogical educational agents could assist adult learners in acquiring basic computer skills. This was done by conducting a usability test in the context of South African adult computer literacy training. A hundred and three participants were randomly assigned to either a control group or a test group, where after all participants received Microsoft Office Word training (pre-test). Only test group participants were introduced to pedagogical agents (experimental treatment). During the usability test both groups were given tasks to perform. Findings showed that computer illiterate adult users could perform better during literacy training with the assistance of educational agents when compared to only being taught through traditional teaching methods. This could open the doors to more effective ways of reaching and teaching a larger group of previously educationally disadvantaged adults in order to give them a better chance at securing employment in the labour market.

Appendices

Appendix A

Post-test Questionnaire (without agents)

The purpose of this questionnaire is to elicit your personal opinions of the Microsoft Office environment you had worked with while carrying out different tasks for evaluation purposes. Please answer all the questions.

For question 1 to question 23 indicate rate your opinion on a scale of 1–5, place a circle around the appropriate number, where: 1 = “Strongly disagree”, 2 = “Disagree”, 3 = “Not sure”, 4 = “Agree”, 5 = “Strongly agree”
No.		SD	D	NS	A	SA
Learning contents provided by the microsoft office
1.	Microsoft office had functionalities I expected it to have	1	2	3	4	5
2.	Microsoft office environment was easy to use	1	2	3	4	5
Effectiveness of the microsoft office help function
3.	I found the microsoft office help function to be useful	1	2	3	4	5
4.	Microsoft office help function provided me with all the necessary information	1	2	3	4	5
5.	Microsoft office help function helped me to quickly learn how to perform a particular task	1	2	3	4	5
6.	Microsoft office help function helped me to recall the different steps involved for a particular task	1	2	3	4	5
7.	Microsoft office help function helped me to complete my tasks quicker	1	2	3	4	5
8.	Microsoft office help function was very practical	1	2	3	4	5
9.	Microsoft office help function assisted me to identify my mistakes when performing a task	1	2	3	4	5
10.	With microsoft office help function, it was quicker and easier for me to recover from a mistake	1	2	3	4	5
11.	I was able to understand the concepts better with the microsoft office help function than I would have without them	1	2	3	4	5
12.	With the help of microsoft office help function I have managed to develop new abilities	1	2	3	4	5
13.	Microsoft office help function helped me to feel more confident about my computer skills	1	2	3	4	5
14.	I trusted the hint from microsoft office help function	1	2	3	4	5
Satisfaction levels about the microsoft office
15.	The interactions with microsoft office were easy	1	2	3	4	5
16.	I was able to use microsoft office successfully	1	2	3	4	5
17.	I felt frustrated working with microsoft office	1	2	3	4	5
18.	I felt nervous when working with microsoft office	1	2	3	4	5
19.	I really had to concentrate to work with microsoft office	1	2	3	4	5
20.	It was exciting working with microsoft office	1	2	3	4	5
21.	Working with microsoft office made me change my attitude towards computers	1	2	3	4	5
22.	My experience with microsoft office encouraged me to learn about other computer programs	1	2	3	4	5
23.	Based on my experience with microsoft office, I can encourage my friends to learn about new concepts	1	2	3	4	5

24. What was the most difficult part when you worked with Microsoft Office? (You may tick more than one option)

Microsoft office environment was distracting
Microsoft office help function instructions were too difficult to follow
I understood very little from the microsoft office help function
Other, please specify:

25. What was the best part when you worked with Microsoft Office? (You may tick more than one option)

Easier to get information needed from microsoft office help function
Easy to understand and follow microsoft office help function instruction
Microsoft office help function instructions were straightforward
With Microsoft office help function it was easier to figure out how to perform a particular task
Other, please specify:

26. Did you enjoy being part of this study?

27. Provide reasons for your answer in question 26.

____________________________________________________________________________

28. How much did you learn from the study overall?

A lot	sufficient	Average	Poor	Nothing
1	2	3	4	5

29. Any general comments or suggestions:

___________________________________________________________________________________

30. I would like to participate in a similar research project in future.

31. If you answered ‘YES’ in question 30, please provide your cell phone number:_________________

Thank you very much for your input in this research.

Appendix B

Post-test Questionnaire (with agents)

The purpose of this questionnaire is to elicit your personal opinions of the agents you had worked with while carrying out different tasks for evaluation purposes. Please answer all the questions.

For question 1 to question 25 rate your opinion on a scale of 1-5, circling the appropriate number, where: 1 = “Strongly disagree”, 2 = “Disagree”, 3 = “Not sure”, 4 = “Agree”, 5 = “Strongly agree”
No.		SD	D	NS	A	SA
Learning contents provided by the agents
1.	The agents had functions and capabilities I expected	1	2	3	4	5
2.	The agents used a language that was familiar to me	1	2	3	4	5
3.	The agents’ hints provided all the necessary information	1	2	3	4	5
Effectiveness of the agents
4.	The agents’ hints helped me to quickly learn how to perform a particular task	1	2	3	4	5
5.	The agents’ hints helped to recall the different steps involved for a particular task	1	2	3	4	5
6.	The agents’ hints helped me to complete my tasks quicker	1	2	3	4	5
7.	The agents’ hints were very practical	1	2	3	4	5
8.	The agents’ hints assisted me to identify my mistakes when performing a task	1	2	3	4	5
9.	With the agents, it was quicker and easier for me to recover from a mistake	1	2	3	4	5
10.	I was able to understand the concepts better with the agents than I would have without them	1	2	3	4	5
11.	With the help of the agents I have managed to develop new abilities	1	2	3	4	5
Satisfaction levels about the agents
12.	The interactions with the agents were easy	1	2	3	4	5
13.	I was able to use the agents successfully	1	2	3	4	5
14.	I trusted the advice from the agents	1	2	3	4	5
15.	I found the agents to be intelligent	1	2	3	4	5
16.	I found the agents to be friendly	1	2	3	4	5
17.	I felt frustrated working with the agents	1	2	3	4	5
18.	I felt nervous when working with the agents	1	2	3	4	5
19.	I really had to concentrate to work with the agents	1	2	3	4	5
20.	It was exciting working with the agents	1	2	3	4	5
21.	The agents’ hints helped me to feel more confident about my computer skills	1	2	3	4	5
22.	Working with the agents made me change my attitude towards computers	1	2	3	4	5
23.	My experience with these agents encouraged me to find out more about them	1	2	3	4	5
24.	Based on my experience with the agents, I can encourage my friends to use them when learning about new concepts	1	2	3	4	5
25.	I would like to consider using agents when learning other concepts in real life	1	2	3	4	5

For question 26 to question 29 select whether the statement is true of false in terms of your preference
26.	I prefer a male agent to a female agent	True	False
27.	I prefer a cartoon agent to a realistic agent	True	False
28.	I prefer a dog agent to a human agent	True	False
29.	I prefer a text agent to a text and audio agent	True	False

For question 30 to question 32 select the one agent that you liked the most based on the criteria listed below
No.	Liking levels of the agents’ temperament
		Male text	Female text	Male Text and audio	Female Text and audio	Male cartoon dog	Female cartoon dog	Male cartoon human	Female cartoon human	Male realistic dog	Female realistic dog
30.	Appearance
31.	Voice
32.	Movement

33. For each of these agents, indicate whether you enjoyed working with them or not. Please provide suggestions for improvement.

		Enjoyable	Frustrating	Suggestions
33.1	Male text agent
33.2	Female text agent
33.3	Male text and audio agent
33.4	Female text and audio agent
33.5	Male cartoon dog agent
33.6	Female cartoon dog agent
33.7	Male cartoon human agent
33.8	Female cartoon human agent
33.9	Male realistic dog agent
33.10	Female realistic dog agent

34. Select your first choice of agent in terms of your overall preference and indicate it with an ‘X’ (select only one)

34.1	Male text agent
34.2	Female text agent
34.3	Male text and audio agent
34.4	Female text and audio agent
34.5	Male cartoon dog agent
34.6	Female cartoon dog agent
34.7	Male cartoon human agent
34.8	Female cartoon human agent
34.9	Male realistic dog agent
34.10	Female realistic dog agent

35. Please give a brief reason for your 1st choice rating in question 34 (e.g. 1st choice agent was friendly, intelligent, attractive, etc.).

1st _______________________________________________________________________________

36. What was the most difficult part when you worked with agents? (You may tick more than one option)

Agents were distracting
Agents were speaking too fast
I understood very little of what the agents said
Agents were saying the same things over and over again
Other, please specify:

37. What was the best part when you worked with agents? (You may tick more than one option)

Easier to get information needed
Easy to understand what the agents said
Agents’ help and hints were straightforward
Easier to figure out how to perform a particular task
Other, please specify:

38. Did you enjoy being part of this study?

39. Provide reasons for your answer in question 38.

_________________________________________________________________________________

40. How much did you learn from the study overall?

A lot	sufficient	Average	Poor	Nothing
1	2	3	4	5

41. Any general comments or suggestions:

___________________________________________________________________________________

___________________________________________________________________________________

42. I would like to participate in a similar research project in future.

43. If you answered ‘YES’ in question 42, please provide your cell phone number: _____________________

Thank you very much for your input in this research.