Analyzing Methods, Instruments, and Tools for Evaluating the Customer eXperience

Abstract

There are multiple evaluation approaches (methods, instruments, and tools) that can be used for evaluating the CX0. However, these may not be considered by different companies/organizations because they assume that these are not related to the CX. This research analyze 29 evaluation approaches identified in a previous study (Rojas and Quiñones 2021) used in the areas of usability, user experience (UX), and satisfaction. We differentiate and examine these evaluation approaches indicating: (1) the type of participants required to apply them (experts or users); (2) the overall costs needed to use them (cheap or expensive); (3) some disadvantages or potential risks of them; and (4) the CX dimensions that could be evaluated. We found that: (1) most evaluation approaches (69%) require representative users rather than expert evaluators; (2) most evaluation approach (86,2%) are inexpensive to use since they do not need equipment or training; and (3) the most evaluated CX dimension corresponded to “sensorial”, while the least evaluated CX dimension was “emotional”.

Appendices

Appendix A: Participants, Costs and Disadvantages of Each Evaluation Approach

Evaluation approach	Participant	Cost	Disadvantages
Affinity diagram [14]	Experts	Cheap	•It requires having previously collected information •It may require a lot of time depending on the variety of information
AttrakDiff [25]	Users	Cheap	•Not much information is available about this instrument •The original publication is in Deutsch
Automated usability evaluation software [15]	Experts	Expensive	•Usually, these software are not free to use •Software with lack of flexibility and customization
Blind finger tracking [16]	Users	Cheap	•A screenshot must be taken each time the user performs a function on the mobile device
Card sorting [17]	Users	Cheap	•This method may not go deep enough •The results can be very variable and inconsistent
Cause and effect diagram [34]	Experts	Cheap	•It is difficult to represent the interrelated nature of problems and causes •It is not very useful to represent complex problems
Co-discovery [18]	Users	Cheap	•The researcher cannot control the direction of the discussion •Not all problems can be identified, only those related to defined tasks
Cognitive walkthrough [19]	Experts	Cheap	•Experts’ experience may cause them to overlook problems •Not all problems are identified, only those related to defined tasks
Critical to quality (CTQ) tree [34]	Experts	Cheap	•It depends on other methods to identify needs •It is not very useful to represent complex problems
Emocards [26]	Users	Cheap	•Difficulty in differentiating emotions by users •The method do not measure the actual emotion, it only measure the perceived pleasantness and arousal
Eye tracking [20]	Users	Expensive	•Eye tracking equipment and training can be expensive •It is difficult for users to control eye position accurately all times
Focus group [18]	Users	Expensive	•Participants usually know they are being observed so the answers might be dishonest •This method is not as efficient in covering maximum depth on a particular issue
Hedonic utility scale [27]	Users	Cheap	•The scale can be interpreted subjectively •Its results may vary depending to the mood of the participant
Heuristic evaluation [21]	Experts	Cheap	•The relevance of the problems identified depends on the experience of the evaluators •Its costs can easily increase if many expert evaluators are required
Interview [18]	Users	Cheap	•Participants usually know they are being observed so the answers might be dishonest •Its costs can easily increase making them expensive
Observations [22]	Experts	Cheap	•The observer had limited control over physical situation •The relevance of the data observed depends on the experience of the evaluator
Positive and negative affect schedule (PANAS) [28]	Users	Cheap	•The scale can be interpreted subjectively •Its results may vary depending to the mood of the participant
Post-study system usability questionnaire (PSSUQ) [29]	Users	Cheap	•There is not as much information available for this instrument as others
Quality function deployment (QFD) [34]	Experts	Cheap	•Categories are based on qualitative aspects and appears to be vague and not very clear
Questionnaire [18]	Users	Cheap	•Participants may give wrong or unanswered answers •Participants answers might be dishonest
Self-assessment manikin (SAM) [30]	Users	Cheap	•The scale can be interpreted subjectively •Its results may vary depending to the mood of the participant
SERVQUAL [31]	Users	Cheap	•It evaluates customer perception in a general way •It only focuses only on service delivery and not on the outcomes
Supply, Input, Process, Output and Customer (SIPOC) [34]	Experts	Cheap	•The tool is not applicable to all processes •Not very useful to represent complex problems
Survey [23]	Users	Cheap	•Participants may give wrong or unanswered answers •Its costs can easily increase making them expensive
System usability scale (SUS) [32]	Users	Cheap	•The instrument is a subjective measure of perceived usability •The instrument only provide quantitative data so it is difficult to know why participants assigned certain scores
Thinking aloud [18]	Users	Cheap	•Participants limit their responses because they are observed •Not all problems can be identified, only those related to defined tasks
UX questionnaire (UEQ) [33]	Users	Cheap	•Participants may have problems to interpret the items of the scale
Valence method [24]	Users	Cheap	•Participants should use the product or prototype for the first time during the evaluation •Its results may vary depending to the mood of the participant
Web usage analysis [15]	Users	Expensive	•There is not as much information available for this method as others

Appendix B: Association Between CX Dimensions and Evaluation Approaches

Evaluation approach	CX dimensions
	Sensorial	Emotional	Cognitive	Pragmatic	Lifestyle	Relational
Affinity diagram [14]	X		X		X	X
AttrakDiff [25]	X			X
Automated usability evaluation software [15]				X
Blind finger tracking [16]	X		X	X
Card sorting [17]	X		X	X
Cause and effect diagram [34]			X			X
Co-discovery [18]	X		X	X		X
Cognitive walkthrough [19]	X		X	X
Critical to quality (CTQ) tree [34]			X
Emocards [26]	X	X
Eye tracking [20]			X	X
Focus group [18]	X	X			X	X
Hedonic utility scale [27]	X	X			X
Heuristic evaluation [21]			X	X
Interview [18]	X	X	X	X	X	X
Observations [22]	X		X		X	X
Positive and negative affect schedule (PANAS) [28]	X	X
Post-study system usability questionnaire (PSSUQ) [29]	X		X	X
Quality function deployment (QFD) [34]			X			X
Questionnaire [18]	X	X	X	X	X	X
Self-assessment manikin (SAM) [30]	X	X
SERVQUAL [31]	X		X		X
Supply, Input, Process, Output and Customer (SIPOC) [34]			X			X
Survey [23]	X	X	X	X	X	X
System usability scale (SUS) [32]	X			X
Thinking aloud [18]	X		X	X
UX questionnaire (UEQ) [33]	X		X		X
Valence method [24]	X	X			X
Web usage analysis [15]			X	X
Total	21	9	20	15	10	10