1 Introduction

The term user experience (UX) appeared more than a decade ago becoming, after a few years, a phenomenon, a field of study and a practice [1], a core for experts and researchers in the field of Human-Computer Interaction (HCI), and even penetrating other areas of knowledge such as psychology [2], medicine [3], learning [4], and advertisement [5]. At first, researchers and practitioners dedicated a few years to obtain a common definition of what is UX, what does it imply, how is it done, and what is its scope [6,7,8].

The ISO 9241-210 [9] standardized the concept of UX in 2010. Before this happened, for defining the scope of this emerging area, researchers and practitioners took into account different standpoints, which we can summarize as: (a) usability before UX, (b) UX beyond completing tasks, (c) UX as an emotional aspect, and (d) designing and evaluating for UX. Although there are other considerations on UX approaches, we will not address them in this paper.

  1. (a)

    Usability before UX: the traditional usability framework focused primarily on user’s ability to understand (user cognition) and to use (user performance) an ‘artifact’ in HCI [10]. Usability remained as a necessary condition in the context of interactive products and software, therefore the premise was “it is not sufficient to make a user happy”. Positioned in this way, most researchers argue that UX emerged as an extension of usability to accommodate the fuzzy quality attributes of experience such as enjoyment, pleasure or fun, whereas others argue that UX exceeded usability by including it [8].

  2. (b)

    UX beyond completing tasks: the pragmatic/hedonic model from Hassenzahl [11] gave the UX two dimensions for measuring it, differentiating from a focus on the product (pragmatic quality) and a focus on the self (hedonic quality), i.e., on the subjective side of the product [12]. This model links product attributes with human needs (i.e., personal growth, self-expression, and self-maintenance) and values (i.e., increasing knowledge, skills, and memories) [13].

  3. (c)

    UX as an emotional aspect: there is a common understanding of the holistic nature of UX as it makes emphasis on the emotion, motivation, and action, in a given physical and social context. In addition, UX is subjective; it focuses on the “felt experiences” rather than the product attributes [14]. In UX, there are two basic ways in dealing with emotions: one emphasizes the importance of emotions as consequence of product usage; the other line focuses on their importance as previous circumstances of product usage and evaluative judgment [13]. Thüring and Mahlke’s framework [15] explicitly defines emotional reactions as an integral component of the user experience and not as a consequence. On the other hand, Mandryk’s approach explored physiological data as a direct indication of UX through mathematically modeling emotion [16]. Both models measure physiological reactions; these measures are valued for being unobtrusive and therefore able to monitor, in a constant way, indices of emotion, instead of asking the users to stop their experience, reflect on their emotion, and then disclose it [17].

  4. (d)

    Designing and evaluating UX: both design and evaluation methods have interest in finding ways to evaluate UX of current concept ideas, design details, prototypes, or final products [18]. The primary effort of evaluation methods is to support and help in selecting the best design so development is on the right path, or to measure whether the final product meets and comply with the original UX targets [19]. Kort’s framework studies the sense-making process, the UX aspects, and their relationships with design elements intended to create specific experiences [20]. However, this is not an operational framework.

With the arrival of the ISO standard (9241-210) in 2010, and previous work on handling the standard since 2009 (e.g., [21, 22]), researchers and practitioners from the HCI community were divided. There are those who accepted the norm, stuck to it and still do it, and those who, based on their experience in academia and industry, built their own definition and work(ed) around it, always keeping in mind that the concept of UX is complex [1]. New researchers in UX area have seen in the ISO definition a safer way to start walking the long road built to date of what UX is and how we should do it.

According to the ISO 9241-210 standard, which was last reviewed and confirmed in 2015, the UX is defined as a “person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service”. This standard also clarifies that: first, the emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviors and accomplishments from users are included in the UX and they can occur before, during and after use. Second, UX is a consequence of three factors:

  1. (i)

    The interactive system characterized by the brand image, presentation, functionality, system performance, interactive behavior and assistive capabilities.

  2. (ii)

    The prior experience, attitudes, skills, and personality as causes for the user’s internal and physical state.

  3. (iii)

    The context of use.

Third, usability criteria, interpreted from the perspective of the users’ personal goals, can use perceptual and emotional aspects from UX to assess it [9].

As we have demonstrated, UX involves many aspects and different perspectives, so it is difficult to evaluate the whole set of what UX represents. Thus, to the best of our knowledge, there is no clear definition of the UX evaluation concerning the identification of the different evaluative aspects according to the perspectives, and even taking into account the final needs of UX evaluation according to a given context of use.

As a first step towards a clearer definition, we propose a conceptual framework that allows for identifying the difference between the evaluation perspectives and the measurable aspects in each of them. Eventually, this proposal will allow us to settle our measurement needs on those UX aspects that offer a lower risk of bias when carrying out the measurement and evaluation processes.

The remainder of this paper is organized as follows: in the Methods section, we present the qualitative method for building conceptual framework on which our contribution is based on. In the Results section, we highlight the proposed conceptual framework and the description of its usage scenario according to a contextualized problem (namely, the UX evaluation of a video game for children with specific learning disorders). The Discussion section explains the implications of the proposed framework and its comparison with other existing frameworks. Finally, we present the Conclusions section of this paper.

2 Methods

According to [23], a conceptual framework is a plane or a network of interlinked concepts providing a comprehensive understanding of a phenomenon, and it is not a simple collection of concepts; each concept plays a representative role. The proposed methodology follows eight key phases: (1) mapping the data sources, (2) categorizing the selected data, (3) identifying concepts, (4) categorizing concepts, (5) integrating concepts, (6) synthesis and resynthesis, (7) validating the conceptual framework, and (8) rethinking the conceptual framework.

  1. (1)

    Mapping the data sources: in this first part, sources of data were mapped according to the time division created by the ISO appearance:

    • Before ISO: data from workshops and special groups of interest from HCI conferences such as ACM CHI Conference on Human Factors in Computing Systems.

    • After ISO: the standard itself and papers with direct references to it.

    • Scientific databases like Scopus and ACM digital library for reviewing primary studies.

    • Papers on secondary studies [24,25,26,27,28].

    • The SQuaRE (Systems and Software Quality Requirements and Evaluation) series of standards [29].

  2. (2)

    Categorizing the selected data: to categorize the information found in the data sources, four aspects were taken into account:

    • UX design

    • UX evaluation of the user

    • UX evaluation of the system (product or service)

    • Quality evaluation of the system.

  3. (3)

    Identifying concepts: the identification of concepts was made explicitly and implicitly, naming those concepts that were “discovered”.

  4. (4)

    Categorizing concepts: in this phase, the main attributes, characteristics, assumptions, and roles from the concepts were identified and organized according to common features.

  5. (5)

    Integrating concepts: the aim of this phase is to integrate and group together concepts that have similarities through association of concepts or relationships.

  6. (6)

    Synthesis and resynthesis: the theorization process of the grouped concepts is iterative and includes repetitive synthesis, in order to recognize a general framework that makes sense.

Regarding phases (7) validating the conceptual framework and (8) rethinking the conceptual framework, precisely, the scope of this research paper is to validate the presented framework, discuss it with practitioners/researchers, receive feedback, and therefore, rethink the framework knowing that UX as a phenomenon is a dynamic area within the HCI field.

3 Results

In this section, aiming to synthesize the discovered information into one conceptual framework, we analyzed the results of the found data from the selected sources. The section has three parts: first, we define the main concepts in our framework by analyzing the domain of UX evaluation, taking as reference the ISO 9241-210 standard and relating and introducing other concepts derived from psychophysiology. Second, we formalize the conceptual framework relating its main concepts and their associations; third, we introduce a usage scenario for applying the conceptual framework: perform the evaluation of UX from the interaction between a video game for rehabilitation that focuses on memory and concentration therapies and children with specific learning disorders.

3.1 Main Concepts in UX Evaluation

We analyze the UX evaluation domain from two perspectives: (a) the system (product or service) and (b) the person (user).

  1. (a)

    From the system’s perspective, several authors declare to perform UX evaluations when developing usability tests during verification processes of software quality [27]; Miki’s evaluation framework of usability and UX [30] tries to separate objective measures represented in effectiveness and efficiency from subjective measures represented in perceptions and satisfaction. The author in [21] discusses, analyzes and conceptualizes in three different ways the relationship between usability and UX. From the software engineering standards, usability reduces UX to a simple aspect of it. Consequently, the pleasure measure in ISO/IEC 25022 [31] is equivalent to UX and the recommended methods to find out user’s pleasure are questionnaires.

  2. (b)

    From the user’s perspective, the main categories of the measurement methods to evaluate UX are: observational methods, where an expert estimates the user’s reactions; psychometric scales, in which the user evaluates himself/herself; and psychophysiological measures, measuring the body’s responses to determine user reactions. In the first two categories, we consider the methods as subjective, while in the third, the methods are objective [28].

Since the analysis of the ISO UX definition will provide us a good basis for a better understanding of UX, we take a deeper look into some of its main concepts, beginning with the “person’s perceptions and responses” from the perspective of psychophysiology. As entries of a psychophysiological process, we have the perceptions occurring in the nervous system [32], and the responses being the outcomes of this kind of process.

More precisely, psychophysiology is a research field that tries to understand the underlying mental reactions/functions (psychological responses) connected to the body signals (physiological responses) [33]. The central nervous system (CNS) and the peripheral nervous system (PNS) control the physiological responses: the CNS manages all the information received from the body, whereas the PNS includes all nerve cells external to the CNS. The PNS transmits most of the physical sensations; therefore, on the skin is easier to measure these reactions [34]. On the other side, responses related to emotions and mental workload of the users are the psychological ones, in its most basic definition [16].

3.2 Conceptual Framework for UX Evaluation

As part of the initial thinking of the proposed conceptual framework, the concepts and their associations took part in the plane facilitating the preliminary identification of the basics of the evaluation process in UX (see Fig. 1). Three sources acquired high relevance in the building of the framework: the ISO 9241-210 [9], the SQuaRE standards for Quality Measure Elements [35] and Measurement of quality in use [31]. Additionally, the research field of psychophysiology was relevant.

Fig. 1.
figure 1

Conceptual framework for UX evaluation

Full size image

The clarifying notes given within the definition of UX from the standards mentioned in Sect. 1, helped in the consolidation of the framework, especially in the designation of the causal relationships between the key concepts. Thus, in the first part of the here proposed framework, we declared that user and system are the cause of the UX, particularly the user’s states and the properties of the system.

A third cause of the UX comes from the context of use. We omitted this perspective within the conceptual framework, given that we assume the UX evaluation as being made or performed on tangible ‘assets’, that is, the user and the system, and although the context of use must have the same importance as the other perspectives, we consider it as something intangible and therefore, not measurable over the ‘real world’. It is also important to consider that we see the context of use as the interaction (not tangible) therefore as determinant defining the UX. Physical and psychological responses and perceptions represent the user’s states (internal and physical) as well as the six properties assigned to the system.

From the user’s perspective, psychophysiology collects these states. As psychophysiology is the “scientific study of social, psychological, and behavioral phenomena as related to and revealed through physiological principles and events in functional organisms” [33], we concluded it gathers measures of physiological signals and psychological events separately, in order to obtain its counterpart. Nowadays, psychophysiology emphasizes their research in the map of the relationships between physiological responses and psychological events; the most common case when performing physiological evaluation is where one body response could be associated with two or more mental effects or processes (the many-to-one relationship) [34].

According to Bernhaupt, from an HCI position, the overall goal of UX is to understand the role of affect as an antecedent, a consequence and a mediator of technology, hence, the concept of UX focuses rather on emotional outcomes [36]. Bearing this in mind, from psychological events such as behaviors, cognitive activity and emotions, we approached emotions from their own theory: we represented emotions in our framework inside the set of the ‘basic emotions’ defined by Izard [37] and Ekman [38]: surprise, sadness, anger, disgust, fear, contempt, pleasure and happiness.

In the case of the perspective from the system, and based on the quality requirements set forth in ISO 25020, we grouped the properties of the system. The measurement of quality requirements includes their assessment in the process, internally and externally, and when the system is in real or simulated use [29]. Based on the effect when a system is actually used, the quality assurance of systems comprises the quality-in-use measures. ISO 25022 typifies the quality requirements in use in terms of effectiveness, efficiency, satisfaction, freedom from risk and context coverage and, in turn, there are quality sub-requirements in three of these requirements; specifically: usefulness, trust, pleasure, and comfort from satisfaction requirement [31].

It is important to highlight that we can also find some of these concepts in the usability definition: “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [9], since this concept comes from the HCI.

Finally, in our conceptual framework, we reached a common point between both perspectives; this common point is the emotion and quality sub-characteristic “pleasure”: looking at it from the user, it is just another emotion included in a large set of variables to measure from the user and to take into account in the UX evaluation. On the contrary, digging a little deeper into the sub-characteristic from system’s perspective, it is the one variable defining all UX.

3.3 Usage Scenario of the Conceptual Framework

HapHop-Physio [39] is a project developed from a technology transfer process to support rehabilitation therapies for children with specific learning disorders in the city of Popayán, Colombia. Through a video game of memory and concentration, we encourage children to complete the therapies while having fun. Evaluation of the UX is particularly important in ICT solutions for health since users/patients must maintain the motivation to continue using the technology, in order to guarantee adherence to their treatments or interventions.

Given that our users are children in cognitive rehabilitation processes, the UX evaluation would become a key factor in understanding the results of the child’s therapy sessions. However, the measurement of aspects of UX is a difficult and exhausting task for test participants. With methods such as psychometric scales (like Fun toolkit or This-or-that [40]), children are easily distracted, are not impartial in the evaluation, and their reflection processes are not mature enough which would prevent the externalization of their feelings in a rational way: regarding observational methods, children tend to reject what is intrusive and can inhibit their natural performance.

Therefore, and for our case, we need an evaluation method that takes into account that end users are children, that we need measurements and responses that are as close as possible to objectivity (without risk of bias), and that allows children behave naturally in an environment that is closest to reality. In an extensive search of repositories (e.g., allaboutUX.com) and compilations in the literature of evaluation methods in UX, we found that psychophysiological measures are the closest to cover our needs. Nevertheless, studies using these types of evaluation methods have not done it so with children and the measurements are intrusive and sometimes annoying for users [27].

In the search for an objective evaluation method of the UX, we find in ISO/IEC 25022 [31] a guide that interrelates concepts of quality measurement in a simple and understandable diagram. Starting from this point and without any other method that was suitable for us, the built conceptual framework helped us in the definition of our own objective method (see Fig. 2).

Fig. 2.
figure 2

Adapted and modified from ISO/IEC 25022.

Objective method for the UX evaluation.

Full size image

Our method is simplistic: physiological signals are the convergence point between the physical state of the user and the measurement of emotions such as pleasure and happiness. Psychophysiology emphasizes their research in the map of the relationships between signals and emotions, and a convenient way to understand the potential of these relations is to consider a many-to-one relationship, where one body signal could be associated with two or more emotions in our case [33]. Particularly, several studies consider the electrodermal activity and the cardiovascular physiology, as the simplest of the signals and although, providing multiple resources in order to map emotions [41,42,43,44,45].

4 Discussion

In relation to the construction of the conceptual framework, we can argue in the first place that the proposed framework differentiates between the elements involved in an interaction: the user and the system. Additionally, it considers the context of use. Within the HCI community, some researchers and practitioners tend to overlap concepts, which, although they are difficult to separate due to their dependent nature, it is important to recognize their individualities and strengths in their singularity.

Second, this framework relates the different forms of UX evaluation from the user’s perspective. We take the process of expansion of the aspects to be evaluated in the UX to its “most basic” form (based on the selected concepts), allowing us to think about a whole lot of exploratory combinations to obtain the desired measurement. We can think about the observation of behavior or analysis of cognitive activity through psychological methods.

Thirdly, it relates the standards to determine the quality of the system from the user’s point of view because we base its conception on cascading concepts with general causality relationships. The levels of abstraction of the framework help in the easy connection of concepts of the user with its counterpart in the system to form the sense of the framework. This affirmation is also true with the definition of the objective evaluation method.

The framework also provides clarity regarding the procedure followed to evaluate the UX in a given context. Based on this fourth implication, we support the reason why we do not consider the evaluation of UX from the perspective of the context of use. Everything changes when defining that variable and we complement it with the recognition of the specific needs of each researcher and its project. This is also evident in the proposed evaluation method.

It attends as well as a conceptual and experimental basis for the exploration and subsequent implementation of a measurement procedure that allows obtaining objectively the UX in children with specific learning disorders, by measuring their physiological responses at the usage time of HapHop-Physio. This happening, once we are able to understand in a better way the links to its meanings and its related quality describing causes. We already took the first step towards the procedure by defining the objective method.

In previous sections, we made clear that there is a close relationship between usability and UX, which, according to Bevan, one way to conceptualize it is: “UX can be viewed as an elaboration of the satisfaction component of usability” [21]. Moreover, this is exactly what the ISO standard establishes the UX is, giving to researchers, developers, designers, and practitioners, wrong information about what UX constitutes for the HCI community. We prove that by reaching the common point in the conceptual framework. Correctly, many studies have shown with enough arguments that the relationship between usability and user experience is what Bevan defined as: “the umbrella term for all user’s perceptions and responses, whether measured subjectively or objectively” [46].

In comparison with our proposed framework, in the pragmatic/hedonic model of Hassenzahl, the research framework of Turing and Mahlke, the psychophysiological method of Mandryk, and the Kort’s framework we find some flaws. In the first place, Hassenzahl’s model is quite subjective; there is no room for objective alternatives for measuring the UX aspects. Turing and Mahlke’s framework takes into account system properties, user characteristics and context, however, fails in the specification of the causality relationships for the evaluation of the UX measurable aspects. Its framework counts aspects of the user and the system within the components of UX; however, they see the aspects from the user’s perspective, so it does not establish a dependency relationship between the user and the system.

In the same way, Mandryk’s method is context-dependent, which would not allow its instantiation to satisfy different variables of the context of use, the user context, and the system context. Lastly, Kort’s framework encloses UX design and evaluation processes without defining the scope of each one.

Finally, the conceptual framework helped us in the definition of the objective method of UX evaluation thanks to which it allows us to observe more clearly and simply the dynamics of the UX phenomenon, building a method that considers a real application scenario. As part of the future work, we will instantiate the objective method within the described usage scenario, by mapping the electrodermal activity and some cardiovascular physiological measurement with the E4 sensor from Empatica Inc., in order to obtain an objective approximation of the user’s internal state.

5 Conclusions

We proposed a conceptual framework in which we see in a more clearly and precisely way, the differences raised in the phenomenon of the UX evaluation seen from the two considered perspectives: the user and the system under consideration of the context. Likewise, the framework organized the most relevant concepts within the UX field, taking as reference several standards (for example, ISO 9241-210, ISO 25022) and relating other areas of knowledge. At the same time, the framework helped in the construction of an objective method of UX evaluation that takes into account our measurement needs and context of use.

We followed a qualitative method for building conceptual frameworks. The method included several steps: the mapping of the information sources, the categorization of the found information, the identification of the most relevant concepts, the organization of the concepts according to their common characteristics and similarities through their associations, and the iteration of synthesis processes. These steps led us to form a “network” or “plane” of concepts that provide a better overview of the phenomenon studied. From the proposed framework, we obtained an objective method, which we consider as a first step towards the validation of the framework.

The conceptual framework has several implications for the UX community: It

  1. (i)

    helps in differentiating between the elements involved in an interaction: the user and the system.

  2. (ii)

    relates the different forms of UX evaluation from the user’s perspective.

  3. (iii)

    relates the standards to determine the quality of the system from the user’s point of view.

  4. (iv)

    provides clarity regarding the procedure followed to evaluate the UX in a given context.

In addition, this approach attends as a conceptual and experimental basis for the exploration and subsequent implementation of a measurement procedure that allows to objectively obtaining the UX in children with specific learning disorders, by measuring their physiological responses at the use time of a video game for cognitive rehabilitation.