Heuristics to Evaluate the Usability of Ubiquitous Systems

Abstract

While the ubiquitous systems have characteristics that modify the way the user interacts with the systems, Human-Computer Interaction area studies forms of interaction, with usability being one of the main quality criteria. One of the methods used to evaluate usability is Heuristic Evaluation. In the case of ubiquitous systems, that have characteristics such as mobility and context awareness, Nielsen’s heuristics, which are widely used in conventional systems, do not focus on these particularities. Therefore, this work proposes specific heuristics to evaluate the usability of ubiquitous systems. Empirical studies and questionnaires were applied with experts in order to evaluate the proposed heuristics. The results point to improvements in both the way of conducting the evaluation and in the heuristics. From these results, the proposed heuristics were refined.

R.M.C. Andrade—CNPq Productivity Scholarship in Technological Development and Innovative Extension (DT-2).

1 Introduction

Ubiquitous systems are design to be present in people’s lives, helping users in their daily activities and providing access to information at any time and wherever the user may be (Hansmann et al. 2003).

Nowdays, it is common to see the same user with several devices and several applications interacting with them. However, we have to consider the following question: “Who wants to have hundreds of computers around constantly demanding attention and bombarding us with irrelevant information?” (Riekki et al. 2004). A system that does not efficiently help the users in activities of their daily lives, causing discomfort or insecurity, discourages the user to use it, presenting low usability, according to (Santos 2014).

Taking then the usability concept that consider usability one of the quality-of-use criteria that respond if the characteristics of the interactions and interface are adequate (Barbosa and Da Silva 2010). Usability is related to how easy it is to use the interface as well as user satisfaction due to the use of the system (Nielsen 1993). So, usability evaluation becomes a priority for ubiquitous systems since ubiquitous computing directly interferes with the user’s way of connecting to their systems.

One of the methods to evaluate system usability is the Heuristic Evaluation, an inspection method, created by Jakob Nielsen, where experts are guided by a set of usability principles, known as heuristics, to evaluate the elements of system’s interfaces.

To evaluate the usability of specific systems with heuristics, the aspects belonging to the domain of these systems must be considered for more effective results (Moraes and Rosa 2008). In the case of ubiquitous systems, it is necessary to adapt the existing Nielsen’s heuristics and to elaborate new heuristics to cover the specific characteristics of these systems, such as context awareness, transparency, attention, calmness and mobility (Santos 2014).

This paper aims to propose heuristics to evaluate the usability of ubiquitous systems, according to the particular characteristics of these systems. At first, a set of 13 heuristics was elaborated using a methodology based on Rusu et al. (2011). This set of heuristics was used in qualitative evaluations: an empirical study, which had the objective of observing the use of these heuristics in a practical evaluation, where experts performed heuristic evaluations; and a questionnaire, where other experts analyzed the heuristics and the process of creating them, reporting total agreement, partial agreement, disagreements and improvements to be applied.

This paper is structured as follows: Sect. 2 provides the theoretical basis: Ubiquitous Systems and Heuristic Evaluation. Section 3 investigates the work related to this research. In Sect. 4, the Ubiquitous Heuristics are elaborated through the execution of the methodology chosen for this purpose. Section 5 presents the questions and structuring of the qualitative evaluations and in Sect. 6 the results of the evaluations are presented and discussed. Finally, Sect. 7 concludes the paper, presenting the final considerations of this work.

2 Background

2.1 Ubiquitous Systems

Weiser (1991) describes his idealization of Ubiquitous Computing as follows: “Ubiquitous computing aims to improve computer use by making many computers available everywhere, but making them effectively invisible to the user.” He also says that “The most advanced technologies are those that disappear. In ubiquitous computing, computers will be embedded in the surrounding environment, creating a new paradigm of access and manipulation of information”.

Because the ubiquitous systems have differentiated characteristics, the evaluations carried out must take these factors into account. Bezerra et al. (2014) mention three challenges for usability testing in ubiquitous systems: (i) ubiquitous environments have more usability factors that should be evaluated, such as contextual information; (ii) most of the software measures do not consider the factors of ubiquitous applications; (iii) currently, usability testing methods follow the same activities performed in traditional systems.

In Santos (2014) are selected the following characteristics of ubiquitous systems as essential for evaluating the quality of human-computer interaction:

• 1. Context Awareness: corresponds to the ability to collect contextual information and use this information to make adaptations in the systems (Kourouthanassis et al. 2008).

• 2. Transparency: as said by (Satyanarayanan 2001), can be achieved by the proactivity of the system so that the user is minimally distracted.

• 3. Attention: In the ubiquitous environment, computers are hidden and replace user activities, thus enabling the user to focus on the various mental and physical activities such as walking, driving or other real-world interactions (Garlan et al. 2002).

• 4. Calmness: means free of distraction, excitement or disturbance. A quiet application is one that interacts with the user at the right time, only presents relevant information and demand the user’s attention only when necessary (Riekki 2004).

• 5. Mobility: in the ubiquitous computing era there is a search for “seamless” mobility that refers to the continuous or uninterrupted use of computing while the user moves through devices (Yu et al. 2013).

These characteristics were selected by Santos (2014) from the main existing definitions of ubiquitous systems to create measures to evaluate the quality of the human-computer interaction of these systems. Once the heuristic evaluation aims to achieve the quality of the human-computer interaction, these characteristics were selected to be part of the scope of the ubiquitous systems of this paper.

2.2 Human-Computer Interaction and Heuristic Evaluation

For Preece et al. (1994), the Human-Computer Interaction (HCI) area concerns the understanding of how people use computer systems to design new systems that better match user’s needs.

Usability is one of the main criteria of the quality in use of systems. It is related to the easiness of learning and use of the interface and to the user satisfaction in using it (Nielsen 1993). The ISO/IEC 9126 (2001) regulation was the first standard that defined the term Usability as “A set of attributes related to the effort required to use an interactive system, and related to the individual evaluation of such use by a set of specific users”.

To evaluate the usability of systems, methods are proposed to guide the evaluators during the evaluation to maximize the identification of usability defects. One such method is the Heuristic Evaluation.

In 1990, Nielsen and Molich proposed Heuristic Evaluation to find usability issues during the development of interactive systems. This method directs the evaluators to systematically inspect the system interface to identify problems that compromise good usability. The guidance is made by usability guidelines, called “heuristic”, which describes recommendations for interfaces and interaction (Barbosa and da Silva 2010). Nielsen’s heuristics are widely used to evaluate the usability of any type of systems.

Although the heuristics proposed by Nielsen and Molich are the precursors, Moraes and Rosa (2008) affirm that there are several lists of heuristics in literature, principles or ergonomic criteria that can be used for this type of evaluation. Nevertheless, these lists are generic, so adaptation is necessary to achieve a more effective result.

Barbosa and da Silva (2010) and Preece et al. (2002) recommend the heuristic evaluation in three stages:

• 1. Preparation: in this first stage are defined and organized the screens to be evaluated and the list of heuristics to be used. In synthesis, the evaluators are told what and how to perform it.

• 2. Data Collection and Interpretation (evaluation period): each evaluator individually inspects every screen to identify if the guidelines are being followed. If any guideline is violated, then it is considered a potential problem.

• 3. Consolidation and Report of results: at the end of the inspections, all of the evaluators meet to discuss the results and present a single consolidated report, according to the general consensus.

3 Related Work

In order to identify papers that relate heuristic evaluation to ubiquitous systems, our research was carried out through the ACM, IEEE, Springer, Scopus and BDBComp databases, seeking to answer the following research question: “Which existing papers use Heuristic Evaluation to evaluate the usability of ubiquitous systems?”.

The following search string was used: (“heuristic”) and (“evaluation” or “assessment”) and (“ubiquitous” or “ubiquity” or “pervasive”)

As a result, only one paper used Heuristic Evaluation to evaluate a ubiquitous system (Kemp et al. 2008). The authors of this study have developed a framework for the heuristic evaluation of the interface of an e-learning application, which is considered ubiquitous since the authors define the following characteristics for ubiquity: invisibility, usability, universality, and utility. The system evaluated was for desktop and a set of 18 heuristics was generated to evaluate web and ubiquitous systems but adapted to the needs of a distance learning system. They aimed to support learning by minimizing the visibility of the computer so the user could maximize the visibility of teaching content to the student. However, the evaluated system may have compromised issues such as mobility and context awareness, characteristics of the scope of ubiquitous systems of this paper.

As only one paper was selected in our previous research, a new research was conducted to cover other studies that have heuristics to evaluate the usability of mobile and/or context-aware systems. So a new research aimed to answer the following research question: “Which existing works have characteristics or heuristics to evaluate the usability of Ubiquitous systems, mobile or context-awareness systems?”.

The search string has been updated to (“heuristic” or “characteristic”) and (“evaluation” or “assessment”) and (“usability”) and (“ubiquitous” or “ubiquity” or “pervasive” or “context” or “mobile”).

In total, a new research selected 8 papers, from the reading of their titles and abstracts, to analyze the heuristics and characteristics of the ubiquitous systems. They are presented in Sect. 4.1.

One of these papers (Santos 2014) presents the basis of the characteristics that must be present when evaluating the usability of ubiquitous systems. In this paper, a systematic mapping was carried out to identify the characteristics that influence the quality of HCI in ubiquitous systems. The mapping found 134 characteristics, but there are duplicate characteristics with the same name and meaning and also characteristics with the same meaning, but with different names. In order to obtain the final set of characteristics, an analysis of meanings (semantic analysis) was performed considering what was written in the papers found by the systematic mapping and also in the classical literature of the areas involved. In the end, 26 relevant characteristics for HCI evaluation in ubiquitous systems were identified.

The other seven papers, (Bertini et al. 2006; Kemp et al. 2008; Varsaluoma 2009; Bonifácio 2012; Moraveji and Soesanto 2012; Inostroza et al. 2013; Machado Neto and Pimentel 2013), were analyzed to identify new characteristics not contemplated in Santos (2014). Table 3 shows all 31 characteristics captured using the related works presents in this section.

4 Ubiquitous Heuristics

In this section, we present the methodology used in this work to create the ubiquitous heuristics. Also, we present the initial set of 13 ubiquitous heuristics before the refinement step.

4.1 Methodology

For the creation of the ubiquitous heuristics, a methodology based on Rusu et al. (2011) was followed. Among the reasons for choosing this methodology, it is worth mentioning that is a generic methodology, from which heuristics for specific domains have already been created (e.g., virtual worlds systems, touchscreen-based mobile devices). It is also based on the characteristics of the domain application and indicates Nielsen’s Heuristics as the basis for the new heuristics.

One of the steps proposed by Rusu et al. (2011) is the Validation Stage (Step 5), where through heuristic evaluations performed in specific case studies new heuristics against Nielsen’s heuristics are compared. However, in this paper, the focus of the evaluation will not be comparing to the proposed heuristics, but to observe their use through an experimental study and to apply a questionnaire to experts, so this stage was adapted as “Evaluation Stage”.

The methodology used in this work involves 6 steps, summarized as following and represented in Fig. 1:

Fig. 1.

Methodology used for the creation of Ubiquitous Heuristics, adapted from Rusu et al. (2011)

• Step 1: Exploratory Stage - a bibliographical research is done to collect subjects related to the main topics of the research, such as heuristic evaluation and ubiquitous systems.

• Step 2: Descriptive Stage - the most important characteristics of the information collected in Step 1 are highlighted.

• Step 3: Correlational Stage - a filter is performed on the information obtained in Step 2 to identify the characteristics that the ubiquitous heuristics should have.

• Step 4: Explanatory Stage - formally specifies the initial version of the proposed ubiquitous heuristics, associating the characteristics present in each one and the process used for its creation.

• Step 5: Evaluation Stage - observes the use of the proposed heuristics applied in heuristic evaluations performed by experts. In addition, a questionnaire is applied with other experts to evaluate the process of creating the heuristics.

• Step 6: Refinement Stage - based on feedback from the evaluation stage (Step 5), the heuristics defined in the Step 4 are refined.

The following subsections present in detail the execution of steps 1 to 4 of Rusu’s methodology. Once the heuristics are defined in Step 4, the Step 5 of the methodology, which we adapted, is carried out by experts through qualitative evaluations (Sect. 5), and the ubiquitous heuristics are refined in Step 6, as we present in Sect. 6.

4.2 Step 1: Exploratory Stage

For the exploratory stage, a survey of the bibliography related to the research topics was performed, as presented in Sect. 3.

From the selected papers in the researchs carried out, as presented in Sect. 3, Santos (2014) systematically mapped the characteristics that influence the quality of human-computer interaction in ubiquitous systems. In addition to Santos’ work (2014), seven other papers, which include heuristics and characteristics of the ubiquitous systems, were selected: (Bertini et al. 2006; Kemp et al. 2008; Varsaluoma 2009; Bonifácio 2012; Moraveji and Soesanto 2012; Inostroza et al. 2013; Machado Neto and Pimentel 2013).

The selection of these papers, illustrated in Table 1, was carried out according to the reading of their titles and abstracts. After this selection, the full papers were read for content analysis and characteristic identification.

Table 1. Selected papers in the Exploratory Stage and the number of characteristics extracted in the Descriptive Stage.

4.3 Step 2: Descriptive Stage

The descriptive stage highlights the most important characteristics of the information collected earlier. From the complete reading of the selected papers from the Exploratory Stage (Table 1), characteristics related to the evaluation of usability in ubiquitous systems were extracted. Such characteristics must be present in the heuristics to be developed in this work.

From the eight selected papers, Kemp (2008) is present in the systematic mapping performed by Santos (2014), where a list of 26 characteristics was extracted to be present in HCI evaluations of ubiquitous systems. This list is called in this paper of “Santos’s List”.

The characteristics of the “Santos’s List” were selected for the elaboration of ubiquitous heuristics. However, the other 6 selected papers (see Table 1) did not enter into the systematic mapping performed by Santos (2014), because they were found due to the bibliographic research being performed with a focus on usability heuristics, which was not the focus of the mapping previously mentioned. Then, Kemp et al. (2008) the others 6 papers were read and 67 characteristics of usability were extracted to be present in ubiquitous systems to identify new characteristic not yet reported in Santos’s List.

4.4 Step 3: Correlational Stage

For the correlational stage, a filter was performed in the new 67 characteristics identified in the descriptive stage (Subsect. 4.3), since some terms were synonyms or were already present in Santos’s List, or were already present in Nielsen’s 10 Heuristics, which would be taken as the basis for the creation of the ubiquitous heuristics, proposed in this paper.

Table 2 presents a sample of the analysis performed. In the first column, there is the name of the selected work, in the second column there are the characteristics extracted from that work and in the third column it is said if: the characteristic is already part of Santos’s List; the characteristic is present in some Nielsen’s Heuristic; or whether it will be added to the final set of characteristics because it has not yet been identified in any of the previous options.

Table 2. Sample of the analysis of identified characteristics^a.

From the analysis, a final set of 31 essential characteristics were identified to be present in the usability evaluation heuristics of ubiquitous systems. A summary of the characteristics is presented in Table 3.

Table 3. Final set of 31 characteristics identified to be present in the Ubiquitous Heuristics.

It can be observed that the characteristics adaptation, data entry, flexibility, positioning of components and visualization of information were the new characteristics identified, all others belong to the Santos’s List.

Based on the list of 31 characteristics and Nielsen’s 10 heuristics, the proposed set of heuristics for the evaluation of the ubiquitous systems was created. This set is called Ubiquitous Heuristics.

To start the creation process, Nielsen’s 10 heuristics are taken as a basis, as suggested by Rusu et al. (2011). The description of each one is analyzed to verify whether it remains in the set of Ubiquitous Heuristics. For this analysis, the following processes have been defined: (i) Elimination, in which the heuristic is removed because it does not apply to ubiquitous systems; (ii) Junction, in which heuristics are united by addressing the same subject in ubiquitous systems; and (iii) Adaptation, in which heuristics are adapted to suit ubiquitous systems.

During the creation process, we also analysed each characteristic presented in Table 3 to identify which of Nielsen’s heuristics could be associated. Thus, each of these heuristics went through one of three processes and had one or more of the 31 characteristics associated with it.

Some of the inherent characteristics of the evaluation of ubiquitous systems were not contemplated by Nielsen’s heuristics, and therefore, new heuristics had to be elaborated using the definitions of these characteristics. The initial result is a set of 13 Ubiquitous Heuristics, which are presented in the next subsection.

Table 4 presents the “Ubiquitous Heuristics”, their relationship with the Nielsen’s heuristics and the process used to define that heuristics, as well as the characteristics present in each one.

Table 4. Relation between the Ubiquitous Heuristics and Nielsen’s heuristics, the creation process and the characteristics involved

4.5 Step 4: Explanatory Stage

This stage formally specifies the initial version of the set of the proposed heuristics. As illustrated in Table 4, the ubiquitous heuristics from HU1 to HU8 represent the heuristics that were created from the Nielsen’s heuristics analysis and the heuristics from HU9 to HU13 were created from characteristics that were not covered by Nielsen’s heuristics. The following are the heuristics and the characteristics involved:

• HU1 - Calm communication. The system should inform users of what is happening only when needed or when prompted so that it does not disturb the user in their current activity. It should present the exact information that the user needs at the moment, neither more nor less. Communication should be clear, objective and brief, using tone, appropriate emotion, and naturally calm elements. If it is an error message, the problem must be stated accurately and constructively suggesting a solution.

  • Characteristic(s): Calmness, Effectiveness, Attention, Utility, Simplicity.

• HU2 - Correspondence between the application and the real world. The system must speak the language of the user, with words, phrases, and concepts familiar to the user and in order to inspire trust. The conventions of the real world must be followed, making the information seem logical and natural.

  • Characteristic(s): Familiarity, Trust, Information display, Positioning of components.

• HU3 - User Freedom. The user should not feel pressured to perform any task, he has to have the option to interact or not with the application. When the application interacts with the user in a given context, the user should not feel obligated to respond to the interaction and should have the option to ignore it. The user can not feel controlled or overloaded by the application so that the user does not abandon it.

  • Characteristic(s): Calmness, Acceptability, User Satisfaction.

• HU4 - Consistency and standards. Application interfaces, ways of interacting or adapting to the context must be consistent and follow a standard to facilitate the user’s use of the system.

  • Characteristic(s): Ease of Use.

• HU5 - Error prevention. It is necessary to be familiar with the situations that most cause errors and modify the interfaces and interactions so that these errors do not occur. In addition, the application must be able to maintain its functionalities when used in adverse conditions.

  • Characteristic(s): Reliability and Scalability.

• HU6 - Customization. The application should give the users the ability to customize configurations according to their needs and experiences.

  • Characteristic(s): Flexibility.

• HU7 - Minimum need of help and documentation. Ideally the application should be so easy to use (intuitive) that it does not need help or documentation. If needed, the help should be easily accessible, focused on the user’s current activity. The aid guidelines should be objective and not too large.

  • Characteristic(s): Usability and Ease of Use.

• HU8 - Minimal effort. The application must easily reach the intended objective. Efficiently using the effort and resources required.

  • Characteristic(s): Efficiency, Ease of Use, User Satisfaction.

• HU9 - Mobility and Mobile Devices. Ubiquitous applications should suit the physical displacement of the user and the limitations of mobile devices. Aspects such as, wireless connection, connection between devices, small screen, limited hardware and memory capabilities, and limited power capability are factors that the application needs to circumvent and be natural to the user without causing inconvenience.

  • Characteristic(s): Device capability, network capability, interconnectivity, Mobility.

• HU10 - Privacy and Safety. The application must be able to keep the information protected, so that there is no risk of damage in a context of specific use. Information must be transported and stored securely, as well as the application’s access controls.

  • Characteristic(s): Privacy, Security, Safety, Availability.

• HU11 - Invisibility and Transparency. The system must be able to hide computational components so users do not worry about them. Interactions must take place through natural interfaces.

  • Characteristic(s): Transparency, Predictability.

• HU12 - Context Awareness and Adaptive Interfaces. The ubiquitous application should react according to the user context, the temporal context, and the context of the device. Interfaces must adapt to this context and bring only relevant information.

  • Characteristic(s): Context Awareness, Adaptation.

• HU13 - Sensors and data input. It must be verified whether the data input, whether given by the user or captured by sensors, is being effective and happening in a natural way for the user. The application should work properly in the presence of invalid inputs or stressful environmental conditions.

  • Characteristic(s): Data Input, Robustness.

5 Qualitative Evaluations

Step 5 of the methodology is the evaluation stage, which consists of qualitative evaluations of an exploratory nature. Initially, in Sub-sect. 5.1, we present the empirical study carried out using an observation method, in order to explore the use of the proposed heuristics in heuristic evaluations carried out in practice. In Subsect. 5.2, experts answered a questionnaire in what they analyzed the heuristics and the process of creating them, reporting total agreement, partial agreement, disagreements and suggesting improvements to be applied.

5.1 Empirical Study

The empirical study aims to evaluate the use of the proposed heuristics during the heuristic evaluations. In this subsection, we present the research question and how this study was organized.

Research Question.

What impressions, doubts, facilities, difficulties, and improvements are identified by experts when using the ubiquitous heuristics in a heuristic evaluation performed in practice?

• Objective: The objective of the study was to explore the ubiquitous heuristics proposed in practical situations of use, in other words, to apply in heuristic evaluations performed by experts, in order to look for new considerations to improve the heuristics and the evaluation method.

• Context: The study was carried out in two days. In the first day three experts participated, and in the second day, four others took part. All participants completed a personal questionnaire and signed a Term of Consent. To perform the evaluations, the following were available: (i) the ubiquitous heuristics for the experts; (ii) a space to write their considerations; and (iii) a brief presentation of the ubiquitous application to be evaluated. The experts were also asked to verbally express their doubts and impressions during the evaluation so that what was said was recorded and noted.

• Participants: Seven experts participated in the study. Before performing the evaluation, they answered a personal questionnaire with three questions, in which their answers should follow the scale: 1 = Has no experience with the subject; 2 = Knows the concepts, but have had no experience with the subject; 3 = The expert is familiar with the concepts and has had some experiences with the subject; and 4 = Expert in the subject and had several experiences in the area. Table 5 presents the profile of the participants in the study.

  Table 5. Profile of the experts of the empirical study.

We can observe that most of them are familiar with HCI concepts and usability evaluation methods, some a little more experienced and some of them just beginners. However, only one participant is an expert in Ubiquitous Systems. The other participants neither have experiences with ubiquitous systems nor are experts on the topic. Therefore, a training on ubiquitous systems was held to level the participants’ knowledge and to help them carry out the evaluation. In addition to the seven participants, there were two observers who acted as facilitators.

• Instrumentation¹: The instrumentation was done through a printed worksheet with the heuristics delivered to the participants. On the first day of the study, verification items related to the description of each of the heuristics were used. On the second day only the description of the heuristic was. In addition to the spreadsheet, on both days the material was available to report the result of the analysis (and possible problems identified) and smartphones with the GREat Mute² application installed. GREat Mute (Carvalho et al. 2015) is an application developed in the Android platform, whose main function is to leave the cell phone in silence according to the events of the user in Google Calendar. For example, whenever there is an event with the words “meeting” or “au-la” in the user’s Google Calendar, the mobile is automatically silent if the user is in the location defined in the event, on the day and time. For this, it is enough for the user to register the keywords in GREat Mute.

• Experimental draw: The study was carried out in three stages: 1. pre-evaluation training; 2. execution of heuristic evaluations; and 3. post-evaluation interview (Focus Group). In the Pre-evaluation training, the facilitator starts by explaining the purpose of the experiment and the steps to be taken. The experts were invited to complete a personal questionnaire and sign a Consent Term. After this initial stage, the facilitator explains the concepts of ubiquitous systems to level the knowledge of the evaluators in the subject. Next, the application to be evaluated, GREat Mute, is presented on a mobile device. Finally, the evaluation instruments are presented. In the execution of heuristic evaluations, the evaluation was carried out based on the spreadsheet provided with the ubiquitous heuristics, which serve as a guide for the experts to evaluate that application. The facilitators ask the experts to note the evaluation time and to state their doubts and thoughts during the evaluation. The results of the heuristic evaluation were reported through the analysis whether or not the application was in agreement with the heuristic. If it did not, the problem was reported in such a way that indicate: the location, the description and the severity of the problem. Throughout the evaluation, two observers remained with the evaluators, observing and noting the interactions, their doubts, and the identified problems. Finally, in the post-evaluation interview, the researchers interview the experts to identify their general perceptions about the ubiquitous heuristics and the applied evaluation.

5.2 Questionnaire

A second evaluation was carried out through a questionnaire with experts, aiming to evaluate the process of creating the heuristics and the characteristics associated with them, as well as the process of eliminating the Nielsen’s heuristic “Aesthetics and Minimalist Design”. In this section, we present the research question and how this study was organized.

Research Question.

What are the experts’ opinions about the process of creating the ubiquitous heuristics and the characteristics associated with them? What are your considerations?

• Objective. This evaluation aims at the analysis of experts in the process of creating ubiquitous heuristics and of the heuristics themselves.

• Context. A questionnaire was sent by e-mail to 4 experts. They were asked to analyze the process of creating each of the ubiquitous heuristics and give their opinions. After that, the questionnaire was answered and the results were analysed.

• Expert Profile. Table 6 presents the profile of the experts: a doctor, two doctoral students and a master’s student, all with more than three years of experience in applying methods to evaluate the usability of systems and with familiarity in the subjects of HCI and ubiquitous computing. Even though expert 4 was not familiar with ubiquitous computing, he recorded that studied previously the concepts and characteristics of ubiquitous computing to help in the responses of this evaluation.

  Table 6. Profile of the experts who participated in the Questionnaire evaluation.

• Instrumentation³. We have used the following instruments: a spreadsheet containing the objective of the evaluation; a set of instructions, which explained the whole process of creating the ubiquitous heuristics (as presented in Sect. 4) and how the experts should complete the questionnaire; the set of 31 characteristics, their definitions and references; a mapping, as shown in Table 4; and of the worksheet for each of the ubiquitous heuristics to be evaluated.

• Execution. For each ubiquitous heuristic, there was the corresponding Nielsen’s heuristic, the creation process, its definition, and the characteristics present in it. The evaluator had to answer whether (1) he agrees, (2) partially agrees, or (3) disagrees with any of these factors, and write their observations.

6 Results Analysis

This section presents the results obtained in the two qualitative evaluations: empirial study and questionnaire. Based on these results, we have applied the improvements to refine the initial set of ubiquitous heuristics (see Table 4) following Step 6 of the methodology.

6.1 Empirical Study

As the purpose of this study is to explore the heuristics and the stages of heuristic evaluation for ubiquitous systems, the results of the usability problems identified in the GREat Mute application will not be part of the data analysis of this experiment. The factors analyzed are the perceptions obtained by the experts during the heuristic evaluations. It could be observed that:

• The experiment lasted an average of 2 h, each day.

• The expert 2 had doubts about “HU1 - Calm Communication”, “It was not clear to me that this heuristic corresponds to feedback from Nielsen’s heuristics”, he said. Still on this heuristic, “Even though the application is ubiquitous, it is bad that it does not present feedback from time to time”, the expert 3 said.

• The expert 3 questioned the “HU9 - Mobility and Mobile Devices”: “ I could not evaluate the HU 9 items here inside the room”, the others also agreed. We observed that the evaluation was happening in a controlled environment and thus it does not give the participants the possibility of mobility, or change their context.

• We have also observed that on the first day experts were not attentive to the heuristic and its definition, just for the check items (the questions), they only read the questions without observing which heuristic they belonged, sometimes leaving the context of the item.

• The expert 1 suggested that users would use the application in their daily lives for a certain period, “Some heuristics with ubiquitous characteristics can only be perceived with the use in an outside environment, using day-by-day, for a while”. The expert 2 agreed, “I thought there was not room to evaluate the ubiquitous characteristics.” The expert 3 believes that to evaluate some heuristics, the Usability Test method is appropriate.

• The experts found the heuristics and steps clear but they could not evaluate all heuristics, for example, the expert 6 said “I suggest performing the experiment with other applications to look at all the ubiquitous heuristic”. Also the expert 7 suggested setting out examples of what could be a usability problem in each heuristic, to facilitate understanding, differentiating them.

• The experts who participated in the experiment on the second day said that they prefer to evaluate the application by describing the heuristics and not by checking items as a checklists, as reiterated by the expert 4 “Only the description of the heuristics fit the evaluation, I did not miss a checklist”.

• Although the researchers made it possible for experts to get around during the evaluation, they did not do so because they claimed they would need more time and more sites to perform the tests.

• Facilitators have observed that the ubiquitous heuristics worksheet provided were suitable as a guide.

With these observations, it was concluded that the heuristics worksheet was very efficient to assist the inspector in the inspection that the heuristics were clear and the stages of the evaluation were successful. However, the facilitators of the empirical study observed that not all ubiquitous heuristics could be evaluated. For example, the mobility heuristic, could not be evaluated because the study was done in a controlled environment. In addition, the experiment scenario did not help, ideally, experts should be asked to use the application days before the experiment to assess issues such as mobility and context. It is a lesson learned from this study.

6.2 Questionnaire

The results of the Questionnaire evaluation were analyzed. The main considerations of the experts and the improvements made before the analysis are presented as follows:

• The experts did not agree with the junction occurring in “HU1- Calm communication”, “The junction of these three Nielsen’s heuristics is inconsistent. This junction resulting in HU1 does not cover the aspects intended by the 3 HNs used and in many aspects is not related to them.”, as can be seen in the expert’s speech 3. Therefore, after analyzing all the considerations, we chose the refinement of HU1: for each of the Nielsen’s heuristics that had been joined by the junction process, would have a corresponding ubiquitous heuristic. For example, HN8 became HU8 and no longer HU1.

• The heuristic “HN6 - Minimizing user memory overhead” did not have its elimination and justification approved by the experts. The expert 3 said that “This heuristic should be adapted for use in ubiquitous systems, as the justification for elimination is not adequate. In ubiquitous systems transparency does not necessarily imply the total lack of explicit interaction.”. Expert 4 said that “This is the heuristic that is more relevant for adapting with ubiquitous systems for preventing exactly this minimization of user load.” Therefore, it was decided that HN6 would be adapted for refinement.

• The heuristic “HU 8-Minimum effort” was created to contemplate the characteristic Efficiency. However, after the experts’ comments, for example “I do not understand why this heuristic was formulated. What need does the ubiquitous system present for such purpose? This heuristic reminds usability as a criterion of quality. I did not understand why.”. This characteristic was allocated in other heuristics, then no longer the there is need for HU8. Therefore, for refinement, “HU 8-Minimum effort” has been eliminated.

• The heuristics “HU7 - Minimum need for help and documentation”, “HU11 - Invisibility and Transparency” and “HU 13 - Sensors and data inputs” were accepted in their entirety without any consideration by the experts.

• For the other heuristics, there were minor improvements such as changes in their titles (e.g., “HU6 - Personalization” for “HU7 - Flexibility and Efficiency of use”), in the descriptions and also some characteristics were reallocated.

Based on the presented results, the initial set of heuristics (see Table 4) was refined as will be presented in the next subsection. It is worth mentioning that the results obtained also contributed to improve the way to apply, in future evaluations, the steps of a heuristic evaluation in ubiquitous systems.

6.3 Refinement of Ubiquitous Heuristics

After the evaluations and analysis of the results, it is possible to conduct Step 6 of the methodology, “Refinement Stage”. The final version of the Ubiquitous Heuristics proposed in this paper is then presented as follows:

• HU1 - Visibility of system status. The system should always provide feedback to the user in response to an interaction performed. This feedback should neither disrupt the user in his current activity nor overwhelm the user with information, but must exist in the form of a noticeable change in some of the interaction modalities of the interface.

  • Characteristics: Calmness, Attention, Information display.

• HU2 - Correspondence between the system and the real world. The system must speak the language of the user, with words, symbols, concepts and interactions familiar to the user, instead of being system-oriented. One must follow the conventions of the real world, making the information appear logical and natural and easily reaching the intended goal.

  • Characteristic(s): Familiarity, Information display, Positioning of components, Predictability, Ease of use, Usability.

• HU3 - User control and freedom. The user must feel free to interact with the application or not. When the user wishes to interact with the system, must be in control, and at any time can abort a task or undo an operation and return to the previous state. When the application interacts with the user in a given context, the user should not feel obligated to respond to the interaction and should have the option to ignore it in order to keep the focus on their current activity. All of these actions must be clearly marked on the system and their visualization, if any, should maintain the standard throughout the application.

  • Characteristic(s): Calmness, Acceptability, User Satisfaction, Attention.

• HU4 - Consistency and standards. Application interfaces, data inputs, ways of interacting or adapting to the context, must be consistent and follow conventions and standards familiar to the user, so that the software can be understood, learned and used.

  • Characteristic(s): Usability, Predictability, Data Input, Familiarity.

• HU5 - Error prevention. It is important to know the situations that cause most errors and modify the interfaces and interactions so that users do not make these mistakes when interacting with the application. In addition, the application must be able to keep its services and performance always available when used by one or more users, under specific or adverse conditions.

  • Characteristics(s): Predictability, Flexibility, Reliability, Scalability, Security, Availability.

• HU6 - Recognition rather than recall. When there is a dialog or interaction available, the system should minimize the user’s memory load, leaving objects, actions, and options available to at least one of the user’s senses.

  • Characteristic(s): Information display, Usability, Predictability, Calmness, Attention.

• HU7 - Flexibility and efficiency of use. The application should provide shortcuts to accelerate the interaction, in order to reduce the effort required to achieve the intended goal, especially for the advanced user. Completeness of functionality must be maintained when using shortcuts or not. In addition, the system must be flexible, giving the user the ability to customize settings according to their needs and experiences.

  • Characteristic(s): Flexibility, Utility, Efficiency, Effectiveness.

• HU8 - Aesthetics and Minimalist Design. Dialogues should contain only relevant and necessary information, neither more nor less. Each extra unit of information in a dialogue competes with relevant units of information. The sequence of interaction and access to components and functionalities should be available depending on the context, in a simple and natural way.

  • Characteristic(s): Simplicity, Calmness, Attention, Positioning of components.

• HU9 - Help users recognize, diagnose and recover from errors. Error messages should be simple and expressed in clear language (without codes), accurately indicate the problem and constructively suggest a solution. In addition to texts, messages can be displayed in other formats available on mobile devices, such as image, audio, vibration. Error messages should guide the user with caution, without stress, so that the user does not stop using the system.

  • Characteristic(s): Simplicity, Calmness, Acceptability,

• HU10 - Help and documentation. Ideally, the application should be so easy to use (intuitive) that it does not need help or documentation. If necessary, the help should be easily accessible, centered on the user’s current activity. Help guidelines should be simple and objective.

  • Characteristic(s): Usability, Ease of use.

• HU11 - Mobility and Devices. Ubiquitous applications should maintain their functionality with the physical displacement of the user and on devices with different capacities. Aspects such as wireless networking, device connection, screen size, limited hardware capacity and power capacity are factors what the application needs to take into account to adjust during use without causing inconvenience to the user.

  • Characteristic(s): Device Capability, Network Capability, Interconnectivity, Mobility.

• HU12 - Privacy and Safety. The application must be able to keep the information saved and protected, so that there is no risk of damage in a context of specific use. Information must be transported and stored securely, as well as the application’s access controls.

  • Characteristic(s): Privacy, Secutity, Safety, Trust.

• HU13 - Invisibility and Transparency. The system must be able to hide computational components so users do not worry about them. Interactions must take place through natural interfaces.

  • Characteristic(s): Transparency, Predictability.

• HU14 - Context Awareness and Adaptive Interfaces. The ubiquitous application should react according to the context information that the user encounters. Interfaces must adapt to these contexts and bring only relevant information in a way that facilitates the use of the system.

  • Characteristic(s): Context Awareness, Adaptation, Ease of use.

• HU15 - Sensors and data inputs. It must be checked whether data input, either by the user or captured from the sensor, is being effective and happening naturally to the user. The application should operate correctly in the presence of invalid inputs or stressful environmental conditions.

  • Characteristic(s): Data Input, Robustness.

The new set of heuristics is summarized in Table 7. The table also shows a new relationship between the ubiquitous heuristics and the Nielsen’s heuristics, the process by which the ubiquitous heuristic has been elaborated (e.g., adaption or created) and the characteristics involved in each ubiquitous heuristic.

Table 7. New relation between the ubiquitous heuristics and the Nielsen’s heuristics, the creation process and the characteristics involved.

7 Conclusion

This paper presented heuristics to evaluate the usability of ubiquitous systems. From a methodology found in the literature to develop heuristics for a given domain, the heuristics for the evaluation of ubiquitous systems were specified. The six-step methodology of Rusu et al. (2011) was taken as a basis and executed. The first four steps were followed and, in the end, a set of heuristics was generated for this purpose.

An empirical study was conducted to observe the use of applied heuristics in heuristic evaluations performed in practice. In addition, expert’s analysis through questionnaires were carried out focusing on the evaluation of the creation of heuristics. Several considerations were noted during these evaluations and improvements were applied to better use of the ubiquitous heuristics.

Finally, a final set of 15 ubiquitous heuristics was presented to evaluate the usability of ubiquitous systems, refined from the considerations of the evaluations.

As future work, it is intended to apply the proposed heuristics in an environment, in which it is possible to test the various characteristics contemplated by ubiquitous systems.