Keywords

1 Introduction

Usability heuristics, usually referred to as Heuristic Evaluation (HE), is one of the most widely used methods for evaluating the usability of an interactive system. Heuristic Evaluation became the most popular User Centered Design (UCD) approach in the 1990s, but has become less prominent with the move away from desktop applications [3]. This technique is an effective method to assess user interfaces by taking the recommendations based on UCD principles. These recommendations come in terms of design principles, heuristics, guidelines, best practices or user interface design patterns and standards [1] that are supposed to serve interface designers and evaluators.

Nevertheless, this technique has some pitfalls such as the need for adapting the heuristic set to the specific features of each interactive system. Thus, evaluators must combine different recommendations sources to review the specific application domain. This involves either choosing Nielsen’s list [8] (the most well-known and widely used), or a long reviewing process of guideline collections that inevitably causes conflicts between various resources [4]. In addition, each set uses different scoring systems to score the recommendations, so it is necessary to adjust it in the resultant set.

The process of combining different heuristic sets usually finishes up with an extensive list of duplicated recommendations, similar statements using different terms and potentially conflicting guidelines. Then, a clean-up and selection process require to provide a reliable, consistent and easy to use heuristic set [1].

Furthermore, our experience reveals that often (not to say almost always) when someone uses Nielsen’s list [8] is due to a lack of (deep) knowledge about it and, most worryingly, about the technique itself. This assertion is based on Nielsen’s statement: “these principles are broad rules of thumb and not specific usability guidelines” [8], making the list itself impossible to be used to evaluate.

At the same time, even HE is mainly a qualitative technique, some attempts for quantifying have appeared [1, 2, 6, 10, 12], among others. Or, other more ambitious such as work done by Masip et al. [4, 5], who provided a full framework to enable a semi-automatic process to provide the most adequate set of heuristics for each specific situation. It also classifies the heuristics in terms of different constraints (UX-degree) and enables an automatic classification of the problems found (removing the post-evaluation meeting done by the evaluators) for a better full process. Nevertheless, the rigorousness itself of this process makes it so complex that it is not widely used.

In this paper we propose to analyze the usability evaluation as an important technique. After consulting all main related sources, we will put the focus in two of the most known design principles. They will be precisely analyzed and mixed to propose a new list of principles to be used in the future.

2 Combining Common Heuristic Sets

2.1 Main Sources Consulted

Since that Schneiderman, in 1987, established his well-known Eight Golden Rules of Interface Design [11], and going via the no less well known Nielsen’s Ten general principles for interaction design [8] or Tognazzini’s First Principles of Interaction Design [7], several authors have designed new sets (usually modifying Nielsen’s list and/or adding new principles to evaluate specific aspects not covered) to help user interface designers and/or experts in their goal of enhancing usability of any kind of interactive system. A complete review of several sets of usability heuristics created for specific domains by different authors can be found in [10] (specifically in the appendix A of this reference).

Nevertheless, the truth (mainly in private companies) is that when almost everybody refers to evaluate usability or UX with heuristics they refer only to the Nielsen’s list.

In our case, after more than twenty years of experience involved in evaluation of interfaces, we decided to take Nielsen’s and Tognazzini’s lists to do this present work. We selected only these two because both are excellent references. So, there is no need to spend much time refining lists or providing specific new ones when experience evidences that they are not widely used. Nevertheless, as we said before Nielsen’s list needed something more to be useful at all and Tognazzini’s is too long and detailed to be wide used. It has been confirmed by our own experience, that leads us to complete a new proposal resulting from combining both.

2.2 Methodology

Inspired by recommendations found in Quiñones and Rusu [10] and Quiñones’ [9] recommendations, the process followed for deciding our list of proposal principles follows three steps: revision, compare similarities and integration.

Step 1: Revision of the Two Chosen Lists

The first step is to carefully read all the principles of the Nielsen and Tognazzini lists. The revision has been done in terms of understanding the deep meaning of each principle. Tables 1 and 2 shows the revised lists:

Table 1. Nielsen’s ten general principles for interaction design [8]
Full size table
Table 2. Tognazzi’s first principles of interaction design (revised & expanded) [7]
Full size table

Step 2: Compare Similarities

As the intention is to get a compact and complete solution for evaluating all kind of user interfaces. The first exercise to be done after the revision is to analyze and compare every principle from both chosen lists. This comparison is done by searching their similarities and try to group as many as possible. Our intention is to generate a new short as possible list that will include all design principles. It will look for all design characteristics, from both authors.

We will start this integration following the order of Nielsen’s list. Afterwards, we will continue following the order of Tognazzini’s.

The following paragraphs detail the entire reasoning and argumentation of this comparison:

  • N1 + T19 + T7 ➔ H1 - Visibility and system stateFootnote 1

Visibility of system status (N1) means that the system should always keep users informed about what is going on, through appropriate feedback within reasonable time. Feedback that we can assume as actions done to make navigation visible (T19).

Moreover, Tognazzini himself already relates discoverability (T7) and visible navigation (T19) characteristics, then we proposed to group all them into only one.

  • N2 + T11 + T14 ➔ H2 - Connection between the system and the real world, metaphor usage and human objects

With match between system and the real world (N2) Nielsen means that the system capability for speaking the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Idea that also could be expressed saying that human interface objects (T11).

Additionally, Nielsen completes N2 saying that the system should follow real-world conventions, making information appear in a natural and logical order. Conventions that are best represented by the uses metaphors (T14) of the interface because, metaphors generate users’ minds strong connections to their world, representing their conceptual model.

  • N3 + T9 ➔ H3 - User control and freedom

Under the principle of explorable interfaces (T9) we find characteristics such as the ability to provide users with well-marked paths and reference points so that they can freely control the interface.

Among its features we can also find the ability to prevent users from choosing system functions by mistake, or find an “emergency exit” clearly marked to leave the unwanted state without having to go through an extended dialogue, which is exactly the Nielsen principle user control and freedom (N3). Here, we propose Nielsen’s principle name, it is more descriptive.

  • N4 + T5 ➔ H4 - Consistency and standards

There is no need to argue anything to explain that these two principles can be merged into one: Consistency (T5) is part of consistency and standards (N4).

  • N5 + T2 + T13 ➔ H5 - Recognition rather than memory, learning and anticipation

This is a complex case. The heuristic proposed merges three different principles that, in essence, are the same. Nielsen’s principle, recognition rather than recall (N5) means that minimizing the user’s memory load he/she should not have to remember information from one part to another. This characteristic facilitates the system’s learnability (T2) and user’s anticipation (T13) to all the actions.

Then we propose to merge all them changing “than recall” (from N5) for “than “memory” to strong reinforce the other included characteristics.

  • N6 + T8 ➔ H6 - Flexibility and efficiency of use

Contrary to the previous case, this is easy to argue: efficiency of the user (T8) is clearly included into flexibility and efficiency of use (N6). Principles of both authors are looking for the user’s productivity, not the computer’s. Productivity that is enhanced by means of flexibility elements (i.e. keyboard accelerators) that often speed up the interaction for the user.

  • N9 + T1 + T17 ➔ H9 - Aesthetic and minimalist design

Aesthetics (T1) is included in first part of Aesthetic and minimalist design (N9) principle while simplicity (T17) can be assimilated within the second part of N9. Then our proposal here is to use Nielsen’s definition integrating these two from Tognazzini.

  • T15 + T18 ➔ H11 - Save the state and protect the work

Considering the reasoning of Tognazzini, State (T18) principle: “make clear what you will store & protect the user’s information”, it seems clear that to protect user’s work (T15).

  • T4 & T16 ➔ H12 - Colour and readability

Tognazzini refers to colour (T4) in two aspects: first in terms of accessibility for people that have some form of colour blindness and, second, in terms of vital interface element.

About accessibility he emphasizes about the need for providing secondary cues to convey the information to those who cannot see the colours presented. In the second case he highlights the importance of using colour, but it must be used in clear and readable foreground versus background combinations. Then, we can easily observe that readability (T16) is in the essence of both cases.

  • N7 ➔ H7 - Help users recognize, diagnose and recover from errors

  • N8 ➔ H8 - Error prevention

  • N10 ➔ H10 - Help and documentation

In all previous cases, no Tognazzini principle has been identified to be merged with Nielsen’s. Then, in these three cases, one Nielsen principle is used directly, without changes.

  • T3 ➔ H13 - Autonomy

  • T6 ➔ H14 - Defaults

  • T12 ➔ H15 - Latency reduction

Here there are those Tognazzini principles that remain alone, so, as before, they are used directly, without changes.

Finally, Fitts’s law (T10) principle has not been included in the final list. The reason for excluding is because, according to the author’s own words, it requires a stop watch users test. That is, like so much in the field of HCI, you must do a timed usability study with users to test for Fitts’s Law efficiency. Then, as this list of heuristics is created to be used for evaluators without users, it makes sense to not include it in this proposal.

Step 3: Integrate These Similarities

Previous step enabled us to propose a shorter as possible list of heuristics to be used to evaluate the usability of any user interface. Because it is an integration, the comparison has been done without losing the efficiency of both lists. We have seen that some principles convey to a new one while some others remain the same.

Here, in the integration, there is something as important as the list itself: the questions to be used for analyzing all the principles. Even having a list of principles, without a finite and precise collection of questions for every principle, is quite impossible to make a good usability assessment. For example, many people use Nielsen’s principles but, because they are too generalists, when analyzing, same people have misunderstandings about what it is, making lots of mistakes.

Table 3 shows all the principles generated in previous step with their corresponding set of questions needed to be answered when evaluating an interface. All the questions have been obtained from the original sources used, carefully analyzing all the aspects to be assessed underneath every new principle.

Table 3. Heuristic list proposed with all corresponding evaluation questions
Full size table

We can observe two important aspects to be considered in all the questions:

  1. a.

    each question should be able to be answered with a simple “Yes” or “No”, and

  2. b.

    if the answer is “Yes”, it will mean good usability for this characteristic and “No” means bad usability.

Both considerations are more significative as it might seem. Answering with a Yes or a No facilitates the evaluator’s response, minimizing the meaningless responses that are often done by the “fatigue effect” (as the evaluation is being carried out, due to fatigue of the evaluator, the rigor in the answers decreases).

And, giving the same sense to each answer also minimizes the errors, the evaluator knows from the beginning that a positive or negative answer always means the same.

3 Conclusion and Future Work

In this paper we aim to propose something risky but so ambitious as to propose a new heuristic list to go beyond the famous Nielsen’s one. One can think that it is not necessary but, in our opinion, it is necessary at all. With Nielsen’ list is not enough to conduct good usability evaluations.

Nielsen’ list is very popular because is short, clear and it looks easy to use it. Nevertheless, everyone used to do evaluations knows that this list is not enough, with only the information provided is almost impossible to do a professional job. The list has been unchanged since its creation, twenty-two years ago, nor does it have any specific questions to answer for each principle, making it confusing, completely depending on the evaluators intuition.

Other less popular but more precise and more updated is Tognazzini list. In this case, the list is less known and even less used because its length. Nevertheless, it is much more complete, with more principles and better explanations for every principle. When using this list, it leads to make fewer mistakes, but it is very much more time consuming. In this paper we have analyzed both lists to propose a new one from them.

The obtained list solves all the problems of the previous ones while maintaining all the good characteristics of both. The result is a fifteen principles list that covers all the characteristics enclosed by its predecessors. To be useful in usability evaluations, each principle is accompanied by a set of questions that can be answered with a simple “Yes” or “No”, conducting to increase the confidence of the evaluations.

The work here presented is the first one in a list of others that has the goal of providing a complete usability/UX evaluation methodology for modern and future user interfaces (UI). Our aim is to be able to improve the evaluations, making them easier and clearer that current ones.

At the same time, we also aim to be able to quantify the result of every evaluation, providing a number giving some sort of idea about the usability level of the UI evaluated. We want to find a method for obtaining a quantitative value (that can be named as Usability Percentage) that gives a numeric idea about how usable is the evaluated interface.

To achieve it, next steps to be done are: to propose a rating scale for every question, to quantify the values of this rating scale and to find a formula to provide this usability percentage. A sort of experiments will be the final stage to validate all the presented future steps.