Keywords

1 Introduction

The improvement of the quality of health services for people is a major challenge faced by countries worldwide. It is also one of the core Millennium Development Goals of the World Health Organization [1]. More than four million people on the planet are estimated to be affected by poor health services [1]. The health service crisis is the result of a lack of economic, social and educational policies on health quality for people in a number of countries. Improving the skills of the health workforce so as to meet the demands of the population is one of the key issues to promote health quality services. Thus, programs aiming at educating and training the health workforce will contribute to overcoming the health service crisis.

In line with this, the Brazilian Ministry of Health created in 2008 the Open University of the Unified Health System (UNA-SUS) to provide training to health professionals through distance learning courses. This virtual learning modality made the access to training courses possible for health professionals working in remote and rural areas of Brazil.

The distance learning courses offered by UNA-SUS are developed in partnership with a number of Brazilian universities, one of the most active partners being the UFMA - Federal University of Maranhão [2]. From 2014 to 2016 UNA-SUS/UFMA offered 55 distance learning courses to 179,301 health professionals, among them medical doctors, nurses and dentists. In order to improve the quality of their courses, a research on information design and interaction design was conducted. The aim of the research was to propose recommendations for the courses’ developers in these fields, and it consisted of a heuristic evaluation study, followed by users` experience (UX) testing. This article reports the heuristic evaluation study. This study made the identification and hierarchization of problems in the distance learning courses possible, providing the basis for the UX study.

2 Setting the Ground

The literature on human interaction with digital systems/artifacts has produced a number of approaches, frameworks and tools to analyze, design and evaluate such systems/artifacts [3, 4]. In the scope of education, principles and recommendations have been proposed for designing/developing learning applications [5, 6] to avoid the declining of users’ learning performances. Principles and recommendations are also intended to minimize learners’ cognitive load and to promote motivation. In this sense, Sera and Wong [5], based on Hashim, Ahmad and Rohiza`s principles [6], claim that an application should:

  • Be quickly learnable and user-friendly;

  • Often minimize scrolling;

  • Provide clear, simple and consistent navigation throughout its pages;

  • Keep the positions of similar actions and elements the same (e.g., buttons, icons);

  • Present flexibility of the display in the interface design;

  • Provide users with the necessary information only;

  • Favor the use of animation and of pictorial/graphic representations of information to textual information; and

  • Allow user control over the application.

Researchers have also been interested in understanding users’ experience with digital artifacts [4, 7]. In this regard, Garrett [4] has proposed a classification of the elements of users’ experience into five planes:

  1. 1.

    Strategy plane: the identification of users’ needs and the objectives of the system/artifact developed.

  2. 2.

    Scope plane: functional specifications of the system/courses, along with content requirements to be made available.

  3. 3.

    Structure plane: the information architecture (how the information nodes are interconnected) and the design of the interaction (conceptual model and predominant interaction style).

  4. 4.

    Skeleton plane: objects of the interface, the navigation system (how the user can move inside the system) and diagrams of the system screens.

  5. 5.

    Surface plane: the graphic and/or multimedia attributes that represent the information in the system to users.

According to Garrett [4], a design process which focuses on these planes seeks to ensure that aspects of the user’s experience with digital artifacts are taken into account. For that, it is necessary to employ the User-Centered Design approach to digital artifacts. Such an approach entails: (a) considering what users want to do, rather than what technology can do; and (b) designing for diversity and to better connect people in an effectively way [8]. In this sense, evaluation therefore, becomes fundamental to guarantee a usable system. Engaging users in the design process and/or conducting user testing of digital artifacts/systems are at the core of the User-Centered Design approach.

However, the evaluation of digital artifacts/systems by experts prior conducting user testing can also be a valid way to identify artifacts/systems’ drawbacks, making improvements in the testing material possible. Such evaluation prevents testing poor designed digital artifacts with users. It also allows foreseeing any problems/difficulties users may face when interacting with the artifact/system, which can be accounted for in the testing design. Thus, expert evaluation prior conducting user testing is beneficial to User-Centered Design research.

One of the most popular methods of expert assessment is the heuristic evaluation. Heuristic evaluation is the inspection of the usability of artifacts/systems by specialists, through a set of consolidated principles that lead to the discovery and resolution of problems [3, 9]. Heuristic evaluation may consider the artifact/system information architecture, interaction design and/or information design.

Information architecture refers to the specification of how users find information in the system, and it involves organization, navigation, labeling and search systems [10]. The navigation system specifies ways for the user to move through the informational space. The organization system determines how the organization and categorization of content are presented. The labeling system defines verbal (terminology) and visual (iconic) signs for each element of information and of navigation support to users. Finally, the search system determines the questions users can ask and the set of answers that will be obtained when making a query in the database [10, 11].

Interaction design concerns how actions are taken between users and the system, considering interdependent factors, such as the context of use, type of task and of users. Interaction design aims to reduce the negative aspects of the user experience and, at the same time, to improve the positive ones [12]. In this sense, ergonomic criteria may be employed to minimize ambiguities in the identification and classification of qualities and problems in interaction design evaluation of digital artifacts/systems [13]. The following criteria are considered of relevance for interaction design [14]:

  • User Guidance criterion: it refers to the advising and instructing resources offered to users when interacting with a system, such as alarms and messages.

  • Workload criterion: it considers elements of the interface that may reduce users’ cognitive and perceptual load, and may rise dialogue efficiency.

  • Explicit control criterion: it considers users’ control over the system when it processes their actions, as well as the system processing of explicit users’ actions.

  • Adaptability criterion: it regards the system ability to react according to context and to user’s needs and preferences.

  • Error management criterion: it refers to resources for preventing or reducing the occurrence of errors, as well as for recovering from errors, such as incorrect data entry, or entries with inappropriate formats.

  • Consistency criterion: it refers to ways to conform the elements/aspects of the interface design in similar contexts and to differ them when applied to different contexts, such as codes and formats.

Finally, information design considers the communication and optimization of information conveyed in print and/or digital artifacts/systems [15]. It embraces the analysis, planning, presentation and understanding of a message, taking into account its content and form [16]. The evaluation of artifacts/systems from an information design viewpoint should range from general to specific aspects of representing a content. Thus, the following aspects should be taken into account:

  • The general presentation of the content, that is its broad graphic relations and structure within and between pages, and overall use of color;

  • The presentation of the verbal mode regarding typographic aspects of the text (e.g., font size, column width, text alignment, use of whitespace);

  • The presentation of the pictorial mode (static and/or dynamic images) regarding clarity in depiction, graphic emphasis and text-image relations; and,

  • The presentation of the schematic mode regarding the clarity and emphasis of graphs, diagrams, tables and infographics.

To assess the information design aspects of content representation, principles and recommendations have been proposed, whether for the verbal, pictorial and schematic modes [17,18,19,20,21], or to the overall design of artifacts [22, 23]. These principles and recommendations aim at increasing quality of design projects, focusing on users in order to improve their interaction experience with artifacts/systems. Principles and recommendations are suitable for heuristic expert evaluation, and were, therefore, considered in the assessment of the UNA-SUS/UFMA courses in the scope of information design.

Lipton [22] and Pettersson [23] proposed information design principles concerning essential aspects of content communication that are relevant to digital artifacts/systems. They based their principles upon theories of perception and cognition allied to design good practices and research outcomes. Lipton [22] suggested the principles of Consistency in the way elements/relations are presented; Proximity between related elements; Chunking elements to provide groups of information at a time; Alignment of the elements of a page; Hierarchy of elements differing in importance; Structure of elements to organize the content; and Clarity in language.

Similarly, Pettersson [23] also suggested the principles of Clarity, Structure and Consistency (referred to as Unity) together with others, which he grouped into the categories: Functional, Aesthetics, Cognitive and Administrative. The Administrative category concerns financial, principled and legal aspects of the design process, which the author considered to be the principles of Information Access, Information Cost, Information Ethics, and Securing quality. The Cognitive principles regard the users’ domain, aiming to facilitate their Attention, Perception, Processing and Memory. On the other hand, the Functional and Aesthetics principles regard the designers’ domain and their decision making for message representation.

Since heuristic evaluations are conducted by experts, they are within the designers’ domain. Therefore, the Functional and Aesthetics principles are pertinent to heuristic evaluation on information design. The Aesthetics principles proposed by Pettersson [23] are Harmony and Proportion, which regard the arrangement of elements in a visually balanced manner. For the Functional principles, Pettersson [23] proposed the principles of Simplicity (conciseness and accuracy to improve reading flow and image understanding); and Emphasis (highlighting elements to attract, direct and maintain users’ attention), together with Clarity, Structure and Unit (consistency). It is worth mentioning that Pettersson [23] proposed ‘Defining the problems’ as a functional principle which regard the identification of the aspects/issues to guide and/or constrain the design process.

3 The UNA-SUS/UFMA Distance Learning Courses

The UNA-SUS/UFMA courses are similar in the technical and design aspects. They present a common visual language for illustrations, screen layout, menus and typographic resources for the texts. The courses are in the Moodle platform, and can be accessed by the users enrolled on a course. Users are allowed to manage their profile and interact with other users on the same course. The courses are structured in modules and units, which provide users with learning objects, such as e-books, videos, games and animations, as well as learning activities and virtual discussion forums. With the exception of the Moodle platform’s own profile settings, the other contents of the courses are accessed through external links.

UNA-SUS/UFMA offers postgraduate (specialization/diploma) courses, training courses and self-instructional courses. All courses are free of charge, and some are open to the public, as for instance, the community healthcare. A tutorial is available to get users acquainted with the course/system environment and the virtual rooms.

The development of distance learning courses at UNA-SUS/UFMA involves teams of experts from the fields of health (medical doctors, nurses), education (instructional designers, pedagogues), computer science (information technologist, programmers) and design (graphic designers, illustrators). The design of a course at UNA-SUS/UFMA is an iterative process. However, in general, the production of a course begins with the teams of health and education experts. These experts are responsible for developing the courses’ contents, particularly the texts to be presented on the courses. Depending on the topic of the course, the content developers may vary (e.g., cardiologists, nurses). Once the content of a course has been decided, it goes to the computer science and design teams. The computer science team is in charge of the technical aspects of the course, such as programming requirements and database resources. The design team is responsible for the interface design of the course, choosing/creating its elements (e.g., icons, illustrations, type fonts).

Despite the professional efforts of the UNA-SUS/UFMA’s teams to produce courses for effective learning, users/learners and information designers are not yet engaged in the development of the courses. Since user centered design approach and information design expertise are of relevance to produce useful artifacts/systems, neglecting them may compromise the effectiveness of the courses in communicating contents to users. The study reported herein is part of a research project that attempts to fulfil this gap.

4 The Study on Identification and Hierarchization of Problems in the UNA-SUS/UFMA Courses

A study on identification and hierarchization of problems in the UNA-SUS/UFMA courses was conducted to produce recommendations for improving their quality, allowing the design of appropriate material for testing in the UX study.

To define the scope of the study on identification and hierarchization of problems, the heuristic evaluation regarded the planes of Structure, Skeleton and Surface [4], of the UNA-SUS/UFMA courses. However, when and where necessary, the courses` developers were requested to provide information on the scope and strategy planes.

The heuristic evaluation was carried out by six researchers from the Federal University of Parana with expertise in information design and interaction design. The methodological procedures consisted of:

  1. 1.

    Overall examination of the system used for the courses to identify the main usability weaknesses of the system;

  2. 2.

    Delimitation of the course sample to be analyzed;

  3. 3.

    Selection of the heuristics and recommendations to assess the sample;

  4. 4.

    Evaluation of the sample through specific protocols; and

  5. 5.

    Hierarchization of the problems identified in the sample. The FIP (Frequency, Impact and Persistence) technique was employed to establish the severity of the problems.

4.1 The Sample and Selection of Heuristics

A sample of four distance learning courses was selected by the UNA-SUS/UFMA team for the evaluation. Each course offers an e-book as a supporting learning artifact, a virtual café, a notice board, and a forum of activities. Since the courses have a common visual language and system design, the four-course sample was then representative of the UNA-SUS/UFMA courses. The courses in the sample were the following:

  1. 1.

    Specialization Course: More Medical Doctors Program, Module 2, Health and Society, Unit 1, How to understand health?

  2. 2.

    Self-instructional course: PROVAB, Module 1, Sexual and Reproductive Health 1, Unit 1, Prenatal risk of habitual assistance in humanized childbirth.

  3. 3.

    Self-instructional Course: PROVAB, Module 3, Comprehensive Health Care for Children 1, Unit 3 Food and immunization.

  4. 4.

    Training Course: Portuguese Language, Module 1, Spelling and Accentuation, Unit 2, Use of the hyphen.

To select the heuristics/recommendations for the study, the following tasks were carried out by the researchers: change profile; access course units; visit each available link and, return to the home page. As a result, 88 heuristics/recommendations were selected to evaluate the sample, being:

  • 26 for information architecture regarding navigation, labeling, orientation and information search [11];

  • 27 for interaction design on user guidance, workload, explicit control, adaptability, error management and consistency [14]; and

  • 35 for information design concerning functional and aesthetic principles of content representation [23]Footnote 1. These regard the overall representation of content, and the presentation of the verbal, pictorial and schematic modes. Thus, they covered typographic aspects of text layout, static and dynamic images, text-image relationship, and information visualization.

The heuristics were arranged in rows in a table (evaluative protocol), and the columns were intended for the researchers expert judgment. The researchers should consider the heuristic to have been Complied (C), Not Complied (N) or Partially Complied (P) in each course. Figure 1 shows a detail of the table used for heuristic evaluation in interaction design.

Fig. 1.
figure 1

Detail of the protocol of the heuristic evaluation for interaction design

Full size image

4.2 Levels of Evaluation and Hierarchization of Problems

The evaluation of the sample was conducted at a macro level and at a micro level. At the macro level, general aspects of the virtual leaning environment of the UNA-SUS/UFMA courses were analyzed. This allowed an overview of the courses’ system and its structure, as well as the identification of problems and eventual disparities across the courses. At the micro level, a module and a unit of each course and the e-books of the sample were evaluated. This made it possible to identify specific weaknesses of the courses/e-books, particularly regarding information design aspects of content representation in the sample. The heuristics evaluation on information architecture and interaction design was carried out at the macro and micro levels, and the recommendations on information design at the micro level only.

After the heuristic evaluation, a syntheses of the problems identified at the macro and micro levels was generated to make the hierarchization of the problems possible. The FIP (Frequency, Impact, and Persistence) technique [3] was used for this purpose. Such technique aims at answering the following questions:

  • How often does the problem occur?

  • What is the impact of the problem?

  • How persistent is the problem?

To apply the FIP technique, protocols were developed. In each one of the protocols the researchers evaluated the problems identified, assigning a consensus score ranging from 1 to 10 for each of the three aspects: frequency, impact and persistence. The scores were, then, placed in the following formula so as to calculate the severity of each problem:

$$ \text{severity = }\frac{{\text{score(frequency)} \times \text{score}\,\text{(impact)} \times \sqrt {\text{score}} \,(\text{persistence})}}{{\sqrt {10} }} $$
(1)

In the FIP formula, the scores were in a scale of 0 to 100. The problems ranging from 100 to 70 are highest in severity; those ranging from 69 to 30 were intermediate in severity; and those ranging from 29 to 0.1 were lowest in severity for the UNA-SUS/UFMA courses. The problems which scored 0.0 in severity were disregarded and did not require recommendations for improvements.

The results of the heuristic evaluation and problem hierarchization were analyzed qualitatively. The qualitative approach was considered appropriate, since the results were to produce recommendations for the UNA-SUS/UFMA courses, providing improved material for the UX study. Thus, the numbers herein are only intended to show the occurrence of the results in each item evaluated, with no statistical purpose.

5 Results and Discussion

5.1 Heuristic Evaluation

At the macro level, the heuristics on information architecture and interaction design satisfactorily fulfilled were: 11 in information architecture, with a higher incidence in Navigation (N = 5) and; 13 in interaction design, with a higher incidence in Workload (N = 6). For example, the heuristics for actions/elements positioned in a logic manner on the screen, and adequate clusters of information presented per page to avoid overwhelming users with contents, were considered fulfilled in the four courses.

On the other hand, 11 heuristics on both information architecture and interaction design were not fulfilled. For instance, the heuristic for allowing different ways for searching was not fulfilled (N = 5), and neither were the 06 heuristics on consistency. Those partially fulfilled were 04 on information architecture (Navigation) and 03 on interaction design (User guidance, and Adaptability).

At the micro level, the heuristic evaluation on information architecture and interaction design was carried out for each course of the sample. This generated a total of 104 occurrences on information architecture (26 heuristics × 4 courses) and 108 on interaction design (27 heuristics × 4 courses) across the courses. The highest figure of heuristics satisfactorily fulfilled was 23 occurrences in both Workload (interaction design) and Navigation (information architecture). Navigation also presented a high figure of heuristics not fulfilled (N = 19 occurrences). Moreover, the sample has not fulfilled all heuristics on information architecture for Search (N = 20 occurrences); and those on interaction design presented the highest figure on User guidance (N = 15 occurrences).

The heuristic evaluation on information design (micro/e-book level) generated a total of 140 occurrences (35 heuristics × 4 courses). The highest figure of heuristics satisfactorily fulfilled was in the verbal mode (N = 14 occurrences) followed by the pictorial mode (N = 12 occurrences). The verbal mode had also the highest figure of heuristics not fulfilled (N = 10 occurrences), followed by the schematic mode (N = 07 occurrences). The pictorial mode also presented the highest figure of heuristics partially fulfilled (N = 17 occurrences).

5.2 Hierarchization of Problems

The results of the heuristic evaluation were ordered according to their degree of severity (FIP technique). A total of 52 problems were identified (Table 1): 13 high severity problems (N = 100 to 70), 22 medium severity problems (N = 69 to 30), and 17 low severity problems (N = 29 to 0.1). The problems of high and medium severity (N = 35) were considered to negatively affect the distance learning courses produced by UNA-SUS/UFMA.

Table 1. Occurrence of the degree of severity of the hierarchical problems (FIP)
Full size table

In information architecture the highest severity problems with a maximum FIP score (FIP = 100) were absence of navigation aid (e.g., site map); and of tools for searching didactic content. As for the interaction design, the most severe problems had to do with the lack of tools for customizing the course interface (FIP = 100), and for allowing users to resume reading the e-book (FIP = 90).

Finally, within the scope of information design, the problems of greater severity regard typographic aspects of text, text-image relation and pictorial representation of contents. Justified alignment of text, misuse of bold and italic, and poor hierarchy in headings (FIP = 80) were typographic deficiencies (Figs. 2 and 3) found throughout the courses and which may affect reading/legibility. Windows opening outside the page to display information (FIP = 80) were some of the weaknesses found in text-image relation (Fig. 4). These remove users/learners from the page context and may affect their attention to and understanding of information.

Fig. 2.
figure 2

Examples of inappropriate text alignment that resulted in ‘holes’ in the text lines indicated by the red dots in these screen shots (Copyright permission by UNA-SUS/UFMA). (Color figure online)

Full size image
Fig. 3.
figure 3

Examples of misuse of bold and italic; and poor heading hierarchy which are indicated by the red lines in these screen shots (Copyright permission by UNA-SUS/UFMA). (Color figure online)

Full size image
Fig. 4.
figure 4

Example of an unnecessary window to display the image caption. The red line/arrow shows the image caption in this screen shots (Copyright permission by UNA-SUS/UFMA). (Color figure online)

Full size image

Misuse of ordering cues in images/infographics (FIP = 72) that may lead users to erroneous assumptions about a pictorial sequence of events was also found in the courses. For instance, numbers were employed in an image related to prenatal care to show possibilities for calculating gestational age (Fig. 5), which are non-sequential information. Problems regarding picture style were also considered of high severity (FIP = 70). Flat and oversimplified images, lacking color contrast and contours seem not to be adequate to represent certain contents of the courses. They make visualizing elements/details of the images difficult, which may jeopardize comprehension of information. Figure 6 shows examples of poor visualization of elements in explanatory images due to the picture style used.

Fig. 5.
figure 5

Misuse of numbers to show possibilities for calculating gestational age which is non-sequential information (Copyright permission by UNA-SUS/UFMA). (Color figure online)

Full size image
Fig. 6.
figure 6

(Copyright permission by UNA-SUS/UFMA).

Examples of problems in picture style that make visualization of elements in explanatory images difficult

Full size image

Furthermore, the sample presented other weaknesses in the pictorial representation of information in the e-book pages. Different icons were used to convey the same message, indicating inconsistency in the use of icons across courses. For instance, the concept of ‘bibliographic references’ is conveyed by two icons (a quotation mark and a magnifying lens), whereas the concepts of ‘attention’ and ‘to learn more’ (Fig. 7) are both represented by the same icon: an exclamation point (!). In addition, some images were not related to the content presented in the page, and had a decorative function only. Although such images are usually intended to promote a pleasant visual experience (Fig. 8), they may distract users and affect their learning focus if not carefully placed.

Fig. 7.
figure 7

The same icons to represent ‘Attention’ and ‘To learn more’

Full size image
Fig. 8.
figure 8

(Copyright permission by UNA-SUS/UFMA).

Cartoonlike characters used in e-book pages of the sample

Full size image

The sample also presented cartoonlike characters (Fig. 8), possibly intended to make the contents more enjoyable to users, giving ‘lightness’ and ‘grace’ to their representation. This, however, may constrain the acceptance of pictorial representation by users, who may consider cartoonlike characters to infantilize contents, lacking the seriousness expected in adult learning environments.

6 Conclusions

In general, the outcomes of the heuristic evaluation of the UNA-SUS/UFMA course sample and the hierarchization of problems showed design flaws in the planes of structure, skeleton and surface [4]. Weaknesses were found in information design, information architecture, and interaction design. The courses/e-books were not in accordance with the information design principles [22, 23] of consistency in the way elements are presented (icons), proximity between related elements (text-image relation), and hierarchy of elements differing in importance (headings). In the domain of information architecture, the sample presented weaknesses in the navigation (navigation aid to users), labeling (ambiguous icons), and search (lack of search tools to the didactic content) systems [11]. Finally, in the scope of interaction design, the courses have not fully met the ergonomic criteria [13, 14] of adaptability (limited customization), explicit control (not allow users to go back to a page for continuing the reading), and consistency (variation in the presentation of clickable areas). The courses’ system also seems to fail when it comes to preventing/reducing and recovering from errors (error management criterion).

These deficiencies may compromise the communication and pedagogical effectiveness of the UNA-SUS/UFMA courses/e-books, thus, negatively affecting the user learning experience. Errors or confusion may occur during the interaction with the courses/e-books, and may result in difficulties in, or even lack of understanding of information. In order to improve UNA-SUS/UFMA courses and e-books, recommendations have been made on information architecture, interaction design and information design.

7 Recommendations and Final Considerations

Based on the outcomes of the study, 20 recommendations on information design, 16 on both interaction design and on information architecture were put forward to improve the UNA-SUS/UFMA courses. These recommendations have a flexible character so as to accommodate the variety of situations and contents to be represented. The recommendations should be followed by the courses’ developers, who are capable of interpreting/adapting them to the particularities of each content representation, resulting in useful design solutions. The main recommendations regard the high severity problems (FIP score = 100 to 70), which are shown in Table 2.

Table 2. Recommendations regarding the high severity problems found in the sample
Full size table

The heuristic evaluation followed by the hierarchization of problems with FIP technique have strengthened the research design by supporting decision making for UX testing. They have proved to be advantageous at the beginning of the evaluative research process, preventing the testing of material with flaws on information design, information architecture and interaction design. As a result, the UX testing protocols (e.g., usability tasks) were developed taking into consideration the problems identified in the study.

Moreover, this study ratified the importance of considering information design aspects in the production of UNA-SUS/UFMA distance learning courses, to improve their communication quality.