Quali-Quantitative Review of the Use of Multimodal Interfaces for Cognitive Enhancement in People Who Are Blind

Abstract

Visual disability has a major impact on people’s quality of life. Although there are many technologies to assist people who are blind, most of them do not necessarily guarantee the effectiveness of the intended use. As part of research developed at the University of Chile since 1996, we investigated the interfaces for people who are blind regarding a gap in cognitive impact. We first performed a systematic literature review concerning the cognitive impact evaluation of multimodal interfaces for people who are blind. Based on the papers retrieved from the systematic review, a high diversity of experiments was found. Some of them do not present the data results clearly and do not apply a statistical method to guarantee the results. We conclude that there is a need to better plan and present data from experiments on technologies for cognition of people who are blind. Moreover, we also performed a Grounded Theory qualitative-based data analysis to complement and enrich the systematic review results.

Supported by Research Project of the National Fund for Science and Technology, Fondecyt-Chile #1150898, “Knowing with Multimodal Interfaces in Blind Learner”; as well as being part of the Basal Funds for Centers of Excellence, FB0003, CONICYT-Chile; and Fundação Cearense de Apoio ao Desenvolvimento (Funcap), that financed the master scholarship of author Lana Mesquita.

1 Introduction

In 2018, the “World Health Organization” [24] informs that 1.3 billion people are estimated to have visual disabilities worldwide, of which 36 million are blind. In face of this huge amount, visual disability has a significant impact on the quality of life of people, including their ability to study, work and to develop personal relationships. In this aspect, technologies have been designed to assist people who are blind to support daily life activities. These technologies work as aids to facilitate their independence, autonomy, and safety. Thus, such technologies help to improve the quality of life of people with visual disabilities and could stimulate and develop several skills, such as cognitive skills.

Even though there is technology specialized for people who are blind (e.g., serious game [3]), they are still using applications that are similar to older applications for the sighted population. For example, Battleship was one of the earliest games designed as a computer game with its release in 1979 [7]. AudioBattleShip, a version for both blind children and sighted playing together came in 2004 [18]. In general, the people who are blind have particular human-computer interaction needs, and the user interface should be suitable for them [19].

Considering these aspects, there are many efforts to develop accessible multimodal interfaces for people with visual disabilities, especially in multimodal games [3]. Despite this effort and in contrast to the visual interface evolution of games and applications for sighted people, interfaces for people who are blind explore other ways to interact with the user. In general, technologies for people with visual disabilities combine different sources of perceptual inputs and outputs. Although multimodal interfaces could help to improve the learning skills of people with visual disabilities, most of these technologies have been not wholly validated. Mostly, they remain in the prototype phase without being integrated into the people’s everyday life [6].

This paper is a continuation of the research developed and reported in the proposal [13] and of an initial version of the systematic review [14] yet without including the snowballing process.

2 Background

The impact evaluation of software could use several evidence-based methods. This work focuses on the experiments. To treat the cognitive impact evaluation is necessary to comprehend the experiment design in both cognitive psychology and software engineering areas. The two main theoretical background bibliographies used in this work are the books “Cognitive Psychology” [23] and “Experimentation in Software Engineering” [26]. In this multidisciplinary context, the next two subsections are dedicated to explain each point of view and highlight the differences.

2.1 Design Experiment in Software Engineering

The experiment process includes several steps: Scoping; Planning; Operation; Analysis and interpretation; Presentation and package [26]. This process provides a high level of control, which uses a formal, rigorous and controlled investigation. These steps were used to analyze the data of experiments in the methodology proposed. The main concepts involved in the experiment, shown in Table 1, are used to understand how the cognitive impact is evaluated.

Table 1. The main concepts of experimental design [26]

2.2 Design Experiment in Cognitive Psychology

The research methods in cognitive psychology focus on describing particular cognitive phenomena such as how people preconceive notions regarding what they may find while gathering data [23]. The characteristics used in this work of controlled experiments to explore cognitive phenomena are based on Sternberg and Sternberg [23].

Regarding the experiment concepts, the variables of the experiment are (i) independent variables, that are individually manipulated, or carefully regulated, by the experimenter; or (ii) dependent variables, that are outcome responses, the values of which depend on how one or more independent variables influence or affect the participants in the experiment [23]. This literature also presents the concepts of (iii) irrelevant variables, which affect the outcomes (dependent variable) when manipulated; (iv) control variables, which are held constant; and (v) confounding variables, which affect the dependent variables without be controlled or manipulated and should be avoided.

Independent and dependent variables must be chosen with great care, because what is learned from an experiment will depend almost exclusively on the variables one chooses to isolate from the often complex behavior one is observing [23]. The authors suggest two dependent variables that are used in cognitive psychological research: percent correct (or its additive inverse, error rate) and reaction time. These measures are popular because they can tell the investigator, respectively, the accuracy and speed of mental processing [23].

Among the myriad possibilities for independent variables are characteristics of the situation, of the task, or of the participants [23]. For example, characteristics of the situation may involve the presence versus the absence of particular stimuli or hints during a problem-solving task, as virtual versus real navigation [11]. Characteristics of the task may involve reading versus listening to a series of words and then responding to comprehension questions. Characteristics of the participants may include age differences, differences in educational status, or differences based on test scores. Characteristics of the participant are not easily manipulated experimentally due to the ethical regulation.

3 Systematic Literature Review (SLR)

In contrast to an ad-hoc literature review, the Systematic Literature Review (SLR) is a methodologically rigorous analysis and study of research results. To achieve our goal, the main research question for this first part of the study was: How is the cognitive impact evaluated on multimodal interfaces for people who are blind? For a better understanding, a second goal question was formulated “What are the challenges regarding the impact evaluation on multimodal interfaces for people who are blind on this scenario?”.

The process of a SLR includes three main phases: planning the review; conducting the review and reporting the review [10]. Each of these stages has a qualitative methodological design that aims to offer a better specification and evolution in the development of the SLR. Figure 1 presents the SLR process adopted in this study by using a UML language¹ for Activity Diagram. The process suits the guidelines from [10]. The next subsections describe the planning (the study selection criteria, the research sources selected) and the conducting phase (the search process, the data extraction form fields and the studies quality evaluation). The entire process was stored in an excel worksheet available online².

During all the SLR process, we used the tool StArt (“StArt”, 2016) and the software Microsoft Excel³ as a support to create the protocol, apply the filters, select the papers and show the results. We organized all references on software Mendeley⁴. As the papers retrieved from PubMed Central are in MEDLINE format, we developed the tool Medline2bibtex⁵. It works as a parser to permit the list to be read by both StArt and Mendeley.

Fig. 1.

Systematic literature review process (based on [4])

3.1 Planning: Definition of the Protocol

In the planning phase, we defined a review protocol that specifies the research question being addressed and the methods that will be used to perform the review [10].

Sources Selection. The first suggested digital libraries as sources are: ACM Digital Library; Engineering Village; IEEE Xplore; Scopus; Science Direct; Springer Link; PubMed; Web of Science; Google Scholar, that includes the leading conferences and journals from Computer Science.

The bases chosen are the primary research bases for scientific articles in the research area or the bases that index them. Since Scopus, which index ACM Digital Library and IEEE Xplore, is based on the same database of ScienceDirect. As the research includes visual disabilities, we expected to find some works in PubMed database. The final list of sources to SLR is Scopus, Springer, Web of Science and PubMed Central. We also researched in the Journal of Visual Impairment & Blindness that addresses a variety of topics related to visual impairment. The research in this journal changes a little the string applied to get their proceedings, that are in the scientific base PubMed Central.

Studies Initial Selection. The initial selection occurs by applying a search string into in each source. Before the final version of the string, we simulated many other strings in searching the best solution and results with the principal papers of the area. The final string defined was the string bellow.

Study Selection Criteria. The inclusion and exclusion criteria are according to the goal of the SLR. Table 2 presents the inclusion (I) and exclusion (E) selection criteria. To be accepted, a scientific paper must cover all inclusion criteria and none exclusion criterion.

Table 2. Inclusion and exclusion criteria

The technologies defined in the I.01 criterion include mobile application, computer software, IoT systems, virtual environments or a video game with multimodal interfaces. Also, we accepted technologies that are not specifically for people who are blind or visually impaired with the goal of expanding the results, but the studies present the technology focused on users with visual disabilities. We excluded from all technologies that uses Sensory Substitution Devices (SSD) [16], which substitutes a sense as sensory augmentation, bionic eyes, retinal visual prosthesis, cortical implants and others. This definition is important to plan the methodology proposed and to delimit the focus.

We defined the studies type in the E.03. This criterion excludes all studies type different from primary studies that present technology for people who are blind and its evaluation. We accepted articles, conference papers, short papers, and book sections. This criterion includes documents that have the minimum information to understand the evaluation. We did not cover books because the information is dispersed inside them.

The E.04 criterion defines the scientific articles must be in English, because it is the mandatory language for the main events and scientific journals in the search area. And they must be published between 1st January 1998 and 2nd August 2017. The year 1998 was a milestone due to the paper [12] which concerns with 3D acoustic interfaces for blind children and is the first study known.

3.2 Conducting

Search Process. The conducting phase starts with the initial search in the scientific bases proposed. The string was applied on the metadata of papers, which includes abstract, index terms, and bibliographic citation data (such as document title, publication title, etc.).

The first filter excluded papers duplicated and document types out the scope due their format (E.03). The second filter identified which paper is in and out the scope by reading their titles and abstracts (E.01). A lot of papers were excluded in the first filter because the scientific base PubMed Central (PMC) brings a lot of medical papers focused on disease effectiveness and specific medical statements. Even though the area of this study is computer science, we decided to insert the PMC in the bases’ list due to the nature of the subject.

Next, in the third filter, we evaluated each retrieved paper in its entirety (E.02). If necessary, besides the entire text, we searched more about the technologies and processes described, as project and institutional websites, videos, newspaper articles and others. In the fourth filter, we searched and compared the experiments to find the same experiment described in two or more papers. This occurs when the experiment is not the primary goal of the paper, and more than one paper cites the experiment methodology and results according to the paper goal.

Once we have chosen the select papers, we extracted all data required to achieve the objective. The organization of the data generates data synthesis. The main reason for withdrawing papers in the last filter was the evaluation performed is out the search and at most times related to the system performance, e.g., sensor performance evaluation.

In face the final papers selected, we make a snowballing approach for an opportunistic search for other relevant papers. Snowballing is a manual search using the reference and citations (known as backward and forward snowballing) list of a paper or the citations to the paper select aiming to identify additional papers [25]. Thus, we performed other interaction among the forward and backward snowballing list to select the papers reading the title firstly and abstract and after the whole text.

In the snowballing process, the acceptance rate was high compared to the mainstream search. This acceptance rate is due to that the articles are closely related to the theme of tools for people who are blind and usually use similar processes of validation building.

Figure 2 resumes the conducting phase results. The spreadsheet⁶ details the complete list of selected articles and the process.

Fig. 2.

Filters in the systematic review process

Studies Quality Evaluation. Each paper was qualified into the following quality checklist to assess the studies and measure the weight of each study found on the results (Table 3).

Table 3. Quality assessment form

The results of quality form range from 52% to 96% of expected quality, with average of 74%. The expected quality means the maximum points in each question and none misses in the first question.

Data Extraction Form Fields. The data extraction was designed to answer the main and second questions and to understand the context in which each paper is inserted. We divided the data collected into three categories: (i) General, (ii) Research and (iii) Empirical. The general category comprises bibliographic information. Table 4 shows the data extracted and the categories.

Table 4. Form for data collection

The general category comprises bibliographic information and classifies the papers. We classified the experiment of a scientific paper into two classifications: research type, based on [15], which can be validation research, evaluation research, solution research, philosophical research, opinion paper or experience papers. The empirical method classification is based on the classification of [1]. This classification aims to confirm the papers retrieved are in the search focus, since we look for papers in the “evaluation research” type and that are “Experiments”. Although, we retrieved one paper as a Case Study. For this one, we consider only the experiment data. All papers retrieved are in Evaluation Research category.

The research category comprises the classification that fits the technology presented in the key features of multimodal interfaces for the cognition of people who are blind [3]. This classification is divided into 4-dimension: Interface, Interaction, Cognition, and Evaluation; and it is applied to video games and virtual environments. For our purpose, we classified only in the interaction, interface and cognition dimensions. We covered in the classification more than video games and virtual environments, since we also found these features present in the technologies selected. The research category also shows other strategies used to evaluate, as usability evaluation.

The empirical category provides information specifically about how the empirical method that evaluates the impact of the cognitive impact. This category will be explored in the results section.

4 Grounded Theory

The Grounded Theory [5] analysis was performed to enhance and strengthen the findings of the SLR regarding cognitive evaluation concepts used in the context of this search: multimodal interfaces for people who are blind. The data gathered from the literature became the population used in the analysis. We aimed to analyze in the deepest level of generating theory but as the authors state: knowledge and understanding take many forms [2]. The Grounded Theory analysis was supported by the MAXQDA12⁷ tool in all process, which is composed of the following steps: planning, data collection, coding and reporting results.

In Grounded Theory analysis, we used the method to analyze in detail four items from Empirical category: Hypothesis, Variables, Measures, and Tasks. The final map produced by the Selective Coding phase represents the main idea of the cognitive evaluation in the context of this study and connect all elements.

The planning step aims to identify the area of interest and the research question that will drive the work. In our case, the area of interest is the Cognitive Impact Evaluation. In the current research, the Grounded Theory analysis suits well some characteristics of cognitive impact experiment extracted from the systematic review, as it assists in the interpretation and clarification of the results found.

In the data collection step, we prepared an Excel spreadsheet with data from the SLR. We import the data extraction form from each experiment into the MAXQDA. In this way, each experiment is a document in the MAXQDA analysis. All data from experiments are imported as variables (59 variables), that could be used to quantify your qualitative analysis results or to add additional information to pieces of data. These one are already imported as excerpts coded as the empirical categories. The data retrieved is organized and modified from the paper to answer the data form. Although, in some experiments, we added some excerpt from the paper to facilitate the coding step on that experiment.

The coding step, the heart of the Grounded Theory analysis, is composed of (i) Open Coding, (ii) Axial Coding, and (iii) Selective Coding. On this step, we extract concepts from raw data and relate them to each other until reaching a core concept [27]. In our case, we pursued to extract and relate some concepts the experiments of cognitive impact evaluation. After considering all possible meanings and examining the context carefully produced 91 codes and 808 tags. Figure 3 shows the 6 top categories of the codes.

Fig. 3.

Open coding

In the open coding, constant comparative analysis is a regular procedure to execute it. Whenever coming across another excerpt that seemed to talk about the same concept or shared a common attribute, these were grouped together into the same code.

Axial Coding is stepping to relate concepts to each other [27]. With this step, the fractions of data from the open coding can be reassembled and organized into the categories and subcategories with their descriptions, properties or dimensions. The Axial Code produced 95 codes and 603 tags.

The Selective Coding merge all concepts grounded in the process and others captured in the SLR. As a result, we produced maps of concepts and a map to ground the theory of the cognitive impact evaluation in multimodal interfaces for people who are blind or visually impaired.

5 General Results

Despite we considered as the final result of the SLR the papers selected in the fourth filter, in this analysis of general data, we consider the 58 papers selected in the third filter. Regarding the empirical method, almost all papers use an Experiment as an empirical method (21 papers). Only 1 paper present a Case Study as an evaluation, that justifies the choice due to the small number of participants [21]. This information was encountered in the text explicit or it was deducted from the details presented.

Among the papers selected in third filter, 30 are conference papers, 27 are scientific journals, and one is book sections. The main conferences where the papers selected are published are ACM SIGACCES with 6 papers, ICDVRAT with 5 papers and UAHCI with 3 papers. The leading journal with three papers retrieved is the International Journal on Disability and Human Development.

6 Research Results

About other Strategies of the evaluation applied, many papers apply another approach to evaluate other criteria not covered in our study. Usability evaluation is the most used assessment besides impact evaluation.

6.1 Classification

The result of this classification is shown in Fig. 4. As we can see, the Keyboard is the main mode to interact with the technologies analyzed. It is an instrument of interaction already common as input device [18]. The keyboard does not generate more complexity and expenses, like the Novint Falcon device, that is used in [20]. The Novint Falcon is one of the Force Feedback Device that promotes a Tactile and Kinesthetic Feedback. Mouse and Natural language are less used. The mouse is replaced by buttons or other specific devices for a better interaction, as shown in [22]. The most common Feedback is the Sonorous and the main Audio Interface used is the Iconic Sound, which are sounds associated with each available object and action in the environment [20]. These characteristics are important to understand which kind of interfaces are assessed and how the impact is evaluated on them.

Fig. 4.

Key features in multimodal interfaces (evaluation)

7 Empirical Data Results

The Empirical category aims to answer the research question of the SLR in detail. The data retrieved ground the comprehension of the main research question. Thus, we searched the main aspects concerning the empirical method applied. Notice that in this category, we analyze the 47 papers selected on the fourth filter that brings 52 experiments process since five papers have two different experiments. In addition to the SLR (SLR) method, we used the Grounded Theory method to analyze the empirical data. With this combination of methods, we constructed a concept based on cognitive assessment data. Figure 5 shows the data were chosen to be retrieved and which method had been primarily used.

Fig. 5.

Empirical data extracted from experiments

7.1 Sample

In the sample data analysis, we looked for the sampling strategy and the samples description, including the number of participants (per condition) and the kind of participants (e.g., computer science students). The sample information retrieved from each experiment are divided into 4 categories: number of users, blindness level, gender distribution and age range⁸.

From the number of users to the onset age of blindness, there are many sample combinations in the selected experiments. The sample choice includes previous experience required to do the task and others characteristic controlled in the experiment, like disabilities, the onset of blindness, the etiology of a visual impairment or the presence of another disability. The quantity of users varies as shown in Fig. 6. Most of the experiments (22) are applied to 6 to 10 users. One experiment does not inform the number of users. One experiment [8] expresses the small number of the sample (4 blind participants) as a limitation/implication of the research. Moreover, the paper talks about the limited access to people who are blind, and those who use a smartphone are even more difficult to find.

There is no a guideline about the number of users on an experiment sample in the context of this research. The tradeoff between limitations and quality of the research should define the number of users. The planning phase must propose a plausible solution with the group of participants that are sought to the experiment have a reasoned result, preferably based on statistical data.

The rage of age varies a lot among the experiments. To found a pattern, we adopted age groups based on indications of the Brazilian Child and Adolescent Statute (ECA) and the Brazilian Statute of the Elderly, which defines four groups: child (under age12), adolescent (between age 12 and under 21) , adult (between age 21 and 59) and elderly (over 60 years old). Figure 6 also shows the distribution among age groups.

Fig. 6.

Age distribution by groups

The gender distribution in samples is equilibrated in most cases in the 36 papers that describes this information. The mean of gender proportion, among all samples, is 47% for women and 53% for men.

The distribution between the blindness level is varied. There are experiments where the sample is all formed by people who are blind, and there are samples formed only by people who are blindfolded. The blindness level distribution between the samples is an essential information in the context of the experiments studied here. Although 9 experiments do not inform the proportions of blind, low vision, sighted and blindfolded.

7.2 Instruments

The instruments used in the evaluation had the objective to identify some user ability controlled on the Experiment (as an independent variable), e.g., the mathematics knowledge test or is used to guide the evaluation process, e.g., observation guideline to assess O&M skills. There were 37 experiments that described the instruments used. Among the instruments used, there are 5 Likert-based surveys, 20 checklists (which include guidelines and specific tests), 15 questionnaires (which include surveys), 8 modeling kits (which are manual instruments, as pen and papers or bricks), and 9 logs (which include, in addition to the system log, the video and audio logs). None instrument has been used in more than one study experiment from our list. We speculated this is due to the distinct natures of the skills evaluated.

7.3 Statistical Methods

As a result, in 3 experiments the statistical method was not described or explicit in the text, neither in the references. T-test, which uses statistical concepts to reject or not a null hypothesis, is the most used, followed by ANOVA (one, two and three-way) and Person’s Correlation, that is used to analyze the variation between groups.

The statistical method used depends on the research goal and the data acquired of the experiment. Thus, we prefer not to present the methods according to the type of analysis, such as hypothesis analysis or correlation between variables. Instead, we focused on if the experiment data analysis has any statistical method applied. Our objective was not to discuss how to apply the best method, but to analyze the evidence based on statistical confirmations. The two types of statistical methods highlighted are variables correlation and hypothesis tests.

7.4 Resource Data

In general, the papers do not present resource data of the experiments in the text. 19 experiments describe some information about the resource, as time-period, costs or human resources. Among this information, the time-period presented varies from 2 days to 6 months with the mean of 3 months-period. There were 4 experiments that explain some information about cost, two of them chose some technology due to their low cost. One experiment says pay $25 per hour to participants, and another pays $60 for 3-hour sessions. One paper describes their team to apply the experiment. Almost always there is little or no information about the resources, this can hinder the replication of the experiment, as well as it lacks evidence to the descriptive text about adequacy, limits, qualities, costs and associated risks.

7.5 Ethical Concepts

The ethical concepts also are not well covered in the papers; even it is an essential step to produce an experiment with people who have disabilities [26]. Among papers that threats the ethical concepts, 8 papers mentioned signing consents, two of them also applies to stop rules to enforcing ethical concept; 6 papers present the ethics council approved by legal institutions; two experiments express some information concern the user safety. It should be considered the local laws of ethical concepts and also the published year for each experiment.

7.6 Hypothesis and Groups

Hypothesis is the basis for the statistical analysis of an experiment. The hypothesis is the core of the experiment design. Seeking to understand the hypothesis, we selected which experiments have explicit in the text the Hypothesis and which of them test it. Even if the paper explains the research question or the goal of the experiment in the text, we did not count them. Wohlin divides the Hypothesis formulation from the Goal definition as two different phases of the experiment process. The two codes derived from this category is Explicit hypothesis and hypothesis well explicit. The “Hypothesis well explicit” (4 papers) is coded when the paper presents the hypothesis structured very well, as the experiments from “Model of Cognitive Mobility for Visually Impaired” (11 papers).

7.7 Variables

Almost no paper presents the variables explicitly in the text as dependent variables and independent variables. Thus, along with the text, we looked for these variables to understand how the experiment is designed. After the grounded theory analysis process, we produced three categories of dependent variables and three categories of independent variables. In searching this, we can see that the variables are strongly related to the experiment hypothesis or research objective. As a result, the Code Map⁹ presents the code map generated and next we explain each variable structure.

As cognitive impact evaluation, the dependent variable is related to cognition processes. The main category of dependent variables is the task performance that has many ways to measure (explored in the Sect. 7.9). We coded as skills all dependent variables that work with a cognitive skill affected by some independent variable. Problem-solving skill stands out in the experiments. Only two papers use some user emotion, as user opinion.

The independent variables are related to domain knowledge. The independent variables encountered are characteristics of the situation, the participants and the tasks. Generally, the characteristics of the situation and tasks are factors, while participant characteristics are variables controlled, as etiology of blindness or age of participants. Nevertheless, usually, the levels of blindness are factors in the experiments used to divide the sample into groups.

Beyond the independent and dependent variables, we encountered in the grounded data other topics about the variables. There were 11 experiments with their variables well described in a separated section in the text. There were 8 experiments that work with a control group to compare with the treatment. Also, three experiments stand out concern about the confounding variables. Report these topics show more rigor in the controlled experiments, moreover it is not a usual practice.

7.8 Tasks

The tasks explored in the experiments are related to the technology assessed. The Code Map have five categories¹⁰. The two main categories grounded in the experiment data are: Maps and Problem-Solving, already discussed in the Variables section. There were 31 experiments that work with maps applications aiming to develop Orientation & Mobility skills and other related skills. Not all presented in detail the tasks and measures, but we divide the Maps category into: (i) location (indoor or outdoor maps), (ii) number of spaces (one or more spaces), (iii) known spaces (new or public spaces), (iv) type (virtual or real map tasks), (v) tasks that the participant need to reproduce a map using some modeling kit, as bricks, and (vi) tasks that use the clock [17]. Notice that the same task of the experiment could have more than one code.

7.9 Measures

The measures are defined according to the variables of the experiment. We found five categories in the Measures code map grounded in data of the experiments¹¹, that are measures based on (i) technology, (ii) activity type, (iii) tests, (iv) user behavior, and (v) task performance. The categories “activity type” and “task performance” highlight in front of the others because they are the most used. The “activity type” depends on the technology domain and the variables defined. For example, one measure defined to a map task is the obstacle detection. The task performance measures are the most important cognitive measures because they can be used in several domains, for example, time duration and error rate.

8 Theory of Cognitive Impact Evaluation

After organizing the core category and related the concepts of Cognitive Impact Evaluation, the data was arranged to observe the excerpts related to each concept. The relations found were identified by analyzing patterns and implicit meaning between codes. Once the concepts are related they were organized into a theoretical framework [2], meaning the representation of concepts, together with their definitions and relations among them. Figure 7 presents the cognitive impact evaluation central idea, considering the context of multimodal interfaces for people who are blind or visually impaired.

Fig. 7.

Central idea of cognitive impact evaluation for blind users

9 Conclusion

The applications for people who are blind have many needs due to the target audience and unique characteristics related to multimodal interfaces. Moreover, a lot of the applications for people who are blind or visually impaired aims to improve a cognitive skill, such as cognitive enhancement in O&M, wayfinding, and navigation skills, and thus supporting the user in daily lives. Among so many evidence-based methods, the experiments to evaluate the cognitive impact of an application or a technology present faults in the experiment process.

This perception is based on data of experiments retrieved from the scientific papers found in the SLR which report the experiment, and the empirical study cannot be distinguished from its reporting [26]. The experiments that do not report well their characteristics become weak in one of the main points of the experiment: the repeatability. If the differences in factors and settings are well documented and analyzed, more knowledge may be gained from replicated studies [26].

The importance of the experiment is to consider the formalism in terms of the cognition assumptions underpinning that knowledge. This concern introduces challenges in concepts and formalism of the area of cognition that must be well studied and understood within the experiment. Thus, it is important to involve other disciplines to test an in-depth hypothesis, so as to guarantee rigor in the result, as indicated by [9]. It states that without the link from theory to the hypothesis, empirical results cannot contribute to a broader body of knowledge.

We expected to have created a bibliographic review of the cognitive impact evaluation based on the steps of the SLR approach. It organizes the state-of-the-art of this research area making the information summarized. Having the research findings organized in such a manner that can stimulate research and lead to the extension of knowledge.