Keywords

1 Introduction

The objective of this literature review is an attempt to identify user approaches and research methods to unveil cognitive computing practices to develop Cognitive-computing systems. As machines start to enhance human cognition and help people make better decisions, new issues arise for research. For instance, which actions can we transpose to cognitive systems? How to design information for cognitive systems dialogue with humans? What are the design methods available to investigate the main everyday practices and cognitive human process to make decisions? It is important to understand how research in the design research field may serve as a contribution to develop cognitive systems. It is important to point out the meaning to develop for cognitive systems applying design research. In Design Research area, methods, theories and better development of certain artificial and interactive products are carefully studied to improve the design process, solve problems and extend the knowledge generated to other similar artifacts (Zimmerman 2007). Design methods are tools to help to unveil main practices, cognitive processes and user understanding of information sources available in certain contexts. Collect, organize and present information sources in a way humans will grasp is a challenge for cognitive systems, as those systems may be dialogue based. In order to have a big picture of user research for cognitive systems we propose a literature review. 225 papers available were analyzed in three search databases. The papers were published between 2010 and 2015. The review identified recurring themes and patterns of the most common activities and beneficial user methods for designing future Cognitive Systems. This paper is organized as follows. First a brief overview of Cognitive systems is given. Second, we explain how this review was conducted. Third, we discuss the main results of the review. The limitation of this work and further research was highlighted.

2 Cognitive Systems

Cognitive Computing is the use of computational learning systems to augment cognitive capabilities in solving real world problems. According to Kelly and Hamm (2013:8):

Tomorrow’s cognitive systems will be fundamentally different from the machines that preceded them. While traditional computers must be programmed by humans to perform specific tasks, cognitive systems will learn from their interactions with data and humans and be able to, in a sense, program themselves to perform new tasks. Traditional computers are designed to calculate rapidly; cognitive systems will be designed to draw inferences from data and pursue the objectives they were given. […]. In the cognitive era, computers will adapt to people. They’ll interact with us in ways that are natural to us.

Kelly and Hamm (2013) also emphasize that Cognitive systems will help us to be smarter offering effectiveness processing large amount of information, dealing with complexity; expertise to help see the overall picture to make better decisions; objectivity avoiding bias; imagination helping us explore a broad range of choices to generate ideas; sense using sensors and analytics software to grasp also physical information. Not only is Cognitive computing a fundamentally new computing paradigm for tackling real world problems, exploiting enormous amounts of data using massively parallel machines, but also it engenders a new form of interaction between humans and computers. Cognitive systems bring human-like reasoning to the problems of Big Data, and also permit us to expand into the white space of domains that require human-like cognition but that either exceed human capacity or are impossible for a live human presence (Nahamoo 2014). Noor (2015) explains that computer essentially process a series of conditional equations and suggest answers. Therefore, it has consequences for user decision-making, since probability can be taken in consideration when making choices. Cognitive systems are able to infer information usually based on parameters that use data captured by sensors or/and user input and interaction. According to Lintern (2011) the robustness of a cognitive system is due to the manner in which the human participants in the system integrate their activities. For instance, those systems may learn more user behavior patterns and provide more assertive inferences. In this context, humans collaborate with machines to create knowledge, and issues of trust and collaboration are topics that are being considered to design those new kinds of systems (Baillieul et al. 2012). In this context, the present review emphasizes research methods to design systems aimed to augment human knowledge and enhance user experience. Additionally, papers that present research methods to acquire contextual information to provide a better user experience were selected.

3 Literature Review

The research papers summarized in the review are referred to as primary studies, while the review itself is a secondary study. The accumulation of evidence through secondary studies can be very valuable in offering new insights or in identifying where an issue might be clarified by this review (Brereton 2007). This paper is a first attempt to identify main design processes (methodologies, methods and approaches) to assist and design future cognitive systems. It was a challenge to choose suitable terms to cover the main design practices applied in the Cognitive systems domain. Words such as: intelligent, smart, wise are used in the cognitive systems context. Not always those terms refer only to computer systems, but also interchangeable with human machine interfaces, integrated systems or human robot/agent cooperation. Additionally, terms as advisor, recommender and tailor define technologies based on human preferences and needs. The diversity of terms is also noted in papers referring to user-centered design processes of interactive technologies, such as: human-centered design; user experience, human computer interaction, user research and so on. In this paper we use the term Cognitive system to refer to technologies that learn and/or dialogue with humans and augment their sense of making decisions. The term User-centered design refers to all the design activities involving users during the Design research process.

3.1 Research Questions and Search Strategy

The main research question is: What are the common user research methods and approaches to design Cognitive Computing systems? The research question guided the selection of keywords for the search. The search keywords were (Table 1):

Table 1. Keywords
Full size table

The sources selected for the systematic review were: IEEExplore Digital Library, Elsevier ScienceDirect and ACM Digital Library Each digital library has its own query rules for advanced search tasks, hence, the search strings and operators had to fit in each library. In the ACM digital Library, a general automatic search and a hand search was made of the last 5 years of well-known conference proceedings: UIST - ACM User Interface Software and Technology Symposium, IUI – ACM Conference on Intelligent User Interfaces, CHI - ACM Conference on Computer Human Interaction, DIS – ACM Conference on Designing Interactive Systems. A general search was also conducted in Elsevier Science Direct, although special attention was given to papers published on the Design Studies Journal, with a hand-searched review. This publication is a rich resource of papers relying on design process, design cognition and design research.

3.2 Criteria Strategy

The search string was adapted according to each data source engine. To be included in the analysis, a paper must have been peer reviewed, available online, written in English, and reported on the confluence of cognitive computing, qualitative methods, design process and human computer interaction. The search criteria were focused on the last five years of available knowledge (2010 to 2015). Selected papers should have a user research approach (field study, evaluation study with users) and systems described in the paper should have one or more highlighted characteristics of our adopted definition of Cognitive Systems: Complexity; Expertise; Objectivity; Imaginations and Sense (Kelly 2013). The papers were classified following a two-fold approach. Firstly, the titles and abstracts were analyzed. The papers were included or excluded according to the protocol criteria. Firstly, the researcher applied the search strategy to have a preliminary list, and filtered the papers by abstracts’ content, following by reading the full text and filtering the relevant papers. This preliminary filter was analyzed by the database. In sequence, the papers were categorized based on their relevance of methods described to inform the design process and to evaluate the design process of cognitive systems.

4 Results

The literature review was conducted between June 2015 and January 2016. A total of 225 papers were selected for the first inclusion and 93 were selected for the second inclusion and 82 papers selected for being classified into the Design process phases (methods to inform and methods to evaluate).

Papers were excluded based on their relevance to the criteria. Most of the papers excluded were experiments to validate algorithms or did not have a Cognitive system component (support decision making, provide learnability, augment human reasoning). Additionally, some of the works excluded were technical simulations not described as inspired by user studies. Others were technical descriptions of prototypes with no users involved. Three papers described relevant methods for informing the design process and evaluating cognitive systems (Table 2).

Table 2. Inclusion process
Full size table

4.1 Methods to Inform the Design Process

Over all 31 papers were selected, 18 were Journal Articles and 13 were conference papers. Usually the preliminary research stage combines projects that apply diverse methods to inform the design process and data acquisition applied to understand human reasoning. Papers that include a Participatory design approach were also included in this phase. Those selected papers describe studies with users and project teams before and during the process of developing cognitive systems.

The use of material artifacts to elucidate human thinking was a common trend since obtaining requirements with only user research to design cognitive systems is not a fixed starting phase. Usually those systems use human parameters and users inputs into technological artifacts for self-improvements, applying machine-learning algorithms. Therefore, some papers described prototypes that were used with the intent to gather parameters for the future systems, as experimental investigations, and not to evaluate a prototype that represents a system. Robins et al. (2010) investigated how robotic toys could be used as a play tool to assist in the children’s development. Experimental investigations with artifacts (field trails with children), expert panels and questionnaires (with caretakers) help to develop scenarios for robots to give stimulus for autistic children that may promote further learning. Scenarios to illustrate context to field trails (Chatley et al. 2010) and to envision future use of Cognitive systems was also a common method applied with Protocol analysis and Think aloud techniques (Wilkinson and De Angeli 2014). Some of the projects used a mixed method approach with qualitative data from interviews (Li and Mao 2015) and quantitative data from surveys and questionnaires (Antoniou and Lepouras 2010). Some projects aimed to understand users visual preferences applying quantitative techniques such as the study proposed by Yang et al. (2014). In their research, authors used the Amazon Mechanical Turk to understand users comprehension and preferences to composite visualizations under different condition. As a result, they developed taxonomy of participants’ difficulties in understanding the graphics. Liu et al. (2015) describes two cases that use behavioral data to drive requirements to design new services. Although, this data is helpful to generate design insights, still the space of design alternatives is complex, according to the authors, and more knowledge based approaches with their proposal method can improve system design. Therefore, with those methods to gather user information is possible to know WHAT is wrong or not working effectively but its not usually possible to know WHY those behaviors happen without user research methods (contextual inquiry, observation studies). Group interviews, focus groups (Xu 2011) were also a method applied to understand better user reasoning in this preliminary stage. Additionally, knowledge acquisition from multiple experts in a meeting helped to create domain knowledge for cognitive systems in Vivacqua et al. (2011). The authors created an ontology to understand participant’s behaviors in collaborative design meetings that may be applied to create intelligent systems. The researchers used design sessions videos to understand behaviors also giving attention to non-verbal messages. Methodologies and approaches were also used to investigate problems that Cognitive systems might help to solve. For instance, Distributed cognition with Collaborative learning approach was applied with students to change the perspectives of public transport (Vasiliou et al. 2014). Grounded theory was conducted to understand the application field and improvisational practices in crisis management followed by field observation, group discussions, and individual interviews (e.g. police, fire department, red cross). Authors suggest recommendations to support aggregation and visualization of information for this sector (Ley et al. 2012).

This review also included user studies that helped collect insights of users’ cognitive processes to facilitate new concept generation. Participatory design was a common approach to understand users in the design process. For instance, the work described by Wilkinson and De Angeli (2014) applied participatory design approach to generate new ideas for a new intelligent mobility app for older people. The authors investigated users with simulated observation cases, questionnaires regarding their shopping behavior, user focus groups with lead users, survey of past experiences, semi-structured interviews and cognitive walkthroughs. They found evidences of the psychological impact design has upon self-esteem that might affect product adoption. These results helped designers to choose the tone and information design for the intelligent system that would not stigmatize their users. A participatory design approach and the use of data captured by sensors were illustrated in Lundberg and Gustavsson (2011). Users with cognitive impairments participated in the design process; researchers illustrated two cases with scenarios of people with special needs in their day-today living. Researchers used sensors to infer context and provide semantic information to potential users, showing information through representations (icons, signals) that help in interpretation. Additionally, participatory design was applied with a group of users with dementia and stakeholders to enhance empathy and involvement in the design process of a Dementia care app (Slegers et al. 2013). Care should be taken when considering users with impairments, in this case dementia, researchers found it difficult to keep potential users active in the participatory design activities in the later stages of the design process, since cognitive declines typical for dementia. New methods should be created to involve people with impaired abilities in design activities.

Studies in the area of Affective Computing, understanding user’s emotion, might help to evaluate human perception when interacting with cognitive systems. Zhou et al. (2013) affirms that cognitive and affective factors may influence user experience (UX) design in the decision-making process. The same authors present a case study of aircraft cabin design, it aims create positive UX in the cabin, including a healthier and more comfortable cabin environment. Overall, 20 participants were recruited. Half of them watched a video to elicit fear and the other half watched a video to elicit amusement. The UX outcome was measured on a scale between 100 (extremely unpleasant) and 100 extremely pleasant. Subsequently, participants were required to make decisions between two design profiles. This study helped to create an improved user experience model for decision-making. Behoora et al. (2015) use non-wearable sensors and machine learning algorithms to identify emotions in team meetings. Understanding emotional states of the design team members helps quantify interpersonal interactions and how those interactions might affect resulting design solutions. Participants were invited to a scenario based design meeting and a catalogue of 8 body language poses relevant to emotional states was used as data. Their machine learning algorithms identify individual’s body language and relates to emotional states to quantify design team interactions.

Since cognitive systems aim to augment human reasoning, some of the projects did not consider at first ordinary user knowledge but expert knowledge to acquire enough information to make the system available for future use. 12 papers focused on experts in the domain of the projects. Likewise in the project (Oliver et al. 2012) that involved stakeholders and local farmers in the design process to understand on the ground decisions that can impact on environmental quality and rural economy. Based on a survey questionnaire with 77 farmers they choose 10 farmers to engage and validate answers from the questionnaire and deeper understanding of farmer’s knowledge. Authors present a protocol with seven iterative stages. An interesting phase was the first one, in which researchers established relationships with farmers distributing a leaflet explaining about the project before the farm visit. Researchers also applied survey questionnaires, face-to-face interviews, a paper map based approach to understand user’s farm area and techniques applied in the farm and qualitative validation of findings with farmers and a community of farmers. In those stages, they built a trusting relationship with the farming community and they acquired farmer’s knowledge to develop a graphic user interface for a Decision support system for land and water management.

Qualitative methods were applied in most of the studies selected for this session. Usually quantitative methods were used with qualitative methods in the design process. Nine papers reported work aimed at special requirements audiences (seniors, impaired users, dementia and disadvantaged people). Most of those papers applied Participatory design as the main approach. Three were focused on children. The predominant areas of the studies were Healthcare and Education.

4.2 Methods to Evaluate Cognitive Systems

In this Assessment phase we selected 43 papers concerned with user evaluation methods of cognitive systems. 19 were papers published in conferences and 20 were published in journals. Papers that discuss the assessment methods selected for this session rely on three groups: Dialogue and/or speech based interfaces; Agent based interfaces and Information Visualization.

Papers discussed dialogue and speech based interfaces which usually use comparative studies, to understand different conditions. The aim to understand uncertainty was also a trend in those papers. Piccardi et al. (2014) apply a mixed-initiative user interface that follows the human-in-the-loop perspective, where the algorithm generates solutions and the role of the human is to select what solution to use. A Q&A system calculates the probabilities which questions will be answered by a crowd; the user assesses the system’s output and makes a decision. 20 crowd-managers had to dispatch questions that they believe unlikely to be resolved by a crowd, they performed this task under two conditions in a web-based simulated tool: visualizing the system prediction and without this information. Participants should drag and drop questions selected to operators into the system. In this lab-experiment participants also did a pre-test questionnaire, then participated in a brief training session and filled out the NASA task load questionnaire. As a result, they found that the visualization of the predictor reduced the participant’s workload. A Wizard of OZ technique, where a human (wizard) simulates the intelligent system tasks such as natural language understanding without user awareness, was perceived as one of the main approaches to evaluate cognitive dialogue systems. Forbes-Riley and Litman (2011) applied the Wizard of Oz technique. The system was a spoken language tutoring system in which the wizard performed speech recognition, natural language understanding, and uncertainty annotation, for each student to answer. 81 students participated in the study. The authors also claim it was the first study to show that dynamic responding to student uncertainty can significantly improve learning during computer tutoring. Rieser et al. 2014 applied the Wizard of Oz tool to improve information presentation in natural language generation dialogues; humans simulated the intelligent system that provided recommendations of restaurants to other humans. Their aim was to present enough information to users while keeping the utterances short and understandable. Authors identified the adaptive natural language generation, as well the information presentation, affects perceived or objective task success of the system.

Agent-based system projects were evaluated mostly by simulations and usability and/or user experience questionnaires. The evaluation study proposed by D’Mello (2012) was a within-subjects design where 48 undergraduates completed four biologic lectures in an intelligent (agent) tutor. Half of the lectures were evaluated with a gaze-reactive tutor and the remaining with a non-gaze-reactive version tutor. They evaluate students’ boredom and disengagement by asking students to report their affective state in the Affect Grid (evaluates pleasant and unpleasant feelings) and filling out an engagement questionnaire after each lecture. As a result, students that used the gaze sensitive dialogues reoriented their attention to patterns of the important areas of the interface, was effective in promoting learning gains for questions that require deep reasoning and minimal impact on the students’ motivation and engagement. Some works described systems that combines the use of human and computer agent simulation. For instance Tremblay et al. (2011) evaluated a system with those mixed characteristics (human and agent) for tactical commander. Overall 10 subject matter experts participated in the study four helped create scenarios for user tests and six evaluated the system. The system was designed to improve Situation awareness to help in accuracy and fast decision-making. Participants tested the simulated system in two different scenarios, answered usability questions and a semi-structured interview. The study provides insights into the design process adopted and other findings for designing situation awareness intelligent systems.

Information visualization tools may serve as a platform for data manipulation and exploration, those representations complement and enhance mental abilities (Meirelles 2013). Designing information that unveils data patterns to help in decision-making is not a trivial task. Since the number of variables, parameters and information links are typically large, and, well-chosen representations are needed (distinct colors, shapes, contrasts) to facilitate interpretation. Insights and new relationships may emerge from diverse ways to display the same dataset. Research in the field tries to understand how users interpret those representations of uncertainties based on probabilistic data. Daradkeh (2015) conducted a user qualitative evaluation of an information visualization (RiDeViz) that shows investment alternatives. The aim was to understand the user awareness of risk and uncertainty with bar charts. Observation approach using the think aloud protocol and content analysis were the methods applied with 10 subjects. Participants were asked to choose one investment choice evaluating risk and uncertainty in a bar chat visualization with limited range and a risk explorer table. The system provided different types of information, although participants did not use all for investment decision-making, they focused on small number of salient pieces and concentrated in the perceived consequences of undesirable outcomes. Additional work using bar charts with samples shows charts representing uncertainty that help understand risk in decision-making (Ferreira et al. 2014). In an initial user evaluation, with 7 users authors wanted to know how accurate and confident users would answer to 5 different types of questions to analyze three visual conditions. The study used a repeated measures design that showed 75 questions in random order. Users also rated confidence/certainty in a Likert scale for each question they answered, after that they answered a questionnaire about the overall experience. Authors also measured accuracy by a quantitative analysis of the questions answered right. The authors identified that participants were more confident in their accurate responses. Baur et al. (2010) evaluates visualization techniques with large datasets and recommended-based systems on mobile phones. They evaluated the visualization technique for repeated item selection in the context of music playlist creation. They considered the particularities of the mobile phone devices (orientation) with 12 users to do a user trail. Users selected options and for each option five suggestions were given out, and one should be right selected. Authors measured completion times and error rates. They found that the vertical orientation and interface was faster to interact and had less error rate than the horizontal one. Arshad (2015) compared the confidence of expert users and non-expert users varying level of uncertainty presented on a prediction case study of water-pipe failure. Participants did three groups of tasks and received a viewgraph of overlapping and non-overlapping uncertainty presentations as supplementary material for decision-making. Showing this supplementary material improved user confidence and uncertainty with unknown probabilities decreasing user confidence, although uncertainty with known probabilities can increase expert user confidence but the same is not true for non-experts.

A mix of qualitative and quantitative approaches was identified in this session. The number of participants were more variable in evaluation studies, most of them had less than 30 participants per study, the reason for that might be the kind of evaluation authors needed to perform, with more details about user experience, and also insights and parameters that should be included to improve current systems. Attention was given to user interpretation of cognitive interfaces. The subjects in the studies reviewed were most adults and from variable domain Education, Healthcare, Military, Management, Finance.

5 Limitations of This Review and Further Work

The result of this review should be used carefully, since the keyword string was limited and a new choice of keywords may retrieve different results. Additionally, this review was a first attempt; an exploratory study, to identify design methods for designing Cognitive systems, for this reason only one author reviewed and selected the data. However, in the future investigations and reviews the importance of inter coder reliability would be emphasized to methodological rigor. (Tinsley and Weiss 2000).

6 Conclusion

Methods for designing cognitive systems did not differ hugely from traditional design processes. Although a significant change found in the review is the consideration of expert domain to design intelligent systems. Care should be taken when narrowing down the types of users to consider as informants. For example in the health care area or designing systems for children, the final user is not always the expert, e.g. caretaker, so the final user should also be included in the user research process as informant and testers. When designing new traditional systems designers usually consider the mental maps of users and their cognitive constraints. The difference when designing cognitive system relies on the number of choices that those system present to users and how those choices are represented to influence in decision-making. In spite of applying traditional evaluation methods to assess user understanding of system outputs (e.g. uncertainties) still new methods are needed to refine those interpretations. Cognitive systems use human data input with the intent to improve and learn in an iterative process. Context is crucial and changes over time. Therefore, design process phases are not linear. As seen in this review, several times prototypes are crucial to gather data and inform the design, and many times prototypes are used as experiments to develop the real system. Primary research focuses on context and assembles the basis for the design process, it is not a phase that is isolated as in some traditional design processes were ethnography work can be the first phase and enough to inform the overall design process, it is an iterative process. Evaluation methods are applied in several stages of the design process and not only as the final phase. The evaluation phase never ends, because those systems as much as they learn and are assessed using user information they become more intelligent and user friendly. More studies such as the one reported by Gustavsson (2011) are necessary to understand the suitable tone cognitive systems should use with humans. Matters of trust, confidence and transparence might be related to visual and verbal representation of information. Showing uncertainty and risk, as Arshad (2015) identified, it is not always beneficial to non-expert users. New methods to understand how ordinary people interpret risk and uncertainty should be created. Current methods evaluate presentation studies showing or not showing if prediction and uncertainties is beneficial for users. It would be interesting to understand which kind of interfaces; shapes and color saturations attend this purpose more and are better understood by users.

In short, methods to inform and methods to evaluate cognitive systems were reviewed. The main methods and approaches applied were: interviews, questionnaires, scenarios to test current abilities of future prototypes, and field observations. Moreover, semantic scales were common in comparative studies. Dialogue based systems were usually evaluated applying the Wizard of OZ technique. In the future, a new category of design methods also should be investigated. It is the category of validating output information; the kind of feedback (visual, verbal haptic) cognitive systems should give to help in more informed decision-making.