Keywords

1 Introduction

Various studies on understanding users’ behavior with the product are being carried out using eye-tracking technology to analyze if the behavior that users claim to have or to demonstrate is similar to what they are actually performing at the time of the survey [1]. In general, these technologies do not identify users’ real day-to-day behavior. Very often this behavior also differs when a comparison is made between undertaking an experiment and a user´s real life.

Recently research studies have been developed in the neuroscience area with a view to analyzing and understanding human behavior [2]. In their studies, researchers are using the system of measures performed by Electroencephalogram (EEG) which analyzes a user’s experience in accordance with the record of the electrical signals produced by neurons that are picked up by sensors arranged all over the user’s scalps. The data of the electrical activity detected are recorded and stored by a computer via the Brain Computer Interface (BCI) simplified by means of a wireless hardware device called Emotiv EPOC that lets what the user really does and thinks while performing various tasks be observed.

As a result, ergonomics studies associated with neuroscience have led to neuro-ergonomics, namely the study of the brain and its behavior in the various activities of work in the various activities [3]. This area lets human cognition and behavior be analyzed in various locations and activities.

According to Kirkland [4], neuroscience when used in conjunction with knowledge of cognitive science, has led to the emergence of research studies in neurodesign. In this field of research, the user’s experience based on cerebral activities can be better understood and it becomes possible to explain whether or not such an experience is satisfactory for a particular individual.

In this context, the appearance of new technology has been aiding practice in neurodesign, such as using the EPOC Emotiv headset in research studies associated with product design. The Emotiv EPOC® system (Fig. 1) uses neurotechnology knowledge to perform real-time scanning and storage of the EEG signals emitted by a user’s brain so that they can be stored in the software of the apparatus.

Fig. 1.
figure 1figure 1

Emotiv EPOC

Full size image

It is valid to comment that for every activity we perform we have cerebral areas that are respectively activated and which can be studied via the Brain Computer [5] interface. Therefore, the cognitive load of a human being coming from the tasks performed is detected by combining parameters of the power spectrum, of the electroencephalogram (EEG) produced by the Emotiv EPOC.

To do so, the EEG waves are picked up by electrodes that are placed directly on an individual’s scalp and which are highly sensitive to the changes of the potential of neural action of the cerebral cortex. Therefore, the registration of cerebral activity is transmitted by the difference of the interval of the voltage time between a connected active electrode and the reference electrode placed in different locations on the body or scalp. The detected signals are amplified and processed in real time using graphs and decoded in accordance with the behavior of the activity.

The voltage coming from a standard brain activity has a sensitivity standard of 7 μV/mm and the average of the voltages close to the sensor area is amplified and combined to establish a rhythmic activity that is classified and combined in the frequencies Delta, Theta, Alpha, Beta and Gamma [6].

The Emotiv EPOC is a headset consisting of 14 data channels that are arranged using stems. The main rod is responsible for fixing the other secondary stems and has the format of a “tiara”.

Thus, the product was selected for the study because it has the ability to control objects by using the power of thought and was developed to interact with the computer x human interface. Moreover, the Emotiv EPOC can detect thoughts, feelings, facial expressions via a wireless connection. Thus, to analyze the usability of this equipment is of extreme importance, because the utility and function performed may not be consistent with good levels of usability of the product.

Against this background, this research sets out to analyze the usability of the Emotiv EPOC equipment using heuristic analysis to detect possible problems involved in the interaction between the product and the user.

2 Methodological Procedures

The Emotiv EPOC device was investigated using the following procedures.

In the first part of the study, the product was introduced and a general description of was made in relation to its components and operation. Subsequently, the stage of setting out problematic issues and of analyzing the task associated with video-recording the Methodology for the Systemic Analysis of the Human Task-Machine System - SHTM [7] was engaged on. Additionally, the types of problems and interactions related to the design of the Emotiv EPOC were investigated based on how the SHTM set out the problematic issues.

Thereafter, an analysis of the usability of consumer products was made which examined the supposition, learning, and performance of the user, the potential of the system and re-usability [8].

Finally, the SUS (System Usability Scale) method [9] was applied which had 10 questions and so too was a questionnaire with 26 questions relating to the user’s profile, handling of the product, safety and comfort, placement of the sensors and saline solution, place and use of the USB cable, and feedback from the on/charge button. The SUS method invites the user to list tasks such as the activities performed during the process of using the product. This presents a system to the respondent which has questions that should be marked and answered on a scale of satisfaction in accordance with his/her level of agreement or disagreement with each question.

This scale uses scoring that ranges from zero to one hundred by using five response categories: 1- I totally disagree, 2- I disagree, 3- I neither agree nor disagree, 4- I agree and 5-I totally agree. As the sample used in this study was eleven individuals interviewed and the answers to the extreme categories were, in most cases, few in number, it was decided to merge the following categories: I totally disagree, I disagree with I agree, I totally agree.

The ten SUS questions assessed the following categories: use of the system, complexity of the system, ease of use, assistance to use the system, integrated functions of the system, inconsistency of the system, fast learning, user’s discomfort and complications while using the system, security and confidence when using the system and knowledge of other information so as to use the system.

The SUS questionnaire invites the user to list tasks such as the activities performed during the process of using the product. This presents a system to the respondent which has questions that should be answered and marked on a scale of satisfaction according to the level of agreement or disagreement to each question.

The data obtained from the survey were analyzed qualitatively for information on the users’ performance and attitude while performing the task and for suggestions to solve possible problems.

3 Introduction to and Description of the Product

The Emotiv EPOC is marketed through a web page available on the internet (www.emotiv.com). Since it is a new piece of equipment on the market, users do not have experience with the product and the usability issues were analyzed based on how tasks were undertaken while using the device.

The equipment takes the form of a tiara-like headset. The main rod has an on/charge button and the luminous signal that provides feedback by displaying a blue LED light when turned on and a red one to indicate that the system is being charged via a USB cable connected to the exit of the computer which must have properly installed software. Next to the end of the extremities of the main rod, the other secondary stems are divided into two groups of seven stems each, arranged along the left and right hemispheres of the skull. The material that makes up the equipment allows the stems to have flexible movement so they can be handled and coupling on the skull using a simple and practical interface (Fig. 2).

Fig. 2.
figure 2figure 2

Description of the components of the Emotivo EPOC Headset on the head as seen from the front and side [10].

Full size image

The two reference sensors are fixed below the ears, as shown in Fig. 2, and serve as a reference for identifying the left and right hemispheres of the head and to capture signals from the other sensors arranged on the scalp.

The visual interaction of the Emotiv EPOC in operation is accomplished by using the computer monitor after installing the license for using the program. This interaction is achieved via menus and tabs with different content and information. The Emotiv EPOC offers six different SDK (Software Development Kit) packages that store and save the data and may contain 4 different programs in accordance with the objective of the study. In this study we used the Education edition and the Control Panel in the Headset Setup tab so as to see the connection of the sensors. The software displays an image, in a top view of the human skull with the 16 sensors arranged as shown in Fig. 3.

Fig. 3.
figure 3figure 3

Arrangement of the electrodes of the Emotiv EPOC Headset [10]

Full size image

The electrodes form seven sets of channels are arranged on the scalp. Using the software of the product, each sensor is displayed graphically by circles that appear in different colors in accordance with the quality of the contact. Each sensor is represented by a color code: Black - No signal; Red - Very poor signal; Orange - Poor signal; Yellow - Fair signal; Green - Good signal.

4 Setting Out the Problematic Issues

The investigation of the problem will allow an appropriate analysis in order to identify possible solutions for the dysfunction of the human- task-machine system.

Based on this statement, an investigation was undertaken of the types of problems and interactions and how these relate to the situation of the design of the product by means of a three-stage investigation of the problem [7]:

  1. 1.

    Recognizing the problem: Listing the most serious problems of the product analyzed, in terms of meeting the user’s needs

  2. 2.

    Defining the problem: Classifying the situation of the problems by means of the Analysis of Dysfunction of the product-user interface using photographs.

  3. 3.

    Formulating the problem: Describing the most significant and solvable aspects by considering personal competence, the knowledge available and what the user required.

In the first step, recognizing the problem, the following factors were observed: poor fixation of the sensors on the secondary stems; rapid absorption of the saline solution into the felt of the sensors; connection signal hampered by the presence of the user’s hair; and the fragility of the materials of the product.

The second stage, called defining the problem, was important to detect the following categories of problems:

  • Structural and Movement Problems: Little resistance when moving the sensors on the scalp during handling; Lack of practicality for fixing the sensors when adjusting the signal settings and for preventing their sliding on the scalp; Lack of security and the looseness of the sensors as to fixing them to the secondary stems thus causing failure or absence of the connection signal; Connection failure arising from the lack of sensitivity of the sensors to the presence of hair; The felt of the sensor quickly absorbs the saline solution; The looseness and slanting of the sensors impair the connection signal intermittently and causes transmission error of the connection signal of the system. This failure can be seen when the sensor is not connected to the scalp but the system detects the presence of a connection signal via the Emotiv Control Panel.

  • Problems of Resistance: Lack of resistance of the material covering the secondary stems, caused by handling on the scalp, thereby leaving the internal circuitry seen. Lack of resistance of the felt material of the sensors arising from the attrition caused by docking and undocking.

5 Task Analysis Associated with Video Recording

The strategy for assessing the usability of the product was based on the methodology for analyzing a task given by Moraes and Mont’Alvão [7].

To this end, the analysis considered the users’ inexperience with regard to using the product. Thus, the evaluators made general comments on the equipment and its use, and also distributed a list with instructions on how to use the task.

The operations of the Emotiv EPOC system were selected and displayed in summary form in the functional action-decision flowchart below (Fig. 4).

Fig. 4.
figure 4figure 4

Action-Decision Flow-chart of the Emotiv EPOC system

Full size image

The aspects used for the analysis of the task were with regard to ease of use, learning and the user’s first impressions during his/her first experience using Emotiv EPOC. These aspects served as a reference to evaluate the following proceedings of the user’s interaction: Time spent to carry out the activity; Number of attempts to perform the task; Doubts, questions and comments verbalized during the experiment; Nonverbal aspects perceived because of the user’s behavior.

The aspects related to the task procedure were recorded on video and analyzed according to the time taken to performing the following tasks: the average time of the first part of the task which consists of the moistening plus placement of the sensors was five minutes; the time taken in the second part of the activity corresponding to adjusting the connection signal was six minutes. The time spent on the two tasks was eleven minutes.

The main problem observed in the first stage of the video observation was the falling away of the stems of the secondary sensors when they were being placed and when being put on the user’s scalp.

In the second observation step, the same problem as in the previous step was observed regarding the falling off of the sensors from the stems and all users required the help of the researchers to adjust the signal on their scalp. The adjustment was made after pressing the two Emotiv reference sensors, which are located behind the ears, for 10 s. Subsequently, the most troublesome activity was that of sliding the sensors gently on the scalp until all the sensors were green on the Emotiv Control Panel. This practice required some sensors to be again moistened, the sensors that came loose and fell off to be replaced and to push aside the hair of the area on the scalp that the sensor was occupying.

6 Usability Analysis

The evaluation of usability is a systematic method of analysis of the user’s relations with the product and systems. This methodology is undertaken by gathering information about the particular situations of the consumer products in question [11]. The results of the analysis of the usability of the Emotiv EPOC usability analysis, using the principles of Jordan [8], are shown in Table 1, in order to analyze the user´s level of satisfaction, safety, comfort and well-being.

Table 1. Result of the analysis of usability as per Jordan [8]
Full size table

7 SUS Method

The SUS method [9] was applied followed by video recording the users while they performed the task requested and this was monitored by the researchers of the study. The recordings were the basis for comparing the results of the analysis of the task requested.

The research was conducted in the Center for Arts and Communication, the location of the Design course at the Federal University of Pernambuco, and in CESAR (Center for Studies and Advanced Systems of Recife) located in the city of Recife, Brazil. 11 subjects (students, designers and engineers), nine males and two females, aged 20–30 years, were recruited. None of them had some kind of physical disability. An overview was given to each interviewee on the topic of the research and on the purpose of the questionnaire and then the interviewees received some information on how to use the apparatus and the software. Later, they were given some instructions on the task to be performed with the Emotiv EPOC and then asked to answer a questionnaire on evaluating usability. The task performed was the same as that presented in the schema of the action-decision flowchart of the Emotiv EPOC system (Fig. 4).

Volunteers were asked whether they had previously used any similar apparatus and they all said they had not. Because of this they were given some instructions on how to use the product and how to connect the sensors with the software. Even after having read the instructions, users were still uncertain about how to use the product and their questions were answered by the researchers.

Regarding the use of the headset, they were asked if they considered that in their opinion handling and maneuvering the headset was comfortable and safe. Five disagreed, five said they were neutral, another agreed that it was comfortable but not safe and one agreed that it was safe. One person answered twice. On being asked if they had problems in handling and maneuvering the headset, the problems mentioned were: Form of the secondary stems (n = 6); Dimensioning (n = 2); USB cable docking system (n = 1); On/charge button (n = 2); Poor dimensioning (n = 2); Appears to be fragile (n = 2).

When asked if screwing sensors into and removing them from the secondary stems of the headset was easy, one respondent totally disagreed, four disagreed, three marked this as neutral, one totally agreed and two agreed. As to screwing the sensors into and removing them from the secondary stems, the problems encountered were: Screwing the sensors in (n = 3) and adjusting the sensors (n = 9). One person answered twice. Two participants noted the option of another problem, one reported a threading problem and suggested a plug, while the other referred to the problem of bad contact.

They were asked if it was easy to apply the saline solution on the sensors, One was neutral about this, six agreed it was and four agreed completely with this. Similarly, they were asked if it was easy to moisten the sensors. One marked this as neutral; five agreed it was and five agreed completely with this.

Regarding the effectiveness of moistening, two disagreed it was; three marked as neutral, three agreed and three agreed completely. They were asked if they considered moistening the sensors was practical. Three said they were neutral about this, seven agreed and one totally agreed with this. Another question was about which type of problem was identified when moistening the sensors. The comments were as follows: Little absorption (n = 2); Much absorption (n = 3); Rapid absorption (n = 1). Four people marked the option of other problems and their comments were: “Hair absorbs the liquid and interferes with the moistening”; “Rapid absorption of the solution in the sensors”; “There is no indication of the level of moistening if it has already been wetted”; “Rapid evaporation and deterioration of the sensors” One person did not answered.

When asked if installing and removing the USB cable is considered easy, three respondents agreed it was and six agreed completely. To respondents did not answer. Regarding the type of problem when installing and removing the USB cable, two respondents found it difficult to locate the entry point of the USB cable; another did not identify any problem and eight did not answer this question. They were asked if the information on the switch on/charge button was sufficiently clear enough. Two disagreed, four agreed, and five agreed completely with this. The problems encountered in the Switch on/charge button were: Symbol of switch on/charge is not clear (n = 2); the color of the symbol is inadequate for Switch on/charge (n = 1); Inappropriate contrast of the symbol for switch on/charge (n = 1); Size Three respondents indicated another option and commented as follows: “Location”; “The side of on/off is not identified”; “The place of the symbols are in opposite places” (n = 3) and the other did not detect any problems. Three people did not answer.

On aspects related to security, they were asked whether the headset use was safe to use. One disagreed, five were neutral, three agreed and two agreed completely with this. The types of problems identified as to its safe use were: an uncomfortable feeling (n = 2), a feeling of being unsafe (n = 2), fear of getting a shock (n = 3); other, “It looks like it will fall off.” (n = 1). Some people did not answer all questions.

On aspects related to security, they were asked if they considered it safe to use the equipment. One disagreed with this; five were neutral about it; three agreed and two totally agreed it was. The problems identified in relation to safety were: a feeling of discomfort (n = 2); a feeling of being unsafe (n = 2); a fear of getting a shock (n = 3), and other “It looks as if it´s going to fall off” (n = 1). Three people did not give their opinion.

They were asked if the feedback provided by the luminous signal of switch on/charge was effective. Two disagreed with this; one was neutral; three agreed and five totally agreed it was. The problems identified with the feedback of the luminous signal were as follows: Display indication of switch on is not sufficiently clear (n = 3); Display indication of charge is not sufficiently clear (n = 1); Other: “small symbols” and “inadequate location” (n = 2). Five people did not give their opinion.

On aspects related to comfort when using the headset, they were asked if they considered this was comfortable. Two disagreed; four said they were neutral; four agreed and one totally agreed with this. Regarding comfort, the problems identified were: getting hold of it inadequate (n = 4); difficulty in maneuvering it (n = 5); the material (n = 2); Other, “The sensors keep falling off” and “gripping the neck.” (n = 2): One people answered twice.

Regarding the placement and removal of the headset, they were asked if they considered this was easy. Two totally disagreed; three disagreed; two said they were neutral; one agreed and three totally agreed. They were asked what kind of problem they had identified when putting on and removing the headset on their head. The problems were: The action of placing the headset on their head (n = 1), adjusting the headset on my head (n = 4), Other (n = 1): The sensor falls easily. ”

On aspects related to the use when connecting the headset with the software, they were asked if this was effective (it fulfils the role it is aimed at). Two totally disagreed; two disagreed; two were neutral; one agreed; and three totally agreed. The problems identified when connecting to the software were: Absence of signal (n = 1), oscillating signal (n = 7) and did not answer (n = 3).

Regarding the user’s satisfaction with the connection to the software, the users were asked if they considered they would like to use this system frequently. One person strongly disagreed; three disagreed; two were neutral and five agreed.

When questioned if they found the system unnecessarily complex, one person strongly disagreed; four disagreed; three were neutral; two agreed and one strongly agreed. As to the question I thought the system was easy to use, one person strongly disagreed; seven disagreed; one was neutral and two agreed. The participants were questioned about whether they thought that they would need the support of a technical person to be able to use this system.

The answers were: two respondents disagreed, seven agreed and two strongly agreed. As to the question “Did you find the various functions in this system were well integrated”, two disagreed; five were neutral; three agreed and one strongly agreed. Next, they were asked if they thought there was too much inconsistency in this system. The answers were: four disagreed; five were neutral; one agreed and another strongly agreed. The next question was “Would you imagine that most people would learn to use this system very quickly” and they answered as follows: two people strongly disagreed, four disagreed, one was neutral and four agreed.

The users were questioned about whether the system is very cumbersome to use, and the answers were: three disagreed; two were neutral; five agreed and one strongly agreed. The question “Did you feel very confident when using the system” obtained the following results: one strongly disagreed and one disagreed; seven were neutral; one strongly agreed and one agreed. Finally, the last question about satisfaction with the software was “Did you need to learn a lot of things before you could get going with this system?” The results were: six disagreed; one was neutral and four agreed.

In the last question, the respondents were asked to write their opinion about the Emotiv EPOC. The comments were compiled and summarized as follows:

  • “Interesting and useful but tricky and complicated to use without help. The felts fall off thus making the process difficult.” “The adjustment of the equipment is troublesome. The sensors are not sufficiently well docked to the equipment. It takes practice to use it.” “It’s not practical.”

  • “Very light and good feeling when being put on because it is something that I had never before had contact with. It was kind of hard to assimilate how to adjust it or use it but nothing at all after a little practice in learning about it; it seems effective and logical. Within three days I would be able to master how to handle it.”

  • “It´s a nuisance to put on if you have a lot of hair. I found it very difficult to get all sensors to stay green.” “It does not come up to scratch in its operation due to the sensors.” “The connection between the headset and the head was never made and the sensors are difficult to dock.”

Analyis of the Results and Suggestions. It was noted that no user had used this equipment before. Most users felt the grip and handling of the equipment is not comfortable and safe and encountered problems especially regarding the form of secondary stems. As for docking and removing the sensors, most users disagree that this is easy and reported problems in adjusting the sensors.

Everyone considered that putting the saline solution on and moistening the sensors are easy, practical and effective tasks. On the other hand, most identified problems as to adjusting the sensor due to the insufficient amount of saline on the sensors.

Most participants also did not identify problems as to the clarity of the information of the switch on/charge button nor in installing and removing the USB cable. However, when they were asked to identify problems in the button, some alternatives were marked.

The respondents were asked if they considered the equipment was safe. Most agreed it was. However, some items related to safety issues were marked. Most considered the feedback provided by the luminous signal of switch on/charge was effective.

Most thought using the equipment was comfortable, but they identified some problems.

Most had difficulty as to installing and removing the headset and also identified some problems such as:

Regarding satisfaction with the connection of the headset to the software, the responses were divided: roughly, half agreed and the other half disagreed. However, most agreed that the connection signal oscillated.

Attention should be drawn to the fact that there is disagreement between most respondents stating that they would use this system frequently and at the same time that that they agree that the system is not complex yet they would require support from a technician to use it. It was the unanimous view that the equipment really needs the help of an experienced user if it is to be used properly.

Almost all of the respondents considered that the system functions were well integrated, that the system was difficult to use and that most people would take a long time to learn to use it. We also believe that the system works properly, but through the experience of using it.

Regarding the classification of the scores of the SUS questionnaire, values ​​below 60 represent satisfactory and poor systems and which cause users to be dissatisfied. Scores from 61 and over represent very good experiences with a good satisfaction rating [9]. According to the usability indices obtained by the SUS score, the equipment analyzed had a usability index of 46.5. This index is indicated below that recommended by the author. In other words, the Emotiv EPOC does not have a good usability. Its system level is considered to be poor and can result in users being dissatisfied.

8 Recommendations

As a suggestion from the usability analysis, it was realized there is a need to improve how to attach the sensors to the secondary stems so that they do not easily become detached from the insertion site.

Regarding the adjustment of the connection signal, the oscillation of the connection signal displayed by changing the color code should be improved in relation to the sensitivity of the sensors so that the adjustment is easier, faster and more effective.

The sensitivity of the sensors should also be reviewed given that the presence of the user’s hair interferes with the adjustment of the signal connection.

The practicality of fixing the sensors can be improved by implementing a magnetic locking system.

As for the coating material, it is recommended that the material used in the secondary stems and in the felt be replaced with a tougher one that prevents the internal wiring being seen and that allows a better docking and undocking of the sensors.

9 Conclusion

Usability tests in neuroscience are still in their growth phase but the area has been proving to be quite interesting and promising when developing research to discover solutions for complex problems related to the usability of products.

Therefore, it is important to make clear that research studies in this context can help to analyze the usability of a piece of equipment that furthers understanding of human behavior through the brain as well as to assist the design area with the use of emerging technologies.

The use of SUS [9] proved to be pertinent to the analysis conducted of the product. However, we identified some contradictions in the answers. Therefore, we consider that such contradictions have adversely affected the usability analysis of the product. We believe that this problem could have been minimized if the sample had been larger than that used in the study.

We conclude that the idea of the design of the Emotiv EPOC fulfils facilitating the tests that require EEG analysis. To this end, this study set out to contribute by making suggestions to solve the usability problems so that users may conduct their tests with this equipment in a pleasant and satisfactory manner.