Elsevier

Future Generation Computer Systems

Volume 113, December 2020, Pages 380-390
Future Generation Computer Systems

Characterisation of mobile-device tasks by their associated cognitive load through EEG data processing

https://doi.org/10.1016/j.future.2020.07.013Get rights and content

Highlights

  • HuSBIT-10, a taxonomy focused on usual tasks of user–smartphone​ interaction.

  • Cognitive analysis based on EEG of different tasks performed with smartphones.

  • Usual smartphone tasks can be differentiated by their associated cognitive load.

  • A first step towards the early diagnosis of mild cognitive impairment using smartphones.

Abstract

Interaction with mobile devices serves as a link to the cyber world and allows us to characterise user behaviours. The deep analysis of the interactions with the smartphone, aligned with the principles of the Internet of People, allow us to distinguish between normal and abnormal use. One of the multiple applications of this type of analysis will contribute to the early diagnosis of mild cognitive impairment, based on anomalies in the interaction. This work aims to take the first steps towards that ambitious goal: to determine the cognitive load required for different typical tasks with smartphones. By properly identifying which tasks require a higher cognitive load, we will be able to start studying metrics and indicators that contribute to the early diagnosis of cognitive pathologies. The analysis of cognitive load was carried out after an experiment with 26 users who performed 12 typical tasks with a mobile device while their brain activity was monitored through electroencephalography. The results identify that there are clearly tasks with a higher cognitive demand, with audio production and consumption being the most notable, which has been validated experimentally and statistically.

Introduction

Nowadays, many people have smartphones and other mobile devices. This widespread use makes it possible to provide applications and services to address various problems, such as those in healthcare. The large number of sensors in the smartphone provide data on location, movement, voice, battery, application use and more, which is a source of a great deal of information, especially in assessing the behavioural aspects of users’ daily lives [1]. With regard to health, for example, the analysis of smartphone use allows us to track the locations and paths of the GPS followed by users, which can be used to measure things like anxiety levels to anticipate possible mental health problems [2]. Not only are sensor data relevant, but data from the interactions between users and their own mobile devices (e.g. the mistakes made while writing, the active/passive time in applications, the dual task of using the smartphone while walking) also provide valuable information related to human behaviour. Indeed, the smartphone can be a diagnostic ally [3], but it should play a complementary role in the doctor–patient relationship. In particular, in the case of dementia, Blanka Klimova [4] evidenced the potential of mobile apps for facilitating diagnostic support, minimising examiner bias, increasing patient independence, reducing hospitalisation costs and improving the overall quality of life for the elderly. All of this places analyses of interactions (explicit or implicit) with smartphones in an important position, as they can be very valuable both in the fields of human–computer interactions (HCI) and healthcare, whether for diagnostic purposes or even treatment.

This article is part of the project “Mobile computing-based Multitasking for Mild cognitive impairment Monitoring and early Screening (M4S)”, which aims to contribute to the early diagnosis of mild cognitive impairment (MCI) by monitoring dual day-to-day tasks in terms of interactions with smartphones. MCI is highly related with dementia and Alzheimer’s disease and its early diagnosis can contribute for the detection and intervention of them [5]. In fact, the World Health Organisation (WHO) determines the early diagnosis in order to promote early and optimal management as one of the main five goals for dementia care [6]. In the initial stage of this project, the aim is to determine the cognitive load required to carry out various typical tasks performed on a mobile device, which is the main objective of this work. The characterisation of smartphone typical tasks by their cognitive load will help in selecting which tasks should have their performance affected by MCI. Thus, this is a critical pre-requirement to face the next steps of the M4S project that aims to assess cognitive decline analysing the smartphone daily use. Anyhow, the conclusions and results of this document not only contribute to the above-mentioned project but also to the community of researchers in the field, leading to a better understanding of the cognitive processes associated with the use of mobile devices.

This paper is an extension of [7], and its main contribution is the improvement of the experiments in several aspects: (a) the number of participants, which was 26 compared to 6 in the previous paper; (b) the quality of the instrumentation, as the electroencephalography (EEG) headset is a scientific device with more accuracy, higher sample frequency and more channels; (c) the specification of the protocol, which was improved according to the application of the previous experiences; (d) a deeper analysis of the data obtained. In addition, we added extra information to the fundamentals and background, as well as a discussion section.

To study the cognitive load (i.e. the number of working memory resources or “mental effort” associated with a specific task, concepts explored in depth in Section 2.1), we analysed the EEG activity of users performing a set of typical tasks with a smartphone. The fundamentals of the EEG-based cognitive load analysis are also described in Section 2.1. To determine the set of tasks to study, this paper proposes a taxonomy of smartphone-based actions. We considered the related proposals in the literature (Section 2.2) and identified the significant characteristics of the tasks to classify them. As a result, this paper also proposes the HuSBIT-10 taxonomy: Human–Smartphone​ Basic Interactions Taxonomy for 10-s tasks (Section 3). An experiment with real users was conducted with the dual objective of (i) studying the cognitive load of different typical tasks with the smartphone and (ii) validating the classification made in the taxonomy in terms of the mental effort associated with the identified task categories. The protocol, material, and methods of the experiment, as well as the analysis and results of the data from the experiment, are developed in Section 4. Section 5 discusses the results, their meaning and the possible bias of the experiment. Finally, Section 6 concludes the paper, talking about the goals accomplished and future work.

Section snippets

Fundamentals and background

This section will talk about the background of this study, focusing on cognitive load fundamentals and smartphone–user interaction.

Proposed taxonomy: HuSBIT-10

According to the objectives of our study, we needed to define a set of usual tasks focused on user–smartphone interaction. These tasks are quick, simple and require less than 10 s. The name of the taxonomy is HuSBIT-10: Human–Smartphone Basic Interactions Taxonomy for 10-second tasks.

First, four types of interactions that a user could carry out with their smartphone were identified: (τ) touch, (ι) look, (ς) talk, and (η) listen. All of them are closely related to the human senses, which are

Experiment: Cognitive load in smartphone interactions

This section will explain how the experiment was performed, including the protocol followed, information about participants and materials used. The analysis process followed will also be described, as well as the results of the analysis.

Discussion

This section discusses the scope of the obtained results. It is necessary to note that the EEG signal has certain limitations due to its nature. EEG signals can be very sensitive to noise and pick up unwanted artefacts [50] caused by many factors, such as facial movements like blinking or mouth movements [51]. For this reason, EEG data are not directly analysed as a whole; they must be filtered, processed, validated with metrics or characteristics (as is the case with TAR) and, finally,

Conclusions

With this work, we are aiming to take the first step towards a new research line that aims to contribute to the early diagnosis of MCI through the analysis of everyday interactions with smartphones. One contribution of this paper is HuSBIT-10, the taxonomy of typical tasks with smartphones. The taxonomy is based on similar classifications focused on other devices found in the literature, as well as on the cognitive components related to each of the tasks. Researchers in HCI can use this

Ethical standards

The authors assert that all procedures contributing to this work accomplish with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The project of which this work is a part obtained the approval to carry out this research (c-290-Proc. num 11/2019) by the “Committee on Ethics in Research with Medicines” of the Integrated Care Management Unit at Castilla-La Mancha Health Service.

CRediT authorship contribution statement

Luis Cabañero: Software, Formal analysis, Data curation, Writing - original draft, Investigation, Visualization. Ramón Hervás: Conceptualization, Validation, Investigation, Supervision, Writing - review & editing, Project administration, Funding acquisition. Iván González: Conceptualization, Validation, Writing - review & editing, Supervision. Jesús Fontecha: Conceptualization, Methodology, Writing - review & editing. Tania Mondéjar: Validation, Writing - review & editing. José Bravo:

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was funded by the MINISTERIO DE CIENCIA, INNOVACIÓN Y UNIVERSIDADES, Spain, grant number RTI2018-098780-B-I00 (national research project) and the S1911001D predoctoral contract by JUNTA DE COMUNIDADES DE CASTILLA-LA MANCHA, Spain. The EEG headset was purchased with funding from the UNIVERSITY OF CASTILLA–LA MANCHA, Spain , grant number 2019-AYUDA-27430.

Supplementary Materials

The EEG dataset generated and analysed for this study can be found in //www.esi.uclm.es/www/mami/web/index.php/datasets

Luis Cabañero Gómez is Ph.D. student in Computer Science at the Castilla-La Mancha University (UCLM), Spain and member of the Modelling Ambient Intelligence Research Group (MAmI). He received his master’s degree from the UCLM. He has participated in the organisation of two editions of the International Conference on Ubiquitous Computing and Ambient

References (56)

  • KlimovaB.

    Mobile phone apps in the management and assessment of mild cognitive impairment and/or mild-to-moderate dementia: An opinion article on recent findings

    Front. Hum. Neurosci.

    (2017)
  • ChenW. et al.

    Mild cognitive impairment: a concept useful for early detection and intervention of dementia

    Shanghai Arch. Psychiatry.

    (2013)
  • World Health Organization

    Fact sheets

    Dementia

    (2020)
  • CabañeroL. et al.

    Analysis of cognitive load using EEG when interacting with mobile devices

    Proceedings

    (2019)
  • Andrade-LoteroL.A.

    Teoría de la carga cognitiva, diseño multimedia y aprendizaje: un estado del arte, Vol. 5

    (2012)
  • MayerR.E.

    The Cambridge Handbook of Multimedia Learning

    (2014)
  • ChongT.

    Recent advances in cognitive load theory research: Implications for instructional designers

    Malaysian Online J. Instr. Technol.

    (2005)
  • PaasF. et al.

    Cognitive load measurement as a means to advance cognitive load theory

    Educ. Psychol.

    (2003)
  • SwellerJ.

    Discussion of ‘emerging topics in cognitive load research: using learner and information characteristics in the design of powerful learning environments

    Appl. Cogn. Psychol.

    (2006)
  • ClarkR.C. et al.

    E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning

    (2016)
  • KlepschM. et al.

    Development and validation of two instruments measuring intrinsic, extraneous, and Germane cognitive load

    Front. Psychol.

    (2017)
  • KirchnerW.K.

    Age differences in short-term retention of rapidly changing information

    J. Exp. Psychol.

    (1958)
  • FridmanL. et al.

    Cognitive load estimation in the wild

    (2018)
  • van der WelP. et al.

    Pupil dilation as an index of effort in cognitive control tasks: A review

    Psychon. Bull. Rev.

    (2018)
  • Caroline ChanelP.C. et al.

    Online ECG-based features for cognitive load assessment

  • NourbakhshN. et al.

    Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks

    (2012)
  • OulasvirtaA. et al.

    Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI

  • AntonenkoP. et al.

    Using electroencephalography to measure cognitive load

    Educ. Psychol. Rev.

    (2010)
  • Cited by (17)

    • Studying the generalisability of cognitive load measured with EEG

      2021, Biomedical Signal Processing and Control
      Citation Excerpt :

      Klimesch found that an increase in theta band power through the Fz electrode and a decrease in alpha band power through the Pz electrode occur simultaneously when cognitive load increases. That approach is usually applied with the theta-alpha ratio index [11,12] or event-related de-synchronisation [13]. By contrast, the second approach for estimating cognitive load with EEG consists of training machine-learning models using a set of relevant features [14].

    View all citing articles on Scopus

    Luis Cabañero Gómez is Ph.D. student in Computer Science at the Castilla-La Mancha University (UCLM), Spain and member of the Modelling Ambient Intelligence Research Group (MAmI). He received his master’s degree from the UCLM. He has participated in the organisation of two editions of the International Conference on Ubiquitous Computing and Ambient

    Ramón Hervás is Professor in Computer Science in the Department of Languages and Information Systems at the Castilla-La Mancha University (UCLM), Spain, and member of the Modelling​ Ambient Intelligence Research Group (MAmI). He received his M.Sc. and Ph.D. degrees in Computer Engineering from the UCLM. He has been invited as lecturer by different universities, author of 30 JCR articles, and collaborates on several research projects. Dr. Hervás research interests cover a variety of IT issues including Ubiquitous Computing, eHealth, Serious Games, and Neurosciences. Vicedean of the faculty of Computing Science in charge of quality assurance and promotion.

    Iván González is postdoctoral researcher at the Castilla-La Mancha University (UCLM). He received his M.Sc. and Ph.D. degrees in Advanced Computer Technologies from the same University. As a member of the Modelling Ambient Intelligence Research Group (MAmI), Dr. González is author of 10 JCR articles, and collaborates on several research projects. He is currently performing research efforts focused on Quantitative Gait Analysis (QGA), Frailty assessment and Mild Cognitive Impairment (MCI) screening through mobile technologies and embodied sensors. His research interests also include Ubiquitous Computing, Smart Health, Smart Environments, IoT and Sensor Networks.

    Jesús Fontecha is Assistant Professor in Computer Science (Department of Languages & Information Systems) and researcher at the Castilla-La Mancha University (UCLM), performing research in the Modelling Ambient Intelligence Research Group (MAmI) and Technological Institute of Information Systems (ITSI). He received his Ph.D. in 2014 by the UCLM. Currently, Jesus Fontecha is author of more than 18 JCR articles, and editorial board member of several JCR indexed journals. Last years, he has participated in several Healthcare projects in collaboration with other universities and companies. His research interests include Ubiquitous Computing, Smart Health, mHealth, Context Awareness, AAL, Frailty, Gait Analysis, Data Mining, Visualisation and User Interaction.

    Tania Mondéjar Palomares has a degree in psychology and a postgraduate degree in Neuropsychology. She is also in the final phase of her doctorate in health sciences at the University of Castilla-La Mancha. She is currently the director of a psychology clinic for child and youth specialising in neurodevelopment and new technologies. Also is an associate professor in the Faculty of education in the department of psychology at the University of Castilla la Mancha. She has authored and co-authored several works and articles at an international level and collaborates on projects focused on eHealth, Serious games and neurosciences.

    José Bravo is Professor in Computer Science in the Department of Languages and Information Systems at Castilla-La Mancha University, Spain and Head of the Modelling Ambient Intelligence Research Group (MAmI). He is involved in several research areas such us Ubiquitous Computing, Ambient Intelligence, Ambient Assisted Living Context-Awareness and m-Health. He is author of over 37 JCR articles and main researcher on several projects.

    View full text