Skip to main content
Springer Nature Link
Log in

Affective states recognition through touch dynamics

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This work exploits Touch Dynamics to recognize affective states of a user while using a mobile device. To the aim, the acquired touch pattern is segmented in swipes, successively a wide set of handcrafted features is computed to characterize the swipe. The affective analysis is obtained through machine learning techniques. Data have been collected developing a specific App designed to acquire common unlock Android touch patterns. In this way the user interaction has been preserved as the more natural and neutral possible in real environments. Affective state labels have been obtained adopting a well-known psychological questionnaire. Three affective states have been considered: anxiety, stress and depression. Tests, performed on 115 users, reported an overall accuracy of 73.6% thus demonstrating the viability of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abbey A, Andrews FM (1985) Modeling the psychological determinants of life quality. Soc İndic Res 16:1–34. Springer

  2. Ahmad J, Sajjad M, Jan Z (2017) Analysis of interaction trace maps for active authentication on smart devices. Multimed Tools Appl 76:4069–4087

    Article  Google Scholar 

  3. Ali Z, Payton J, Sritapan V (2016) At your fingertips: considering finger distinctness in continuous touch-based authentication for mobile devices. In: 2016 IEEE Security and Privacy Workshops (SPW), pp 272–275

  4. Angelillo MT, Balducci F, Impedovo D, Pirlo G, Vessio G (2019) Attentional pattern classification for automatic dementia detection. IEEE Access 7:57 706–57 716

    Article  Google Scholar 

  5. Antal M, Nemes G (2016) Gender recognition from mobile biometric data. In: 2016 IEEE 11th international symposium on applied computational intelligence and informatics (SACI), pp 243–248ù

  6. Balducci F, Grana C (2017) Affective classification of gaming activities coming from RPG gaming sessions. In: E-Learning and Games. Edutainment 2017. Lecture Notes in Computer Science, vol 10345. Springer

  7. Bevan C, Fraser DS (2016) Different strokes for different folks? Revealing the physical characteristics of smartphone users from their swipe gestures. Int J Hum Comput Stud 88:51–61

    Article  Google Scholar 

  8. Bhattacharya S (2015) A linear regression model to detect user emotion for touch input interactive systems. In: 2015 International conference on affective computing and intelligent interaction (ACII), pp 970–975

  9. ETA Brown, , Chorpita, B.F., W. Korotitsch, D.H. Barlow, “Psychometric properties of the depression anxiety stress scales (DASS) in clinical samples”, Behav Res Ther, 35, 79–89, 1997.

    Article  Google Scholar 

  10. Can YS, Arnrich B, Ersoy C (2019) Stress detection in daily life scenarios using smart phones and wearable sensors: a survey. J Biomed Inform 92:103139

    Article  Google Scholar 

  11. Canossa A, Drachen A, Sørensen JRM (2011) Arrrgghh!!!: Blending quantitative and qualitative methods to detect player frustration. In: Proceedings of the 6th international conference on foundations of digital games, ser. FDG ‘11. New York, NY, ACM, pp 61–68

  12. Carneiro D, Castillo JC, Novais P, Fernndez-Caballero A, Neves J (2012) Multimodal behavioral analysis for non-invasive stress detection. Exp Syst Appl 39(18):13 376–13 389

    Article  Google Scholar 

  13. Ciman M, Wac K (2018) Individuals stress assessment using human smartphone interaction analysis. IEEE Trans Affect Comput 9(1):51–65

    Article  Google Scholar 

  14. Ciman M, Wac K, Gaggi O (2015) isensestress: Assessing stress through human-smartphone interaction analysis. In: 2015 9th International conference on pervasive computing technologies for healthcare (PervasiveHealth), pp 84–91

  15. Cohen R, Lam DY, Agarwal N, Cormier M, Jagdev J, Jin T, Kukreti M, Liu J, Rahim K, Rawat R, Sun W, Wang D, Wexler M (2014) Using computer technology to address the problem of cyberbullying. SIGCAS Comput Soc, vol 44, no. 2, p 5261, ACM

  16. Coppola SM, Lin MY, Schilkowsky J, Arezes PM, Dennerlein JT (2018) Tablet form factors and swipe gesture designs affect thumb biomechanics and performance during two-handed use. Appl Ergon 69:40–46

    Article  Google Scholar 

  17. Corpus KR, Gonzales RJDL, Morada AS, Vea LA (2016) Mobile user identification through authentication using keystroke dynamics and accelerometer biometrics. In: IEEE/ACM international conference on mobile software engineering and systems (MOBILESoft), Austin, TX, pp 11–12

  18. Davidson RJ, Pizzagalli D, Nitschke JB, Putnam K (2002) Depression: perspectives from affective neuroscience. Ann Rev Psychol 53:545–574

    Article  Google Scholar 

  19. Deb S, Dandapat S (2019) Multiscale amplitude feature and significance of enhanced vocal tract information for emotion classification. Trans Cybern 49(3):802–815

    Article  Google Scholar 

  20. Devilly GJ (n.d.) The Psychological effects of a lifestyle management course on war veterans and their spouses. Clintools. Online: http://www.clintools.com/victims/resources/assessment/affect/dass42.html. last visit: 04/12/2020

  21. Dooley J, Pyżalski J, Cross D (2009) Cyberbullying versus face-to-face bullying: a theoretical and conceptual review. Zeitschrift für Psychologie/Journal of Psychology 217(4):182–188

    Article  Google Scholar 

  22. Draffin B, Zhu J, Zhang J (2014) Keysens: passive user authentication through micro-behavior modeling of soft keyboard interaction. In: Memmi G, Blanke U (eds) Mobile computing, applications, and services. Springer, Cham, pp 184–201

    Chapter  Google Scholar 

  23. Eberz S, Lovisotto G, Patan A, Kwiatkowska M, Lenders V, Martinovic I (2018) When your fitness tracker betrays you: Quantifying the predictability of biometric features across contexts. In: IEEE symposium on security and privacy, pp 889–905

  24. Enamamu TS, Clarke N, Haskell-Dowland P, Li F (2017) Transparent authentication: utilising heart rate for user authentication. In: 12th international conference for internet technology and secured transactions (ICITST), Cambridge, pp 283–289

  25. Gao Y, Bianchi-Berthouze N, Meng H (2012) What does touch tell us about emotions in touchscreen-based gameplay? ACM Trans Comput-Hum Interact 19(4):31:1–31:30

    Article  Google Scholar 

  26. Ghosh S, Hiware K, Ganguly N, Mitra B, De P (2019) Emotion detection from touch interactions during text entry on smartphones. Int J Hum-Comput Stud 130:47–57

    Article  Google Scholar 

  27. Gross JJ, John OP (2003) Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J Pers Soc Psychol 85:348–362

    Article  Google Scholar 

  28. Gupta S, Buriro A, Crispo B (2019) Smarthandle: A novel behavioral biometric-based authentication scheme for smart lock systems. In: Proceedings of the 2019 3rd international conference on biometric engineering and applications, ser. ICBEA 2019. Association for Computing Machinery, p 1522

  29. Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46(1–3):389–422

    Article  Google Scholar 

  30. Heikkilä P, Ainasoja M, Oksman V (2015) The potential of technology in facilitating positive stress experiences. In: Proceedings of the European conference on cognitive ergonomics 2015, New York, ACM

  31. Impedovo D (2019) Velocity-based signal features for the assessment of Parkinsonian handwriting. IEEE Signal Process Lett 26(4):632–636

    Article  Google Scholar 

  32. Impedovo D, Pirlo G (2019) Dynamic handwriting analysis for the assessment of neurodegenerative diseases: a pattern recognition perspective. IEEE Rev Biomed Eng 12:209–220

    Article  Google Scholar 

  33. Impedovo D, Pirlo G (2018) Automatic signature verification in the mobile cloud scenario: survey and way ahead. IEEE Trans Emerg Topics Comput. https://doi.org/10.1109/TETC.2018.2865345

  34. Impedovo D, Pirlo G, Vessio G, Angelillo MT A handwriting-based protocol for assessing neurodegenerative dementia. Cogn Comput 11(4):576–586

  35. Kambourakis G, Damopoulos D, Papamartzivanos D, Pavlidakis E (2016) Introducing touchstroke: keystroke-based authentication system for smartphones. Sec Commun Netw 9(6):542–554

    Article  Google Scholar 

  36. Kanniyan N, Harsha S, Unni V, Beevi AS, Jahan N (2018) Certificate authentication using Periocular biometrics. In: International conference on emerging trends and innovations in engineering and technological research (ICETIETR), Ernakulam, IEEE, pp 1-3

  37. Kim M, Kim H, Lee S, Choi YS (2013) A touch based affective user interface for smartphone. In: 2013 IEEE international conference on consumer electronics (ICCE), pp 606–607

  38. Kulkarni PB, Patil MM (2018) Clinical Depression Detection in Adolescent by Face. 2018 ınternational conference on Smart City and emerging technology (ICSCET), Mumbai, pp 1–4

  39. Li C-T, Cao J, Li TM (2016) Eustress or distress: An empirical study of perceived stress in everyday college life. In: Proceedings of the ACM international joint conference on pervasive and ubiquitous computing: adjunct, pp 1209–1217

  40. Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Cordasco G (2017) EMOTHAW: a novel database for emotional state recognition from handwriting and drawing. IEEE Transactions on Human-Machine Systems 47(2):273–284. https://doi.org/10.1109/THMS.2016.2635441

  41. Lovibond PF, Lovibond SH (1995) The structure of negative emotional states: comparison of the depression anxiety stress scales (dass) with the beck depression and anxiety inventories. Behav Res Ther 33(3):335–343

    Article  Google Scholar 

  42. Majumder A, Behera L, Subramanian VK (2018) Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans Cybern 48(1):103–114

    Article  Google Scholar 

  43. Maramis C, Stefanopoulos L, Chouvarda I, Maglaveras N (2018) Emotion recognition from haptic touch on android device screens. https://doi.org/10.1007/978-981-10-7419-6_34

  44. McColl D, Jiang C, Nejat G (2017) Classifying a Person’s degree of accessibility from natural body language during social human–robot interactions. IEEE Trans Cybern 47(2):524–538

    Article  Google Scholar 

  45. Meng Y, Wong DS, Schlegel R, Kwok L-f (2013) Touch gestures based biometric authentication scheme for touchscreen mobile phones. In: Kutyłowski M, Yung M (eds) Information security and cryptology. Springer Berlin Heidelberg, Heidelberg, pp 331–350

    Chapter  Google Scholar 

  46. Meng W, Li W, Wong DS (2018) Enhancing touch behavioral authentication via cost-based intelligent mechanism on smartphones. Multimed Tools Appl 77:30167–30185. https://doi.org/10.1007/s11042-018-6094-2

    Article  Google Scholar 

  47. Miguel-Hurtado O, Stevenage SV, Bevan C, Guest R (2016) Predicting sex as a soft-biometrics from device interaction swipe gestures. Pattern Recogn Lett 79:44–51

    Article  Google Scholar 

  48. Mitchell KJ, Ybarra M, Finkelhor D (2007) The relative importance of online victimization in understanding depression, delinquency and substance use. Child Maltreatment 12(4):314–324

    Article  Google Scholar 

  49. Mondal S, Bours P (2015) Swipe gesture based continuous authentication for mobile devices. In: 2015 Int. conference on biometrics (ICB), pp 458–465

  50. Pereira MG, Figueiredo AP, Fincham FD (2012) Anxiety, depression, traumatic stress and quality of life in colorectal cancer after different treatments: a study with Portuguese patients and their partners. Eur J Oncol Nurs 16(3):227–232. Elsevier

  51. Rehman AU, Awais M, Shah MA (2017) Authentication analysis using input gestures in touch-based mobile devices. In: 2017 23rd international conference on automation and computing (ICAC), pp 1–5

  52. Rowa FK, Antony MM (2008) Generalized anxiety disorder, psychopathology: history, diagnosis, and empirical foundations, p78

  53. Sapolsky RM (2000) Stress hormones: good and bad. Neurobiol Dis 7(5):540–542

    Article  Google Scholar 

  54. Selye H (1974) Stress without distress. The New American Library, New York

    Google Scholar 

  55. Shen S, Chang C, Lin S, Chien W (2019) The enhanced graphic pattern authentication scheme via handwriting identification. In: IEEE Eurasia conference on IOT, communication and engineering (ECICE), Yunlin, Taiwan, pp 150–153

  56. Smith PK, del Barrio C, Tokunaga RS (2013) Definitions of bullying and cyberbullying: how useful are the terms?. In: Principles of cyberbullying research: Definitions, measures, and methodology, pp 26–40, Routledge/Taylor & Francis Group, pp 978–0–415-89749-5 (Hardcover)

  57. Syed Z, Helmick J, Banerjee S, Cukic B (2015) Effect of user posture and device size on the performance of touch-based authentication systems. In: 2015 IEEE 16th international symposium on high assurance systems engineering, pp 10–17

  58. Syed Z, Helmick J, Banerjee S, Cukic B (2019) Touch gesture-based authentication on mobile devices: the effects of user posture, device size, configuration, and inter-session variability. J Syst Softw 149:158–173

    Article  Google Scholar 

  59. Wang Y, Fischer N, Bry F (2019) Pervasive persuasion for stress self-regulation. 2019 IEEE international conference on pervasive computing and communications workshops (PerCom workshops), Kyoto, Japan, pp 724–730

  60. Yao M, Chelmis C, Zois D-S (2018) Cyberbullying detection on instagram with optimal online feature selection. In: Proceedings of the 2018IEEE/ACM international conference on advances in social networks analysis and mining, ser. ASONAM 18. IEEE Press, pp 401–408

Download references

Acknowledgements

This work is supported by the Italian Ministry of Education, University and Research within the PRIN2017 - BullyBuster project - A framework for bullying and cyberbullying action detection by computer vision and artificial intelligence methods and algorithms. CUP: H94I19000230006.

Author information

Authors and Affiliations

  1. Computer Science Department, University of Bari “A. Moro”, Bari, Italy

    Fabrizio Balducci, Donato Impedovo, Nicola Macchiarulo & Giuseppe Pirlo

Authors
  1. Fabrizio Balducci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donato Impedovo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Macchiarulo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giuseppe Pirlo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donato Impedovo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Balducci, F., Impedovo, D., Macchiarulo, N. et al. Affective states recognition through touch dynamics. Multimed Tools Appl 79, 35909–35926 (2020). https://doi.org/10.1007/s11042-020-09146-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09146-4

Keywords