Skip to main content
SpringerLink
Log in

A systematic literature review about technologies for self-reporting emotional information

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Emotional information is complex to manage by humans and computers alike, so it is difficult for users to express emotional information through technology. Two main approaches are used to gather this type of information: objective (e.g. through sensors or facial recognition) and subjective (reports by users themselves). Subjective methods are less intrusive and may be more accurate, although users may fail to report their emotions or not be entirely truthful about them. The goal of this study is to identify trends in the area of interfaces for the self-report of human emotions, under-served populations of users, and avenues of future research. A systematic literature review was conducted on six search engines, resulting in a set of 863 papers, which were filtered in a systematic way until we established a corpus of 40 papers. We studied the technologies used for emotional self-report as well as the issues regarding these technologies, such as privacy, interaction mechanisms, and how they are evaluated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Angelini L, Caon M, Lalanne D et al (2015) Towards an anthropomorphic lamp for affective interaction. In: proceedings of the ninth international conference on tangible, embedded, and embodied interaction. ACM, New York, pp 661–666

  • Bannon L (2011) Reimagining HCI: toward a more human-centered perspective. Interactions 18:50–57. doi:10.1145/1978822.1978833

    Article  Google Scholar 

  • Bardzell S, Bardzell J, Pace T (2009) Understanding affective interaction: Emotion, engagement, and internet videos. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops (ACII 2009), pp 1–8. doi:10.1109/ACII.2009.5349551

  • Barrett LF (2004) Feelings or words? understanding the content in self-report ratings of experienced emotion. J Pers Soc Psychol 87:266–281

    Article  Google Scholar 

  • Bialoskorski LSS, Westerink JHDM, Broek EL (2009) Mood swings: an affective interactive art system. In: Nijholt A, Reidsma D, Hondorp H (eds) Intelligent technologies for interactive entertainment. Springer, Berlin, pp 181–186

    Chapter  Google Scholar 

  • Broekens J, Brinkman WP (2009) AffectButton: towards a standard for dynamic affective user feedback. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops, pp 1–8

  • Broekens J, Brinkman W-P (2013) AffectButton: a method for reliable and valid affective self-report. Int J Hum Comput Stud 71:641–667. doi:10.1016/j.ijhcs.2013.02.003

    Article  Google Scholar 

  • Broekens J, Pronker A, Neuteboom M (2010) Real time labeling of affect in music using the Affectbutton. In: Proceedings of the 3rd international workshop on affective interaction in natural environments. ACM, New York, pp 21–26

  • Caon M, Khaled OA, Mugellini E et al (2013) Ubiquitous interaction for computer mediated communication of emotions. In: 2013 humaine association conference on affective computing and intelligent interaction (ACII). IEEE, pp 717–718

  • Chambel EOM (2013) Accessing movies based on emotional impact. Multimed Syst 19:559–576. doi:10.1007/s00530-013-0303-7

    Article  Google Scholar 

  • Chen L, Chen G-C, Xu C-Z et al (2008) EmoPlayer: a media player for video clips with affective annotations. Interact Comput 20:17–28. doi:10.1016/j.intcom.2007.06.003

    Article  Google Scholar 

  • Chi T-S, Yeh L-Y, Hsu C-C (2011) Robust emotion recognition by spectro-temporal modulation statistic features. J Ambient Intell Humaniz Comput 3:47–60. doi:10.1007/s12652-011-0088-5

    Article  Google Scholar 

  • Choi S, Yamasaki T, Aizawa K (2015) An interactive system based on yes–no questions for affective image retrieval. In: Proceedings of the 1st international workshop on affect and sentiment in multimedia. ACM, New York, pp 45–50

  • Conner TS, Reid KA (2012) Effects of intensive mobile happiness reporting in daily life. Soc Psychol Pers Sci 3:315–323. doi:10.1177/1948550611419677

    Article  Google Scholar 

  • Desmet P (2005) Measuring emotion: development and application of an instrument to measure emotional responses to products. In: Blythe M, Overbeeke K, Monk A, Wright P (eds) Funology. Springer, The Netherlands, pp 111–123

    Google Scholar 

  • Diener H, Oertel K (2006) Experimental approach to affective interaction in games. In: Proceedings of the first international conference on technologies for e-learning and digital entertainment. Springer, Berlin, pp 507–518

  • Doryab A, Frost M, Faurholt-Jepsen M et al (2015) Impact factor analysis: combining prediction with parameter ranking to reveal the impact of behavior on health outcome. Pers Ubiquit Comput 19:355–365

    Article  Google Scholar 

  • Ekman P, Friesen WV (1986) A new pan-cultural facial expression of emotion. Motiv Emot 10:159–168

    Article  Google Scholar 

  • Fogg BJ (1998) Persuasive computers: perspectives and research directions. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 225–232

  • Frost M, Doryab A, Faurholt-Jepsen M, et al (2013) Supporting disease insight through data analysis: refinements of the monarca self-assessment system. In: Proceedings of the 2013 ACM international joint conference on pervasive and ubiquitous computing. ACM, New York, pp 133–142

  • Fuentes C, Gerea C, Herskovic V et al (2015a) User interfaces for self-reporting emotions: a systematic literature review. In: Ubiquitous computing and ambient intelligence. Sensing, processing, and using environmental information—9th international conference, UCAmI 2015. Springer, pp 321–333

  • Fuentes C, Rodríguez I, Herskovic V (2015b) EmoBall: a study on a tangible interface to self-report emotional information considering digital competences. In: Volume 9456 of the series lecture notes in computer science, pp 189–200

  • Gallacher S, O’Connor J, Bird J et al (2015) Mood squeezer: lightening up the workplace through playful and lightweight interactions. In: Proceedings of the 18th ACM conference on computer supported cooperative work and social computing. ACM, pp 891–902

  • Gross T (2009) Towards a new human-centred computing methodology for cooperative ambient intelligence. J Ambient Intell Humaniz Comput 1:31–42. doi:10.1007/s12652-009-0004-4

    Article  Google Scholar 

  • Hastings J, Brass A, Caine C et al (2014) Evaluating the emotion ontology through use in the self-reporting of emotional responses at an academic conference. J Biomed Semant 5:1–17. doi:10.1186/2041-1480-5-38

    Article  Google Scholar 

  • Isbister K, Höök K, Sharp M, Laaksolahti J (2006) The sensual evaluation instrument: developing an affective evaluation tool. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1163–1172

  • Isbister K, Höök K, Laaksolahti J, Sharp M (2007) The sensual evaluation instrument: developing a trans-cultural self-report measure of affect. Int J Hum Comput Stud 65:315–328

    Article  Google Scholar 

  • Ishii H (2008) The tangible user interface and its evolution. Commun ACM 51:32–36

    Article  Google Scholar 

  • Jacko JA (2012) Human computer interaction handbook: fundamentals, evolving technologies, and emerging applications. CRC Press, Boca Raton

    Book  Google Scholar 

  • Jacob RJK, Girouard A, Hirshfield LM et al (2008) Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’08). ACM, pp 201–210

  • Jain J, Lund A, Wixon D (2011) The future of natural user interfaces. In: Extended abstracts on human factors in computing systems (CHI EA’11), pp 211–214

  • Killingsworth MA, Gilbert DT (2010) A wandering mind is an unhappy mind. Science 330:932. doi:10.1126/science.1192439

    Article  Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-01, Version 2.3. Keele University, Keele, UK and Durham University, Durham, UK

  • Laaksolahti J, Isbister K, Höök K (2009) Using the sensual evaluation instrument. Digit Creat 20:165–175. doi:10.1080/14626260903083603

    Article  Google Scholar 

  • Lahey B, Girouard A, Burleson W, Vertegaal R (2011) PaperPhone: understanding the use of bend gestures in mobile devices with flexible electronic paper displays. In: Proceedings of the 2011 annual conference on human factors in computing systems, pp 1303–1312

  • Lang PJ (1980) Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski JB, Johnson JH, Williams TA (eds) Technology in mental health care delivery systems. Ablex, Norwood, NJ, USA, pp 119–137

    Google Scholar 

  • Laurans G, Desmet P, Hekkert P (2009) The emotion slider: a self-report device for the continuous measurement of emotion. In: Proceedings of the 3rd international conference on affective computing and intelligent interaction and workshops (ACII 2009), pp 1–6

  • Lee J-H, Park J-Y, Nam T-J (2007) Emotional interaction through physical movement. In: Human–computer interaction. HCI intelligent multimodal interaction environments. Springer, pp 401–410

  • Lin C-L, Gau P-S, Lai K-J et al (2009) Emotion caster: tangible emotion sharing device and multimedia display platform for intuitive interactions. In: Proceedings of the 13th international symposium on consumer electronics (ISCE’09). IEEE, pp 988–989

  • Lopatovska I, Arapakis I (2011) Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction. Inf Process Manag 47:575–592

    Article  Google Scholar 

  • Lottridge D, Chignell M (2009a) Emotional majority agreement a psychometric property of affective self-report instruments. In: Science and technology for humanity (TIC-STH), 2009 IEEE Toronto international conference, pp 795–800

  • Lottridge D, Chignell M (2009b) Emotrace: tracing emotions through human-system interaction. In: Proceedings of the human factors and ergonomics society annual meeting, vol, 53, pp 1541–1545. doi:10.1177/154193120905301916

  • Lyytinen K, Yoo Y (2002) Ubiquitous computing. Commun ACM 45:63–96

    Article  Google Scholar 

  • Matthews M, Doherty G, Sharry J, Fitzpatrick C (2008) Mobile phone mood charting for adolescents. Br J Guid Couns 36:113–129. doi:10.1080/03069880801926400

    Article  Google Scholar 

  • Mody RN, Willis KS, Kerstein R (2009) WiMo: location-based emotion tagging. In: Proceedings of the 8th international conference on mobile and ubiquitous multimedia. ACM, New York, pp 14:1–14:4

  • Mora S, Rivera-Pelayo V, Müller L (2011) Supporting mood awareness in collaborative settings. In: 2011 7th International Conference on collaborative computing: networking, applications and worksharing (CollaborateCom), pp 268–277

  • Morris M, Kathawala Q, Leen T et al (2010) Mobile therapy: case study evaluations of a cell phone application for emotional self-awareness. J Med Internet Res 12:e10

    Article  Google Scholar 

  • Nagel F, Kopiez R, Grewe O, Altenmüller E (2007) EMuJoy: software for continuous measurement of perceived emotions in music. Behav Res Methods 39:283–290

    Article  Google Scholar 

  • Negru S (2010) A conceptual architecture of an arduino-based social-emotional interactive system. In: Proceedings of the proceedings of the 2010 IEEE 6th international conference on intelligent computer communication and processing. IEEE Computer Society, Washington, pp 93–98

  • Neyem A, Aracena C, Collazos CA, Alarcón R (2007) Designing emotional awareness devices: what one sees is what one feels. Ingeniare Revista Chilena de Ingeniería 15:227–235

    Article  Google Scholar 

  • Niforatos E, Karapanos E (2015) EmoSnaps: a mobile application for emotion recall from facial expressions. Pers Ubiquit Comput 19:425–444

    Article  Google Scholar 

  • Papastavrou E, Charalambous A, Tsangari H (2016) Exploring the other side of cancer care: the informal caregiver. Eur J Oncol Nurs 13:128–136. doi:10.1016/j.ejon.2009.02.003

    Article  Google Scholar 

  • Pfister H-R, Wollstädter S, Peter C (2011) Affective responses to system messages in human–computer-interaction: effects of modality and message type. Interact Comput 23:372–383

    Article  Google Scholar 

  • Picard RW (2003) Affective computing: challenges. Int J Hum Comput Stud 59:55–64

    Article  Google Scholar 

  • Picard RW, Picard R (1997) Affective computing. MIT Press, Cambridge

    Book  Google Scholar 

  • Plutchik R (1980) Emotion: a psychoevolutionary synthesis. Harpercollins College Division, New York

    Google Scholar 

  • Polzin TS, Waibel A (2000) Emotion-sensitive human–computer interfaces. In: ISCA tutorial and research workshop (ITRW) on speech and emotion

  • Read R, Belpaeme T (2013) Using the AffectButton to measure affect in child and adult–robot interaction. In: 2013 8th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 211–212

  • Reid SC, Kauer SD, Dudgeon P et al (2008) A mobile phone program to track young people’s experiences of mood, stress and coping. Soc Psychiatry Psychiatr Epidemiol 44:501–507. doi:10.1007/s00127-008-0455-5

    Article  Google Scholar 

  • Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178

    Article  Google Scholar 

  • Russell JA, Mehrabian A (1977) Evidence for a three-factor theory of emotions. J Res Pers 11:273–294

    Article  Google Scholar 

  • Sánchez JA, Kirschning I, Palacio JC, Ostróvskaya Y (2005) Towards mood-oriented interfaces for synchronous interaction. In: Proceedings of the 2005 Latin American conference on human–computer interaction (CLIHC’05). ACM, pp 1–7

  • Sánchez JA, Hernández NP, Penagos JC, Ostróvskaya Y (2006) Conveying mood and emotion in instant messaging by using a two-dimensional model for affective states. In: Proceedings of VII Brazilian symposium on human factors in computing systems. ACM, New York, pp 66–72

  • Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inform 44:695–729

    Article  Google Scholar 

  • Schubert E, Ferguson S, Farrar N et al (2013) The six emotion-face clock as a tool for continuously rating discrete emotional responses to music. In: From sounds to music and emotions. Springer, pp 1–18

  • Sebe N (2010) Human-centered computing. In: Nakashima H, Aghajan H, Augusto JC (eds) Handbook of ambient intelligence and smart environments. Springer US, Boston, pp 349–370

    Chapter  Google Scholar 

  • Stovold E, Beecher D, Foxlee R, Noel-Storr A (2014) Study flow diagrams in Cochrane systematic review updates: an adapted PRISMA flow diagram. Syst Rev 3:54. doi:10.1186/2046-4053-3-54

    Article  Google Scholar 

  • Torning K, Oinas-Kukkonen H (2009) Persuasive system design: state of the art and future directions. In: Proceedings of the 4th international conference on persuasive technology. ACM, New York, pp 30:1–30:8

  • Watson D, Tellegen A (1985) Toward a consensual structure of mood. Psychol Bull 98:219

    Article  Google Scholar 

  • Weiser M (1995) The computer for the 21st century: specialized elements of hardware and software, connected by wires, radio waves and infrared, will be so ubiquitous that no one will notice their presence. In: Baecker RM, Grudin J, Buxton WAS, Greenberg S (eds) Readings in human-computer interaction, 2nd edn. Morgan Kaufmann Publishers Inc., San Francisco, pp 933–940

    Google Scholar 

  • Wigdor D, Wixon D (2011) Brave NUI world: designing natural user interfaces for touch and gesture. Morgan Kaufmann, San Francisco

    Google Scholar 

  • Yu S-H, Wang L-S, Chu H-H et al (2011) A mobile mediation tool for improving interaction between depressed individuals and caregivers. Pers Ubiquit Comput 15:695–706

    Article  Google Scholar 

  • Zhao Y, Wang X, Goubran M et al (2012) Human emotion and cognition recognition from body language of the head using soft computing techniques. J Ambient Intell Humaniz Comput 4:121–140. doi:10.1007/s12652-012-0107-1

    Article  Google Scholar 

Download references

Acknowledgements

This project was partially funded by CONICYT Chile PhD scholarship (CONICYT-PCHA/Doctorado Nacional/2013-21130661,2014-63140077), CONICIT and MICIT Costa Rica PhD scholarship and Universidad de Costa Rica, and Fondecyt (Chile) Project No. 1150365.

Author information

Authors and Affiliations

  1. Department of Computer Science, Pontificia Universidad Católica de Chile, Santiago, Chile

    Carolina Fuentes, Valeria Herskovic, Iyubanit Rodríguez & Carmen Gerea

  2. Department of Computer Science, Universidad de Chile, Santiago, Chile

    Maíra Marques

  3. Department of Computer Science, Universidad Católica de la Santísima Concepción, Concepción, Chile

    Pedro O. Rossel

Authors
  1. Carolina Fuentes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valeria Herskovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iyubanit Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carmen Gerea
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Maíra Marques
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pedro O. Rossel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valeria Herskovic.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fuentes, C., Herskovic, V., Rodríguez, I. et al. A systematic literature review about technologies for self-reporting emotional information. J Ambient Intell Human Comput 8, 593–606 (2017). https://doi.org/10.1007/s12652-016-0430-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-016-0430-z

Keywords

Search

Navigation