research-article

EmotionSense: a mobile phones based adaptive platform for experimental social psychology research

Authors:

Kiran K. Rachuri,

Mirco Musolesi,

Cecilia Mascolo,

Peter J. Rentfrow,

Chris Longworth,

Andrius AucinasAuthors Info & Claims

UbiComp '10: Proceedings of the 12th ACM international conference on Ubiquitous computing

Pages 281 - 290

https://doi.org/10.1145/1864349.1864393

Published: 26 September 2010 Publication History

Abstract

Today's mobile phones represent a rich and powerful computing platform, given their sensing, processing and communication capabilities. Phones are also part of the everyday life of billions of people, and therefore represent an exceptionally suitable tool for conducting social and psychological experiments in an unobtrusive way.

de the ability of sensing individual emotions as well as activities, verbal and proximity interactions among members of social groups. Moreover, the system is programmable by means of a declarative language that can be used to express adaptive rules to improve power saving. We evaluate a system prototype on Nokia Symbian phones by means of several small-scale experiments aimed at testing performance in terms of accuracy and power consumption. Finally, we present the results of real deployment where we study participants emotions and interactions. We cross-validate our measurements with the results obtained through questionnaires filled by the users, and the results presented in social psychological studies using traditional methods. In particular, we show how speakers and participants' emotions can be automatically detected by means of classifiers running locally on off-the-shelf mobile phones, and how speaking and interactions can be correlated with activity and location measures.

References

[1]

}}T. Abdelzaher, Y. Anokwa, P. Boda, J. Burke, D. Estrin, L. Guibas, A. Kansal, S. Madden, and J. Reich. Mobiscopes for Human Spaces. IEEE Pervasive Computing, 6:20--29, 2007.

Digital Library

[2]

}}Audacity. http://audacity.sourceforge.net/.

[3]

}}R. Banse and K. R. Scherer. Acoustic Profiles in Vocal Emotion Expression. Journal of Personality and Social Psychology, 70(3):614--636, 1996.

[4]

}}V. Bezooijen, Otto, and Heenan. Recognition of Vocal Expressions of Emotion: A Three-Nation Study to Identify Universal Characteristics. Journal of Cross-Cultural Psychology, 14:387--406, 1983.

[5]

}}C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, 2006.

Digital Library

[6]

}}N. Bolger, A. Davis, and E. Rafaeli. Diary Methods: Capturing Life as it is Lived. Annual Review of Psychology, 54(1):579--616, 2003.

[7]

}}W. Campbell, D. Sturim, and D. Reynolds. Support Vector Machines using GMM-Supervectors for Speaker Verification. IEEE Signal Processing Letters, 13:308--311, 2006.

[8]

}}T. Choudhury and A. Pentland. Sensing and Modeling Human Networks using the Sociometer. In Proceedings of ISWC '03, pages 216--222, 2003.

Digital Library

[9]

}}L. A. Clark and D. Watson. Mood and the Mundane: Relations Between Daily Life Events and Self-Reported Mood. Journal of Personality and Social Psychology, 54.2:296--308, 1988.

[10]

}}L. Feldman Barrett and J. Russell. Independence and Bipolarity in the Structure of Current Affect. Journal of Personality and Social Psychology, 74:967--984, 1998.

[11]

}}J. Froehlich, M. Y. Chen, S. Consolvo, B. Harrison, and J. A. Landay. MyExperience: A System for In situ Tracing and Capturing of User Feedback on Mobile Phones. In Proceedings of MobiSys '07, pages 57--70, 2007.

Digital Library

[12]

}}H. Hermansky. Perceptual Linear Predictive (PLP) Analysis of Speech. Journal of the Acoustical Society of America, 87(4), 1990.

[13]

}}Hidden Markov Model Toolkit. http://htk.eng.cam.ac.uk/.

[14]

}}D. Jayagopi, H. Hung, C. Yeo, and D. Gatica-Perez. Modeling Dominance in Group Conversations Using Nonverbal Activity Cues. IEEE Transactions on Audio, Speech, and Language Processing, 17(3):501--513, 2009.

Digital Library

[15]

}}M. Kockmann, L. Burget, and J. H. Cernocky. Brno University of Technology System for Interspeech 2009 Emotion Challenge. In Proceedings of Interspeech '09, 2009.

[16]

}}J. Kukkonen, E. Lagerspetz, P. Nurmi, and M. Andersson. BeTelGeuse: A Platform for Gathering and Processing Situational Data. IEEE Pervasive Computing, 8(2):49--56, 2009.

Digital Library

[17]

}}M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell. Emotional Prosody Speech and Transcripts, 2002.

[18]

}}lightblue. http://lightblue.sourceforge.net/.

[19]

}}H. Lu, W. Pan, N. D. Lane, T. Choudhury, and A. T. Campbell. SoundSense: Scalable Sound Sensing for People-centric Applications on Mobile Phones. In Proceedings of MobiSys '09, pages 165--178, 2009.

Digital Library

[20]

}}M. R. Mehl, S. D. Gosling, and J. W. Pennebaker. Personality in Its Natural Habitat: Manifestations and Implicit Folk Theories of Personality in Daily Life. Journal of Personality and Social Psychology, 90(5):862--877, 2006.

[21]

}}M. R. Mehl and J. W. Pennebaker. The Sounds of Social Life: A Psychometric Analysis of Students' Daily Social Environments and Natural Conversations. Journal of Personality and Social Psychology, 84(4):857--870, 2003.

[22]

}}E. Miluzzo, N. D. Lane, K. Fodor, R. Peterson, H. Lu, M. Musolesi, S. B. Eisenman, X. Zheng, and A. T. Campbell. Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application. In Proceedings of SenSys '08, pages 337--350, 2008.

Digital Library

[23]

}}E. M. Morris, Q. Kathawala, K. T. Leen, E. E. Gorenstein, F. Guilak, M. Labhard, and W. Deleeuw. Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness. Journal of Medical Internet Research, 12(2):e10, 2010.

[24]

}}A. S. Pentland. Honest Signals: How They Shape Our World. The MIT Press, 2008.

Digital Library

[25]

}}Pyke. http://pyke.sourceforge.net/.

[26]

}}D. A. Reynolds. An Overview of Automatic Speaker Recognition Technology. In Proceedings of ICASSP'02, pages 300--304, 2002.

[27]

}}B. Schuller, G. Rigoll, and M. Lang. Hidden Markov Model-based Speech Emotion Recognition. In Proceedings of ICME '03, pages 401--404, 2003.

Digital Library

[28]

}}A. Stolcke, L. Ferrer, S. Kajarekar, E. Shriberg, and A. Venkataraman. MLLR Transforms as Features in Speaker Recognition. In Proceedings of Interspeech'05, pages 2425--2428, 2005.

[29]

}}D. Tacconi, O. Mayora, P. Lukowicz, B. Arnrich, C. Setz, G. Troester, and C. Haring. Activity and Emotion Recognition to Support Early Diagnosis of Psychiatric Diseases. In Proceedings of Pervasive Healthcare '08, pages 100--102, 2008.

[30]

}}Y. Wang, J. Lin, M. Annavaram, Q. A. Jacobson, J. Hong, B. Krishnamachari, and N. Sadeh. A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition. In Proceedings of MobiSys '09, pages 179--192, 2009.

Digital Library

[31]

}}D. Wyatt, J. Bilmes, T. Choudhury, and J. A. Kitts. Towards the Automated Social Analysis of Situated Speech Data. In Proceedings of UbiComp '08, pages 168--171, 2008.

Digital Library

Cited By

Sarkar MLee TSahoo P(2024)Smart Healthcare: Exploring the Internet of Medical Things with Ambient IntelligenceElectronics10.3390/electronics1312230913:12(2309)Online publication date: 13-Jun-2024
https://doi.org/10.3390/electronics13122309
Henry LHansen EChimoff JPokstis KKiderman MNaim RKossowsky JByrne MLopez-Guzman SKircanski KPine DBrotman M(2024)Selecting an Ecological Momentary Assessment Platform: Tutorial for ResearchersJournal of Medical Internet Research10.2196/5112526(e51125)Online publication date: 4-Jan-2024
https://doi.org/10.2196/51125
Vigliocco GConvertino LDe Felice SGregorians LKewenig VMueller MVeselic SMusolesi MHudson-Smith ATyler NFlouri ESpiers H(2024)Ecological brain: reframing the study of human behaviour and cognitionRoyal Society Open Science10.1098/rsos.24076211:11Online publication date: 8-Nov-2024
https://doi.org/10.1098/rsos.240762
Show More Cited By

Index Terms

EmotionSense: a mobile phones based adaptive platform for experimental social psychology research

Recommendations

EmotionSense: An Adaptive Emotion Recognition System Based on Wearable Smart Devices
Special Issue on Wearable Technologies for Smart Health: Part 1

With the recent surge of smart wearable devices, it is possible to obtain the physiological and behavioral data of human beings in a more convenient and non-invasive manner. Based on such data, researchers have developed a variety of systems or ...
Employing both gender and emotion cues to enhance speaker identification performance in emotional talking environments

Speaker recognition performance in emotional talking environments is not as high as it is in neutral talking environments. This work focuses on proposing, implementing, and evaluating a new approach to enhance the performance in emotional talking ...
Controlling Home and Office Appliances with Smart Phones

Most home and office appliances contain microprocessors. All these appliances have some user interface, but many users become frustrated with their appliances' difficult, complex functions. However, a new framework, the personal universal controller (...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '10: Proceedings of the 12th ACM international conference on Ubiquitous computing

September 2010

366 pages

ISBN:9781605588438

DOI:10.1145/1864349

General Chairs:
Jakob E. Bardram
IT University of Copenhagen, Denmark
,
Marc Langheinrich
University of Lugano, Switzerland
,
Program Chairs:
Khai N. Truong
University of Toronto, Canada
,
Paddy Nixon
University College Dublin, Ireland

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

University of Florida: University of Florida

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 September 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Ubicomp '10

Sponsor:

Ubicomp '10: The 2010 ACM Conference on Ubiquitous Computing

September 26 - 29, 2010

Copenhagen, Denmark

Acceptance Rates

UbiComp '10 Paper Acceptance Rate 39 of 202 submissions, 19%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

340
Total Citations
View Citations
2,831
Total Downloads

Downloads (Last 12 months)155
Downloads (Last 6 weeks)12

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sarkar MLee TSahoo P(2024)Smart Healthcare: Exploring the Internet of Medical Things with Ambient IntelligenceElectronics10.3390/electronics1312230913:12(2309)Online publication date: 13-Jun-2024
https://doi.org/10.3390/electronics13122309
Henry LHansen EChimoff JPokstis KKiderman MNaim RKossowsky JByrne MLopez-Guzman SKircanski KPine DBrotman M(2024)Selecting an Ecological Momentary Assessment Platform: Tutorial for ResearchersJournal of Medical Internet Research10.2196/5112526(e51125)Online publication date: 4-Jan-2024
https://doi.org/10.2196/51125
Vigliocco GConvertino LDe Felice SGregorians LKewenig VMueller MVeselic SMusolesi MHudson-Smith ATyler NFlouri ESpiers H(2024)Ecological brain: reframing the study of human behaviour and cognitionRoyal Society Open Science10.1098/rsos.24076211:11Online publication date: 8-Nov-2024
https://doi.org/10.1098/rsos.240762
Beatton TChan HDulleck URistl ASchaffner MTorgler B(2024)Positive affect and heart rate variability: a dynamic analysisScientific Reports10.1038/s41598-024-57279-514:1Online publication date: 25-Mar-2024
https://doi.org/10.1038/s41598-024-57279-5
Mallick SPanda M(2024)Predicting Mental Health Disorders in the Technical Workplace: A Study on Feature Selection and Classification AlgorithmsData Management, Analytics and Innovation10.1007/978-981-97-3242-5_13(169-183)Online publication date: 23-Jul-2024
https://doi.org/10.1007/978-981-97-3242-5_13
Xie YGuo XWang YCheng JChen Y(2024)Universal Targeted Adversarial Attacks Against mmWave-Based Human Activity RecognitionNetwork Security Empowered by Artificial Intelligence10.1007/978-3-031-53510-9_7(177-211)Online publication date: 24-Feb-2024
https://doi.org/10.1007/978-3-031-53510-9_7
Piga BRainisio NStancato GBoffi M(2023)Mapping the In-Motion Emotional Urban Experiences: An Evidence-Based MethodSustainability10.3390/su1510796315:10(7963)Online publication date: 12-May-2023
https://doi.org/10.3390/su15107963
Liang DZhang AThomaz E(2023)Automated Face-To-Face Conversation Detection on a Commodity Smartwatch with Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108827:3(1-29)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610882
Prajwal MRaj ASen SSaha SGhosh S(2023)Towards Efficient Emotion Self-report Collection Using Human-AI CollaborationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962697:2(1-23)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596269
Schneegans TNiclas AGreiner KBenke IMädche ABeigl M(2023)Annotating Affect in the Field: A Case Study on the Usability of a Minimalist Smartwatch User Interface for Affect AnnotationProceedings of the 2nd Empathy-Centric Design Workshop10.1145/3588967.3588977(1-5)Online publication date: 23-Apr-2023
https://dl.acm.org/doi/10.1145/3588967.3588977
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten