skip to main content
10.1145/2070481.2070485acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Mining multimodal sequential patterns: a case study on affect detection

Published: 14 November 2011 Publication History

Abstract

Temporal data from multimodal interaction such as speech and bio-signals cannot be easily analysed without a preprocessing phase through which some key characteristics of the signals are extracted. Typically, standard statistical signal features such as average values are calculated prior to the analysis and, subsequently, are presented either to a multimodal fusion mechanism or a computational model of the interaction. This paper proposes a feature extraction methodology which is based on frequent sequence mining within and across multiple modalities of user input. The proposed method is applied for the fusion of physiological signals and gameplay information in a game survey dataset. The obtained sequences are analysed and used as predictors of user affect resulting in computational models of equal or higher accuracy compared to the models built on standard statistical features.

References

[1]
R. Agrawal and R. Srikant. Mining sequential patterns. In icde, page 3. Published by the IEEE Computer Society, 1995.
[2]
A. Ben-Hur and D. Brutlag. Remote homology detection: a motif based approach. Bioinformatics, 19(suppl 1):i26-i33, 2003.
[3]
A. Drachen, L. Nacke, G. N. Yannakakis, and A. L. Pedersen. Correlation between heart rate, electrodermal activity and player experience in First-Person Shooter games. In SIGGRAPH'10, 2010.
[4]
P. Ferreira and P. Azevedo. Protein sequence classification through relevant sequence mining and bayes classifiers. Progress in AI, pages 236--247, 2005.
[5]
J. Fürnkranz and E. Hüllermeier. Preference learning. Künstliche Intelligenz, 19(1):60--61, 2005.
[6]
J. Goldberger, S. Challapalli, R. Tung, M. Parker, and A. Kadish. Relationship of heart rate variability to parasympathetic effect. Circulation, 103(15):1977, 2001.
[7]
J. Han and M. Kamber. Data mining: concepts and techniques. Morgan Kaufmann, 2006.
[8]
N. Lesh, M. Zaki, and M. Ogihara. Mining features for sequence classification. In Proc. of the fifth ACM SIGKDD Int. Conf. on Knowledge discovery and data mining, pages 342--346. ACM, 1999.
[9]
R. Mandryk, K. Inkpen, and T. Calvert. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology, 25(2):141--158, 2006.
[10]
R.L. Mandryk, and M.S Atkins. A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4):329--347, 2007.
[11]
H. P. Martinez, K. Hullett, and G. N. Yannakakis. Extending Neuro-evolution Preference Learning through Player Modeling. In Proc. of 2010 IEEE CIG, 2010.
[12]
H. P. Martínez and G. N. Yannakakis. Genetic search feature selection for affective modeling: a case study on reported preferences. In Proc. of the 3rd AFFINE workshop, pages 15--20. ACM, 2010.
[13]
S. Mcquiggan, B. Mott, and J. Lester. Modeling self-efficacy in intelligent tutoring systems: An inductive approach. UMUAI, 18(1):81--123, 2008.
[14]
M. Pantic and L. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370--1390, 2003.
[15]
M. Pantic, M. Valstar, R. Rademaker, and L. Maat. Web-based database for facial expression analysis. In 2005 IEEE International Conference on Multimedia and Expo, page 5. IEEE, 2005.
[16]
N. Ravaja, T. Saari, J. Laarni, K. Kallinen, M. Salminen, J. Holopainen, and A. Järvinen. The psychophysiology of video gaming: Phasic emotional responses to game events. In Proc. of DiGRA'05, 2005
[17]
N. Ravaja, T. Saari, M. Salminen, J. Laarni, and K. Kallinen. Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology, 8(4):343--367, 2006.
[18]
R. Srikant and R. Agrawal. Mining sequential patterns: Generalizations and performance improvements. Advances in Database Technology - EDBT'96, pages 1--17, 1996.
[19]
G. N. Yannakakis. Preference Learning for Affective Modeling. In Proc. of ACII'09, 2009.
[20]
G. N. Yannakakis, H. P. Martínez, and A. Jhala. Towards affective camera control in games. UMUAI, 20:313--340, 2010.

Cited By

View all
  • (2024)Automatic Context-Aware Inference of Engagement in HMI: A SurveyIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327870715:2(445-464)Online publication date: Apr-2024
  • (2023)Affective Game Computing: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3315689111:10(1423-1444)Online publication date: Oct-2023
  • (2022)Building a Behavioral Profile and Assessing the Skill of Video Game PlayersIEEE Sensors Journal10.1109/JSEN.2021.312708322:1(481-488)Online publication date: 1-Jan-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '11: Proceedings of the 13th international conference on multimodal interfaces
November 2011
432 pages
ISBN:9781450306416
DOI:10.1145/2070481
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. game events
  2. heart rate variability
  3. preference learning
  4. sequence classification
  5. sequence pattern mining
  6. skin conductance

Qualifiers

  • Research-article

Conference

ICMI'11
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Automatic Context-Aware Inference of Engagement in HMI: A SurveyIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327870715:2(445-464)Online publication date: Apr-2024
  • (2023)Affective Game Computing: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3315689111:10(1423-1444)Online publication date: Oct-2023
  • (2022)Building a Behavioral Profile and Assessing the Skill of Video Game PlayersIEEE Sensors Journal10.1109/JSEN.2021.312708322:1(481-488)Online publication date: 1-Jan-2022
  • (2022)Predicting Mood from Digital Footprints Using Frequent Sequential Context Patterns FeaturesInternational Journal of Human–Computer Interaction10.1080/10447318.2022.207332139:10(2061-2075)Online publication date: 14-Jun-2022
  • (2019)Modeling Dyadic and Group Impressions with Intermodal and Interperson FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/326575415:1s(1-30)Online publication date: 24-Jan-2019
  • (2019)A Methodology for the Automatic Extraction and Generation of Non-Verbal Signals Sequences Conveying Interpersonal AttitudesIEEE Transactions on Affective Computing10.1109/TAFFC.2017.275377710:4(585-598)Online publication date: 1-Oct-2019
  • (2019)Your Gameplay Says It All: Modelling Motivation in Tom Clancy’s The Division2019 IEEE Conference on Games (CoG)10.1109/CIG.2019.8848123(1-8)Online publication date: Aug-2019
  • (2019)The Newborn World: Guiding Creativity in a Competitive Storytelling Game2019 IEEE Conference on Games (CoG)10.1109/CIG.2019.8848116(1-8)Online publication date: Aug-2019
  • (2019)Fusing Level and Ruleset Features for Multimodal Learning of Gameplay Outcomes2019 IEEE Conference on Games (CoG)10.1109/CIG.2019.8848015(1-8)Online publication date: Aug-2019
  • (2019)Rich Representations for Analyzing Learning Trajectories: Systematic Review on Sequential Data Analytics in Game-Based Learning ResearchData Analytics Approaches in Educational Games and Gamification Systems10.1007/978-981-32-9335-9_2(27-53)Online publication date: 11-Sep-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media