skip to main content
10.1145/3462244.3479916acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article
Open access

Dynamic Mode Decomposition with Control as a Model of Multimodal Behavioral Coordination

Published: 18 October 2021 Publication History

Abstract

Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination.

References

[1]
Lauren B Adamson and Janet E Frick. 2003. The still face: A history of a shared experimental paradigm. Infancy 4, 4 (2003), 451–473.
[2]
Victor Ardulov, Madelyn Mendlen, Manoj Kumar, Neha Anand, Shanna Williams, Thomas Lyon, and Shrikanth Narayanan. 2018. Multimodal interaction modeling of child forensic interviewing. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 179–185.
[3]
P Boersma and D Weenink. 2002. Praat 4.0: a system for doing phonetics with the computer [Computer software]. Amsterdam: Universiteit van Amsterdam(2002).
[4]
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. IEEE transactions on pattern analysis and machine intelligence 43, 1(2019), 172–186.
[5]
Jeffrey F Cohn and Edward Z Tronick. 1987. Mother–infant face-to-face interaction: The sequence of dyadic states at 3, 6, and 9 months.Developmental psychology 23, 1 (1987), 68.
[6]
R Feldman. 1998. Coding interactive behavior (CIB). Unpublished manual, Department of Psychology, Bar-Ilan University, Ramat-Gan, Israel (1998).
[7]
Ruth Feldman. 2003. Infant–mother and infant–father synchrony: The coregulation of positive arousal. Infant Mental Health Journal: Official Publication of The World Association for Infant Mental Health 24, 1 (2003), 1–23.
[8]
Andrew Gianino and Edward Z Tronick. 1988. The mutual regulation model: The infant’s self and interactive regulation and coping and defensive capacities.(1988).
[9]
Zakia Hammal, Jeffrey F Cohn, and Daniel S Messinger. 2015. Head movement dynamics during play and perturbed mother-infant interaction. IEEE transactions on affective computing 6, 4 (2015), 361–370.
[10]
Patrik N Juslin and Klaus R Scherer. 2005. Vocal expression of affect. The new handbook of methods in nonverbal behavior research (2005), 65–135.
[11]
Lauren Klein, Victor Ardulov, Yuhua Hu, Mohammad Soleymani, Alma Gharib, Barbara Thompson, Pat Levitt, and Maja J Matarić. 2020. Incorporating Measures of Intermodal Coordination in Automated Analysis of Infant-Mother Interaction. In Proceedings of the 2020 International Conference on Multimodal Interaction. 287–295.
[12]
Chloë Leclère, Marie Avril, S Viaux-Savelon, N Bodeau, Catherine Achard, Sylvain Missonnier, Miri Keren, R Feldman, M Chetouani, and David Cohen. 2016. Interaction and behaviour imaging: a novel method to measure mother–infant interaction using video 3D reconstruction. Translational Psychiatry 6, 5 (2016), e816–e816.
[13]
Chloë Leclère, Sylvie Viaux, Marie Avril, Catherine Achard, Mohamed Chetouani, Sylvain Missonnier, and David Cohen. 2014. Why synchrony matters during mother-child interactions: a systematic review. PloS one 9, 12 (2014), e113571.
[14]
Daniel M Messinger, Paul Ruvolo, Naomi V Ekas, and Alan Fogel. 2010. Applying machine learning to infant interaction: The development is in the details. Neural Networks 23, 8-9 (2010), 1004–1016.
[15]
Center on the Developing Child at Harvard University.2012. The science of neglect: The persistent absence of responsive care disrupts the developing brain.
[16]
Joshua L Proctor, Steven L Brunton, and J Nathan Kutz. 2016. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems 15, 1 (2016), 142–161.
[17]
Seokwoo Song, Seungho Kim, John Kim, Wonjeong Park, and Dongsun Yim. 2016. TalkLIME: mobile system intervention to improve parent-child interaction for children with language delay. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 304–315.
[18]
Rainer Stiefelhagen and Jie Zhu. 2002. Head orientation and gaze direction in meetings. In CHI’02 Extended Abstracts on Human Factors in Computing Systems. 858–859.
[19]
Cynthia A Stifter and Michael Rovine. 2015. Modeling dyadic processes using hidden Markov models: A time series approach to mother–infant interactions during infant immunization. Infant and child development 24, 3 (2015), 298–321.
[20]
Edward Tronick, Heidelise Als, Lauren Adamson, Susan Wise, and T Berry Brazelton. 1978. The infant’s response to entrapment between contradictory messages in face-to-face interaction. Journal of the American Academy of Child psychiatry 17, 1(1978), 1–13.
[21]
Edward Z Tronick and Andrew Gianino. 1986. Interactive mismatch and repair: challenges to the coping infant.Zero to Three (1986).
[22]
Omri Weisman, Mohamed Chetouani, Catherine Saint-Georges, Nadege Bourvis, Emilie Delaherche, Orna Zagoory-Sharon, David Cohen, and Ruth Feldman. 2016. Dynamics of Non-Verbal Vocalizations and Hormones during Father-Infant Interaction.IEEE Trans. Affect. Comput. 7, 4 (2016), 337–345.
[23]
Hao Zhang, Clarence W Rowley, Eric A Deem, and Louis N Cattafesta. 2019. Online dynamic mode decomposition for time-varying systems. SIAM Journal on Applied Dynamical Systems 18, 3 (2019), 1586–1609.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
October 2021
876 pages
ISBN:9781450384810
DOI:10.1145/3462244
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. dyadic interaction
  2. dynamical systems models
  3. infant-mother interaction
  4. multimodal behavioral coordination

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

ICMI '21
Sponsor:
ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
October 18 - 22, 2021
QC, Montréal, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 556
    Total Downloads
  • Downloads (Last 12 months)201
  • Downloads (Last 6 weeks)36
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media