Continuous emotion recognition (CER) is the task of determining the emotional content of speech from audio or multimedia recordings. Training targets for machine learning must be generated by human annotation, generally as a time series of emotional parameter values. In typical contemporary CER systems and challenges, the mean over a pool of annotators is taken to represent this ground truth, but is this an appropriate model for the emotional content of speech? Using the RECOLA dataset, the primary contribution of this research is to show that a correlation exists between the time-varying disagreement from independent groups of annotators. Because the groups are completely isolated except via the speech signal, this agreement-about-disagreement demonstrates that there is a component of annotator disagreement which arises systematically from the signal itself, which qualitatively implies that the perceived emotional content of speech can exhibit some degree of inherent ambiguity. Additionally, we show that these human annotations exhibit a degree of temporal smoothness. Neither of these characteristics is represented by the standard series-of-means ground-truth model, so we propose two alternative ground-truth models: a mean-variance model that incorporates ambiguity and a more general Gaussian process model that incorporates ambiguity and temporal smoothness in a well-defined probability distribution.
Cite as: Atcheson, M., Sethu, V., Epps, J. (2018) Demonstrating and Modelling Systematic Time-varying Annotator Disagreement in Continuous Emotion Annotation. Proc. Interspeech 2018, 3668-3672, doi: 10.21437/Interspeech.2018-1933
@inproceedings{atcheson18_interspeech,
author={Mia Atcheson and Vidhyasaharan Sethu and Julien Epps},
title={{Demonstrating and Modelling Systematic Time-varying Annotator Disagreement in Continuous Emotion Annotation}},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={3668--3672},
doi={10.21437/Interspeech.2018-1933}
}