Analysis of Native Listeners’ Facial Microexpressions While Shadowing Non-Native Speech — Potential of Shadowers’ Facial Expressions for Comprehensibility Prediction

Tasavat Trisitichoke, Shintaro Ando, Daisuke Saito, Nobuaki Minematsu

Recently, researchers’ attention has been paid to pronunciation assessment not based on comparison between L2 speech and native models, but based on comprehensibility of L2 speech [1, 2, 3]. In our previous studies [4, 5, 6], native listeners’ shadowing of L2 speech was examined and it was shown that delay of shadowing and accuracy of articulation in shadowing utterances, both of which were acoustically calculated, are strongly influenced by the amount of cognitive load imposed for understanding L2 speech, especially when it is with strong accents. In this paper, aside from acoustic analysis of shadowings, we focus on shadowers’ facial microexpressions and examine how they are correlated with perceived comprehensibility. To extract facial expression features, two methods are tested. One is a computer-vision-based method and recorded videos of shadowers’ facial expressions are analyzed. The other is a method using a physiological sensor that can detect subtle movements of facial muscles. In experiments, four shadowers’ facial expressions are analyzed, each of whom shadowed approximately 800 L2 utterances. Results show that some of shadowers’ facial expressions are highly correlated with perceived comprehensibility, and that those facial expressions are strongly shadower-dependent. These results indicate a high potential of shadowers’ facial expressions for comprehensibility prediction.

doi: 10.21437/Interspeech.2019-1953

Cite as: Trisitichoke, T., Ando, S., Saito, D., Minematsu, N. (2019) Analysis of Native Listeners’ Facial Microexpressions While Shadowing Non-Native Speech — Potential of Shadowers’ Facial Expressions for Comprehensibility Prediction. Proc. Interspeech 2019, 1861-1865, doi: 10.21437/Interspeech.2019-1953

@inproceedings{trisitichoke19_interspeech,
  author={Tasavat Trisitichoke and Shintaro Ando and Daisuke Saito and Nobuaki Minematsu},
  title={{Analysis of Native Listeners’ Facial Microexpressions While Shadowing Non-Native Speech — Potential of Shadowers’ Facial Expressions for Comprehensibility Prediction}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={1861--1865},
  doi={10.21437/Interspeech.2019-1953}
}