Disfluencies and Fine-Tuning Pre-Trained Language Models for Detection of Alzheimer’s Disease

Jiahong Yuan, Yuchen Bian, Xingyu Cai, Jiaji Huang, Zheng Ye, Kenneth Church

Disfluencies and language problems in Alzheimer’s Disease can be naturally modeled by fine-tuning Transformer-based pre-trained language models such as BERT and ERNIE. Using this method, we achieved 89.6% accuracy on the test set of the ADReSS (Alzheimer’s Dementia Recognition through Spontaneous Speech) Challenge, a considerable improvement over the baseline of 75.0%, established by the organizers of the challenge. The best accuracy was obtained with ERNIE, plus an encoding of pauses. Robustness is a challenge for large models and small training sets. Ensemble over many runs of BERT/ERNIE fine-tuning reduced variance and improved accuracy. We found that um was used much less frequently in Alzheimer’s speech, compared to uh. We discussed this interesting finding from linguistic and cognitive perspectives.

doi: 10.21437/Interspeech.2020-2516

Cite as: Yuan, J., Bian, Y., Cai, X., Huang, J., Ye, Z., Church, K. (2020) Disfluencies and Fine-Tuning Pre-Trained Language Models for Detection of Alzheimer’s Disease. Proc. Interspeech 2020, 2162-2166, doi: 10.21437/Interspeech.2020-2516

@inproceedings{yuan20_interspeech,
  author={Jiahong Yuan and Yuchen Bian and Xingyu Cai and Jiaji Huang and Zheng Ye and Kenneth Church},
  title={{Disfluencies and Fine-Tuning Pre-Trained Language Models for Detection of Alzheimer’s Disease}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={2162--2166},
  doi={10.21437/Interspeech.2020-2516}
}