Neural Rasch Model: How Do Word Embeddings Adjust Word Difficulty?

Abstract

The Rasch model is a probabilistic model for analyzing the psychological test response data such as second language tests; it is especially useful to automatically obtain a common scale of difficulty for various types of test questions by fitting the model to the response data of the test takers obtained from various methods such as reading comprehension questions and vocabulary questions. Because a test-taker can answer only some hundreds of vocabulary questions from tens of thousands of words in second language vocabulary, there exists a strong need to estimate the difficulty of the words that were not used during the test. For this purpose, the word frequency in a large corpus was previously used as a major clue. Although recent advancements in natural language processing enable us to obtain considerable semantic information about a word in the form of word embeddings, the manner in which such embeddings can be utilized while adjusting the word difficulty estimation has not been considerably investigated. Herein, we investigate how to effectively leverage word embeddings for adjusting the word difficulty estimates. We propose a novel neural model to fit the test response data. Further, we use the trained weights of our neural model to estimate the difficulty of the words that were not tested. The quantitative and qualitative experimental results denote that our model effectively leverages word embeddings to adjust simple frequency-based word difficulty estimates.

This work was supported by JST, ACT-I Grant Number JPMJPR18U8, Japan. We used the ABCI infrastructure by AIST for computational resources.