Human Versus Machine and Human-Machine Teaming on Masked Language Modeling Tasks

Abstract

The objective of AI-based masked language modeling (MLM) is to mask one or more words in a sentence and have the Natural Language Processing (NLP) model identify the masked words given the other words (representing context) in a sentence. In this study, using real examples collected from an online translation study group, we identify multiple human strategies to perform masked language modeling tasks: looking up definitions; comparing/contrasting; relying on common sense and knowledge; relying on statistical properties of past experiences; building an augmented context (using a list of keywords); and using it on a web search engine. In terms of human versus machine performance, the MLM algorithm’s performance is equal to the level of an average human expert, but it still cannot compete with the best human performance. The human experts’ strengths are the awareness of global knowledge, deep understanding of concepts, events, public opinion, etc. The human experts’ usual weaknesses, on the other hand, are the lack of domain knowledge and human biases. The machine’s strength is its comprehensive and encompassing coverage gleaned by learning from a large corpus, so that it can sometimes fill human experts’ knowledge gaps and correct human bias. But the machine could suffer from lack of true understanding and machine bias due to misleading statistical patterns. One common trait shared by human experts and the MLM algorithm is that they both can make decisions based on statistical observations. Therefore, it stands to reason that a human and machine can form a team to achieve better overall performance. In most cases, humans are not aware of their knowledge limitation or bias, so AI algorithm should take a proactive role in making suggestions, not a reactive role to be activated when human feels the need. In addition, it would be beneficial if the AI algorithm lists definition and sample usages of the word they suggest because humans need to be educated. The important skills demonstrated by human experts seem to be their ability to manipulate context for sensitivity analyses and/or the ability to gauge context-word interactions to understand the context. To improve human-machine teaming, it would be beneficial to incorporate human creativity into interface and interaction designs—humans can quickly input different context manipulation and word-context combinations, and the machine can provide quick feedback based on its extensive knowledge based from a large corpus. This teaming arrangement helps facilitate the joining of forces between human creativity and machine intelligence.