Task-Related Pretraining with Whole Word Masking for Chinese Coherence Evaluation

Abstract

This paper presents an approach for evaluating coherence in Chinese middle school student essays, addressing the challenges of time-consuming and inconsistent essay assessment. Previous approaches focused on linguistic features, but coherence, crucial for essay organization, has received less attention. Recent works utilized neural networks, such as CNN, LSTM, and transformers, achieving good performance with labeled data. However, labeling coherence manually is costly and time-consuming. To address this, we propose a method that pretrains RoBERTa with whole word masking (WWM) on a low-resource dataset of middle school essays, followed by finetuning for coherence evaluation. The WWM pretraining is unsupervised and captures general characteristics of the essays, adding little cost to the low-resource setting. Experimental results on Chinese essays demonstrate that this strategy improves coherence evaluation compared to naive finetuning on limited data. We also explore variants of their method, including pseudo labeling and additional neural networks, providing insights into potential performance trade-offs. The contributions of this work include the collection and curation of a substantial dataset, the proposal of a cost-effective pretraining method, and the exploration of alternative approaches for future research.

Appendix

During the process of improving overall accuracy, we have also experimented with some new model architectures, including the Cross Task Grader model mentioned below.

1.1 PFT+HAN

We proposed a multi-layer coherence evaluation model, depicted in Fig. 1, which firstly utilized pre-trained RoBERTa to extract features from the articles, followed by an attention pooling layer. Then, we concatenated punctuation-level embeddings and passed them through another attention pooling layer. Finally, we obtained the ultimate coherence score by using a classifier.

Fig. 1.

PFT+HAN

Fig. 2.

Cross Task Grader

Pre-trained Encoder. A sequence of words $s_i=\{w_1,w_2,\ldots ,w_m\}$is encoded with the pre-trained RoBERTa.

Paragraph Representation Layer. An attention pooling layer applied to the output of the pre-trained encoder layer is designed to capture the paragraph representations and is defined as follows:

$$\begin{aligned} m_{i}=tanh({W_m}\cdot {x_i}+{b_m}) \end{aligned}$$

(2)

$$\begin{aligned} u_i=\frac{e^{{w_u}\cdot {m_i}}}{\sum \limits _{j=1}^{m} e^{{w_u}\cdot {m_j}}} \end{aligned}$$

(3)

$$\begin{aligned} p=\sum \limits _{i=1}^{m} {u_i}\cdot {x_i} \end{aligned}$$

(4)

where $W_m$ is a weights matrix, $w_u$ is a weights vector, $m_i$ is the attention vector for the i-th word, $u_i$ is the attention weight for the i-th word, and p is the paragraph representation.

Essay Representation Layer. We incorporated punctuation representations to enhance the model’s performance. We encoded the punctuation information for each paragraph, obtaining the punctuation representation $pu_i$ for each paragraph. Then, we concatenated this representation $pu_i$ with the content representation $p_i$ of each paragraph:

$$\begin{aligned} c_i=concatenate(p_i,pu_i) \end{aligned}$$

(5)

where $c_i$ represents the representation of the concatenated i-th paragraph. Next, we use another layer of attention pooling to obtain the representation of the entire essay and is defined as follows:

$$\begin{aligned} a_{i}=tanh({W_a}\cdot {c_i}+{b_a}) \end{aligned}$$

(6)

$$\begin{aligned} v_i=\frac{e^{{w_v}\cdot {a_i}}}{\sum \limits _{j=1}^{a} e^{{w_v}\cdot {a_j}}} \end{aligned}$$

(7)

$$\begin{aligned} E=\sum \limits _{i=1}^{a} {v_i}\cdot {c_i} \end{aligned}$$

(8)

where $W_a$ is a weights matrix, $w_v$ is a weights vector, $a_i$ is the attention vector for the i-th paragraph, $v_i$ is the attention weight for the i-th paragraph, and E is the essay representation.

1.2 Cross Task Grader

We also used Multi-task Learning(MTL) in our experiment, which is depicted in Fig. 2.

We used both target data and some pseudo-labeled essays from various grade and created a separate PFT+HAN model for each. To facilitate multi-task learning, we adopted the Hard Parameter Sharing approach, sharing the pre-trained encoder layer and the first layer of attention pooling among all the models.Additionally, we added a cross attention layer before the classifier.

Cross Attention Layer. After obtaining the essay representation, we added a cross attention layer to learn the connections between different essays, defined as follows:

$$\begin{aligned} A=[E_1,E_2,\ldots ,E_N] \end{aligned}$$

(9)

$$\begin{aligned} \alpha ^{i}_{j}=\frac{e^{score(E_i,A_{i,j})}}{\sum \limits _i^{l} e^{score(E_i,A_{i,l})}} \end{aligned}$$

(10)

$$\begin{aligned} P_i=\sum \limits {\alpha ^{i}_{j}}\cdot {A_{i,j}} \end{aligned}$$

(11)

$$\begin{aligned} y_i=concatenate(E_i,P_i) \end{aligned}$$

(12)

where A is a concatenation of the representations for each task $[E_1,E_2,\ldots ,E_N]$, and $\alpha ^{i}_{j}$, is the attention weight. We then calculate attention vector $P_i$ through a summation of the product of each weight $\alpha ^{i}_{j}$ and $A_{i,j}$. The final representation $y_i$ is a concatenation of $E_i$ and $P_i$.