Text Classification Network Based on Token Dynamic Decay

The existing pre-trained models have made significant strides in reducing computation time, with a substantial number of these models expediting inference by eliminating less impactful words from the text. However, these models tend to overlook the generalization effect of simple samples on high-level model parameters. Moreover, the direct deletion of words in these models can result in a notable number of misdeletions, thereby impacting the original semantic information of the text. To address these issues, this paper introduces a model for Sample Influence Adaptation Training and Token Dynamically Attenuation Inferred. Specifically, during the training phase, we dynamically adjust the text masking by assessing the semantic analysis difficulty of the text, thereby diminishing the generalization ability of simple samples on high-level models. In the inference phase, a double-layer superimposed influence score method is employed to dynamically decay and remove unimportant tokens. Subsequently, various dimensions of textual information are extracted to calculate the influence score for each token, ultimately eliminating tokens with reduced influence. Extensive experiments conducted on text classification datasets demonstrate that our approach achieves comparable inference speeds to other models while maintaining higher accuracy across multiple datasets.