Abstract
Humans are prone to making typos in writing, which, though, doesn’t affect understanding the whole sentence. However, neural models in natural language processing(NLP) would collapse when confronted with such tiny mistakes. This problem results from that neural models incline to entangle information, i.e., replacing a single aspect of the input text leads to significant changes in all components of the representation. Therefore, a trivial noise in a sentence can bring about a dramatic performance drop of the model. In this paper, we propose a novel and general framework to enhance the robustness of a model. The whole framework is trained in an adversarial style, which enables the model to encode the original sentence and the sentence refined by a lexical distiller to a similar sentence representation. We verify the effectiveness of the proposed framework in auto-encoder task. Experimental results show that our framework enhances the robustness of the model in different aspects.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Cheng, Y., Tu, Z., Meng, F., Zhai, J., Liu, Y.: Towards robust neural machine translation. arXiv preprint arXiv:1805.06130 (2018)
Goodfellow, I.J., et al.: Generative adversarial nets, pp. 2672–2680 (2014)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. In: International Conference on Learning Representations (2017)
Jia, R., Liang, P.: Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328 (2017)
Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: AAAI, vol. 333, pp. 2267–2273 (2015)
Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp. 3104–3112 (2014)
Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
Tang, Y., Eliasmith, C.: Deep networks for robust visual recognition. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1055–1062. Citeseer (2010)
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)
Wang, Y., Bansal, M.: Robust machine comprehension models via adversarial training. arXiv preprint arXiv:1804.06473 (2018)
Acknowledgments
We thank the reviewers for their valuable comments. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106602).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Qin, W., Zhao, D. (2021). Enhancing Model Robustness via Lexical Distilling. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13029. Springer, Cham. https://doi.org/10.1007/978-3-030-88483-3_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-88483-3_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88482-6
Online ISBN: 978-3-030-88483-3
eBook Packages: Computer ScienceComputer Science (R0)
