Abstract
Word embeddings have a significant impact on natural language processing. In morpheme writing systems, most Chinese word embeddings take a word as the basic unit, or directly use the internal structure of words. However, these models still neglect the rich relevant derivative meanings in the internal structure of Chinese characters. Based on our observations, the relevant derivative meanings of the main-components in Chinese characters are very helpful for improving Chinese word embeddings learning. In this paper, we focus on employing the relevant derivative meanings of the main-components in the Chinese characters to train and enhance the Chinese word embeddings. To this end, we propose two main-component enhanced word embedding models named MCWE-SA and MCWE-HA respectively, which incorporate the relevant derivative meanings of the main-components during the training process based on the attention mechanism. Our models can fine-grained enhance the precision of word embeddings without generating additional vectors. Experiments on word similarity and syntactic analogy tasks are conducted to validate the feasibility of our models. Furthermore, the results show that our models have a certain improvement in the similarity task over most baselines, and have nearly 3% improvement in Chinese analogical reasoning dataset compared with the state-of-the-art model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
References
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., Qin, B.: Learning sentiment-specific word embedding for twitter sentiment classification. In: ACL, pp. 1555–1565 (2014)
Conneau, A., Schwenk, H., Barrault, L., Lecun, Y.: Very deep convolutional networks for natural language processing. arXiv preprint arXiv:1606.01781 (2016)
Jean, S., Cho, K., Memisevic, R., Bengio, Y.: On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007 (2015)
Kyunghyun, C., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv: 1406.1078 (2014)
Bonggun, S., Timothy, L., Jinho, D.: Lexicon integrated cnn models with attention for sentiment analysis. arXiv preprint arXiv:1610.06272 (2016)
Tang, D., Wei, F., Qin, B., Yang, N., Liu, T., Zhou, M.: Sentiment embeddings with applications to sentiment analysis. In: IEEE Transactions on Knowledge and Data Engineering, pp. 496–509 (2016)
Guangyou, Z., Tingting, H., Jun, Z, Po, H.: Learning continuous word embedding with metadata for question retrieval in community question answering. In: ACL, pp. 250–259 (2015)
Antoine, B., Sumit, C., Jason, W.: Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676 (2014)
Tomas, M., Kai, C., Greg, C., Jeffrey, D.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013a)
Tomas, M., Ilya, S., Kai, C., Greg, S., Jeff, D.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013b)
Tomas, M., Wen-tau, Y., Geoffrey, Z.: Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 746–751 (2013c)
Jeffrey, P., Richard, S., Christopher, M.: Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543 (2014)
Harris, Z.: Distributional structure. Word 10(23), 146–162 (1954). https://doi.org/10.1080/00437956.1954.11659520
Li, Y., Li, W., Sun, F., Li, S.: Component-enhanced Chinese character embeddings. In: Proceedings of EMNLP, pp. 829–834 (2015)
Chen, X., Xu, L., Liu, Z., Sun, M., Luan, H.: Joint learning of character and word embeddings. In: IJCAI, pp. 1236–1242 (2015)
Rongchao, Y., Quan, W., Peng, L., Rui, L., Bin, W.: Multi-granularity Chinese word embedding. In: Proceedings of EMNLP, pp. 981–986 (2016)
Xu, J., Liu, J., Zhang, L., Li, Z., Chen, H.: Improve Chinese word embeddings by exploiting internal structure. In: NAACL, pp. 1041–1050 (2016)
Yu, J., Jian. X., Xin, H., Song, Y.: Joint embeddings of Chinese words, characters, and fine-grained subcharacter components. In: EMNLP, pp. 286–291 (2017)
Tzu-Ray, S., Hung-Yi, L.: Learning Chinese word representations from glyphs of characters. In: EMNLP (2017)
Cao, S., Lu, W., Zhou, J., Li, X.: cw2vec: Learning Chinese word embeddings with stroke n-gram information. In: AAAI (2018)
Kelvin, X., et al.: Show, attend and tell: neural image caption generation with visual attention. In: ICML (2015)
Xu, Y., Liu, J., Yang, W., Huang, L.: Incorporating latent meanings of morphological compositions to enhanceword embeddings. In: ACL, pp. 1232–1242 (2018)
Lai, S., Liu, K., Xu, L., Zhao, J.: How to generate a good word embedding?. arXiv preprint arXiv:1507.05523 (2015)
Piotr, B., Edouard, G., Armand, J., Tomas, M.: Enriching word vectors with subword information. In: ACL, pp. 135–146 (2017)
Chen, Z., Hu, K.: Radical enhanced Chinese word embedding. In: CCL(The Seventeenth China National Conference on Computational Linguistics) (2018)
Acknowledgments
The authors are grateful to the reviewers for constructive feedback. We would like to thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Natural Science Foundation of China (No. 61572456) and the Anhui Initiative in Quantum Information Technologies (No. AHY150300).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Su, X., Yang, W., Wang, J. (2019). Enhancing Chinese Word Embeddings from Relevant Derivative Meanings of Main-Components in Characters. In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds) Chinese Computational Linguistics. CCL 2019. Lecture Notes in Computer Science(), vol 11856. Springer, Cham. https://doi.org/10.1007/978-3-030-32381-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-32381-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32380-6
Online ISBN: 978-3-030-32381-3
eBook Packages: Computer ScienceComputer Science (R0)