Abstract
The literature in the field of traditional Chinese medicine (TCM) contains a large amount of knowledge of traditional Chinese medicine. Such knowledge plays an important role in the automatic diagnosis and treatment of TCM. In order to obtain the above knowledge, the word segmentation of TCM texts is crucial and fundamental. However, the corpus of TCM texts presents mostly in the form of classical Chinese, which is different from modern Chinese, but there are still some cases where the path of modern Chinese and classical Chinese comes in cross. So it is very difficult to use the current universal word segmentation. The lack of manually labeled corpus for TCM word segmentation which is also hard to get makes it difficult to use a supervised method to train the tokenizer applied in the Chinese medicine. In order to solve this problem, we use an unsupervised method which uses entropy as goodness together with traditional Chinese medicine domain dictionary to construct a TCM text-specific tokenizer. Finally, the effectiveness of this method has been proved through experiments on 280 MB TCM texts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xiong, Z., Shen, Q., Wang, Y., Zhu, C.: Paragraph vector representation based on word to vector and CNN learning. CMC: Comput. Mater. Continua 55(2), 213–227 (2018)
Zhang, Y., Wang, Q., Li, Y., Wu, X.: Sentiment classification based on piecewise pooling convolutional neural network. CMC: Comput. Mater. Continua 56, 285–297 (2018)
Sproat, R., Shih, C.: A statistical method for finding word boundaries in Chinese text. Comput. Process. Chin. Orient. Lang. 4(4), 336–351 (1990)
Feng, H., Chen, K., Deng, X., Zheng, W.: Accessor variety criteria for Chinese word extraction. Comput. Linguist. 30(1), 75–93 (2004)
Goldwater, S., Griffiths, T.L., Johnson, M.: Contextual dependencies in unsupervised word segmentation. In: 44th Annual Meeting of the Association for Computational Linguistics Joint with the 21st International Conference on Computational Linguistics, pp. 673–680. Affiliated Organizations, Sydney (2006)
Mochihashi, D., Yamada, T., Ueda, N.: Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling. In: the 47th Annual Meeting of the ACL Joint with the 4th International Joint Conference on Natural Language Processing of the AFNLP, Volume 1-Volume 1, pp. 100–108 (2009)
Chang, J.S., Lin, T.: Unsupervised word segmentation without dictionary. In: ROCLING 2003 Poster Papers, pp. 355–359 (2003)
Magistry, P., Sagot, B.: Unsupervized word segmentation: the case for Mandarin Chinese. In: Meeting of the Association for Computational Linguistics, Short Papers-Volume 2, pp. 383–387 (2012)
Magistry, P., Sagot, B.: Can MDL improve unsupervised Chinese word segmentation? In: 6th International Joint Conference on Natural Language Processing, Sighan Workshop, p. 2 (2013)
Jin, Z., Tanaka-Ishii, K.: Unsupervised segmentation of Chinese text by use of branching entropy. In: The COLING/ACL on Main Conference Poster Sessions, pp. 428–435 (2006)
Harris, Z.S.: From phoneme to morpheme. Language 31(2), 190–222 (1955)
Zhao, H., Kit, C.: An empirical comparison of goodness measures for unsupervised Chinese word segmentation with a unified framework. In: 3th International Joint Conference on Natural Language Processing, Volume-I (2008)
Chen, S., Xu, Y., Chang, H.: A simple and effective unsupervised word segmentation approach. In: AAAI Conference on Artificial Intelligence, pp. 866–871. AAAI Press (2011)
Wang, H., Zhu, J., Tang, S., Fan, X.: A new unsupervised approach to word segmentation. Comput. Linguist. 37(3), 421–454 (2011)
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3(Feb), 1137–1155 (2003)
Qinjun, Q., Zhong, X., Liang, W.: A cyclic self-learning Chinese word segmentation for the geoscience domain. Geomatica 72(1), 16–26 (2018)
Qiu, Q., Xie, Z., Wu, L., Li, W.: DGeoSegmenter: a dictionary-based Chinese word segmenter for the geoscience domain. Comput. Geosci. 121, 1–11 (2018)
19(3), 536–541 (2017)Heafield, K.: KenLM: faster and smaller language model queries. In: The Workshop on Statistical Machine Translation, pp. 187–197 (2011)
19(3), 536–541 (2017)Acknowledgment
This work is supported by the National Key Research and Development Program of China under Grant 2017YFB1002304. We would also like to thank the anonymous reviewers for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Jia, Q., Xie, Y., Xu, C., Zhou, Y., Zhang, D. (2019). Unsupervised Traditional Chinese Medicine Text Segmentation Combined with Domain Dictionary. In: Sun, X., Pan, Z., Bertino, E. (eds) Artificial Intelligence and Security. ICAIS 2019. Lecture Notes in Computer Science(), vol 11632. Springer, Cham. https://doi.org/10.1007/978-3-030-24274-9_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-24274-9_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-24273-2
Online ISBN: 978-3-030-24274-9
eBook Packages: Computer ScienceComputer Science (R0)