Abstract
Punctuation prediction, as a key step of connecting speech recognition and natural language processing, has a profound impact on subsequent tasks. Although methods of bidirectional long short term memory and conditional random fields (BiLSTM+CRF) are proposed and remain advanced for a long time, it still suffers from the lack of ability of capturing long-distance interactions among words and extracting useful semantic information, especially in Chinese punctuation prediction. In this paper, considering the characteristic of Chinese punctuation symbols, we propose a novel method of punctuation standardization. In our BiLSTM+CRF based network, adaptive attention and dependency parsing tree are utilized to capture the long distance interactions and extract useful semantic information, and thus enhancing the word representation. As for the performance, the first proposal of Chinese punctuation prediction outperforms BiLSTM+CRF with a gain of 0.292% and 0.127% on accuracy in two datasets respectively. The second proposal outperforms existing methods with a gap of above 4.5% of accuracy and reaches state-of-the-art performance in two datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xu, K., Xie, L., Yao, K.: Investigating LSTM for punctuation prediction. International Symposium on Chinese Spoken Language Processing (2017)
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 (2016)
Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF. In: Proceedings of ACL (2016)
Santos, C.N.D., Zadrozny, B.: Learning character-level representations for part-of-speech tagging. In: International Conference on Machine Learning (2014)
Passos, A., Kumar, V., Mccallum, A.: Lexicon infused phrase embeddings for named entity resolution. In: Computer Science (2014)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. In: Computer Science (2015)
Wang, L., Li, S., Wong, D.F., Chao, L.S.: A joint chinese named entity recognition and disambiguation system. In: Proceedings of Empirical Methods in Natural Language Processing (2015)
Strubell, E., Verga, P., Belanger, D., Mccallum, A.: Fast and accurate sequence labeling with iterated dilated convolutions. arXiv preprint arXiv:1702.02098 (2017)
Cucchiarelli, A., Velardi, P.: Unsupervised named entity recognition using syntactic and semantic contextual evidence. Comput. Linguist. 27, 123–131 (2001)
Sasano, R., Kurohashi, S.: Japanese named entity recognition using structural natural language processing. International Joint Conference on Natural Language Processing (2008)
Ling, X., Weld, D.S.: Fine-grained entity recognition. In: AAAI (2012)
Liu, Y., Stolcke, A., Shriberg, E., Harper, M.: Using conditional random fields for sentence boundary detection in speech. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 451–458 (2005)
Xu, C., Xie, L., Huang, G., Xiao, X., Chng, E.S., Li, H.: A deep neural network approach for sentence boundary detection in broadcast news. International Speech Communication Association, pp. 2887–2891 (2014)
Kim, J.-H., Woodland, P.C.: The use of prosody in a combined system for punctuation generation and speech recognition. In: International Speech Communication Association, pp. 2757–2760 (2001)
Christensen, H., Gotoh, Y., Renals, S.: Punctuation annotation using statistical prosody models. In: International Symposium on Computer Architecture (2001)
Huang, J., Zweig, G.: Maximum entropy model for punctuation annotation from speech. In: Seventh International Conference on Spoken Language Processing (2002)
Lu, W., Ng, H.T.: Better punctuation prediction with dynamic conditional random fields. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 177–186 (2010)
Zhang, D., Wu, S., Yang, N., Li, M.: Punctuation prediction with transition-based parsing. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 752–760 (2013)
Graves, A., Mohamed, A.-.R, Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (2013)
Wang, X., Sim, K.C., Ng, H.T.: Combining punctuation and disfluency prediction: an empirical study. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 121–130 (2014)
Tilk, O., Alumäe, T.: LSTM for punctuation restoration in speech transcripts. In: Sixteenth Annual Conference of the International Speech Communication Association (2015)
Jie, Z., Lu, W.: Dependency-guided LSTM-CRF for named entity recognition. In: Proceedings of Empirical Methods in Natural Language Processing (2019)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. Computer Science (2014)
Luo, L., et al.: An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics 34, 1381–1388 (2017)
Vaswani, A., et al.: Attention is all you need (2017)
Wang, X., Jiang, Y., Bach, N., et al.: Structure-level knowledge distillation for multilingual sequence labeling. In: Processings of ACL (2020)
He, K., Zhang, X., Ren, S., Jian, S.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
Zhang, Y., Yang, J.: Chinese NER using lattice LSTM. In: Proceedings of ACL (2018)
Zhang, Y., Li, Z., Zhang, M.: Efficient second-order TreeCRF for neural dependency parsing. In: Proceedings of ACL (2020)
Sun, K., Zhang, R., Mensah, S., Mao, Y., Liu, X.: Aspect-level sentiment analysis via convolution over dependency tree. In: Proceedings of Empirical Methods in Natural Language Processing (2019)
Vania, C., Kementchedjhieva, Y., Søgaard, A., Lopez, A.: A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. In: Proceedings of Empirical Methods in Natural Language Processing (2019)
Acknowledgement
This paper is supported by National Key Research and Development Program of China under grant No. 2018YFB1003500, No. 2018YFB0204400 and No. 2017YFB1401202.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yan, Z., Wang, J., Cheng, N., Wu, T., Xiao, J. (2021). Chinese Punctuation Prediction with Adaptive Attention and Dependency Tree. In: Chen, H., Liu, K., Sun, Y., Wang, S., Hou, L. (eds) Knowledge Graph and Semantic Computing: Knowledge Graph and Cognitive Intelligence. CCKS 2020. Communications in Computer and Information Science, vol 1356. Springer, Singapore. https://doi.org/10.1007/978-981-16-1964-9_1
Download citation
DOI: https://doi.org/10.1007/978-981-16-1964-9_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-1963-2
Online ISBN: 978-981-16-1964-9
eBook Packages: Computer ScienceComputer Science (R0)