Abstract
Opinion question machine reading comprehension (MRC) requires a machine to answer questions by analyzing corresponding passages. Compared with traditional MRC tasks where the answer to every question is a segment of text in corresponding passages, opinion question MRC is more challenging because the answer to an opinion question may not appear in corresponding passages but needs to be deduced from multiple sentences. In this study, a novel framework based on neural networks is proposed to address such problems, in which a new hybrid embedding training method combining text features is used. Furthermore, extra attention and output layers which generate auxiliary losses are introduced to jointly train the stacked recurrent neural networks. To deal with imbalance of the dataset, irrelevancy of question and passage is used for data augmentation. Experimental results show that the proposed method achieves state-of-the-art performance. We are the biweekly champion in the opinion question MRC task in Artificial Intelligence Challenger 2018 (AIC2018).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bajaj P, Campos D, Craswell N, et al., 2016. MS MARCO: a human generated MAchine Reading COmprehension dataset. https://arxiv.org/abs/1611.09268
Devlin J, Chang MW, Lee K, et al., 2018. BERT: pretraining of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805
He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
He W, Liu K, Liu J, et al., 2018. DuReader: a Chinese machine reading comprehension dataset from real-world applications. Proc Workshop on Machine Reading for Question Answering, p.37–46. https://doi.org/10.18653/v1/W18-2605
Hermann KM, Kočiský T, Grefenstette E, et al., 2015. Teaching machines to read and comprehend. Proc 28th Int Conf on Neural Information Processing Systems, p.1693–1701.
Hochreiter S, Schmidhuber J, 1997. Long short-term memory. Neur Comput, 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Joshi M, Choi E, Weld DS, et al., 2017. TriviaQA: a large scale distantly supervised challenge dataset for reading c omprehension. https://arxiv.org/abs/1705.03551
Kingma DP, Ba J, 2014. Adam: a method for stochastic optimization. https://arxiv.org/abs/1412.6980
Liu JH, Wei W, Sun MS, et al., 2018. A multi-answer multi-task framework for real-world machine reading comprehension. Proc Conf on Empirical Methods in Natural Language Processing, p.2109–2118. https://doi.org/10.18653/v1/D18-1235
Mikolov T, Sutskever I, Chen K, et al., 2013a. Distributed representations of words and phrases and their compositionality. Proc 26th Int Conf on Neural Information Processing Systems, p.3111–3119.
Mikolov T, Chen K, Corrado G, et al., 2013b. Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781
Pan YH, 2016. Heading toward artificial intelligence 2.0. Engineering, 2(4):409–413. https://doi.org/10.1016/J.ENG.2016.04.018
Pascanu R, Mikolov T, Bengio Y, 2012. Understanding the exploding gradient problem. https://arxiv.org/abs/1211.5063v1
Rajpurkar P, Zhang J, Lopyrev K, et al., 2016. SQuAD: 100 000+ questions for machine comprehension of text. Proc Conf on Empirical Methods in Natural Language Processing, p.2383–2392. https://doi.org/10.18653/v1/D16-1264
Richardson M, Burges CJC, Renshaw E, 2013. MCTest: a challenge dataset for the open-domain machine comprehension of text. Proc Conf on Empirical Methods in Natural Language Processing, p.193–203.
Seo M, Kembhavi A, Farhadi A, et al., 2016. Bidirectional attention flow for machine comprehension. https://arxiv.org/abs/1611.01603
Srivastava N, Hinton G, Krizhevsky A, et al., 2014. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res, 15(1):1929–1958.
Sutskever I, Vinyals O, Le QV, 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, p.3104–3112.
Tan CQ, Wei FR, Wang WH, et al., 2018. Multiway attention networks for modeling sentence pairs. Proc 27th Int Joint Conf on Artificial Intelligence, p.4411–4417. https://doi.org/10.24963/ijcai.2018/613
Vinyals O, Fortunato M, Jaitly N, 2015. Pointer networks. Advances in Neural Information Processing Systems, p.2692–2700.
Wang SH, Jiang J, 2016. Learning natural language inference with LSTM. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.1442–1451. https://doi.org/10.18653/v1/N16-1170
Wang W, Yang N, Wei F, et al., 2017. R-NET: Machine Reading Comprehension with Self-matching Networks. Technical Report, Natural Language Computing Group, Microsoft Research Asia, Beijing, China. https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf
Wu HC, Luk RWP, Wong KF, et al., 2008. Interpreting TF-IDF term weights as making relevance decisions. ACM Trans Inform Syst, 26(3):13. https://doi.org/10.1145/1361684.1361686
Wu YH, Schuster M, Chen ZF, et al., 2016. Google’s neural machine translation system: bridging the gap between human and machine translation. https://arxiv.org/abs/1609.08144
Yang Y, Yih WT, Meek C, 2015. WikiQA: a challenge dataset for open-domain question answering. Proc Conf on Empirical Methods in Natural Language Processing, p.2013–2018. https://doi.org/10.18653/v1/D15-1237
Yu AW, Dohan D, Luong MT, et al., 2018. QANet: combining local convolution with global self-attention for reading comprehension. https://arxiv.org/abs/1804.09541
Zhuang YT, Wu F, Chen C, et al., 2017. Challenges and opportunities: from big data to knowledge in AI 2.0. Front Inform Technol Electron Eng, 18(1):3–14. https://doi.org/10.1631/FITEE.1601883
Author information
Authors and Affiliations
Corresponding author
Additional information
Deceased
Project supported by the China Knowledge Centre for Engineering Sciences and Technology (No. CKCEST-2019-1-12) and the National Natural Science Foundation of China (No. 61572434)
Contributors
Xiang-zhou HUANG, Si-liang TANG, Yin ZHANG, and Bao-gang WEI designed the research. Xiang-zhou HUANG processed the data and drafted the manuscript. Si-liang TANG, Yin ZHANG, and Bao-gang WEI helped organize the manuscript. Xiang-zhou HUANG revised and finalized the paper.
Compliance with ethics guidelines
Xiang-zhou HUANG, Si-liang TANG, Yin ZHANG, and Bao-gang WEI declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Huang, Xz., Tang, Sl., Zhang, Y. et al. Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension. Front Inform Technol Electron Eng 21, 1346–1355 (2020). https://doi.org/10.1631/FITEE.1900571
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.1900571