Cited By
View all- Baradaran RGhiasi RAmirkhani H(2022)A Survey on Machine Reading Comprehension SystemsNatural Language Engineering10.1017/S1351324921000395(1-50)Online publication date: 19-Jan-2022
Unsupervised Story Comprehension with Hierarchical Encoder-Decoder
Authors: 
Bingning Wang, Ting Yao, Qi Zhang, Jingfang Xu, Kang Liu, Zhixing Tian, Jun ZhaoAuthors Info & Claims
Bingning Wang, Ting Yao, Qi Zhang, Jingfang Xu, Kang Liu, Zhixing Tian, Jun ZhaoAuthors Info & ClaimsPages 93 - 100
Abstract
Commonsense understanding is a long-term goal of natural language processing yet to be resolved. One standard testbed for commonsense understanding isStory Cloze Test (SCT) \citemostafazadeh2016corpus, In SCT, given a 4-sentences story, we are expected to select the proper ending out of two proposed candidates. The training set in SCT only contains unlabeled stories, previous works usually adopt the small labeled development data for training, which ignored the sufficient training data and, essentially, not reveal the commonsense reasoning procedure. In this paper, we propose an unsupervised sequence-to-sequence method for story reading comprehension, we only adopt the unlabeled story and directly model the context-target inference probability. We propose a loss-reweight training strategy for the seq-to-seq model to dynamically tuning the training process. Experimental results demonstrate the advantage of the proposed model and it achieves the comparable results with supervised methods on SCT.
References
[1]
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In COLING-ACL .
[2]
Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay Attention to the Ending: Strong Neural Baselines for the ROC Story Cloze Task. In ACL .
[3]
Nathanael Chambers and Daniel Jurafsky. 2008. Unsupervised Learning of Narrative Event Chains. In ACL, Vol. 94305. Citeseer, 789--797.
[4]
Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In ACL. Association for Computational Linguistics, 602--610.
[5]
Ming-Wei Chang, Kristina Toutanova, Kenton Lee, and Jacob Devlin. 2019. Language Model Pre-training for Hierarchical Document Representations. arXiv preprint arXiv:1901.09128 (2019).
[6]
Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story Comprehension for Predicting What Happens Next. In EMNLP .
[7]
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task. In Association for Computational Linguistics (ACL) .
[8]
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proc. ACL .
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[10]
Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In NIPS .
[11]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS .
[12]
Edouard Grave, Tomas Mikolov, Armand Joulin, and Piotr Bojanowski. 2017. Bag of Tricks for Efficient Text Classification. In EACL .
[13]
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. 1684--1692.
[14]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd CIKM. ACM, 2333--2338.
[15]
Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. CoRR, Vol. abs/1602.02410 (2016).
[16]
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems. 3294--3302.
[17]
Hongyu Lin, Le Sun, and Xianpei Han. 2017b. Reasoning with Heterogeneous Knowledge for Commonsense Machine Comprehension. In EMNLP . 2022--2033. http://aclweb.org/anthology/D17--1215
[18]
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017a. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 (2017).
[19]
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In EMNLP .
[20]
Todor Mihaylov and Anette Frank. 2017. Story Cloze Ending Selection Baselines and Data Examination. arXiv preprint arXiv:1703.04330 (2017).
[21]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS . 3111--3119.
[22]
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. Proceedings of NAACL HLT (2016).
[23]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268 (2016).
[24]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In EMNLP .
[25]
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018).
[26]
Karl Pichotta and Raymond J Mooney. 2016. Learning Statistical Scripts with LSTM Recurrent Neural Networks. In AAAI . 2800--2806.
[27]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf (2018).
[28]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000
[29]
Questions for Machine Comprehension of Text. In EMNLP .
[30]
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. In EMNLP, Vol. 1. 2.
[31]
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In EMNLP. 1681--1686.
[32]
Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. 2017. The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task. arXiv preprint arXiv:1702.01841 (2017).
[33]
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence .
[34]
Siddarth Srinivasan, Richa Arora, and Mark O. Riedl. 2018. A Simple and Effective Approach to the Story Cloze Test. In NAACL-HLT .
[35]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. [n.d.]. Dropout: A simple way to prevent neural networks from overfitting. JMLR ( [n.,d.]).
[36]
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What Makes Reading Comprehension Questions Easier? EMNLP (2018).
[37]
Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa. 2017. Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability. In ACL .
[38]
Simon Suster and Walter Daelemans. 2018. CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension. CoRR, Vol. abs/1803.09720 (2018).
[39]
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. NewsQA: A Machine Comprehension Dataset. arXiv preprint arXiv:1611.09830 (2016).
[40]
Bingning Wang, Kang Liu, and Jun Zhao. 2017. Conditional Generative Adversarial Networks for Commonsense Machine Comprehension. In IJCAI .
[41]
Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint (2012).
[42]
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In The IEEE International Conference on Computer Vision (ICCV) .
Index Terms
- Unsupervised Story Comprehension with Hierarchical Encoder-Decoder
Recommendations
STED-Net: Self-taught encoder-decoder network for unsupervised feature representation
AbstractCompared with the great successes achieved by supervised learning, e.g. convolutional neural network (CNN), unsupervised feature learning is still a highly-challenging task suffering from no training labels. Because of no training labels for ...
Unsupervised Person Re-Identification via Multi-Label Classification
AbstractThe challenge of unsupervised person re-identification (ReID) lies in learning discriminative features without true labels. Most of previous works predict single-class pseudo labels through clustering. To improve the quality of generated pseudo ...
A novel double-layer sparse representation approach for unsupervised dictionary learning
We propose a DLSR approach for dictionary learning.The DLSR formulation enhances reconstructive and discriminative abilities of dictionary.A DLSR-OMP algorithm is developed to solve the DLSR formulation. This paper presents a novel double-layer sparse ...
Comments
Information & Contributors
Information
Published In
September 2019
273 pages
ISBN:9781450368810
DOI:10.1145/3341981
- General Chairs:
- Yi Fang,
- Yi Zhang,
- Program Chairs:
- James Allan,
- Krisztian Balog,
- Ben Carterette,
- Jiafeng Guo
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 September 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICTIR '19
Sponsor:
ICTIR '19: The 2019 ACM SIGIR International Conference on the Theory of Information Retrieval
October 2 - 5, 2019
CA, Santa Clara, USA
Acceptance Rates
ICTIR '19 Paper Acceptance Rate 20 of 41 submissions, 49%;
Overall Acceptance Rate 235 of 527 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 113Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)1
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Baradaran RGhiasi RAmirkhani H(2022)A Survey on Machine Reading Comprehension SystemsNatural Language Engineering10.1017/S1351324921000395(1-50)Online publication date: 19-Jan-2022
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in