tutorial

Public Access

Knowledge-Augmented Methods for Natural Language Processing

Authors:

Bill Yuchen Lin,

Wenhao YuAuthors Info & Claims

WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

Pages 1228 - 1231

https://doi.org/10.1145/3539597.3572720

Published: 27 February 2023 Publication History

Abstract

Knowledge in NLP has been a rising trend especially after the advent of large-scale pre-trained models. Knowledge is critical to equip statistics-based models with common sense, logic and other external information. In this tutorial, we will introduce recent state-of-the-art works in applying knowledge in language understanding, language generation and commonsense reasoning.

Supplementary Material

MP4 File (wsdm2023_tutorial_language_processing_01.mp4-streaming.mp4)

Knowledge-Augmented Methods for Natural Language Processing

Download
3675.54 MB

References

[1]

Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. In 8th International Conference on Learning Representations, ICLR 2020.

[2]

Aaron Chan, Soumya Sanyal, Bo Long, Jiashu Xu, Tanishq Gupta, and Xiang Ren. 2021. SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning. ArXiv, Vol. abs/2104.08793 (2021).

[3]

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and Play Language Models: A Simple Approach to Controlled Text Generation. In 8th International Conference on Learning Representations.

[4]

Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. 2020. Differentiable Reasoning over a Virtual Knowledge Base. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26--30, 2020.

[5]

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In 7th International Conference on Learning Representations.

[6]

Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive Graph for Multi-Hop Reading Comprehension at Scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 2694--2703. https://doi.org/10.18653/v1/P19--1259

[7]

Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuanjing Huang, Nan Duan, and Ruofei Zhang. 2020. An Enhanced Knowledge Injection Model for Commonsense Generation. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 2014--2025. https://doi.org/10.18653/v1/2020.coling-main.182

[8]

Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 1295--1309. https://doi.org/10.18653/v1/2020.emnlp-main.99

[9]

Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as Experts: Sparse Memory Access with Entity Supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 4937--4951. https://doi.org/10.18653/v1/2020.emnlp-main.400

[10]

Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020a. A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation. Transactions of the Association for Computational Linguistics, Vol. 8 (2020), 93--108. https://doi.org/10.1162/tacl_a_00302

[11]

Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020b. A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation. Transactions of the Association for Computational Linguistics, Vol. 8 (2020), 93--108.

[12]

Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story Ending Generation with Incremental Encoding and Commonsense Knowledge. In The Thirty-Third AAAI Conference on Artificial Intelligence.

[13]

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909 (2020).

[14]

Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. 2022. Empowering Language Models with Knowledge Graph Reasoning for Question Answering. Conference on Empirical Methods in Natural Language Processing (EMNLP) (2022).

[15]

Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, Lianhui Qin, Xiaodan Liang, Haoye Dong, and Eric P. Xing. 2018. Deep Generative Models with Learnable Knowledge Constraints. In Advances in Neural Information Processing Systems.

[16]

Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs. In AAAI.

[17]

Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, and Minlie Huang. 2020a. Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths. In Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and International Joint Conference on Natural Language (AACL).

[18]

Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020b. Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online.

[19]

Mingxuan Ju, Wenhao Yu, Tong Zhao, Chuxu Zhang, and Yanfang Ye. 2022. Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering. Conference on Empirical Methods in Natural Language Processing (EMNLP) (2022).

[20]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online.

[21]

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

[22]

Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kü ttler, Mike Lewis, Wen-tau Yih, Tim Rockt"a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems.

[23]

Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Hong Kong, China.

[24]

Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William Cohen. 2021a. Differentiable Open-Ended Commonsense Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. Online.

[25]

Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021b. RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge. In ACL.

[26]

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.

[27]

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Minneapolis, Minnesota.

[28]

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: Enabling Language Representation with Knowledge Graph. In The Thirty-Fourth AAAI Conference on Artificial Intelligence.

[29]

Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. 2020. Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering. In The Thirty-Fourth AAAI Conference on Artificial Intelligence.

[30]

Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. 2019. Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing. Association for Computational Linguistics, Hong Kong, China, 22--32. https://doi.org/10.18653/v1/D19--6003

[31]

Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge Enhanced Contextual Word Representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China.

[32]

Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03681 (2019).

[33]

Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020. Backpropagation-based Decoding for Unsupervised Counterfactual and Abductive Reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 794--805.

[34]

Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

[35]

Mrigank Raman, Siddhant Agarwal, Peifeng Wang, Aaron Chan, Hansen Wang, Sungchul Kim, Ryan A. Rossi, Handong Zhao, Nedim Lipka, and Xiang Ren. 2021. Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation. ArXiv, Vol. abs/2010.12872 (2021).

[36]

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.

[37]

Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019).

[38]

Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-On What Language Model Pre-training Captures. Transactions of the Association for Computational Linguistics (2020), 743--758.

[39]

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Minneapolis, Minnesota.

[40]

Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval Enhanced Model for Commonsense Generation. In Annual Meeting of Association for Computational Linguistics.

[41]

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019. KEPLER: A unified model for knowledge embedding and pre-trained language representation. arXiv preprint arXiv:1911.06136 (2019).

[42]

Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model. In 8th International Conference on Learning Representation.

[43]

Jun Yan, Mrigank Raman, Tianyu Zhang, Ryan A. Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. 2020. Learning Contextualized Knowledge Structures for Commonsense Reasoning. ArXiv, Vol. abs/2010.12873 (2020).

[44]

Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2022b. Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering. Annual Meeting of the Association for Computational Linguistics (ACL) (2022).

[45]

Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2022d. Jaket: Joint pre-training of knowledge graph and language understanding. AAAI Conference on Artificial Intelligence (AAAI) (2022).

[46]

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022a. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063 (2022).

[47]

Wenhao Yu, Mengxia Yu, Tong Zhao, and Meng Jiang. 2020a. Identifying referential intention with heterogeneous contexts. In Proceedings of The Web Conference 2020. 962--972.

Digital Library

[48]

Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2020b. A survey of knowledge-enhanced text generation. ACM Computing Survey (CSUR) (2020).

[49]

Wenhao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, and Meng Jiang. 2022c. Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts. In Annual Meeting of the Association for Computational Linguistics (ACL).

[50]

Hongming Zhang, Xin Liu, Haojie Pan, Hao Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2021. ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities. ArXiv, Vol. abs/2104.02137 (2021).

[51]

Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020a. Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 2031--2043. https://doi.org/10.18653/v1/2020.acl-main.184

[52]

Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020b. Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 2031--2043. https://doi.org/10.18653/v1/2020.acl-main.184

[53]

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 1441--1451. https://doi.org/10.18653/v1/P19--1139

[54]

Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense Knowledge Aware Conversation Generation with Graph Attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence.

Cited By

Lin XSu THuang ZXue SLiu HChen EChua TNgo CKa-Wei Lee RKumar RLauw H(2024)A Knowledge-Injected Curriculum Pretraining Framework for Question AnsweringProceedings of the ACM Web Conference 202410.1145/3589334.3645406(1986-1997)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645406
Jian ZLi JWu QYao J(2024)Retrieval Contrastive Learning for Aspect-Level Sentiment ClassificationInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10353961:1Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.ipm.2023.103539
Panchendrarajan RZubiaga A(2024)Synergizing machine learning & symbolic methodsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124097251:COnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124097
Show More Cited By

Index Terms

Knowledge-Augmented Methods for Natural Language Processing
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
    2. Natural language processing

Recommendations

Natural Language Programming Based on Knowledge
AICI '10: Proceedings of the 2010 International Conference on Artificial Intelligence and Computational Intelligence - Volume 02

A new programming language—Quasi-Natural Language and an implementation of this language—Kaimeng language processing platform is introduced in this paper. This language focuses on knowledge representation. Knowledge appears mainly in two forms in this ...
Overview of knowledge reasoning for knowledge graph
Abstract
Knowledge graphs are large-scale semantic networks that considerably impact knowledge representation. Mining hidden knowledge from existing data, including triplet knowledge reasoning, is a primary objective of knowledge graphs. With the ...
Using natural language to represent knowledge in an intelligent tutoring system
K-CAP '09: Proceedings of the fifth international conference on Knowledge capture

Developing intelligent tutoring systems (ITS) for a target domain includes authoring tasks requiring long time and efforts. Letting authors use natural language to represent various kinds of knowledge is a way that any authors can add and update ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

February 2023

1345 pages

ISBN:9781450394079

DOI:10.1145/3539597

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Hady Lauw
Singapore Management University
,
Program Chairs:
Luo Si
Salesforce
,
Evimaria Terzi
Boston University
,
Panayiotis Tsaparas
University of Ioannina

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Tutorial

Funding Sources

DARPA MCS program
ONR
NSF

Conference

WSDM '23

Sponsor:

WSDM '23: The Sixteenth ACM International Conference on Web Search and Data Mining

February 27 - March 3, 2023

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
964
Total Downloads

Downloads (Last 12 months)314
Downloads (Last 6 weeks)32

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lin XSu THuang ZXue SLiu HChen EChua TNgo CKa-Wei Lee RKumar RLauw H(2024)A Knowledge-Injected Curriculum Pretraining Framework for Question AnsweringProceedings of the ACM Web Conference 202410.1145/3589334.3645406(1986-1997)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645406
Jian ZLi JWu QYao J(2024)Retrieval Contrastive Learning for Aspect-Level Sentiment ClassificationInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10353961:1Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.ipm.2023.103539
Panchendrarajan RZubiaga A(2024)Synergizing machine learning & symbolic methodsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124097251:COnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124097
Zhao YHuang YCheng B(2024)KiProL: A Knowledge-Injected Prompt Learning Framework for Language GenerationAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2266-2_6(70-82)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1007/978-981-97-2266-2_6
Yu WTong LShi WPeng NJiang MSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)The Second Workshop on Knowledge-Augmented Methods for Natural Language ProcessingProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599233(5899-5900)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599233
Khan WDaud AKhan KMuhammad SHaq R(2023)Exploring the frontiers of deep learning and natural language processing: A comprehensive overview of key challenges and emerging trendsNatural Language Processing Journal10.1016/j.nlp.2023.1000264(100026)Online publication date: Sep-2023
https://doi.org/10.1016/j.nlp.2023.100026

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten