Abstract
The Winograd Schema Challenge (WSC) has attracted much attention recently as common sense is recognized to be not only the key to human-level intelligence but also a bottleneck faced by recent progress. Although neural language models (LMs) have achieved state-of-the-art (SOTA) performance on WSC, they fall short on interpretability and robustness against adversarial attacks. Contrarily, methods with structured representation and explicit reasoning suffer from the difficulty of knowledge acquisition and the rigidness of representation. In this paper, we look back on the current model-free and model-based approaches, pointing out the missing ingredients towards solving the WSC. We report our preliminary exploration of formalizing the WSC problems using a variant of first-order language and our first-hand findings of indispensable capabilities of human-level commonsense reasoning. The issues we encounter suggest that a full spectrum of representation tools and reasoning abilities are called for.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Formulae in this paper are all universally quantified. For brevity, we omit the UNAs.
- 2.
This solution only applies to deterministic actions without ramification; In our cases, we have no trouble with this limitation.
References
Bender, D.: Establishing a human baseline for the Winograd schema challenge. In: MAICS, pp. 39–45 (2015)
Davis, E., Marcus, G.: Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58(9), 92–103 (2015)
Davis, E., Morgenstern, L., Ortiz, C.: The Winograd schema challenge (2016). https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html
de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24
Golovin, D., Claßen, J., Schwering, C.: Reasoning about conditional beliefs for the Winograd schema challenge. In: COMMONSENSE 2017 (2017)
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S.R., Smith, N.A.: Annotation artifacts in natural language inference data. In: NAACL-HLT, pp. 107–112 (2018)
Kocijan, V., Cretu, A., Camburu, O., Yordanov, Y., Lukasiewicz, T.: A surprisingly robust trick for the Winograd schema challenge. In: ACL, pp. 4837–4842 (2019)
Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017)
Lenat, D.B.: CYC: a large-scale investment in knowledge infrastructure. Commun. ACM 38(11), 32–38 (1995)
Levesque, H.J., Davis, E., Morgenstern, L.: The Winograd schema challenge. In: KR 2012(2012)
Liu, Q., Jiang, H., Ling, Z.H., Zhu, X., Wei, S., Hu, Y.: Combing context and commonsense knowledge through neural networks for solving Winograd schema problems. In: 2017 AAAI Spring Symposium Series (2017)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692 (2019)
McCarthy, J.: Generality in artificial intelligence. Commun. ACM 30(12), 1030–1035 (1987)
McCarthy, J.: Notes on formalizing context. In: Proceedings of the 13th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI 1993, pp. 555–560. Morgan Kaufmann Publishers Inc., San Francisco (1993)
Prakash, A., Sharma, A., Mitra, A., Baral, C.: Combining knowledge hunting and neural language models to solve the Winograd schema challenge. In: ACL, pp. 6110–6119 (2019)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Reiter, R.: Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems. MIT Press, Cambridge (2001)
Richard-Bollans, A., Álvarez, L.G., Cohn, A.G.: The role of pragmatics in solving the Winograd schema challenge. In: COMMONSENSE 2017 (2017)
Russell, S.J., Norvig, P.: Artificial Intelligence - A Modern Approach. Third International Edition. Pearson Education, London (2010)
Sakaguchi, K., Bras, R.L., Bhagavatula, C., Choi, Y.: Winogrande: an adversarial Winograd schema challenge at scale. In: AAAI 2020, pp. 8732–8740 (2020)
Sap, M., Rashkin, H., Chen, D., LeBras, R., Choi, Y.: SocialIQA: commonsense reasoning about social interactions. In: EMNLP (2019)
Schüller, P.: Tackling Winograd schemas by formalizing relevance theory in knowledge graphs. In: KR 2014 (2014)
Sharma, A., Vo, N.H., Aditya, S., Baral, C.: Towards addressing the Winograd schema challenge - building and using a semantic parser and a knowledge hunting module. IJCAI 2015, 1319–1325 (2015)
Trichelair, P., Emami, A., Trischler, A., Suleman, K., Cheung, J.C.K.: How reasonable are common-sense reasoning tasks: a case-study on the Winograd schema challenge and SWAG. In: EMNLP-IJCNLP, November 2019
Trinh, T.H., Le, Q.V.: A simple method for commonsense reasoning. arXiv (2018)
Tsuchiya, M.: Performance impact caused by hidden bias of training data for recognizing textual entailment. In: LREC 2018 (2018)
Wang, A., et al.: SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In: NeurIPS 2019, pp. 3261–3275 (2019)
Zhang, H., Zhao, X., Song, Y.: WinoWhy: a deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge. In: ACL 2020, pp. 5736–5745 (2020)
Acknowledgement
We thank Prof. Yongmei Liu for her guidance and insightful advice, and we thank Yu Dong for his effort. We acknowledge support from the National Natural Science Foundation of China (No. 61572535) and the Guangdong Basic and Applied Basic Research Foundation (2020A1515010642).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
He, W., Xiao, Z. (2021). Towards Solving the Winograd Schema Challenge: Model-Free, Model-Based and a Spectrum in Between. In: Qiu, H., Zhang, C., Fei, Z., Qiu, M., Kung, SY. (eds) Knowledge Science, Engineering and Management . KSEM 2021. Lecture Notes in Computer Science(), vol 12816. Springer, Cham. https://doi.org/10.1007/978-3-030-82147-0_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-82147-0_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82146-3
Online ISBN: 978-3-030-82147-0
eBook Packages: Computer ScienceComputer Science (R0)