Alignment Rationale for Query-Document Relevance
Pages 2489 - 2494
Abstract
Deep neural networks are widely used for text pair classification tasks such as as adhoc information retrieval. These deep neural networks are not inherently interpretable and require additional efforts to get rationale behind their decisions. Existing explanation models are not yet capable of inducing alignments between the query terms and the document terms -- which part of the document rationales are responsible for which part of the query? In this paper, we study how the input perturbations can be used to infer or evaluate alignments between the query and document spans, which best explain the black-box ranker's relevance prediction. We use different perturbation strategies and accordingly propose a set of metrics to evaluate the faithfulness of alignment rationales to the model. Our experiments show that the defined metrics based on substitution-based perturbation are more successful in preferring higher-quality alignments, compared to the deletion-based metrics.
References
[1]
Samuel Carton, Anirudh Rathore, and Chenhao Tan. 2020. Evaluating and Characterizing Human Rationales. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 9294--9307. https://doi.org/10.18653/v1/2020.emnlp-main.747
[2]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. [n.d.]. OVERVIEW OF THE TREC 2019 DEEP LEARNING TRACK. ( [n.,d.]).
[3]
Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 985--988.
[4]
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A Benchmark to Evaluate Rationalized NLP Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4443--4458. https://doi.org/10.18653/v1/2020.acl-main.408
[5]
Zeon Trevor Fernando, Jaspreet Singh, and Avishek Anand. 2019. A Study on the Interpretability of Neural Retrieval Models Using DeepSHAP. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR'19). 1005--1008.
[6]
Peter Hase, Harry Xie, and Mohit Bansal. 2021. The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations. Advances in Neural Information Processing Systems, Vol. 34 (2021).
[7]
Marti A Hearst. 1995. Tilebars: Visualization of term distribution information in full text information access. In Proceedings of the SIGCHI conference on Human factors in computing systems. 59--66.
[8]
Orland Hoeber and Xue Dong Yang. 2006. The Visual Exploration ofWeb Search Results Using HotMap. Tenth International Conference on Information Visualisation (IV'06) (2006), 157--165.
[9]
Zhongtao Jiang, Yuanzhe Zhang, Zhao Yang, Jun Zhao, and Kang Liu. 2021. Alignment Rationale for Natural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 5372--5387.
[10]
Youngwoo Kim, Myungha Jang, and James Allan. 2020. Explaining text matching on neural natural language inference. ACM Transactions on Information Systems (TOIS), Vol. 38, 4 (2020), 1--23.
[11]
Youngwoo Kim, Razieh Rahimi, Hamed Bonab, and James Allan. 2021. Query-Driven Segment Selection for Ranking Long Documents .Association for Computing Machinery, New York, NY, USA, 3147--3151. https://doi.org/10.1145/3459637.3482101
[12]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS .
[13]
Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the Behaviors of BERT in Ranking. arXiv preprint arXiv:1904.07531 (2019).
[14]
Razieh Rahimi, Youngwoo Kim, Hamed Zamani, and James Allan. 2021. Explaining Documents' Relevance to Search Queries. arXiv preprint arXiv:2111.01314 (2021).
[15]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). Association for Computing Machinery, New York, NY, USA, 1135--1144. https://doi.org/10.1145/2939672.2939778
[16]
Procheta Sen, Debasis Ganguly, Manisha Verma, and Gareth J.F. Jones. 2020. The Curious Case of IR Explainability: Explaining Document Scores within and across Ranking Models . 2069--2072.
[17]
Jaspreet Singh and Avishek Anand. 2018. Posthoc Interpretability of Learning to Rank Models using Secondary Training Data. In Workshop on ExplainAble Recommendation and Search (EARS 2018) at SIGIR 2018 .
[18]
Jaspreet Singh and Avishek Anand. 2019. Exs: Explainable search using local model agnostic interpretability. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining . 770--773.
[19]
Jaspreet Singh, Megha Khosla, Wang Zhenye, and Avishek Anand. 2021. Extracting per Query Valid Explanations for Blackbox Learning-to-Rank Models. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. Association for Computing Machinery, New York, NY, USA, 203--210. https://doi.org/10.1145/3471158.3472241
[20]
Manisha Verma and Debasis Ganguly. 2019. LIRME: Locally Interpretable Ranking Model Explanation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR'19). 1281--1284.
[21]
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained Transformers for Text Ranking: BERT and Beyond .Association for Computing Machinery, New York, NY, USA, 1154--1156. https://doi.org/10.1145/3437963.3441667
[22]
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. An Analysis of BERT in Document Ranking .Association for Computing Machinery, New York, NY, USA, 1941--1944. https://doi.org/10.1145/3397271.3401325
[23]
Honglei Zhuang, Xuanhui Wang, Michael Bendersky, Alexander Grushetsky, Yonghui Wu, Petr Mitrichev, Ethan Sterling, Nathan Bell, Walker Ravina, and Hai Qian. 2021. Interpretable Ranking with Generalized Additive Models. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining . 499--507.
Index Terms
- Alignment Rationale for Query-Document Relevance
Recommendations
A document query search using an extended centrality with the word2vec
ICEC '16: Proceedings of the 18th Annual International Conference on Electronic Commerce: e-Commerce in Smart connected WorldWhile everyday document search is done by keyword-based queries to search engines, we have situations that need deep search of documents such as scrutinies of patents, legal documents, and so on. In such cases, using document queries, instead of keyword-...
On perfect document rankings for expert search
SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrievalExpert search systems often employ a document search component to identify on-topic documents, which are then used to identify people likely to have relevant expertise. This work investigates the impact of the retrieval effectiveness of the underlying ...
Information Retrieval by Document Re-ranking using Term Association Graph
ICONIAAC '14: Proceedings of the 2014 International Conference on Interdisciplinary Advances in Applied ComputingMost of the Information Retrieval techniques are based on representing the documents using the traditional vector space model i.e. bag-of-words model. In this paper, associations among words in the documents are assessed and it is expressed in term ...
Comments
Information & Contributors
Information
Published In
July 2022
3569 pages
ISBN:9781450387323
DOI:10.1145/3477495
- General Chairs:
- Enrique Amigo,
- Pablo Castells,
- Julio Gonzalo,
- Program Chairs:
- Ben Carterette,
- J. Shane Culpepper,
- Gabriella Kazai
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 July 2022
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
SIGIR '22
Sponsor:
SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 11 - 15, 2022
Madrid, Spain
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 344Total Downloads
- Downloads (Last 12 months)110
- Downloads (Last 6 weeks)27
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in