short-paper

Adversarial Domain Adaptation for Cross-lingual Information Retrieval with Multilingual BERT

Authors:

Qing HeAuthors Info & Claims

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Pages 3498 - 3502

https://doi.org/10.1145/3459637.3482050

Published: 30 October 2021 Publication History

Abstract

Transformer-based language models (e.g. BERT, RoBERT, GPT, etc) have shown remarkable performance in many natural language processing tasks and their multilingual variants make it easier to handle cross-lingual tasks without using machine translation system. In this paper, we apply multilingual BERT in cross-lingual information retrieval (CLIR) task with triplet loss to learn the relevance between queries and documents written in different languages. Moreover, we align the token embeddings from different languages via adversarial networks to help the language model to learn cross-lingual sentence representation. We achieve the state-of-the-art result on the newly published CLIR dataset: CLIRMatrix. Furthermore, we show that the adversarial multilingual BERT can also get the competitive result in the zero-shot setting in some specific languages when we are lack of CLIR training data in a specific language.

Supplementary Material

MP4 File (CIKM21-rgsp2243.mp4)

Presentation video

Download
13.39 MB

References

[1]

Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087 (2017).

[2]

Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 985--988.

Digital Library

[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[4]

Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao. 2020. Adver-sarial and domain-aware bert for cross-domain sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4019--4028.

[5]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096--2030.

Digital Library

[6]

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014).

[7]

Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM international on conference on information and knowledge management. 55--64.

Digital Library

[8]

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. 2333--2338.

Digital Library

[9]

Zhuolin Jiang, Amro El-Jaroudi, William Hartmann, Damianos Karakos, and Lingjun Zhao. 2020. Cross-lingual Information Retrieval with BERT. arXiv preprint arXiv:2004.13005 (2020).

[10]

Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and NER. arXiv preprint arXiv:1909.00153 (2019).

[11]

Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 39--48.

Digital Library

[12]

Lukas Lange, Anastasiia Iurshina, Heike Adel, and Jannik Strötgen. 2020. Ad-versarial alignment of multilingual models for extracting temporal expressions from text. arXiv preprint arXiv:2005.09392 (2020).

[13]

Robert Litschko, Goran Glava?, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1253--1256.

Digital Library

[14]

Chi-Liang Liu, Tsung-Yuan Hsu, Yung-Sung Chuang, and Hung-Yi Lee. 2020. A study of cross-lingual ability and language-specific information in multilingual BERT. arXiv preprint arXiv:2004.09205 (2020).

[15]

Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168 (2013).

[16]

Jian-Yun Nie. 2010. Cross-language information retrieval. Synthesis Lectures on Human Language Technologies 3, 1 (2010), 1--125.

Digital Library

[17]

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30.

Digital Library

[18]

Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multi-lingual bert? arXiv preprint arXiv:1906.01502 (2019).

[19]

Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the Behaviors of BERT in Ranking. arXiv preprint arXiv:1904.07531 (2019).

[20]

Shuo Sun and Kevin Duh. 2020. CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 4160--4170.

[21]

Ivan Vulić and Marie-Francine Moens. 2015. Monolingual and cross-lingual in-formation retrieval models based on (bilingual) word embeddings. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 363--372.

Digital Library

[22]

Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. arXiv preprint arXiv:1904.09077 (2019).

[23]

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval. 55--64.

Digital Library

[24]

Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Applying BERT to document retrieval with birch. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. 19--24.

[25]

Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1959--1970.

[26]

Yuyu Zhang, Ping Nie, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. 2020. DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding. arXiv preprint arXiv:2002.12591 (2020).

Cited By

Guo PRen YHu YCao YLi YHuang HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Steering Large Language Models for Cross-lingual Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657819(585-596)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657819
Guo PHu YCao YRen YLi YHuang HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Query in Your Tongue: Reinforce Large Language Models with Retrievers for Cross-lingual Search Generative ExperienceProceedings of the ACM Web Conference 202410.1145/3589334.3645701(1529-1538)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645701
Guoming J(2024)Computer Network Information Retrieval Algorithm Integrating Data Structure Fusion Optimization2024 International Conference on Inventive Computation Technologies (ICICT)10.1109/ICICT60155.2024.10544673(1632-1638)Online publication date: 24-Apr-2024
https://doi.org/10.1109/ICICT60155.2024.10544673
Show More Cited By

Index Terms

Adversarial Domain Adaptation for Cross-lingual Information Retrieval with Multilingual BERT
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Semantic morphological variant selection and translation disambiguation for cross-lingual information retrieval
Abstract
Cross-Lingual Information Retrieval (CLIR) enables a user to query in a language which is different from the target documents language. CLIR incorporates a translation technique based on either a manual dictionary or a probabilistic dictionary ...
Refined stop-words and morphological variants solutions applied to Hindi-English cross-lingual information retrieval
Soft Computing and Intelligent Systems: Techniques and Applications

Cross-Lingual Information Retrieval (CLIR) enables a user to query in a language which is different than the target documents language. CLIR incorporates a machine translation technique, like, Statistical Machine Translation (SMT) and Neural Machine ...
Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings
SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval

We propose a new unified framework for monolingual (MoIR) and cross-lingual information retrieval (CLIR) which relies on the induction of dense real-valued word vectors known as word embeddings (WE) from comparable data. To this end, we make several ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

October 2021

4966 pages

ISBN:9781450384469

DOI:10.1145/3459637

General Chairs:
Gianluca Demartini
The University of Queensland, Australia
,
Guido Zuccon
The University of Queensland, Australia
,
Program Chairs:
J. Shane Culpepper
RMIT University, Australia
,
Zi Huang
The University of Queensland, Australia
,
Hanghang Tong
University of Illinois at Urbana-Champaign, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

National Natural Science Foundation of China

Conference

CIKM '21

Sponsor:

CIKM '21: The 30th ACM International Conference on Information and Knowledge Management

November 1 - 5, 2021

Queensland, Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
421
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo PRen YHu YCao YLi YHuang HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Steering Large Language Models for Cross-lingual Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657819(585-596)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657819
Guo PHu YCao YRen YLi YHuang HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Query in Your Tongue: Reinforce Large Language Models with Retrievers for Cross-lingual Search Generative ExperienceProceedings of the ACM Web Conference 202410.1145/3589334.3645701(1529-1538)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645701
Guoming J(2024)Computer Network Information Retrieval Algorithm Integrating Data Structure Fusion Optimization2024 International Conference on Inventive Computation Technologies (ICICT)10.1109/ICICT60155.2024.10544673(1632-1638)Online publication date: 24-Apr-2024
https://doi.org/10.1109/ICICT60155.2024.10544673
Kim JChung SChi S(2024)Cross-Lingual Information Retrieval from Multilingual Construction Documents Using Pretrained Language ModelsJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-14273150:6Online publication date: Jun-2024
https://doi.org/10.1061/JCEMD4.COENG-14273
Wang DFeng XLiu ZWang C(2024)2M-NER: contrastive learning for multilingual and multimodal NER with language and modal fusionApplied Intelligence10.1007/s10489-024-05490-254:8(6252-6268)Online publication date: 9-May-2024
https://dl.acm.org/doi/10.1007/s10489-024-05490-2
V JS A(2023)Cross-lingual Sentiment Analysis of Tamil Language Using a Multi-stage Deep Learning ArchitectureACM Transactions on Asian and Low-Resource Language Information Processing10.1145/363139122:12(1-28)Online publication date: 19-Dec-2023
https://dl.acm.org/doi/10.1145/3631391
Wang HWang YWu FXue HGao JSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Macular: A Multi-Task Adversarial Framework for Cross-Lingual Natural Language UnderstandingProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599864(5061-5070)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599864
Zhu JWu JLuo XLiu J(2023)Semantic matching based legal information retrieval system for COVID-19 pandemicArtificial Intelligence and Law10.1007/s10506-023-09354-x32:2(397-426)Online publication date: 14-Mar-2023
https://dl.acm.org/doi/10.1007/s10506-023-09354-x
Chen DZhang XZhang S(2023)Narrowing the language gap: domain adaptation guided cross-lingual passage re-rankingNeural Computing and Applications10.1007/s00521-023-08803-735:28(20735-20748)Online publication date: 25-Jul-2023
https://dl.acm.org/doi/10.1007/s00521-023-08803-7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten