research-article

Brute force and indexed approaches to pairwise document similarity comparisons with MapReduce

Author:

Jimmy LinAuthors Info & Claims

SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

Pages 155 - 162

https://doi.org/10.1145/1571941.1571970

Published: 19 July 2009 Publication History

Abstract

This paper explores the problem of computing pairwise similarity on document collections, focusing on the application of "more like this" queries in the life sciences domain. Three MapReduce algorithms are introduced: one based on brute force, a second where the problem is treated as large-scale ad hoc retrieval, and a third based on the Cartesian product of postings lists. Each algorithm supports one or more approximations that trade effectiveness for efficiency, the characteristics of which are studied experimentally. Results show that the brute force algorithm is the most efficient of the three when exact similarity is desired. However, the other two algorithms support approximations that yield large efficiency gains without significant loss of effectiveness.

References

[1]

A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. CACM, 51(1):117--122, 2008.

Digital Library

[2]

V. Anh, O. de Kretser, and A. Moffat. Vector-space ranking with effective early termination. In SIGIR, 35--42, 2001.

Digital Library

[3]

R. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. In WWW, 131--140, 2007.

Digital Library

[4]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 137--150, 2004.

Digital Library

[5]

T. Elsayed, J. Lin, and D. Oard. Pairwise document similarity in large collections with MapReduce. In ACL, Companion Volume, 265--268, 2008.

Digital Library

[6]

S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In SOSP, 29--43, 2003.

Digital Library

[7]

W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. TREC 2005 Genomics Track overview. In TREC, 2005.

[8]

N. Lester, A. Moffat, W. Webber, and J. Zobel. Space-limited ranked query evaluation using adaptive pruning. In WISE, 470--477, 2005.

Digital Library

[9]

J. Lin and M. Smucker. How do users find things with ? Towards automatic utility evaluation with user simulations. In SIGIR, 19--26, 2008.

Digital Library

[10]

J. Lin and W. J. Wilbur. related articles: A probabilistic topic-based model for content similarity. BMC Bioinformatics, 8:423, 2007.

[11]

A. Moffat and J. Zobel. Self-indexing inverted files for fast text retrieval. ACM TOIS, 14(4):349--379, 1996.

Digital Library

[12]

M. Persin, J. Zobel, and R. Sacks-Davis. Filtered document retrieval with frequency-sorted indexes. JASIS, 47(10):749--764, 1996.

[13]

T. Strohman and W. Croft. Efficient document retrieval in main memory. In SIGIR, 175--182, 2007.

Digital Library

[14]

I. Witten, A. Moffat, and T. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, San Francisco, 1999.

Digital Library

[15]

C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In SIGIR, 334--342, 2001.

Digital Library

Cited By

Dhanani JMehta RRana D(2021)Legal document recommendation system: A cluster based pairwise similarity computationJournal of Intelligent & Fuzzy Systems10.3233/JIFS-189871(1-13)Online publication date: 1-Apr-2021
https://doi.org/10.3233/JIFS-189871
Patil KBhavsar RPawar B(2021)Word Suggestions for non-word Text Errors using Similarity Measure2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS)10.1109/ICACCS51430.2021.9441858(892-897)Online publication date: 19-Mar-2021
https://doi.org/10.1109/ICACCS51430.2021.9441858
Ma YZhang RCui ZLin C(2020)Projection Based Large Scale High-Dimensional Data Similarity Join Using MapReduce FrameworkIEEE Access10.1109/ACCESS.2020.30070288(121665-121677)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3007028
Show More Cited By

Index Terms

Brute force and indexed approaches to pairwise document similarity comparisons with MapReduce
1. Information systems
  1. Information retrieval

Recommendations

Effective measures for inter-document similarity
CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge Management

While supervised learning-to-rank algorithms have largely supplanted unsupervised query-document similarity measures for search, the exploration of query-document measures by many researchers over many years produced insights that might be exploited in ...
Investigating Measures for Pairwise Document Similarity
Efficient top-k similarity document search utilizing distributed file systems and cosine similarity

Document similarity has important real life applications such as finding duplicate web sites and identifying plagiarism. While the basic techniques such as k-similarity algorithms have been long known, overwhelming amount of data, being collected such ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

July 2009

896 pages

ISBN:9781605584836

DOI:10.1145/1571941

General Chairs:
James Allan
University of Massachusetts Amherst, USA
,
Javed Aslam
Northeastern University, USA
,
Program Chairs:
Mark Sanderson
University of Sheffield, UK
,
ChengXiang Zhai
University of Illinois at Urbana-Champaign, USA
,
Justin Zobel
University of Melbourne, Australia

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '09

Sponsor:

SIGIR '09: The 32nd International ACM SIGIR conference on research and development in Information Retrieval

July 19 - 23, 2009

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
1,354
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dhanani JMehta RRana D(2021)Legal document recommendation system: A cluster based pairwise similarity computationJournal of Intelligent & Fuzzy Systems10.3233/JIFS-189871(1-13)Online publication date: 1-Apr-2021
https://doi.org/10.3233/JIFS-189871
Patil KBhavsar RPawar B(2021)Word Suggestions for non-word Text Errors using Similarity Measure2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS)10.1109/ICACCS51430.2021.9441858(892-897)Online publication date: 19-Mar-2021
https://doi.org/10.1109/ICACCS51430.2021.9441858
Ma YZhang RCui ZLin C(2020)Projection Based Large Scale High-Dimensional Data Similarity Join Using MapReduce FrameworkIEEE Access10.1109/ACCESS.2020.30070288(121665-121677)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3007028
Yu ZWu LYang J(2020)Quantitative Evaluation of Promotion of Traditional National Sports Events Based on Probity Multiple Regression AnalysisArtificial Intelligence and Security10.1007/978-981-15-8083-3_21(232-243)Online publication date: 13-Sep-2020
https://doi.org/10.1007/978-981-15-8083-3_21
Zheng L(2019)Human-autonomous devices for English network teaching system based on artificial intelligence and WBIETS systemJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-019-01427-7Online publication date: 28-Aug-2019
https://doi.org/10.1007/s12652-019-01427-7
Aydar MAyvaz S(2019)An improved method of locality-sensitive hashing for scalable instance matchingKnowledge and Information Systems10.1007/s10115-018-1199-558:2(275-294)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s10115-018-1199-5
Ma YZhang RJia SZhang YMeng X(2019)An efficient similarity join approach on large‐scale high‐dimensional data using random projectionConcurrency and Computation: Practice and Experience10.1002/cpe.530331:20Online publication date: 7-May-2019
https://doi.org/10.1002/cpe.5303
Feng YTang JLiu MWang CXie J(2018)Fast Document Cosine Similarity Self-Join on GPUs2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI.2018.00040(205-212)Online publication date: Nov-2018
https://doi.org/10.1109/ICTAI.2018.00040
Lv PYang PDong YGu L(2018)Document Nearest Neighbors Query Based on Pairwise Similarity with MapReduceAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-05051-1_3(34-45)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-3-030-05051-1_3
Tang XAlabduljalil MJin XYang T(2017)Partitioned Similarity Search with Cache-Conscious Data TraversalACM Transactions on Knowledge Discovery from Data10.1145/301406011:3(1-38)Online publication date: 14-Apr-2017
https://dl.acm.org/doi/10.1145/3014060
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten