skip to main content
10.1145/2838931.2838939acmotherconferencesArticle/Chapter ViewAbstractPublication PagesadcsConference Proceedingsconference-collections
short-paper

Data Fusion for Japanese Term and Character N-gram Search

Published: 08 December 2015 Publication History

Abstract

Term segmentation plays a vital role in building effective information retrieval systems. In particular, languages such as Japanese and Chinese require a morphological analyzer or a word segmenter to identify potential terms. The alternative approach to indexing a segmented collection is n-gram search, where every n-length sequence of symbols is indexed. Both approaches have strengths and weaknesses when applied to non-English collections. In this study, we explore data fusion techniques to answer the following question: if there are multiple ranked lists of documents from both word and n-gram indexes, can we improve overall effectiveness by combining them? We consider three empirical methods for combining search results using eight different search indexes and twenty-one different search models with and without automatic query expansion. Our approach is language independent; however, we focus on Japanese test collections -- NTCIR IR4QA -- as our testbed for the current experiments. Our experimental results demonstrate that the combination of the two different segmentation approaches has the potential to significantly outperform the best word-segmented search methods.

References

[1]
ChaSen. https://osdn.jp/projects/chasen-legacy/, 2012.
[2]
JUMAN. http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN, 2012.
[3]
KAKASI. http://kakasi.namazu.org/index.html.en, 2013.
[4]
KyTea. http://www.phontron.com/kytea/, 2013.
[5]
MeCab. http://taku910.github.io/mecab/, 2013.
[6]
Terrier IR Platform. http://terrier.org/, 2014.
[7]
S. Abdou and J. Savoy. Monolingual experiments with Far-East languages in NTCIR-6. In Proc. of the 6th NTCIR Workshop Meeting, pages 52--59, 2007.
[8]
K. Abdulahhad, J. Chevallet, and C. Berrut. Matching fusion with conceptual indexing. In Proc. of RISE2012, 2012.
[9]
G. Amati. Probability Models for Information Retrieval based on Divergence from Randomness. PhD thesis, University of Glasgow, Glasgow, 2003.
[10]
E. Fox and J. Shaw. Combination of multiple searches. In Proc. of TREC-2, pages 243--252, 1994.
[11]
J. Mayfield and P. McNamee. Combining methods for the TREC 2003 robust track. In Working Notes of TREC 2003, 2003.
[12]
P. McNamee. Experiments in the retrieval of unsegmented Japanese text at the NTCIR-2 workshop. In Proc. of the 2nd NTCIR Workshop Meeting, 2001.
[13]
P. McNamee, C. K. Nicholas, and J. Mayfield. Addressing morphological variation in alphabetic languages. In Proc. of SIGIR '09, pages 75--82, 2009.
[14]
D. Metzler and W. B. Croft. A markov random field model for term dependencies. In Proc. of SIGIR '05, pages 472--479, 2005.
[15]
C. Monz, J. Kamps, and M. de Rijke. The university of Amsterdam at CLEF 2002. In CLEF Working Notes, 2002.
[16]
Y. Ogawa and T. Matsuda. Overlapping statistical word indexing: A new indexing method for Japanese text. In Proc. of SIGIR '97, pages 226--234, 1997.
[17]
T. Sakai, N. Kando, et al. Overview of the NTCIR-7 ACLIA IR4QA task. In Proc. of the 7th NTCIR Workshop Meeting, pages 77--114, 2008.
[18]
T. Sakai, H. Shima, et al. Overview of NTCIR-8 ACLIA IR4QA. In Proc. of the 8th NTCIR Workshop Meeting, pages 63--93, 2010.
[19]
J. Savoy. Comparative study of monolingual and multilingual search models for use with asian languages. TALIP, 4(2):163--189, 2005.
[20]
T. Shima and T. Mitamura. Bootstrap pattern learning for open-domain CLQA. In Proc. of the 8th NTCIR Workshop Meeting, pages 37--42, 2010.
[21]
S. Tomlinson. Experiments in finding Chinese and Japanese answer documents at NTCIR-7. In Proc. of the 7th NTCIR Workshop Meeting, pages 177--184, 2008.
[22]
M. Wu, D. Hawking, A. Turpin, and F. Scholer. Using anchor text for homepage and topic distillation search tasks. JASIST, 63(6):1235--1255, 2012.
[23]
S. Wu. Data Fusion in Information Retrieval. Springer, 2012.

Cited By

View all
  • (2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018

Index Terms

  1. Data Fusion for Japanese Term and Character N-gram Search

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ADCS '15: Proceedings of the 20th Australasian Document Computing Symposium
    December 2015
    72 pages
    ISBN:9781450340403
    DOI:10.1145/2838931
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 December 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. morphological analysis
    2. n-gram search
    3. term segmentation

    Qualifiers

    • Short-paper
    • Research
    • Refereed limited

    Funding Sources

    • KAKENHI, JSPS
    • Discovery Projects Scheme, Australian Research Council
    • DECRA Research Fellowship, Australian Research Council

    Conference

    ADCS '15
    ADCS '15: The 20th Australasian Document Computing Symposium
    December 8 - 9, 2015
    NSW, Parramatta, Australia

    Acceptance Rates

    ADCS '15 Paper Acceptance Rate 5 of 14 submissions, 36%;
    Overall Acceptance Rate 30 of 57 submissions, 53%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)4
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media