skip to main content
10.1145/1390334.1390518acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
poster

Don't have a stemmer?: be un+concern+ed

Published: 20 July 2008 Publication History

Abstract

The choice of indexing terms used to represent documents crucially determines how e ective subsequent retrieval will be. IR systems commonly use rule-based stemmers to normalize surface word forms to combat the problem of not finding documents that contain words related to query terms by inflectional or derivational morphology. But such stemmers are not available in all languages. In this paper we explore the effectiveness of unsupervised morphological segmentation as an alternative to stemming using test sets in thirteen European languages. We find that unsupervised segmentation is significantly better than unnormalized words, in several cases by more than 20%. However, rule-based stemming, if available, is better in low complexity languages. We also compare these methods to the use of character n-grams, finding that on average n-grams yield the best performance.

References

[1]
M. Creutz and K. Lagus. Unsupervised discovery of morphemes. In ACL-02 Workshop on Morphological and Phonological Learning, pages 21--30, 2002.
[2]
G. M. Di Nunzio, N. Ferro, T. Mandl, and C. Peters. CLEF 2007: Ad hoc track overview. In CLEF 2007 Working Notes, 2007.
[3]
D. Harman. How effective is stemming? JASIS, 42(1):7--15, 1991.
[4]
D. A. Hull. Stemming algorithms: A case study for detailed evaluation. JASIS, 47(1):70--84, 1996.
[5]
R. Krovetz. Viewing morphology as an inference process. In ACM SIGIR 1993, pages 191--202, 1993.
[6]
M. Kurimo, M. Creutz, and V. Turunen. Overview of Morpho Challenge in CLEF 2007. In Working Notes of the CLEF 2007 Workshop, 2007.
[7]
P. McNamee and J. Mayfield. Character n-gram tokenization for european language text retrieval. Information Retrieval, 7(1-2):73--97, 2004.
[8]
M. F. Porter. An algorithm for suffix stripping. Program, 14:130--137, 1980.

Cited By

View all
  • (2015)Language Independent Tokenization vs. Stemming in Automated Detection of Health Websites’ HONcode Conformity: An EvaluationProcedia Computer Science10.1016/j.procs.2015.08.48464(224-231)Online publication date: 2015
  • (2012)Fast and effective soft linksSoftware: Practice and Experience10.1002/spe.212243:5(577-593)Online publication date: 11-Apr-2012
  • (2010)EMMAProceedings of the 23rd International Conference on Computational Linguistics10.5555/1873781.1873897(1029-1037)Online publication date: 23-Aug-2010
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
July 2008
934 pages
ISBN:9781605581644
DOI:10.1145/1390334
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. character n-grams
  2. stemming
  3. tokenization
  4. unsupervised morphological segmentation

Qualifiers

  • Poster

Conference

SIGIR '08
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)1
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2015)Language Independent Tokenization vs. Stemming in Automated Detection of Health Websites’ HONcode Conformity: An EvaluationProcedia Computer Science10.1016/j.procs.2015.08.48464(224-231)Online publication date: 2015
  • (2012)Fast and effective soft linksSoftware: Practice and Experience10.1002/spe.212243:5(577-593)Online publication date: 11-Apr-2012
  • (2010)EMMAProceedings of the 23rd International Conference on Computational Linguistics10.5555/1873781.1873897(1029-1037)Online publication date: 23-Aug-2010
  • (2010)Query-based text normalization selection models for enhanced retrieval accuracyProceedings of the NAACL HLT 2010 Workshop on Semantic Search10.5555/1867767.1867770(19-26)Online publication date: 5-Jun-2010
  • (2009)JHU Ad Hoc Experiments at CLEF 2008Evaluating Systems for Multilingual and Multimodal Information Access10.1007/978-3-642-04447-2_21(170-177)Online publication date: 2009
  • (2008)JHU ad hoc experiments at CLEF 2008Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access10.5555/1813809.1813836(170-177)Online publication date: 17-Sep-2008

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media