Skip to main content
Springer Nature Link
Log in

Dual-Sorted Inverted Lists

  • Conference paper
String Processing and Information Retrieval (SPIRE 2010)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 6393))

Included in the following conference series:

Abstract

Several IR tasks rely, to achieve high efficiency, on a single pervasive data structure called the inverted index. This is a mapping from the terms in a text collection to the documents where they appear, plus some supplementary data. Different orderings in the list of documents associated to a term, and different supplementary data, fit widely different IR tasks. Index designers have to choose the right order for one such task, rendering the index difficult to use for others.

In this paper we introduce a general technique, based on wavelet trees, to maintain a single data structure that offers the combined functionality of two independent orderings for an inverted index, with competitive efficiency and within the space of one compressed inverted index. We show in particular that the technique allows combining an ordering by decreasing term frequency (useful for ranked document retrieval) with an ordering by increasing document identifier (useful for phrase and Boolean queries). We show that we can support not only the primitives required by the different search paradigms (e.g., in order to implement any intersection algorithm on top of our data structure), but also that the data structure offers novel ways of carrying out many operations of interest, including space-free treatment of stemming and hierarchical documents.

Funded in part by Fondecyt Grant 1-080019, Chile (first author) and by the Australian Research Council (second author).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Anh, V., de Kretser, O., Moffat, A.: Vector-space ranking with effective early termination. In: Proc. 24th SIGIR, pp. 35–42 (2001)

    Google Scholar 

  2. Anh, V., Moffat, A.: Inverted index compression using word-aligned binary codes. Inf. Retr. 8(1), 151–166 (2005)

    Article  Google Scholar 

  3. Anh, V., Moffat, A.: Pruned query evaluation using pre-computed impacts. In: Proc. 29th SIGIR, pp. 372–379 (2006)

    Google Scholar 

  4. Baeza-Yates, R.: A fast set intersection algorithm for sorted sequences. In: Sahinalp, S.C., Muthukrishnan, S.M., Dogrusoz, U. (eds.) CPM 2004. LNCS, vol. 3109, pp. 400–408. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  5. Baeza-Yates, R., Moffat, A., Navarro, G.: Searching Large Text Collections, pp. 195–244. Kluwer Academic, Dordrecht (2002)

    MATH  Google Scholar 

  6. Baeza-Yates, R., Ribeiro, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)

    Google Scholar 

  7. Baeza-Yates, R., Salinger, A.: Experimental analysis of a fast intersection algorithm for sorted sequences. In: Consens, M.P., Navarro, G. (eds.) SPIRE 2005. LNCS, vol. 3772, pp. 13–24. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  8. Barbay, J., Kenyon, C.: Adaptive intersection and t-threshold problems. In: Proc. 13th SODA, pp. 390–399 (2002)

    Google Scholar 

  9. Barbay, J., López-Ortiz, A., Lu, T., Salinger, A.: An experimental investigation of set intersection algorithms for text searching. ACM J. Exp. Alg. 14, article 7 (2009)

    Google Scholar 

  10. Brisaboa, N., Fariña, A., Navarro, G., Esteller, M.: S,C-dense coding: an optimized compression code for natural language text databases. In: Nascimento, M.A., de Moura, E.S., Oliveira, A.L. (eds.) SPIRE 2003. LNCS, vol. 2857, pp. 122–136. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  11. Buettcher, S., Clarke, C., Cormack, G.V.: Information Retrieval: Implementing and Evaluating Search Engines. MIT Press, Cambridge (2010)

    MATH  Google Scholar 

  12. Croft, B., Metzler, D., Strohman, T.: Search Engines: Information Retrieval in Practice. Pearson Education, London (2009)

    Google Scholar 

  13. Culpepper, J.S., Moffat, A.: Compact set representation for information retrieval. In: Ziviani, N., Baeza-Yates, R. (eds.) SPIRE 2007. LNCS, vol. 4726, pp. 137–148. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  14. Culpepper, J.S., Navarro, G., Puglisi, S.J., Turpin, A.: Top-k ranked document search in general text databases. In: Proc. 18th ESA (2010) (to appear)

    Google Scholar 

  15. Demaine, E., Munro, I.: Adaptive set intersections, unions, and differences. In: Proc. 11th SODA, pp. 743–752 (2000)

    Google Scholar 

  16. Gagie, T., Puglisi, S., Turpin, A.: Range quantile queries: Another virtue of wavelet trees. In: Karlgren, J., Tarhio, J., Hyyrö, H. (eds.) SPIRE 2009. LNCS, vol. 5721, pp. 1–6. Springer, Heidelberg (2009)

    Google Scholar 

  17. Grossi, R., Gupta, A., Vitter, J.: High-order entropy-compressed text indexes. In: Proc. 14th SODA, pp. 841–850 (2003)

    Google Scholar 

  18. Gupta, A.: Succinct Data Structures. PhD thesis, Duke University, USA (2007)

    Google Scholar 

  19. Heaps, H.: Information Retrieval - Computational and Theoretical Aspects. Academic Press, London (1978)

    MATH  Google Scholar 

  20. Hon, W.-K., Shah, R., Vitter, J.S.: Space-efficient framework for top-k string retrieval problems. In: Proc. 50th IEEE FOCS, pp. 713–722 (2009)

    Google Scholar 

  21. Hull, D.A.: Stemming algorithms: A case study for detailed evaluation. J. Amer. Soc. Inf. Sci. 47(1), 70–84 (1996)

    Article  Google Scholar 

  22. Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  23. Navarro, G., Mäkinen, V.: Compressed full-text indexes. ACM Comp. Surv. 39(1), article 2 (2007)

    Google Scholar 

  24. Navarro, G., Moura, E., Neubert, M., Ziviani, N., Baeza-Yates, R.: Adding compression to block addressing inverted indexes. Inf. Retr. 3(1), 49–77 (2000)

    Article  Google Scholar 

  25. Persin, M., Zobel, J., Sacks-Davis, R.: Filtered document retrieval with frequency-sorted indexes. J. Amer. Soc. Inf. Sci. 47(10), 749–764 (1996)

    Article  Google Scholar 

  26. Raman, R., Raman, V., Rao, S.: Succinct indexable dictionaries with applications to encoding k-ary trees and multisets. In: Proc. 13th SODA, pp. 233–242 (2002)

    Google Scholar 

  27. Sanders, P., Transier, F.: Intersection in integer inverted indices. In: Proc. 9th ALENEX (2007)

    Google Scholar 

  28. Sanders, P., Transier, F.: Compressed inverted indexes for in-memory search engines. In: Proc. 10th ALENEX, pp. 3–12 (2008)

    Google Scholar 

  29. Scholer, F., Williams, H., Yiannis, J., Zobel, J.: Compression of inverted indexes for fast query evaluation. In: Proc. 25th SIGIR, pp. 222–229 (2002)

    Google Scholar 

  30. Strohman, T., Croft, B.: Efficient document retrieval in main memory. In: Proc. 30th SIGIR, pp. 175–182 (2007)

    Google Scholar 

  31. Witten, I., Moffat, A., Bell, T.: Managing Gigabytes, 2nd edn. Morgan Kaufmann, San Francisco (1999)

    MATH  Google Scholar 

  32. Xu, J., Croft, W.B.: Corpus-based stemming using cooccurrence of word variants. ACM Trans. Inf. Sys. 16(1), 61–81 (1998)

    Article  Google Scholar 

  33. Yan, H., Ding, S., Suel, T.: Inverted index compression and query processing with optimized document ordering. In: Proc. 18th WWW, pp. 401–410 (2009)

    Google Scholar 

  34. Zipf, G.: Human Behaviour and the Principle of Least Effort. Addison-Wesley, Reading (1949)

    Google Scholar 

  35. Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comp. Surv. 38(2), article 6 (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Chile, Santiago, Chile

    Gonzalo Navarro

  2. School of Comp. Sci. & Inf. Tech., Royal Melbourne Institute of Technology, Melbourne, Australia

    Simon J. Puglisi

Authors
  1. Gonzalo Navarro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon J. Puglisi
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Physics and Mathematics, Edificio "B", Universidad Michoacana, Ciudad Universitaria, 5800, Morelia, Mich., Mexico

    Edgar Chavez

  2. Dept. of Computer Science and Enginerring, University of California, 92521, Riverside, CA, USA

    Stefano Lonardi

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Navarro, G., Puglisi, S.J. (2010). Dual-Sorted Inverted Lists. In: Chavez, E., Lonardi, S. (eds) String Processing and Information Retrieval. SPIRE 2010. Lecture Notes in Computer Science, vol 6393. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-16321-0_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-16321-0_33

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-16320-3

  • Online ISBN: 978-3-642-16321-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics