Abstract
The scale and growth rate of text collection bring new challenges for index construction. To tackle this problem, an efficient indexing algorithm, the Fine Granularity Pipeline (FGP), is proposed to improve the indexing performance for multi-core platform. Compared to the Simple Pipeline (SP) algorithm, compressing and parsing are divided in the FGP to get more balanced pipeline stages. Evaluations for three collections from Terabyte track in the TREC 2011 over Intel Woodcrest platform showed that the performance improvements of the FGP were over 45% and 17% compared to indexing in Indri 2.4 and the SP with three cores.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Melnik, S., Raghavan, S., Yang, B., Garcia-Molina, H.: Building a distributed full-text index for the web. ACM Transactions on Information Systems 19(3), 217–241 (2001)
Golomb, S.W.: Run-length Encodings: IEEE Transactions on Information Theory IT-12, 399–401
Elias, P.: Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory IT-21, 194–203
Scholer, F., Williams, H.E., Yiannis, J., Zobel, J.: Compression of inverted indexes for fast query evaluation. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 222–229 (2002)
Anh, V.N., Moffat, A.: Inverted Index Compression Using Word-Aligned Binary Codes. Information Retrieval 8(1), 151–166 (2005)
Anh, V.N., Moffat, A.: Improved Word-Aligned Binary Compression for Text Indexing. IEEE Transactions on Knowledge and Data Engineering 18(6), 857–861 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, SQ., Li, JR. (2012). Fine Granularity Pipeline Indexing Algorithm for Multi-core Platform. In: Liu, C., Wang, L., Yang, A. (eds) Information Computing and Applications. ICICA 2012. Communications in Computer and Information Science, vol 307. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34038-3_61
Download citation
DOI: https://doi.org/10.1007/978-3-642-34038-3_61
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34037-6
Online ISBN: 978-3-642-34038-3
eBook Packages: Computer ScienceComputer Science (R0)