Abstract
As very large digital library collections become more commonplace, software tools must adapt appropriately. This paper reports on an evolution of the Greenstone Digital Library software to support parallel processing during the collection building phase. A series of experiments were conducted to first establish a basic speed-up factor, and then deconstruct the parallelisation process to understand the execution profile of the application. Several bottlenecks were identified and resolved to further improve the performance. The adaptation of Greenstone confirms that the build phase is indeed a suitable candidate for parallelisation; and suggests that parallelisation of processing is a new avenue for exploration in emerging digital library architectures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Adams, D.: Papers past: browse access for online New Zealand newspapers. Microform & Imaging Review 34, 22–27 (2005)
Bainbridge, D., Witten, I.H., Boddie, S., Thompson, J.: Stress-testing general purpose digital library software. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) ECDL 2009. LNCS, vol. 5714, pp. 203–214. Springer, Heidelberg (2009), http://portal.acm.org/citation.cfm?id=1812799.1812828
Barroso, L.A., Dean, J., Hölzle, U.: Web search for a planet: The Google cluster architecture. IEEE Micro 23, 22–28 (2003), http://portal.acm.org/citation.cfm?id=776692.776716
Goetz, B.: Java theory and practice: Thread pools and work queues. Tech. Rep. j-jtp0730, IBM, New York, United States (2002), http://www.ibm.com/developerworks/library/j-jtp0730/index.html
Manghi, P., Pagano, P., Ioannidis, Y.: Second Workshop on Very Large Digital Libraries: in conjunction with the European Conference on Digital Libraries. SIGMOD Rec. 38, 46–48 (2010), http://doi.acm.org/10.1145/1815948.1815959
Rasmussen, E.M.: Introduction: parallel processing and information retrieval. Inf. Process. Manage. 27, 255–263 (1991), http://portal.acm.org/citation.cfm?id=117658.117659
Stanfill, C.: Partitioned posting files: a parallel inverted file structure for information retrieval. In: Proceedings of the 13th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1990, pp. 413–428. ACM, New York (1990), http://doi.acm.org/10.1145/96749.98247
Witten, I.H., Bainbridge, D., Nichols, D.M.: How to Build a Digital Library, 2nd edn. Morgan Kaufmann Publishers Inc., San Francisco (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Thompson, J., Bainbridge, D., Suleman, H. (2011). Towards Very Large Scale Digital Library Building in Greenstone Using Parallel Processing. In: Xing, C., Crestani, F., Rauber, A. (eds) Digital Libraries: For Cultural Heritage, Knowledge Dissemination, and Future Creation. ICADL 2011. Lecture Notes in Computer Science, vol 7008. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24826-9_41
Download citation
DOI: https://doi.org/10.1007/978-3-642-24826-9_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-24825-2
Online ISBN: 978-3-642-24826-9
eBook Packages: Computer ScienceComputer Science (R0)