Abstract
Digital libraries and information management systems are increasingly being developed according to component models with well-defined APIs and often with Web-accessible interfaces. In parallel with metadata access and harvesting, Web 2.0 mashups have demonstrated the flexibility of developing systems as independent distributed components. It can be argued that such distributed components also can be an enabler for scalability of service provision in medium to large systems. To test this premise, this article discusses how an existing component framework was modified to include support for scalability. A set of lightweight services and extensions were created to migrate and replicate services as the load changes. Experiments with the prototype system confirm that this system can in fact be quite effective as an enabler of transparent and efficient scalability, without the need to resort to complex middleware or substantial system reengineering. Finally, specific problems areas have been identified as future avenues for exploration at the crucial intersection of digital libraries and high-performance computing.
Similar content being viewed by others
References
Anderson, D.P., Cobb, J., Korpela, E., Lebofsky M., Werthimer, D.: SETI@home: an experiment in public-resource computing. Commun. ACM 45(11), 56–61. ACM Press, New York (2002)
Andresen, D., Yang, T., Egecioglu, O., Ibarra, O.H., Smith, T.R.: Scalability issues for high performance digital libraries on the world wide web. Technical Report, March 1996, Department of Computer Science, University of California, Santa Barbara (1996)
Bar, M.: openMosix, a Linux Kernel Extension for Single System Image Clustering. In: Proceedings of Linux Kongress: 10th International Linux System Technology Conference, Saarbrücken, Germany, 15–16 October, pp. 94–102 (2003)
Candela, L., Castelli, D., Pagano, P.: A reference architecture for digital library systems. In: DELOS Conference on Digital Libraries. Grand Hotel Continental, Tirrenia, Pisa, Italy, 13–14 February (2007)
Castelli, D., Pagano, P.: A system for building expandable digital libraries. In: Delcambre, L., Henry, G. (ed.) Third ACM/IEEE-CS Joint Conference on Digital Libraries, Houston, USA, 27–31 May, 335–345. IEEE Computer Society, Washington, DC, USA (2003)
Catalyst IT.: Technical Evaluation of selected Open Source Repository Systems. Open Access Repositories in New Zealand. http://eduforge.org/docman/view.php/131/1062/Repository%20Evaluation%20Document.pdf (2006)
Don, K.J., Bainbridge, D., Witten, I.H.: The design of Greenstone 3: an agent based dynamic digital library. Technical report, December 2002. Department of Computer Science, University of Waikato, Hamilton, New Zealand. http://www.greenstone.org/manuals/gs3design.pdf (2002)
Foster, I.: Globus toolkit version 4: software for service-oriented systems. In: Gaudiot, J., Ni, L. (eds.) IFIP International Conference on Network and Parallel Computing, Beijing, China, 30 November–3 December, LNCS 3779, pp. 2–13. Springer, Berlin (2005)
Glasner, J.: Britannica Redux. Wired Magazine, March 1999. http://www.wired.com/techbiz/media/news/1999/11/32278
Haedstrom, M.: Research challenges in digital archiving and long-term preservation. In: Proceedings of NSF Post Digital Library Futures Workshop, Cape Cod, 15–17 June. http://www.sis.pitt.edu/~dlwkshop/paper_hedstrom.html (2003)
About OpenPBS. http://www.openpbs.org/about.html
IBM World Community Grid. http://www.worldcommunitygrid.org/about_us/viewAboutUs.do
Imafouo, A.: A scalability survey in IR and DL. TCDL Bulletin, 2, 2. http://www.ieee-tcdl.org/Bulletin/v2n2/imafouo/imafouo.html
Lagoze, C., Davis, J. R.: Dienst—an architecture for distributed document libraries. Commun. ACM, 38(4), 47. ACM Press, New York (1995)
Lagoze, C., Van de Sompel, H.: The open archives initiative: building a low barrier interoperability framework. In: Borgman, C. (ed.) JCDL 2001, Roanoke, VA., USA, 17–23 June, pp. 54–62. ACM Press, New York, NY (2001)
MPI-2: Extensions to the Message-Passing Interface. MPI-1.2 and MPI-2 standards. http://www.mpi-forum.org/docs/mpi2-report.pdf
Naughton, T., Scott, S.L., Barrett, B., Squyres, J., Lumsdaine. A, Fang, Y., Mashayekhi, V.: Looking Inside the OSCAR Cluster Toolkit. In: Dell Powers Solutions. http://www.csm.ornl.gov/PR/dell_1.html (2002)
Omar, M.: Component-based digital library scalability using clusters. MSc Thesis, University of Cape Town (2007)
Papadopoulos, P.M., Katz, M.J., Bruno, G.: NPACI rocks: tools and techniques for easily deploying manageable Linux clusters. In: 3rd IEEE International Conference on Cluster Computing (CLUSTER’01), Sutton Place Hotel, Newport Beach, California, USA, 8–11 October, 258. IEEE (2001)
Staples, T., Wayland, R., Payette, S.: The Fedora Project: An Open-source Digital Object Repository System. D-Lib Magazine, 9, 4. http://www.dlib.org/dlib/april03/staples/04staples.html April (2003)
Suleman, H., Fox, E.A.: A Framework for Building Open Digital Libraries. D-Lib Magazine, 7, 12. http://www.dlib.org/dlib/december01/suleman/12suleman.html December (2001)
Suleman, H.: Analysis and Evaluation of Service-Oriented Architectures for Digital Libraries. In: Turker, C., Agosti, M., Schek, H. (eds.) Peer-to-Peer, Grid, and Service-Orientation in Digital Library Architectures, 6th Thematic Workshop of the EU Network of Excellence DELOS, Cagliari, Italy, 24–25 June 2004, Revised Selected Papers. Lecture Notes in Computer Science 3664, 130–146. Springer, Berlin. http://pubs.cs.uct.ac.za/archive/00000278/01/delos_2005_paper_eval_full_revised.pdf (2005)
Suleman, H.: Parallelising harvesting. In: Sugimoto, S., Hunter, J., Rauber, A., Morishima, A. (eds.) Proceedings of 9th International Conference on Asian Digital Libraries (ICADL 2006), Tokyo, Japan, 27–30 November, 81–90. Springer, Berlin. http://pubs.cs.uct.ac.za/archive/00000328/01/icadl_2006_oaihpc.pdf (2006)
Suleman, H., Fox, E.A.: Designing protocols in support of digital library componentization. In: Agosti, M., Thanos, C. (eds.) Proceedings of 6th European Conference on Research and Advanced Technology for Digital Libraries (ECDL2002), LNCS 2458, Rome, Italy, 16–18 September, pp. 568–582. Springer, Berlin. http://www.husseinsspace.com/publications/ecdl_2002_paper_odl.pdf (2002)
Sunderam V.S.: PVM: A framework for parallel distributed computing. Concurrency Practice Exp. 2, 315–339 (1990)
Su, A., Choffnes, D.R., Kuzmanovic, A., Bustamante, F.E.: Drafting behind Akamai (travelocity-based detouring). In: Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM 2006), Pisa, Italy, 11–15 September, pp. 435–446. ACM, New York (2006)
Tannenbaum, T., Wright, D., Miller, K., Livny, M.: Condor—A Distributed Job Scheduler. In: Sterling, T. (ed.) Beowulf Cluster Computing with Linux. The MIT Press, Cambridge, MA, USA (2002)
van de Sompel, H., Bekaert J., Liu, X.: aDORe: a modular and standards-based digital object repository at the los alamos national laboratory. In: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’05), 367, ACM, New York (2005)
Author information
Authors and Affiliations
Corresponding author
Additional information
This project was made possible by funding from University of Cape Town and NRF (Grant number: GUN2073203).
Rights and permissions
About this article
Cite this article
Suleman, H., Parker, C. & Omar, M. Lightweight component-based scalability. Int J Digit Libr 9, 115–124 (2008). https://doi.org/10.1007/s00799-008-0047-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00799-008-0047-8