Abstract
With the increasing amount of data the research community is facing problems with methods of effectively accessing, storing, and processing data in large scale and geographically distributed environments. This paper addresses major data management issues, in particular use cases and scenarios (on the basis of Polish research community organized around the PLGrid PLUS Project) and discusses architectures of data storage management systems available in both PL-Grid and other similar federated environments. On that basis, a concept of a new meta storage system, named VeilFS, is presented. The proposed system unifies file access methods for geographically distributed large scale systems and hides complexity of data access and management in such environments. However, it should be emphasized that the main purpose of this article is identification and discussion about users’ requirements and existing solutions. The VeilFS system will be described in detail in the future.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
FUSE: Filesystem in Userspace. http://fuse.sourceforge.net/ (2013). Accessed 21 April 2013 (Online)
PLATON Storage Service U4. http://www.storage.pionier.net.pl/ (2013). Accessed 21 April 2013 (Online)
Tachyon Project. http://tachyon-project.org/ (2013). Accessed 21 April 2013
Atkinson, M., et al.: Data-intensive research workshop report. Technical Report, e-Science Institute. http://research.nesc.ac.uk/files/DIRWS.pdf (2010)
Baud, J.P.B., Caey, J., Lemaitre, S., Nicholson, C., Smith, D., Stewart, G.: LCG data management: from EDG to EGEE. In: UK e-Science All Hands Meeting, Nottingham, UK (2005)
Benedyczak, K., Rekawek, T., Rybicki, J., Schuller, B.: UNICORE data management: recent advancements. In: Romberg, M., Bala, P., Mller-Pfefferkorn, R., Mallmann, D. (eds.) UNICORE Summit 2011 Proceedings, Torun, Poland, 7–8 July 2011. IAS Series, vol. 9, pp. 24–27, Forschungszentrum Jülich (2011)
Braam, P.J., Schwan, P.: Lustre: the intergalactic file system. In: Ottawa Linux Symposium, June 2002
Dutka, Ł., Kitowski, J.: Application of component-expert technology for selection of data-handlers in CrossGrid. In: Kranzlmüller, D., Kacsuk, P., Dongarra, J., Volkert, J. (eds.) PVM/MPI 2002. LNCS, vol. 2474, pp. 25–32. Springer, Heidelberg (2002)
Grid File Access Library 2.0 official page. https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2 (2013). Accessed 14 April 2013
Hey, A., Tansley, S., Tolle, K.: The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, Redmond (2009)
Hunich, D., Muller-Pfefferkorn, R.: Managing large datasets with iRODS: a performance analysis. In: Proceedings of the 2010 International Multiconference on Computer Science and Information Technology (IMCSIT), pp. 647–654 (2010)
Kitowski, J., et al.: Polish computational research space for international scientific collaborations. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2011, Part I. LNCS, vol. 7203, pp. 317–326. Springer, Heidelberg (2012)
Kitowski, J., Turała, M., Wiatr, K., Dutka, Ł.: Pl-grid: foundations and perspectives of national computing infrastructure. In: Bubak, M., Szepieniec, T., Wiatr, K. (eds.) PL-Grid 2011. LNCS, vol. 7136, pp. 1–14. Springer, Heidelberg (2012)
Lustre. http://www.whamcloud.com/lustre/ (2013). Accessed 10 January 2013
Mills, S., Lucas, S., Irakliotis, L., Rappa, M., Carlson, T., Perlowitz, B.: DEMYSTIFYING BIG DATA: a practical guide to transforming the business of Government. Technical report. http://www.ibm.com/software/data/demystifying-big-data/ (2012)
OpenStack Object Storage (“Swift”). https://wiki.openstack.org/wiki/Swift (2013). Accessed 14 April 2013
PLGrid Plus project. http://www.plgrid.pl/en#section-1t (2013). Accessed 14 April 2013
Roblitz, T.: Towards implementing virtual data infrastructures a case study with iRODS. Comput. Sci. 13(4), 21–33 (2012). http://journals.agh.edu.pl/csci/article/view/43
Shafer, J., Rixner, S., Cox, A.L.: The Hadoop distributed filesystem: balancing portability and performance. In: ISPASS, pp. 122–133, March 2010
Słota, R., Nikolow, D., Skitał, Ł., Kitowski, J.: Implementation of replication methods in the grid environment. In: Sloot, P.M.A., Hoekstra, A.G., Priol, T., Reinefeld, A., Bubak, M. (eds.) EGC 2005. LNCS, vol. 3470, pp. 474–484. Springer, Heidelberg (2005)
Słota, R.: Storage QOS provisioning for execution programming of data-intensive applications. Sci. Program. 20(1), 69–80 (2012)
Słota, R., Król, D., Skałkowski, K., Orzechowski, M., Nikolow, D., Kryza, B., Wrzeszcz, M., Kitowski, J.: A toolkit for storage QOS provisioning for data-intensive applications. Comput. Sci. 13(1), 63–73 (2012). http://journals.agh.edu.pl/csci/article/view/26
Słota, R., Nikolow, D., Kitowski, J., Król, D., Kryza, B.: FiVO/QStorMan semantic toolkit for supporting data-intensive applications in distributed environments. Comput. Inform. 31(5), 1003–1024 (2012)
Stewart, G.A., Cameron, D., Cowan, G.A., McCance, G.: Storage and data management in EGEE. In: Proceedings of the fifth Australasian Symposium on ACSW frontiers, ACSW’07, Australia, vol. 68, pp. 69–77. Australian Computer Society Inc, Darlinghurst (2007)
Thain, D., Livny, M.: Parrot: an application environment for data-intensive computing. J. Parallel Distrib. Comput. Pract. 6(3), 9–18 (2005)
Worldwide LHC Computing Grid. http://wlcg.web.cern.ch/ (2013). Accessed 10 April 2013
Zhou, K., Wang, H., Li, C.: Cloud storage technology and its application. ZTE Commun. 16(4), 24–27 (2010)
Acknowledgments
This research is supported partly by the European Regional Development Fund program no. POIG.02.03.00-00-096/10 as part of the PLGrid PLUS project and AGH-UST grants no. 11.11.230.015 and 15.11.230.097.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Słota, R. et al. (2014). Storage Management Systems for Organizationally Distributed Environments PLGrid PLUS Case Study. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds) Parallel Processing and Applied Mathematics. PPAM 2013. Lecture Notes in Computer Science(), vol 8384. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-55224-3_68
Download citation
DOI: https://doi.org/10.1007/978-3-642-55224-3_68
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-55223-6
Online ISBN: 978-3-642-55224-3
eBook Packages: Computer ScienceComputer Science (R0)