Skip to main content
SpringerLink
Log in

IKAROS: An HTTP-Based Distributed File System, for Low Consumption & Low Specification Devices

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

We present the design of IKAROS: an HTTP-based distributed file system, which provides file access scalability and targets a large variety of operating systems and storage systems. IKAROS bypasses the server bottleneck enabling clients to access storage directly, while supporting the usage of multiple types of meta-data. It enables low-consumption, low-specification and low-cost devices to achieve a high throughput data transfer, responding to highly demanding applications. We present data transfer results comparing IKAROS, NFS, PVFS2 and HDFS on a Small Office/Home Office Network Attached Storage infrastructure. We show that IKAROS architecture satisfies and outperforms the data rate demands of high performance applications. We also present experimental results which compare IKAROS and GridFTP using the European Grid Infrastructure. IKAROS performs better in most cases while being competitive at the rest.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Allcock, W., Bester, J., Bresnahan, J., Chervenak, A., Liming, L., Meder, S., Tuecke, S.: GridFTP Protocol Specification, GFD-R-P.047, openGridForum (2005)

  2. Carns, P.H., Ligon III, W.B., Ross, R.B., Thakur, R.: PVFS: a parallel file system for Linux clusters. In: Proceedings of the 4th Annual Linux Show-case and Conference, pp. 317327. USENIX Association, Atlanta, GA (2000)

  3. Schwan, P.: Lustre: building a file system for 1000-node clusters. In: Proc. of the 2003 Linux Symposium (2003)

  4. Schmuck, F., Haskin, R.: GPFS: a shared-disk file system for large computing clusters. In: Proceedings of the FAST 2002 Conference on File and Storage Technologies. IBM Almaden Research Center, San Jose, CA (2002)

  5. Hildebrand, D., Honeyman, P.: Exporting storage systems in a scalable manner with pNFS. In: MSST ’05 Proceedings of the 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies

  6. Hildebrand, D., Honeyman, P.: Direct-pNFS: scalable, transparent, and versatile access to parallel file systems. In: HPDC ’07 Proceedings of the 16th International Symposium on High Performance Distributed Computing

  7. Stern, H.: Managing NFS and NIS. OReilly and Associates, Inc (1991)

  8. Braam, P.J.: The Coda distributed file system. Linux Journal 50, 46–51 (1998)

    Google Scholar 

  9. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: 19th ACM SOSP (2003)

  10. Nagle, D., Serenyi, D., Matthews, A.: The panasas activescale storage cluster: delivering scalable high bandwidth storage. In: SC 04: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (2004)

  11. Microsoft Inc.: Distributed file system. http://www.microsoft.com/windowsserversystem/dfs/default.mspx/ (2011). Accessed 3 Sept 2011

  12. GlusterFS: http://www.gluster.com/ (2011). Accessed 3 Sept 2011

  13. Isilon Systems: OneFS. http://www.isilon.com/ (2012). Accessed 5 June 2012

  14. POHMELFS: parallel optimized host message exchange layered file system. http://www.ioremap.net/projects/pohmelfs/ (2012). Accessed 4 Sept 2013

  15. Hupfeld, F., Cortes, T., Kolbeck, B., Focht, E., Hess, M., Malo, J., Marti, J., Stender, J., Cesario, E.: XtreemFS—a case for object-based storage in Grid data management. VLDB Workshop on Data Management in Grids (2007)

  16. Yang, H., Wenjun, Z., Qian, L.: MapReduce workload modeling with statistical approach. J. Grid Computing 10, 279310 (2012)

    Article  Google Scholar 

  17. Gu, Y., Grossman, R., Szalay, A., Thakar, A.: Distributing the Sloan Digital Sky Survey Using UDT and Sector. e-Science (2006)

  18. CloudStore: http://code.google.com/p/kosmosfs/ (2012). Accessed 4 Sept 2013

  19. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scalable, highperformance distributed file system. In: Proceedings of the 7th OSDI (2006)

  20. Tatebe, O., et al.: Gfarm V2: A Grid File System that Supports High-Performance Distributed and Parallel Data Computing, CHEP 04 (2004)

  21. Xiaohui, W., Li, W.W., Tatebe, O., Gaochao, X., Liang, H., Jiubin, J.: Implementing Data Aware Scheduling in Gfarm Using LSF Scheduler Plugin Mechanism, GCA05 (2005)

  22. Wei, X., Li Wilfred, W., Osamu, T., Xu, G., Hu, L., Ju, J.: Integrating Local Job Scheduler LSF with Gfarm, ISPA05, vol. 3758/2005 (2005)

  23. MooseFS: http://www.moosefs.org/ (2012). Accessed 4 Sept 2013

  24. Thain, D., Moretti, C., Hemmes, J.: Chirp: A Practical Global Filesystem for Cluster and Grid Computing, JGC. Springer (2008)

  25. Al-Kiswany, S., Gharaibeh, A., Ripeanu, M.: The case for a versatile storage system. In: Workshop on Hot Topics in Storage and File Systems (HotStorage09) (2009)

  26. Druschel, P., Rowstron, A.: Past: persistent and anonymous storage in a peer-to-peer networking environment. In: Proceedings of the 8th IEEE Workshop on Hot Topics in Operating Systems (HotOS) (2001)

  27. Circle: http://savannah.nongnu.org/projects/circle/ (2012). Accessed 4 Sept 2013

  28. Ousterhout, J., et al.: The case for RAMclouds: scalable high-performance storage entirely in DRAM. In: Operating System Review (2009)

  29. Andreeva, J., Campana, S., Fanzago, F., Herrala, J.: High-energy physics on the Grid: the ATLAS and CMS experience. J. Grid Computing 6, 313 (2008)

    Article  Google Scholar 

  30. KM3Net: http://www.km3net.org/ (2012). Accessed 4 Sept 2013

  31. LHC: http://lcg.web.cern.ch/lcg/ (2012). Accessed 4 Sept 2013

  32. Tuura, L., Barrass, T., Bonacorsi, D., et al.: PhEDEx high-throughput data transfer management system. CHEP 06, Mumbai, India (2006)

  33. Apache DSO: http://httpd.apache.org/docs/2.0/dso.html/ (2012). Accessed 20 Jan 2012

  34. Cristina, A., et al.: Design and implementation of the gLite CREAM job management service. Futur. Gener. Comput. Syst. 26, 654–667 (2009)

    Google Scholar 

  35. Tantisiriroj, W., Patil, S., Gibson, G., Woo Son, S., Lang, S.J., Ross, R.B.: On the duality of data-intensive file system design: reconciling HDFS and PVFS. SC11, Seattle, Washington USA (2011)

  36. CURL: http://curl.haxx.se/ (2012). Accessed 4 Sept 2013

  37. Borthakur, D.: The Hadoop distributed file system: architecture and design. http://hadoop.apache.org/core/docs/r0.16.4/hdfsdesign.html/ (2012). Accessed 20 Jan 2012

  38. Joukov, N., Sipek, J.: GreenFS: Making enterprise computers greener by protecting them better. ACM SIGOPS Operating Systems Review – EuroSys 42(4), 69–80 (2008)

    Article  Google Scholar 

  39. Filippidis, C., Cotronis, Y., Markou, C.: Design and implementation of the mobile Grid resource management system. Comput. Sci. 13(1), 17–24. ISSN 1508-2806 (2012)

    Article  Google Scholar 

  40. Raicu, I., Foster, I., Beckman, P.: Making a Case for Distributed File Systems at Exascale, LSAP11, 8 June (2011)

  41. Raicu, I., Zhang, Z., Wilde, M., Foster, I., Beckman, P., Iskra, K., Clifford, B.: Toward Loosely Coupled Programming on Petascale Systems, IEEE SC (2008)

  42. Domenici, A., Donno, F.: Static and dynamic data models for the storage resource manager v2.2. J. Grid Computing 7, 115133 (2009)

    Article  Google Scholar 

  43. Rimal, B., Jukan, A., Katsaros, D., Goeleven, Y.: Architectural requirements for cloud computing systems: an enterprise cloud approach. J. Grid Computing 9, 326 (2011)

    Article  Google Scholar 

  44. Ponciano, L., Brasileiro, F.: Assessing green strategies in peer-to-peer opportunistic Grids. J. Grid Computing 11, 129–148 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics & Telecommunications, University of Athens, Athens, Greece

    Christos Filippidis & Yiannis Cotronis

  2. Institute of Nuclear and Particle Physics, “NCSR” Demokritos, Agia Paraskevi, Greece

    Christos Filippidis & Christos Markou

Authors
  1. Christos Filippidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiannis Cotronis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christos Markou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christos Filippidis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Filippidis, C., Cotronis, Y. & Markou, C. IKAROS: An HTTP-Based Distributed File System, for Low Consumption & Low Specification Devices. J Grid Computing 11, 681–698 (2013). https://doi.org/10.1007/s10723-013-9271-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-013-9271-6

Keywords

Search

Navigation