Skip to main content
SpringerLink
Log in

Implementation of a Distributed Data Storage System with Resource Monitoring on Cloud Computing

  • Conference paper
Book cover Advances in Grid and Pervasive Computing (GPC 2012)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 7296))

Included in the following conference series:

Abstract

This paper focuses on cloud computing infrastructure, and especially data services. The goal of this paper is to implement a high performance and load balancing, and able-to-be-replicated system that provides data storage for private cloud users through a virtualization system. The DaaS extends the functionality of the Hadoop distributed system (HDFS). The proposed approach also implements a resource monitor of machine status factors such as CPU, memory, and network usage to help optimize the virtualization system and data storage system. To prove and extend the usability of this design, a synchronize app was also developed running on Android based on our distributed data storage (DDS).

This paper was supported in part by the National Science Council, Taiwan ROC, under grant numbers NSC 100-2218-E-029-004 and NSC 100-2622-E-029-008-CC3.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Cloud computing, http://en.wikipedia.org/wiki/Cloud_computing#Infrastructure

  2. Milojičić, D., Llorente, I.M., Montero, R.S.: OpenNebula: A Cloud Management Tool. IEEE Internet Computing 15(2) (2011)

    Google Scholar 

  3. Sempolinski, P., Thain, D.: A Comparison and Critique of Eucalyptus, OpenNebula and Nimbus. In: 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CloudCom), December 3, pp. 417–426 (2010)

    Google Scholar 

  4. Cordeiro, T., Damalio, D., Pereira, N., Endo, P., Palhares, A., Gonçalves, G., Sadok, D., Kelner, J., Melander, B., Souza, V., Mångs, J.-E.: Open Source Cloud Computing Platforms. In: 2010 9th International Conference on Grid and Cooperative Computing (GCC), pp. 366–371 (2010)

    Google Scholar 

  5. Truong, H.L., Dustdar, S.: On analyzing and specifying concerns for data as a service. In: IEEE Asia-Pacific on Services Computing Conference, APSCC 2009, pp. 87–94 (2009)

    Google Scholar 

  6. Truong, H.-L., Dustdar, S.: On Evaluating and Publishing Data Concerns for Data as a Service. In: 2010 IEEE Asia-Pacific Services Computing Conference (APSCC), pp. 363–370 (2010)

    Google Scholar 

  7. Ju, D., Liu, C., Wang, D., Liu, H., Tang, Z.: Performance Comparison of IP-Networked Storage. Tsinghua Science & Technology 14(1), 29–40 (2009)

    Article  Google Scholar 

  8. Wang, D.: Meeting Green Computing Challenges. In: International Symposium on High Density packaging and Microsystem Integration, HDP 2007, pp. 1–4 (2007)

    Google Scholar 

  9. Mackey, G., Sehrish, S., Wang, J.: Improving metadata management for small files in HDFS. In: IEEE International Conference on Cluster Computing and Workshops, CLUSTER 2009, pp. 1–4 (2009)

    Google Scholar 

  10. Shafer, J., Rixner, S., Cox, A.L.: The Hadoop Distributed Filesystem: Balancing Portability and Performance, pp. 122–133. IEEE, Houstan (2010)

    Google Scholar 

  11. Jiang, L., Li, B., Song, M.: THE optimization of HDFS based on small files. In: 2010 3rd IEEE International Conference on Broadband Network and Multimedia Technology (IC-BNMT), pp. 122–133 (2010)

    Google Scholar 

  12. Barlet-Ros, P., Iannaccone, G., Sanjuas-Cuxart, J., Sole-Pareta, J.: Predictive Resource Management of Multiple Monitoring Applications. IEEE/ACM Transactions on Networking 19(3), 788–801 (2011)

    Article  Google Scholar 

  13. Cheng, G., Gong, J.: A Resource-Efficient Flow Monitoring System. IEEE Communications Letters 11(6), 558–560 (2007)

    Article  Google Scholar 

  14. An Adaptive Resource Monitoring Method for Distributed Heterogeneous Computing Environment. In: 2009 IEEE International Symposium on Parallel and Distributed Processing with Applications, pp. 40–44. Sch. of Comput. Sci., Northwestern Polytech. Univ., Xi’an, China (2009)

    Google Scholar 

  15. Miettinen, T., Pakkala, D., Hongisto, M.: A Method for the Resource Monitoring of OSGi-based Software Components. In: 34th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2008, pp. 100–107 (2008)

    Google Scholar 

  16. Wang, C.-C., Chen, Y.-M., Weng, C.-H., Chung, T.-Y.: An overlay resource monitor system. In: The 8th International Conference on Advanced Communication Technology, ICACT 2006, vol. 3, p. 5 (2006)

    Google Scholar 

  17. Düllmann, D., Hoschek, W., Jaen-Martinez, J., Segal, B.: Model for Replica Synchronization and Consistency in a Data Grid. In: The IEEE International Symposium on High Performance Distributed Computing, San Francisco, CA, USA, pp. 67–75 (2001)

    Google Scholar 

  18. Xu, P., Huang, X., Wu, Y., Liu, L., Zheng, W.: Campus Cloud for Data Storage and Sharing. In: Eighth International Conference on Grid and Cooperative Computing, GCC 2009, pp. 244–249 (2009)

    Google Scholar 

  19. Zeng, W., Zhao, Y., Song, W.: Research on Cloud Storage Architecture and Key Technologies. In: ICIS 2009, November 24-26. ACM (2009)

    Google Scholar 

  20. Zhan, Y., Sun, Y.: Cloud Storage Management Technology. In: Proceedings of the 2009 Second International Conference on Information and Computing Science, May 21-22, pp. 309–311 (2009)

    Google Scholar 

  21. Hirofuchi, T., Nakada, H., Ogawa, H., Itoh, S., Sekiguchi, S.: A live storage migration mechanism over wan and its performance evaluation. In: Proceedings of the 3rd International Workshop on Virtualization Technologies in Distributed Computing, Barcelona, Spain, June 15 (2009)

    Google Scholar 

  22. Bertino, E., Maurino, A., Scannapieco, M.: Guest editors’ introduction: Data quality in the internet aera. IEEE Internet Computing 14, 11–13 (2010)

    Article  Google Scholar 

  23. Carns, P., Lang, S., Ross, R., Vilayannur, M., Kunkel, J., Ludwig, T.: Small-File Access in Parallel File Systems. In: Proceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium, pp. 1–11 (April 2009)

    Google Scholar 

  24. Amazon S3, http://en.wikipedia.org/wiki/Amazon_S3

  25. Amazon Simple Storage Service, http://aws.amazon.com/s3/

  26. EgeCast, http://www.edgecast.com/

  27. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. ACM Transactions on Computer Systems 26(2), Article 4 (June 2008)

    Google Scholar 

  28. Ghemawat, S., Gobioff, H., Leung, S.-T.: The google file system. In: SOSP 2003: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, pp. 29–43. ACM Press, New York (2003)

    Chapter  Google Scholar 

  29. Ceph, http://ceph.newdream.net/

  30. Gu, Y., Lu, L., Grossman, R., Yoo, A.: Processing Massived Sized Graphs using Sector/Sphere. In: 3rd Workshop on Many-Task Computing on Grids and Supercomputers, co-located with SC10, New Orleans, LA, November 15 (2010)

    Google Scholar 

  31. Gu, Y., Grossman, R.: Sector and Sphere: The Design and Implementation of a High Performance Data Cloud. Theme Issue of the Philosophical Transactions of the Royal Society A: Crossing Boundaries: Computational Science, E-Science and Global E-Infrastructure 367(1897), 2429–2445 (2009)

    Google Scholar 

  32. SAN, http://en.wikipedia.org/wiki/Storage_area_network

  33. NAS, http://en.wikipedia.org/wiki/Network-attached_storage

  34. iSCSI, http://en.wikipedia.org/wiki/ISCSI

  35. NFS, http://en.wikipedia.org/wiki/Network_File_System_protocol

  36. OpenNeBula, http://opennebula.org/

  37. Ctrix XenServer, http://www.citrix.com/

  38. Lo, C.-T.D., Qian, K.: Green Computing Methodology for Next Generation Computing Scientists. In: 2010 IEEE 34th Annual Computer Software and Applications Conference (COMPSAC), pp. 250–251 (2010)

    Google Scholar 

  39. Giroire, F., Guinand, F., Lefevre, L., Torres, J.: Energy-aware, power-aware, and Green Computing for large distributed systems and applications. In: 2010 International Conference on High Performance Computing and Simulation, HPCS, pp. lv – lxvii (2010)

    Google Scholar 

  40. Zhong, B., Feng, M., Lung, C.-H.: A Green Computing Based Architecture Comparison and Analysis. In: 2010 IEEE/ACM Int’l Conference on & Int’l Conference on Cyber, Physical and Social Computing (CPSCom), Green Computing and Communications (GreenCom), pp. 386–391 (2010)

    Google Scholar 

  41. Richardson, L., Ruby, S.: Restful Web Services, 1st edn., O’Reilly Media ( May15, 2007)

    Google Scholar 

  42. RFC 2616, http://tools.ietf.org/html/rfc2616

  43. Fielding, R.T., Gettys, J., Mogul, J.C., Nielsen, H.F., Masinter, L., Leach, P.J., Berners-Lee.: RFC 2616: Hypertext Transfer Protocol – HTTP/1.1

    Google Scholar 

  44. NIO, http://en.wikipedia.org/wiki/New_I/O

  45. Hadoop, http://hadoop.apache.org

  46. dd, http://en.wikipedia.org/wiki/Dd_Unix

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Tunghai University, Taichung, 40704, Taiwan ROC

    Chao-Tung Yang & Chih-Lin Huang

  2. Department of Applied Informatics and Multimedia, Asia University, Taichung, 41354, Taiwan ROC

    Wen-Chung Shih

Authors
  1. Chao-Tung Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wen-Chung Shih
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chih-Lin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Science and Technology, Huazhong University of Science and Technology, 1037 Luoyu Road, 430074, Wuhan, China

    Ruixuan Li

  2. Department of Computing, Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China

    Jiannong Cao

  3. University of Franche-Comte, FEMTO-ST, 1 cours Leprince-Ringuet, 25200, Montbéliard, France

    Julien Bourgeois

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yang, CT., Shih, WC., Huang, CL. (2012). Implementation of a Distributed Data Storage System with Resource Monitoring on Cloud Computing. In: Li, R., Cao, J., Bourgeois, J. (eds) Advances in Grid and Pervasive Computing. GPC 2012. Lecture Notes in Computer Science, vol 7296. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30767-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-30767-6_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-30766-9

  • Online ISBN: 978-3-642-30767-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation