Abstract
Owing to performance improvement of mobile devices, number of mobile applications and their variety has increased exponentially in recent years. However, many of these mobile applications are not executed alone and need server-side Internet services which require computing functions such as processing, networking, and storage. The server-side Internet services are usually provided using computing resources at Cloud data center because mobile applications are rapidly increasing in number and they tend to be more and more complex in nature. In addition, the conventional data managing framework, like 3-tier architecture, face additional problems such as heterogeneous external data to import and the vast amount of data to process. In this paper, we propose a data processing framework for mobile applications based on OGSA-DAI for heterogeneous external data import and MapReduce for large data processing. We designed and implemented a data connector based on OGSA-DAI middleware which can access and integrate heterogeneous data in a distributed environment, supporting various data management functions. And then we deployed a data processing framework (we call this data connector) into a Cloud system for mobile applications. We also used MapReduce programming model for data connector. Finally, we conducted various experiments and showed that our proposed framework can be used to access heterogeneous external data and to process large data with negligible or no system overhead.
This research was supported by the MKE, Korea, under the ITRC(Information Technology Research Center) support program supervised by the NIPA(NIPA-2011-C1090-1101-0008). This work was supported by the Dongguk University Research Fund of 2011.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kovachev., D., Renzel., D., Klamma, R., Cao, Y.: Mobile community cloud computing: emerges and evolves. In: Proc. 1st Intl. Workshop on Mobile Cloud Computing (MDM 2010), pp. 393–395 (2010)
Smartphone, Wikipedia, http://en.wikipedia.org/wiki/Smartphone/
Marinelli, E.: Hyrax: cloud computing on mobile devices using MapReduce. Master thesis, Carnegie Mellon University (2009)
Wang, G., Ng, T.: The impact of virtualization on network performance of amazon ec2 data center. In: Proc.of INFOCOMM, pp. 1–9 (2010)
Wang, J., Varman, P., Xie, C.: Avoiding performance fluctuation in cloud storage. In: Proc., of HiPC (2010)
Brito, M., Kakugawa, F., Sato, L., Correa, P.: An Architecture for Integrating Databases with Replication Support Based on the OGSA-DAI Middleware. In: Proc. of International Conference on Computational Science and Engineering (2009)
Jiang., L., Li., B., Song, M.: THE optimization of HDFS based on small files. In: Proc. of 3rd IEEE International Conference on Broadband Network and Multimedia Technology, pp. 912–915 (2010)
Mackey, G., Sehrish, S., Wang, J.: Improving metadata management for small files in HDFS. In: Proc. of IEEE International Conference on Cluster Computing, pp. 1–4 (2009)
Hadoop, A.: http://hadoop.apache.org/
Huang, L., Wang, X.-W., Zhai, Y.-D., Yang, B.: Extraction of User Profile Based on the Hadoop Framework. In: Proc. of IEEE Conf. on Wireless Communications, Networking and Mobile Computing, pp. 1–6 (2009)
Gunarathne, T., Wu, T.-L., Qiu, J., Fox, G.: MapReduce in the Clouds for Science. In: Proc. of IEEE Conf. on Second International Conference, pp. 565–572 (2010)
Mackey, G., Sehrish, S., Bent, J., Lopez, J., Habib, S., Wang, J.: Introducing map-reduce to high end computing. In: Proc. of Petascale Data Storage Workshop, pp. 1–6 (2008)
Dean, J.: Experiences with mapreduce, an abstraction for large-scale computation. In: Proc. of the 15th International Conference on Parallel Architectures and Compilation Techniques, New York, pp. 1–1 (2006)
Montero., R.S., Huedo., E., Llorente, I.M.: Dynamic deployment of custom execution environments in Grids. In: 2nd International Conference on Advanced Engineering Computing and Applications in Sciences, pp. 33–38 (2008)
Foster, I., Kesselman, C., Tuecke, S.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal of High Performance Computing Applications 15(3), 200–222 (2001)
Grant, A., Antonioletti, M., Hume, A.C., Krause, A., Dobrzelecki, B., Jackson, M.J., Parsons, M., Atkinson, M.P., Theocharopoulos, E.: OGSA-DAI: Middleware for Data Integration: Selected Applications. In: Proc. of IEEE Fourth International Conference on eScience, pp. 343–343 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kim, H., Kim, W., Lee, K., Kim, Y. (2011). A Data Processing Framework for Cloud Environment Based on Hadoop and Grid Middleware. In: Kim, Th., et al. Grid and Distributed Computing. GDC 2011. Communications in Computer and Information Science, vol 261. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27180-9_63
Download citation
DOI: https://doi.org/10.1007/978-3-642-27180-9_63
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-27179-3
Online ISBN: 978-3-642-27180-9
eBook Packages: Computer ScienceComputer Science (R0)