Abstract
At present, the amount of data generated by high-energy physics experiments is increasing. When Hadoop, a large data processing platform, is used to process high-energy physics data, it is faced with the actual needs of data migration. However, the existing migration tools do not support data transmission between HDFS and other file systems, and their performance has obvious defects. Starting from the requirements of synchronization and archiving of high energy physical data, this paper designs and implements a universal mass data migration system. By extending the access mode of HDFS data, MapReduce is used to migrate data directly between HDFS data nodes and other storage systems/media. In addition, the system designs and implements the dynamic priority scheduling model, and evaluates and chooses the dynamic priority of multi-task. The system has been applied to data migration in large high altitude air shower Observatory (LHAASO) cosmic ray and other physical experiments. The actual operation results show that the system has good performance and can meet the data migration requirements of various experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sahadat MN, Alreja A, Mikail N et al (2018) Comparing the use of single versus multiple combined abilities in conducting complex computer tasks hands-free. IEEE Trans Neural Syst Rehabil Eng 26(9):1868–1877
Glushkova D, Jovanovic P, Abello A (2019) Mapreduce performance model for Hadoop 2.x. Inf Syst 79:32–43
Lin CY, Lin YC (2017) An overall approach to achieve load balancing for hadoop distributed file system. Int J Web Grid Serv 13(4):448–466
Khalil Y, Alshayeji M, Ahmad I (2019) Distributed whale optimization algorithm based on MapReduce. Concur Comput-Pract Exp 31(1):e4872
Zhang H, Huang T, Lv ZH et al (2018) MCRS: a course recommendation system for MOOCs. Multimedia Tools Appl 77(6):7051–7069
Rashti MJ, Sabin G, Kettimuthu R (2016) Long-haul secure data transfer using hardware-assisted GridFTP. Future Gener Comput Syst-Int J Escience 56:265–276
Suryanarayana SM, Kotaleski JH, Grillner S et al (2019) Roles for globus pallidus externa revealed in a computational model of action selection in the basal ganglia. Neural Netw 109:113–136
Nunez-Gaona MA, Marcelin-Jimenez R, Gutierrez-Martinez J et al (2018) A dependable massive storage service for medical imaging. J Digit Imaging 31(5):628–639
Cao Z, Zheng XY, Liu YS et al (2018) Exploring the changing patterns of China’s migration and its determinants using census data of 2000 and 2010. Habitat Int 82:72–82
Komarasamy D, Muthuswamy V (2018) Priority scheduling with consolidation based backfilling algorithm in cloud. World Wide Web-Internet Web Inf Syst 21(6):1453–1471
Acknowledgments
This work was supported by grants from The National Natural Science Foundation of China (No. 61862056), the Guangxi Natural Science Foundation (No. 2017GXNSFAA198148) foundation of Wuzhou University (No. 2017B001) and Guangxi Colleges and Universities Key Laboratory of Professional Software Technology, Wuzhou University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, M., Zhuo, M. (2020). Development of Hadoop Massive Data Migration System. In: Xu, Z., Choo, KK., Dehghantanha, A., Parizi, R., Hammoudeh, M. (eds) Cyber Security Intelligence and Analytics. CSIA 2019. Advances in Intelligent Systems and Computing, vol 928. Springer, Cham. https://doi.org/10.1007/978-3-030-15235-2_199
Download citation
DOI: https://doi.org/10.1007/978-3-030-15235-2_199
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-15234-5
Online ISBN: 978-3-030-15235-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)