research-article

Dynamic Load Balancing Method for Urban Surveillance Video Big Data Storage Based on HDFS

Author:

Yue LiAuthors Info & Claims

ICCAI '21: Proceedings of the 2021 7th International Conference on Computing and Artificial Intelligence

Pages 160 - 167

https://doi.org/10.1145/3467707.3467730

Published: 24 September 2021 Publication History

Abstract

HDFS has been widely used by many video service websites, but its load balancing tool does not consider the bandwidth consumption characteristics of video file online playback and the heterogeneous performance difference of NameNode in metadata allocation problem. The dynamic load imbalance of cluster makes the utilization of bandwidth resources low. In this paper, a HDFS NameNode dynamic load balancing tool (NDLBT) for city monitoring video in urban surveillance video big data storage in cloud storage environment is proposed. method. Firstly, it analyzes the relationship between the bandwidth consumption and the bit rate, data block size and access heat of the video file when the video file is played online, and a new load evaluation model is established. On this basis, it adds consideration to the bandwidth consumption factor in the load scheme generation and load scheduling, and through the dynamic adaptive backup of multi-replica heterogeneous nodes of metadata. The dynamic distribution of metadata is realized under the consideration of node performance and load, and the performance of metadata server cluster is guaranteed. Finally, combined with cache strategy and automatic recovery mechanism, the reading and writing of metadata is improved. The simulation results show that compared with the proposed method, we can effectively avoid the aggregation of high bandwidth consumption data blocks. In the experimental scenario where high bandwidth consumption video files are used as service access hotspots, the proposed method is superior to the original load balancing method in 90% scenarios, and can reduce the bandwidth peak value of bottleneck nodes in data node clusters by 20%.

References

[1]

Adhikari M, Amgoth T. Heuristic-based load-balancing algorithm for IaaS cloud[J]. Future Generation Computer Systems, 2018, 81: 156-165.

Digital Library

[2]

Neghabi A A, Navimipour N J, Hosseinzadeh M, Load balancing mechanisms in the software defined networks: a systematic and comprehensive review of the literature[J]. IEEE Access, 2018, 6: 14159-14178.

[3]

Wan J, Chen B, Wang S, Fog computing for energy-aware load balancing and scheduling in smart factory[J]. IEEE Transactions on Industrial Informatics, 2018, 14(10): 4548-4556.

[4]

Ding S, Chen C, Xin B, A bi-objective load balancing model in a distributed simulation system using NSGA-II and MOPSO approaches[J]. Applied Soft Computing, 2018, 63: 249-267.

Digital Library

[5]

Lin Y D, Wang C C, Lu Y J, Two-tier dynamic load balancing in SDN-enabled Wi-Fi networks[J]. Wireless Networks, 2018, 24(8): 2811-2823.

Digital Library

[6]

Shao X, Jibiki M, Teranishi Y, An efficient load-balancing mechanism for heterogeneous range-queriable cloud storage[J]. Future Generation Computer Systems, 2018, 78: 920-930.

Digital Library

[7]

Lieber M, Nagel W E. Highly scalable SFC-based dynamic load balancing and its application to atmospheric modeling[J]. Future Generation Computer Systems, 2018, 82: 575-590.

[8]

Alarabi L, Mokbel M F, Musleh M. St-hadoop: A mapreduce framework for spatio-temporal data[J]. GeoInformatica, 2018, 22(4): 785-813.

Digital Library

[9]

Rodrigues R A, Lima Filho L A, Gonçalves G S, Integrating NoSQL, relational database, and the hadoop ecosystem in an interdisciplinary project involving big data and credit card transactions[M]//Information Technology-New Generations. Springer, Cham, 2018: 443-451.

[10]

Tang Z, Zhang X, Li K, An intermediate data placement algorithm for load balancing in Spark computing environment[J]. Future Generation Computer Systems, 2018, 78: 287-301.

[11]

Song J, He H Y, Wang Z, Modulo based data placement algorithm for energy consumption optimization of MapReduce system[J]. Journal of Grid Computing, 2018, 16(3): 409-424.

Digital Library

[12]

Uta A, Danner O, van der Weegen C, MemEFS: A network-aware elastic in-memory runtime distributed file system[J]. Future Generation Computer Systems, 2018, 82: 631-646.

[13]

Shah A, Padole M. Load Balancing through Block Rearrangement Policy for Hadoop Heterogeneous Cluster[C]//2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE, 2018: 230-236.

[14]

Wu X, Zhang C, Zhang R, A distributed intrusion detection model via nondestructive partitioning and balanced allocation for big data[J]. CMC-COMPUTERS MATERIALS & CONTINUA, 2018, 56(1): 61-72.

[15]

Ammar K, Özsu M T. Experimental analysis of distributed graph systems[J]. Proceedings of the VLDB Endowment, 2018, 11(10): 1151-1164.

Digital Library

[16]

Shabestari F, Rahmani A M, Navimipour N J, A taxonomy of software-based and hardware-based approaches for energy efficiency management in the Hadoop[J]. Journal of Network and Computer Applications, 2019, 126: 162-177.

[17]

Huang Z. Frame-groups based fractal video compression and its parallel implementation in Hadoop cloud computing environment[J]. Multidimensional Systems and Signal Processing, 2018, 29(3): 961-978.

Digital Library

[18]

Guerrero C, Lera I, Bermejo B, Multi-objective Optimization for Virtual Machine Allocation and Replica Placement in Virtualized Hadoop[J]. IEEE Transactions on Parallel and Distributed Systems, 2018, 29(11): 2568-2581.

Digital Library

[19]

Zhu J. Research on data mining of electric power system based on Hadoop cloud computing platform[J]. International Journal of Computers and Applications, 2019, 41(4): 289-295.

[20]

Lu X, Phang K. An enhanced Hadoop heartbeat mechanism for MapReduce task scheduler using dynamic calibration[J]. China Communications, 2018, 15(11): 93-110.

[21]

Berlińska J, Drozdowski M. Comparing load-balancing algorithms for MapReduce under Zipfian data skews[J]. Parallel Computing, 2018, 72: 14-28.

[22]

Guerrero C, Lera I, Juiz C. Migration-aware genetic optimization for MapReduce scheduling and replica placement in Hadoop[J]. Journal of Grid Computing, 2018, 16(2): 265-284.

Digital Library

Cited By

Sundara Kumar MMohan H(2024)Improving big data analytics data processing speed through map reduce scheduling and replica placement with HDFS using genetic optimization techniquesJournal of Intelligent & Fuzzy Systems10.3233/JIFS-24006946:4(10863-10882)Online publication date: 18-Apr-2024
https://doi.org/10.3233/JIFS-240069

Recommendations

An Adaptive Feedback Load Balancing Algorithm in HDFS
INCOS '13: Proceedings of the 2013 5th International Conference on Intelligent Networking and Collaborative Systems

As one of the most popular cloud storage distributed file systems, Hadoop Distributed File System (HDFS) has the characteristics of open source, low-cost, high fault tolerance and high scalability. Thus, HDFS plays a critical role in cloud storage. ...
Access-Load-Aware Dynamic Data Balancing for Cloud Storage Service
IDCS 2013: Proceedings of the 6th International Conference on Internet and Distributed Computing Systems - Volume 8223

Cloud storage is the typical way for storing massive data in Big Data Era. Dynamic data balancing is important for cloud storage since it aims to improve the utilization of computing resource and the performance of data process. However, storage-load-...
Preventing Data Popularity Concentration in HDFS based Cloud Storage
UCC '19 Companion: Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing Companion

Hadoop Distributed File System(HDFS) often experiences skew in data storage over time, mainly because of random data block allocation policy, datanode failure, replica reconstruction, and client activity, leading to utilization and load imbalance in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICCAI '21: Proceedings of the 2021 7th International Conference on Computing and Artificial Intelligence

April 2021

498 pages

ISBN:9781450389501

DOI:10.1145/3467707

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Major Cultivation of Scientific and Technological Achievements Transformation in Sichuan Education Department

Conference

ICCAI '21

ICCAI '21: 2021 7th International Conference on Computing and Artificial Intelligence

April 23 - 26, 2021

Tianjin, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
46
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sundara Kumar MMohan H(2024)Improving big data analytics data processing speed through map reduce scheduling and replica placement with HDFS using genetic optimization techniquesJournal of Intelligent & Fuzzy Systems10.3233/JIFS-24006946:4(10863-10882)Online publication date: 18-Apr-2024
https://doi.org/10.3233/JIFS-240069

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten