Abstract
Current data allocation algorithms neglect the problems of unsatisfactory allocation results and long execution time caused by the redundancy of full comparative data and the complexity of data types. To solve these problems, a real-time allocation algorithm of full comparison data based on storm is proposed. Firstly, the phase unwrapping algorithm of minimum spanning tree is used to remove redundant data in full comparison data; then, the distributed data clustering algorithm and storm framework are used to realize the full comparison data clustering after redundancy removal. Several main factors affecting the selection of statistical information are summarized according to the clustering results. Then the communication cost of data loading and transaction processing is determined, and the trade-off between read-only transaction and update transaction cost is achieved. By judging whether the total cost of read-only transaction and update transaction is reduced or not, the replica is eliminated, and a full comparison data allocation algorithm with minimum total cost of read-only transaction and update transaction is proposed to realize real-time allocation of full-comparative data. The example analysis shows that the proposed algorithm can meet the user’s needs in terms of execution time, acceleration ratio, storage efficiency and cost. Compared with the reference algorithm, the proposed algorithm has the lowest execution time, the highest acceleration ratio and the closest allocation cost to the ideal overhead.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Wang ZC, Li HX, Hou HH (2020) Sender data allocation scheme for reducing disorder degree in CMT system. Application Research of Computers 37:551–554
Ma XX, Lu G, Fu BZ et al (2020) Implementation methods and performance analysis of non-contiguous data communication in network. Chinese Journal of Computers. 43:1123–1138
Lin M, He ZF (2019) Big data allocation mechanism for wireless sensor communication based on cooperative Kalman filter. Electronic Measurement Technology 42:123–127
Zhang YM, Jiang JB, Lu JW et al (2019) An iterative data partitioning strategy for MapReduce. Chinese Journal of Computers 42:1873–1885
L. P. Xiang, H. J. Yang. Dynamic resource allocation algorithm based on big data stream characteristic and improved SOM clustering. Computer Applications and Software. 36 (2019) 262–268+,280
Ding ZC, Ge HW (2020) Density peaks clustering with optimized allocation strategy. Journal of Frontiers of Computer Science & Technology 14:792–802
Wang X (2020) Research on dynamic traffic data resource allocation algorithm of ship mobile network. Ship Science and Technology 42:184–186
Xv J, Wang XY, Cai ZX et al (2020) Imbalance classification based on informative instances selection. Journal of Frontiers of Computer Science & Technology. 14:401–409
L Shuai, W Shuai, L Xinyu, et al. (2021) Human memory update strategy: a multi-layer template update mechanism for remote visual monitoring, IEEE Transactions on Multimedia 23:2188–2198
Raju S, Chandrasekaran M (2019) Performance analysis of efficient data distribution in P2P environment using hybrid clustering techniques. Soft Comput 23:9253–9263
Ahmad SG, Khan HU, Ijaz S et al (2020) Use case-based evaluation of workflow optimization strategy in real-time computation system. J Supercomput 76:708–725
S Wang, X Liu, S Liu, et al. Human Short-Long Term Cognitive Memory Mechanism for Visual Monitoring in IoT-Assisted Smart Cities. IEEE Internet of Things Journal, 2021, online first, doi: https://doi.org/10.1109/JIOT.2021.3077600
Shui-Hua W, Xiaosheng W, Yu-dong Z et al (2020) Diagnosis of COVID-19 by wavelet Renyi entropy and three-segment biogeography-based optimization. International Journal of Computational Intelligence Systems 13(1):1332–1344
Liu S, Liu D, Muhammad K et al (2021) Effective template update mechanism in visual tracking with background clutter. Neurocomputing 458:615–625
Yu-Dong Z, Zhengchao D, Shui-HuaW, et al. (2020) Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Information Fusion 64:149–187
Pourpanah F, Shi YH, Lim CP et al (2019) Feature selection based on brain storm optimization for data classification. Applied Soft ComputingVolume 80:761–775
Wang SH, Govindaraj VV, Górriz JM (2021) Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network. Information Fusion 67:208–229
Liu S, Wang S, Liu X et al (2021) Fuzzy detection aided real-time and robust visual tracking under complex environments. IEEE Trans Fuzzy Syst 29(1):90–102
Castro-Medina F, Rodriguez-Mazahua L, Lopez-Chau A et al (2020) FRAGMENT: a web application for database fragmentation. Allocation and Replication over a Cloud Environment IEEE Latin America Transactions 18:1126–1134
Rossi ALD, Soares C, Souza BFD, Ponce de Leon Ferreira de Carvalho AC (2021) Micro-MetaStream: algorithm selection for time-changing data. Inf Sci 565:262–277
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dong, Cq., Chen, C., Ren, N. et al. Real-Time Distribution Algorithm for Fully Comparison Data Based on Storm. Mobile Netw Appl 27, 588–597 (2022). https://doi.org/10.1007/s11036-021-01824-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11036-021-01824-3