Skip to main content
SpringerLink
Log in

An optimized cluster storage method for real-time big data in Internet of Things

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Data storage, especially big data storage, is a research hot spot in Internet of Things (IoT) system today. In traditional data storage methods, the fault-tolerant algorithm for data copies is adjusted with whole data file, which causes huge redundancy because there are less utilization and more cost of data storage when only a part of data blocks in the file are accessed. Therefore, an optimized cluster storage method for big data in IoT is proposed in this paper. First, weights of data blocks in each historical accessing period are calculated by temporal locality of data access, and the access frequencies of the data block in next period are predicted by the weights. Second, the hot spot of a data block is determined with a threshold which is calculated by previous data access. Meantime, in order to improve the data access efficiency and resource utilization, as well as to reduce the copy storage costs, copy of data block is dynamically adjusted and stored in different groups with high-performance and low-load nodes for data balance. Finally, experimental results show that the storage cost of proposed method is 70% less than that of traditional methods, which means that the proposed method effectively improves the data access speed, reduces storage space, and balances the storage load.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Chen B, Ting KM, Washio T et al (2015) Half-space mass: a maximally robust and efficient data depth method. Mach Learn 100(2–3):677–699. https://doi.org/10.1007/s10994-015-5524-x

    Article  MathSciNet  MATH  Google Scholar 

  2. Cherubini G, Jelitto J, Venkatesan V (2016) Cognitive storage for big data. Computer 49(4):43–51. https://doi.org/10.1109/MC.2016.117

    Article  Google Scholar 

  3. Cody BM, Bau D, Gonzalez-Nicolas A (2015) Stochastic injection-strategy optimization for the preliminary assessment of candidate geological storage sites. Hydrogeol J 23(6):1229–1245. https://doi.org/10.1007/s10040-015-1250-5

    Article  Google Scholar 

  4. El-Rabiaey MA, Areed NFF, Obayya SSA (2016) Novel plasmonic data storage based on nematic liquid crystal layers. J Lightwave Technol 34(16):3726–3732. https://doi.org/10.1109/JLT.2016.2582838

    Article  Google Scholar 

  5. Gaocheng L, Shuai L, Khan M et al (2018) Object tracking in vary lighting conditions for fog based intelligent surveillance of public spaces. IEEE Access 6:29283–29296

    Article  Google Scholar 

  6. Hong W, Yang Y, Li Y et al (2015) Systematic optimization of long gradient chromatography mass spectrometry for deep analysis of brain proteome. J Proteome Res 14(2):829–838. https://doi.org/10.1021/pr500882h

    Article  Google Scholar 

  7. Ishii T, Shimada K, Hoshizawa T et al (2015) Analysis of vibration effects on holographic data storage system. Jpn J Appl Phys 54(9S):09MA04. https://doi.org/10.7567/jjap.54.09ma04

    Article  Google Scholar 

  8. Kaneko K, Kawamoto Y, Nishiyama H et al (2015) An efficient utilization of intermittent surface–satellite optical links by using mass storage device embedded in satellites. Perform Eval 87(C):37–46. https://doi.org/10.1016/j.peva.2015.01.004

    Article  Google Scholar 

  9. Liu S, Fu W, He L et al (2017) Distribution of primary additional errors in fractal encoding method. Multimed Tools Appl 76(4):5787–5802

    Article  Google Scholar 

  10. Liu S, Bai W, Liu C et al (2018) Parallel fractal compression method for big video data. Complexity 2018:2016976. https://doi.org/10.1155/2018/2016976

    Article  MATH  Google Scholar 

  11. Shuai L, Xiaochun C, Caihe L et al (2013) Fractal property of generalized M-set with rational number exponent. Appl Math Comput 220:668–675

    MathSciNet  MATH  Google Scholar 

  12. Shuai L, Xiaochun C, Weina F et al (2014) Numeric characteristics of generalized M-set with its asymptote. Appl Math Comput 243:767–774

    MathSciNet  MATH  Google Scholar 

  13. Wang J, Huang CG, Liu SK et al (2016) Research of distributed storage method for agricultural scientific data. Comput Eng Appl 52(11):248–253

    Article  Google Scholar 

  14. Wijetunge CD, Saeed I, Boughton BA et al (2015) Exims an improved data analysis pipeline based on a new peak picking method for exploring imaging mass spectrometry data. Bioinformatics 31(19):3198–3206. https://doi.org/10.1093/bioinformatics/btv356

    Article  Google Scholar 

  15. Xu Z, Schrama E, Wal WVD (2015) Optimization of regional constraints for estimating the Greenland mass balance with grace level-2 data. Geophys J Int 202(1):381–393. https://doi.org/10.1093/gji/ggv146

    Article  Google Scholar 

  16. Yang G, Liu S (2014) Distributed cooperative algorithm for k − M set with negative integer k by fractal symmetrical property. Int J Distrib Sensor Netw 12:3. https://doi.org/10.1155/2014/398583

    Article  Google Scholar 

  17. Yang CT, Shih WC, Huang CL et al (2016) On construction of a distributed data storage system in cloud. Computing 98(1–2):93–118. https://doi.org/10.1007/s00607-014-0399-4

    Article  MathSciNet  Google Scholar 

  18. Zhang Y, Bhamber R, Riba-Garcia I et al (2015) Streaming visualisation of quantitative mass spectrometry data based on a novel raw signal decomposition method. Proteomics 15(8):1419–1427. https://doi.org/10.1002/pmic.201400428

    Article  Google Scholar 

  19. Zhao D, Katsouras I, Asadi K et al (2016) Retention of intermediate polarization states in ferroelectric materials enabling memories for multi-bit data storage. Appl Phys Lett 108(23):1040–1090. https://doi.org/10.1063/1.4953199

    Article  Google Scholar 

  20. Zheng P, Shuai L, Weina F (2017) A review of visual moving target tracking. Multimed Tools Appl 76(16):16989–17018

    Article  Google Scholar 

  21. Zheng P, Shuai L, Arun KS et al (2018) Visual attention feature (VAF): a novel strategy for visual tracking based on cloud platform in intelligent surveillance systems. J Parallel Distrib Comput 120:182–194

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Programs of National Natural Science Foundation of China (No: 61502254), Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region (No. NJYT-18-B10), Teaching Reform Project Fund of University of Electronic Science and Technology of China Zhongshan Institute (No: 418YKQN02), Scientific Research Fund of Guangdong Provincial Education Department under Grant (No: 416YCQ01), Teaching Reform Project Fund of Hunan province under Grant (No: 2016[400]), Scientific Research Fund of Hunan Provincial Education Department under Grant Nos: 16C0299, 17C0295), Natural Science Foundation of Hunan Province, China (No. 2018JJ2023). We want to thank Prof Zhang from University of Leicester for his careful checking of language to this paper.

Author information

Authors and Affiliations

  1. College of Mechanical Electrical Engineering, University of Electronic Science and Technology of China, Zhongshan Institute, Zhongshan, China

    Li Tu & Ping Li

  2. College of Computer Science, Inner Mongolia University, Hohhot, China

    Shuai Liu

  3. School of Information and Electronic Engineering, Hunan City University, Yiyang, China

    Yan Wang & Chi Zhang

  4. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Changchun, China

    Li Tu, Shuai Liu & Yan Wang

Authors
  1. Li Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuai Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tu, L., Liu, S., Wang, Y. et al. An optimized cluster storage method for real-time big data in Internet of Things. J Supercomput 76, 5175–5191 (2020). https://doi.org/10.1007/s11227-019-02773-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-019-02773-1

Keywords

Search

Navigation