Skip to main content
SpringerLink
Log in

A Hadoop Processing Method for Massive Sensor Network Data Based on Internet of Things

  • Published:
International Journal of Wireless Information Networks Aims and scope Submit manuscript

Abstract

Based on the analysis of the architecture of the Internet of Things service platform and the key technologies of cloud computing, a massive sensing information processing scheme based on the Internet of Things service platform is proposed. The scheme first proposes a system architecture model that can satisfy the massive sensor information processing in an open platform environment, and designs multiple functional unit modules of the system. By combining these functional units, service configurability can be realized, facing thousands of services and Tenant. Then, Hadoop open source framework is used to realize the distributed computing of the system, which makes full use of the processing advantages of MapReduce computing model, HBase distributed database and HDFS distributed file system in Hadoop framework, and uses Oracle database as a supplement to realize the system high. Finally, the mass sensor information was deployed and tested. The effectiveness of the Hadoop processing method was verified by analyzing the results of MapReduce parallel computing experiments. The average cache hit rate is 93.1%, which has a high cache hit rate, greatly reduces MySQL database I/O, and improves system performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. R. D. Fátima, M. R .D. Andrade, A. C. Zucchi, et al., Distributed processing from large scale sensor network using Hadoop. In IEEE International Congress on Big Data, IEEE Computer Society, Washington, D.C., pp. 417–418, 2013.

  2. L. Cai, X. Guan, P. Chi, et al., Big data visualization collaborative filtering algorithm based on RHadoop, International Journal of Distributed Sensor Networks, Vol. 11, p. 3, 2015.

    Google Scholar 

  3. D. Wu, L. I. Zhuorong, R. Bie, et al., Research on database massive data processing and mining method based on Hadoop cloud platform. In International Conference on Identification, Information and Knowledge in the Internet of Things, IEEE, New York, pp. 107–110, 2015.

  4. X. Song, H. He, S. Niu, et al., A data streams analysis strategy based on Hoeffding tree with concept drift on Hadoop system. In International Conference on Advanced Cloud and Big Data, IEEE, New York, pp. 45–48, 2017.

  5. B. Li. Study on massive e-government data cloud storage scheme based on Hadoop. In IEEE International Conference on Software Engineering and Service Science, IEEE, New York, pp. 434–437, 2013.

  6. C. Yang, X. Zhang, C. Zhong, et al., A spatiotemporal compression based approach for efficient big data processing on cloud, Journal of Computer & System Sciences, Vol. 80, No. 8, pp. 1563–1583, 2014.

    Article  MathSciNet  Google Scholar 

  7. S. Chen, T. Bednarz, P. Szul, et al., Galaxy + Hadoop: toward a collaborative and scalable image processing toolbox in cloud. In International Conference on Service-Oriented Computing, Springer, Cham, pp. 339–351, 2013.

  8. X. Song, H. He, S. Niu, et al., A data streams analysis strategy based on Hoeffding tree with concept drift on Hadoop system. In International Conference on Advanced Cloud & Big Data, IEEE, New York, 2017.

  9. Z. Chen, J. Guo, and Q. Liu. DBSCAN algorithm clustering for massive AIS data based on the Hadoop platform. In 2017 International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration (ICIICII), IEEE Computer Society, Washington, D.C., 2017.

  10. S. Li, D. Shen, Y. Kou, et al., Query optimization for massive RDF data based on Spark. In IEEE 2018 4th International Conference on Big Data Computing and Communications (BIGCOM)—Chicago, IL (2018.8.7-2018.8.9)] 2018 4th International Conference on Big Data Computing and Communications (BIGCOM), pp. 219–224, 2018.

  11. W. Huang, L. Meng, D. Zhang, et al., In-memory parallel processing of massive remotely sensed data using an Apache Spark on Hadoop YARN model, IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, Vol. 10, No. 1, pp. 3–19, 2017.

    Article  Google Scholar 

  12. L. Zhu, and Y. Li. Distributed storage and analysis of massive urban road traffic flow data based on Hadoop. In Web Information System and Application Conference, IEEE, New York, pp. 75–78, 2016.

  13. Z. Chen, J. Guo, and Q. Liu. DBSCAN algorithm clustering for massive AIS data based on the Hadoop platform. In International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration, IEEE Computer Society, Washington, D.C., pp. 25–28, 2017.

  14. M. A. Li, L. I. Shu-Gang, P. Xiao, et al., Massive data storing technique of coal mine emergency management in cloud computing, Journal of Xian University of Science & Technology, Vol. 34, No. 05, pp. 596–601, 2014.

    Google Scholar 

  15. M. M. Rathore, A. Ahmad, and A. Paul. The Internet of Things based medical emergency management using Hadoop ecosystem. In Sensors, IEEE, New York, pp. 1–4, 2015.

  16. L. Sun, M. Hu, Q. Meng, et al., The solution for performance improvement of electric distribution network line loss based on Hadoop big data technology. In International Conference on Computer Science and Network Technology, IEEE, New York, pp. 500–506, 2016.

  17. B. Ma, A new kind of parallel K_NN network public opinion classification algorithm based on Hadoop platform, Applied Mechanics & Materials, Vol. 644–650, pp. 2018–2021, 2014.

    Article  Google Scholar 

  18. S. Zhang, J. Liu, Z. M. Lei, et al., Characterizing and modeling microblog traffic in cellular data network based on massive data analysis, Advanced Materials Research, Vol. 926–930, No. 926–930, pp. 2781–2785, 2014.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, Hunan Women’s University, Changsha, 410004, Hunan, China

    Yanxin Zhang

Authors
  1. Yanxin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanxin Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y. A Hadoop Processing Method for Massive Sensor Network Data Based on Internet of Things. Int J Wireless Inf Networks 27, 299–306 (2020). https://doi.org/10.1007/s10776-019-00455-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10776-019-00455-6

Keywords

Search

Navigation