Abstract
The scale of meteorological sensor data increases at TB level every week. Traditional relational database is inefficient in storing and processing such data and cannot satisfy many soft requirements. However, the heterogeneity and diversity of the numerous existing NoSQL systems impede the well-informed comparison and selection of a data store appropriate for a given application context. Implementing a meteorological sensor data storage mechanism is a key challenge. Therefore, a meteorological sensor data storage mechanism based on TimescaleDB and Kafka is proposed. In this solution, meteorological sensor data was acquired and transmitted by Kafka and was sent to TimescaleDB for storage and analysis. Based on simulated meteorological sensor dataset, it compared the solution with other NoSQL stores such as Redis, MongoDB, Cassandra, HBase and Riak TS. The experimental results show that the storage mechanism proposed is superior in the storage and processing of massive meteorological sensor data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Wamba, S.F., Akter, S., Edwards, A.: How ‘big data’ can make big impact: findings from a systematic review and a longitudinal case study. Int. J. Prod. Econ. 165, 234–236 (2015)
Yang, M., Yang, H., Chen, Q., Xiao, Y., Gao, Z., Zeng, Y.: Meteorological data cloud data storage technology and application. Meteorol. Sci. Technol. 45(06), 1017–1021 (2017)
Wang, R., Huang, X., Zhang, B., Wang, J., Luo, B.: Design and implementation of real-time analysis and storage system for massive meteorological data. Comput. Eng. Sci. 37(11), 2045–2054 (2015)
Redis. https://redis.io/. Accessed 22 Jan 2019
Jiang, H., Shen, F., Chen, S., Li, K.C., Jeong, Y.S.: A secure and scalable storage system for aggregate data in IoT. Future Gener. Comput. Syst. 49, 133–141 (2015)
Apache HBase. http://hbase.apache.org/. Accessed 22 Jan 2019
Apache Cassandra. http://cassandra.apache.org/. Accessed 22 Jan 2019
Riak TS. http://basho.com/products/riak-ts/. Accessed 22 Jan 2019
Teng, S., et al.: A cooperative multi-classifier method for local area meteorological data mining. In: Proceedings of the 2014 IEEE 18th International Conference on Computer Supported [::Cooperative::] Work in Design (CSCWD), pp. 435–440
Shao, L., Liu, J., Dong, G., Mu, Y., Guo, P.: The establishment and data mining of meteorological data warehouse. In: 2014 IEEE International Conference on Mechatronics and Automation, pp. 2049–2052 (2014)
Jiang, X., Chen, W., Wang, Y.: The adaptive research of data layout in large-scale meteorological data storage system. In: 2013 IEEE Third International Conference on Information Science and Technology (ICIST), pp. 1016–1020 (2013)
Xu, X., Yang, Z., Ma, T.: Query optimization of meteorological structured data based on HBase. Comput. Eng. Appl. (9), 80–84 (2017)
Wang, L.P., Munoz Lopez, C., Homg, T.C., et al.: A convective rain cell database based upon high-resolution radar images: unravelling convection patterns. In: EGU General Assembly Conference Abstracts, vol. 20, p. 362 (2018)
Chandra, G.: Deka: BASE analysis of NoSQL database. Future Gener. Comput. Syst. 52, 13–21 (2015)
TimescaleDB. https://docs.timescale.com/v1.1/introduction. Accessed 28 Jan 2019
Apache Kafka. http://kafka.apache.org/. Accessed 28 Jan 2019
Acknowledgement
This work is partly supported by the 2018 Jiangsu Province Key Research and Development Program (Modern Agriculture) Project under Grant No. BE2018301, 2017 Jiangsu Province Postdoctoral Research Funding Project under Grant No. 1701020C, 2017 Six Talent Peaks Endorsement Project of Jiangsu under Grant No. XYDXX-078, the Fundamental Research Funds for the Central Universities under Grant No. 2013B01814.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Shen, L., Lou, Y., Chen, Y., Lu, M., Ye, F. (2019). Meteorological Sensor Data Storage Mechanism Based on TimescaleDB and Kafka. In: Cheng, X., Jing, W., Song, X., Lu, Z. (eds) Data Science. ICPCSEE 2019. Communications in Computer and Information Science, vol 1058. Springer, Singapore. https://doi.org/10.1007/978-981-15-0118-0_11
Download citation
DOI: https://doi.org/10.1007/978-981-15-0118-0_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-0117-3
Online ISBN: 978-981-15-0118-0
eBook Packages: Computer ScienceComputer Science (R0)