Skip to main content
SpringerLink
Log in

Hydrological stream data pipeline framework based on IoTDB

  • SPECIAL ISSUE PAPER
  • Published:
Service Oriented Computing and Applications Aims and scope Submit manuscript

Abstract

With the increasing amount of hydrological data in Chuhe river basin, the traditional relational database has been unable to meet the needs of users, which not only makes it difficult to achieve low latency and high throughput in the real-time transmission of hydrological data, but also causes the phenomenon of long time or even system crash when querying large amount of annual water-level data. To solve this problem, this paper proposes a stream data pipeline framework based on timeseries databases IoTDB and Kafka, which can provide services for hydrological early warning and anomaly detection researchers. Based on the hydrological sensor data of Chuhe river, the processing scenarios of sensor stream data are set and compared with other NoSQL (HBase, MongoDB, RiakTS and Redis) in different scenarios. The performance and workload of different NoSQL in this data pipeline are tested. Finally, it is docked with Flink real-time stream data processing platform and compared with other data pipelines. The experimental results show that the stream data pipeline composed of IoTDB, Kafka and Flink is outstanding in data acquisition, transmission, incremental query and data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Tang E, Fan Y (2017) Performance comparison between five NoSQL databases. In: International conference on cloud computing & big data. IEEE

  2. Kang L, Deolalikar V, Pradhan N (2015) Big data gathering and mining pipelines for CRM using open-source. In: IEEE international conference on big data

  3. Raj P (2018) A detailed analysis of NoSQL and NewSQL databases for bigdata analytics and distributed computing. Adv Comput 109:1–48

    Article  Google Scholar 

  4. Lawlor B, Lynch R, Mac MA, Walsh P (2018) Field of genes: using apache kafka as a bioinformatic data repository. Gigascience 7(4):giy036

    Article  Google Scholar 

  5. Nazeer H, Iqbal W, Bokhari F (2017) Real-time text analytics pipeline using open-source big data tools. arXiv:1712.04344v1

  6. Freire SM, Teodoro D, Wei-Kleiner F, Sundvall E, Karlsson D, Lambrix P (2016) Comparing the performance of nosql approaches for managing archetype-based electronic health record data. PLoS ONE 11(3):e0150069

    Article  Google Scholar 

  7. Nguyen CN, Kim JS, Hwang S (2016) KOHA: building a kafka-based distributed queue system on the fly in a Hadoop cluster. Foundations and applications of self* systems. In: IEEE International Workshops on IEEE

  8. Yi M, Ting X, Shao-Bin L (2017) Research on NoSQL distributed big data mining method in complex attribute environment. Sci Technol Eng

  9. O’Donovan P, Leahy K, Bruton K (2015) An industrial big data pipeline for data-driven analytics maintenance applications in large-scale smart manufacturing facilities. J. Big Data 2(1):25

    Article  Google Scholar 

  10. Nallakaruppan MK, Kumaran US (2018) Quick fix for obstacles emerging in management recruitment measure using IOT-based candidate selection. Serv. Oriented Comput Appl 12(3–4):275–284

    Article  Google Scholar 

  11. Zhang Q, Li S, Li Z (2015) CHARM: a cost-efficient multi-cloud data hosting scheme with high availability. IEEE Trans Cloud Comput 3(3):1

    Article  Google Scholar 

  12. Al-Sakran A, Qattous H, Hijjawi M (2018) A proposed performance evaluation of NoSQL databases in the field of IoT. In: The 8th international conference on computer science and information technology (CSIT 2018). IEEE Computer Society

  13. Veloudis S, Paraskakis I, Petsos C (2017) Cloud service broker-age: enhancing resilience in virtual enterprises through service governance and quality assurance. Serv. Oriented Comput Appl 11(4):445–458

    Article  Google Scholar 

  14. Feng Y, Peng Z, Sheng G, Yong C (2019) Intelligent Chuhe system based on the new generation of big data processing engine Flink. Water Resour Prot 2:90–94

    Google Scholar 

  15. Reniers V, Rafique A, Van Landuyt D, Joosen W (2017) Object-nosql database mappers: a benchmark study on the performance overhead. J Internet Serv Appl 8(1):1

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by the 2018 Jiangsu Province Key Research and Development Program (Modern Agriculture) Project under Grant No. 20195013812, 2017 Jiangsu Province Postdoctoral Research Funding Project under Grant No. 1701020C, 2017 Six Talent Peaks Endorsement Project of Jiangsu under Grant No. XYDXX- 078, the Fundamental Research Funds for the Central Universities under Grant No. 2013B01814.

Author information

Authors and Affiliations

  1. College of Computer and Information, Hohai University, Nanjing, China

    YuanSheng Lou, Yu Qin & Feng Ye

  2. Jiangsu Water Resources Department, Nanjing, China

    Feng Ye & Yong Chen

  3. Postdoctoral Centre, Nanjing Longyuan Micro-Electronic Company, Nanjing, China

    Peng Zhang

Authors
  1. YuanSheng Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feng Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Ye.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lou, Y., Qin, Y., Ye, F. et al. Hydrological stream data pipeline framework based on IoTDB. SOCA 13, 287–295 (2019). https://doi.org/10.1007/s11761-019-00267-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11761-019-00267-9

Keywords

Search

Navigation