Abstract
The Internet of Things (IoT) and Artificial Intelligence of Things (AIoT) are emerging as promising paradigms for enabling ubiquitous and intelligent applications across various domains. However, managing and utilizing the massive and heterogeneous data generated by IoT and AIoT devices poses significant challenges for traditional data management systems. In this paper, we present a new data management paradigm called IoT Lakehouse, which aims to integrate the best practices of data warehouse and data lake to provide a unified, scalable and efficient platform for IoT and AIoT data. We define the concept and characteristics of IoT Lakehouse, and compare it with other existing data management paradigms. We present a refercence architecture and key technologies of IoT Lakehouse, and discuss how it supports various needs and scenarios of AIoT. We also analyze the main challenges and future directions of IoT Lakehouse research and development.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Artificial Intelligence in IoT Market by Component. https://www.marketdataforecast.com/market-reports/artificial-intelligence-in-iot-market. Accessed 7 May 2023
Delta: Build Lakehouses with Delta Lake. https://delta.io/. Accessed 7 May 2023
Hudia:a transactional data lake platform. https://hudi.apache.org/. Accessed 7 May 2023
Iceberg: The open table format for analytic datasets. https://iceberg.apache.org/. Accessed 7 May 2023
What is a Data Lakehouse? - Databricks. https://www.databricks.com/glossary/data-lakehouse. Accessed 8 May 2023
What is the Databricks Lakehouse? - Azure Databricks. https://learn.microsoft.com/en-us/azure/databricks/lakehouse/. Accessed 8 May 2023
Number of IoT connected devices worldwide 2019–2030. https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide/. Accessed 7 May 2023
Akidau, T., et al.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endow. 8(12), 1792–1803 (2015). https://doi.org/10.14778/2824032.2824076
Asad, U., Mohammed, A.S.: Deep learning and industrial internet of things to improve smart city safety. In: 2023 International Conference on Business Analytics for Technology and Security (ICBATS), pp. 1–10 (2023). https://doi.org/10.1109/ICBATS57792.2023.10111164
Azevedo, R., Silva, J.P., Lopes, N., Curado, A., Nunes, L.J., Lopes, S.I.: Designing an IoT-enabled data warehouse for indoor radon time series analytics. In: 2022 17th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6 (2022). https://doi.org/10.23919/CISTI54924.2022.9820540
Bubeck, S., et al.: Sparks of artificial general intelligence: Early experiments with gpt-4 (2023), https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/
Cunha, B., Sousa, C.: On the definition of intelligible IIoT architectures. In: 2021 16th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6 (2021). https://doi.org/10.23919/CISTI52073.2021.9476342
Dehghani, Z.: Data Mesh. O’Reilly Media (2022). https://books.google.de/books?id=jmZjEAAAQBAJ
Gallas, E.J., Malon, D., Hawkings, R.J., Albrand, S., Torrence, E.: An integrated overview of metadata in atlas (2010)
Harby, A.A., Zulkernine, F.: From data warehouse to lakehouse: A comparative review. In: 2022 IEEE International Conference on Big Data (Big Data), pp. 389–395 (2022). https://doi.org/10.1109/BigData55660.2022.10020719
Li, H.: Alluxio: A Virtual Distributed File System. Ph.D. thesis, EECS Department, University of California, Berkeley (2018). https://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-29.html
Macagnano, D., Destino, G., Abreu, G.: Indoor positioning: a key enabling technology for IoT applications. In: 2014 IEEE World Forum on Internet of Things (WF-IoT), pp. 117–118 (2014). https://doi.org/10.1109/WF-IoT.2014.6803131
Sethi, R., et al.: Presto: SQL on everything. In: 2019 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1802–1813 (2019). https://doi.org/10.1109/ICDE.2019.00196
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. p. 2. NSDI’12, USENIX Association, USA (2012)
Zaharia, M., Ghodsi, A., Xin, R., Armbrust, M.: Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. In: 11th Conference on Innovative Data Systems Research, CIDR 2021, Virtual Event, 11–15 January 2021, Online Proceedings. https://www.cidrdb.org (2021), https://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, G. et al. (2023). IoT Lakehouse: A New Data Management Paradigm for AIoT. In: Zhang, S., Hu, B., Zhang, LJ. (eds) Big Data – BigData 2023. BigData 2023. Lecture Notes in Computer Science, vol 14203. Springer, Cham. https://doi.org/10.1007/978-3-031-44725-9_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-44725-9_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44724-2
Online ISBN: 978-3-031-44725-9
eBook Packages: Computer ScienceComputer Science (R0)