Skip to main content
SpringerLink
Log in
Book cover

International Conference on Cloud Computing and Services Science

CLOSER 2019: Cloud Computing and Services Science pp 150–176Cite as

A Containerized Edge Cloud Architecture for Data Stream Processing

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1218))

Abstract

Internet of Things (IoT) devices produce large volumes of data, which creates challenges for the supporting, often centralised cloud infrastructure that needs to process and store the data. We consider here an alternative, more centralised approach, based on the edge cloud computing model. Here, filtering and processing of data happens locally before transferring it to a central cloud infrastructure. In our work, we use a low-power and low-cost cluster of single board computers (SBC) to apply common models and technologies from the big data domain. The benefit is reducing the volume of data that is transferred.

We implement the system using a cluster of Raspberry Pis and Docker to containerize and deploy an Apache Hadoop and Apache Spark data streaming processing cluster. We evaluate the performance, but of trust support of the system, showing that by using containerization increased fault tolerance and ease of maintenance can be achieved. The analysis of the performance takes into account the resource usage of the proposed solution with regards to the constraints imposed by the devices. Our trust management solution relies on blockchain technologies.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.carberry.it.

  2. 2.

    http://www.instructables.com/id/OBD-Pi/.

  3. 3.

    https://www.pololu.com/product/2506.

  4. 4.

    Docker Engine, https://docs.docker.com/engine.

  5. 5.

    Docker Swarm, https://docs.docker.com/engine/swarm/.

  6. 6.

    Nodejs, https://nodejs.org.

  7. 7.

    Veracity Global Camswitch 8, http://www.veracityglobal.com/products/networked-video-integration-devices/camswitch-mobile.aspx.

  8. 8.

    Hypriot OS, https://blog.hypriot.com/about/.

  9. 9.

    Ansible, https://www.ansible.com/.

  10. 10.

    Prometheus exporters: Armexporter, Node exporter, cAdvisor.

  11. 11.

    Spark sample applications, https://github.com/apache/spark/tree/master/examples.

  12. 12.

    Prometheus, https://prometheus.io/.

  13. 13.

    Google cAdvisor, https://github.com/google/cadvisor.

  14. 14.

    Node exporter, https://github.com/prometheus/ node_exporter.

  15. 15.

    Grafana, https://grafana.com/.

  16. 16.

    Gitlab, https://about.gitlab.com/.

  17. 17.

    QEMU, https://www.qemu.org/.

  18. 18.

    The model used is a HP 355 G2, with AMD A8-6410 2 GHz CPU and 12 GB memory.

  19. 19.

    https://www.jeffgeerling.com/blog/2018/raspberry-pi-3-b-review-and-performance-comparison.

References

  1. Scolati, R., Fronza, I., El Ioini, N., Samir, A., Pahl, C.: A containerized big data streaming architecture for edge cloud computing on clustered single-board devices. In: International Conference on Cloud Computing and Services Science (2019)

    Google Scholar 

  2. Ambroz, M., Hudomalj, U., Marinsek, A., Kamnik, R.: Raspberry pi-based low-cost connected device for assessing road surface friction. Electronics 8(3), 341 (2019)

    Google Scholar 

  3. Apache: Hadoop (2019). https://hadoop.apache.org. Accessed June 2019

  4. Apache: Spark (2019). https://spark.apache.org. Accessed June 2019

  5. Arabnejad, H., Pahl, C., Jamshidi, P., Estrada, G.: A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling. In: 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (2017)

    Google Scholar 

  6. Baldeschwieler, E.: Yahoo! launches world’s largest hadoop production application (2018). http://yahoohadoop.tumblr.com/post/98098649696/ yahoo-launches-worlds-largest-hadoop-production. Accessed September 2018

  7. Baumgartl, R., Muller, D.: Raspberry pi as an inexpensive platform for real-time traffic jam analysis on the road. In: Federated Conference on Computer Science and Information Systems (2018)

    Google Scholar 

  8. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: OSDI 2004 Symposium on Operating System Design and Implementation (2004)

    Google Scholar 

  9. Docker (2018). https://docs.docker.com/. Accessed September 2018

  10. El Ioini, N., Pahl, C.: A review of distributed ledger technologies. In: Panetto, H., Debruyne, C., Proper, H.A., Ardagna, C.A., Roman, D., Meersman, R. (eds.) OTM 2018. LNCS, vol. 11230, pp. 277–288. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-02671-4_16

    Chapter  Google Scholar 

  11. El Ioini, N., Pahl, C.: Trustworthy orchestration of container based edge computing using permissioned blockchain. In: Fifth International Conference on Internet of Things: Systems, Management and Security (IoTSMS) (2018)

    Google Scholar 

  12. Femminella, M., Pergolesi, M., Reali, G.: Performance evaluation of edge cloud computing system for big data applications. In: 5th IEEE International Conference on Cloud Networking (Cloudnet), pp. 170–175 (2016)

    Google Scholar 

  13. Fowley, F., Pahl, C., Jamshidi, P., Fang, D., Liu, X.: A classification and comparison framework for cloud service brokerage architectures. IEEE Trans. Cloud Comput. 6(2), 358–371 (2018)

    Article  Google Scholar 

  14. Gillam, L., Katsaros, K., Dianati, M., Mouzakitis, A.: Exploring edges for connected and autonomous driving. In: Conference on Computer Communications WS (2018)

    Google Scholar 

  15. Haramaki, T., Nishino, H.: A safe driving support system based on distributed cooperative edge computing. In: International Conference on Consumer Electronics (2018)

    Google Scholar 

  16. Heinrich, R., et al.: Performance engineering for microservices: research challenges and directions. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion (2017)

    Google Scholar 

  17. Jamshidi, P., Sharifloo, A., Pahl, C., Arabnejad, A., Metzger, A., Estrada, G.: Fuzzy self-learning controllers for elasticity management in dynamic cloud architectures. In: 12th International ACM Conference on Quality of Software Architectures (2016)

    Google Scholar 

  18. Jamshidi, P., Sharifloo, A., Pahl, C., Metzger, A., Estrada, G.: Self-learning cloud controllers: Fuzzy q-learning for knowledge evolution. arXiv preprint arXiv:1507.00567 (2015)

  19. Jamshidi, P., Pahl, C., Mendonca, N.C., Lewis, J., Tilkov, S.: Microservices: the journey so far and challenges ahead. IEEE Softw. 35(3), 24–35 (2018)

    Article  Google Scholar 

  20. Jamshidi, P., Pahl, C., Mendonca, N.C.: Managing uncertainty in autonomic cloud elasticity controllers. IEEE Cloud Comput. 3, 50–60 (2016)

    Google Scholar 

  21. Jamshidi, P., Pahl, C., Chinenyeze, S., Liu, X.: Cloud migration patterns: a multi-cloud service architecture perspective. In: Toumani, F., et al. (eds.) ICSOC 2014. LNCS, vol. 8954, pp. 6–19. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22885-3_2

    Chapter  Google Scholar 

  22. Jamshidi, P., Pahl, C., Mendonca, N.C.: Pattern-based multi-cloud architecture migration. Softw.: Practice Experience 47(9), 1159–1184 (2017)

    Google Scholar 

  23. Javed, M., Abgaz, Y.M., Pahl, C.: Ontology change management and identification of change patterns. J. Data Semant. 2(2–3), 119–143 (2013)

    Article  Google Scholar 

  24. Johnston, S.J., et al.: Commodity single board computer clusters and their applications. Future Gen. Comput. Syst. 89, 201–212 (2018)

    Article  Google Scholar 

  25. Hentschel, K., Jacob, D., Singer, J., Chalmers, M.: Supersensors: Raspberry pi devices for smart campus infrastructure. In: IEEE International Conference on Future Internet of Things and Cloud (FiCloud) (2016)

    Google Scholar 

  26. Morabito, R.: A performance evaluation of container technologies on internet of things devices. In: IEEE Conference on Computer Communications Workshops (2016)

    Google Scholar 

  27. Morabito, R., Farris, I., Iera, A., Taleb, T.: Evaluating performance of containerized iot services for clustered devices at the network edge. IEEE Internet Things J. 4(4), 1019–1030 (2017)

    Article  Google Scholar 

  28. Morabito, R., Petrolo, R., Loscri, V., Mitton, N., Ruggeri, G., Molinaro, A.: Lightweight virtualization as enabling technology for future smart cars. In: Symposium on Integrated Network and Service Management, pp. 1238–1245. IEEE (2017)

    Google Scholar 

  29. Naik, N.: Docker container-based big data processing system in multiple clouds for everyone. In: International Systems Engineering Symposium (ISSE), pp. 1–7 (2017)

    Google Scholar 

  30. Pahl, C., Lee, B.: Containers and clusters for edge cloud architectures - a technology review. In: IEEE International Conference on Future Internet of Things and Cloud (2015)

    Google Scholar 

  31. Pahl, C., El Ioini, N., Helmer, S., Lee, B.: An architecture pattern for trusted orchestration in IoT edge clouds. In: International Conference Fog and Mobile Edge Computing (2018)

    Google Scholar 

  32. Pahl, C., Jamshidi, P., Zimmermann, O.: Architectural principles for cloud software. ACM Trans. Internet Technol. (TOIT) 18(2), 17 (2018)

    Article  Google Scholar 

  33. Pahl, C., Helmer, S., Miori, L., Sanin, J., Lee, B.: A container-based edge cloud PaaS architecture based on raspberry pi clusters. In: IEEE International Conference on Future Internet of Things and Cloud Workshops (2016)

    Google Scholar 

  34. Pahl, C.: An ontology for software component matching. In: Pezzè, M. (ed.) FASE 2003. LNCS, vol. 2621, pp. 6–21. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36578-8_2

    Chapter  MATH  Google Scholar 

  35. Pahl, C., Brogi, A., Soldani, J., Jamshidi, P.: Cloud container technologies: a state-of-the-art review. IEEE Trans. Cloud Comput. 7, 677–692 (2017)

    Google Scholar 

  36. Renner, T., Meldau, M., Kliem, A.: Towards container-based resource management for the internet of things. In: International Conference on Software Networking (2016)

    Google Scholar 

  37. Renner, M.: Testing high availability of docker swarm on a raspberry pi cluster. https://blog.hypriot.com/post/high-availability-with-docker. Accessed 09 2018

  38. Renner, M.: Evaluation of high availability performance of kubernetes and docker swarm on a raspberry pi cluster. In: Highload++ Conference (2016)

    Google Scholar 

  39. Raspberry Pi Foundation (2018). https://www.raspberrypi.org/products/raspberry-p i-2-model-b/. Accessed Sept 2018

  40. Stager, A., Bhan, L., Malikopoulos, A., Zhao, L.: A scaled smart city for experimental validation of connected and automated vehicles. IFAC 51(9), 130–135 (2018)

    Google Scholar 

  41. Stevens, A., et al.: Cooperative automation through the cloud: the CARMA project. In: Proceedings of 12th ITS European Congress (2017)

    Google Scholar 

  42. Taibi, D., Lenarduzzi, V., Pahl, C.: Architectural patterns for microservices: a systematic mapping study. In: International Conference, Cloud Computing and Services Science (2018)

    Google Scholar 

  43. Tso, F.P., White, D.R., Jouet, S., Singer, J., Pezaros, D.P.: The glasgow raspberry pi cloud: a scale model for cloud computing infrastructures. In: IEEE 33rd International Conference on Distributed Computing Systems Workshops (2013)

    Google Scholar 

  44. Turner, V.: The digital universe of opportunities: rich data and the increasing value of the internet of things. IDC Report (2014)

    Google Scholar 

  45. Vilalta, R., et al.: Control and management of a connected car using YANG/RESTCONF and cloud computing. In: International Conference on the Network of the Future (2017)

    Google Scholar 

  46. von Leon, D., Miori, L., Sanin, J., El Ioini, N., Helmer, S., Pahl, C.: A performance exploration of architectural options for a middleware for decentralised lightweight edge cloud architectures. In: International Conference Internet of Things, Big Data & Security (2018)

    Google Scholar 

  47. von Leon, D., Miori, L., Sanin, J., El Ioini, N., Helmer, S., Pahl, C.: A lightweight container middleware for edge cloud architectures. In: Fog and Edge Computing: Principles and Paradigms, pp. 145–170. Wiley (2019)

    Google Scholar 

  48. Wang, Y., Goldstone, R., Yu, W., Wang, T.: Characterization and optimization of memory-resident mapreduce on HPC systems. In: IEEE 28th International Parallel and Distributed Processing Symposium (2014)

    Google Scholar 

Download references

Acknowledgements

This work has received funding from the EU’s Horizon 2020 research and innovation programme under grant agreement 825012 - Project 5G-CARMEN.

Author information

Authors and Affiliations

  1. Free University of Bozen-Bolzano, Bolzano, Italy

    Remo Scolati, Ilenia Fronza, Nabil El Ioini, Areeg Samir, Hamid Reza Barzegar & Claus Pahl

Authors
  1. Remo Scolati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilenia Fronza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nabil El Ioini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Areeg Samir
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hamid Reza Barzegar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Claus Pahl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claus Pahl .

Editor information

Editors and Affiliations

  1. Columbia University, New York, NY, USA

    Donald Ferguson

  2. Escola d’Enginyeria, Bellaterra, Barcelona, Spain

    Víctor Méndez Muñoz

  3. Free University of Bozen-Bolzano, Bolzano, Bolzano, Italy

    Claus Pahl

  4. Dublin City University, Dublin, Ireland

    Markus Helfert

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Scolati, R., Fronza, I., El Ioini, N., Samir, A., Barzegar, H.R., Pahl, C. (2020). A Containerized Edge Cloud Architecture for Data Stream Processing. In: Ferguson, D., Méndez Muñoz, V., Pahl, C., Helfert, M. (eds) Cloud Computing and Services Science. CLOSER 2019. Communications in Computer and Information Science, vol 1218. Springer, Cham. https://doi.org/10.1007/978-3-030-49432-2_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49432-2_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49431-5

  • Online ISBN: 978-3-030-49432-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation