Skip to main content
SpringerLink
Log in

Multi-level just-enough elasticity for MQTT brokers of Internet of Things applications

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Applications for the Internet of Things (IoT) are rapidly having an impact on all areas of daily life. Every day, its embedded devices generate loads of data that requires efficient network infrastructure. The integration of lightweight communication protocols such as Message Queuing Telemetry Transport (MQTT) is to send millions of IoT messages back and forth with as few errors as possible. In practice, IoT big data analytic systems are often deployed with highly regarded MQTT solutions to handle huge amounts of dynamic data and achieve scalability. However, these solutions do not adapt well to fluctuations in workload, so they are not elastic yet. This article introduces a novel framework that provides just-enough elasticity for MQTT brokers with multiple levels of virtualization and its implementation using EMQX MQTT broker, Kubernetes container-orchestration system and OpenStack cloud environment. Various experiments based on a real life IoT application are conducted to validate our proposed framework and its elastic functionality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availibility

The data used to support the findings of this study are available from the corresponding author upon request.

Notes

  1. https://github.com/fimocode/cluster_controller.

  2. https://www.emqx.io/docs/en/v3.0/tune.html.

References

  1. Sharma, N., Panwar, D.: Green IoT: Advancements and Sustainability with Environment by 2050. In: 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, pp. 1127–1132 (2020)

  2. Turner, V., Reinsel, D., Gantz, J.F., Minton, S.: The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things. IDC Report (2014)

  3. Message Queuing Telemetry Transport. http://mqtt.org. Accessed 30 April 2021

  4. Mell, P., Grance, T.: The NIST definition of cloud computing. In: NIST special publication, vol. 800, pp. 145 (2011)

  5. Eugster, P.Th., Felber, P.A., Guerraoui, R., Kermarrec, A.: The many faces of publish/subscribe. In: ACM Computing. Survey. 35, 2, 114-131 (2003)

  6. EMQX Broker. https://docs.emqx.io/broker/latest/en/. Accessed 30 April 2021

  7. Kawaguchi, R., Bandai, M.: Edge Based MQTT Broker Architecture for Geographical IoT Applications. In: International Conference on Information Networking (ICOIN), Barcelona, Spain, pp. 232-235 (2020)

  8. Gupta, V., Khera, S., Turk, N.: MQTT protocol employing IOT based home safety system with ABE encryption. In: Multimedia Tools and Applications (2020)

  9. Mukambikeshwari, A. Poojary: Smart Watering System Using MQTT Protocol in IoT. In: Advances in Artificial Intelligence and Data Engineering. Advances in Intelligent Systems and Computing, Vol. 1133. Springer, Singapore (2020)

  10. See, Y.C., Ho, E.X.: IoT-Based Fire Safety System Using MQTT Communication Protocol. In: IJIE, Vol. 12(6), pp. 207–215 (2020)

  11. Nazir, S., Kaleem, M.: Reliable Image Notifications for Smart Home Security with MQTT. In: International Conference on Information Science and Communication Technology (ICISCT), Karachi, Pakistan, pp. 1–5 (2019)

  12. Alqinsi, P., Edward, I.J.M., Ismail, N., Darmalaksana, W.: IoT-Based UPS Monitoring System Using MQTT Protocols. In: 4th International Conference on Wireless and Telematics (ICWT), Nusa Dua, pp. 1–5 (2018)

  13. Comparison of MQTT Brokers. https://tewarid.github.io/2019/03/21/comparison-of-mqtt-brokers.html. Accessed 30 April 2021

  14. Schmitt, A., Carlier, F., Renault, V.: Data Exchange with the MQTT Protocol: Dynamic Bridge Approach. In: IEEE 89th Vehicular Technology Conference (VTC2019-Spring), Kuala Lumpur, Malaysia, pp. 1–5 (2019)

  15. Zambrano A.M.V., Zambrano M.V., Mejía, E.L.O., Calderón X.H.: SIGPRO: A Real-Time Progressive Notification System Using MQTT Bridges and Topic Hierarchy for Rapid Location of Missing Persons. In: IEEE Access, Vol. 8, pp. 149190–149198 (2020)

  16. The features that various MQTT servers (brokers) support. https://github.com/mqtt/mqtt.github.io/wiki/server-support. Accessed 30 April 2021

  17. Jutadhamakorn, P., Pillavas, T., Visoottiviseth, V., Takano, R., Haga, J., Kobayashi, D.: A scalable and low-cost MQTT broker clustering system. In: 2nd International Conference on Information Technology (INCIT), Nakhonpathom, pp. 1-5 (2017)

  18. Thean, Z. Y., Voon Yap, V., Teh, P. C.: Container-based MQTT Broker Cluster for Edge Computing. In: 4th International Conference and Workshops on Recent Advances and Innovations in Engineering (ICRAIE), Kedah, Malaysia, pp. 1–6 (2019)

  19. Detti, A., Funari, L., Blefari-Melazzi, N.: Sub-linear scalability of MQTT clusters in topic-based publish-subscribe applications. IEEE Trans. Network Serv. Manag. 17(3), 1954–1968 (2020)

    Article  Google Scholar 

  20. Ullah, A., Li, J., Hussain, A.: Design and evaluation of a biologically-inspired cloud elasticity framework. Clust. Comput. 23(4), 3095–3117 (2020)

    Article  Google Scholar 

  21. Righi, R.R., Correa, E., Gomes, M.M., Costa, C.A.: Enhancing performance of IoT applications with load prediction and cloud elasticity. Future Gener. Comput. Syst. 109, 689–701 (2020)

    Article  Google Scholar 

  22. Fourati, M. H., Marzouk, S., Drira, K., Jmaiel, M.: DOCKERANALYZER : Towards Fine Grained Resource Elasticity for Microservices-Based Applications Deployed with Docker. In: 20th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), Gold Coast, Australia, pp. 220–225 (2019)

  23. Nardelli, M., Cardellini, V., Casalicchio, E.: Multi-Level Elastic Deployment of Containerized Applications in Geo-Distributed Environments. In: IEEE 6th International Conference on Future Internet of Things and Cloud (FiCloud), Barcelona, pp. 1–8 (2018)

  24. Rodrigues, V.F., Wendt, I.G., Righi, R.R., Costa, C.A., Barbosa, J.L.V., Alberti, A.M.: Brokel: Towards enabling multi-level cloud elasticity on publish/subscribe brokers. Int. J. Distrib. Sens. Networks (2017). https://doi.org/10.1177/1550147717728863

    Article  Google Scholar 

  25. Vavassori, S., Soriano, J., Fernández, R.: Enabling large-scale IoT-based services through elastic publish/subscribe. Sensors 17, 2148 (2017)

    Article  Google Scholar 

  26. Yadav, M.P., Yadav, D.K.: Maintaining container sustainability through machine learning. Clust. Comput. 24(4), 3725–3750 (2021)

    Article  Google Scholar 

  27. Dawoud W., Takouna I., Meinel C.: Elastic Virtual Machine for Fine-Grained Cloud Resource Provisioning. In: Krishna P.V., Babu M.R., Ariwa E. (eds) Global Trends in Computing and Communication Systems. ObCom 2011. Communications in Computer and Information Science, vol 269. Springer, Berlin (2012)

  28. Nardelli M., Russo Russo G., Cardellini V., Lo Presti F.: A multi-level elasticity framework for distributed data stream processing. In: Mencagli, G., et al. (Eds.), Euro-Par 2018: Parallel Processing Workshops. Euro-Par 2018. Lecture Notes in Computer Science, Vol. 11339. Springer, Cham (2019)

  29. Russo Russo, G., Nardelli, M., Cardellini, V., Lo Presti, F.: Multi-level elasticity for wide-area data streaming systems: a reinforcement learning approach. Algorithms 11(9), 134 (2018)

    Article  Google Scholar 

  30. Linh Manh Pham, Truong-Thang Nguyen, Tien-Quang Hoang, “Towards an Elastic Fog-Computing Framework for IoT Big Data Analytics Applications”, Wireless Communications and Mobile Computing, vol. 2021, Article ID 3833644, 16 pages, 2021

  31. Roure, D., Goble, C.: Software design for empowering scientists. IEEE Softw. 26(01), 88–95 (2009)

    Article  Google Scholar 

  32. A distributed, reliable key-value store. https://etcd.io/docs/v3.4.0/. Accessed 30 April 2021

  33. Kubernetes. https://kubernetes.io/. Accessed 30 April 2021

  34. HAProxy. https://www.haproxy.com/solutions/load-balancing/. Accessed 30 April 2021

  35. OpenStack: Open Source Cloud Computing Infrastructure. https://www.openstack.org/. Accessed 30 April 2021

  36. OpenStack Heat. https://docs.openstack.org/heat/latest/. Accessed 30 April 2021

  37. OpenStack Ceilometer. https://docs.openstack.org/ceilo-meter/latest/. Accessed 30 April 2021

  38. OpenStack Aodh. https://docs.openstack.org/aodh/latest/. Accessed 30 April 2021

  39. Gnocchi - Metric as a Service. https://gnocchi.xyz/. Accessed 30 April 2021

  40. RabbitMQ. https://www.rabbitmq.com/. Accessed 30 April 2021

  41. Apache Jmeter. https://jmeter.apache.org/. Accessed 30 April 2021

  42. Pham, L.M., Nguyen, T.T., Tran, M.D.: A benchmarking tool for elastic MQTT brokers in IoT applications. Int. J. Inf. Commun. Sci. 4(4), 59–67 (2019)

    Google Scholar 

  43. Prometheus. https://prometheus.io/. Accessed 30 April 2021

  44. Grafana. https://grafana.com/. Accessed 30 April 2021

Download references

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

  1. VNU University of Engineering and Technology, 144 Xuan Thuy, Cau Giay, Hanoi, Vietnam

    Linh Manh Pham & Xuan-Truong Nguyen

  2. Thuyloi University, 175 Tay Son, Dong Da, Hanoi, Vietnam

    Nguyen-Tuan-Thanh Le

  3. Hanoi Pedagogical University 2, 32 Nguyen Van Linh, Xuan Hoa, Phuc Yen, Vinh Phuc, Vietnam

    Xuan-Truong Nguyen

Authors
  1. Linh Manh Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nguyen-Tuan-Thanh Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuan-Truong Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by LMP, N-T-TL and X-TN. The first draft of the manuscript was written by LMP and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Linh Manh Pham.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pham, L.M., Le, NTT. & Nguyen, XT. Multi-level just-enough elasticity for MQTT brokers of Internet of Things applications. Cluster Comput 25, 3961–3976 (2022). https://doi.org/10.1007/s10586-022-03636-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-022-03636-w

Keywords

Search

Navigation