skip to main content
10.1145/3628034.3628043acmotherconferencesArticle/Chapter ViewAbstractPublication PageseuroplopConference Proceedingsconference-collections
research-article

Joint Learning: A Pattern for Reliable and Efficient Decision-Making in Self-Adaptive Internet of Things

Published: 05 February 2024 Publication History

Abstract

An Internet-of-Things (IoT) system typically comprises many small computing elements (nodes) that are battery-powered and communicate over a wireless network. These elements monitor properties in the environment and send the data to client applications via gateways. The wireless networks used by the elements are subject to uncertainties that are difficult to predict upfront, such as dynamic objects (swaying trees, cars,...) and changing weather conditions that may deteriorate the transmissions. To ensure reliable communication over a wireless network of energy-constrained elements, recent research has proposed self-adaptive IoT systems. Such a self-adaptive system equips the network of elements – referred to as the managed system – with a feedback loop – the managing system. The managing system monitors the changing conditions and adapts the transmission settings of the IoT network to ensure the system’s quality goals. Leveraging and consolidating the existing knowledge in this area, we present a pattern that we coined Joint Learning that provides a solution to the decision-making problem of large, distributed self-adaptive IoT systems. With this pattern, elements use a joint learner to make adaptation decisions for individual elements while yielding reliable communication of the overall network. The pattern is applied to two cases to show that the solutions realize the system goals while operating under uncertainties.

References

[1]
Sawsan AbdulRahman, Hanine Tout, Hakima Ould-Slimane, Azzam Mourad, Chamseddine Talhi, and Mohsen Guizani. 2020. A survey on federated learning: The journey from centralized to distributed on-site learning and beyond. IEEE Internet of Things Journal 8, 7 (2020), 5476–5497.
[2]
Ferran Adelantado, Xavier Vilajosana, Pere Tuset-Peiro, Borja Martinez, Joan Melia-Segui, and Thomas Watteyne. 2017. Understanding the limits of LoRaWAN. IEEE Communications magazine 55, 9 (2017), 34–40.
[3]
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. 2018. Emergent Complexity via Multi-Agent Competition. In International Conference on Learning Representations. https://openreview.net/forum?id=Sy0GnUxCb
[4]
Paolo Bellavista, Luca Foschini, and Alessio Mora. 2021. Decentralised learning in federated deployment environments: A system-level survey. ACM Computing Surveys (CSUR) 54, 1 (2021), 1–38.
[5]
Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, and Marc Tommasi. 2018. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 473–481.
[6]
Martin C Bor, Utz Roedig, Thiemo Voigt, and Juan M Alonso. 2016. Do LoRa low-power wide-area networks scale?. In Proceedings of the 19th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems. 59–67.
[7]
R. Calinescu, L. Grunske, M. Kwiatkowska, R. Mirandola, and G. Tamburrelli. 2011. Dynamic QoS Management and Optimization in Service-Based Systems. IEEE Transactions on Software Engineering 37, 3 (2011). https://doi.org/10.1109/TSE.2010.92
[8]
Javier Cámara, Gabriel A. Moreno, David Garlan, and Bradley Schmerl. 2016. Analyzing Latency-Aware Self-Adaptation Using Stochastic Games and Simulations. ACM Trans. Auton. Adapt. Syst. 10, 4, Article 23 (Jan. 2016), 28 pages. https://doi.org/10.1145/2774222
[9]
Betty H. C. Cheng, Rogério de Lemos, Holger Giese, Paola Inverardi, Jeff Magee, Jesper Andersson, Basil Becker, Nelly Bencomo, Yuriy Brun, Bojan Cukic, Giovanna Di Marzo Serugendo, Schahram Dustdar, Anthony Finkelstein, Cristina Gacek, Kurt Geihs, Vincenzo Grassi, Gabor Karsai, Holger M. Kienle, Jeff Kramer, Marin Litoiu, Sam Malek, Raffaela Mirandola, Hausi A. Müller, Sooyong Park, Mary Shaw, Matthias Tichy, Massimo Tivoli, Danny Weyns, and Jon Whittle. 2009. Software Engineering for Self-Adaptive Systems: A Research Roadmap. Springer Berlin Heidelberg, Berlin, Heidelberg, 1–26. https://doi.org/10.1007/978-3-642-02161-9_1
[10]
Li Da Xu, Wu He, and Shancang Li. 2014. Internet of things in industries: A survey. IEEE Transactions on industrial informatics 10, 4 (2014), 2233–2243.
[11]
Rogério de Lemos, Holger Giese, Hausi A. Müller, Mary Shaw, Jesper Andersson, Marin Litoiu, Bradley Schmerl, Gabriel Tamura, Norha M. Villegas, Thomas Vogel, Danny Weyns, Luciano Baresi, Basil Becker, Nelly Bencomo, Yuriy Brun, Bojan Cukic, Ron Desmarais, Schahram Dustdar, Gregor Engels, Kurt Geihs, Karl M. Göschka, Alessandra Gorla, Vincenzo Grassi, Paola Inverardi, Gabor Karsai, Jeff Kramer, Antónia Lopes, Jeff Magee, Sam Malek, Serge Mankovskii, Raffaela Mirandola, John Mylopoulos, Oscar Nierstrasz, Mauro Pezzè, Christian Prehofer, Wilhelm Schäfer, Rick Schlichting, Dennis B. Smith, João Pedro Sousa, Ladan Tahvildari, Kenny Wong, and Jochen Wuttke. 2013. Software Engineering for Self-Adaptive Systems: A Second Research Roadmap. Springer Berlin Heidelberg, Berlin, Heidelberg, 1–32. https://doi.org/10.1007/978-3-642-35813-5_1
[12]
Pamela J Derfus, Patrick G Maggitti, Curtis M Grimm, and Ken G Smith. 2008. The Red Queen effect: Competitive actions and firm performance. Academy of Management Journal 51, 1 (2008), 61–80.
[13]
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. 2020. A theoretical analysis of deep Q-learning. In Learning for Dynamics and Control. PMLR, 486–489.
[14]
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. 2016. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems 29 (2016).
[15]
David Garlan, Shang-Wen Cheng, An-Cheng Huang, Bradley Schmerl, and Peter Steenkiste. 2004. Rainbow: Architecture-Based Self-Adaptation with Reusable Infrastructure. Computer 37, 10 (oct 2004), 46–54. https://doi.org/10.1109/MC.2004.175
[16]
Les Gasser, Nicholas F Rouquette, Randall W Hill, and John Lieb. 1989. Representing and using organizational knowledge in distributed AI systems. In Distributed artificial intelligence. Elsevier, 55–78.
[17]
Orestis Georgiou and Usman Raza. 2017. Low power wide area network analysis: Can LoRa scale?IEEE Wireless Communications Letters 6, 2 (2017), 162–165.
[18]
Omid Gheibi, Danny Weyns, and Federico Quin. 2021. Applying Machine Learning in Self-Adaptive Systems: A Systematic Literature Review. ACM Trans. Auton. Adapt. Syst. 15, 3, Article 9 (aug 2021), 37 pages. https://doi.org/10.1145/3469440
[19]
Jetmir Haxhibeqiri, Eli De Poorter, Ingrid Moerman, and Jeroen Hoebeke. 2018. A survey of LoRaWAN for IoT: From technology to application. Sensors 18, 11 (2018), 3995.
[20]
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. 2016. Opponent modeling in deep reinforcement learning. In International conference on machine learning. PMLR, 1804–1813.
[21]
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. 2019. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems 33, 6 (2019), 750–797.
[22]
Fatima Hussain, Syed Ali Hassan, Rasheed Hussain, and Ekram Hossain. 2020. Machine learning for resource management in cellular and IoT networks: Potentials, current solutions, and open challenges. IEEE communications surveys & tutorials 22, 2 (2020), 1251–1275.
[23]
Usman Iftikhar and Danny Weyns. 2014. ActivFORMS: Active Formal Models for Self-Adaptation. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (Hyderabad, India) (SEAMS 2014). ACM, 125–134. https://doi.org/10.1145/2593929.2593944
[24]
Didac Gil De La Iglesia and Danny Weyns. 2015. MAPE-K formal templates to rigorously design behaviors for self-adaptive systems. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 10, 3 (2015), 1–31.
[25]
Jeffrey O Kephart and David M Chess. 2003. The vision of autonomic computing. Computer 36, 1 (2003), 41–50.
[26]
Latif U Khan, Walid Saad, Zhu Han, Ekram Hossain, and Choong Seon Hong. 2021. Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Communications Surveys & Tutorials 23, 3 (2021), 1759–1799.
[27]
Jeff Kramer and Jeff Magee. 2007. Self-Managed Systems: an Architectural Challenge. FoSE 2007: Future of Software Engineering, 259–268. https://doi.org/10.1109/FOSE.2007.19
[28]
Christian Krupitzer, Timur Temizer, Thomas Prantl, and Claudia Raibulet. 2020. An overview of design patterns for self-adaptive systems in the context of the internet of things. IEEE Access 8 (2020), 187384–187399.
[29]
Franklin F Kuo. 1974. The ALOHA system. ACM SIGCOMM Computer Communication Review 4, 1 (1974), 5–8.
[30]
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations. https://openreview.net/forum?id=Hk8N3Sclg
[31]
Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037 (2017).
[32]
Kai Li, Wei Ni, Eduardo Tovar, and Abbas Jamalipour. 2020. Deep Q-learning based resource management in UAV-assisted wireless powered IoT networks. In ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 1–6.
[33]
Shancang Li, Li Da Xu, and Shanshan Zhao. 2015. The internet of things: a survey. Information systems frontiers 17 (2015), 243–259.
[34]
Yongxin Liao, Eduardo de Freitas Rocha Loures, and Fernando Deschamps. 2018. Industrial Internet of Things: A systematic literature review and insights. IEEE Internet of Things Journal 5, 6 (2018), 4515–4525.
[35]
Roger A Light. 2017. Mosquitto: server and client implementation of the MQTT protocol. Journal of Open Source Software 2, 13 (2017), 265.
[36]
Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. 2018. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1774–1783.
[37]
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017).
[38]
Igor Mordatch and Pieter Abbeel. 2018. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
[39]
Angelika Musil, Juergen Musil, Danny Weyns, Tomas Bures, Henry Muccini, and Mohammad Sharaf. 2017. Patterns for Self-Adaptation in Cyber-Physical Systems. Springer International Publishing, Cham, 331–368. https://doi.org/10.1007/978-3-319-56345-9_13
[40]
Peyman Oreizy, Michael M. Gorlick, Richard N. Taylor, D. Heimhigner, G. Johnson, Nenad Medvidovic, Alex Quilici, David S. Rosenblum, and Alexander L. Wolf. 1999. An architecture-based approach to self-adaptive software. IEEE Intelligent Systems and their Applications 14, 3 (1999), 54–62. https://doi.org/10.1109/5254.769885
[41]
Emanuele Pesce and Giovanni Montana. 2020. Improving coordination in small-scale multi-agent deep reinforcement learning through memory-driven communication. Machine Learning 109, 9-10 (2020), 1727–1747.
[42]
Michiel Provoost and Danny Weyns. 2019. DingNet: A self-adaptive internet-of-things exemplar. In 2019 IEEE/ACM 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 195–201.
[43]
Federico Quin, Danny Weyns, and Omid Gheibi. 2022. Reducing large adaptation spaces in self-adaptive systems using classical machine learning. Journal of Systems and Software 190 (2022), 111341. https://doi.org/10.1016/j.jss.2022.111341
[44]
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. 2018. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning. PMLR, 4257–4266.
[45]
A. Ramírez and B. Cheng. 2010. Design patterns for developing dynamically adaptive systems. In International Symposium on Software Engineering for Adaptive and Self-Managing Systems.
[46]
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research 21, 1 (2020), 7234–7284.
[47]
Lukas Reinfurt, Uwe Breitenbücher, Michael Falkenthal, Frank Leymann, and Andreas Riegg. 2016. Internet of things patterns. In Proceedings of the 21st European Conference on Pattern Languages of Programs. 1–21.
[48]
Stefano Savazzi, Monica Nicoli, and Vittorio Rampa. 2020. Federated learning with cooperating devices: A consensus approach for massive IoT networks. IEEE Internet of Things Journal 7, 5 (2020), 4641–4654.
[49]
Emiliano Sisinni, Abusayeed Saifullah, Song Han, Ulf Jennehag, and Mikael Gidlund. 2018. Industrial internet of things: Challenges, opportunities, and directions. IEEE transactions on industrial informatics 14, 11 (2018), 4724–4734.
[50]
Sainbayar Sukhbaatar, Rob Fergus, 2016. Learning multiagent communication with backpropagation. Advances in neural information processing systems 29 (2016).
[51]
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. 2017. Multiagent cooperation and competition with deep reinforcement learning. PloS one 12, 4 (2017), e0172395.
[52]
Ming Tan. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning. 330–337.
[53]
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. 2020. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 (2020).
[54]
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. 2021. {DOP}: Off-Policy Multi-Agent Decomposed Policy Gradients. In International Conference on Learning Representations. https://openreview.net/forum?id=6FqKiVAdI3Y
[55]
Danny Weyns. 2020. Introduction to Self-Adaptive Systems: A Contemporary Software Engineering Perspective. Wiley. ISBN 978-1-119-57494-1.
[56]
Danny Weyns, Ilias Gerostathopoulos, Nadeem Abbas, Jesper Andersson, Stefan Biffl, Premek Brada, Tomas Bures, Amleto Di Salle, Matthias Galster, Patricia Lago, Grace Lewis, Marin Litoiu, Angelika Musil, Juergen Musil, Panos Patros, and Patrizio Pelliccione. 2023. Self-Adaptation in Industry: A Survey. ACM Trans. Auton. Adapt. Syst. (mar 2023). https://doi.org/10.1145/3589227 Just Accepted.
[57]
Danny Weyns, Omid Gheibi, Federico Quin, and Jeroen Van Der Donckt. 2022. Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-Adaptive Systems. ACM Trans. Auton. Adapt. Syst. 17, 1–2, Article 1 (jul 2022), 42 pages. https://doi.org/10.1145/3530192
[58]
Danny Weyns, M. Usman Iftikhar, Danny Hughes, and Nelson Matthys. 2018. Applying Architecture-Based Adaptation to Automate the Management of Internet-of-Things. In Software Architecture, Carlos E. Cuesta, David Garlan, and Jennifer Pérez (Eds.). Springer, 49–67.
[59]
Danny Weyns and Usman Iftikhar. 2022. ActivFORMS: A Formally Founded Model-Based Approach to Engineer Self-Adaptive Systems. ACM Transactions on Software Engineering and Methodology 1, 12 (2022).
[60]
Danny Weyns, Usman Iftikhar, and Joakim Söderlund. 2013. Do External Feedback Loops Improve the Design of Self-adaptive Systems? A Controlled Experiment. In 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. IEEE Press, 10 pages. http://dl.acm.org/citation.cfm?id=2487336.2487341
[61]
Danny Weyns, Sam Malek, and Jesper Andersson. 2010. FORMS: A Formal Reference Model for Self-Adaptation. In Proceedings of the 7th International Conference on Autonomic Computing (Washington, DC, USA) (ICAC ’10). Association for Computing Machinery, New York, NY, USA, 205–214. https://doi.org/10.1145/1809049.1809078
[62]
Danny Weyns, Michiel Provoost, Dimitri Van Landuyt, Sam Michiels, and Tomáš Bureš. 2023. A comparison of learning techniques in a distributed setting. (2023). https://people.cs.kuleuven.be/ danny.weyns/papers/2023DRL.pdf unpublished.
[63]
Danny Weyns, Bradley Schmerl, Vincenzo Grassi, Sam Malek, Raffaela Mirandola, Christian Prehofer, Jochen Wuttke, Jesper Andersson, Holger Giese, and Karl M Göschka. 2013. On patterns for decentralized control in self-adaptive systems. In Software Engineering for Self-Adaptive Systems II: International Seminar, Dagstuhl Castle, Germany, October 24-29, 2010 Revised Selected and Invited Papers. Springer, 76–107.
[64]
Dongbin Zhao, Haitao Wang, Kun Shao, and Yuanheng Zhu. 2016. Deep reinforcement learning with experience replay based on SARSA. In 2016 IEEE symposium series on computational intelligence (SSCI). IEEE, 1–6.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EuroPLoP '23: Proceedings of the 28th European Conference on Pattern Languages of Programs
July 2023
451 pages
ISBN:9798400700408
DOI:10.1145/3628034
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 February 2024

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

EuroPLoP 2023

Acceptance Rates

Overall Acceptance Rate 216 of 354 submissions, 61%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 35
    Total Downloads
  • Downloads (Last 12 months)35
  • Downloads (Last 6 weeks)2
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media