Skip to main content
SpringerLink
Log in

Handling uncertainty in self-adaptive systems: an ontology-based reinforcement learning model

  • Original Article
  • Published:
Journal of Reliable Intelligent Environments Aims and scope Submit manuscript

Abstract

Ubiquitous and pervasive systems interact with each other and perform actions favoring the emergence of a global desired behavior. To function well, these systems need to be self-adaptive to handle noisy data and partially observable dynamic environments. However, facing unpredictable and rare events while only accessing incomplete information about the environment causes uncertainty in the adaptation process. Such uncertainty results in inconsistent decisions and unexpected system behavior. Currently, SAS handles such unpredictable conditions using adaptive modeling mechanisms to select default actions or exploiting reinforcement learning (RL) algorithms to learn new actions. However, the current mechanisms do not address rare events in an environment. This paper improves the system’s decision-making when facing rare events by providing extra information about alternative adaptation actions using domain ontologies, which provide a thorough understanding of a domain. In this paper, we propose an Ontology-based unCertainty handling model (OnCertain), which enables the RL-based system to augment its observation and reason about the rare event using prior ontological knowledge. The overall aim of this model is to improve the system’s decision-making process under conditions of uncertainty. Our model is evaluated in a traffic signal control system and an edge computing environment. The results show that the OnCertain model can improve the RL-based systems’ observation and, consequently, their performance in such environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Availability of data and materials/Code availability

The code of the OnCertain-DDPG algorithm and the simulated data and results are publicly available: https://github.com/saeedehghanadbashi/ontology-based-RL.

Notes

  1. The code of OnCertain-DDPG algorithm and the simulated results are publicly available: https://github.com/saeedehghanadbashi/ontology-based-RL.

References

  1. Calinescu R, Mirandola R, Perez-Palacin D, Weyns D (2020) Understanding uncertainty in self-adaptive systems. In: International conference on autonomic computing and self-organizing systems (ACSOS). IEEE, Washington, pp 242–251

  2. Van Der Donckt J, Weyns D, Quin F, Van Der Donckt J, Michiels S (2020) Applying deep learning to reduce large adaptation spaces of self-adaptive systems with multiple types of goals. In: International symposium on software engineering for adaptive and self-managing systems (SEAMS). IEEE/ACM, Seoul, pp 20–30

  3. Wang Y, Yang X, Liang H, Liu Y (2018) A review of the self-adaptive traffic signal control system based on future traffic environment. J Adv Transp 2018

  4. Safavifar Z, Ghanadbashi S, Golpayegani F (2021) Adaptive workload orchestration in pure edge computing: a reinforcement-learning model. In: International conference on tools with artificial intelligence (ICTAI). IEEE, Washington, pp 856–860

  5. Ciampi M, Coronato A, Naeem M, Silvestri S (2022) An intelligent environment for preventing medication errors in home treatment. Expert Syst Appl 193:116434

    Article  Google Scholar 

  6. Naeem M, Coronato A (2022) An AI-empowered home-infrastructure to minimize medication errors. J Sens Actuator Netw 11(1):13

    Article  Google Scholar 

  7. Schmerl B, Andersson J, Vogel T, Cohen MB, Rubira CM, Brun Y, Gorla A, Zambonelli F, Baresi L (2017) Challenges in composing and decomposing assurances for self-adaptive systems. In: De Lemos R, Garlan D, Ghezzi C, Giese H (eds) Software engineering for self-adaptive systems III. Assurances. Springer, Cham, pp 64–89

  8. Esfahani N, Malek S (2013) Uncertainty in self-adaptive software systems. In: De Lemos R, Giese H, Müller HA, Shaw M (eds) Software engineering for self-adaptive systems. Springer, Berlin, pp 214–238

  9. Mahdavi-Hezavehi S, Avgeriou P, Weyns D (2017) A classification framework of uncertainty in architecture-based self-adaptive systems with multiple quality requirements. In: Mistrik I, Ali N, Kazman R, Grundy J, Schmerl B (eds) Managing trade-offs in adaptable software architectures. Elsevier, Berlin, pp 45–77

  10. Calinescu R, Grunske L, Kwiatkowska M, Mirandola R, Tamburrelli G (2010) Dynamic QoS management and optimization in service-based systems. Trans Softw Eng 37(3):387–409

    Article  Google Scholar 

  11. Cámara J, Peng W, Garlan D, Schmerl B (2017) Reasoning about sensing uncertainty in decision-making for self-adaptation. In: International conference on software engineering and formal methods. Springer, Cham, pp 523–540

  12. Moreno G.A, Cámara J, Garlan D, Klein M (2018) Uncertainty reduction in self-adaptive systems. In: International conference on software engineering for adaptive and self-managing systems (SEAMS). ACM, Gothenburg, pp 51–57

  13. Gheibi O, Weyns D, Quin F (2021) Applying machine learning in self-adaptive systems: a systematic literature review. Trans Auton Adapt Syst 15(3):1–37

    Google Scholar 

  14. Metzger A, Quinton C, Mann Z.Á, Baresi L, Pohl K (2022) Realizing self-adaptive systems via online reinforcement learning and feature-model-guided exploration. Computing 104:1–22

  15. Zouaq A, Nkambou R (2010) A survey of domain ontology engineering: methods and tools. In: Nkambou R, Bourdeau J, Mizoguchi R (eds) Advances in intelligent tutoring systems. Springer, Berlin, pp 103–119

  16. Fong ACM, Hong G, Fong B (2019) Augmented intelligence with ontology of semantic objects. In: International conference on contemporary computing and informatics (IC3I). IEEE, Singapore, pp 1–4

  17. Golpayegani F, Dusparic I, Clarke S (2019) Using social dependence to enable neighbourly behaviour in open multi-agent systems. Trans Intell Syst Technol 10(3):1–31

    Article  Google Scholar 

  18. Alkhabbas F, Murturi I, Spalazzese R, Davidsson P, Dustdar S (2020) A goal-driven approach for deploying self-adaptive IoT systems. In: International conference on software architecture (ICSA). IEEE, Salvador, pp 146–156

  19. Qureshi NA, Liaskos S, Perini A (2011) Reasoning about adaptive requirements for self-adaptive systems at runtime. In: International workshop on Requirements@Run.Time (RE@RunTime). IEEE, Trento, pp 16–22

  20. Morandini M (2011) Goal-oriented development of self-adaptive systems. PhD thesis, University of Trento, Italy. http://eprints-phd.biblio.unitn.it/511

  21. Whittle J, Sawyer P, Bencomo N, Cheng BH, Bruel J-M (2010) RELAX: a language to address uncertainty in self-adaptive systems requirement. Requir Eng 15(2):177–196

    Article  Google Scholar 

  22. Ding Z, Zhou Y, Zhou M (2015) Modeling self-adaptive software systems with learning Petri nets. Trans Syst Man Cybern Syst 46(4):483–498

    Article  Google Scholar 

  23. Qin Y, Xu C, Yu P, Lu J (2016) SIT: sampling-based interactive testing for self-adaptive apps. J Syst Softw 120:70–88

    Article  Google Scholar 

  24. Sykes D, Corapi D, Magee J, Kramer J, Russo A, Inoue K (2013) Learning revised models for planning in adaptive systems. In: International conference on software engineering (ICSE). IEEE, San Francisco, pp 63–71

  25. Vassev E, Hinchey M, Balasubramaniam D, Dobson S (2011) An ASSL approach to handling uncertainty in self-adaptive systems. In: Annual software engineering workshop (SEW). IEEE, Limerick, pp 11–18

  26. Solano GF, Caldas RD, Rodrigues GN, Vogel T, Pelliccione P (2019) Taming uncertainty in the assurance process of self-adaptive systems: a goal-oriented approach. In: International symposium on software engineering for adaptive and self-managing systems (SEAMS). IEEE/ACM, Montreal, pp 89–99

  27. Elkhodary A, Esfahani N, Malek S (2010) FUSION: a framework for engineering self-tuning self-adaptive software systems. In: International symposium on foundations of software engineering (FSE). ACM, Santa Fe, pp 7–16

  28. Calinescu R, Kwiatkowska M (2009) Using quantitative analysis to implement autonomic IT systems. In: International conference on software engineering (ICSE). IEEE, Vancouver, pp 100–110

  29. Cheng S-W, Garlan D (2007) Handling uncertainty in autonomic systems. In: International Workshop on Living with Uncertainties (IWLU’07), colocated with International Conference on Automated Software Engineering (ASE’07). Citeseer, Atlanta,

  30. Baresi L, Pasquale L, Spoletini P (2010) Fuzzy goals for requirements-driven adaptation. In: International requirements engineering conference (RE). IEEE, Sydney, pp 125–134

  31. Whittle J, Sawyer P, Bencomo N, Cheng BH, Bruel J-M (2009) Relax: incorporating uncertainty into the specification of self-adaptive systems. In: International requirements engineering conference (RE). IEEE, Atlanta, pp 79–88

  32. Esfahani N, Kouroshfar E, Malek S (2011) Taming uncertainty in self-adaptive software. In: International symposium on foundations of software engineering (FSE). ACM, Szeged, pp 234–244

  33. Han D, Xing J, Yang Q, Li J, Wang H (2016) Handling uncertainty in self-adaptive software using self-learning fuzzy neural network. In: Annual computer software and applications conference (COMPSAC), vol 2. IEEE, Atlanta, pp 540–545

  34. Sharifloo AM, Metzger A, Quinton C, Baresi L, Pohl K (2016) Learning and evolution in dynamic software product lines. In: International symposium on software engineering for adaptive and self-managing systems (SEAMS). IEEE/ACM, Austin, pp 158–164

  35. Kephart JO, Chess DM (2003) The vision of autonomic computing. Computer 36(1):41–50

    Article  MathSciNet  Google Scholar 

  36. Weyns D, Schmerl B, Kishida M, Leva A, Litoiu M, Ozay N, Paterson C, Tei K (2021) Towards better adaptive systems by combining MAPE, control theory, and machine learning. In: International symposium on software engineering for adaptive and self-managing systems (SEAMS). IEEE/ACM, Madrid, pp 217–223

  37. Rodrigues A, Caldas RD, Rodrigues GN, Vogel T, Pelliccione P (2018) A learning approach to enhance assurances for real-time self-adaptive systems. In: International symposium on software engineering for adaptive and self-managing systems (SEAMS). IEEE/ACM, Gothenburg, pp 206–216

  38. Zavala E, Franch X, Marco J, Knauss A, Damian D (2018) SACRE: supporting contextual requirements’ adaptation in modern self-adaptive systems in the presence of uncertainty at runtime. Expert Syst Appl 98:166–188

    Article  Google Scholar 

  39. Knauss A, Damian D, Franch X, Rook A, Müller HA, Thomo A (2016) ACon: a learning-based approach to deal with uncertainty in contextual requirements at runtime. Inf Softw Technol 70:85–99

    Article  Google Scholar 

  40. Mao X, Dong M, Liu L, Wang H (2014) An integrated approach to developing self-adaptive software. Inf Sci Eng 30(4):1071–1085

  41. Wu T, Li Q, Wang L, He L, Li Y (2018) Using reinforcement learning to handle the runtime uncertainties in self-adaptive software. In: International conference on software technologies: applications and foundations (STAF). Springer, Berlin, pp 387–393

  42. Teimourikia M, Fugini M (2017) Ontology development for run-time safety management methodology in smart work environments using ambient knowledge. Future Gener Comput Syst 68:428–441

    Article  Google Scholar 

  43. Poggi F, Rossi D, Ciancarini P, Bompani L (2016) An application of semantic technologies to self adaptations. In: International forum on research and technologies for society and industry (RTSI). IEEE, Bologna, pp 1–6

  44. Qureshi NA, Perini A, Ernst NA, Mylopoulos J (2010) Towards a continuous requirements engineering framework for self-adaptive systems. In: International workshop on Requirements@Run.Time (RE@RunTime). IEEE, Sydney, pp 9–16

  45. Cheng W, Li Q, Wang L, He L (2018) Handling uncertainty online for self-adaptive systems. In: International conference on soft computing and machine intelligence (ISCMI). IEEE, Nairobi, pp 135–139

  46. Hao J, Bouzouane A, Gaboury S (2019) An incremental learning method based on formal concept analysis for pattern recognition in nonstationary sensor-based smart environments. Pervasive Mob Comput 59:101045

    Article  Google Scholar 

  47. Civitarese G, Bettini C, Sztyler T, Riboni D, Stuckenschmidt H (2019) newNECTAR: collaborative active learning for knowledge-based probabilistic activity recognition. Pervasive Mob Comput 56:88–105

    Article  Google Scholar 

  48. Sanabria AR, Ye J (2020) Unsupervised domain adaptation for activity recognition across heterogeneous datasets. Pervasive Mob Comput 64:101147

    Article  Google Scholar 

  49. Ghanadbashi S, Golpayegani F (2021) An ontology-based intelligent traffic signal control model. In: International intelligent transportation systems conference (ITSC). IEEE, Indianapolis, pp 2554–2561

  50. Palm A, Metzger A, Pohl K (2020) Online reinforcement learning for self-adaptive information systems. In: International conference on advanced information systems engineering (CAiSE). Springer, Berlin, pp 169–184

  51. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT, Cambridge, pp 154–157

    Google Scholar 

  52. Thanh-Tung D, Flood B, Wilson C, Sheahan C, Bao-Lam D (2006) Ontology-MAS for modelling and robust controlling enterprises. In: International conference on theories and applications of computer science (ICTACS). World Scientific, Ho Chi Minh, pp 116–123

  53. Horrocks I, Patel-Schneider PF, Boley H, Tabet S, Grosof B, Dean M et al (2004) SWRL: a semantic web rule language combining OWL and RuleML. W3C Memb Submiss 21(79):1–31

    Google Scholar 

  54. Belohlavek R, Macko J (2011) Selecting important concepts using weights. In: International conference on formal concept analysis (ICFCA). Springer, Nicosia, pp 65–80

  55. Saleena B, Srivatsa S (2015) Using concept similarity in cross ontology for adaptive e-Learning systems. J King Saud Univ Comput Inf Sci 27(1):1–12

    Google Scholar 

  56. Vitkute-Adzgauskiene D, Markievicz I, Krilavicius T, Tamosiunaite M. (2022) Learning and execution of action categories (ACAT). https://if.vdu.lt/en/research/projects/project-learning-and-execution-of-action-categories-acat. Accessed 2022

  57. Li C, Tian G (2020) Transferring the semantic constraints in human manipulation behaviors to robots. Appl Intell 50:1711–1724

  58. Vallejo M, Corne DW (2016) Evolutionary algorithms under noise and uncertainty: a location-allocation case study. In: Viktorovic M, Yang D, De Vries B (eds) Symposium series on computational intelligence (SSCI). IEEE, Athens, pp 1–10

  59. Viktorović M, Yang D, de Vries B (2020) Connected Traffic Data Ontology (CTDO) for intelligent urban traffic systems focused on connected (semi) autonomous vehicles. Sensors 20(10):2961

    Article  ADS  PubMed Central  PubMed  Google Scholar 

  60. Wei H, Zheng G, Gayah V, Li Z (2019) A survey on traffic signal control methods. Comput Res Repos. arXiv:1904.08117

  61. Mazak A, Schandl B, Lanzenberger M (2010) iweightings: Enhancing structure-based ontology alignment by enriching models with importance weighting. In: International conference on complex, intelligent and software intensive systems (CISIS). IEEE, Krakow, pp 992–997

  62. Wang J, Liu Y, Li B (2020) Reinforcement learning with perturbed rewards. In: AAAI conference on artificial intelligence. AAAI Press, New York, vol 34, pp 6202–6209

  63. Kiran BR, Sobh I, Talpaert V, Mannion P, Al Sallab AA, Yogamani S, Pérez P (2022) Deep reinforcement learning for autonomous driving: a survey. Trans Intell Transp Syst 23(6):4909–4926

  64. Alegre LN (2019) SUMO-RL. https://github.com/LucasAlegre/sumo-rl. Accessed 2022

  65. Han M, Lee Y, Moon SB, Jang K, Lee D (2008) CRAWDAD dataset kaist/wibro (v. 2008-06-04). https://crawdad.org/kaist/wibro/20080604. https://doi.org/10.15783/C72S3B

  66. Chen D (2020) Resources allocation in the edge computing environment using reinforcement learning. https://github.com/davidtw0320/ Resources-Allocation-in-The-Edge-Computing-Environment-Using-Reinforcement-Learning. Accessed 2022

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University College Dublin, Belfield, Dublin 4, D04 V1W8, Ireland

    Saeedeh Ghanadbashi, Zahra Safavifar & Fatemeh Golpayegani

  2. School of Computer Science, Islamic Azad University, Azarshahr Street, Tehran, 1584743311, Iran

    Farshad Taebi

Authors
  1. Saeedeh Ghanadbashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zahra Safavifar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farshad Taebi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fatemeh Golpayegani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saeedeh Ghanadbashi.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghanadbashi, S., Safavifar, Z., Taebi, F. et al. Handling uncertainty in self-adaptive systems: an ontology-based reinforcement learning model. J Reliable Intell Environ 10, 19–44 (2024). https://doi.org/10.1007/s40860-022-00198-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40860-022-00198-x

Keywords

Search

Navigation