Skip to main content
SpringerLink
Log in

Meaningful Update and Repair of Markov Decision Processes for Self-Adaptive Systems

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Self-adaptive systems are able to adjust their behaviour in response to environmental condition changes and are widely deployed as Internetwares. Considered as a promising way to handle the ever-growing complexity of software systems, they have seen an increasing level of interest and are covering a variety of applications, e.g., autonomous car systems and adaptive network systems. Many approaches for the construction of self-adaptive systems have been developed, and probabilistic models, such as Markov decision processes (MDPs), are one of the favoured. However, the majority of them do not deal with the problems of the underlying MDP being obsolete under new environments or unsatisfactory to the given properties. This results in the generated policies from such MDP failing to guide the self-adaptive system to run correctly and meet goals. In this article, we propose a systematic approach to updating an obsolete MDP by exploring new states and transitions and removing obsolete ones, and repairing an unsatisfactory MDP by adjusting its structure in a more meaningful way rather than arbitrarily changing the transition probabilities to values not in line with reality. Experimental results show that the MDPs updated and repaired by our approach are more competent in guiding the self-adaptive systems’ correct running compared with the original ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Esfahani N, Malek S. Uncertainty in self-adaptive software systems. In Proc. the International Seminar on Software Engineering for Self-Adaptive Systems, October 2010, pp.214-238. https://doi.org/10.1007/978-3-642-35813-5_9.

  2. Cámara J, Schmerl B, Moreno G A, Garlan D. MOSAICO: Offline synthesis of adaptation strategy repertoires with flexible trade-offs. Automated Software Engineering, 2018, 25(3): 595–626. https://doi.org/10.1007/s10515-018-0234-9.

    Article  Google Scholar 

  3. Weyns D, Schmerl B, Grassi V, Malek S, Mirandola R, Prehofer C, Wuttke J, Andersson J, Giese H, Göschka K M. On patterns for decentralized control in self-adaptive systems. In Lecture Notes in Computer Science 7475, de Lemos R, Giese H, Wüller M A et al. (eds.), Springer Berlin Heidelberg, 2013, pp.76-107. https://doi.org/10.1007/978-3-642-35813-5_4.

  4. Krupitzer C, Roth F M, VanSyckel S, Schiele G, Becker C. A survey on engineering approaches for self-adaptive systems. Pervasive Mob. Comput., 2015, 17: 184-206. https://doi.org/10.1016/j.pmcj.2014.09.009.

    Article  Google Scholar 

  5. Wang Q. Towards a rule model for self-adaptive software. SIGSOFT Softw. Eng. Notes, 2005, 30(1): Article No. 8. https://doi.org/10.1145/1039174.1039198.

  6. Sama M, Rosenblum D S, Wang Z, Elbaum S. Model-based fault detection in context-aware adaptive applications. In Proc. the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, November 2008, pp.261-271. https://doi.org/10.1145/1453101.1453136.

  7. Filieri A, Hoffmann H, Maggio M. Automated design of self-adaptive software with control-theoretical formal guarantees. In Proc. the 36th International Conference on Software Engineering, May 31-June 7, 2014, pp.299-310. https://doi.org/10.1145/2568225.2568272.

  8. Filieri A, Hoffmann H, Maggio M. Automated multiobjective control for self-adaptive software design. In Proc. the 10th Joint Meeting on Foundations of Software Engineering, August 30-September 4, 2015, pp.13-24. https://doi.org/10.1145/2786805.2786833.

  9. Shevtsov S, Weyns D. Keep it SIMPLEX: Satisfying multiple goals with guarantees in control-based self-adaptive systems. In Proc. the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, November 2016, pp. 229-241. https://doi.org/10.1145/2950290.2950301.

  10. Cámara J, De Lemos R. Evaluation of resilience in self-adaptive systems using probabilistic model-checking. In Proc. the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2012, pp.53-62. https://doi.org/10.1109/SEAMS.2012.6224391.

  11. Franco J M, Correia F, Barbosa R, Zenha-Rela M, Schmerl B, Garlan D. Improving self-adaptation planning through software architecture-based stochastic modeling. J. Syst. Softw., 2016, 115: 42-60. https://doi.org/10.1016/j.jss.2016.01.026.

    Article  Google Scholar 

  12. Filieri A, Tamburrelli G. Probabilistic verification at runtime for self-adaptive systems. In Assurances for Self-Adaptive Systems: Principles, Models, and Techniques, Cámara J, De Lemos R, Ghezzi C, Lopes A (eds.), Springer, 2013, pp.30-59. https://doi.org/10.1007/978-3-642-36249-1_2.

  13. Ghezzi C, Pinto L S, Spoletini P, Tamburrelli G. Managing non-functional uncertainty via model-driven adaptivity. In Proc. the 2013 International Conference on Software Engineering, May 2013, pp.33-42. https://doi.org/10.1109/ICSE.2013.6606549.

  14. Brechtel S, Gindele T, Dillmann R. Probabilistic MDP-behavior planning for cars. In Proc. the 14th International IEEE Conference on Intelligent Transportation Systems, Oct. 2011, pp.1537-1542. https://doi.org/10.1109/ITSC.2011.6082928.

  15. Kwiatkowska M, Parker D. Automated verification and strategy synthesis for probabilistic systems. In Proc. the 11th International Symposium on Automated Technology for Verification and Analysis, October 2013, pp.5-22. https://doi.org/10.1007/978-3-319-02444-8_2.

  16. Bartocci E, Grosu R, Katsaros P, Ramakrishnan C R, Smolka S A. Model repair for probabilistic systems. In Proc. the 17th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, March 26-April 3, 2011, pp.326-340. https://doi.org/10.1007/978-3-642-19835-9_30.

  17. Chen T, Hahn E M, Han T, Kwiatkowska M, Qu H, Zhang L. Model repair for Markov decision processes. In Proc. the 7th International Symposium on Theoretical Aspects of Software Engineering, July 2013, pp.85-92. https://doi.org/10.1109/TASE.2013.20.

  18. Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41–50. https://doi.org/10.1109/MC.2003.1160055.

    Article  MathSciNet  Google Scholar 

  19. Sykes D, Corapi D, Magee J, Kramer J, Russo A, Inoue K. Learning revised models for planning in adaptive systems. In Proc. the 2013 International Conference on Software Engineering, May 2013, pp.63-71. https://doi.org/10.1109/ICSE.2013.6606552.

  20. Cheng B H C, Sawyer P, Bencomo N, Whittle J. A goal-based modeling approach to develop requirements of an adaptive system with environmental uncertainty. In Proc. the 12th International Conference on Model Driven Engineering Languages and Systems, October 2009, pp.468-483. https://doi.org/10.1007/978-3-642-04425-0_36.

  21. Cámara J, Garlan D, Schmerl B, Pandey A. Optimal planning for architecture-based self-adaptation via model checking of stochastic games. In Proc. the 30th Annual ACM Symposium on Applied Computing, April 2015, pp.428-435. https://doi.org/10.1145/2695664.2695680.

  22. Moreno G A, Cámara J, Garlan D, Schmerl B. Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In Proc. the 10th Joint Meeting on Foundations of Software Engineering, August 30-September 4, 2015, pp.1-12. https://doi.org/10.1145/2786805.2786853.

  23. Kwiatkowska M, Norman G, Parker D. PRISM 4.0: Verification of probabilistic real-time systems. In Proc. the 23rd International Conference on Computer Aided Verification, July 2011, pp.585-591. https://doi.org/10.1007/978-3-642-22110-1_47.

  24. Clarke E M, Emerson E A. Design and synthesis of synchronization skeletons using branching time temporal logic. In Proc. the Workshop on Logics of Programs, May 1981, pp.52-71. https://doi.org/10.1007/BFb0025774.

  25. Iftikhar M U, Ramachandran G S, Bollansée P, Weyns D, Hughes D. DeltaIoT: A self-adaptive Internet of Things exemplar. In Proc. the 12th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-Managing Systems, May 2017, pp.76-82. https://doi.org/10.1109/SEAMS.2017.21.

  26. Puterman M L. Markov Decision Processes: Discrete Stochastic Dynamic Programming (1st edition). John Wiley & Sons, 1994. https://doi.org/10.1002/9780470316887.

  27. Sama M, Elbaum S, Raimondi F, Rosenblum D S, Wang Z. Context-aware adaptive applications: Fault patterns and their automated identification. IEEE Transactions on Software Engineering, 2010, 36(5): 644-661. https://doi.org/10.1109/TSE.2010.35.

    Article  Google Scholar 

  28. Yang W, Xu C, Liu Y, Cao C, Ma X, Lu J. Verifying self-adaptive applications suffering uncertainty. In Proc. the 29th ACM/IEEE International Conference on Automated Software Engineering, September 2014, pp.199-210. https://doi.org/10.1145/2642937.2642999.

  29. Yang W, Xu C, Pan M, Cao C, Ma X, Lu J. Efficient validation of self-adaptive applications by counterexample probability maximization. Journal of Systems and Software, 2018, 138: 82-99. https://doi.org/10.1016/j.jss.2017.12.009.

    Article  Google Scholar 

  30. Wilcoxon F. Individual comparisons by ranking methods. Biometrics Bulletin, 1945, 1(6): 80-83. https://doi.org/10.2307/3001968.

    Article  Google Scholar 

  31. Abdi H. The bonferonni and Šidák corrections for multiple comparisons. In Encyclopedia of Measurement and Statistics, Salkind N (ed.), SAGE, 2007, pp.103-107.

  32. Salehie M, Tahvildari L. Self-adaptive software: Landscape and research challenges. ACM Trans. Auton. Adapt. Syst., 2009, 4(2): Article No. 4.

  33. Zhao T. The generation and evolution of adaptation rules in requirements driven self-adaptive systems. In Proc. the 24th IEEE International Requirements Engineering Conference, Sept. 2016, pp.456-461. https://doi.org/10.1109/RE.2016.18.

  34. Cheng S W, Huang A C, Garlan D, Schmerl B, Steenkiste P. Rainbow: Architecture-based self-adaptation with reusable infrastructure. In Proc. the 1st International Conference on Autonomic Computing, May 2004, pp.276-277. https://doi.org/10.1109/ICAC.2004.46.

  35. Chen B, Peng X, Liu Y, Song S, Zheng J, Zhao W. Architecture-based behavioral adaptation with generated alternatives and relaxed constraints. IEEE Transactions on Services Computing, 2019, 12(1): 73-87. https://doi.org/10.1109/TSC.2016.2593459.

    Article  Google Scholar 

  36. Howard R A. Dynamic Programming and Markov Processes (1st edition). The MIT Press, 1960.

  37. Sutton R S, Barto A G. Reinforcement Learning: An Introduction (2nd edition). Bradford Books, 2018.

  38. Calinescu R, Ghezzi C, Kwiatkowska M, Mirandola R. Self-adaptive software needs quantitative verification at runtime. Commun. ACM, 2012, 55(9): 69-77. https://doi.org/10.1145/2330667.2330686.

    Article  Google Scholar 

  39. Su G, Chen T, Feng Y, Rosenblum D S, Thiagarajan P S. An iterative decision-making scheme for Markov decision processes and its application to self-adaptive systems. In Proc. the 19th International Conference on Fundamental Approaches to Software Engineering, April 2016, pp.269-286. https://doi.org/10.1007/978-3-662-49665-7_16.

  40. Filieri A, Grunske L, Leva A. Lightweight adaptive filtering for efficient learning and updating of probabilistic models. In Proc. the 37th International Conference on Software Engineering, May 2015, pp.200-211. https://doi.org/10.1109/ICSE.2015.41.

  41. Nahabedian L, Braberman V, D’Ippolito N, Honiden S, Kramer J, Tei K, Uchitel S. Dynamic update of discrete event controllers. IEEE Transactions on Software Engineering, 2018, 46(11): 1220-1240. https://doi.org/10.1109/TSE.2018.2876843.

    Article  Google Scholar 

  42. Ghezzi C, Greenyer J, Manna V P L. Synthesizing dynamically updating controllers from changes in scenario-based specifications. In Proc. the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2012, pp.145-154. https://doi.org/10.1109/SEAMS.2012.6224401.

  43. Hahn E M, Han T, Zhang L. Synthesis for PCTL in parametric Markov decision processes. In Proc. the 3rd International Symposium on NASA Formal Methods, April 2011, pp.146-161. https://doi.org/10.1007/978-3-642-20398-5_12.

  44. Cubuktepe M, Jansen N, Junges S, Katoen J P, Papusha I, Poonawala H A, Topcu U. Sequential convex programming for the efficient verification of parametric MDPs. In Proc. the 23rd International Conference on Tools and Algorithms for the Construction and Analysis of Systems, April 2017, pp.133-150. https://doi.org/10.1007/978-3-662-54580-5_8.

  45. Arming S, Bartocci E, Sokolova A. SEA-PARAM: Exploring schedulers in parametric MDPs. In Proc. the 15th Workshop on Quantitative Aspects of Programming Languages and Systems, April 2017, pp.25-38. https://doi.org/10.4204/EPTCS.250.3.

  46. Pathak S, Ábrahám E, Jansen N, Tacchella A, Katoen J P. A greedy approach for the efficient repair of stochastic models. In Proc. the 7th International Symposium on NASA Formal Methods, April 2015, pp.295-309. https://doi.org/10.1007/978-3-319-17524-9_21.

  47. Chatzieleftheriou G, Katsaros P. Abstract model repair for probabilistic systems. Information and Computation, 2018, 259: 142-160. https://doi.org/10.1016/j.ic.2018.02.019.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Wen-Hua Yang, Yu Zhou & Zhi-Qiu Huang

  2. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210093, China

    Wen-Hua Yang, Min-Xue Pan & Yu Zhou

  3. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing University, Nanjing, 210023, China

    Wen-Hua Yang

Authors
  1. Wen-Hua Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min-Xue Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhi-Qiu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen-Hua Yang.

Supplementary Information

ESM 1

(PDF 379 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, WH., Pan, MX., Zhou, Y. et al. Meaningful Update and Repair of Markov Decision Processes for Self-Adaptive Systems. J. Comput. Sci. Technol. 37, 106–127 (2022). https://doi.org/10.1007/s11390-021-1484-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1484-8

Keywords

Search

Navigation