skip to main content
10.1145/3412841.3441894acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
research-article

Using regression learners to predict performance problems on software updates: a case study on elevators dispatching algorithms

Published: 22 April 2021 Publication History

Abstract

Remote software deployment and updating has long been commonplace in many different fields, but now, the increasing expansion of IoT and CPSoS (Cyber-Physcal System of Systems) has highlighted the need for additional mechanisms in these systems, to ensure the correct behaviour of the deployed software version after deployment. In this sense, this paper investigates the use of Machine Learning algorithms to predict acceptable behaviour in system performance of a new software release. By monitoring the real performance, eventual unexpected problems can be identified. Based on previous knowledge and actual run-time information, the proposed approach predicts the response time that can be considered acceptable for the new software release, and this information is used to identify problematic releases. The mechanism has been applied to the post-deployment monitoring of traffic algorithms in elevator systems. To evaluate the approach, we have used performance mutation testing, obtaining good results. This paper makes two contributions. First, it proposes several regression learners that have been trained with different types of traffic profiles to efficiently predict response time of the traffic dispatching algorithm. This prediction is then compared with the actual response time of the new algorithm release, and provides a verdict about its performance. Secondly, a comparison of the different learners is performed.

References

[1]
Jon Ayerdi, Aitor Garciandia, Aitor Arrieta, Wasif Afzal, Eduard Enoiu, and Aitor Agirre. Towards a Taxonomy for Eliciting Design-Operation Continuum Requirements of Cyber-Physical Systems. In IEEE 28th International Requirements Engineering Conference. IEEE, 2020.
[2]
Jon Ayerdi, Sergio Segura, Aitor Arrieta, Goiuria Sagardui, and Maite Arratibel. Qos-aware metamorphic testing: An elevation case study. In International Symposium on Software Reliability Engineering (ISSRE 2020). IEEE, 2020.
[3]
Sreram Balasubramaniyan, Seshadhri Srinivasan, Furio Buonopane, B. Subathra, Jüri Vain, and Srini Ramaswamy. Design and verification of Cyber-Physical Systems using TrueTime, evolutionary optimization and UPPAAL. Microprocessors and Microsystems, 42(2016):37--48, 2016.
[4]
G. C. Barney. 2003.
[5]
Wing Kwong Chan, Jeffrey CF Ho, and TH Tse. Finding failures from passed test cases: Improving the pattern classification approach to the testing of mesh simplification programs. Software Testing, Verification and Reliability, 20(2):89--120, 2010.
[6]
Vladimir Cherkassky and Yunqian Ma. Practical selection of SVM parameters and noise estimation for SVM regression. Neural Networks, 17(1):113--126, 2004.
[7]
Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
[8]
Pedro Delgado-Pérez, Ana Belén Sánchez, Sergio Segura, and Inmaculada Medina-Bulo. Performance mutation testing. Software Testing Verification and Reliability, 2020.
[9]
Giovanni Denaro, Andrea Polini, and Wolfgang Emmerich. Early performance testing of distributed software applications. Proceedings of the Fourth International Workshop on Software and Performance, WOSP'04, pages 94--103, 2004.
[10]
Vincenzo Ferme and Cesare Pautasso. Towards holistic continuous software performance assessment. ICPE 2017 - Companion of the 2017 ACM/SPEC International Conference on Performance Engineering, pages 159--164, 2017.
[11]
Ahmet Esat Genç, Hasan Sözer, M Furkan Kiraç, and Bariş Aktemur. Advisor: An adjustable framework for test oracle automation of visual output systems. IEEE Transactions on Reliability, 2019.
[12]
Anton Gulenko, Marcel Wallschlager, Florian Schmidt, Odej Kao, and Feng Liu. Evaluating machine learning algorithms for anomaly detection in clouds. Proceedings - 2016 IEEE International Conference on Big Data, Big Data 2016, pages 2716--2721, 2016.
[13]
Liang Hu, Xi Long Che, and Si Qing Zheng. Online system for grid resource monitoring and machine learning-based prediction. IEEE Transactions on Parallel and Distributed Systems, 23:134--145, 2012.
[14]
Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. ACM SIGPLAN Notices, 47:77--87, 2012.
[15]
René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 654--665. ACM, 2014.
[16]
Rakesh Kumar Lenka, Pranali Bhanse, and Utkalika Satapathy. Load performance testing on cloud platform. Proceedings - IEEE 2018 International Conference on Advances in Computing, Communication Control and Networking, ICACCCN 2018, pages 414--419, 2018.
[17]
Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Microsoft Azure, Peng Huang, Johns Hopkins University, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Microsoft Research, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), 2020.
[18]
Yepang Liu, Chang Xu, and Shing Chi Cheung. Characterizing and detecting performance bugs for smartphone applications. Proceedings - International Conference on Software Engineering, (1):1013--1024, 2014.
[19]
Elena Markoska and Sanja Lazarova-Molnar. Towards smart buildings performance testing as a service. 2018 3rd International Conference on Fog and Mobile Edge Computing, FMEC 2018, pages 277--282, 2018.
[20]
Oswaldo Olivo, Isil Dillig, and Calvin Lin. Static detection of asymptotic performance bugs in collection traversals. Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2015-June:369--378, 2015.
[21]
Carla Sauvanaud, Mohamed Kaâniche, Karama Kanoun, Kahina Lazri, and Guthemberg Da Silva Silvestre. Anomaly detection and diagnosis for cloud services: Practical experiments and lessons learned. Journal of Systems and Software, 139:84--106, 2018.
[22]
Sergio Segura, Javier Troya, Amador Durán, and Antonio Ruiz-Cortés. Performance metamorphic testing: motivation and challenges. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, pages 7--10. IEEE Press, 2017.
[23]
ML Siikonen. On traffic planning methodology. Lift Report, (March), 2001.
[24]
Zhen Song, Philippe Labalette, Robin Burger, Wolfram Klein, Sudev Nair, Suhas Suresh, Ling Shen, and Arquimedes Canedo. Model-based cyber-physical system integration in the process industry. IEEE International Conference on Automation Science and Engineering, 2015-October(September 2016):1012--1017, 2015.
[25]
Elaine J. Weyuker. Experience with performance testing of software systems: issues, an approach, and case study. IEEE Transactions on Software Engineering, 26:1147--1156, 2000.
[26]
Philipp Wieder, Edwin Yaqub, Ramin Yahyapour, and Ali Imran Jehangiri. Distributed predictive performance anomaly detection for virtualised platforms. International Journal of High Performance Computing and Networking, 11:279, 2018.
[27]
Chen Zhang, Jiaxin Li, Dongsheng Li, and Xicheng Lu. Understanding and Statically Detecting Synchronization Performance Bugs in Distributed Cloud Systems. IEEE Access, 7:99123--99135, 2019.
[28]
Shenglin Zhang, Ying Liu, Dan Pei, Yu Chen, Xianping Qu, Shimin Tao, Zhi Zang, Xiaowei Jing, and Mei Feng. FUNNEL: Assessing Software Changes in Web-Based Services. IEEE Transactions on Services Computing, 11(1):34--48, 2018.

Cited By

View all
  • (2023)Performance-Driven Metamorphic Testing of Cyber-Physical SystemsIEEE Transactions on Reliability10.1109/TR.2022.319307072:2(827-845)Online publication date: Jun-2023
  • (2023)DevOps for Cyber-Physical Systems: Objectives, Results and Lessons Learned from the Adeptness H2020 Project2023 26th Euromicro Conference on Digital System Design (DSD)10.1109/DSD60849.2023.00035(184-189)Online publication date: 6-Sep-2023
  • (2023)A microservice-based framework for multi-level testing of cyber-physical systemsSoftware Quality Journal10.1007/s11219-023-09639-z32:1(193-223)Online publication date: 31-May-2023
  • Show More Cited By
  1. Using regression learners to predict performance problems on software updates: a case study on elevators dispatching algorithms

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied Computing
    March 2021
    2075 pages
    ISBN:9781450381048
    DOI:10.1145/3412841
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 April 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cyber-physical systems
    2. machine learning
    3. performance bugs

    Qualifiers

    • Research-article

    Funding Sources

    • European Union
    • Basque Government

    Conference

    SAC '21
    Sponsor:
    SAC '21: The 36th ACM/SIGAPP Symposium on Applied Computing
    March 22 - 26, 2021
    Virtual Event, Republic of Korea

    Acceptance Rates

    Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

    Upcoming Conference

    SAC '25
    The 40th ACM/SIGAPP Symposium on Applied Computing
    March 31 - April 4, 2025
    Catania , Italy

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Performance-Driven Metamorphic Testing of Cyber-Physical SystemsIEEE Transactions on Reliability10.1109/TR.2022.319307072:2(827-845)Online publication date: Jun-2023
    • (2023)DevOps for Cyber-Physical Systems: Objectives, Results and Lessons Learned from the Adeptness H2020 Project2023 26th Euromicro Conference on Digital System Design (DSD)10.1109/DSD60849.2023.00035(184-189)Online publication date: 6-Sep-2023
    • (2023)A microservice-based framework for multi-level testing of cyber-physical systemsSoftware Quality Journal10.1007/s11219-023-09639-z32:1(193-223)Online publication date: 31-May-2023
    • (2023)The integration of machine learning into automated test generation: A systematic mapping studySoftware Testing, Verification and Reliability10.1002/stvr.184533:4Online publication date: 2-May-2023
    • (2022)A Digital-Twin Based Architecture for Software Longevity in Smart Homes2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS54860.2022.00070(669-679)Online publication date: Jul-2022
    • (2022)Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithmsJournal of Software: Evolution and Process10.1002/smr.246534:11Online publication date: 25-May-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media