Abstract
Finding the best configurations for a highly configurable system is challenging. Existing studies learned regression models to predict the performance of potential configurations. Such learning suffers from the low accuracy and the high effort of examining the actual performance for data labeling. A recent approach uses an iterative strategy to sample a small number of configurations from the training pool to reduce the number of sampled ones. In this paper, we conducted a comparative study on the rank-based approach of configurable systems with four regression methods. These methods are compared on 21 evaluation scenarios of 16 real-world configurable systems. We designed three research questions to check the impacts of different methods on the rank-based approach. We find out that the decision tree method of Classification And Regression Tree (CART) and the ensemble learning method of Gradient Boosted Regression Trees (GBRT) can achieve better ranks among four regression methods under evaluation; the sampling strategy in the rank-based approach is useful to save the cost of sampling configurations; the measurement, i.e., rank difference correlates with the relative error in several evaluation scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Ranks in the experiment are zero-based; that is, the MAR value of the best configuration is zero.
References
Breiman, L.: Classification and Regression Trees. Routledge, Abingdon (2017)
Drucker, H., Burges, C.J., Kaufman, L., Smola, A.J., Vapnik, V.: Support vector regression machines. In: Advances in Neural Information Processing Systems, pp. 155–161 (1997)
Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38(4), 367–378 (2002)
Gu, Y., Chen, Y., Jia, X., Xuan, J.: Multi-objective configuration sampling for performance ranking in configurable systems. In: Proceedings of the the 26th Asia-Pacific Software Engineering Conference (APSEC 2019), Putrajaya, Malaysia, 2–5 December 2019 (2019)
Gu, Y., et al.: Does the fault reside in a stack trace? Assisting crash localization by predicting crashing fault residence. J. Syst. Softw. 148, 88–104 (2019). https://doi.org/10.1016/j.jss.2018.11.004
Guo, J., Czarnecki, K., Apel, S., Siegmund, N., Wasowski, A.: Variability-aware performance prediction: a statistical learning approach. In: 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, 11–15 November 2013, pp. 301–311 (2013). https://doi.org/10.1109/ASE.2013.6693089
Jamshidi, P., Casale, G.: An uncertainty-aware approach to optimal configuration of stream processing systems. In: 24th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2016, London, United Kingdom, 19–21 September 2016, pp. 39–48 (2016). https://doi.org/10.1109/MASCOTS.2016.17
Nair, V., Menzies, T., Siegmund, N., Apel, S.: Using bad learners to find good configurations. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, 4–8 September 2017, pp. 257–267 (2017). https://doi.org/10.1145/3106237.3106238
Nair, V., Menzies, T., Siegmund, N., Apel, S.: Faster discovery of faster system configurations with spectral learning. Autom. Softw. Eng. 25(2), 247–277 (2018). https://doi.org/10.1007/s10515-017-0225-2
Nair, V., Yu, Z., Menzies, T., Siegmund, N., Apel, S.: Finding faster configurations using FLASH. CoRR abs/1801.02175 (2018). http://arxiv.org/abs/1801.02175
Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., von Luxburg, U., Rätsch, G. (eds.) ML -2003. LNCS (LNAI), vol. 3176, pp. 63–71. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28650-9_4
Sarkar, A., Guo, J., Siegmund, N., Apel, S., Czarnecki, K.: Cost-efficient sampling for performance prediction of configurable systems (T). In: 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, 9–13 November, pp. 342–352 (2015). https://doi.org/10.1109/ASE.2015.45
Siegmund, N., Grebhahn, A., Apel, S., Kästner, C.: Performance-influence models for highly configurable systems. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, 30 August–4 September 2015, pp. 284–294 (2015). https://doi.org/10.1145/2786805.2786845
Siegmund, N., et al.: Predicting performance via automated feature-interaction detection. In: 34th International Conference on Software Engineering, ICSE 2012, Zurich, Switzerland, 2–9 June 2012, pp. 167–177 (2012). https://doi.org/10.1109/ICSE.2012.6227196
Valov, P., Guo, J., Czarnecki, K.: Empirical comparison of regression methods for variability-aware performance prediction. In: Proceedings of the 19th International Conference on Software Product Line, SPLC 2015, Nashville, TN, USA, 20–24 July 2015, pp. 186–190 (2015). https://doi.org/10.1145/2791060.2791069
Valov, P., Petkovich, J., Guo, J., Fischmeister, S., Czarnecki, K.: Transferring performance prediction models across different hardware platforms. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ICPE 2017, L’Aquila, Italy, 22–26 April 2017, pp. 39–50 (2017). https://doi.org/10.1145/3030207.3030216
Xu, Y., Jia, X., Xuan, J.: Writing tests for this higher-order function first: automatically identifying future callings to assist testers. In: Proceedings of the 11th Asia-Pacific Symposium on Internetware (Internetware 2019), Fukuoka, Japan, 28–29 October 2019 (2019)
Xuan, J., Cornu, B., Martinez, M., Baudry, B., Seinturier, L., Monperrus, M.: B-refactoring: automatic test code refactoring to improve dynamic analysis. Inf. Softw. Technol. 76, 65–80 (2016). https://doi.org/10.1016/j.infsof.2016.04.016
Xuan, J., Gu, Y., Ren, Z., Jia, X., Fan, Q.: Genetic configuration sampling: learning a sampling strategy for fault detection of configurable systems. In: Proceedings of the 5th International Workshop on Genetic Improvement (GI@GECCO 2018), Kyoto, Japan, 15–19 July 2018 (2018). https://doi.org/10.1145/3205651.3208267
Zhang, Y., Guo, J., Blais, E., Czarnecki, K.: Performance prediction of configurable software systems by Fourier learning (T). In: 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, 9–13 November 2015, pp. 365–373 (2015). https://doi.org/10.1109/ASE.2015.15
Zuluaga, M., Krause, A., Püschel, M.: e-pal: an active learning approach to the multi-objective optimization problem. J. Mach. Learn. Res. 17, 1–104:32 (2016). http://jmlr.org/papers/v17/15-047.html
Acknowledgments
The work is supported by the National Key R&D Program of China under Grant No. 2018YFB1003901, the National Natural Science Foundation of China under Grant Nos. 61872273 and 61502345, the Open Research Fund Program of CETC Key Laboratory of Aerospace Information Applications under Grant No. SXX18629T022, and the Advance Research Projects of Civil Aerospace Technology, Intelligent Distribution Technology of Domestic Satellite Information, under Grant No. B0301.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, Y., Gu, Y., He, L., Xuan, J. (2020). Regression Models for Performance Ranking of Configurable Systems: A Comparative Study. In: Miao, H., Tian, C., Liu, S., Duan, Z. (eds) Structured Object-Oriented Formal Language and Method. SOFL+MSVL 2019. Lecture Notes in Computer Science(), vol 12028. Springer, Cham. https://doi.org/10.1007/978-3-030-41418-4_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-41418-4_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41417-7
Online ISBN: 978-3-030-41418-4
eBook Packages: Computer ScienceComputer Science (R0)