Pavel Valov
Jianmei Guo
ABSTRACT
Many software systems provide configuration options relevant to users, which are often called features. Features influence functional properties of software systems as well as non-functional ones, such as performance and memory consumption. Researchers have successfully demonstrated the correlation between feature selection and performance. However, the generality of these performance models across different hardware platforms has not yet been evaluated.
We propose a technique for enhancing generality of performance models across different hardware environments using linear transformation. Empirical studies on three real-world software systems show that our approach is computationally efficient and can achieve high accuracy (less than 10% mean relative error) when predicting system performance across 23 different hardware platforms. Moreover, we investigate why the approach works by comparing performance distributions of systems and structure of performance models across different platforms.
- NIST/SEMATECH e-Handbook of Statistical Methods. http://www.itl.nist.gov/div898/handbook/.Google Scholar
- J. Antony. Design of Experiments for Engineers and Scientists. Butterworth-Heinemann, 2003.Google Scholar
- S. Balsamo and M. Marzolla. Performance Evaluation of UML Software Architectures with Multiclass Queueing Network Models. In Proceedings of the 5th International Workshop on Software and Performance. ACM, 2005. Google ScholarDigital Library
- V. Cortellessa and R. Mirandola. PRIMA-UML: a performance validation incremental methodology on early UML diagrams. Science of Computer Programming, July 2002. Google ScholarDigital Library
- J. Guo, K. Czarnecki, S. Apel, N. Siegmund, and A. Wasowski. Variability-aware performance prediction: A statistical learning approach. In Proc. ASE. IEEE, 2013. Google ScholarDigital Library
- K. Hoste, A. Phansalkar, L. Eeckhout, A. Georges, L. K. John, and K. De Bosschere. Performance prediction based on inherent program similarity. In Proceedings of the 15th International Conference on Parallel Architectures and Compilation Techniques, PACT '06, pages 114--122, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence, 206(0):79--111, Jan. 2014. Google ScholarDigital Library
- F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown. Algorithm runtime prediction: Methods & evaluation (extended abstract). In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), July 2015.Google Scholar
- M. Kowal, M. Tschaikowski, M. Tribastone, and I. Schaefer. Scaling Size and Parameter Spaces in Variability-Aware Software Performance Models (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2015.Google ScholarDigital Library
- D. Montgomery. Design and Analysis of Experiments. John Wiley & Sons, 2008.Google ScholarDigital Library
- J. C. Petkovich, A. Oliveira, Y. Zhang, T. Reidemeister, and S. Fischmeister. DataMill: A Distributed Heterogeneous Infrastructure For Robust Experimentation. Software: Practice and Experience, pages n/a--n/a, 2015.Google Scholar
- W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C (2nd Ed.): The Art of Scientific Computing. Cambridge Univ. Press, 1992.Google ScholarDigital Library
- B. D. Ripley. Stochastic simulation, volume 316. John Wiley & Sons, 2009.Google Scholar
- I. Sobol and Y. Levitan. A pseudo-random number generator for personal computers. Computers and Mathematics with Applications, 37(4--5):33--40, 1999.Google Scholar
- SQLite. SQLite. https://www.sqlite.org/. Accessed April. 15th, 2016.Google Scholar
- The Tukaani Project. XZ Utils. http://tukaani.org/xz/. Accessed April. 17th, 2016.Google Scholar
- E. Thereska, B. Doebel, A. X. Zheng, and P. Nobel. Practical performance models for complex, popular applications. SIGMETRICS Perform. Eval. Rev., 38(1):1--12, June 2010. Google ScholarDigital Library
- P. Valov, J. Guo, and K. Czarnecki. Empirical comparison of regression methods for variability-aware performance prediction. In Proceedings of the 19th International Conference on Software Product Line, SPLC '15, pages 186--190, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- VideoLAN Organization. x264, the best H.264/AVC encoder. http://www.videolan.org/developers/x264.html. Accessed April. 15th, 2016.Google Scholar
- D. Westermann, J. Happe, R. Krebs, and R. Farahbod. Automated inference of goal-oriented performance prediction functions. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pages 190--199, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- Y. Zhang, J. Guo, E. Blais, and K. Czarnecki. Performance prediction of configurable software systems by fourier learning. In Proceedings of the International Conference on Automated Software Engineering (ASE), 2015.Google ScholarDigital Library
Index Terms
- Transferring Performance Prediction Models Across Different Hardware Platforms
Recommendations
Transferring Pareto Frontiers across Heterogeneous Hardware Environments
ICPE '20: Proceedings of the ACM/SPEC International Conference on Performance EngineeringSoftware systems provide user-relevant configuration options called features. Features affect functional and non-functional system properties, whereas selections of features represent system configurations. A subset of configuration space forms a Pareto ...
Analytical Performance Estimation for Large-Scale Reconfigurable Dataflow Platforms
Next-generation high-performance computing platforms will handle extreme data- and compute-intensive problems that are intractable with today’s technology. A promising path in achieving the next leap in high-performance computing is to embrace ...
Temporal Performance Modelling of Serverless Computing Platforms
WoSC '20: Proceedings of the 2020 Sixth International Workshop on Serverless ComputingAnalytical performance models have been shown very efficient in analyzing, predicting, and improving the performance of distributed computing systems. However, there is a lack of rigorous analytical models for analyzing the transient behaviour of ...
Comments