Skip to main content
SpringerLink
Log in

What should your run-time configuration framework do to help developers?

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The users or deployment engineers of a software system can adapt such a system to a wide range of deployment and usage scenarios by changing the value of configuration options, for example by disabling unnecessary features, tweaking performance-related parameters or specifying GUI preferences. However, the literature agrees that the flexibility of such options comes at a price: misconfigured options can lead a software system to crash in the production environment, while even in the absence of such configuration errors, a large number of configuration options makes a software system more complicated to deploy and use. In earlier work, we also found that developers who intend to make their application configurable face 22 challenges that impact their configuration engineering activities, ranging from technical to management-related or even inherent to the domain of configuration engineering. In this paper, we use a prototyping approach to derive and empirically evaluate requirements for tool support able to deal with 13 (primarily technical) configuration engineering challenges. In particular, via a set of interviews with domain experts, we identify four requirements by soliciting feedback on an incrementally evolving prototype. The resulting “Config2Code” prototype, which implements the four requirements, is then empirically evaluated via a user study involving 55 participants that comprises 10 typical configuration engineering tasks, ranging from the creation, comprehension, refactoring, and reviewing of configuration options to the quality assurance of options and debugging of configuration failures. A configuration framework satisfying the four requirements enables developers to perform more accurately and more swiftly in 70% and 60% (respectively) of the configuration engineering tasks than a state-of-the-practice framework not satisfying the requirements. Furthermore, such a framework allows to reduce the time taken for these tasks by up to 94.62%, being slower for only one task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Ansible (2018) Core java preferences api, http://www.ansible.com/. Accessed March 08, 2018

  • Apel S, Batory D, Kästner C, Saake G (2013) Feature-oriented software product lines: Concepts and implementation. Springer, Berlin/Heidelberg, p 308. ISBN 978-3-642-37520-0. http://www.springer.com/computer/swe/book/978-3-642-37520-0, Tech Rep

    Book  Google Scholar 

  • Bass L, Weber I, Zhu L (2015) Devops: a software architect’s perspective, 1st edn. Addison-Wesley Professional, Reading

    Google Scholar 

  • Behrang F, Cohen MB, Orso A (2015) Users beware: Preference inconsistencies ahead. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’15, New York, NY, USA, pp 295–306

  • Buse RP, Sadowski C, Weimer W (2011) Benefits and barriers of user evaluation in software engineering research. In: Proceedings of the International Conference on Object Oriented Programming Systems Languages and Applications, ser OOPSLA’11, pp 643–656

  • Cassoli J (2016) Web application with spring annotation-driven configuration: Rapidly develop lightweight Java web applications using Spring with annotations, 1st edn. CreateSpace Independent Publishing Platform, Scotts Valley

    Google Scholar 

  • Dong Z, Andrzejak A, Shao K (2015) Practical and accurate pinpointing of configuration errors using static analysis, Bremen, Germany, automated debugging;Configuration errors;Configuration options;Correlation degree;Misconfigurations;Run-time information;Software program;Testing oracles;. Online. Available https://doi.org/10.1109/ICSM.2015.7332463

  • Dong Z, Andrzejak A, Lo D, Costa D (2016) Orplocator: Identifying read points of configuration options via static analysis. In: Proceedings of the 27th international symposium on software reliability engineering, ser. ISSRE’16, pp 185–195

  • Huang P, Bolosky WJ, Singh A, Zhou Y (2015) Confvalley: a systematic configuration validation framework for cloud services. In: Proceedings of the tenth European conference on computer systems, ser. Eurosys ’15, NY, USA, pp 1–16

  • Hubaux A, Xiong Y, Czarnecki K (2012) A user survey of configuration challenges in linux and ecos. In: Proceedings of the 6th Int’l workshop on variability modeling of software-intensive systems, pp 149–155

  • Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional, Reading

    Google Scholar 

  • Jiang Y, Adams B (2015) Co-evolution of infrastructure and source code: an empirical study. In: Proceedings of the 12th Conference on Mining Software Repositories, ser. MSR’15, pp 45–55

  • Jin D, Qu X, Cohen MB, Robinson B (2014) Configurations everywhere: Implications for testing and debugging in practice. In: Proceedings of the 36th international conference on soft. eng., ser. ICSE’14, pp 215–224

  • Jin D, Cohen MB, Qu X, Robinson B (2014) Preffinder: Getting the right preference in configurable software systems. In: Proceedings of the 29th international conference on automated software engineering, ser. ASE ’14, pp 151–162

  • Khan M, Huang Z, Li M, Taylor GA, Khan M (2017) Optimizing hadoop parameter settings with gene expression programming guided pso. Concurrency and Computation: Practice and Experience 29(3):186–197

    Article  Google Scholar 

  • Ko AJ, LaToza T, Burnett M (2015) A practical guide to controlled experiments of software engineering tools with human participants. Empir Softw Eng 20(1):110–141

    Article  Google Scholar 

  • Krum S, Van Hevelingen W, Kero B, Turnbull J, McCune J (2013) Pro Puppet, 2nd edn. Apress, New York

    Book  Google Scholar 

  • Li W, Li S, Liao X, Xu X, Zhou S, Jia Z (2017) Conftest: Generating comprehensive misconfiguration for system reaction ability evaluation. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering, ser. EASE’17, NY, USA, pp 88–97

  • van der Linden F, Schmid K, Rommes E (2007) Software product lines in action: the best industrial practice in product line engineering. Springer, Berlin. Online. Available: https://books.google.ca/books?id=PC4LyoSNNAkC

    Book  Google Scholar 

  • Lwakatare LE, Kuvaja P, Oivo M (2016) Relationship of devops to agile, lean and continuous deployment. In: Abrahamsson P., Jedlitschka A., Nguyen Duc A., Felderer M., Amasaki S., Mikkonen T. (eds) Product-focused software process improvement. Springer International Publishing, Cham, pp 399–415

    Chapter  Google Scholar 

  • Medeiros F, Kästner C, Ribeiro M, Gheyi R, Apel S (2016) A comparison of 10 sampling algorithms for configurable systems. In: Proceedings of the 38th international conference on software engineering, ser. ICSE ’16. Online. Available: https://doi.org/10.1145/2884781.2884793. ACM, New York, pp 643–654

  • Meng X, Foong PS, Perrault S, Zhao S (2016) 5-step approach to designing controlled experiments. In: Proceedings of the international working conference on advanced visual interfaces, pp 358–359

  • Morris K (2016) Infrastructure as code: managing servers in the cloud, 1st edn. O’Reilly Media, Sebastopol

    Google Scholar 

  • Nadi S, Berger T, Kästner C, Czarnecki K (2014) Mining configuration constraints: Static analyses and empirical results. In: Proceedings of the 36th international conference on Soft. Eng., ser. ICSE’14, pp 140–151

  • Nadi S, Berger T, Kästner C, Czarnecki K (2015) Where do configuration constraints stem from? an extraction approach and an empirical study. IEEE Trans Softw Eng 41(8):820–841

    Article  Google Scholar 

  • Pohl K, Böckle G, van der Linden F (2005) Software product line engineering: foundations, principles and techniques. Springer, Berlin. Online. Available: https://books.google.ca/books?id=lsX8_O_TRkEC

    Book  Google Scholar 

  • Qu X, Acharya M, Robinson B (2011) Impact analysis of configuration changes for test case selection, ser. ISSRE’11, Hiroshima, Japan, pp 140–149

  • Rabkin A, Katz R (2011) Precomputing possible configuration error diagnoses, ser. ASE’11, Lawrence, KS, United States, pp 193–202

  • Rabiser R, Dhungana D (2007) Integrated support for product configuration and requirements engineering in product derivation, ser. EUROMICRO’07, Piscataway, USA, pp 193–200

  • Rahman MT, Querel L-P, Rigby PC, Adams B (2016) Feature toggles: a case study and survey. In: Proceedings of the 13th IEEE working conference on mining software repositories (MSR), Austin, TX, pp 201–211

  • Sarma A, Bortis G, van der Hoek A (2007) Towards supporting awareness of indirect conflicts across software configuration management workspaces. In: Proceedings of the 22 international conference on automated software engineering, ser. ASE’07, pp 94–103

  • Sayagh M, Adams B (2018) Videos, http://mcis.polymtl.ca/msayagh/userStudy/Config2Code/

  • Sayagh M, Kerzazi N, Adams B (2017a) On cross-stack configuration errors. In: Proceedings of the 39th international conference on software engineering, ser. ICSE ’17. IEEE Press, Piscataway, pp 255–265

  • Sayagh M, Dong Z, Andrzejak A, Adams B (2017b) Does the choice of configuration framework matter for developers? empirical study on 11 java configuration frameworks. In: Proceedings of the 17th international working conference on source code analysis and manipulation, ser. SCAM’17, pp 41–50

  • Sayagh M, Kerzazi N, Adams B, Petrillo F (2018) Software configuration engineering in practice - interviews, survey, and systematic literature review. Transactions on Software Engineering (TSE). Online. Available: mcis.polymtl.ca/publications/2018/tse_mohammed.pdf

  • Sedlmair M, Meyer M, Munzner T (2012) Design study methodology: Reflections from the trenches and the stacks. IEEE Trans Vis Comput Graphics 18 (12):2431–2440. Online. Available:

    Article  Google Scholar 

  • Sharma T, Fragkoulis M, Spinellis D (2016) Does your configuration code smell?. In: Proceedings of the 13th international conference on mining software repositories, ser. MSR’16, pp 189–200

  • Sommerville I (2010) Software engineering, 9th edn. Addison-Wesley Publishing Company, USA

    MATH  Google Scholar 

  • Wagner S, Fernández DM (2015) Analysing text in software projects. arXiv:1612.00164

    Chapter  Google Scholar 

  • Wettel R, Lanza M, Robbes R (2011) Software systems as cities: a controlled experiment. In: 2011 33rd international conference on software engineering (ICSE), pp 551–560

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell

    Book  Google Scholar 

  • Xu T, Zhou Y (2015) Systems approaches to tackling configuration errors: a survey. ACM Computing Surveys 47(4):70:1–70:41

    Article  Google Scholar 

  • Xu T, Jin L, Fan X, Zhou Y, Pasupathy S, Talwadker R (2015) Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’2015, NY, USA, pp 307–319

  • Xu T, Zhang J, Huang P, Zheng J, Sheng T, Yuan D, Zhou Y, Pasupathy S (2013) Do not blame users for misconfigurations. In: Proceedings of the 24 symposium on operating systems principles, pp 244–259

  • Yin Z, Ma X, Zheng J, Zhou Y, Bairavasundaram LN, Pasupathy S (2011) An empirical study on configuration errors in commercial and open source systems. In: Proceedings of the twenty-third symposium on operating systems principles, ser. SOSP’11, pp 159–172

  • Zhang S, Ernst M (2013) Automated diagnosis of software configuration errors. Piscataway, NJ, USA, pp 312–21, software configuration error diagnosis;ConfDiagnoser;configuration error identification;static analysis;dynamic profiling;statistical analysis;noncrashing configuration errors;configurable software system;Java;. Online. Available: https://doi.org/10.1109/ICSE.2013.6606577

  • Zhang S, Ernst M (2014) Which configuration option should i change?. In: Proceedings of the 36th international conference on software engineering, ser. ICSE’14, NY, USA, pp 152–163

Download references

Author information

Authors and Affiliations

  1. Queen’s University, Kingston, ON, K7L 3N6, Canada

    Mohammed Sayagh

  2. ENSIAS, Mohammed V University, Rabat, Morocco

    Noureddine Kerzazi

  3. Université du Québec à Chicoutimi, Chicoutimi, QC, G7H 2B1, Canada

    Fabio Petrillo

  4. Polytechnique Montreal, Montreal, QC, H3T 1J4, Canada

    Khalil Bennani & Bram Adams

Authors
  1. Mohammed Sayagh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noureddine Kerzazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fabio Petrillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Khalil Bennani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bram Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammed Sayagh.

Additional information

Communicated by: Romain Robbes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sayagh, M., Kerzazi, N., Petrillo, F. et al. What should your run-time configuration framework do to help developers?. Empir Software Eng 25, 1259–1293 (2020). https://doi.org/10.1007/s10664-019-09790-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09790-x

Keywords

Search

Navigation