Abstract
Current experimentation platforms for online controlled experimentation focus on the technical execution of an experiment. This makes them specific to the application domain, the expected infrastructure, and the used technology. Moreover, the experiment definitions include numerous implicit assumptions about the platform’s implementation. As a result, experiments are difficult to replicate or compare across platforms or even platform versions.
This paper presents an experimentation infrastructure based on platform-independent experimentation of software changes. Experiments are defined technology-independently and the experimentation platform’s role is reduced to execution. The explicit definition of experiments in an independent artifact and the modular architecture of the services make experimentation replicable and the architecture open to change. Additionally, a lightweight approach to include the knowledge of past experiments is demonstrated. The infrastructure is presented by a running example experiment and its prototypical implementation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Convert Experiments and Pathmapp, tried to access on 2020-07-27.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
References
Auer, F., Felderer, M.: Current state of research on continuous experimentation: a systematic mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335–344. IEEE (2018)
Auer, F., Felderer, M.: Shifting quality assurance of machine learning algorithms to live systems. In: Tichy, M., Bodden, E., Kuhrmann, M., Wagner, S., Steghöfer, J.P. (eds.) Software Engineering und Software Management 2018, pp. 211–212. Gesellschaft für Informatik, Bonn (2018)
Auer, F., Felderer, M.: Evaluating the usefulness and ease of use of an experimentation definition language (S). In: GarcĂa-Castro, R. (ed.) The 32nd International Conference on Software Engineering and Knowledge Engineering, SEKE 2020, KSIR Virtual Conference Center, USA, 9–19 July 2020, pp. 158–163. KSI Research Inc. (2020). https://doi.org/10.18293/SEKE2020-067
Auer, F., Felderer, M.: Platform-independent online controlled experimentation. In: Winkler, D., Biffl, S., Mendez, D., Wimmer, M., Bergsmann, J. (eds.) Software Quality: Quality Intelligence in Software and Systems Engineering - 13th International Conference, SWQD 2021. Springer, Switzerland (2021). https://doi.org/10.1007/978-3-030-65854-0
Auer, F., Lee, C.S., Felderer, M.: Characteristics of continuous experiment definitions. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE (2020)
Fabijan, A., Dmitriev, P., Holmstrom Olsson, H., Bosch, J.: The online controlled experiment lifecycle. IEEE Softw. 37(2), 60–67 (2020)
Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development: From data to a data-driven organization at scale. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE (2017). https://doi.org/10.1109/icse.2017.76
Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: Online controlled experimentation at scale: an empirical survey on the current state of a/b testing. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 68–72. IEEE (2018)
Fernández, D.M., Graziotin, D., Wagner, S., Seibold, H.: Open science in software engineering. ArXiv abs/1904.06499 (2019)
Giaimo, F., Berger, C.: Design criteria to architect continuous experimentation for self-driving vehicles. In: 2017 IEEE International Conference on Software Architecture (ICSA), pp. 203–210. IEEE (2017)
Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., Xu, Y.: Trustworthy online controlled experiments: Five puzzling outcomes explained. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 786–794 (2012)
Lettner, F., Holzmann, C., Hutflesz, P.: Enabling A/B testing of native mobile applications by remote user interface exchange. In: Moreno-DĂaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2013. LNCS, vol. 8112, pp. 458–466. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-53862-9_58
Mattos, D.I., Bosch, J., Olsson, H.H.: Your system gets better every day you use it: towards automated continuous experimentation. In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 256–265. IEEE (2017)
Rissanen, O., Münch, J.: Continuous experimentation in the B2B domain: a case study. In: 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, pp. 12–18. IEEE (2015)
Schermann, G., Schöni, D., Leitner, P., Gall, H.C.: Bifrost: supporting continuous deployment with automated enactment of multi-phase live testing strategies. In: Proceedings of the 17th International Middleware Conference, pp. 1–14 (2016)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Auer, F., Felderer, M. (2021). An Infrastructure for Platform-Independent Experimentation of Software Changes. In: Bureš, T., et al. SOFSEM 2021: Theory and Practice of Computer Science. SOFSEM 2021. Lecture Notes in Computer Science(), vol 12607. Springer, Cham. https://doi.org/10.1007/978-3-030-67731-2_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-67731-2_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67730-5
Online ISBN: 978-3-030-67731-2
eBook Packages: Computer ScienceComputer Science (R0)