skip to main content
10.1145/3175731.3176175acmotherconferencesArticle/Chapter ViewAbstractPublication Pagessisos-at-ecsaConference Proceedingsconference-collections
research-article

Towards systematic live experimentation in software-intensive systems of systems

Published: 29 November 2016 Publication History

Abstract

As the size, variation, and sophistication of software-intensive systems-of-systems grows, so does the uncertainty inherent to their design and development. To deal with this issue, we propose a framework for systematic experimentation based on declarative specification connected with system architecture. The focus is on how to specify experiments that allow systematic exploration of the space of alternative configurations at runtime. Since such experiments should be launched on live systems, extra care needs to be taken in preventing damages when experimenting with the systems. Therefore, we also focus on how to quantify the direct and the indirect cost associated with each experiment execution (which needs to be included in a cost-benefit analysis for system adaptation) and on how to gradually roll out an experiment via a number of different stages. We use the development of a route planner system as an example to motivate and exemplify our approach.

References

[1]
Bird, C. et al. 2011. Empirical Software Engineering at Microsoft Research. Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (New York, NY, USA, 2011), 143--150.
[2]
Boehm, B.W. 2006. Value-Based Software Engineering: Overview and Agenda. Value-Based Software Engineering. S. Biffl et al., eds. Springer Berlin Heidelberg. 3--14.
[3]
Bosch, J. and Olsson, H.H. 2016. Data-driven Continuous Evolution of Smart Systems. Proc. of SEAMS '16 (2016), 28--34.
[4]
Eichelberger, H. and Schmid, K. 2014. Flexible resource monitoring of Java programs. Journal of Systems and Software. 93, (Jul. 2014), 163--186.
[5]
Esfahani, N. et al. 2011. Taming uncertainty in self-adaptive software. Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (2011), 234--244.
[6]
van Hoorn, A. et al. 2009. Continuous Monitoring of Software Services: Design and Application of the Kieker Framework. Department of Computer Science, Kiel University, Germany.
[7]
Horký, V. et al. 2016. Analysis of Overhead in Dynamic Java Performance Monitoring. Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering (New York, NY, USA, 2016), 275--286.
[8]
Humble, J. and Farley, D. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley.
[9]
Kohavi, R. et al. 2009. Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice (2009).
[10]
Kreps, J. et al. 2011. Kafka: A distributed messaging system for log processing. Proceedings of the 6th International Workshop on Networking Meets Databases (NetDB'11) (2011), 1--7.
[11]
Malony, A.D. 1990. Performance Observability. University of Illinois at Urbana-Champaign.
[12]
Malony, A.D. and Shende, S.S. 2004. Overhead Compensation in Performance Profiling. Euro-Par 2004 Parallel Processing. M. Danelutto et al., eds. Springer Berlin Heidelberg. 119--132.
[13]
Parsons, T. et al. 2006. Non-intrusive end-to-end runtime path tracing for J2EE systems. Software, IEE Proceedings. 153, 4 (Aug. 2006), 149--161.
[14]
Schmid, S. et al. 2017. Self-Adaptation Based on Big Data Analytics: A Model Problem and Tool. Proc. of 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS'17) (Buenos Aires, Argentina, May 2017), 102--108.
[15]
Waller, J. et al. 2014. Application Performance Monitoring: Trade-Off between Overhead Reduction and Maintainability. Proceedings of the Symposium on Software Performance 2014 (Stuttgart, Germany, Nov. 2014), 1--24.
[16]
Wolf, T.D. 2007. Analysing and Engineering Self-Organising Emergent Applications. Katholieke Universiteit Leuven.

Cited By

View all
  • (2018)A Declarative Approach for Performance Tests Execution in Continuous Software Development EnvironmentsProceedings of the 2018 ACM/SPEC International Conference on Performance Engineering10.1145/3184407.3184417(261-272)Online publication date: 30-Mar-2018
  • (2018)Cost-Aware Stage-Based Experimentation: Challenges and Emerging Results2018 IEEE International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C.2018.00027(72-75)Online publication date: Apr-2018

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SiSoS@ECSA '16: Proceedings of the International Colloquium on Software-intensive Systems-of-Systems at 10th European Conference on Software Architecture
November 2016
69 pages
ISBN:9781450363990
DOI:10.1145/3175731
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 November 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. system architecture
  2. systematic experimentation
  3. uncertainty

Qualifiers

  • Research-article

Conference

SiSoS@ECSA 2016

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2018)A Declarative Approach for Performance Tests Execution in Continuous Software Development EnvironmentsProceedings of the 2018 ACM/SPEC International Conference on Performance Engineering10.1145/3184407.3184417(261-272)Online publication date: 30-Mar-2018
  • (2018)Cost-Aware Stage-Based Experimentation: Challenges and Emerging Results2018 IEEE International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C.2018.00027(72-75)Online publication date: Apr-2018

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media