Skip to main content
SpringerLink
Log in

ACE: Easy Deployment of Field Optimization Experiments

  • Conference paper
  • First Online:
Software Architecture (ECSA 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11681))

Included in the following conference series:

Abstract

Optimization of software parameters is a recurring activity in the life-cycle of many software products, from prototypes and simulations, test beds and hardware-in-the-loop scenarios, field calibrations to the evolution of continuous deployment cycles. To perform this activity, software companies require a combination of software developers and optimization experts with domain specific knowledge. Moreover, in each of life-cycle steps, companies utilize a plethora of different tools, tailored for specific domains or development stages. To most companies, this scenario leads to an excessive cost in the optimization of smaller features or in cases where it is not clear what the returned value will be.

In this work we present a new optimization system based on field experiments, that is aimed to facilitate the adoption of optimization in all stages of development. We provide two main contributions. First, we present the architecture of a new optimization system that allows existing software systems to perform optimization procedures in different domains and in different development stages. This optimization system utilizes domain-agnostic interfaces to allow existing systems to perform optimization procedures with minimal invasiveness and optimization expertise. Second, we provide an overview of the deployments, discuss the advantages and limitations and evaluate the optimization system in three empirical scenarios: (1) offline optimization with simulations; (2) optimization of a communication system in a test bed in collaboration with Ericsson; (3) live optimization of a mobile application in collaboration with Sony Mobile. We aim to provide practitioners with a single optimization tool that can leverage their optimization activities from offline to live systems, with minimal invasiveness and optimization expertise.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://mathworks.com.

  2. 2.

    https://vwo.com/.

  3. 3.

    https://optimizely.com.

  4. 4.

    https://www.software-center.se/.

  5. 5.

    https://www.docker.com.

  6. 6.

    https://www.nginx.com/.

  7. 7.

    http://flask.pocoo.org/.

  8. 8.

    https://ec.europa.eu/commission/priorities/justice-and-fundamental-rights/data-protection/2018-reform-eu-data-protection-rules_en.

References

  1. Piyaratna, S., et al.: Digital RF processing system for Hardware-in-the-loop simulation. In: 2013 International Conference on Radar, pp. 554–559 (2013)

    Google Scholar 

  2. Scholz, D., von Stryk, O.: Efficient design parameter optimization for musculoskeletal bipedal robots combining simulated and hardware-in-the-loop experiments. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 512–518, December 2015

    Google Scholar 

  3. Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2010, p. 17 (2010)

    Google Scholar 

  4. Bakshy, E., Park, M., Eckles, D., Park, M., Bernstein, M.S.: Designing and deploying online field experiments. In: Proceedings of 23rd International Conference of World Wide Web - WWW 2014, pp. 283–292, September 2014

    Google Scholar 

  5. Kohavi, R., Deng, A., Longbotham, R., Xu, Y.: Seven rules of thumb for web site experimenters. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2014, pp. 1857–1866 (2014)

    Google Scholar 

  6. Xu, Y., Duan, W., Huang, S.: SQR : balancing speed, quality and risk in online experiments. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining - KDD 2018, vol. 1, pp. 895–904 (2018)

    Google Scholar 

  7. Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems, pp. 2503–2511 (2015)

    Google Scholar 

  8. Issa Mattos, D., Bosch, J., Olsson, H.H.: Multi-armed bandits in the wild: pitfalls and strategies in online experiments. Inf. Softw. Technol. 113, 68–81 (2019)

    Article  Google Scholar 

  9. Mattos, D.I., Mårtensson, E., Bosch, J., Olsson, H.H.: Optimization experiments in the continuous space. In: Colanzi, T.E., McMinn, P. (eds.) SSBSE 2018. LNCS, vol. 11036, pp. 293–308. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99241-9_16

    Chapter  Google Scholar 

  10. Mattos, D.I., Bosch, J., Olsson, H.H., Dakkak, A., Bergh, K.: Automated optimization of software parameters in a long term evolution radio base station. In: IEEE 13th Annual International Systems Conference, pp. 1–8 (2019)

    Google Scholar 

  11. Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Disc. 18(1), 140–181 (2009). https://doi.org/10.1007/s10618-008-0114-1

    Article  MathSciNet  Google Scholar 

  12. Burtini, G., Loeppky, J., Lawrence, R.: A survey of online experiment design with the stochastic multi-armed bandit. pp. 1–49 (2015) arXiv:1510.00757

  13. Urban, G.L., et al.: Morphing banner advertising. Mark. Sci. 33(1), 27–46 (2014)

    Article  MathSciNet  Google Scholar 

  14. Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World wide web - WWW 2010, p. 661 (2010)

    Google Scholar 

  15. Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., Sculley, D.: Google vizier. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2017, pp. 1487–1495 (2017)

    Google Scholar 

  16. Bubeck, S., Munos, R., Stoltz, G., Szepesvari, C.: X-armed bandits. Theor. Comput. Sci. 412(19), 1832–1852 (2010)

    Article  Google Scholar 

  17. Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2016)

    Article  Google Scholar 

  18. Shang, X., Kaufmann, E., Valko, M.: Hierarchical bandits for “Black Box” optimization, Lille (2015)

    Google Scholar 

  19. Tamburrelli, G., Margara, A.: Towards automated A/B testing. In: Le Goues, C., Yoo, S. (eds.) SSBSE 2014. LNCS, vol. 8636, pp. 184–198. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09940-8_13

    Chapter  Google Scholar 

  20. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28(1), 75 (2004)

    Article  Google Scholar 

  21. Gregor, S., Hevner, A.R.: Positioning and presenting design science research for maximum impact. MIS Q. 37(2), 337–355 (2013)

    Article  Google Scholar 

  22. Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. Wiley, Hoboken (2012)

    Google Scholar 

  23. Krettek, J., Schauten, D., Hoffmann, F., Bertram, T.: Evolutionary hardware-in-the-loop optimization of a controller for cascaded hydraulic valves. In: 2007 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 1–6 (2007)

    Google Scholar 

  24. Zhao, Y., Dong, W., Zou, X., Tong, L., Zhu, G.: Analysis and design of power hardware-in-the-loop testing for 400-Hz inverters. In: Proceedings of 2017 12th IEEE Conference on Industrial Electronics and Applications. ICIEA 2017, pp. 1122–1126, February 2018

    Google Scholar 

  25. Gerostathopoulos, I., Uysal, A.N., Prehofer, C., Bures, T.: A tool for online experiment-driven adaptation. In: Proceedings - 2018 IEEE 3rd International Workshop on Foundations and Applications of Self Systems FAS*W 2018, pp. 100–105 (2019)

    Google Scholar 

  26. Gerostathopoulos, I., Prehofer, C., Bulej, L., Bures, T., Horky, V., Tuma, P.: Cost-aware stage-based experimentation : challenges and emerging results. In: 2018 IEEE International Conference on Software Architecture Companion, pp. 72–75 (2018)

    Google Scholar 

  27. Mattos, D.I., Bosch, J., Olsson, H. H.: Your system gets better every day you use it: towards automated continuous experimentation. In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), no. Ml, pp. 256–265 (2017)

    Google Scholar 

  28. Mattos, I., Bosch, J., Olsson, H.H.: More for less: automated experimentation in software-intensive systems. In: Felderer, M., Méndez Fernández, D., Turhan, B., Kalinowski, M., Sarro, F., Winkler, D. (eds.) PROFES 2017. LNCS, vol. 10611, pp. 146–161. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69926-4_12

    Chapter  Google Scholar 

  29. Deng, A., Shi, X.: Data-driven metric development for online controlled experiments. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2016, pp. 77–86 (2016)

    Google Scholar 

Download references

Acknowledgments

This work was partially supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and by the Software Center. The authors would also like to express their gratitude for all the support provided by Ericsson and Sony Mobile. We also would like to thank to the support and help from Anas Dakkak, Krister Bergh and Erling Mårtensson.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Chalmers University of Technology, Hörselgången 11, 412 96, Gothenburg, Sweden

    David Issa Mattos & Jan Bosch

  2. Department of Computer Science and Media Technology, Malmö University, Nordenskiöldsgatan, 211 19, Malmö, Sweden

    Helena Holmström Olsson

Authors
  1. David Issa Mattos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Bosch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Helena Holmström Olsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Issa Mattos .

Editor information

Editors and Affiliations

  1. Charles University, Prague, Czech Republic

    Tomas Bures

  2. University of Lille, Villeneuve d’Ascq, France

    Laurence Duchien

  3. University of L’Aquila, L’Aquila, Italy

    Paola Inverardi

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mattos, D.I., Bosch, J., Holmström Olsson, H. (2019). ACE: Easy Deployment of Field Optimization Experiments. In: Bures, T., Duchien, L., Inverardi, P. (eds) Software Architecture. ECSA 2019. Lecture Notes in Computer Science(), vol 11681. Springer, Cham. https://doi.org/10.1007/978-3-030-29983-5_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-29983-5_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-29982-8

  • Online ISBN: 978-3-030-29983-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation