Abstract
Context: Agile methods have become mainstream even in large-scale systems engineering companies that need to accommodate different development cycles of hardware and software. For such companies, requirements engineering is an essential activity that involves upfront and detailed analysis which can be at odds with agile development methods.
Objective: This paper presents a multiple case study with seven large-scale systems companies, reporting their challenges, together with best practices from industry. We also analyse literature about two popular large-scale agile frameworks, SAFe® and LeSS, to derive potential solutions for the challenges.
Method: Our results are based on 20 qualitative interviews, five focus groups, and eight cross-company workshops which we used to both collect and validate our results.
Results: We found 24 challenges which we grouped in six themes, then mapped to solutions from SAFe®, LeSS, and our companies, when available.
Conclusion: In this way, we contribute a comprehensive overview of RE challenges in relation to large-scale agile system development, evaluate the degree to which they have been addressed, and outline research gaps. We expect these results to be useful for practitioners who are responsible for designing processes, methods, or tools for large scale agile development as well as guidance for researchers.
Reprinted with permission from Elsevier. Originally published in Journal of Systems and Software, Volume 172, 2021. DOI: 10.1016/j.jss.2020.110851
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Ablett, R., Sharlin, E., Maurer, F., Denzinger, J., Schock, C.: Buildbot: Robotic monitoring of agile software development teams. In: RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication, pp. 931–936. IEEE (2007)
Abrahamsson, P.: Is management commitment a necessity after all in software process improvement? In: Proceedings of the 26th Euromicro Conference. EUROMICRO 2000. Informatics: Inventing the Future, vol. 2, pp. 246–253. IEEE (2000)
Abrahamsson, P.: Measuring the success of software process improvement: the dimensions. arXiv preprint arXiv:1309.4645 (2013)
Abrahamsson, P., Warsta, J., Siponen, M., Ronkainen, J.: New directions on agile methods: a comparative analysis. In: Proceedings of the International Conference on Software Engineering, pp. 244–254 (2003). DOI 10.1109/ICSE.2003.1201204
Abran, A.: Software metrics and software metrology. John Wiley & Sons (2010)
Agarwal, A., Shankar, R., Tiwari, M.: Modeling the metrics of lean, agile and leagile supply chain: An anp-based approach. European Journal of Operational Research 173(1), 211–225 (2006)
Agarwal, P.: Continuous scrum: agile management of saas products. In: Proceedings of the 4th India Software Engineering Conference, pp. 51–60 (2011)
Aghabozorgi, S., Shirkhorshidi, A.S., Wah, T.Y.: Time-series clustering–a decade review. Information Systems 53, 16–38 (2015)
Albuquerque, C., Antonino, P., Nakagawa, E.: An investigation into agile methods in embedded systems development. In: Computational Science and Its Applications, Lecture Notes in Computer Science, vol. 7335, pp. 576–591. Springer (2012). URL http://www.springerlink.com/content/38uk703767811277/abstract/
Allamanis, M., Barr, E.T., Bird, C., Sutton, C.: Learning natural coding conventions. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 281–293. ACM (2014)
Alshayeb, M., Li, W.: An empirical study of system design instability metric and design evolution in an agile software process. Journal of Systems and Software 74(3), 269–274 (2005)
Alyahya, S., Ivins, W.K., Gray, W.: A holistic approach to developing a progress tracking system for distributed agile teams. In: 2012 IEEE/ACIS 11th International Conference on Computer and Information Science, pp. 503–512. IEEE (2012)
Ambler, S.: Agile modeling: effective practices for extreme programming and the unified process. John Wiley & Sons (2002)
Ambler, S.W., Lines, M.: Disciplined Agile Delivery, 1 edn. IBM Press (2012). URL http://disciplinedagiledelivery.wordpress.com/
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. IEEE (2019)
Antinyan, V., Staron, M.: Rendex: A method for automated reviews of textual requirements. Journal of Systems and Software 131, 63–77 (2017)
Arazy, O., Kopak, R.: On the measurability of information quality. Journal of the American Society for Information Science and Technology 62(1), 89–99 (2011)
Arpteg, A., Brinne, B., Crnkovic-Friis, L., Bosch, J.: Software engineering challenges of deep learning. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50–59. IEEE (2018)
Auer, F., Felderer, M.: Current state of research on continuous experimentation: a systematic mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335–344. IEEE (2018)
Avgeriou, P., Kruchten, P., Ozkaya, I., Seaman, C.: Managing technical debt in software engineering (dagstuhl seminar 16162). In: Dagstuhl Reports, vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)
Axelsson, S., Baca, D., Feldt, R., Sidlauskas, D., Kacan, D.: Detecting defects with an interactive code review tool based on visualisation and machine learning. In: the 21st International Conference on Software Engineering and Knowledge Engineering (SEKE 2009) (2009)
Bach, J.: Exploratory Testing. https://www.satisfice.com/exploratory-testing (2020). [Online; accessed July 18, 2020]
Baldassarre, M.T., Caivano, D., Visaggio, G.: Comprehensibility and efficiency of multiview framework for measurement plan design. In: Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on, pp. 89–98. IEEE (2003)
Bardsiri, V.K., Jawawi, D.N.A., Hashim, S.Z.M., Khatibi, E.: Increasing the accuracy of software development effort estimation using projects clustering. IET Software 6(6), 461–473 (2012)
Barik, T., DeLine, R., Drucker, S., Fisher, D.: The bones of the system: A case study of logging and telemetry at microsoft. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 92–101. IEEE (2016)
Baskerville, R., Wood-Harper, A.T.: A Critical Perspective on Action Research as a Method for Information Systems Research. Journal of Information Technology 11(2), 235–246 (1996)
Basri, S., Dominic, D.D., Murugan, T., Almomani, M.A.: A proposed framework using exploratory testing to improve software quality in sme’s. In: International Conference of Reliable Information and Communication Technology, pp. 1113–1122. Springer (2018)
Batsaikhan, O., Lin, Y.: Building a shared understanding of customer value in a large-scale agile organization: A case study. Master’s thesis, Chalmers—University of Gothenburg, Dept. of Computer Science and Engineering (2018)
Baumeister, J., Reutelshoefer, J.: Developing knowledge systems with continuous integration. In: Proceedings of the 11th International Conference on Knowledge Management and Knowledge Technologies, pp. 1–4 (2011)
Beaumont, O., Bonichon, N., Courtès, L., Dolstra, E., Hanin, X.: Mixed data-parallel scheduling for distributed continuous integration. In: 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pp. 91–98. IEEE (2012)
Beck, K.: Embracing change with extreme programming. Computer 32(10), 70–77 (1999)
Beck, K.: Extreme programming explained: embrace change. addison-wesley professional (2000)
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., Thomas, D.: Manifesto for the Agile Software Development (2001)
Berger, C., Eklund, U.: Expectations and challenges from scaling agile in mechatronics-driven companies–a comparative case study. In: International Conference on Agile Software Development, pp. 15–26. Springer (2015)
Bernardi, L., Mavridis, T., Estevez, P.: 150 successful machine learning models: 6 lessons learned at booking. com. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1743–1751 (2019)
Besker, T., Martini, A., Bosch, J.: A systematic literature review and a unified model of atd. In: 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 189–197. IEEE (2016)
Besker, T., Martini, A., Bosch, J.: The pricey bill of technical debt: When and by whom will it be paid? In: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 13–23. IEEE (2017)
Bisio, R., Malabocchia, F.: Cost estimation of software projects through case base reasoning. In: International Conference on Case-Based Reasoning, pp. 11–22. Springer (1995)
Bjarnason, E., Unterkalmsteiner, M., Borg, M., Engström, E.: A multi-case study of agile requirements engineering and the use of test cases as requirements. Information and Software Technology 77, 61–79 (2016)
Bjarnason, E., Wnuk, K., Regnell, B.: A case study on benefits and side-effects of agile practices in large-scale requirements engineering. In: proceedings of the 1st workshop on agile requirements engineering, pp. 1–5 (2011)
Boehm, B.: Get ready for agile methods, with care. Computer 35(1), 64–69 (2002)
Boehm, B.W., et al.: Software engineering economics, vol. 197. Prentice-hall Englewood Cliffs (NJ) (1981)
Boetticher, G., Menzies, T., Ostrand, T.: Promise repository of empirical software engineering data. West Virginia University, Department of Computer Science (2007)
Booch, G.: Object oriented design with applications. Benjamin-Cummings Publishing Co., Inc. (1990)
Bosch, J.: Building products as innovation experiment systems. In: International Conference of Software Business, pp. 27–39. Springer (2012)
Bosch, J., Eklund, U.: Eternal embedded software: Towards innovation experiment systems. In: International Symposium On Leveraging Applications of Formal Methods, Verification and Validation, pp. 19–31. Springer (2012)
Bosch, J., Olsson, H.H., Crnkovic, I.: It takes three to tango: Requirement, outcome/data, and ai driven development. In: SiBW, pp. 177–192 (2018)
Bosch, J., Olsson, H.H., Crnkovic, I.: Engineering ai systems: A research agenda. In: Artificial Intelligence Paradigms for Smart Cyber-Physical Systems, pp. 1–19. IGI Global (2021)
Bosch-Sijtsema, P., Bosch, J.: User involvement throughout the innovation process in high-tech industries. Journal of Product Innovation Management 32(5), 793–807 (2015)
Bowyer, J., Hughes, J.: Assessing undergraduate experience of continuous integration and test-driven development. In: Proceedings of the 28th international conference on Software engineering, pp. 691–694 (2006)
Brar, H.K., Kaur, P.J.: Static analysis tools for security: A comparative evaluation. International Journal 5(7) (2015)
Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative research in psychology 3(2), 77–101 (2006)
Briand, L., El Emam, K., Morasca, S.: Theoretical and empirical validation of software product measures. International Software Engineering Research Network, Technical Report ISERN-95-03 (1995)
Briand, L.C.: Novel applications of machine learning in software testing. In: 2008 The Eighth International Conference on Quality Software, pp. 3–10. IEEE (2008)
Briand, L.C., Morasca, S., Basili, V.R.: Property-based software engineering measurement. Software Engineering, IEEE Transactions on 22(1), 68–86 (1996)
Briand, L.C., Wüst, J., Daly, J.W., Victor Porter, D.: Exploring the relationships between design measures and software quality in object-oriented systems. Journal of systems and software 51(3), 245–273 (2000)
Brooks, G.: Team pace keeping build times down. In: Agile 2008 Conference, pp. 294–297. IEEE (2008)
Brown, N., Cai, Y., Guo, Y., Kazman, R., Kim, M., Kruchten, P., Lim, E., MacCormack, A., Nord, R., Ozkaya, I., et al.: Managing technical debt in software-reliant systems. In: Proceedings of the FSE/SDP workshop on Future of software engineering research, pp. 47–52 (2010)
Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: Proceedings of the 26th International Conference on Software Engineering, ICSE ’04, pp. 480–490. IEEE Computer Society, Washington, DC, USA (2004). URL http://dl.acm.org/citation.cfm?id=998675.999452
Bruneliere, H., Burger, E., Cabot, J., Wimmer, M.: A feature-based survey of model view approaches. Software & Systems Modeling (2017). DOI 10.1007/s10270-017-0622-9
Buglione, L., Abran, A.: Introducing root-cause analysis and orthogonal defect classification at lower cmmi maturity levels. Proc. MENSURA p. 29 (2006)
Bures, M., Frajtak, K., Ahmed, B.S.: Tapir: Automation support of exploratory testing using model reconstruction of the system under test. IEEE Transactions on Reliability 67(2), 557–580 (2018)
Calpur, M.C., Arca, S., Calpur, T.C., Yilmaz, C.: Model dressing for automated exploratory testing. In: 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 577–578. IEEE (2017)
Campbell-Pretty, E.: Tribal unity: Getting from teams to tribes by creating a one team culture (2016)
Cannizzo, F., Clutton, R., Ramesh, R.: Pushing the boundaries of testing and continuous integration. In: Agile 2008 Conference, pp. 501–505. IEEE (2008)
Castellion, G.: Do it wrong quickly: how the web changes the old marketing rules by mike moran (2008)
Catal, C., Diri, B.: A systematic review of software fault prediction studies. Expert systems with applications 36(4), 7346–7354 (2009)
Chappelly, T., Cifuentes, C., Krishnan, P., Gevay, S.: Machine learning for finding bugs: An initial report. In: Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE), IEEE Workshop on, pp. 21–26. IEEE (2017)
Chow, T., Cao, D.B.: A survey study of critical success factors in agile software projects. Journal of systems and software 81(6), 961–971 (2008)
Cicchetti, A., Ciccozzi, F., Pierantonio, A.: Multi-view approaches for software and system modelling: a systematic literature review. Software & Systems Modeling pp. 1–27 (2019). DOI 10.1007/s10270-018-00713-w
Clancy, T.: The standish group report. Chaos report (1995)
Cockburn, A.: Agile software development: the cooperative game. Pearson Education (2006)
Codabux, Z., Williams, B.: Managing technical debt: An industrial case study. In: 2013 4th International Workshop on Managing Technical Debt (MTD), pp. 8–15. IEEE (2013)
Cohan, S.: Successful customer collaboration resulting in the right product for the end user. In: Agile 2008 Conference, pp. 284–288. IEEE (2008)
Cook, T.D., Campbell, D.T., Day, A.: Quasi-experimentation: Design & analysis issues for field settings, vol. 351. Houghton Mifflin Boston (1979)
Cossio, M., et al.: A/b testing-the most powerful way to turn clicks into customers, vol (2012)
Mascarenhas Hornos da Costa, J., Oehmen, J., Rebentisch, E., Nightingale, D.: Toward a better comprehension of lean metrics for research and product development management. R&D Management (2014)
Crook, T., Frasca, B., Kohavi, R., Longbotham, R.: Seven pitfalls to avoid when running controlled experiments on the web. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105–1114. ACM (2009)
Cunningham, W.: The wycash portfolio management system. ACM SIGPLAN OOPS Messenger 4(2), 29–30 (1992)
Cusomano, M., Selby, R.: Microsoft secrets—how the world’s most powerful software company creates technology, shapes markets, and manages people (1995)
Dagan, I., Engelson, S.P.: Committee-based sampling for training probabilistic classifiers. In: Machine Learning Proceedings 1995, pp. 150–157. Elsevier (1995)
Dahlmeier, D.: On the challenges of translating nlp research into commercial products. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 92–96 (2017)
Dajsuren, Y., Gerpheide, C., Serebrenik, A., Wijs, A., Vasilescu, B., van den Brand, M.: Formalizing Correspondence Rules for Automotive Architecture Views. In: Proceedings of the 10th international ACM Sigsoft conference on Quality of software architectures, pp. 129–138. ACM (2014). DOI 10.1145/2602576.2602588
Daskalantonakis, M.K., Yacobellis, R.H., Basili, V.R.: A method for assessing software measurement technology. Quality Engineering 3(1), 27–40 (1990)
Davis, A.M.: Just Enough Requirements Management: Where Software Development Meets Marketing. Dorset House Publishing (2005)
Desharnais, J.M., Abran, A.: How to succesfully implement a measurement program: From theory to practice. In: Metrics in Software Evolution, pp. 11–38. Oldenbourg Verlag, Oldenburg (1995)
Dess, G.G., Shaw, J.D.: Voluntary turnover, social capital, and organizational performance. Academy of Management Review 26(3), 446–456 (2001)
D’haeseleer, P.: What are dna sequence motifs? Nature biotechnology 24(4), 423 (2006)
Di Nucci, D., Palomba, F., Tamburri, D.A., Serebrenik, A., De Lucia, A.: Detecting code smells using machine learning techniques: are we there yet? In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 612–621. IEEE (2018)
Diaz-Ley, M., Garcia, F., Piattini, M.: Implementing a software measurement program in small and medium enterprises: a suitable framework. IET Software 2(5), 417–436 (2008)
Dikert, K., Paasivaara, M., Lassenius, C.: Challenges and success factors for large-scale agile transformations: A systematic literature review. Journal of Systems and Software 119, 87–108 (2016)
Dingsyr, T., Nerur, S., Balijepally, V., Moe, N.B.: A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software 85(6), 1213–1221 (2012). DOI 10.1016/j.jss.2012.02.033. URL http://www.sciencedirect.com/science/article/pii/S0164121212000532
Dösinger, S., Mordinyi, R., Biffl, S.: Communicating continuous integration servers for increasing effectiveness of automated testing. In: 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 374–377. IEEE (2012)
Downs, J., Hosking, J., Plimmer, B.: Status communication in agile software teams: A case study. In: 2010 Fifth International Conference on Software Engineering Advances, pp. 82–87. IEEE (2010)
Downs, J., Plimmer, B., Hosking, J.G.: Ambient awareness of build status in collocated software teams. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 507–517. IEEE (2012)
Dubinsky, Y., Talby, D., Hazzan, O., Keren, A.: Agile metrics at the israeli air force. In: Agile Conference, 2005. Proceedings, pp. 12–19. IEEE (2005)
Durelli, V.H., Durelli, R.S., Borges, S.S., Endo, A.T., Eler, M.M., Dias, D.R., Guimarães, M.P.: Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 68(3), 1189–1212 (2019)
Durisic, D., Staron, M., Tichy, M., Hansson, J.: Assessing the impact of meta-model evolution: a measure and its automotive application. Software & Systems Modeling 18(2), 1419–1445 (2019)
Duvall, P.M., Matyas, S., Glover, A.: Continuous integration: improving software quality and reducing risk. Pearson Education (2007)
Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: A systematic review. Information and Software Technology 50(9-10), 833–859 (2008). DOI 10.1016/j.infsof.2008.01.006. URL http://www.sciencedirect.com/science/article/pii/S0950584908000256
Dyer, R., Nguyen, H.A., Rajan, H., Nguyen, T.N.: Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 422–431. IEEE Press (2013)
Dzamashvili Fogelström, N., Gorschek, T., Svahnberg, M., Olsson, P.: The impact of agile principles on market-driven software product development. Journal of software maintenance and evolution: Research and practice 22(1), 53–80 (2010)
Ebert, C., Paasivaara, M.: Scaling agile. Ieee Software 34(6), 98–103 (2017)
Egyed, A.: Automatically Detecting and Tracking Inconsistencies in Software Design Models. IEEE Transactions on Software Engineering 37(2), 188–204 (2010). DOI 10.1109/tse.2010.38
Ehrig, H., Ehrig, K., Hermann, F.: From Model Transformation to Model Integration based on the Algebraic Approach to Triple Graph Grammars. Electronic Communications of the EASST 10 (2008)
Eiffel protocol. https://github.com/eiffel-community/eiffel
Eisenhardt, K.M.: Building theories from case study research. Academy of management review 14(4), 532–550 (1989)
Eklund, U., Olsson, H.H., Strøm, N.J.: Industrial challenges of scaling agile in mass-produced embedded systems. In: International Conference on Agile Software Development, pp. 30–42. Springer (2014)
Emanuelsson, P., Nilsson, U.: A comparative study of industrial static analysis tools. Electronic notes in theoretical computer science 217, 5–21 (2008)
Ernst, N.A., Bellomo, S., Ozkaya, I., Nord, R.L., Gorton, I.: Measure it? manage it? ignore it? software practitioners and technical debt. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 50–60 (2015)
Esling, P., Agon, C.: Time-series data mining. ACM Computing Surveys (CSUR) 45(1), 12 (2012)
ETSI: 3GPP Technical Specification Release 14 - ETSI TS 136 300. Tech. Rep. Release 14, ETSI, Valbonne, France (2017)
Evbota, F., Knauss, E., Sandberg, A.: Scaling up the planning game: Collaboration challenges in large-scale agile product development. In: International Conference on Agile Software Development, pp. 28–38. Springer, Cham (2016)
Fabijan, A., Dmitriev, P., McFarland, C., Vermeer, L., Holmström Olsson, H., Bosch, J.: Experimentation growth: Evolving trustworthy a/b testing capabilities in online software companies. Journal of Software: Evolution and Process 30(12), e2113 (2018)
Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 770–780. IEEE (2017)
Fabijan, A., Olsson, H.H., Bosch, J.: Customer feedback and data collection techniques in software r&d: a literature review. In: International Conference of Software Business, pp. 139–153. Springer (2015)
Fabijan, A., Olsson, H.H., Bosch, J.: The lack of sharing of customer data in large software organizations: challenges and implications. In: International Conference on Agile Software Development, pp. 39–52. Springer (2016)
Fabijan, A., Olsson, H.H., Bosch, J.: Time to say’good bye’: Feature lifecycle. In: 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 9–16. IEEE (2016)
Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st international workshop on rapid continuous software engineering, pp. 26–35 (2014)
Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The right model for continuous experimentation. Journal of Systems and Software 123, 292–305 (2017)
Fatima, A., Bibi, S., Hanif, R.: Comparative study on static code analysis tools for c/c++. In: Applied Sciences and Technology (IBCAST), 2018 15th International Bhurban Conference on, pp. 465–469. IEEE (2018)
Feldmann, S., Herzig, S., Kernschmidt, K., Wolfenstetter, T., Kammerl, D., Qamar, A., Lindemann, U., Krcmar, H., Paredis, C., Vogel-Heuser, B.: A Comparison of Inconsistency Management Approaches Using a Mechatronic Manufacturing System Design Case Study. In: 2015 IEEE International Conference on Automation Science and Engineering (CASE), pp. 158–165. IEEE (2015). DOI 10.1109/coase.2015.7294055
Feldmann, S., Wimmer, M., Kernschmidt, K., Vogel-Heuser, B.: A Comprehensive Approach for Managing Inter-Model Inconsistencies in Automated Production Systems Engineering. In: 2016 IEEE International Conference on Automation Science and Engineering (CASE), pp. 1120–1127. IEEE (2016). DOI 10.1109/coase.2016.7743530
Fenton, N., Bieman, J.: Software metrics: a rigorous and practical approach. CRC Press (2014)
Feyh, M., Petersen, K.: Lean software development measures and indicators-a systematic mapping study. In: Lean Enterprise Software and Systems, pp. 32–47. Springer (2013)
Fisher, R.A.: On the Interpretation of χ2 from Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society 85(1), 87 (1922). DOI 10.2307/2340521. URL http://www.jstor.org/stable/2340521?origin=crossref
Fitzgerald, B., Stol, K.J.: Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123, 176–189 (2017)
Fitzgerald, B., Stol, K.J., O’Sullivan, R., O’Brien, D.: Scaling agile methods to regulated environments: An industry case study. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 863–872. IEEE (2013)
Flick, U.: An introduction to qualitative research. Sage Publications Ltd (2009)
Flick, U.: Designing qualitative research. Sage (2018)
Fontana, F.A., Mäntylä, M.V., Zanoni, M., Marino, A.: Comparing and experimenting machine learning techniques for code smell detection. Empirical Software Engineering 21(3), 1143–1191 (2016)
Fontana, F.A., Roveda, R., Zanoni, M.: Tool support for evaluating architectural debt of an existing system: An experience report. In: Proceedings of the 31st Annual ACM Symposium on Applied Computing, pp. 1347–1349 (2016)
Fontana, F.A., Zanoni, M., Marino, A., Mantyla, M.V.: Code smell detection: Towards a machine learning-based approach. In: Software Maintenance (ICSM), 2013 29th IEEE International Conference on, pp. 396–399. IEEE (2013)
Fowler, K.: Mission-critical and safety-critical development. IEEE Instrumentation & Measurement Magazine 7(4), 52–59 (2004)
Fowler, M.: Continuous Integration. https://martinfowler.com/articles/continuousIntegration.html (2006). [Online; accessed 30-January-2013]
Frajtak, K., Bures, M., Jelinek, I.: Model-based testing and exploratory testing: Is synergy possible? In: 2016 6th International Conference on IT Convergence and Security (ICITCS), pp. 1–6. IEEE (2016)
Frajtak, K., Bures, M., Jelinek, I.: Exploratory testing supported by automated reengineering of model of the system under test. Cluster Computing 20(1), 855–865 (2017)
Bernard Nicolau de França, B., Horta Travassos, G.: Simulation based studies in software engineering: A matter of validity. CLEI electronic journal 18(1), 5–5 (2015)
Freitas, A.A.: Comprehensible classification models: a position paper. ACM SIGKDD explorations newsletter 15(1), 1–10 (2014)
Fu, Q., Zhu, J., Hu, W., Lou, J.G., Ding, R., Lin, Q., Zhang, D., Xie, T.: Where do developers log? an empirical study on logging practices in industry. In: Companion Proceedings of the 36th International Conference on Software Engineering, pp. 24–33 (2014)
Fu, T.c.: A review on time series data mining. Engineering Applications of Artificial Intelligence 24(1), 164–181 (2011)
Fu, Y., Zhu, X., Li, B.: A survey on instance selection for active learning. Knowledge and information systems 35(2), 249–283 (2013)
Gatrell, M., Counsell, S., Hall, T.: Empirical support for two refactoring studies using commercial c# software. In: 13th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1–10 (2009)
Gebizli, C.S., Sözer, H.: Improving models for model-based testing based on exploratory testing. In: 2014 IEEE 38th International Computer Software and Applications Conference Workshops, pp. 656–661. IEEE (2014)
Gebizli, C.Ş., Sözer, H.: Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal 25(3), 979–1005 (2017)
Gebizli, C.Ş., Sözer, H.: Impact of education and experience level on the effectiveness of exploratory testing: An industrial case study. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 23–28. IEEE (2017)
Geels, F.W., Kemp, R.: Dynamics in socio-technical systems: Typology of change processes and contrasting case studies. Technology in Society 29(4), 441 – 455 (2007). DOI http://dx.doi.org/10.1016/j.techsoc.2007.08.009
Gestwicki, P.: The entity system architecture and its application in an undergraduate game development studio. In: Proceedings of the International Conference on the Foundations of Digital Games, pp. 73–80 (2012)
Ghazi, A.N., Garigapati, R.P., Petersen, K.: Checklists to support test charter design in exploratory testing. In: International Conference on Agile Software Development, pp. 251–258. Springer (2017)
Ghazi, A.N., Petersen, K., Bjarnason, E., Runeson, P.: Levels of exploration in exploratory testing: From freestyle to fully scripted. IEEE Access 6, 26416–26423 (2018)
Gibbs, G.R.: Analyzing qualitative data, vol. 6. Sage (2018)
Gilb, T.: Software metrics. Winthrop Publishers (1977)
Goldratt, E.M., Cox, J.: The goal: a process of ongoing improvement. Routledge (2016)
Goodman, D., Elbaz, M.: ”it’s not the pants, it’s the people in the pants” learnings from the gap agile transformation what worked, how we did it, and what still puzzles us. In: Agile 2008 Conference, pp. 112–115. IEEE (2008)
Goodman, L.A.: Snowball Sampling. The Annals of Mathematical Statistics 32(1), 148–170 (1961)
Goodman, P.: Practical implementation of software metrics. International software quality assurance series. McGraw-Hill, London (1993). Lc92042989 Paul Goodman
Goodman, P.S., Bazerman, M., Conlon, E.: Institutionalization of planned organizational change. In: Research in Organizational Behavior, pp. 215–246. JAI Press,Greenwich (1980)
Goodman, P.S., Dean Jr, J.W.: Creating long-term organizational change. In: Change In Organizations. Carnegie-Mellon Univ Pittsburgh, PA, Graduate School of Industiral Administration (1982)
Goodman, R.M., Steckler, A.: A framework for assessing program institutionalization. Knowledge in Society 2(1), 57–71 (1989)
Gould, E., Marcus, A.: Company culture audit to improve development team’s collaboration, communication, and cooperation. In: Design, user experience, and usability. Theory, methods, tools and practice, pp. 415–424. Springer (2011)
Gregory, J., Crispin, L.: More agile testing: learning journeys for the whole team. Addison-Wesley Professional (2014)
Gryce, C., Finkelstein, A., Nentwich, C.: Lightweight Checking for UML Based Software Development. In: Workshop on Consistency Problems in UML-based Software Development., Dresden, Germany (2002)
Guinan, P.J., Cooprider, J.G., Faraj, S.: Enabling software development team performance during requirements definition: A behavioral versus technical approach. Information Systems Research 9(2), 101–125 (1998)
Guo, Y., Seaman, C., Gomes, R., Cavalcanti, A., Tonin, G., Da Silva, F.Q., Santos, A.L., Siebra, C.: Tracking technical debt—an exploratory case study. In: 2011 27th IEEE international conference on software maintenance (ICSM), pp. 528–531. IEEE (2011)
Guo, Y., Spínola, R.O., Seaman, C.: Exploring the costs of technical debt management–a case study. Empirical Software Engineering 21(1), 159–182 (2016)
Gyimothy, T., Ferenc, R., Siket, I.: Empirical validation of object-oriented metrics on open source software for fault prediction. Software Engineering, IEEE Transactions on 31(10), 897–910 (2005)
Hadar, E., Hassanzadeh, A.: Big data analytics on cyber attack graphs for prioritizing agile security requirements. In: 2019 IEEE 27th International Requirements Engineering Conference (RE), pp. 330–339 (2019). DOI 10.1109/RE.2019.00042
Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S.: A systematic literature review on fault prediction performance in software engineering. Software Engineering, IEEE Transactions on 38(6), 1276–1304 (2012)
Hanssen, G.K., Haugset, B., Stålhane, T., Myklebust, T., Kulbrandstad, I.: Quality assurance in scrum applied to safety critical software. In: International Conference on Agile Software Development, pp. 92–103. Springer, Cham (2016)
Hartmann, D., Dymond, R.: Appropriate agile measurement: using metrics and diagnostics to deliver business value. In: Agile Conference, 2006, pp. 6–pp. IEEE (2006)
Hatcher, W.G., Yu, W.: A survey of deep learning: Platforms, applications and emerging research trends. IEEE Access 6, 24411–24432 (2018)
Hause, M.: The SysML Modelling Language. In: Fifteenth European Systems Engineering Conference, vol. 9, pp. 1–12. Citeseer (2006)
Heidenberg, J., Porres, I.: Metrics functions for kanban guards. In: Engineering of Computer Based Systems (ECBS), 2010 17th IEEE International Conference and Workshops on, pp. 306–310. IEEE (2010)
Heidenberg, J., Weijola, M., Mikkonen, K., Porres, I.: A metrics model to measure the impact of an agile transformation in large software development organizations. In: International Conference on Agile Software Development, pp. 165–179. Springer (2013)
Heikkilä, V.T., Damian, D., Lassenius, C., Paasivaara, M.: A mapping study on requirements engineering in agile software development. In: 2015 41st Euromicro conference on software engineering and advanced applications, pp. 199–207. IEEE (2015)
Heikkilä, V.T., Paasivaara, M., Lasssenius, C., Damian, D., Engblom, C.: Managing the requirements flow from strategy to release in large-scale agile development: a case study at ericsson. Empirical Software Engineering 22(6), 2892–2936 (2017)
Hellmann, T.D., Maurer, F.: Rule-based exploratory testing of graphical user interfaces. In: 2011 Agile Conference, pp. 107–116. IEEE (2011)
Hendrickson, E.: Explore it!: reduce risk and increase confidence with exploratory testing. Pragmatic Bookshelf (2013)
Herzig, S., Qamar, A., Paredis, C.: An approach to Identifying Inconsistencies in Model-Based Systems Engineering. Procedia Computer Science 28, 354–362 (2014). DOI 10.1016/j.procs.2014.03.044
Hetzel, B.: Making software measurement work: Building an effective measurement program. John Wiley & Sons, Inc. (1993)
Highsmith, J., Cockburn, A.: Agile software development: The business of innovation. Computer 34(9), 120–127 (2001)
Hill, J.H., Schmidt, D.C., Porter, A.A., Slaby, J.M.: Cicuts: combining system execution modeling tools with continuous integration environments. In: 15th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ecbs 2008), pp. 66–75. IEEE (2008)
Hochstein, L., Basili, V.R., Zelkowitz, M.V., Hollingsworth, J.K., Carver, J.: Combining self-reported and automatic data to improve programming effort measurement. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 356–365. ACM (2005)
Hoda, R., Noble, J., Marshall, S.: Self-organizing roles on agile software development teams. IEEE Transactions on Software Engineering 39(3), 422–444 (2013). DOI 10.1109/TSE.2012.30
Hoffman, B., Cole, D., Vines, J.: Software process for rapid development of hpc software using cmake. In: 2009 DoD high performance computing modernization program users group conference, pp. 378–382. IEEE (2009)
Hohnhold, H., O’Brien, D., Tang, D.: Focusing on the long-term: It’s good for users and business. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1849–1858 (2015)
Holck, J., Jørgensen, N., et al.: Continuous integration and quality assurance: A case study of two open source projects. Australasian Journal of Information Systems 11(1) (2003)
Holmes, A., Kellogg, M.: Automating functional tests using selenium. In: AGILE 2006 (AGILE’06), pp. 6–pp. IEEE (2006)
Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”. In: Proceeding of the Euromicro Conference on Software Engineering and Advanced Applications. Cesme, Izmir, Turkey (2012)
Holvitie, J., Leppänen, V.: Debtflag: Technical debt management with a development environment integrated tool. In: 2013 4th International Workshop on Managing Technical Debt (MTD), pp. 20–27. IEEE (2013)
Holvitie, J., Leppänen, V., Hyrynsalmi, S.: Technical debt and the effect of agile software development practices on it-an industry practitioner survey. In: 2014 Sixth International Workshop on Managing Technical Debt, pp. 35–42. IEEE (2014)
Horkoff, J., Lindman, J., Hammouda, I., Knauss, E.: Experiences applying e3 value modeling in a cross-company study. In: International conference on conceptual modeling, pp. 610–625. Springer (2018)
Huang, H.Y., Liu, H.H., Li, Z.J., Zhu, J.: Surrogate: A simulation apparatus for continuous integration testing in service oriented architecture. In: 2008 IEEE International Conference on Services Computing, vol. 2, pp. 223–230. IEEE (2008)
Huang, Q., Shihab, E., Xia, X., Lo, D., Li, S.: Identifying self-admitted technical debt in open source projects using text mining. Empirical Software Engineering 23(1), 418–451 (2018)
Hudson, J., Denzinger, J.: Risk management for self-adapting self-organizing emergent multi-agent systems performing dynamic task fulfillment. Autonomous Agents and Multi-Agent Systems 29(5), 973–1022 (2015)
Humble, J., Farley, D.: Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education (2010)
Humphrey, W.S., Chick, T.A., Nichols, W.R., Pomeroy-Huff, M.: Team software process(tsp) body of knowledge (bok). Tech. rep., Carnegie Mellon University (2010)
Huzar, Z., Kuzniarz, L., Reggio, G., Sourrouille, J.L.: Consistency Problems in UML-Based Software Development. In: UML Modeling Languages and Applications, pp. 1–12. Springer (2005). DOI 10.1007/978-3-540-31797-5_1
Idri, A., Abran, A.: Evaluating software project similarity by using linguistic quantifier guided aggregations. In: IFSA World Congress and 20th NAFIPS International Conference, 2001. Joint 9th, vol. 1, pp. 470–475. IEEE (2001)
Idri, A., azzahra Amazal, F., Abran, A.: Analogy-based software development effort estimation: A systematic mapping and review. Information and Software Technology 58, 206–230 (2015)
Idri, A., Zahi, A., Abran, A.: Software cost estimation by fuzzy analogy for web hypermedia applications. In: Proceedings of the International Conference on Software Process and Product Measurement, Cadiz, Spain, pp. 53–62 (2006)
IEEE Standard Glossary of Software Engineering Terminology (1990). IEEE Standards Board/American National Standards Institute, Std. 610.12-1990
Inayat, I., Salim, S.S., Marczak, S., Daneva, M., Shamshirband, S.: A systematic literature review on agile requirements engineering practices and challenges. Computers in human behavior 51, 915–929 (2015)
International vocabulary of basic and general terms in metrology (1993). International Organization for Standardization
Irwin, W., Churcher, N.: A generated parser of c++. NZ Journal of Computing 8(3), 26–37 (2001)
ISO: Iso 26262: 2018:“road vehicles—functional safety” (2018)
ISO/IEC/IEEE Systems and software engineering – Architecture description (2011). DOI 10.1109/IEEESTD.2011.6129467
ISO/IEC 15939: Systems and Software Engineering - Measurement Process (2007)
Itkonen, J., Mantyla, M.V., Lassenius, C.: How do testers do it? an exploratory study on manual testing practices. In: 2009 3rd International Symposium on Empirical Software Engineering and Measurement, pp. 494–497. IEEE (2009)
Itkonen, J., Mäntylä, M.V., Lassenius, C.: The role of the tester’s knowledge in exploratory software testing. IEEE Transactions on Software Engineering 39(5), 707–724 (2012)
Jacquet, J.P., Abran, A.: From software metrics to software measurement methods: a process model. In: Third IEEE International Software Engineering Standards Symposium and Forum – Emerging International Standards, ISESS, pp. 128–135. IEEE (1997)
Janus, A., Dumke, R., Schmietendorf, A., Jäger, J.: The 3c approach for agile quality assurance. In: 2012 3rd International Workshop on Emerging Trends in Software Metrics (WETSoM), pp. 9–13. IEEE (2012)
Jenkins. http://jenkins-ci.org. [Online; accessed 30-January-2013]
John, M.M., Olsson, H.H., Bosch, J.: Ai on the edge: Architectural alternatives. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 21–28. IEEE (2020)
John, M.M., Olsson, H.H., Bosch, J.: Developing ml/dl models: A design framework. In: Proceedings of the International Conference on Software and System Processes, pp. 1–10 (2020)
Johnson, D.E.: Crossover experiments. Wiley Interdisciplinary Reviews: Computational Statistics 2(5), 620–625 (2010)
Johnson, P.M.: Project hackystat: Accelerating adoption of empirically guided software development through non-disruptive, developer-centric, in-process data collection and analysis. Department of Information and Computer Sciences, University of Hawaii 22 (2001)
Johnson, P.M., Kou, H., Agustin, J., Chan, C., Moore, C., Miglani, J., Zhen, S., Doane, W.E.: Beyond the personal software process: Metrics collection and analysis for the differently disciplined. In: Proceedings of the 25th international Conference on Software Engineering, pp. 641–646. IEEE Computer Society (2003)
Johnson, T., Kerzhner, A., Paredis, C., Burkhart, R.: Integrating Models and Simulations of Continuous Dynamics into SysML. Journal of Computing and Information Science in Engineering 12 (2012). DOI 10.1115/1.4005452
Jorgensen, M.: Software quality measurement. Advances in Engineering Software 30(12), 907–912 (1999)
Jørgensen, M.: Do agile methods work for large software projects? In: International Conference on Agile Software Development, pp. 179–190. Springer (2018)
Jung, H.W., Kim, S.G., Chung, C.S.: Measuring software product quality: A survey of iso/iec 9126. IEEE software 21(5), 88–92 (2004)
Kahkonen, T.: Agile methods for large organizations-building communities of practice. In: Agile development conference, pp. 2–10. IEEE (2004)
Kai, G.: Virtual measurement system for muzzle velocity and firing frequency. In: 8th International Conference on Electronic Measurement and Instruments, pp. 176–179 (2001)
Kaisti, M., Mujunen, T., Mäkilä, T., Rantala, V., Lehtonen, T.: Agile principles in the embedded system development. In: Agile Processes in Software Engineering and Extreme Programming, Lecture Notes in Business Information Processing, vol. 179, pp. 16–31. Springer, Rome, Italy (2014). DOI 10.1007/978-3-319-06862-6_2
Kaner, C.: Testing computer software. TAB Books (1988)
Kaner, C., Bach, J., Pettichord, B.: Lessons learned in software testing. John Wiley & Sons (2001)
Kaplan, B., Maxwell, J.A.: Qualitative research methods for evaluating computer information systems. In: Evaluating the organizational impact of healthcare information systems, pp. 30–55. Springer (2005)
Kaplan, R.S., Norton, D.P.: Putting the balanced scorecard to work. Performance measurement, management, and appraisal sourcebook 66 (1995)
Kasauli, R., Knauss, E., Kanagwa, B., Nilsson, A., Calikli, G.: Safety-critical systems and agile development: A mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 470–477 (2018). DOI 10.1109/SEAA.2018.00082
Kasauli, R., Knauss, E., Nilsson, A., Klug, S.: Adding value every sprint: A case study on large-scale continuous requirements engineering. In: REFSQ Workshops (2017)
Kasauli, R., Wohlrab, R., Knauss, E., Steghöfer, J.P., Horkoff, J., Maro, S.: Charting coordination needs in large-scale agile organisations with boundary objects and methodological islands. In: Proceedings of the International Conference on Software and System Processes, ICSSP ’20, p. 51–60. Association for Computing Machinery, New York, NY, USA (2020). DOI 10.1145/3379177.3388897. URL https://doi.org/10.1145/3379177.3388897
Kazman, R., Cai, Y., Mo, R., Feng, Q., Xiao, L., Haziyev, S., Fedak, V., Shapochka, A.: A case study in locating the architectural roots of technical debt. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, pp. 179–188. IEEE (2015)
Keogh, E., Lin, J.: Clustering of time-series subsequences is meaningless: implications for previous and future research. Knowledge and information systems 8(2), 154–177 (2005)
Kerievsky, J.: Industrial XP: Making XP work in large organizations. Executive Report Vol. 6, No. 2, Cutter Consortium (2005). URL http://www.cutter.com/content-and-analysis/resource-centers/agile-project-management/sample-our-research/apmr0502.html
Kettunen, P.: Adopting key lessons from agile manufacturing to agile software product development—a comparative study. Technovation 29(6), 408–422 (2009)
Kettunen, P., Laanti, M.: Combining agile software projects and large-scale organizational agility. Software Process: Improvement and Practice 13(2), 183–193 (2008). DOI 10.1002/spip.354. URL http://onlinelibrary.wiley.com/doi/10.1002/spip.354/abstract
Khurum, M., Gorschek, T., Wilson, M.: The software value map—an exhaustive collection of value aspects for the development of software intensive products. Journal of software: Evolution and Process 25(7), 711–741 (2013)
Kilpi, T.: Implementing a software metrics program at nokia. IEEE Software 18(6), 72–77 (2001)
Kim, D.K., Lee, L.S.: Reverse engineering from exploratory testing to specification-based testing. International Journal of Software Engineering and Its Applications 8(11), 197–208 (2014)
Kim, E.H., Na, J.C., Ryoo, S.M.: Implementing an effective test automation framework. In: 2009 33rd Annual IEEE International Computer Software and Applications Conference, vol. 2, pp. 534–538. IEEE (2009)
Kim, E.H., Na, J.C., Ryoo, S.M.: Test automation framework for implementing continuous integration. In: 2009 Sixth International Conference on Information Technology: New Generations, pp. 784–789. IEEE (2009)
Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The emerging role of data scientists on software development teams. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 96–107. IEEE (2016)
Kim, S., Park, S., Yun, J., Lee, Y.: Automated continuous integration of component-based software: An industrial experience. In: 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, pp. 423–426. IEEE (2008)
Kitchenham, B.: Procedures for performing systematic reviews. Keele, UK, Keele University 33(2004), 1–26 (2004)
Kitchenham, B.: What’s up with software metrics?–a preliminary mapping study. Journal of systems and software 83(1), 37–51 (2010)
Klaine, P.V., Imran, M.A., Onireti, O., Souza, R.D.: A survey of machine learning techniques applied to self-organizing cellular networks. IEEE Communications Surveys & Tutorials 19(4), 2392–2431 (2017)
Knaster, R., Leffingwell, D.: SAFe 4.0 distilled: applying the Scaled Agile Framework for lean software and systems engineering. Addison-Wesley Professional (2017)
Knauss, E., Liebel, G., Horkoff, J., Wohlrab, R., Kasauli, R., Lange, F., Gildert, P.: T-reqs: Tool support for managing requirements in large-scale agile system development. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 502–503. IEEE (2018)
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1168–1176 (2013)
Kohavi, R., Deng, A., Longbotham, R., Xu, Y.: Seven rules of thumb for web site experimenters. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1857–1866 (2014)
Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18(1), 140–181 (2009)
Kolovos, D., Paige, R., Polack, F.: The Epsilon Object Language (EOL). In: European Conference on Model Driven Architecture-Foundations and Applications, pp. 128–142. Springer (2006). DOI 10.1007/11787044_11
Kolovos, D., Paige, R., Polack, F.: Detecting and Repairing Inconsistencies Across Heterogeneous Models. In: 2008 1st International Conference on Software Testing, Verification, and Validation, pp. 356–364. IEEE (2008). DOI 10.1109/icst.2008.23
Kruchten, P., Nord, R.L., Ozkaya, I.: Technical debt: From metaphor to theory and practice. Ieee software 29(6), 18–21 (2012)
Kuhn, A.: On extracting unit tests from interactive live programming sessions. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 1241–1244. IEEE (2013)
Kuhrmann, M., Diebold, P., Münch, J., Tell, P., Garousi, V., Felderer, M., Trektere, K., McCaffery, F., Linssen, O., Hanser, E., Prause, C.R.: Hybrid software and system development in practice: Waterfall, scrum, and beyond. In: Proceedings of the 2017 International Conference on Software and System Process, ICSSP 2017, p. 30–39. Association for Computing Machinery, New York, NY, USA (2017). DOI 10.1145/3084100.3084104. URL https://doi.org/10.1145/3084100.3084104
Kumar, S., Wallace, C.: Guidance for exploratory testing through problem frames. In: 2013 26th International Conference on Software Engineering Education and Training (CSEE&T), pp. 284–288. IEEE (2013)
Kunz, R.F., Kasmala, G.F., Mahaffy, J.H., Murray, C.J.: On the automated assessment of nuclear reactor systems code accuracy. Nuclear Engineering and Design 211(2-3), 245–272 (2002). TY - JOUR
Laanti, M., Salo, O., Abrahamsson, P.: Agile methods rapidly replacing traditional methods at nokia: A survey of opinions on agile transformation. Information and Software Technology 53(3), 276–290 (2011)
Lacoste, F.J.: Killing the gatekeeper: Introducing a continuous integration system. In: 2009 agile conference, pp. 387–392. IEEE (2009)
Lagerberg, L., Skude, T., Emanuelsson, P., Sandahl, K., Ståhl, D.: The impact of agile principles and practices on large-scale software development projects: A multiple-case study of two projects at ericsson. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 348–356. IEEE (2013)
Larman, C.: Scaling lean & agile development: thinking and organizational tools for large-scale Scrum. Pearson Education India (2008)
Larman, C., Vodde, B.: Large-scale scrum: More with LeSS. Addison-Wesley Professional (2016)
Lauesen, S.: Software requirements: styles and techniques. Pearson Education (2002)
Lauesen, S.: Guide to requirements SL-07. Lauesen Publishing (2017)
Layman, L., Williams, L., Cunningham, L.: Motivations and measurements in an agile case study. Journal of Systems Architecture 52(11), 654–667 (2006)
Lee, C.L., Yang, H.J.: Organization structure, competition and performance measurement systems and their joint effects on performance. Management Accounting Research 22(2), 84–104 (2011)
Leffingwell, D.: Agile software requirements: lean requirements practices for teams, programs, and the enterprise. Addison-Wesley Professional (2010)
Leffingwell, D., et al.: Scaled agile framework 3.0 (2014)
Li, Z., Avgeriou, P., Liang, P.: A systematic mapping study on technical debt and its management. Journal of Systems and Software 101, 193–220 (2015)
Lier, F., Wrede, S., Siepmann, F., Lütkebohle, I., Paul-Stueve, T., Wachsmuth, S.: Facilitating research cooperation through linking and sharing of heterogenous research artefacts: cross platform linking of semantically enriched research artefacts. In: Proceedings of the 8th International Conference on Semantic Systems, pp. 157–164 (2012)
Lin, J., Keogh, E., Lonardi, S., Chiu, B.: A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, pp. 2–11. ACM (2003)
Lin, J., Keogh, E., Lonardi, S., Patel, P.: Finding motifs in time series. In: In the 2nd Workshop on Temporal Data Mining, at the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 53–68 (2002)
Lin, J., Keogh, E., Wei, L., Lonardi, S.: Experiencing sax: a novel symbolic representation of time series. Data Mining and knowledge discovery 15(2), 107–144 (2007)
Lin, J., Kolcz, A.: Large-scale machine learning at twitter. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 793–804 (2012)
Lindgren, E., Münch, J.: Raising the odds of success: the current state of experimentation in product development. Information and Software Technology 77, 80–91 (2016)
Lindman, J., Horkoff, J., Hammouda, I., Knauss, E.: Emerging perspectives of application programming interface strategy: A framework to respond to business concerns. IEEE Software 37(2), 52–59 (2020). DOI 10.1109/MS.2018.2875964
Lindvall, M., Muthig, D., Dagnino, A., Wallin, C., Stupperich, M., Kiefer, D., May, J., Kahkonen, T.: Agile software development in large organizations. Computer 37(12), 26–34 (2004)
Liu, H., Li, Z., Zhu, J., Tan, H., Huang, H.: A unified test framework for continuous integration testing of soa solutions. In: 2009 IEEE International Conference on Web Services, pp. 880–887. IEEE (2009)
Liu, S., Xiao, F., Ou, W., Si, L.: Cascade ranking for operational e-commerce search. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1557–1565 (2017)
Lokan, C., Mendes, E.: Cross-company and single-company effort models using the isbsg database: A further replicated study. In: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering, ISESE ’06, pp. 75–84. ACM, New York, NY, USA (2006). DOI 10.1145/1159733.1159747. URL http://doi.acm.org/10.1145/1159733.1159747
Lokan, C., Wright, T., Hill, P.R., Stringer, M.: Organizational benchmarking using the isbsg data repository. IEEE Software 18(5), 26–32 (2001)
Long, B.: Managing module dependencies to facilitate continuous testing. Information processing letters 108(3), 127–131 (2008)
Lucas, F., Molina, F., Toval, A.: A systematic review of UML model consistency management. Information and Software Technology 51(12), 1631–1645 (2009). DOI 10.1016/j.infsof.2009.04.009
Lucassen, G., Dalpiaz, F., van der Werf, J.M.E., Brinkkemper, S.: Forging high-quality user stories: Towards a discipline for agile requirements. In: 2015 IEEE 23rd International Requirements Engineering Conference (RE), pp. 126–135 (2015). DOI 10.1109/RE.2015.7320415
Luckow, A., Cook, M., Ashcraft, N., Weill, E., Djerekarov, E., Vorster, B.: Deep learning in the automotive industry: Applications and tools. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 3759–3768. IEEE (2016)
Lwakatare, L.E., Raj, A., Bosch, J., Olsson, H.H., Crnkovic, I.: A taxonomy of software engineering challenges for machine learning systems: An empirical investigation. In: International Conference on Agile Software Development, pp. 227–243. Springer, Cham (2019)
Maguire, M., Delahunt, B.: Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. All Ireland Journal of Higher Education 9(3) (2017)
van Manen, H., van Vliet, H.: Organization-wide agile expansion requires an organization-wide agile mindset. In: Product-Focused Software Process Improvement, Lecture Notes in Computer Science, pp. 48–62. Springer, Helsinki, Finland (2014). URL http://link.springer.com/chapter/10.1007/978-3-319-13835-0_4
Mantere, M., Uusitalo, I., Roning, J.: Comparison of static code analysis tools. In: Emerging Security Information, Systems and Technologies, 2009. SECURWARE’09. Third International Conference on, pp. 15–22. IEEE (2009)
Manzi, J.: Uncontrolled: The surprising payoff of trial-and-error for business, politics, and society. Basic Books (AZ) (2012)
Mårtensson, T., Martini, A., Ståhl, D., Bosch, J.: Excellence in exploratory testing: Success factors in large-scale industry projects. In: International Conference on Product-Focused Software Process Improvement, pp. 299–314. Springer (2019)
Mårtensson, T., Ståhl, D., Bosch, J.: Exploratory testing of large-scale systems–testing in the continuous integration and delivery pipeline. In: International Conference on Product-Focused Software Process Improvement, pp. 368–384. Springer (2017)
Mårtensson, T., Ståhl, D., Bosch, J.: Enable more frequent integration of software in industry projects. Journal of Systems and Software 142, 223–236 (2018)
Mårtensson, T., Ståhl, D., Bosch, J.: Test activities in the continuous integration and delivery pipeline. Journal of Software: Evolution and Process 31(4), e2153 (2019)
Martin, R.C.: Agile software development: principles, patterns, and practices. Prentice Hall (2002)
Martini, A., Besker, T., Bosch, J.: The introduction of technical debt tracking in large companies. In: 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), pp. 161–168. IEEE (2016)
Martini, A., Bosch, J.: The danger of architectural technical debt: Contagious debt and vicious circles. In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp. 1–10. IEEE (2015)
Martini, A., Bosch, J.: An empirically developed method to aid decisions on architectural technical debt refactoring: Anacondebt. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 31–40. IEEE (2016)
Martini, A., Bosch, J.: A multiple case study of continuous architecting in large agile companies: current gaps and the caffea framework. In: 2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA), pp. 1–10. IEEE (2016)
Martini, A., Bosch, J.: The magnificent seven: towards a systematic estimation of technical debt interest. In: Proceedings of the XP2017 Scientific Workshops, pp. 1–5 (2017)
Martini, A., Bosch, J., Chaudron, M.: Investigating architectural technical debt accumulation and refactoring over time: A multiple-case study. Information and Software Technology 67, 237–253 (2015)
Maruping, L.M., Zhang, X., Venkatesh, V.: Role of collective ownership and coding standards in coordinating expertise in software project teams. European Journal of Information Systems 18(4), 355–371 (2009)
Masters, J.: The history of action research. Action research electronic reader 22, 2005 (1995)
Masuda, S., Ono, K., Yasue, T., Hosokawa, N.: A survey of software quality for machine learning applications. In: 2018 IEEE International conference on software testing, verification and validation workshops (ICSTW), pp. 279–284. IEEE (2018)
Matsumoto, K., Kibe, S., Uehara, M., Mori, H.: Design of development as a service in the cloud. In: 2012 15th International Conference on Network-Based Information Systems, pp. 815–819. IEEE (2012)
Mattos, D.I., Bosch, J., Olsson, H.H.: Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. In: 19th International Conference on Agile Software Development (2018)
Maximilien, E.M., Williams, L.: Assessing test-driven development at ibm. In: Software Engineering, 2003. Proceedings. 25th International Conference on, pp. 564–569. IEEE (2003)
Maxwell, J.A.: Qualitative research design: An interactive approach, vol. 41. Sage publications (2012)
Maxwell, K.D., Forselius, P.: Benchmarking software development productivity. IEEE Software 17(1), 80–88 (2000). DOI 10.1109/52.820015
Mayring, P.: Qualitative content analysis–research instrument or mode of interpretation. The role of the researcher in qualitative psychology 2(139-148) (2002)
McConnell, S.: Managing technical debt presentation at icse 2013 (2013)
McGarry, J.: Practical software measurement: objective information for decision makers. Addison-Wesley Professional (2002)
McIntosh, S., Kamei, Y., Adams, B., Hassan, A.E.: The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects. In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 192–201. ACM (2014)
McMahon, P.: Extending agile methods: A distributed project and organizational improvement perspective. In: Systems and Software Technology Conference (2005)
Melão, N., Pidd, M.: A conceptual framework for understanding business processes and business process modelling. Information systems journal 10(2), 105–129 (2000)
Mellado, R.P., Montini, D.Á., Dias, L.A.V., da Cunha, A.M., et al.: Software product measurement and analysis in a continuous integration environment. In: 2010 Seventh International Conference on Information Technology: New Generations, pp. 1177–1182. IEEE (2010)
Mendes, E., Lokan, C., Harrison, R., Triggs, C.: A replicated comparison of cross-company and within-company effort estimation models using the isbsg database. In: 11th IEEE International Software Metrics Symposium (METRICS’05), pp. 10 pp.–36 (2005). DOI 10.1109/METRICS.2005.4
Menzies, T., Butcher, A., Cok, D., Marcus, A., Layman, L., Shull, F., Turhan, B., Zimmermann, T.: Local versus global lessons for defect prediction and effort estimation. IEEE Transactions on software engineering 39(6), 822–834 (2013)
Meyer, B.: The ugly, the hype and the good: an assessment of the agile approach. In: Agile!, pp. 149–154. Springer (2014)
Mi, Q., Keung, J., Xiao, Y., Mensah, S., Gao, Y.: Improving code readability classification using convolutional neural networks. Information and Software Technology 104, 60–71 (2018)
Micallef, M., Porter, C., Borg, A.: Do exploratory testers need formal training? an investigation using hci techniques. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 305–314. IEEE (2016)
Mihindukulasooriya, N., Rizzo, G., Troncy, R., Corcho, O., García-Castro, R.: A two-fold quality assurance approach for dynamic knowledge bases: The 3cixty use case. In: (KNOW@ LOD/CoDeS)@ ESWC (2016)
Miles, M.B., Huberman, A.M.: Qualitative data analysis: An expanded sourcebook. sage (1994)
Miller, A.: A hundred days of continuous integration. In: Agile 2008 conference, pp. 289–293. IEEE (2008)
Moha, N., Gueheneuc, Y.G., Duchien, A.F., et al.: Decor: A method for the specification and detection of code and design smells. IEEE Transactions on Software Engineering (TSE) 36(1), 20–36 (2010)
Moitra, D.: Managing change for software process improvement initiatives: a practical experience-based approach. Software Process: Improvement and Practice 4(4), 199–207 (1998)
Mueen, A., Keogh, E., Zhu, Q., Cash, S., Westover, B.: Exact discovery of time series motifs. In: Proceedings of the 2009 SIAM international conference on data mining, pp. 473–484. SIAM (2009)
Mujtaba, S., Feldt, R., Petersen, K.: Waste and lead time reduction in a software product customization process with value stream maps. In: 2010 21st australian software engineering conference, pp. 139–148. IEEE (2010)
Müller, M., Sazama, F., Debou, C., Dudzic, P., Abowd, P.: Survey – State of Practice “Agile in Automotive”. Tech. rep., KUGLER MAAG CIE GmbH (2014). URL http://www.kuglermaag.com/improvement-concepts/agile-in-automotive/state-of-practice.html
Munappy, A., Bosch, J., Olsson, H.H., Arpteg, A., Brinne, B.: Data management challenges for deep learning. In: 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 140–147. IEEE (2019)
Munappy, A.R., Mattos, D.I., Bosch, J., Olsson, H.H., Dakkak, A.: From ad-hoc data analytics to dataops. In: Proceedings of the International Conference on Software and System Processes, pp. 165–174 (2020)
Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology 48(3), 443–453 (1970)
Nentwich, C., Emmerich, W., Finkelstein, A., Ellmer, E.: Flexible Consistency Checking. ACM Transactions on Software Engineering and Methodology (TOSEM) 12(1), 28–63 (2003). DOI 10.1145/839268.839271
Niessink, F., van Vliet, H.: Measurements should generate value, rather than data. In: 6th International Software Metrics Symposium, pp. 31–38 (2000)
Niessink, F., van Vliet, H.: Measurement program success factors revisited. Information and Software Technology 43(10), 617–628 (2001). TY - JOUR
Nilsson, A., Bosch, J., Berger, C.: The civit model in a nutshell: Visualizing testing activities to support continuous integration. In: Continuous software engineering, pp. 97–106. Springer (2014)
Niven, P.R.: Balanced scorecard step-by-step: maximizing performance and maintaining results. John Wiley & Sons (2002)
Novak, J., Krajnc, A., ontar, R.: Taxonomy of static code analysis tools. In: MIPRO, 2010 Proceedings of the 33rd International Convention, pp. 418–422. IEEE (2010)
Ochodek, M., Staron, M., Bargowski, D., Meding, W., Hebig, R.: Using machine learning to design a flexible loc counter. In: Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE), IEEE Workshop on, pp. 14–20. IEEE (2017)
Offen, R.J., Jeffery, R.: Establishing software measurement programs. Software, IEEE 14(2), 45–53 (1997)
Olsson, H.H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”–a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: Software Engineering and Advanced Applications (SEAA), 2012 38th EUROMICRO Conference on, pp. 392–399. IEEE (2012)
Olsson, H.H., Bosch, J.: From opinions to data-driven software r&d: A multi-case study on how to close the ’open loop’ problem. In: 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 9–16. IEEE (2014)
Olsson, H.H., Bosch, J.: The hypex model: from opinions to data-driven software development. In: Continuous software engineering, pp. 155–164. Springer (2014)
Olsson, H.H., Bosch, J.: Towards continuous customer validation: A conceptual model for combining qualitative customer feedback with quantitative customer observation. In: International Conference of Software Business, pp. 154–166. Springer (2015)
Olszewska, M., Heidenberg, J., Weijola, M., Mikkonen, K., Porres, I.: Quantitatively measuring a large-scale agile transformation. Journal of Systems and Software 117, 258 – 273 (2016). URL http://www.sciencedirect.com/science/article/pii/S016412121600087X
Organization, I.S., Commission, I.E.: Software and systems engineering, software measurement process. Tech. rep., ISO/IEC (2007)
Paasivaara, M., Lassenius, C.: Challenges and success factors for large-scale agile transformations: A research proposal and a pilot study. In: Proceedings of the Scientific Workshop Proceedings of XP2016, pp. 1–5 (2016)
Paetsch, F., Eberlein, A., Maurer, F.: Requirements engineering and agile software development. In: WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003., pp. 308–313. IEEE (2003)
Paige, R., Brooke, P., Ostroff, J.: Metamodel-Based Model Conformance and Multi-view Consistency Checking. ACM Transactions on Software Engineering and Methodology (TOSEM) 16(3), 11 (2007). DOI 10.1145/1243987.1243989
Pantazos, K., Shollo, A., Staron, M., Meding, W.: Presenting software metrics indicators-a case study. In: Proceedings of IWSM/Mensura conference (2010)
Patil, D.: Building data science teams. “ O’Reilly Media, Inc.” (2011)
Peach, R.W.: The ISO 9000 handbook. Irwin Professional Publishing (1995)
Pernstål, J., Magazinius, A., Gorschek, T.: A study investigating challenges in the interface between product development and manufacturing in the development of software-intensive automotive systems. International Journal of Software Engineering and Knowledge Engineering 22(07), 965–1004 (2012)
Persson, M., Torngren, M., Qamar, A., Westman, J., Biehl, M., Tripakis, S., Vangheluwe, H., Denil, J.: A Characterization of Integrated Multi-View Modeling in the Context of Embedded and Cyber-Physical Systems. In: Embedded Software (EMSOFT), 2013 Proceedings of the International Conference on, pp. 1–10. IEEE (2013). DOI 10.1109/emsoft.2013.6658588
Pesola, J.P., Tanner, H., Eskeli, J., Parviainen, P., Bendas, D.: Integrating early v&v support to a gse tool integration platform. In: 2011 IEEE Sixth International Conference on Global Software Engineering Workshop, pp. 95–101. IEEE (2011)
Petersen, K.: A palette of lean indicators to detect waste in software maintenance: A case study. In: Agile processes in software engineering and extreme programming, pp. 108–122. Springer (2012)
Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering. sn (2008)
Petersen, K., Wohlin, C.: A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of Systems and Software 82(9), 1479–1490 (2009). DOI 10.1016/j.jss.2009.03.036
Pfahl, D., Yin, H., Mäntylä, M.V., Münch, J.: How is exploratory testing used? a state-of-the-practice survey. In: Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement, pp. 1–10 (2014)
Pichler, J., Ramler, R.: How to test the intangible properties of graphical user interfaces? In: 2008 1st International Conference on Software Testing, Verification, and Validation, pp. 494–497. IEEE (2008)
Raappana, P., Saukkoriipi, S., Tervonen, I., Mäntylä, M.V.: The effect of team exploratory testing–experience report from f-secure. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 295–304. IEEE (2016)
Radatz, J., Geraci, A., Katki, F.: Ieee standard glossary of software engineering terminology. IEEE Std 610121990(121990), 3 (1990)
Radjenović, D., Heričko, M., Torkar, R., Živkovič, A.: Software fault prediction metrics: A systematic literature review. Information and Software Technology 55(8), 1397–1418 (2013)
Raj, A., Bosch, J., Olsson, H.H., Wang, T.J.: Modelling data pipelines. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 13–20. IEEE (2020)
Ramasubbu, N., Cataldo, M., Balan, R.K., Herbsleb, J.D.: Configuring global software teams: a multi-company analysis of project productivity, quality, and profits. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 261–270. ACM (2011)
Ramesh, B., Cao, L., Baskerville, R.: Agile requirements engineering practices and challenges: an empirical study. Information Systems Journal 20(5), 449–480 (2010)
Rana, R., Staron, M., Berger, C., Hansson, J., Nilsson, M., Törner, F., Meding, W., Höglund, C.: Selecting software reliability growth models and improving their predictive accuracy using historical projects data. Journal of Systems and Software 98, 59–78 (2014)
Rashmi, N., Suma, V.: Defect detection efficiency of the combined approach. In: ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India-Vol II, pp. 485–490. Springer (2014)
Rasmusson, J.: Long build trouble shooting guide. In: Conference on Extreme Programming and Agile Methods, pp. 13–21. Springer (2004)
Reis, J., Mota, A.: Aiding exploratory testing with pruned gui models. Information Processing Letters 133, 49–55 (2018)
Ries, E.: The lean startup: How today’s entrepreneurs use continuous innovation to create radically successful businesses. Crown Business Publishing (2011)
Rissanen, O., Münch, J.: Continuous experimentation in the b2b domain: a case study. In: 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, pp. 12–18. IEEE (2015)
Roberts, M.: Enterprise continuous integration using binary dependencies. In: International Conference on Extreme Programming and Agile Processes in Software Engineering, pp. 194–201. Springer (2004)
Robson, C., McCartan, K.: Real world research. John Wiley & Sons (2016)
Rodden, K., Hutchinson, H., Fu, X.: Measuring the user experience on a large scale: user-centered metrics for web applications. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 2395–2398 (2010)
Rodríguez, P., Haghighatkhah, A., Lwakatare, L.E., Teppola, S., Suomalainen, T., Eskeli, J., Karvonen, T., Kuvaja, P., Verner, J.M., Oivo, M.: Continuous deployment of software intensive products and services: A systematic mapping study. Journal of Systems and Software 123, 263–291 (2017)
Rogers, R.O.: Cruisecontrol. net: Continuous integration for. net. In: International Conference on Extreme Programming and Agile Processes in Software Engineering, pp. 114–122. Springer (2003)
Rogers, R.O.: Scaling continuous integration. In: International conference on extreme programming and agile processes in software engineering, pp. 68–76. Springer (2004)
Ruhe, G.: Software engineering decision support–a new paradigm for learning software organizations. In: Advances in Learning Software Organizations, pp. 104–113. Springer (2003)
Ruhe, G., Saliu, M.O.: The art and science of software release planning. Software, IEEE 22(6), 47–53 (2005)
Rumpe, B.: Agile modeling with the uml. In: M. Wirsing, A. Knapp, S. Balsamo (eds.) Radical Innovations of Software and Systems Engineering in the Future, pp. 297–309. Springer Berlin Heidelberg, Berlin, Heidelberg (2004)
Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14(2), 131–164 (2009)
Runeson, P., Host, M., Rainer, A., Regnell, B.: Case study research in software engineering: Guidelines and examples. John Wiley & Sons (2012)
Runeson, P., Host, M., Rainer, A., Regnell, B.: Case study research in software engineering: Guidelines and examples. John Wiley & Sons (2012)
Salo, O., Abrahamsson, P.: Agile methods in european embedded software development organisations: a survey on the actual use and usefulness of extreme programming and scrum. IET software 2(1), 58–64 (2008)
Sandberg, A., Pareto, L., Arts, T.: Agile collaborative research: Action principles for industry-academia collaboration. Software, IEEE 28(4), 74–83 (2011)
Savolainen, J., Kuusela, J., Vilavaara, A.: Transition to agile development-rediscovery of important requirements engineering practices. In: 2010 18th IEEE International Requirements Engineering Conference, pp. 289–294. IEEE (2010)
Schaefer, C.J., Do, H.: Model-based exploratory testing: a controlled experiment. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, pp. 284–293. IEEE (2014)
Schermann, G., Cito, J., Leitner, P., Zdun, U., Gall, H.C.: We’re doing it live: A multi-method empirical study on continuous experimentation. Information and Software Technology 99, 41–57 (2018)
Schmidt, D.C.: Model-driven engineering. IEEE Computer 39(2), 25 (2006)
Schuh, P.: Integrating agile development in the real world. Charles River Media Hingham (2005)
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F., Dennison, D.: Hidden technical debt in machine learning systems. Advances in neural information processing systems 28, 2503–2511 (2015)
Seaman, C., Guo, Y., Zazworka, N., Shull, F., Izurieta, C., Cai, Y., Vetrò, A.: Using technical debt data in decision making: Potential decision approaches. In: 2012 Third International Workshop on Managing Technical Debt (MTD), pp. 45–48. IEEE (2012)
Sedano, T., Ralph, P., Praire, C.: The product backlog. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 200–211 (2019). DOI 10.1109/ICSE.2019.00036
Sehmi, A., Jones, N., Wang, S., Loudon, G.: Knowledge-based systems for neuroelectric signal processing. IEE Proceedings-Science, Measurement and Technology 141(3), 215–23 (2003)
Senin, P., Lin, J., Wang, X., Oates, T., Gandhi, S., Boedihardjo, A.P., Chen, C., Frankenstein, S., Lerner, M.: Grammarviz 2.0: a tool for grammar-based pattern discovery in time series. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 468–472. Springer (2014)
Shadish, W.R., Cook, T.D., Campbell, D.T., et al.: Experimental and quasi-experimental designs for generalized causal inference/William R. Shedish, Thomas D. Cook, Donald T. Campbell. Boston: Houghton Mifflin, (2002)
Shah, A., Kerzhner, A., Schaefer, D., Paredis, C.: Multi-view Modeling to Support Embedded Systems Engineering in SysML. In: Graph transformations and model-driven engineering, pp. 580–601. Springer (2010). DOI 10.1007/978-3-642-17322-6_25
Shah, S.M.A., Gencel, C., Alvi, U.S., Petersen, K.: Towards a hybrid testing process unifying exploratory testing and scripted testing. Journal of software: Evolution and Process 26(2), 220–250 (2014)
Shah, S.M.A., Torchiano, M., Vetrò, A., Morisio, M.: Exploratory testing as a source of technical debt. IT Professional 16(3), 44–51 (2013)
Shahnewaz, S., Ruhe, G.: Relrea-an analytical approch for evaluating release readiness. In: SEKE (2014)
Shalloway, A., Beaver, G., Trott, J.R.: Lean-agile software development: achieving enterprise agility. Pearson Education (2009)
Shaukat, R., Shahoor, A., Urooj, A.: Probing into code analysis tools: A comparison of c# supporting static code analyzers. In: Applied Sciences and Technology (IBCAST), 2018 15th International Bhurban Conference on, pp. 455–464. IEEE (2018)
Shen, M., Yang, W., Rong, G., Shao, D.: Applying agile methods to embedded software development: A systematic review. In: Proceedings of the International Workshop on Software Engineering for Embedded Systems, pp. 30–36. IEEE (2012). DOI 10.1109/SEES.2012.6225488
Shoaib, L., Nadeem, A., Akbar, A.: An empirical evaluation of the influence of human personality on exploratory software testing. In: 2009 IEEE 13th International Multitopic Conference, pp. 1–6. IEEE (2009)
Shull, F., Singer, J., Sjøberg, D.I.K. (eds.): Guide to Advanced Empirical Software Engineering. Springer London, London (2008). DOI 10.1007/978-1-84800-044-5. URL http://www.springerlink.com/index/10.1007/978-1-84800-044-5
Silhavy, P., Silhavy, R., Prokopova, Z.: Categorical variable segmentation model for software development effort estimation. IEEE Access 7, 9618–9626 (2019). DOI 10.1109/ACCESS.2019.2891878
Silhavy, R., Silhavy, P., Prokopova, Z.: Improving algorithmic optimisation method by spectral clustering. In: Computer Science On-line Conference, pp. 1–10. Springer (2017)
Silhavy, R., Silhavy, P., Prokopová, Z.: Evaluating subset selection methods for use case points estimation. Information and Software Technology 97, 1–9 (2018)
Singh, D., Sekar, V.R., Stolee, K.T., Johnson, B.: Evaluating how static analysis tools can reduce code review effort. In: Visual Languages and Human-Centric Computing (VL/HCC), 2017 IEEE Symposium on, pp. 101–105. IEEE (2017)
Sinnema, M., Deelstra, S., Nijhuis, J., Bosch, J.: Covamof: A framework for modeling variability in software product families. In: International Conference on Software Product Lines, pp. 197–213. Springer (2004)
Smit, M., Gergel, B., Hoover, H.J., Stroulia, E.: Maintainability and source code conventions: An analysis of open source projects. University of Alberta, Department of Computing Science, Tech. Rep. TR11-06 (2011)
Sommerville, I.: Software engineering. 6th. Ed., Harlow, UK.: Addison-Wesley (2001)
Sommerville, I.: Software Engineering, 10th edn. Pearson (2015)
Sorrell, S., et al.: Digitalisation of goods: a systematic review of the determinants and magnitude of the impacts on energy consumption. Environmental Research Letters 15(4), 043001 (2020)
Ståhl, D., Bosch, J.: Experienced benefits of continuous integration in industry software product development: A case study. In: The 12th iasted international conference on software engineering,(innsbruck, austria, 2013), pp. 736–743 (2013)
Ståhl, D., Bosch, J.: Continuous integration flows. In: Continuous software engineering, pp. 107–115. Springer (2014)
Ståhl, D., Bosch, J.: Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87, 48–59 (2014)
Ståhl, D., Bosch, J.: Industry application of continuous integration modeling: a multiple-case study. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 270–279. IEEE (2016)
Ståhl, D., Bosch, J.: Cinders: The continuous integration and delivery architecture framework. Information and Software Technology 83, 76–93 (2017)
Ståhl, D., Hallén, K., Bosch, J.: Achieving traceability in large scale continuous integration and delivery deployment, usage and validation of the eiffel framework. Empirical Software Engineering 22(3), 967–995 (2017)
Stahl, D., Martensson, T., Bosch, J.: Continuous practices and devops: beyond the buzz, what does it all mean? In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 440–448. IEEE (2017)
Staron, M.: Critical role of measures in decision processes: Managerial and technical measures in the context of large software development organizations. Information and Software Technology 54(8), 887–899 (2012)
Staron, M.: Software complexity metrics in general and in the context of ISO 26262 software verification requirements. In: Scandinavian Conference on Systems Safety. http://gup.ub.gu.se/records/fulltext/233026/233026.pdf (2016)
Staron, M.: Action Research in Software Engineering. Springer (2020)
Staron, M., Hansson, J., Feldt, R., Henriksson, A., Meding, W., Nilsson, S., Hoglund, C.: Measuring and visualizing code stability–a case study at three companies. In: Software Measurement and the 2013 Eighth International Conference on Software Process and Product Measurement (IWSM-MENSURA), 2013 Joint Conference of the 23rd International Workshop on, pp. 191–200. IEEE (2013)
Staron, M., Meding, W.: Predicting short-term defect inflow in large software projects–an initial evaluation. 11th International Conference on Evaluation and Assessment in Software Engineering, EASE (2007)
Staron, M., Meding, W.: Predicting weekly defect inflow in large software projects based on project planning and test status. Information and Software Technology p. (available online) (2007)
Staron, M., Meding, W.: Ensuring reliability of information provided by measurement systems. In: Proceedings of the International Conferences on Software Process and Product Measurement. Springer Berlin / Heidelberg (2009)
Staron, M., Meding, W.: Factors determining long-term success of a measurement program: an industrial case study. e-Informatica Software Engineering Journal pp. 7–23 (2011)
Staron, M., Meding, W.: Software Development Measurement Programs: Development, Management and Evolution. Springer (2018)
Staron, M., Meding, W., Caiman, M.: Improving completeness of measurement systems for monitoring software development workflows. In: Software Quality. Increasing Value in Software and Systems Development, pp. 230–243. Springer (2013)
Staron, M., Meding, W., Hansson, J., Höglund, C., Niesel, K., Bergmann, V.: Dashboards for continuous monitoring of quality for software product under development. System Qualities and Software Architecture (SQSA) (2013)
Staron, M., Meding, W., Karlsson, G., Nilsson, C.: Developing measurement systems: an industrial case study. Journal of Software Maintenance and Evolution: Research and Practice 23(2), 89–107 (2011)
Staron, M., Meding, W., Nilsson, C.: A framework for developing measurement systems and its industrial evaluation. Information and Software Technology 51(4), 721–737 (2008)
Staron, M., Meding, W., Palm, K.: Release readiness indicator for mature agile and lean software development projects. In: Agile Processes in Software Engineering and Extreme Programming, pp. 93–107. Springer (2012)
Staron, M., Meding, W., Söderqvist, B.: A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation. Information and Software Technology 52(10), 1069–1079 (2010)
Staron, M., Ochodek, M., Meding, W., Söder, O., Rosenberg, E.: Machine learning to support code reviews in continuous integration. In: Artificial Intelligence Methods For Software Engineering, pp. 141–167. World Scientific (2021)
Steghöfer, J.P., Knauss, E., Horkoff, J., Wohlrab, R.: Challenges of scaled agile for safety-critical systems. In: X. Franch, T. Männistö, S. Martínez-Fernández (eds.) Product-Focused Software Process Improvement, pp. 350–366. Springer International Publishing, Cham (2019)
Stolberg, S.: Enabling agile testing through continuous integration. In: 2009 agile conference, pp. 369–374 (2009)
Sturdevant, K.F.: Cruisin’and chillin’: Testing the java-based distributed ground data system” chill” with cruisecontrol system” chill” with cruisecontrol. In: 2007 IEEE Aerospace Conference, pp. 1–8. IEEE (2007)
Subramanyam, R., Krishnan, M.S.: Empirical analysis of ck metrics for object-oriented design complexity: Implications for software defects. Software Engineering, IEEE Transactions on 29(4), 297–310 (2003)
Sunindyo, W.D., Moser, T., Winkler, D., Biffl, S.: Foundations for event-based process analysis in heterogeneous software engineering environments. In: 2010 36th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 313–322. IEEE (2010)
Suryadevara, J., Tiwari, S.: Adopting MBSE in Construction Equipment Industry: An Experience Report. In: 25th Asia-Pacific Software Engineering Conference APSEC (2018). DOI 10.1109/apsec.2018.00066
Susman, G., Evered, R.: An Assessment of the Scientific Merits of Action Research. Journal of Administrative Science Quarterly 23(4), 582–603 (1978)
Susman, G.I.: Action research: a sociotechnical systems perspective. Beyond method: Strategies for social research pp. 95–113 (1983)
Susman, G.I., Evered, R.D.: An assessment of the scientific merits of action research. Administrative science quarterly pp. 582–603 (1978)
Sutherland, J., Frohman, R.: Hitting the wall: What to do when high performing scrum teams overwhelm operations and infrastructure. In: 2011 44th Hawaii International Conference on System Sciences, pp. 1–6. IEEE (2011)
Sviridova, T., Stakhova, D., Marikutsa, U.: Exploratory testing: Management solution. In: 2013 12th International Conference on the Experience of Designing and Application of CAD Systems in Microelectronics (CADSM), pp. 361–361. IEEE (2013)
Tamburri, D.A., Kruchten, P., Lago, P., van Vliet, H.: What is social debt in software engineering? In: 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 93–96. IEEE (2013)
Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure: More, better, faster experimentation. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 17–26 (2010)
Tingling, P., Saeed, A.: Extreme programming in action: a longitudinal case study. In: International Conference on Human-Computer Interaction, pp. 242–251. Springer (2007)
Tom, E., Aurum, A., Vidgen, R.: An exploration of technical debt. Journal of Systems and Software 86(6), 1498–1516 (2013)
Torunski, E., Shafiq, M.O., Whitehead, A.: Code style analytics for the automatic setting of formatting rules in ides: A solution to the tabs vs. spaces debate. In: Digital Information Management (ICDIM), 2017 Twelfth International Conference on, pp. 6–14. IEEE (2017)
Tosun, A., Turhan, B., Bener, A.: Practical considerations in deploying ai for defect prediction: a case study within the turkish telecommunication industry. In: Proceedings of the 5th International Conference on Predictor Models in Software Engineering, pp. 1–9 (2009)
Trist, E.: The evolution of socio-technical systems. Occasional paper 2, 1981 (1981)
Tsai, W., Heisler, K., Volovik, D., Zualkernan, I.: A critical look at the relationship between ai and software engineering. In: [Proceedings] 1988 IEEE Workshop on Languages for Automation@ m_Symbiotic and Intelligent Robotics, pp. 2–18. IEEE (1988)
Tuomikoski, J., Tervonen, I.: Absorbing software testing into the scrum method. In: International Conference on Product-Focused Software Process Improvement, pp. 199–215. Springer (2009)
Uludag, Ö., Kleehaus, M., Caprano, C., Matthes, F.: Identifying and structuring challenges in large-scale agile development based on a structured literature review. In: 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC), pp. 191–197. IEEE (2018)
Umarji, M., Emurian, H.: Acceptance issues in metrics program implementation. In: H. Emurian (ed.) 11th IEEE International Symposium Software Metrics, pp. 10–17 (2005)
Unterkalmsteiner, M., Gorschek, T., Islam, A., Cheng, C.K., Permadi, R.B., Feldt, R.: A conceptual framework for spi evaluation. Journal of Software: Evolution and Process 26(2), 251–279 (2014)
Unterkalmsteiner, M., Gorschek, T., Islam, A.M., Cheng, C.K., Permadi, R.B., Feldt, R.: Evaluation and measurement of software process improvement—a systematic literature review. Software Engineering, IEEE Transactions on 38(2), 398–424 (2012)
Van Der Linden, F., Bosch, J., Kamsties, E., Känsälä, K., Obbink, H.: Software product family evaluation. In: International Conference on Software Product Lines, pp. 110–129. Springer (2004)
Van Der Storm, T.: Continuous release and upgrade of component-based software. In: Proceedings of the 12th international workshop on Software configuration management, pp. 43–57 (2005)
Van Der Storm, T.: The sisyphus continuous integration system. In: 11th European Conference on Software Maintenance and Reengineering (CSMR’07), pp. 335–336. IEEE (2007)
Van Der Storm, T.: Backtracking incremental continuous integration. In: 2008 12th European Conference on Software Maintenance and Reengineering, pp. 233–242. IEEE (2008)
Van Nostrand, R.C.: Design of experiments using the taguchi approach: 16 steps to product and process improvement (2002)
Vidgen, R., Wang, X.: Coevolving systems and the organization of agile software development. Information Systems Research 20(3), 355–376 (2009)
van Waardenburg, G., van Vliet, H.: When agile meets the enterprise. Information and Software Technology 55(12), 2154–2171 (2013). DOI 10.1016/j.infsof.2013.07.012. URL http://www.sciencedirect.com/science/article/pii/S0950584913001584
Walsham, G.: Interpretive case studies in is research: nature and method. European Journal of information systems 4(2), 74–81 (1995)
Watanabe, W.M., Fortes, R.P., Dias, A.L.: Using acceptance tests to validate accessibility requirements in ria. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, pp. 1–10 (2012)
Weippl, E.R.: Security in data warehouses. IGI Global, Data Ware-housing Design and Advanced Engineering Applications (2010)
Westerman, G., Tannou, M., Bonnet, D., Ferraris, P., McAfee, A.: The digital advantage: How digital leaders outperform their peers in every industry. MITSloan Management and Capgemini Consulting, MA 2, 2–23 (2012)
Weyuker, E.J.: Evaluating software complexity measures. Software Engineering, IEEE Transactions on 14(9), 1357–1365 (1988)
Whittaker, J.A.: Exploratory software testing: tips, tricks, tours, and techniques to guide test design. Pearson Education (2009)
Wieringa, R., Daneva, M.: Six strategies for generalizing software engineering theories. Science of computer programming 101, 136–152 (2015)
Wiklund, K., Sundmark, D., Eldh, S., Lundqvist, K.: Impediments in agile software development: An empirical investigation. In: International Conference on Product Focused Software Process Improvement, pp. 35–49. Springer (2013)
Williams, L., Cockburn, A.: Agile software development: it’s about feedback and change. IEEE computer 36(6), 39–43 (2003)
Wisell, D., Stenvard, P., Hansebacke, A., Keskitalo, N.: Considerations when designing and using virtual instruments as building blocks in flexible measurement system solutions. In: P. Stenvard (ed.) IEEE Instrumentation and Measurement Technology Conference, pp. 1–5 (2007)
Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., Grahn, H., Henningsson, K., Kagstrom, S., Low, G., et al.: The success factors powering industry-academia collaboration. IEEE software 29(2), 67–73 (2012)
Wohlin, C., Runeson, P., Host, M., Ohlsson, M.C., Regnell, B., Wessln, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publisher, Boston MA (2000)
Wohlrab, R., Knauss, E., Pelliccione, P.: Why and how to balance alignment and diversity of requirements engineering practices in automotive. Journal of Systems and Software 162, 110516 (2020). DOI https://doi.org/10.1016/j.jss.2019.110516. URL https://www.sciencedirect.com/science/article/pii/S0164121219302900
Wohlrab, R., Pelliccione, P., Knauss, E., Larsson, M.: Boundary objects and their use in agile systems engineering. J. Softw. Evol. Process. 31(5) (2019)
Wood, W., Tam, L., Witt, M.G.: Changing circumstances, disrupting habits. Journal of personality and social psychology 88(6), 918 (2005)
Woskowski, C.: Applying industrial-strength testing techniques to critical care medical equipment. In: International Conference on Computer Safety, Reliability, and Security, pp. 62–73. Springer (2012)
Xenos, M., Christodoulakis, D.: Measuring perceived software quality. Information and software technology 39(6), 417–424 (1997)
Yaman, S.G., Fagerholm, F., Munezero, M., Münch, J., Aaltola, M., Palmu, C., Männistö, T.: Transitioning towards continuous experimentation in a large software product and service development organisation–a case study. In: International Conference on Product-Focused Software Process Improvement, pp. 344–359. Springer (2016)
Yaman, S.G., Munezero, M., Münch, J., Fagerholm, F., Syd, O., Aaltola, M., Palmu, C., Männistö, T.: Introducing continuous experimentation in large software-intensive product and service organisations. Journal of Systems and Software 133, 195–211 (2017)
Yin, R.K.: Case study research design and methods third edition. Applied social research methods series 5 (2003)
Yin, R.K.: Case study research and applications: Design and methods. Sage publications (2017)
Yli-Huumo, J., Maglyas, A., Smolander, K.: How do software development teams manage technical debt?–an empirical study. Journal of Systems and Software 120, 195–218 (2016)
Yuan, D., Park, S., Zhou, Y.: Characterizing logging practices in open-source software. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 102–112. IEEE (2012)
Yuksel, H.M., Tuzun, E., Gelirli, E., Biyikli, E., Baykal, B.: Using continuous integration and automated test techniques for a robust c4isr system. In: 2009 24th International Symposium on Computer and Information Sciences, pp. 743–748. IEEE (2009)
Zaborovsky, A.N., Danilov, D.O., Leonov, G.V., Mescheriakov, R.V.: Software and hardware for measurements systems. In: D.O. Danilov (ed.) The IEEE-Siberian Conference on Electron Devices and Materials, pp. 53–57. IEEE (2007)
Zazworka, N., Spínola, R.O., Vetro’, A., Shull, F., Seaman, C.: A case study on effectively identifying technical debt. In: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, pp. 42–47 (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Elsevier Inc. All rights reserved
About this chapter
Cite this chapter
Kasauli, R., Knauss, E., Horkoff, J., Liebel, G., Neto, F.G.d.O. (2020). Chapter 10 Requirements Engineering Challenges and Practices in Large-Scale Agile System Development. In: Bosch, J., Carlson, J., Holmström Olsson, H., Sandahl, K., Staron, M. (eds) Accelerating Digital Transformation. Springer, Cham. https://doi.org/10.1007/978-3-031-10873-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-10873-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-10872-3
Online ISBN: 978-3-031-10873-0
eBook Packages: Computer ScienceComputer Science (R0)