Skip to main content
SpringerLink
Log in

On rapid releases and software testing: a case study and a semi-systematic literature review

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases in the context of a case study on Mozilla Firefox, and performs a semi-systematic literature review. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firefox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulates our findings with a Mozilla QA engineer. We find that rapid releases have a narrower test scope that enables a deeper investigation of the features and regressions with the highest risk. Furthermore, rapid releases make testing more continuous and have proportionally smaller spikes before the main release. However, rapid releases make it more difficult to build a large testing community , and they decrease test suite diversity and make testing more deadline oriented. In addition, our semi-systematic literature review presents the benefits, problems and enablers of rapid releases from 24 papers found using systematic search queries and a similar amount of papers found through other means. The literature review shows that rapid releases are a prevalent industrial practice that are utilized even in some highly critical domains of software engineering, and that rapid releases originated from several software development methodologies such as agile, open source, lean and internet-speed software development. However, empirical studies proving evidence of the claimed advantages and disadvantages of rapid releases are scarce.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  • Agarwal P (2011) Continuous scrum: Agile management of saas products, pp 51–60

  • Azham Z, Ghani I, Ithnin N (2011) Security backlog in scrum security practices, pp 414–417

  • Baskerville R, Ramesh B, Levine L, Pries-Heje J (2006) High-speed software development practices: What works, what doesn’t. IT Professional 8:29–36

    Article  Google Scholar 

  • Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th Working Conf. on Mining Software Repositories (MSR), pp 238–241

  • Baysal O, Kononenko O, Holmes R, Godfrey M (2012) The secret life of patches: a firefox case study, pp 447–455

  • Beck K, Andres C (2004) Extreme Programming explained: embrace change, 2nd edn. Addison-Wesley

  • Begel A, Nagappan N (2007) Usage and perceptions of agile software development in an industrial context: an exploratory study, pp 255–264

  • Bell R, Ostrand T, Weyuker E (2006) Looking for bugs in all the right places, vol. 2006, pp 61–71

  • Berner S, Weber R, Keller RK (2005) Observations and lessons learned from automated testing. In: Proceedings of the 27th international conference on Software engineering, ACM, pp 571–579

  • Boshuizen C, Marshall W, Bridges C, Kenyon S, Klupar P (2011) Learning to follow: embracing commercial technologies and open source for space missions 10:8097–8101

  • Brown AW (2011) A case study in agile-at-scale delivery. In: Proceeding of the 12th Intl. Conf. on Agile Processes in Software Engineering and Extreme Programming (XP), vol 77, pp 266–281

  • Byrt T, Bishop J, Carlin JB (1993) Bias, prevalence and kappa. J Clin Epidemiol 46:423–429

    Article  Google Scholar 

  • Carney PA, Sickles EA, Monsees BS, Bassett LW, Brenner RJ, Feig SA, Smith RA, Rosenberg RD, Bogart TA, Browning S et al (2010) Identifying minimally acceptable interpretive performance criteria for screening mammography 1. Radiology 255:354–361

    Article  Google Scholar 

  • Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43:551–558

    Article  Google Scholar 

  • Codabux Z, Williams B (2013) Managing technical debt: an industrial case study, pp 8–15

  • Cohan S (2007) Successful integration of agile development techniques within disa, pp 255–260

  • Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Academic Press

  • Cusumano MA, Yoffie DB (1999) Competing on internet time: Lessons from netscape and its battle with microsoft. Simon and Schuster

  • Gamma E (2005) Agile, open source, distributed, and on-time – inside the eclipse development process. Keynote at the 27th Intl. Conf. on Software Engineering (ICSE)

  • Greiler M, van Deursen A, Zaidman A (2012) Measuring test case similarity to support test suite understanding. In: Objects, models components patterns. Springer, pp 91–107

  • Gundebahar M, Kus Khalilov M (2013) A hybrid deployment model for financial systems with service oriented architecture (soa): Running from client via branch server, pp 365–370

  • Heikkilä V, Rautiainen K, Jansen S (2010) A revelatory case study on scaling agile release planning, pp 289–296

  • Hemmati H, Arcuri A, Briand L (2013) Achieving scalable model-based testing through test case diversity. ACM Trans Softw Eng Methodol (TOSEM) 22:6

    Article  Google Scholar 

  • Hodgetts P, Phillips D (2002) 30. In: eXtreme Adoption eXperiences of a B2B Start Up. Addison-Wesley Longman Publishing Co., Inc. Extreme Programming Perspectives

  • Hollander M, Wolfe DA (1999) Nonparametric statistical methods, 2nd edn. John Wiley and Sons inc

  • HP (2012) Shorten release cycles aby bringing developers to application lifecycle management. HP applications handbook, Retrieved on February 08, 2012

  • Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional

  • InvestmentWatch (2011) Mozilla puts out firefox 5.0 web browser which carries over 1,000 improvements in just about 3 months of development. http://bit.ly/aecRrL

  • Iwai A, Aoyama M (2011) Automotive cloud service systems based on service-oriented architecture and its evaluation, pp 638–645

  • Jansen S, Brinkkemper S (2006) Ten misconceptions about product software release management explained using update cost/value functions. In: Proceedings of the International Workshop on Software Product Management, pp 44–50

  • Jenkins J (2011) Velocity culture (the unmet challenge in ops). Presentation at O’Reilly Velocity Conference

  • Jones G, Leung V (2005) Visual surveillance: a systems engineering approach for rapid development. Number 2005-11033, pp 161–166

  • Kaply M (2012) Why do companies stay on old technology? Retrieved on January 12, 2012

  • Kettunen V, Kasurinen J, Taipale O, Smolander K (2010) A study on agility and testing processes in software organizations. In: Proc. of the 19th Intl. Symp. on Software Testing and Analysis (ISSTA), pp 231–240

  • Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? An empirical case study of mozilla firefox. In: MSR, pp 179–188

  • Kim S (2012) ppcor: Partial and semi-partial (part) correlation. http://bit.ly/XkWuyn

  • Kitchenham B (2004) Procedures for performing systematic reviews, vol. 33:2004

  • Kong S, Kendall JE, Kendall KE (2009) The challenge of improving software quality: Developers’ beliefs about the contribution of agile practices. In: Proc. of the Americas Conf. on Information Systems (AMCIS), p 12

  • Kühner G, Bluhm T, Heimann P, Hennig C, Kroiss H, Krom J, Laqua H, Lewerentz M, Maier J, Schacht J, Spring A, Werner A, Zilker M (2012) Progress on standardization and automation in software development on w7x. Fusion Eng Des 87:2232–2237

    Article  Google Scholar 

  • Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics159–174

  • Lavoie T, Merlo E (2013) How much really changes? a case study of firefox version evolution using a clone detector, pp 83–89

  • Li J, Moe NB, Dybå T (2010) Transition from a plan-driven process to scrum: a longitudinal case study on software quality. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 13:1–13:10

  • Li X, Li Y, Xie M, Ng S (2011) Reliability analysis and optimal version-updating for open source software. Inf Softw Technol 53:929–936

    Article  Google Scholar 

  • Ltd. RS (2013) Web browsers (global marketshare). http://bit.ly/81klgi

  • Maalej W (2009) Task-first or context-first? tool integration revisited, pp 344–355

  • Mäntylä MV, Itkonen J (2013) More testers âĂŞ the effect of crowd size and time restriction in software testing. Inf Softw Technol 55:986–1003

    Article  Google Scholar 

  • Mantyla M, Khomh F, Adams B, Engstrom E, Petersen K (2013) On rapid releases and software testing. In: Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM), pp 20–29

  • Marschall M (2007) Transforming a six month release cycle to continuous flow. In: Proc. of the conf. on AGILE, pp 395–400

  • Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) ’good’organisational reasons for’bad’software testing: An ethnographic study of testing in a small software company. In: Software Engineering, 2007. ICSE 2007. 29th International Conference on, IEEE, pp 602–611

  • McDaniel LS (1990) The effects of time pressure and audit program structure on audit performance. J Account Res 28:267–285

    Article  Google Scholar 

  • Middleton P (2001) Lean software development two case studies. Software Qual J 9:241–252

    Article  Google Scholar 

  • Middleton P, Joyce D (2012) Lean software management: Bbc worldwide case study. IEEE Trans Eng Manag 59:20–32

    Article  Google Scholar 

  • Mozilla (2013) Litmus wiki. http://mzl.la/evJmTW

  • Mozilla (2013) Moztrap wiki. http://bit.ly/XBGMfu

  • Mozilla (2013) Mozilla source code mercurial repositories

  • Olsson H, Alahyari H, Bosch J (2012) Climbing the “stairway to heaven” - a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software, pp 392–399

  • Olsson H, Bosch J, Alahyari H (2013) Towards r&d as innovation experiment systems: a framework for moving beyond agile software development, pp 798–805

  • Otte T, Moreton R, Knoell HD (2008) Applied quality assurance methods under the open source development model. In: Proceeding of the 32nd Annual IEEE Intl. Computer Software and Applications Conf. (COMPSAC), pp 1247–1252

  • Oza N, Ebert C, Abrahamsson P (2012) Lean software development. IEEE Softw 29:0022–25

    Google Scholar 

  • Paasivaara M, Lassenius C (2003) Collaboration practices in global inter-organizational software development projects. Softw Process Improv Pract 8:183–199

    Article  Google Scholar 

  • Patel C, Lycett M, Macredie R, De Cesare S (2006) Perceptions of agility and collaboration in software development practice, vol. 1, p 10c

  • Paul R (2011) Mozilla outlines 16-week firefox development cycle. http://bit.ly/fLHEfo

  • Paul R (2012) Firefox extended support will mitigate rapid release challenges. http://ars.to/M2TbFQ

  • Pérez-Castillo R, Sánchez-González L, Piattini M, García F, Garcia-Rodriguez de Guzman I (2011) Obtaining thresholds for the effectiveness of business process mining. In: Empirical software engineering and measurement (ESEM), 2011 International Symposium on IEEE, pp 453–462

  • Petersen K, Wohlin C (2009) A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. J Syst Softw 82:1479–1490

    Article  Google Scholar 

  • Petersen K, Wohlin C (2010) The effect of moving from a plan-driven to an incremental software development approach with agile practices. Empir Softw Eng 15:654–693

    Article  Google Scholar 

  • Poppendieck M (2007) Lean software development. In: Companion to the proceedings of the 29th International Conference on Software Engineering, IEEE Computer Society, pp 165–166

  • Porter A, Yilmaz C, Memon AM, Krishna AS, Schmidt DC, Gokhale A (2006) Techniques and processes for improving the quality and performance of open-source software. Softw Process Improv Pract 11:163–176

    Article  Google Scholar 

  • Rafi DM, Moses KRK, Petersen K, Mäntylä MV (2012) Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In: Automation of Software Test (AST), 2012 7th International Workshop on, IEEE, pp 36–42

  • Raymond ES (1999) The Cathedral and the Bazaar, 1st edn. O’Reilly & Associates, Inc., Sebastopol

    Google Scholar 

  • Reinertsen DG (2009) The principles of product development flow: second generation lean product development. Celeritas Redondo Beach, Canada

    Google Scholar 

  • Rogmann JJ (2013) Orddom: Ordinal dominance statistics. http://bit.ly/Y0K0eo

  • Ruhe G, Saliu MO (2005) The art and science of software release planning. IEEE Softw 22:47–53

    Article  Google Scholar 

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14:131–164

    Article  Google Scholar 

  • Schwaber K (1995) Scrum development process. In: Proceedings of the 10th Annual ACM Conference on Object Oriented Programming Systems, Languages, and Applications (OOPSLA), pp 117–134

  • Shankland S (2010) Google ethos speeds up chrome release cycle. http://cnet.co/wlS24U

  • Shankland S (2011) Rapid-release firefox meets corporate backlash. http://cnet.co/ktBsUU

  • Sicore D (2011) New channels for firefox rapid releases. http://bit.ly/hc1zmY

  • Sonnentag S, Brodbeck FC, Heinbokel T, Stolte W (1994) Stressor-burnout relationship in software development teams. J Occup Organ Psychol 67:327–341

    Article  Google Scholar 

  • Sundmark D, Petersen K, Larsson S (2011) An exploratory case study of testing in an automotive electrical system release process, pp 166–175

  • Szőke Á (2011) Conceptual scheduling model and optimized release scheduling for agile environments. Inf Softw Technol 53:574–591

    Article  Google Scholar 

  • Torkar R, Minoves P, Garrigós J (2011) Adopting free/libre/open source software practices, techniques and methods for industrial use. J Assoc Inf Syst 12:88–122

    Google Scholar 

  • Wikipedia (2013) Firefox release history. http://bit.ly/Ngvfln

  • Wikipedia (2013) Extended support release. http://bit.ly/ZlgqoM#Extended_Support_Release

  • Wikipedia (2013) Locale. http://bit.ly/2iJLwB

  • VersionOne (2012) 7th annual state of agile survey. http://www.versionone.com/pdf/7th-Annual-State-of-Agile-Development-Survey.pdf

Download references

Acknowledgment

We would like to thank the Mozilla QA engineer who provided feedback to our findings. His responses are accounts of personal experience and opinion, and are in no means whatsoever an official statement from Mozilla. This work has been partially supported by EU FP7 Grant 318082 (U-QASAR, http://www.uqasar.eu/), ELLIIT (the StrategicArea for ICT research, funded by the Swedish Government, http://www.liu.se/elliit), N4S (research program managed by DIGILE and funded by TEKES, http://www.n4s.fi/), and NSERC (Natural Sciences and Engineering Research Council of Canada).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Aalto University, Espoo, Finland

    Mika V. Mäntylä

  2. MCIS, Polytechnique Montréal, Québec, Canada

    Bram Adams

  3. SWAT, Polytechnique Montréal, Québec, Canada

    Foutse Khomh

  4. Department of Computer Science, Lund University, Lund, Sweden

    Emelie Engström

  5. School of Computing, Blekinge Institute of Technology, Karlskrona, Sweden

    Kai Petersen

Authors
  1. Mika V. Mäntylä
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bram Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Foutse Khomh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emelie Engström
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kai Petersen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mika V. Mäntylä.

Additional information

Communicated by: Yann-Gaél Guéhéneuc and Tom Mens

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mäntylä, M.V., Adams, B., Khomh, F. et al. On rapid releases and software testing: a case study and a semi-systematic literature review. Empir Software Eng 20, 1384–1425 (2015). https://doi.org/10.1007/s10664-014-9338-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-014-9338-4

Keywords

Search

Navigation