Alexander Tarvo
Peter F. Sweeney
ABSTRACT
Canary testing is an emerging technique that offers to minimize the risk of deploying a new version of software. It does so by slowly transferring load from the current to the new ("canary") version. As this ramp-up progresses, a human compares the performance and correctness of the two versions, and assesses whether to abort the canary version. For canary testing to be effective, a plethora of metrics must be analyzed, including CPU utilization and logged errors, across hundreds to thousands of machines. Performing this analysis manually is both time consuming and error prone. In this paper, we present CanaryAdvisor, a tool for automatic canary testing of cloud-based applications. CanaryAdvisor continuously monitors the deployed versions of an application and detects degradations in correctness, performance, and/or scalability. We describe our design and implementation of the CanaryAdvisor and outline open challenges.
- G. Casella and R. L. Berger. Statistical inference, volume 2. Duxbury Pacific Grove, CA, 2002.Google Scholar
- R. T. Fielding and R. N. Taylor. Principled design of the modern web architecture. ACM Trans. Internet Technol., 2(2):115–150, May 2002. Google ScholarDigital Library
- A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous java performance evaluation. In Proc. of the 22Nd Annual ACM Conference on Object-oriented Programming Systems and Applications, OOPSLA ’07, pages 57–76, 2007. Google ScholarDigital Library
- J. Humble and D. Farley. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. 1st edition, 2010. Google ScholarDigital Library
- O. Moser, F. Rosenberg, and S. Dustdar. Non-intrusive monitoring and service adaptation for ws-bpel. In Proc. of 17th Intl. Conference on World Wide Web, WWW ’08, pages 815–824, 2008. Google ScholarDigital Library
- T. Nguyen, B. Adams, Z. M. Jiang, A. Hassan, M. Nasser, and P. Flora. Automated detection of performance regressions using statistical process control techniques. In Proc. of the 3rd ACM. International Conference on Performance Engineering, ICPE ’12, pages 299–310, 2012. Google ScholarDigital Library
- M. Pradel, M. Huggler, and T. R. Gross. Performance regression testing of concurrent classes. In Proc. of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pages 13–25, 2014. Google ScholarDigital Library
- Q. Wang, J. Shao, F. Deng, and Y. Liu. An online monitoring approach for web service requirements. IEEE Trans. on Services Computing, 2:338–351, August 2009. Google ScholarDigital Library
Index Terms
- CanaryAdvisor: a statistical-based tool for canary testing (demo)
Recommendations
Examining Penetration Tester Behavior in the Collegiate Penetration Testing Competition
Penetration testing is a key practice toward engineering secure software. Malicious actors have many tactics at their disposal, and software engineers need to know what tactics attackers will prioritize in the first few hours of an attack. Projects like ...
Dynamic Canary Randomization for Improved Software Security
CISRC '16: Proceedings of the 11th Annual Cyber and Information Security Research ConferenceStack canaries are a well-known and effective technique for detecting and defeating stack overflow attacks. However, they are not perfect. For programs compiled using gcc, the reference canary value is randomly generated at program invocation and fixed ...
TSTL: a language and tool for testing (demo)
ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and AnalysisWriting a test harness is a difficult and repetitive program- ming task, and the lack of tool support for customized auto- mated testing is an obstacle to the adoption of more sophis- ticated testing in industry. This paper presents TSTL, the Template ...
Comments