ABSTRACT
Most of our current understanding of how programmers perform various software maintenance and evolution tasks is based on controlled studies or interviews, which are inherently limited in size, scope, and realism. Replicating controlled studies in the field can both explore the findings of these studies in wider contexts and study new factors that have not been previously encountered in the laboratory setting. While replicating controlled studies in the field seems like an obvious next step in scientific progress, it is a step that has rarely been attempted, in part due to its complexity, which requires not only the industrial knowhow to implement a robust, scalable system, but the academic knowledge of how to design rigorous studies. In this talk, I will describe a few examples of successfully scaled studies, contrast them with less successful cases (including our own), and provide lessons learned. I will share the importance of collecting targeted information instead of generic logs, the insight that automated data collection paired with followup surveys is a powerful tool, and the nuances around what researchers can and cannot expect working developers to tolerate for the sake of research.
- S. Amann, S. Proksch, S. Nadi, and M. Mezini. A study of visual studio usage in practice. In Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER '16), 2016.Google ScholarCross Ref
- S. K. Bajracharya and C. V. Lopes. Analyzing and mining a code search engine usage log. Empirical Software Engineering, 17(4-5):424--466, Aug. 2012. Google ScholarDigital Library
- K. Damevski, D. Shepherd, and L. Pollock. A field study of how developers locate features in source code. Empirical Software Engineering, pages 1--24, Jan 2015. Google ScholarDigital Library
- M. Kersten and G. C. Murphy. Mylar: A degree-of-interest model for ides. In Proceedings of the 4th International Conference on Aspect-oriented Software Development, AOSD '05, pages 159--168, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- G. C. Murphy, M. Kersten, and L. Findlater. How are java software developers using the eclipse ide? IEEE Software, 23(4):76--83, July 2006. Google ScholarDigital Library
- E. Murphy-Hill, C. Parnin, and A. P. Black. How we refactor, and how we know it. In Proceedings of the 31st International Conference on Software Engineering, ICSE '09, pages 287--297, Washington, DC, USA, 2009. IEEE Computer Society. Google ScholarDigital Library
- S. Negara, M. Codoban, D. Dig, and R. E. Johnson. Mining Fine-grained Code Changes to Detect Unknown Change Patterns. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, pages 803--813, New York, NY, USA, 2014. ACM. Google ScholarDigital Library
- W. Snipes, V. Augustine, A. R. Nair, and E. M. Hill. Towards recognizing and rewarding efficient developer work patterns. In Proceedings of the 2013 International Conference on Software Engineering, pages 1277--1280, Piscataway, NJ, USA, 2013. IEEE Press. Google ScholarDigital Library
- W. Snipes, A. R. Nair, and E. Murphy-Hill. Experiences gamifying developer adoption of practices and tools. In Proceedings of the 36th International Conference on Software Engineering, pages 105--114, New York, NY, USA, 2014. ACM. Google ScholarDigital Library
- M. Vakilian and R. E. Johnson. Alternate refactoring paths reveal usability problems. pages 1106--1116. ACM Press, 2014. Google ScholarDigital Library
Index Terms
- Experiences in scaling field studies of software developer behavior: keynote for the software engineering research & industrial practice workshop
Comments