skip to main content
10.1145/2889160.2889231acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

The bones of the system: a case study of logging and telemetry at Microsoft

Published: 14 May 2016 Publication History

Abstract

Large software organizations are transitioning to event data platforms as they culturally shift to better support data-driven decision making. This paper offers a case study at Microsoft during such a transition. Through qualitative interviews of 28 participants, and a quantitative survey of 1,823 respondents, we catalog a diverse set of activities that leverage event data sources, identify challenges in conducting these activities, and describe tensions that emerge in data-driven cultures as event data flow through these activities within the organization. We find that the use of event data span every job role in our interviews and survey, that different perspectives on event data create tensions between roles or teams, and that professionals report social and technical challenges across activities.

References

[1]
J. K. Blomkvist, J. Persson, and J. Åberg. Communication through boundary objects in distributed agile teams. In CHI '15, pages 1875--1884, Apr. 2015.
[2]
J. A. Carlson. Avoiding traps in member checking. The Qualitative Report, 15(5):1102--1113, Sept. 2010.
[3]
J. Cito, P. Leitner, T. Fritz, and H. C. Gall. The making of cloud applications: An empirical study on software development for the cloud. In ESEC/FSE '15, pages 393--403, Aug. 2015.
[4]
M. Curtin and E. Fossey. Appraising the trustworthiness of qualitative studies: Guidelines for occupational therapists. Australian Occupational Therapy Journal, 54(2):88--94, June 2007.
[5]
D. Fisher, R. DeLine, M. Czerwinski, and S. Drucker. Interactions with big data analytics. Interactions, 19(3):50--59, 2012.
[6]
Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie. Where do developers log? An empirical study on logging practices in industry. In ICSE '14, pages 24--33, May 2014.
[7]
S. Hove and B. Anda. Experiences from conducting semi-structured interviews in empirical software engineering research. In METRICS '05, pages 23--32, 2005.
[8]
S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer. Enterprise data analysis and visualization: An interview study. Visualization and Computer Graphics, 18(12):2917--2926, 2012.
[9]
M. Kim, T. Zimmermann, R. DeLine, and A. Begel. The emerging role of data scientists on software development teams. Available as Microsoft Tech Report MSR-TR-2015-30, 2015.
[10]
C. P. Lee. Boundary negotiating artifacts: Unbinding the routine of boundary objects and embracing chaos in collaborative work. CSCW '07, 16(3):307--339, Apr. 2007.
[11]
W. G. Lutters and C. B. Seaman. Revealing actual documentation usage in software maintenance through war stories. Information and Software Technology, 49(6):576--587, June 2007.
[12]
T. Menzies and T. Zimmermann. Software analytics: so what? IEEE Software, 30(4):31--37, 2013.
[13]
R. Musson, J. Richards, D. Fisher, C. Bird, B. Bussone, and S. Ganguly. Leveraging the crowd: How 48,000 users helped improve Lync performance. IEEE Software, 30(4):38--45, July 2013.
[14]
S. Nadella. A data culture for everyone. http://blogs.microsoft.com/blog/2014/04/15/a-data-culture-for-everyone/, 2014.
[15]
A. Pecchia, M. Cinque, G. Carrozza, and D. Cotroneo. Industry practices and event logging: Assessment of a critical software development process. In ICSE '15, pages 169--178, May 2015.
[16]
J. Saldaña. The Coding Manual for Qualitative Researchers. SAGE Publications, 2009.
[17]
J. Sapsed. Postcards from the edge: Local communities, global programs and boundary objects. Organization Studies, 25(9):1515--1534, Nov. 2004.
[18]
F. L. Schmidt, J. E. Hunter, and A. N. Outerbridge. Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71(3):432--439, 1986.
[19]
W. Shang, M. Nagappan, A. E. Hassan, and Z. M. Jiang. Understanding log lines using development knowledge. In ICSME '14, pages 21--30, Sept. 2014.
[20]
S. L. Star and J. R. Griesemer. Institutional ecology, 'translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3):387--420, Aug. 1989.
[21]
A. L. Strauss. Qualitative Analysis for Social Scientists. Cambridge University Press, 1987.
[22]
D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. SherLog: Error diagnosis by connecting clues from run-time logs. In ACM SIGARCH News, volume 38, pages 143--154, 2010.
[23]
D. Yuan, S. Park, and Y. Zhou. Characterizing logging practices in open-source software. In ICSE '12, pages 102--112, 2012.
[24]
D. Yuan, J. Zheng, S. Park, Y. Zhou, and S. Savage. Improving software diagnosability via log enhancement. ACM Transactions on Computer Systems (TOCS), 30(1):4, 2012.
[25]
D. Zhang, S. Han, Y. Dang, J.-G. Lou, H. Zhang, and T. Xie. Software analytics in practice. IEEE Software, 30(5):30--37, 2013.

Cited By

View all
  • (2024)Towards robust log parsing using self-supervised learning for system security analysisIntelligent Data Analysis10.3233/IDA-23013328:4(1093-1113)Online publication date: 17-Jul-2024
  • (2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
  • (2023)Log-related Coding Patterns to Conduct Postmortems of Attacks in Supervised Learning-based ProjectsACM Transactions on Privacy and Security10.1145/356802026:2(1-24)Online publication date: 12-Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion
May 2016
946 pages
ISBN:9781450342056
DOI:10.1145/2889160
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. boundary object
  2. collaboration
  3. developer tools
  4. logging
  5. practices
  6. telemetry

Qualifiers

  • Research-article

Conference

ICSE '16
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)51
  • Downloads (Last 6 weeks)6
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Towards robust log parsing using self-supervised learning for system security analysisIntelligent Data Analysis10.3233/IDA-23013328:4(1093-1113)Online publication date: 17-Jul-2024
  • (2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
  • (2023)Log-related Coding Patterns to Conduct Postmortems of Attacks in Supervised Learning-based ProjectsACM Transactions on Privacy and Security10.1145/356802026:2(1-24)Online publication date: 12-Apr-2023
  • (2023)How Do Developers' Profiles and Experiences Influence their Logging Practices? An Empirical Study of Industrial PractitionersProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00080(855-867)Online publication date: 14-May-2023
  • (2023)A/B Integrations: 7 Lessons Learned from Enabling A/B Testing as a Product FeatureProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00033(304-314)Online publication date: 17-May-2023
  • (2023)The view on systems monitoring and its requirements from future Cloud-to-Thing applications and infrastructuresFuture Generation Computer Systems10.1016/j.future.2022.11.024141(243-257)Online publication date: Apr-2023
  • (2023)An interview study about the use of logs in embedded software engineeringEmpirical Software Engineering10.1007/s10664-022-10258-828:2Online publication date: 11-Feb-2023
  • (2022)An empirical study of log analysis at MicrosoftProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558963(1465-1476)Online publication date: 7-Nov-2022
  • (2022)QuLogProceedings of the 30th IEEE/ACM International Conference on Program Comprehension10.1145/3524610.3527906(275-286)Online publication date: 16-May-2022
  • (2022)A Survey On Log Research Of AIOps: Methods and TrendsMobile Networks and Applications10.1007/s11036-021-01832-3Online publication date: 5-Feb-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media