Understanding stack overflow code quality: A recommendation of caution

Abstract

Community Question and Answer (CQA) platforms use the power of online groups to solve problems, or gain information. While these websites host useful information, it is critical that the details provided on these platforms are of high quality, and that users can trust the information. This is particularly necessary for software development, given the ubiquitous use of software across all sections of contemporary society. Stack Overflow is the leading CQA platform for programmers, with a community comprising over 10 million contributors. While research confirms the popularity of Stack Overflow, concerns have been raised about the quality of answers that are provided to questions on Stack Overflow. Code snippets often contained in these answers have been investigated; however, the quality of these artefacts remains unclear. This could be problematic for the software engineering community, as evidence has shown that Stack Overflow snippets are frequently used in both open source and commercial software. This research fills this gap by evaluating the quality of code snippets on Stack Overflow. We explored various aspects of code snippet quality, including reliability and conformance to programming rules, readability, performance and security. Outcomes show variation in the quality of Stack Overflow code snippets for the different dimensions; however, overall, quality issues in Stack Overflow snippets were not always severe. Vigilance is encouraged for those reusing Stack Overflow code snippets.

Introduction

Community Question and Answer (CQA) platforms facilitate the use of the power of the crowd (i.e., online communities) to solve problems [32]. Platforms such as Stack Overflow and Yahoo!Answers¹ provide a service that benefits those who look to the internet to answer questions or find particular information [56]. In particular, such platforms benefit software practitioners seeking information, as it is likely that many other people have faced a similar problem, and so, a relevant question may have already been asked that has invoked a suitable answer. On the other hand, these platforms also allow new questions to be created, and experts can lend their specific experience, which allows them to solve a problem and gain the respect of their peers in the community.

While these websites host useful information, it is critical that the information provided on these platforms is of good quality, and that users can trust the information. This justifies a research agenda to afford this form of quality assurance. However, as this research area is still developing, even the terminologies used to identify such platforms are inconsistent. Srba and Bielikova [61] found that multiple terms are used for referring to CQA platforms. For example, while this paper refers to the types of community question and answer platforms as CQA, it is common for such platforms to be labelled Q&A, social Q&A and community forums [55], [57]. Beyond the terms used to identify CQA portals, Krüger et al. [41] conducted a secondary study of CQA papers and suggest that the quality of questions and answers is a central challenge of CQA systems. In addition, Srba and Bielikova [61] completed a CQA survey and noted preservation of long-term sustainability as a key area for future research of these platforms. For CQA platforms to have long-term sustainability, however, establishing the quality of these sites is key to ensuring that users trust the platform and feel that they can continue to participate. Other approaches for encouraging sustainability of CQA platforms in inspiring user participation are to provide tools and employ gamification techniques [33].

Stack Overflow has been noted as a successful platform due to its user participation, suggesting that its employment of gamification techniques indeed contributes to its prominence [16]. However, there are questions surrounding the quality of answers on Stack Overflow [25], [64]. In addition, given that developers use many of the code snippets in posts on Stack Overflow during development [62], it is important to evaluate the quality of these artefacts. In particular, while the Stack Overflow community's collective surveillance may help to identify and improve errors in code snippets on this platform and users may appropriately use code snippets by adapting them to their specific problem/task, this is not always the case. Wu et al. [68] examined how Stack Overflow code snippets were used in open-source projects and found that only 44% of the files containing Stack Overflow snippets showed that these snippets were modified prior to reuse by software developers. In fact, Bi [11] has shown that even the throwing/catching of generic exceptions is missed by software developers reusing Stack Overflow code snippets. We thus set out to understand the quality of code snippets that are often provided in Stack Overflow posts. The findings of this research could be helpful for directing future research on Stack Overflow, by facilitating investigations aimed at providing mechanisms to further scrutinise the aspects of quality that Stack Overflow code snippets do not meet. Additionally, developers selecting code snippets from Stack Overflow during software development will have a better understanding of the potential limitations of the code they use.

However, addressing the objective of this research project requires code snippet quality to be defined. This is done by assessing research and considering code snippet quality in relation to well established and understood software quality measurements [35]. This has led to our consideration of code reliability and conformance to programming rules, readability, performance and security (refer to Section 2.1 for discussion on this issue). Given our definition of quality, consisting of multiple dimensions, code snippets are extracted from Stack Overflow and analysed against these criteria. We then provide results at multiple levels of granularity, covering all violations (or errors), snippet specific violations and qualitative analysis assessing the implications for the presence of violations, and Stack Overflow community's efforts towards addressing these. The outcomes provided in this work are a survey of Stack Overflow code snippets' violations against these quality dimensions, as well as the types of violation that are evident in code provided by contributors. We also outline the basis for how the software development community may craft an agenda towards maintaining software quality, notwithstanding the utility that Stack Overflow provides.

The remaining sections of this study are structured as follows. Section 2 considers the literature relating to Stack Overflow, and outlines subsequent research questions. In Section 3, the proposed methodology of this research is introduced and discussed. This leads to Section 4, which presents the results of the study in relation to the associated research questions. Section 5 provides a discussion of the results along with the implications. The threats of the research are next discussed in Section 6. Finally, Section 7 concludes the work and outlines potential areas for future research.