Keywords

1 Introduction

The use of design patterns [1,2,3] in software development is a practice that brings several advantages, including code reuse, better code organization, ease of maintenance, better scalability, and software quality [4, 5]. In the context of agile methods, these characteristics are essential to keep the code ready to change, enabling adjustments in requirements in small iterations. However, becoming proficient in understanding and applying these patterns can take time and effort [6], requiring a high degree of abstraction and practice in object-oriented programming. Educational institutions use various learning techniques [7, 8] to engage students with real-world problems in recognizing and interpreting these patterns.

ChatGPT is an AI-assisted tool capable of answering questions in a conversational way. Recently, this kind of tool has received a lot of attention from the media and the general public, raising questions about the potential of its application in several fields, such as public health [9] and academia [10]. Some recent works suggest its use for software engineering applications, such as solving programming bugs [11] and improving other kinds of activities, like requirements engineering and code quality [12]. GitHub Co-PilotFootnote 1 is an example of another tool in this direction.

The goal of this paper is to investigate if an AI-assisted tool can help developers to make design decisions, like suggesting appropriate design patterns to apply in a given scenario. To reach that goal, we performed an exploratory study considering 56 exercises and exam questions obtained from educational institutions and public tenders, which present a concrete context and ask which pattern should be applied. We used ChatGPT for this study, and we evaluated how many answers it answered correctly based on the question’s official answer and what elements that can help in the pattern implementation is present in the answers. For wrong answers, we also assessed if they could mislead the developer to the wrong path.

We found that the AI-assisted tool answered 93% of the questions correctly, always referring to specific details of the scenario and, in several cases providing guidance to the pattern implementation, including the classes and interfaces that should be created. In the few cases where the tool provided wrong answers, we found that they could be misleading and guide the developers in the wrong direction. Based on that, we consider the results of this initial assessment promising, pointing to the potential of such tools to assist in software design decisions. However, we also believe an experienced professional is still needed to evaluate the suitability of the suggested solution.

2 Research Design

Based on the goal of investigating if an AI-assisted tool can help developers in choosing the appropriate design pattern to apply in a given scenario, we formulated three more specific research questions:

  1. 1.

    \(RQ_1\) - How is the success rate of the AI-assisted tool in suggesting design patterns?

  2. 2.

    \(RQ_2\) - Do elements provided in the answer of the AI-assisted tool help in the pattern implementation?

  3. 3.

    \(RQ_3\) - In questions where the wrong pattern is pointed, does the AI-assisted tool answer mislead the developers?

To answer these research questions, we collected preexisting questions prepared by higher education professors or public tenders in which a scenario is described, and it asks which design pattern should be applied. Since these questions are used to evaluate the student’s knowledge of patterns, we believe they can also be used to evaluate the tool’s performance. We chose ChatGPT as the AI-assisted tool for the study and submitted the selected questions to it, registering the answers. The content of the answers was analyzed, not only comparing to the exercise correct answer (\(RQ_1\)) but also analyzing what elements that can help the developers in the pattern implementation were present (\(RQ_2\)). A qualitative analysis was done on the wrong answers to evaluate if they could mislead the developers (\(RQ_3\)).

To select the answers, we contacted professors that teach design patterns in universities asking for exam questions and exercises they give students. We’ve received the material from three professors from different countries: Italy, the United States, and Portugal. We also analyzed available exams of public tenders for 26 Brazilian institutions. As a question inclusion criteria for this study, we considered the following criteria: (a) the solution should be one of the “Gang of Four” (GoF) patterns; (b) the question should describe a scenario and ask which pattern should be applied; and (c) the question should describe a concrete scenario and not just give general requirements.

The first criterion was included to restrict the scope of the study to these patterns, which are well known by the software development community and taught in university courses, which enabled finding a good number of questions. The two other criteria were considered to choose questions that better simulate the decision of applying a pattern in a real scenario.

The qualitative analysis inspected each correct answer and evaluated the following criteria: (C1) If the answer mentions the described context (instead of just giving a generic answer); (C2) If the answer explains the pattern;(C3) If the answer describes the classes and interfaces that need to be created to implement the pattern; (C4) If the answer describes the methods that should be created to implement the pattern; (C5) If the answers included a UML diagram to represent the solution; (C6) If the answer described any trade-off or negative consequence in using of the proposed pattern.

3 Questions Database

In selecting questions, we obtained a total of 56 Design Pattern questions. There are 19 questions from university courses: 9 questions from the United States, 6 from Portugal, and 4 from Italy. All of these questions were formulated in English. We also found 37 questions in Portuguese from public tenders in Brazil, inspecting selection exams of twenty-nine different institutions. Since the questions obtained from teachers are used in exams and some asked to keep them private, we did not make the questions dataset open.

From the selected questions, all patterns in the study scope were covered with at least one question. Figure 1 presents how many questions have each pattern as a correct answer. Considering the pattern categories, there are 25 questions about behavioral pattern, 22 about structural patterns, and 17 about creational patterns. We consider that the question database is reasonably balanced among the pattern categories, having a good amount of questions that cover each one. Two questions accepted more than one pattern as a correct answer; in such cases, we consider any one of them as right. Four questions asked for a combination of patterns as an answer; in this case, we considered that the right answer should include all of them.

Fig. 1.
figure 1

Number of questions collected by design patterns.

Full size image

4 Results

The questions were sent to ChatGPT (version 3.5) in the period comprised between 2023-03-03 and 2023-04-16. Based on the results, ChatGPT was generally effective in answering our questions, with an overall accuracy rate of 93%. Most of the questions were in Portuguese, with 31 of the 34 questions answered correctly, while 21 of the 22 in English were answered correctly.

The answers have an average number of 1033 chars with a standard deviation of 657, varying from a minimum of 247 to a maximum of 3127. Considering the number of words, the average is 155, with a standard deviation of 89, varying from a minimum of 37 to a maximum of 390 words. We used the Pearson correlation coefficient to evaluate if the number of words on the question would be related to the number of words in the answer. The value of 0,09 obtained showed that there was no significant relation.

Information Present in the Answers. A qualitative analysis evaluated the six criteria proposed in Sect. 2. The following are the results:

  • (C1) 100% of the correct answers (52/52) mention the described context (instead of just giving a generic answer)

  • (C2) 57.69% of the correct answers (30/52) explain the pattern;

  • (C3) 32.69% of the correct answers (17/52) describe the classes and interfaces that need to be created to implement the pattern;

  • (C4) 26.92% of the correct answers (14/52) describe the methods that should be created to implement the pattern;

  • (C5) 3.85% of the correct answers (2/52) include a UML diagram to represent the solution;

  • (C6) None of the correct answers (0/52) describe any trade-off or negative consequence in using the proposed pattern.

Assessment of Wrong Answers. This section presents an analysis of the questions that received a wrong answer (the ID will be used to refer to that question later):

  • (WA1) The question presented a problem in creating a solution for reports needing two dimensions: the report type and the report format. While the correct answer was using the Visitor pattern, the tool suggested Strategy. In the answer, one of the dimensions of the question was ignored, and the proposed solution focused only on the report format.

  • (WA2) The question asked how the representation of more complex devices could be dynamically created by reusing the behavior of more simple devices that are part of it. While the correct answer was Composite, the given answer was Factory Method. In this case, the solution described does not help in solving the design problem presented.

  • (WA3) The question presented the context of exchanging messages between different cell phone operating systems platforms, handling distinct message formats transparently. While the correct answer was Adapter, the answer given was Bridge. In this case, even if the patterns share some structural similarities, the described solution does not help solve the problem.

  • (WA4) The question presented an abstract class that is responsible for transparently creating objects corresponding to different types of databases. Concrete subclasses are responsible for instantiating specific connections and queries for each type of database. The correct answer was Abstract Factory, and the given answer was Factory Method. Even if both are creational patterns, the given answer does not help to solve the problem because it was necessary to create a family of objects for each type of database.

5 Discussion

This section performs a discussion based on the obtained results, answering each research question. We also point out some limitations of the present study.

5.1 \(RQ_1\) - How is the Success Rate of the AI-Assisted Tool in Suggesting Design Patterns?

The study showed an efficient result with 93% correct answers in 56 questions. We highlight that the questions selected do not just ask for information about the patterns but present a specific and concrete scenario. To correctly answer the question by choosing the right pattern, the relevant factors that point to the usage of that specific pattern need to be identified in the context description. We also highlight that in the four questions that required a combination of patterns as an answer, the correct answer was provided by the AI-assisted tool.

ChatGPT performance can be considered good compared to other works that proposed more specific techniques for pattern suggestion [13, 14]. Consequently, it can be considered a suitable assisting tool for developers, which can describe the design scenario in their application and receive suggestions from the tool.

5.2 \(RQ_2\) - Do Elements Provided in the Answer of the AI-Assisted Tool Help in the Pattern Implementation?

One positive point is that all responses mentioned the problem’s context instead of just pointing to the pattern and providing a generic explanation. Around one-third of the answers also provided the name of the classes and interfaces that should be created in that case. Additionally, around one-quarter described the methods that should be created. Few answers, only 2, also provided UML diagrams with the structure that should be used.

Most of the answers, but not all, brought a more theoretical explanation of the pattern, which can be useful in case the developer is unfamiliar with the pattern. A piece of important information that was not present in any of the answers is the trade-off in the usage of the suggested pattern since knowing possible bad consequences is important to make design decisions.

To avoid any kind of bias, the questions used in this study were not modified; however, we acknowledge that if we explicitly included in the question what information we wanted in the answer, the tool would include it. We did some exploratory experiments on some of the questions, rephrasing them to ask explicitly for some missing information. As a result, the missing information was included correctly in the new answers. That fact highlights the importance of how the questions are formulated, a new disciple called prompt engineering [12].

5.3 \(RQ_3\) in Questions Where the Wrong Pattern is Pointed, Does the AI-Assisted Tool Answer Mislead the Developers?

We analyzed the four questions that received wrong answers to see if there was any inconsistency or ambiguity in the way they were formulated, and all authors agreed that they were clearly described and there was no doubt about the correct answer. For all the patterns that were the correct answer for these questions, there is at least another question with the same pattern answered correctly.

In the case of WA1, the answer given by ChatGPT considered the described context partially, and the Strategy implementation that it described could be evolved to a Visitor. However, in WA2, WA3, and WA4, the answer leads to a way that does not solve the problem. The answers are assertive and well-described, and we believe can easily be accepted by a less experienced developer. Because of that, we advise that the answers given by an AI-assisted tool should be reviewed and understood instead of being blinded implemented. In other words, there should be a professional that should be responsible for its analysis and adoption.

We also noticed that some of these questions used terms and expressions closely connected to other patterns. So, we suggest avoiding expressions like “family of algorithms” or “composite objects” that can drive the tool to a specific pattern.

5.4 Study Limitations

Our study focused only on “Gang of Four” (GoF) design patterns, which are well-known and widely used in the software development community. The performance of ChatGPT may differ for other design patterns that are newer and have less material available.

Our study was conducted in a controlled setting using preexisting questions. These questions were formulated by teachers with software design knowledge, focused on scenarios suitable for applying one of the patterns, and included all the necessary information relevant to direct the answer to one of the patterns. Using such a tool in a real project would require skills to identify the relevant forces in the scenario and formulate them properly [12]. Asking which pattern should be used might force the choice of one pattern in cases where none of them is suitable.

6 Conclusion

This paper described an exploratory study that investigated using an AI-assisted tool, in our case ChatGPT, to suggest design patterns using scenarios from existing exam questions. The tool answers have good accuracy, being correct in 93% of the cases, and always mention specific details of the scenarios, which provide guidance for the pattern implementation. In some cases, the answers also included information on classes, interfaces, and methods to be created, which we believe could be improved even more using prompt engineering techniques [12]. Another finding is that wrong answers, even if given in only a few cases, could be misleading and lead the developers in a direction that does not help the design problem described. In conclusion, our study suggests that AI-assisted tools can be a valuable resource for developers in understanding and applying design patterns; however, they should not replace human expertise, and the solutions should be reviewed before their adoption.

Future studies in this direction can focus on a different set of patterns that have less documentation and material available on the Internet. The result of this new study can be compared with the one made with the GoF patterns to assess if the AI-assisted tool can have a similar performance. Other studies can be conducted in more realistic scenarios using the description of design challenges faced by development teams and comparing the solutions suggested by an AI-assisted tool with the ones adopted in the projects.