Influencing factors of defect removal: an empirical study based on industrial data

Software testing is an important process for software quality control, and it is also costly and time-consuming. It can easily take 50% of development time [1]. To gain high quality software, a frequently-adopted approach is to try to minimize the number of defects in the final product, and it depends on software testing effectiveness. Many studies show that testing process is among one of the most difficult management practices. It is the last step to remove the defects before the software product to be released. But how to evaluate the quality of testing, a common way is to measure the defects detected during testing. When a defect was detected, if we can know the probability of the defect can be fixed, we can economize the limited time to fix more defects. But how we know the defect can be fixed or not before fixing. In our study, we want to find out which factors affect the defect removal.

Whether the defects can be removed may be affected by many factors, for example, the engineers' experience, the defect severity, and the quality of modules that the defects belonged to. In this study, we only analyzed these three factors.

a. The engineers' experience. In the testing process, the personnel capability and experience of the people involved is important factor not only for defects detected but also for the defects removal. The experience of a person can reflect his proficiency in defects detected or defects removal, it can reflected the capability of the person to a certain extent. In the datasets we used, the engineer that be assigned a defect is the programmer who is responsible to the code that produce the defect.

b. The defect severity. It is one of important attributes for defects. It can reflect the degree of the risk or damage for software. In general, a defect has the more severe will have a greater probability to be removed.

c. The quality of modules. In this study, we only considered the number of detected defects as an indicator for the quality of modules.

We use ratio of defect removal as major indicator, analyze "which factors have impact on the defect removal?" This question can be split into the following three specific research questions: RQ1: Fixing defect is a human intensive activity, whether engineers' experience has significant impact on the defect removal?

RQ2: Regarding the defect severity, whether the different severity has significant difference for the ratio of defect removal? And does the defect severity affect the results in RQ1?

RQ3: Regarding the software module, size, complexity and quality are used to measure the module. Whether its difference has significant difference for the ratio of defect removal? And does it affect the results in RQ1?

Aiming at investigating the influencing factors of defect removal with three factors: engineers' experience, defect severity and quality of modules, we got data from ten industrial test projects, which executed system testing for five embedded software products and their ten versions, and we gain the following three findings:

a. Engineers' experience has significant impact on RODR. Usually it is said the more experience will have better defect removal. But our case showed a contrary finding. Multiple factors mixture analysis looks more benefit.

b. The higher severity defects have a higher RODR. Combine with experience, in most severity the more engineers' experience, the higher the RODR. And the range of RODR for EY3 is quite lager. Probably, it is why the lower average of RODR in this level.

The quality of modules has impact on RODR, but it is quite different in each quality level. In MC2 and MC3, the modules have more defect, we assume they will have more complexity and lower quality. The average of RODR descended with the experience increase and the range is quite large.

The project managers said the skilled engineers were assigned to complex module. The more complexity defect, the more difficult to be fixed. It is why the lower defect removal. But the large range means the performance of engineers is not consistent even though they are in the same level. It will be a meaningful improvement point.