Abstract
Collecting product and process measures in software development projects, particularly in education and training environments, is important as a basis for assessing current performance and opportunities for improvement. However, analyzing the collected data manually is challenging because of the expertise required, the lack of benchmarks for comparison, the amount of data to analyze, and the time required to do the analysis. ProcessPAIR is a novel tool for automated performance analysis and improvement recommendation; based on a performance model calibrated from the performance data of many developers, it automatically identifies and ranks potential performance problems and root causes of individual developers. In education and training environments, it increases students’ autonomy and reduces instructors’ effort in grading and feedback. In this article, we present the results of a controlled experiment involving 61 software engineering master students, half of whom used ProcessPAIR in a Personal Software Process (PSP) performance analysis assignment, and the other half used a traditional PSP support tool (Process Dashboard) for performing the same assignment. The results show significant benefits in terms of students’ satisfaction (average score of 4.78 in a 1–5 scale for ProcessPAIR users, against 3.81 for Process Dashboard users), quality of the analysis outcomes (average grades achieved of 88.1 in a 0–100 scale for ProcessPAIR users, against 82.5 for Process Dashboard users), and time required to do the analysis (average of 252 min for ProcessPAIR users, against 262 min for Process Dashboard users, but with much room for improvement).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alperowitz, L., Dzvonyar, D., Bruegge, B. (2016). Metrics in agile project courses. In Proceedings of the 38th international conference on software engineering companion (ICSE ’16) (pp. 323–326). New York: ACM, DOI https://doi.org/10.1145/2889160.2889183, (to appear in print).
Alves, T. (2012). Benchmark-based software product quality evaluation. PhD Thesis, U. Minho.
Alves, T., Ypma, C., Visser, J. (2010). Deriving metric thresholds from benchmark data. In 2010 IEEE international conference on software maintenance (ICSM’10) (pp. 1–10), IEEE. https://doi.org/10.1109/ICSM.2010.5609747.
Basili, V.R. (2007). The role of controlled experiments in software engineering research, empirical software engineering issues. Critical assessment and future directions. In Basili, V.R., Rombach, D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R. (Eds.) LNCS (pp. 33–37). Berlin: Springer.
Beck, K., & Andres, C. (2004). Extreme programming explained: embrace change, 2nd Edn. Addison-Wesley.
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J. (1983). Classification and regression trees. Belmont.
Fowler, M., Beck, K., Brant, J., Opdyke, W., Roberts, D. (1999). Refactoring: improving the design of existing code. Addison-Wesley.
Hackystat Development Team. (2010). Hackystat [online], available: http://code.google.com/p/hackystat/ (last release: January 2010; last visited October 2016).
Humphrey, W. (2005). PSP SM: a self-improvement process for software engineers. Addison-Wesley Professional.
Humphrey, W. (2009). The software quality profile. White paper, SEI.
Jedlitschka, A., Ciolkowski, M., Pfahl, D. (2008). Reporting experiments in software engineering. Guide to advanced empirical software engineering.
Ko, A.J., Latoza, T.D., Burnett, M.M. (2015). A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering, 20(1), 110–141.
Kumar, V., Kinshuk, Somasundaram, T., Harris, S., Boulanger, D., Seanosky, J., Paulmani, G., Panneerselvam, K. (2015). An approach to measure coding competency evolution: toward learning analytics, smart learning environments. In Chang, M. & Li, Y. (Eds.) Lecture notes in educational technology (pp. 27–43). Springer. https://doi.org/10.1007/978-3-662-44447-4_2.
Raza, M. (2017). Automated software process performance analysis and improvement recommendation. PhD Thesis, Faculty of Engineering of the University of Porto.
Raza, M., & Faria, J. (2016a). A model for analyzing performance problems and root causes in the personal software process. Journal of Software: Evolution and Process, 28(4), 254–271.
Raza, M., & Faria, J. (2016b). ProcessPAIR: a tool for automated performance analysis and improvement recommendation in software development. In Proceedings of the 31st IEEE/ACM international conference on automated software engineering (ASE 2016) (pp. 798–803). ACM. https://doi.org/10.1145/2970276.2970284.
Raza, M., Faria, J.J., Salazar, R. (2016). Empirical evaluation of the ProcessPAIR tool for automated performance analysis. In 28th international conference on software engineering and knowledge engineering (SEKE 2016). https://doi.org/10.18293/SEKE2016-205.
Raza, M., Faria, J., Salazar, R. (2017). Helping software engineering students analyzing their performance data: tool support in an educational environment. Published in 39th international conference on software engineering (ICSE). Buenos Aires, Argentina.
Rong, G., Zhang, H., Qi, S., Shao, D. (2016). Can engineering students program defect-free? An educational approach. In Proceedings of the 38th international conference on software engineering companion (ICSE ’16) (pp. 364–373). ACM. https://doi.org/10.1145/2889160.2889189.
Saltelli, A., Chan, K., Scott, E. (2008). Sensitivity analysis. Wiley.
Shalizi, C. (2009). Classification and regression trees, 36–350, Data Mining. Standford University, (lecture notes).
Sjøberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N.-K., Rekdal, A. (2005). A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31.9, 733–753.
Shadish, W.R., Cook, T.D., Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Shin, H., Choi, H., Baik, J. (2007). Jasmine: a PSP supporting tool. In Proceedings of the international conference on software process (ICSP 2007), LNCS 4470 (pp. 73–83). Springer. https://doi.org/10.1007/978-3-540-72426-1_7.
Thisuk, S., & Ramingwong, S. (2014). WBPS: a new web based tool for personal software process. In 2014 11th international conference on electrical engineering/electronics, computer, telecommunications and information technology. IEEE.
Tuma Solutions LLC. (2015). Process Dashboard [online], available: http://www.processdash.com (last release: December 2015; last visited October 2016).
Visser, J. (2015). Building maintainable software. O’Reilly, ISBN13: 9781491940662.
Acknowledgements
The authors would like to acknowledge the SEI and Tec de Monterrey for facilitating the access to the PSP data for performing this research and AWKUM for their partial initial grant. The authors would also like to acknowledge the students of Tec de Monterrey who participated in the controlled experiment. This work is partially financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within the project POCI-01-0145-FEDER-006961, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia as part of project UID/EEA/50014/2013 and research grant SFRH/BD/85174/2012.
Funding
This work is partially financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within the project POCI-01-0145-FEDER-006961, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia as part of project UID/EEA/50014/2013 and research grant SFRH/BD/85174/2012.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix: examples of support provided by ProcessPAIR and process dashboard for the PSP final report assignment
Appendix: examples of support provided by ProcessPAIR and process dashboard for the PSP final report assignment
In order to better understand and distinguish the support provided by ProcessPAIR and Process Dashboard for conducting the tasks in the controlled experiment, we next show the most relevant analysis views provided by both tools for answering some of the performance analysis questions to be addressed by the subjects (see Table 4). The questions belong to the “Analysis of time estimating accuracy” category.
B2: What percentage over or under the actual time was the estimated time for each program? What are your average, maximum, and minimum values for these?
Whilst Process Dashboard (Fig. 8, bottom) simply provides the raw data (Time Estimating Error), Process PAIR (Fig. 8, top) also provides the requested statistics and, most importantly, allows the user to assess his performance in comparison with thresholds derived from the historical dataset. In this example, the Time Estimation Accuracy is classified with the ’yellow’ semaphore, meaning that there is some room for improvement.
Relevant views in ProcessPAIR (top) and Process Dashboard (bottom) to answer question B2 in Table 4. In these charts, the Time Estimating Error is defined as (actual time - estimated time)/(estimated time), while the Time Estimation Accuracy is defined as (actual time)/(estimated time)
B5: Is my productivity stable? Why or why not?
This question requires the students to analyse one of the factors that affects the Time Estimation Accuracy. Whilst Process Dashboard (Fig. 9, bottom) simply provides the Productivity values, without any benchmarks for stability comparison, Process PAIR (Fig. 9, top) provides a specific indicator for the Productivity Stability (with defined values starting in program 3) and semaphore ranges derived from the historical dataset. In this example, the Productivity Stability is classified with the “green” semaphore, so it can be considered reasonably stable (as compared to the historical dataset). Furthermore, with Process PAIR, the user can also drill down to inspect the stability per phase and determine the problematic phases.
Relevant views in ProcessPAIR (top) and Process Dashboard (bottom) to answer question B5 in Table 4. The zero value in the bottom chart is in fact an undefined value
B7: How much are my time estimates affected by the accuracy of my size estimates?
This question requires the students to analyse the other factor that affects Time Estimation Accuracy —the Size Estimation Accuracy. With Process Dashboard (Fig. 10, bottom), the best thing that can be done is to inspect the “Size Estimating Error” and “Time Estimating Error” charts. By contrast, Process PAIR (Fig. 10, first on top) automatically performs a causal analysis, and points out the “Size Estimation Accuracy” as the most relevant factor affecting the “Time Estimation Accuracy”, with a moderate impact. The user can also do an interactive causal analysis, by drilling down the “Time Estimation Accuracy” indicator in the tree browser (Fig. 10, second on top). In this case, the only relevant factor is the “Size Estimation Accuracy” indicator, because it is the only one without a green semaphore. To visualize the relationship between both indicators, the user can plot both indicators in the same chart (Fig. 10, second on top). The linear correlation coefficient is automatically computed, showing a moderate value in this example (r= 0.68).
Relevant views in ProcessPAIR (top) and Process Dashboard (bottom) to answer question B7 in Table 4
Rights and permissions
About this article
Cite this article
Raza, M., Faria, J.P. & Salazar, R. Assisting software engineering students in analyzing their performance in software development. Software Qual J 27, 1209–1237 (2019). https://doi.org/10.1007/s11219-018-9433-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-018-9433-7