Assessment of software developed by a third-party: A case study and comparison
Introduction
Improvement of software and software development process requires sufficient level of understanding of what exactly is happening and why. As such, analysis of produced software and of the processes related to its development play a critical role both in the scientific area of software engineering and in practice. Afterall, it is very difficult, if not impossible, to control what cannot be understood. Unsurprisingly, a lot of attention is being spent by researchers on different low level aspects of software (e.g., code metrics) and on development process itself (e.g., process models and process metrics). Unfortunately, the amount of reported case-studies and experiences from the industry is relatively small. Furthermore, such studies primarily focus on a scenario where the main recipient of the analysis is the developer of the software. However, in practice there is also a different scenario that is quite common: when the assessment of the software is commissioned by non-developer. In such cases, the typical recipient of the analysis is either the main customer of the software system considered or a third-party, such as the court handling a legal case. Such assessments are quite different from those where the client is the developer. While the core of the analysis is still performed on the same target software, the focus and the objectives of such assessments are what set them apart. In order to better highlight and understand these differences, we propose to look at such cases as a separate type of software assessments. Within the scope of this paper, we classify software assessments into two categories:
- 1.
“Developer-oriented” – assessments ordered by the developer of the software.
- 2.
“Customer-oriented” – assessments ordered by a third-party.
This paper describes one such customer-oriented assessment and also provides a comparison between the two assessment types. The goal is to contribute to the body of knowledge on this topic by presenting a case-study from a public administration, and to encourage discussion and research focused more on the conceptual aspects of software assessments.
The industrial software assessment presented in this paper took place in Italy. It was commissioned by one of the public administrations. The target of the assessment was a complex supplier and data management system that was developed by an independent software development company that had won the public competition.
The provided comparison of assessment types is based on the personal experience of the authors gained from multiple previous software and software development process assessment projects with industrial partners.
The paper is structured as follows: In Section 2, covers important related works by other researchers as well as previous publications on the topic written by the authors; Section 3 describes the details of the assessment; Section 4 provides the general comparison between developer-oriented and customer-oriented type of assessments; Section 5 concludes with a brief summary of the paper.
Section snippets
Related work
As far as we are aware, there are no scientific publications about the assessments of software when such assessments are ordered by non-developer. However, a number of papers highlight certain aspects that we feel are important to such type of analysis or illustrative of its specific characteristics.
Case study
This section describes the details of the industrial software assessment project, including information related to the general context, assessment method, details on different parts of the performed analysis, and the final results.
Developer-oriented vs. customer-oriented software assessments
This section provides an analysis of our assessment experiences in different companies from the perspective of the client. It is meant to share our personal experiences and to facilitate further discussion on the merits of such classification and its implication for research.
Note that there are a number of different terms that are being used in research and industry that relate to the analysis of software. For example, IEEE Standard for Software Reviews and Audits [35] defines the following
Summary
This paper described a case-study in Italian public administration on the assessment of software system that was developed and delivered by a third-party. The goal of the assessment was to provide expert driven analysis of the system according to a set of predefined (by the customer) objectives, including the evaluation of the use of specific programming technologies (Spring framework and Dojo Toolkit), security assessment, stress testing, and so on. The paper also provided a general comparison
References (37)
- et al.
Supporting CMMI assessment using distributed, non-invasive measurement and process mining
- et al.
Continuous CMMI assessment using non-invasive measurement and process mining
Int. J. Softw. Eng. Knowl. Eng.
(2013) - et al.
Influence of confirmation biases of developers on software quality: an empirical study
Softw. Qual. J.
(2013) - et al.
An empirical exploratory study on inferring developers activities from low-level data
- et al.
Automated identification of tasks in development sessions
- et al.
Investigating the usefulness of pair-programming in a mature agile team
- et al.
A case-study on using an automated in-process software engineering measurement and analysis system in an industrial environment
- et al.
Software assurance practices for mobile applications
Computing
(March 2014) - et al.
Pair programming and software defects–a large, industrial case study
IEEE Trans. Softw. Eng.
(2013) - et al.
Usability estimation of software system by using object-oriented metrics
ACM SIGSOFT Softw. Eng. Notes
(2011)
Assessment of maintainability metrics for object-oriented software system
ACM SIGSOFT Softw. Eng. Notes
Comparison of software quality models: an analytical approach
Int. J. Emerg. Technol. Adv. Eng.
Identifying thresholds for object-oriented software metrics
J. Syst. Softw.
Failure prediction based on log files using random indexing and support vector machines
J. Syst. Softw.
Open source software for the public administration
A benchmarking-inspired approach to determine threshold values for metrics
ACM SIGSOFT Softw. Eng. Notes
Software process support over the internet
Software quality metrics aggregation in industry
J. Softw. Evol. Process
Cited by (11)
An Introduction to Programming Languages: Simultaneous Learning in Multiple Coding Environments
2023, Synthesis Lectures on Computer ScienceExamining Readiness of E-Learning Implementation using Information System Readiness Impact Model
2020, 2020 8th International Conference on Cyber and IT Service Management, CITSM 2020Comparison of systematically derived software metrics thresholds for object-oriented programming languages
2020, Computer Science and Information SystemsThe concept of constructing the designer for processes of algorithms
2019, Advances in Intelligent Systems and ComputingAn initial characterization of bug-injecting development sessions
2018, Proceedings - International Conference on Software EngineeringIagile: Mission critical military software development
2017, Proceedings - 2017 International Conference on High Performance Computing and Simulation, HPCS 2017