Elsevier

Information Sciences

Volume 328, 20 January 2016, Pages 237-249
Information Sciences

Assessment of software developed by a third-party: A case study and comparison

https://doi.org/10.1016/j.ins.2015.08.028Get rights and content

Abstract

Most of the research effort in the area of software analysis is focused on the perspective of the developer (as in “software developing company”) and the ways how the software development process could be improved. However, that is not the only type of software assessment common in the industry. There are also assessments that are commissioned by other parties, such as the primary recipients of the software solutions or courts dealing with legal cases that are related to software products or services. This work presents one such case-study that was performed for a public administration in Italy. The paper describes the assessment itself and also points out the need for more focused research by providing a comparison between developer-oriented and customer-oriented assessment types.

Introduction

Improvement of software and software development process requires sufficient level of understanding of what exactly is happening and why. As such, analysis of produced software and of the processes related to its development play a critical role both in the scientific area of software engineering and in practice. Afterall, it is very difficult, if not impossible, to control what cannot be understood. Unsurprisingly, a lot of attention is being spent by researchers on different low level aspects of software (e.g., code metrics) and on development process itself (e.g., process models and process metrics). Unfortunately, the amount of reported case-studies and experiences from the industry is relatively small. Furthermore, such studies primarily focus on a scenario where the main recipient of the analysis is the developer of the software. However, in practice there is also a different scenario that is quite common: when the assessment of the software is commissioned by non-developer. In such cases, the typical recipient of the analysis is either the main customer of the software system considered or a third-party, such as the court handling a legal case. Such assessments are quite different from those where the client is the developer. While the core of the analysis is still performed on the same target software, the focus and the objectives of such assessments are what set them apart. In order to better highlight and understand these differences, we propose to look at such cases as a separate type of software assessments. Within the scope of this paper, we classify software assessments into two categories:

  • 1.

    “Developer-oriented” – assessments ordered by the developer of the software.

  • 2.

    “Customer-oriented” – assessments ordered by a third-party.

This paper describes one such customer-oriented assessment and also provides a comparison between the two assessment types. The goal is to contribute to the body of knowledge on this topic by presenting a case-study from a public administration, and to encourage discussion and research focused more on the conceptual aspects of software assessments.

The industrial software assessment presented in this paper took place in Italy. It was commissioned by one of the public administrations. The target of the assessment was a complex supplier and data management system that was developed by an independent software development company that had won the public competition.

The provided comparison of assessment types is based on the personal experience of the authors gained from multiple previous software and software development process assessment projects with industrial partners.

The paper is structured as follows: In Section 2, covers important related works by other researchers as well as previous publications on the topic written by the authors; Section 3 describes the details of the assessment; Section 4 provides the general comparison between developer-oriented and customer-oriented type of assessments; Section 5 concludes with a brief summary of the paper.

Section snippets

Related work

As far as we are aware, there are no scientific publications about the assessments of software when such assessments are ordered by non-developer. However, a number of papers highlight certain aspects that we feel are important to such type of analysis or illustrative of its specific characteristics.

Case study

This section describes the details of the industrial software assessment project, including information related to the general context, assessment method, details on different parts of the performed analysis, and the final results.

Developer-oriented vs. customer-oriented software assessments

This section provides an analysis of our assessment experiences in different companies from the perspective of the client. It is meant to share our personal experiences and to facilitate further discussion on the merits of such classification and its implication for research.

Note that there are a number of different terms that are being used in research and industry that relate to the analysis of software. For example, IEEE Standard for Software Reviews and Audits [35] defines the following

Summary

This paper described a case-study in Italian public administration on the assessment of software system that was developed and delivered by a third-party. The goal of the assessment was to provide expert driven analysis of the system according to a set of predefined (by the customer) objectives, including the evaluation of the use of specific programming technologies (Spring framework and Dojo Toolkit), security assessment, stress testing, and so on. The paper also provided a general comparison

References (37)

  • AstromskisS. et al.

    Supporting CMMI assessment using distributed, non-invasive measurement and process mining

  • AstromskisS. et al.

    Continuous CMMI assessment using non-invasive measurement and process mining

    Int. J. Softw. Eng. Knowl. Eng.

    (2013)
  • ÇalıklıG. et al.

    Influence of confirmation biases of developers on software quality: an empirical study

    Softw. Qual. J.

    (2013)
  • ComanI. et al.

    An empirical exploratory study on inferring developers activities from low-level data

  • ComanI. et al.

    Automated identification of tasks in development sessions

  • ComanI.D. et al.

    Investigating the usefulness of pair-programming in a mature agile team

  • ComanI.D. et al.

    A case-study on using an automated in-process software engineering measurement and analysis system in an industrial environment

  • CorralL. et al.

    Software assurance practices for mobile applications

    Computing

    (March 2014)
  • di BellaE. et al.

    Pair programming and software defects–a large, industrial case study

    IEEE Trans. Softw. Eng.

    (2013)
  • DubeyS.K. et al.

    Usability estimation of software system by using object-oriented metrics

    ACM SIGSOFT Softw. Eng. Notes

    (2011)
  • DubeyS.K. et al.

    Assessment of maintainability metrics for object-oriented software system

    ACM SIGSOFT Softw. Eng. Notes

    (2011)
  • DubeyS.K. et al.

    Comparison of software quality models: an analytical approach

    Int. J. Emerg. Technol. Adv. Eng.

    (2012)
  • FerreiraK.A. et al.

    Identifying thresholds for object-oriented software metrics

    J. Syst. Softw.

    (2012)
  • FronzaI. et al.

    Failure prediction based on log files using random indexing and support vector machines

    J. Syst. Softw.

    (2013)
  • KovácsG.L. et al.

    Open source software for the public administration

  • LochmannK.

    A benchmarking-inspired approach to determine threshold values for metrics

    ACM SIGSOFT Softw. Eng. Notes

    (2012)
  • MaurerF. et al.

    Software process support over the internet

  • MordalK. et al.

    Software quality metrics aggregation in industry

    J. Softw. Evol. Process

    (2013)
  • Cited by (11)

    View all citing articles on Scopus
    View full text