Skip to main content
Springer Nature Link
Log in

Empirical Study on Benchmarking Software Development Tasks

  • Conference paper
Software Process Dynamics and Agility (ICSP 2007)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 4470))

Included in the following conference series:

  • 1073 Accesses

Abstract

Benchmarking is one of the most important methods to learn the best practices for software process improvement. However, in current software process context, benchmarking is mainly for projects rather than software development tasks. Can we benchmark software development tasks? If so, how to? Moreover, benchmarking software development tasks has to deal with multivariate and variable return to scale (VRS). This paper reports practical experience of benchmarking software development tasks under multivariate and VRS constraints using Data Envelopment Analysis (DEA). The analysis of experience data in Institute of Software, Chinese Academy of Sciences (ISCAS) indicates that the ideas and techniques of benchmarking software projects can be deployed at the software development task level. Moreover, results also show that DEA VRS model allows the developers to gain new insight about how to identify the relatively efficient tasks as the task performance benchmark and how to establish different reference sets for each relatively inefficient task under multivariate and VRS constraints. We thus recommend DEA VRS model be used as the default technique for appropriately benchmarking software development tasks. Our results are beneficial to software process improvement. To the best of our knowledge, we believe that it is the first time to report such comprehensive and repeatable results of benchmarking software development tasks using DEA.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Stensrud, E., Myrtveit, I.: Identifying High Performance ERP Projects. IEEE Transactions on Software Engineering 29(5), 398–416 (2003)

    Article  Google Scholar 

  2. Humphrey, W.S.: Introduction to the Team Software Process. Addison Wesley Professional, Reading (1999)

    Google Scholar 

  3. Stark, J.A., Crocker, R.: Trends in Software Process: The PSP and Agile Methods. IEEE Software 20(3), 89–91 (2003)

    Article  Google Scholar 

  4. Maxwell, K.D., Forselius, P.: Benchmarking Software Development Productivity. IEEE Software 17(1), 80–88 (2000)

    Article  Google Scholar 

  5. Farris, J.A., et al.: Evaluating the Relative Performance of Engineering Design Projects: A Case Study Using Data Envelopment Analysis. IEEE Transactions on Engineering Management 53(3), 471–482 (2006)

    Article  Google Scholar 

  6. ISBSG: Worldwide Software Development – the Benchmark. Technical Report Release 5, International Software Benchmarking Standards Group (1998)

    Google Scholar 

  7. Myrtveit, I., Stensrud, E.: Benchmarking COTS Projects Using Data Envelopment Analysis. In: Sixth International Software Metrics Symposium, pp. 269–278 (1999)

    Google Scholar 

  8. Malaiya, Y.K., Denton, J.: Module Size Distribution and Defect Density. Software Reliability Engineering, 52–71 (2000)

    Google Scholar 

  9. Institute, P.M.: A Practice Standard for Earned Value Management, New Square, PA (2005)

    Google Scholar 

  10. Qing, W., Li, M.S.: Measuring and Improving Software Process in China. In: Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE’05), Australia, November 2005, pp. 17–18 (2005)

    Google Scholar 

  11. Scheel, H.: Undesirable Outputs in Efficiency Valuations. European Journal of Operation Reseach 132(2), 400–410 (2001)

    Article  MATH  Google Scholar 

  12. Charnes, A., Cooper, W.W., Rhodes, E.: Measuring the Efficiency of Decision Making Units. European Journal of Operation Research 2, 429–444 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  13. Banker, R.D., Charnes, A., Cooper, W.: Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment. Management Science 30, 1078–1092 (1984)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Software, Chinese Academy of Sciences, Beijing 100080, China

    Li Ruan, Yongji Wang, Qing Wang, Mingshu Li, Yun Yang, Lizi Xie, Dapeng Liu, Haitao Zeng, Shen Zhang, Junchao Xiao, Lei Zhang, M. Wasif Nisar & Jian Dai

  2. Graduate University, Chinese Academy of Sciences, Beijing 100039, China

    Li Ruan, Lizi Xie, Dapeng Liu, Haitao Zeng, Shen Zhang, Junchao Xiao, Lei Zhang, M. Wasif Nisar & Jian Dai

  3. Center for Information Technology Research, Swinburne University of Technology, Australia

    Yun Yang

Authors
  1. Li Ruan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongji Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingshu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yun Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lizi Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dapeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Haitao Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Shen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Junchao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. M. Wasif Nisar
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Jian Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Qing Wang Dietmar Pfahl David M. Raffo

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Berlin Heidelberg

About this paper

Cite this paper

Ruan, L. et al. (2007). Empirical Study on Benchmarking Software Development Tasks. In: Wang, Q., Pfahl, D., Raffo, D.M. (eds) Software Process Dynamics and Agility. ICSP 2007. Lecture Notes in Computer Science, vol 4470. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72426-1_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-72426-1_19

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-72425-4

  • Online ISBN: 978-3-540-72426-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics