Skip to main content
Springer Nature Link
Log in

Component-Based Software Certification Based on Experimental Risk Assessment

  • Conference paper
Dependable Computing (LADC 2007)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 4746))

Included in the following conference series:

  • 532 Accesses

Abstract

Third-party software certification should attest that the software product satisfies the required confidence level according to certification standards such as ISO/IEC 9126, ISO/IEC 14598 or ISO/IEC 25051. In many application areas, especially in mission-critical applications, certification is essential or even mandatory. However, the certification of software products using common off-the-shelf (COTS) components is difficult to attain, as detailed information about COTS is seldom available. Nevertheless, software products are increasingly being based on COTS components, which mean that traditional certification processes should be enhanced to take COTS into account in an effective way. This paper proposes a mean to help in the certification of component-based systems through an experimental risk assessment methodology based on fault injection and statistical analysis. Using the proposed methodology the certification authority or the system integrator can compare among components available the one that best fit for the system that is assembling a component that provides a specific functionality. Based on the results it is also possible to decide whether a software product may be considered certified or not in what concerns the risk of using a COTS into the system. The proposed approach is demonstrated and evaluated using a space application running on top of two alternative COTS real-time operating systems: RTEMS and RTLinux.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Amland, S.: Risk-based Testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. The Journal of Systems and Software 53, 287–295 (2000)

    Article  Google Scholar 

  2. Arlat, J., et al.: Fault Injection and Dependability Evaluation of Fault Tolerant Systems. IEEE Transaction on Computers 42(8), 919–923 (1993)

    Article  Google Scholar 

  3. Bach, J.: Heuristic Risk-Based Testing. In: Software Testing and Engineering Magazine (1999)

    Google Scholar 

  4. Basili, V., Briand, L., Melo, W.: Measuring the Impact of Reuse on Quality and Productivity in Object-Oriented Systems. Technical Report, University of Maryland, Dep. Of Computer Science, CS-TR-3395 (1995)

    Google Scholar 

  5. Chillarege, R., Orthogonal Defect Classification, Ch. 9 of Handbook of Software Reliability Engineering, M. Lyu Ed., IEEE Computer Society, McGraw-Hill (1995)

    Google Scholar 

  6. Christmansson, J., Chillarege, R.: Generation of an Error Set that Emulates Software Faults-Based on Fields Data. In: Proc. of 26th Int. Symp. on Fault-Tolerant Computing, Sendai, Japan, pp. 304–313 (1996)

    Google Scholar 

  7. Colombo, R., Guerra, A.: The Evaluation Method for Software Product. In: ICSSEA 2002. Proc. of Int. Conf. on Software & Systems Engineering & Applications, Paris, France (2002)

    Google Scholar 

  8. Councill, B.: Third-Party Certification and Its Required Elements. In: Proc. of The 4th Workshop on Component-Based Software Engineering (CBSE), Springer, Heidelberg, Canada. Lecture Notes in Computer Science (LNCS) (2001)

    Google Scholar 

  9. Durães, J.: Madeira, H. Definition of Software Fault Emulation Operators: A Field Data Study. In: Proc. of The International Conference on Dependable Systems and Networks - DSN2003, pp. 105-114, San Francisco, USA (2003) (William Carter Award for the best student paper)

    Google Scholar 

  10. Durães, J., Madeira, H.: Emulation of Software Faults: A Field Data Study and a Practical Approach. IEEE Transactions on Software Engineering 32(11) (November 2006), ISSN: 0098-558

    Google Scholar 

  11. El Emam, K., Benlarbi, S., Goel, N., Rai, S.: Comparing Case-based Reasoning Classifiers for Predicting High Risk Software Components. Journal of Systems and Software 55(3), 301–320 (2001)

    Article  Google Scholar 

  12. EN 45020 General Terms and Definitions Concerning Standardization and Related Activities. CEN, Brussels (1993)

    Google Scholar 

  13. Fenton, N., Ohlsson, N.: Software Metrics and Risk. In: FESMA 1999. Proc. of The 2nd European Software Measurement Conference (1999)

    Google Scholar 

  14. Halstead, M.: Elements of Software Science. Elsevier Science Inc., New York (1977)

    MATH  Google Scholar 

  15. Hosmer, D., Lemeshow, S.: Applied Logistic Regression. John Wiley & Sons, Chichester (1989)

    Google Scholar 

  16. Health & Safety Commission The use of computers in Safety-critical Applications. Technical Report, UK (1998)

    Google Scholar 

  17. Hudepohl, et al.: EMERALD: A Case Study in Enhancing Software Reliability. Proc. of IEEE Eight Int. Symposium on Software Reliability Engineering - ISSRE98 98, 85–91 (1998)

    Google Scholar 

  18. ISO/IEC 12119. International Organization For Standardization ISO/IEC 12119, Information Technology - Software packages - Quality requirements and testing, p. 16, Geneve (1994)

    Google Scholar 

  19. ISO/IEC 14598-1. International Organization For Standardization ISO/IEC 14598-1 Information Technology - Software product evaluation - Part 1: General Overview; Geneve ISO (1999)

    Google Scholar 

  20. ISO/IEC 9126-1. International Organization For Standardization ISO/IEC 9126-1, Software Engineering - Software product quality - Part 1: Quality Model; Geneve ISO (2001)

    Google Scholar 

  21. ISO/IEC 25051 Software Engineering - Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing, Final Draft International Standard (2006)

    Google Scholar 

  22. Iyer, R.: Experimental Evaluation. In: Special Issue FTCS-25 Silver Jubilee, 25th IEEE Symposium on Fault Tolerant Computing, pp. 115–132 (1995)

    Google Scholar 

  23. Jacobson, I., Griss, M., Jonsson, P.: Software Reuse: Architecture, Process and Organization for Business Success. Addison-Wesley, Longman (1997)

    Google Scholar 

  24. Khoshgoftaar, et al.: Process Measures for Predicting Software Quality. In: Proc of High Assurance System Engineering Workshop - HASE 1997 (1997)

    Google Scholar 

  25. Leveson, N.: Safeware, System Safety and Computers. Addison-Wesley Publishing Company, Reading (1995)

    Google Scholar 

  26. The linux kernel. Accessed on Feb/06 (2006), http://www.kernel.org

  27. Lyu, M., Chen, J., Avizienis, A.: Experience in Metrics and Measurements for N-Version Programming. Int. Journal of Reliability, Quality and Safety Engineering 1(1), 41–62 (1994)

    Article  Google Scholar 

  28. Lyu, M.: Handbook of Software Reliability Engineering. IEEE omputer Society Press, McGraw-Hill, Los Alamitos (1996)

    Google Scholar 

  29. Madeira, H., Vieira, M., Costa, D.: On the Emulation of Software Faults by Software Fault Injection. In: Proc. of The Int. Conf. on Dependable Systems and Networks, NY, USA (2000)

    Google Scholar 

  30. Menzies, T., Greenwald, J., Frank, A.: Learning Defect Predictors. Journal (submitted 2006) (accessed, February 2006), http://menzies.us/

  31. Moraes, R., Durães, J., Martins, E., Madeira, H.: A field data study on the use of software metrics to define representative fault distribution. In: DSN 2006. Proc. of The International Conference on Dependable Systems & Networks, IEEE Computer Society Pres, Los Alamitos (2006)

    Google Scholar 

  32. Moraes, R., Durães, J., Barbosa, R., Martins, E., Madeira, H.: Experimental Risk Assessment and Comparison using Software Fault Injection. In: The International Conference on Dependable Systems and Networks - DSN 07, Edimburgo (2007)

    Google Scholar 

  33. Morris, J., Lee, G., Parker, K., Bundell, G., Lam, C.: Software Component Certification. IEEE Computer 34(9), 30–36 (2001)

    Google Scholar 

  34. Musa, J.: Software Reliability Engineering. McGraw-Hill, New York (1996)

    Google Scholar 

  35. Munson, J., Khoshgoftaar, T.: Software Metrics for Reliability Assessment. In: Michael, R. (ed.) Handbook of Software Reliability Engineering, IEEE Comp. Society Press, Los Alamitos (1995)

    Google Scholar 

  36. Kitchenham, B., Pfleeger, S., Fenton, N.: Towards a framework for software measurement validation. IEEE Transactions on Software Engineering 21(12), 929–944 (1995)

    Article  Google Scholar 

  37. Rodríguez-Dapena, P.: Software Safety Certification: A Multidomain Problem. IEEE Software 16(4), 31–38 (1999)

    Article  Google Scholar 

  38. Rome Laboratory (RL). Methodology for Software Reliability Prediction and Assessment. Technical Report RL-TR-92-52, vol. 1 and 2 (1992)

    Google Scholar 

  39. Rosenberg, L., Stapko, R., Gallo, A.: Risk-based Object Oriented Testing. In: Proc of. 13th International Software / Internet Quality Week-QW, San Francisco, California, USA 2 (2000)

    Google Scholar 

  40. Resource Standard Metrics, Version 6.1 (2005), Last access: http://msquaredtechnologies.com/m2rsm/rsm.htm

  41. Real-Time Operating System for Multiprocessor Systems. (February 2006) (accessed), http://www.rtems.com

  42. Rushby, John Modular Certification. Langley Research Center. Report Number: NAS 1.26212130, NASA CR-2002-212130, SRI-11003

    Google Scholar 

  43. Sherer, S.: A Cost-Effective Approach to Testing. IEEE Software 8(2), 34–40 (1991)

    Article  Google Scholar 

  44. Singpurwalla, N.: Statistical Methods in Software Engineering: Reliability and Risk, 1st edn. Springer, Heidelberg (1999)

    MATH  Google Scholar 

  45. Stafford, J., Wallnau, K.: Is Third-Party Certification Necessary? In: Proceedings of the 4th ICSE Workshop on Component-Based Software Engineering, Toronto, Canada, May, Toronto, Canada, pp. 13–17 (2001)

    Google Scholar 

  46. Systems Integration Requirements Task Group Certification Considerations for Highly-Integrated or Complex Aircraft Systems, Technical Report AS-1C, ASD, SAE (1996)

    Google Scholar 

  47. Tang, M., Kao, M., Chen, M.: An Empirical Study on Object-Oriented Metrics. In: Proceedings of the Sixth International Software Metrics Symposium, pp. 242–249 (1999)

    Google Scholar 

  48. Nuclear Safety Directorate Computer Based Safety Systems. Technical Assessment Guide T/AST/046, UK (2000)

    Google Scholar 

  49. Testwell Oy Ltd. (2006) (accessed on March 2006), http://www.testwell.fi

  50. Voas, J.: Certifying Off-the-Shelf Software Components. IEEE Computer 31(6), 53–59 (1998)

    Google Scholar 

  51. Voas, J.: Certifying Software for High-Assurance Environments. IEEE Software 16(4), 48–54 (1999)

    Article  Google Scholar 

  52. Weyuker, E.: Testing Component-Based Software: A Cautionary Tale. IEEE Software (1998)

    Google Scholar 

  53. Yang, Y., Boehm, B., Clark, B.: Assessing COTS Integration Risk Using Cost Estimation Inputs. In: Proc. of 28th International Conference on Software Engineering, Shangai, China (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State University of Campinas, UNICAMP, São Paulo, Brazil 13.084-971 – Campinas – SP, Brasil

    Regina Moraes & Eliane Martins

  2. CISUC, University of Coimbra, Portugal 3030-290 – Coimbra, Portugal

    Henrique Madeira

  3. CISUC, ISEC, 3030-290 – Coimbra, Portugal

    João Durães

Authors
  1. Regina Moraes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. João Durães
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eliane Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Henrique Madeira
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Andrea Bondavalli Francisco Brasileiro Sergio Rajsbaum

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Moraes, R., Durães, J., Martins, E., Madeira, H. (2007). Component-Based Software Certification Based on Experimental Risk Assessment. In: Bondavalli, A., Brasileiro, F., Rajsbaum, S. (eds) Dependable Computing. LADC 2007. Lecture Notes in Computer Science, vol 4746. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75294-3_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-75294-3_14

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-75293-6

  • Online ISBN: 978-3-540-75294-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics