skip to main content
research-article

Software testing with an operational profile: OP definition

Published: 01 February 2014 Publication History

Abstract

This article is devoted to the survey, analysis, and classification of operational profiles (OP) that characterize the type and frequency of software inputs and are used in software testing techniques. The survey follows a mixed method based on systematic maps and qualitative analysis. This article is articulated around a main dimension, that is, OP classes, which are a characterization of the OP model and the basis for generating test cases. The classes are organized as a taxonomy composed of common OP features (e.g., profiles, structure, and scenarios), software boundaries (which define the scope of the OP), OP dependencies (such as those of the code or in the field of interest), and OP development (which specifies when and how an OP is developed). To facilitate understanding of the relationships between OP classes and their elements, a meta-model was developed that can be used to support OP standardization. Many open research questions related to OP definition and development are identified based on the survey and classification.

References

[1]
S. Arora, R. B. Misra, and V. M. Kumre. 2005. Software reliability improvement through operational profile driven testing. In Proc. of the 53rd Annual Reliability and Maintainability Symposium. 621--627.
[2]
L. D. Bousquet, F. Ouabdesselam, and J.-L. Richier. 1998. Expressing and implementing operational profiles for reactive software validation. In Proc. of the 9th International Symposium on Software Reliability Engineering (ISSRE’98). 222--230.
[3]
P. A. Brooks and A. M. Memon. 2007. Automated GUI testing guided by usage profiles. In Proc. of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE’07). 333--342.
[4]
K. Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage, Thousand Oaks, CA.
[5]
J. Corbin and A. Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory 3rd Ed. Sage, Thousand Oaks, CA.
[6]
D. Cramer. 1997. Basic Statistics for Social Research: Step-by-Step Calculations and Computer Techniques Using Minitab. Psychology Press.
[7]
C. Csallner and Y. Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Software Practice and Experience 34, 11, 1025--1050.
[8]
W. Dulz and F. Zhen. 2003. MaTeLo—Statistical usage testing by annotated sequence diagrams, Markov chains and TTCN-3. In Proc. of the 3rd International Conference on Quality Software. 336--342.
[9]
S. Elbaum and S. Narla. 2001. A methodology for operational profile refinement. In Proc. of the 2001 Annual Reliability and Maintainability Symposium (RAMS’01). 142--149.
[10]
D. F. Fabio, A. Carlomusto, A. Petrillo, and A. Ramondo. 2012. Human reliability analysis: A review of the state of the art. International Journal of Research in Management and Technology, 2, 1.
[11]
M. Gittens, H. Lutfiyya, and M. Bauer. 2004. An extended operational profile model. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 314--325.
[12]
B. G. Glaser and A. L. Strauss. 2009. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine de Gruyter, New York.
[13]
K. Gwet. 2002. Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical Methods for Inter-rater Reliability Assessment 1--6.
[14]
H. L. Guen and T. Thelin. 2003. Practical experiences with statistical usage testing. In Proc. of the 11th Annual Workshop on Software Test and Reliability Estimation Process at Software Technology and Engineering Practice (STEP’03). 87--93.
[15]
H. L. Guen, R. Marie, and T. Thelin. 2004. Reliability estimation for statistical usage testing using Markov chains. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 54--65.
[16]
D. Hamlet, D. Mason, and D. Woit. 2001. Theory of software reliability based on components. In Proc. of the 23rd International Conference on Software Engineering (ICSE’01). 361--370.
[17]
J. M. Hammersley and D. C. Handscomb. 1964. Monte Carlo Methods (Methuen's Monographs on Applied Probability and Statistics). Methuen, London.
[18]
D. Harel. 1987. Statecharts: A visual formalism for complex systems. Science of Computer Programming 8, 231--274.
[19]
B. Huang, M. Rodríguez, M. Li, J. Bernstein, and C. Smidts. 2011. Hardware error likelihood induced by the operation of software. IEEE Transactions on Reliability 60, 3, 622--639.
[20]
B. Huang, M. Rodríguez, M. Li, and C. Smidts. 2007. On the development of fault injection profiles. In Proc. of the 53nd Annual Reliability and Maintainability Symposium (RAMS’07). 226--231.
[21]
IEEE/ISO/IEC 24765. 2010. System and Software Engineering Vocabulary. Standards.
[22]
M. Jørgensen and M. Shepperd. 2007. A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering 33, 33--53.
[23]
B. D. Juhlin. 1992. Implementing operational profiles to measure system reliability. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’02). 286--295.
[24]
B. Kitchenham. 2004. Procedures for Performing Systematic Reviews. Technical Report. Keele University TR/SE-0401, NICTA Technical Report 0400011T.1, Keele University.
[25]
P. Koopman and J. Devale. 1999. Comparing the robustness of POSIX operating systems. In Proc. of the 29th IEEE International Symposium on Fault-Tolerant Computing. 30--37.
[26]
K. S. Kumar, R. B. Misra, and N. K. Goyal. 2008. Development of fuzzy software operational profile. International Journal of Reliability, Quality, and Safety Engineering 15, 581--597.
[27]
T. Muhr and S. Friese. 2004. User's Manual for ATLAS. ti 6.0. ATLAS. ti Scientific Software Development GmbH, Berline.
[28]
J. D. Musa. 1992. The operational profile in software reliability engineering: An overview. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’92). 140--154.
[29]
J. D. Musa. 1993. Operational profiles in software-reliability engineering. IEEE Software 10, 2, 14--32.
[30]
F. Ouabdesselam and I. Parissis. 1995. Constructing operational profiles for synchronous critical software. In Proc. of the 6th International Symposium on Software Reliability Engineering (ISSRE’95). 286--293.
[31]
K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. 71--80.
[32]
S. J. Prowell. 2003. JUMBL: A tool for model-based statistical testing. In Proc. of the 36th Hawaii International Conference on System Sciences (HICSS’03).
[33]
M. Popovic and J. Kovacevic. 2007. A statistical approach to model-based robustness testing. In Proc. of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07). 485--494.
[34]
A. Rosenberg and E. Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proc. of HLT-NAACL 2004: Short Papers. Association for Computational Linguistics, 77--80.
[35]
P. Runeson and C. Wohlin. 1993. Statistical usage testing for software reliability certification and control. In Proc. of the 1st European International Conference on Software Testing, Analysis and Review (EuroSTAR’93). 309--323
[36]
J. Saldana. 2012. The Coding Manual for Qualitative Researchers (No. 14). Sage.
[37]
B. Shneideman and C. Plaisant. 2010. Designing the User Interface: Strategies for Effective Human Interaction. Addison-Wesley.
[38]
R. Shukla, D. Carrington, and P. Strooper. 2004a. Systematic operational profile development for software components. In Proc. of the 11th Asia-Pacific Software Engineering Conference (APSEC’04). 528--537.
[39]
R. Y. Shukla, P. A. Strooper, and D. A. Carrington. 2004b. A framework for reliability assessment of software components. In Proc. of the 7th International Symposium on Component-based Software Engineering (CBSE7). 272--279.
[40]
J. Sim and C. C. Wright. 2005. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy 85, 3, 257--268.
[41]
C. Taylor, T. C. Gibbs and A. Lewins. 2005. Quality of Qualitative Analysis. Retrieved from http://onlineqda.hud.ac.uk/Intro_QDA/qualitative_analysis.php.
[42]
G. H. Walton, J. H. Poore, and C. J. Trammell. 1995. Statistical testing of software based on a usage model. Software—Practice and Experience 25, 1, 97--108.
[43]
Y. Wei, M. Rodríguez, and C. Smidts. 2010. PRA framework for software propagation analysis of failures. Journal of Risk and Reliability 224, 2, 113--135.
[44]
E. J. Weyuker. 1998. Testing component-based software: A cautionary tale. IEEE Software 15, 5, 54--59.
[45]
J. A. Whittaker and J. Voas. 2000. Toward a more reliable theory of software reliability. IEEE Computer 33, 12, 36--42.
[46]
J. A. Whittaker and J. H. Poore. 1993. Markov analysis of software specifications. ACM Transactions on Software Engineering and Methodology. 2, 1, 93--106.
[47]
J. A. Whittaker. and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Transactions on Software Engineering 20, 10, 812--824.
[48]
D. M. Woit. 1993. Specifying Operational Profiles for Modules. In Proc. of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’93). 2--10.
[49]
J. Yan, J. Wang, and H. W. Chen. 2004. Automatic generation of Markov chain usage models from real-time software UML Models. In Proc. of the 4th International Conference on Quality Software (QSIC’04). 22--31.
[50]
F. Zhen and C. Peng. 2004. A system test methodology based on the Markov chain usage model. In Proc. of the 8th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2004). 160--165.

Cited By

View all
  • (2024)Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous SystemsIEEE Robotics and Automation Letters10.1109/LRA.2024.33644719:4(3299-3306)Online publication date: Apr-2024
  • (2023)Measuring Reliability of A Web Portal Based on Testing ProfileComputers, Materials & Continua10.32604/cmc.2023.03145974:3(6641-6663)Online publication date: 2023
  • (2023)Operational profile development methodology for normative multi-agent systemsMultiagent and Grid Systems10.3233/MGS-22150719:1(1-22)Online publication date: 8-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys
ACM Computing Surveys  Volume 46, Issue 3
January 2014
507 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/2578702
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2014
Accepted: 01 August 2013
Revised: 01 July 2013
Received: 01 January 2011
Published in CSUR Volume 46, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Software testing
  2. operational profile
  3. software reliability
  4. taxonomy
  5. usage models

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)5
Reflects downloads up to 25 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous SystemsIEEE Robotics and Automation Letters10.1109/LRA.2024.33644719:4(3299-3306)Online publication date: Apr-2024
  • (2023)Measuring Reliability of A Web Portal Based on Testing ProfileComputers, Materials & Continua10.32604/cmc.2023.03145974:3(6641-6663)Online publication date: 2023
  • (2023)Operational profile development methodology for normative multi-agent systemsMultiagent and Grid Systems10.3233/MGS-22150719:1(1-22)Online publication date: 8-Jun-2023
  • (2021)Extending the Space of Software Test Monitoring: Practical ExperienceIEEE Access10.1109/ACCESS.2021.31361389(166166-166183)Online publication date: 2021
  • (2021)Assessing test artifact quality—A tertiary studyInformation and Software Technology10.1016/j.infsof.2021.106620139:COnline publication date: 1-Nov-2021
  • (2020)Testing Relative to Usage ScopeACM Transactions on Software Engineering and Methodology10.1145/338912629:3(1-24)Online publication date: 1-Jun-2020
  • (2020)Reliability Assessment of Service-based Software under Operational Profile UncertaintyReliability Engineering & System Safety10.1016/j.ress.2020.107193(107193)Online publication date: Aug-2020
  • (2020)Mobile Application Testing and AssessmentTheory and Applications of Dependable Computer Systems10.1007/978-3-030-48256-5_28(283-292)Online publication date: 22-May-2020
  • (2020)Hybrid Is Better: Why and How Test Coverage and Software Reliability Can Benefit Each OtherWeb Information Systems and Technologies10.1007/978-3-030-35330-8_2(25-38)Online publication date: 17-Jan-2020
  • (2019)Adaptive Test Case Allocation, Selection and Generation using Coverage Spectrum and Operational ProfileIEEE Transactions on Software Engineering10.1109/TSE.2019.2906187(1-1)Online publication date: 2019
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media