research-article

Software testing with an operational profile: OP definition

Authors:

Manuel Rodríguez,

Matthew J. GerberAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 46, Issue 3

Article No.: 39, Pages 1 - 39

https://doi.org/10.1145/2518106

Published: 01 February 2014 Publication History

Abstract

This article is devoted to the survey, analysis, and classification of operational profiles (OP) that characterize the type and frequency of software inputs and are used in software testing techniques. The survey follows a mixed method based on systematic maps and qualitative analysis. This article is articulated around a main dimension, that is, OP classes, which are a characterization of the OP model and the basis for generating test cases. The classes are organized as a taxonomy composed of common OP features (e.g., profiles, structure, and scenarios), software boundaries (which define the scope of the OP), OP dependencies (such as those of the code or in the field of interest), and OP development (which specifies when and how an OP is developed). To facilitate understanding of the relationships between OP classes and their elements, a meta-model was developed that can be used to support OP standardization. Many open research questions related to OP definition and development are identified based on the survey and classification.

References

[1]

S. Arora, R. B. Misra, and V. M. Kumre. 2005. Software reliability improvement through operational profile driven testing. In Proc. of the 53rd Annual Reliability and Maintainability Symposium. 621--627.

[2]

L. D. Bousquet, F. Ouabdesselam, and J.-L. Richier. 1998. Expressing and implementing operational profiles for reactive software validation. In Proc. of the 9th International Symposium on Software Reliability Engineering (ISSRE’98). 222--230.

Digital Library

[3]

P. A. Brooks and A. M. Memon. 2007. Automated GUI testing guided by usage profiles. In Proc. of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE’07). 333--342.

Digital Library

[4]

K. Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage, Thousand Oaks, CA.

[5]

J. Corbin and A. Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory 3rd Ed. Sage, Thousand Oaks, CA.

[6]

D. Cramer. 1997. Basic Statistics for Social Research: Step-by-Step Calculations and Computer Techniques Using Minitab. Psychology Press.

[7]

C. Csallner and Y. Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Software Practice and Experience 34, 11, 1025--1050.

Digital Library

[8]

W. Dulz and F. Zhen. 2003. MaTeLo—Statistical usage testing by annotated sequence diagrams, Markov chains and TTCN-3. In Proc. of the 3rd International Conference on Quality Software. 336--342.

Digital Library

[9]

S. Elbaum and S. Narla. 2001. A methodology for operational profile refinement. In Proc. of the 2001 Annual Reliability and Maintainability Symposium (RAMS’01). 142--149.

[10]

D. F. Fabio, A. Carlomusto, A. Petrillo, and A. Ramondo. 2012. Human reliability analysis: A review of the state of the art. International Journal of Research in Management and Technology, 2, 1.

[11]

M. Gittens, H. Lutfiyya, and M. Bauer. 2004. An extended operational profile model. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 314--325.

Digital Library

[12]

B. G. Glaser and A. L. Strauss. 2009. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine de Gruyter, New York.

[13]

K. Gwet. 2002. Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical Methods for Inter-rater Reliability Assessment 1--6.

[14]

H. L. Guen and T. Thelin. 2003. Practical experiences with statistical usage testing. In Proc. of the 11th Annual Workshop on Software Test and Reliability Estimation Process at Software Technology and Engineering Practice (STEP’03). 87--93.

Digital Library

[15]

H. L. Guen, R. Marie, and T. Thelin. 2004. Reliability estimation for statistical usage testing using Markov chains. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 54--65.

Digital Library

[16]

D. Hamlet, D. Mason, and D. Woit. 2001. Theory of software reliability based on components. In Proc. of the 23rd International Conference on Software Engineering (ICSE’01). 361--370.

Digital Library

[17]

J. M. Hammersley and D. C. Handscomb. 1964. Monte Carlo Methods (Methuen's Monographs on Applied Probability and Statistics). Methuen, London.

[18]

D. Harel. 1987. Statecharts: A visual formalism for complex systems. Science of Computer Programming 8, 231--274.

Digital Library

[19]

B. Huang, M. Rodríguez, M. Li, J. Bernstein, and C. Smidts. 2011. Hardware error likelihood induced by the operation of software. IEEE Transactions on Reliability 60, 3, 622--639.

[20]

B. Huang, M. Rodríguez, M. Li, and C. Smidts. 2007. On the development of fault injection profiles. In Proc. of the 53nd Annual Reliability and Maintainability Symposium (RAMS’07). 226--231.

[21]

IEEE/ISO/IEC 24765. 2010. System and Software Engineering Vocabulary. Standards.

[22]

M. Jørgensen and M. Shepperd. 2007. A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering 33, 33--53.

Digital Library

[23]

B. D. Juhlin. 1992. Implementing operational profiles to measure system reliability. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’02). 286--295.

[24]

B. Kitchenham. 2004. Procedures for Performing Systematic Reviews. Technical Report. Keele University TR/SE-0401, NICTA Technical Report 0400011T.1, Keele University.

[25]

P. Koopman and J. Devale. 1999. Comparing the robustness of POSIX operating systems. In Proc. of the 29th IEEE International Symposium on Fault-Tolerant Computing. 30--37.

Digital Library

[26]

K. S. Kumar, R. B. Misra, and N. K. Goyal. 2008. Development of fuzzy software operational profile. International Journal of Reliability, Quality, and Safety Engineering 15, 581--597.

[27]

T. Muhr and S. Friese. 2004. User's Manual for ATLAS. ti 6.0. ATLAS. ti Scientific Software Development GmbH, Berline.

[28]

J. D. Musa. 1992. The operational profile in software reliability engineering: An overview. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’92). 140--154.

[29]

J. D. Musa. 1993. Operational profiles in software-reliability engineering. IEEE Software 10, 2, 14--32.

Digital Library

[30]

F. Ouabdesselam and I. Parissis. 1995. Constructing operational profiles for synchronous critical software. In Proc. of the 6th International Symposium on Software Reliability Engineering (ISSRE’95). 286--293.

[31]

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. 71--80.

Digital Library

[32]

S. J. Prowell. 2003. JUMBL: A tool for model-based statistical testing. In Proc. of the 36th Hawaii International Conference on System Sciences (HICSS’03).

Digital Library

[33]

M. Popovic and J. Kovacevic. 2007. A statistical approach to model-based robustness testing. In Proc. of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07). 485--494.

Digital Library

[34]

A. Rosenberg and E. Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proc. of HLT-NAACL 2004: Short Papers. Association for Computational Linguistics, 77--80.

Digital Library

[35]

P. Runeson and C. Wohlin. 1993. Statistical usage testing for software reliability certification and control. In Proc. of the 1st European International Conference on Software Testing, Analysis and Review (EuroSTAR’93). 309--323

[36]

J. Saldana. 2012. The Coding Manual for Qualitative Researchers (No. 14). Sage.

[37]

B. Shneideman and C. Plaisant. 2010. Designing the User Interface: Strategies for Effective Human Interaction. Addison-Wesley.

[38]

R. Shukla, D. Carrington, and P. Strooper. 2004a. Systematic operational profile development for software components. In Proc. of the 11th Asia-Pacific Software Engineering Conference (APSEC’04). 528--537.

Digital Library

[39]

R. Y. Shukla, P. A. Strooper, and D. A. Carrington. 2004b. A framework for reliability assessment of software components. In Proc. of the 7th International Symposium on Component-based Software Engineering (CBSE7). 272--279.

[40]

J. Sim and C. C. Wright. 2005. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy 85, 3, 257--268.

[41]

C. Taylor, T. C. Gibbs and A. Lewins. 2005. Quality of Qualitative Analysis. Retrieved from http://onlineqda.hud.ac.uk/Intro_QDA/qualitative_analysis.php.

[42]

G. H. Walton, J. H. Poore, and C. J. Trammell. 1995. Statistical testing of software based on a usage model. Software—Practice and Experience 25, 1, 97--108.

Digital Library

[43]

Y. Wei, M. Rodríguez, and C. Smidts. 2010. PRA framework for software propagation analysis of failures. Journal of Risk and Reliability 224, 2, 113--135.

[44]

E. J. Weyuker. 1998. Testing component-based software: A cautionary tale. IEEE Software 15, 5, 54--59.

Digital Library

[45]

J. A. Whittaker and J. Voas. 2000. Toward a more reliable theory of software reliability. IEEE Computer 33, 12, 36--42.

Digital Library

[46]

J. A. Whittaker and J. H. Poore. 1993. Markov analysis of software specifications. ACM Transactions on Software Engineering and Methodology. 2, 1, 93--106.

Digital Library

[47]

J. A. Whittaker. and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Transactions on Software Engineering 20, 10, 812--824.

Digital Library

[48]

D. M. Woit. 1993. Specifying Operational Profiles for Modules. In Proc. of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’93). 2--10.

Digital Library

[49]

J. Yan, J. Wang, and H. W. Chen. 2004. Automatic generation of Markov chain usage models from real-time software UML Models. In Proc. of the 4th International Conference on Quality Software (QSIC’04). 22--31.

Digital Library

[50]

F. Zhen and C. Peng. 2004. A system test methodology based on the Markov chain usage model. In Proc. of the 8th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2004). 160--165.

Cited By

Dong YZhao XWang SHuang X(2024)Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous SystemsIEEE Robotics and Automation Letters10.1109/LRA.2024.33644719:4(3299-3306)Online publication date: Apr-2024
https://doi.org/10.1109/LRA.2024.3364471
Muhammad Ali Shahid MSulaiman SAl-Sarem MUr Rahman AIqbal SNawaz Bashir RAhmad Khan AM. Alrawi MR. Marie RPoochaya S(2023)Measuring Reliability of A Web Portal Based on Testing ProfileComputers, Materials & Continua10.32604/cmc.2023.03145974:3(6641-6663)Online publication date: 2023
https://doi.org/10.32604/cmc.2023.031459
Menassel YMarir TMokhati F(2023)Operational profile development methodology for normative multi-agent systemsMultiagent and Grid Systems10.3233/MGS-22150719:1(1-22)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.3233/MGS-221507
Show More Cited By

Index Terms

Recommendations

Operational Profiles in Software-Reliability Engineering

A systematic approach to organizing the process of determining the operational profile for guiding software development is presented. The operational profile is a quantitative characterization of how a system will be used that shows how to increase ...
Software Operational Profile vs. Test Profile
SBQS '19: Proceedings of the XVIII Brazilian Symposium on Software Quality

Software Operational Profile (SOP) is a software specification based on how users use the software. This specification corresponds to a quantitative representation of software that identifies the most used software parts. As software reliability depends ...
Sensitivity of Two Coverage-Based Software Reliability Models to Variations in the Operational Profile
SSIRI '10: Proceedings of the 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement

Software in field use may be utilized by users with diverse profiles. The way software is used affects the reliability perceived by its users, that is, software reliability may not be the same for different operational profiles. Two software reliability ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 46, Issue 3

January 2014

507 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/2578702

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2014

Accepted: 01 August 2013

Revised: 01 July 2013

Received: 01 January 2011

Published in CSUR Volume 46, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
1,069
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)5

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dong YZhao XWang SHuang X(2024)Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous SystemsIEEE Robotics and Automation Letters10.1109/LRA.2024.33644719:4(3299-3306)Online publication date: Apr-2024
https://doi.org/10.1109/LRA.2024.3364471
Muhammad Ali Shahid MSulaiman SAl-Sarem MUr Rahman AIqbal SNawaz Bashir RAhmad Khan AM. Alrawi MR. Marie RPoochaya S(2023)Measuring Reliability of A Web Portal Based on Testing ProfileComputers, Materials & Continua10.32604/cmc.2023.03145974:3(6641-6663)Online publication date: 2023
https://doi.org/10.32604/cmc.2023.031459
Menassel YMarir TMokhati F(2023)Operational profile development methodology for normative multi-agent systemsMultiagent and Grid Systems10.3233/MGS-22150719:1(1-22)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.3233/MGS-221507
Lasynskyi MSosnowski J(2021)Extending the Space of Software Test Monitoring: Practical ExperienceIEEE Access10.1109/ACCESS.2021.31361389(166166-166183)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3136138
Tran HUnterkalmsteiner MBörstler JAli N(2021)Assessing test artifact quality—A tertiary studyInformation and Software Technology10.1016/j.infsof.2021.106620139:COnline publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.infsof.2021.106620
Miranda BBertolino A(2020)Testing Relative to Usage ScopeACM Transactions on Software Engineering and Methodology10.1145/338912629:3(1-24)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1145/3389126
Pietrantuono RPopov PRusso S(2020)Reliability Assessment of Service-based Software under Operational Profile UncertaintyReliability Engineering & System Safety10.1016/j.ress.2020.107193(107193)Online publication date: Aug-2020
https://doi.org/10.1016/j.ress.2020.107193
Jeleński MSosnowski J(2020)Mobile Application Testing and AssessmentTheory and Applications of Dependable Computer Systems10.1007/978-3-030-48256-5_28(283-292)Online publication date: 22-May-2020
https://doi.org/10.1007/978-3-030-48256-5_28
Bertolino AMiranda BPietrantuono RRusso S(2020)Hybrid Is Better: Why and How Test Coverage and Software Reliability Can Benefit Each OtherWeb Information Systems and Technologies10.1007/978-3-030-35330-8_2(25-38)Online publication date: 17-Jan-2020
https://doi.org/10.1007/978-3-030-35330-8_2
Bertolino AMiranda BPietrantuono RRusso S(2019)Adaptive Test Case Allocation, Selection and Generation using Coverage Spectrum and Operational ProfileIEEE Transactions on Software Engineering10.1109/TSE.2019.2906187(1-1)Online publication date: 2019
https://doi.org/10.1109/TSE.2019.2906187
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents