Abstract
Reproducing and learning from failures in deployed software is costly and difficult. Those activities can be facilitated, however, if the circumstances leading to a failure can be recognized and properly captured. To anticipate failures we propose to monitor system field behavior for simple trace instances that deviate from a baseline behavior experienced in-house. In this work, we empirically investigate the effectiveness of various simple anomaly detection schemes to identify the conditions that precede failures in deployed software. The results of our experiment provide a preliminary assessment of these schemes, and expose the tradeoffs between different anomaly detection algorithms applied to several types of observable attributes under varying levels of in-house testing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Barbara D, Wu N, Jajodia S (2001) Detecting novel network intrusions using bayes estimators. In: Proceedings of the international conference on data mining, pp 15–24
Basili V, Perricone B, (1984) Software errors and complexity: an empirical investigation. Commun ACM 27(1):42–52
Briand L, Emam KE, Freimut B, Laitenberger, O (2000) A comprehensive evaluation of capture-recapture models for estimating software defect content. IEEE Trans Softw Eng 26(6):234–244
Chillarege R, Bhandar I, Chaar J, Halliday M, Moebus D, Ray B, Wong M.-Y (1992) Orthogonal defect classification—a concept for in-process measurements. IEEE Trans Softw Eng 18(11): 325–345
Debar H, Becker M, Siboni D (1992) A neural network component for intrusion detection techniques. In: Proceedings of the symposium on research in security and privacy, pp 240–250
Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Softw Eng 10(4):405–435
Eick S, Loader C, Long M, Votta L, Wiel S (1992) Estimating software fault content before coding. In: Proceedings of the international conference on software engineering, pp 59–65
Elbaum S, Hardojo M (2004) An empirical study of profiling strategies for released software and their impact on testing activities. In: Proceedings of the symposium on software testing, analysis, and verification, pp 65–75
Forrest S, Hofmeyr S, Somayaji A, Longstaff T (1996) A sense of self for unix processes. In: Proceedings of the symposium on security and privacy, pp 120–128
Goodwin G, Sin K (1984) Adaptive Fitering, Prediction and Control. Prentice-Hall, Englewood Cliffs, NJ
Graves T, Karr A, Marron J, Siy H (2000) Predicting fault incidence using software change history. IEEE Trans Softw Eng 26(7):653–661
Hangal S, Lam M (2002) Tracking down software bugs using automatic anomaly detection. In: Proceedings of the international conference on software engineering, pp 291–301
Havelund K, Rosu G (2001) Java pathexplorer—a runtime verification tool. In: Proceedings international symposium on artificial intelligence, robotics and automation in space
Hutchins M, Foster H, Goradia T, Ostrand T (1994) Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: Proceedings of the international conference on software engineering, pp 191–200
IEEE Computer Society (2002) (ed) IEEE standard classification for software anomalies, pp 1044–1993. The Institute of Electrical and Electronic Engineers, New York
Jha S, Tan K, Maxion R (2001) Markov chains, classifiers, and intrusion detection. In: Proceedings of the computer security foundation workshop, pp 206–219
Kanduri S, Elbaum S (2002) An empirical study of tracing techniques from a failure analysis perspective. In: Proceedings of the international symposium of software reliability engineering, pp 280–291
Khoshgoftaar T, Munson J (1990) Predicting software development errors using complexity metrics. IEEE J Sel Areas Commun 8(2):253–261
Lane T, Brodley CE (1997) An application of machine learning to anomaly detection. In: Proceedings of the NIST-NCSC national information systems security conference, pp 366–380
Lee I, Iyer K, Mehta A (1994) Identifying software problems using symptoms. In: Proceedings of the international symposium on fault-tolerant computing, pp 320–329
Liblit B, Aiken A (2002) Building a better backtrace: techniques for postmortem program analysis. Technical Report UCB/CSD:-02-1203, University of California–Berkely
Liblit B, Aiken A, Zheng A (2003) Distributed program sampling. In: Proceedings of the conference on programming language design and implementation, pp 1–11
Mahoney M, Chan P (2002) Learning nonstationary models of normal network traffic for detecting novel attacks. In: Proceedings of the international conference on knowledge discovery and data mining, pp 376–385
Maxion R, Tan K (2000) Benchmarking anomaly-based detection systems. In: International conference on dependable systems and networks, pp 623–630
Memon A, Porter A, Yilmaz C, Nagarajan A, Schmidt D, Natarajan B (2004) Skoll: distributed continuous quality assurance. In: Proceedings of the international conference on software engineering, pp 449–458
Munson J, Elbaum, S (1999) Software reliability as a function of user execution patterns. In: Hawaiian international conference on system sciences
Musa, J (1996) Software faults, software failures, software reliability modeling. IEEE Softw 6(2):85–91
Neumann P, Porras P (1999) Experience with EMERALD to date. In: Proceedings of the USENIX workshop on intrusion detection and network monitoring, pp 73–80
Raz O, Koopman P, Shaw M (2002) Semantic anomaly detection in online data sources. In: Proceedings of the international conference on software engineering, pp 302–312
Sekar R, Bendre M, Dhurjati D, Bollineni P (2001) A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the symposium on security and privacy, pp 144–155
Swets J, Pickett R, (1982) Evaluation of diagnstic systems: methods from signal detection theory. Academic, New York
Tan K, Maxion R (2002) Why 6? defining the operational limits of stide, an anomaly-based intrusion detector. In: Proceedings of the symposium on security and privacy, pp 188–201
Tan K, Maxion R (2005) The effects of algorithmic diversity on anomaly detector performance. In: Proceedings of the international conference on dependable systems and networks, pp 216–225
Teng H, Chen K, Lu S (1990) Security audit trail analysis using inductively generated predictive rules. In: Proceedings of the conference on AI applications, pp 24–29
Warrender C, Forrest S, Perlmutter B (1999) Detecting intrusions using system calls: alternative data models. In: Proceedings of the symposium on security and privacy, pp 302–145
Wiel VS, Votta L (1993) Assessing software designs using capture-recapture methods. IEEE Trans Softw Eng 19(11):1045–1054
Wohlin C, Runeson P (1995) An experimental evaluation of capture-recapture in software inspections. Softw Test Verif Reliab 5(4):213–232
Yu T, Shen V, Dunsmore H (1988) An analysis of several software defect models. IEEE Trans Softw Eng 14(9):1261–1270
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Elbaum, S., Kanduri, S. & Andrews, A. Trace anomalies as precursors of field failures: an empirical study. Empir Software Eng 12, 447–469 (2007). https://doi.org/10.1007/s10664-007-9042-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-007-9042-8