Abstract
Transactional systems are the core of the information systems of most organizations. Although there is general acknowledgement that failures in these systems often entail significant impact both on the proceeds and reputation of companies, the benchmarks developed and managed by the Transaction Processing Performance Council (TPC) still maintain their focus on reporting bare performance. Each TPC benchmark has to pass a list of dependability-related tests (to verify ACID properties), but not all benchmarks require measuring their performances. While TPC-E measures the recovery time of some system failures, TPC-H and TPC-C only require functional correctness of such recovery. Consequently, systems used in TPC benchmarks are tuned mostly for performance. In this paper we argue that nowadays systems should be tuned for a more comprehensive suite of dependability tests, and that a dependability metric should be part of TPC benchmark publications. The paper discusses WHY and HOW this can be achieved. Two approaches are introduced and discussed: augmenting each TPC benchmark in a customized way, by extending each specification individually; and pursuing a more unified approach, defining a generic specification that could be adjoined to any TPC benchmark.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Transaction Processing Performance Council, http://www.tpc.org/
IFIP WG10.4 on Dependable Computing And Fault Tolerance, http://www.dependability.org/wg10.4/
Laprie, J.C.: Dependable Computing: Concepts, Limits, Challenges. In: 25th Int. Symp. On Fault-Tolerant Computing: FTCS-25. IEEE Press, Los Alamitos
Avizienis, A., Laprie, J.-C., Randell, B.: Fundamental Concepts of Dependability: LAAS Research Report, N°1145 (April 2001)
Trivedi, K.S., Haverkort, B.R., Rindos, A., Mainkar, V.: Methods and Tools for Reliability and Performability: Problems and Perspectives. In: 7th Intl. Conf. on Techniques and Tools for Computer Performance Evaluation
Jenn, E., Arlat, J., Rimén, M., Ohlsson, J., Karlsson, J.: Fault Injection into VHDL Models: The MEFISTO Tool. In: Randell, B., Laprie, J.-C., Kopetz, H., Littlewood, B. (eds.) Predictably Dependable Computing Systems
Gray, J.: A Census of Tandem System Availability Between 1985 and 1990. IEEE Transactions on Reliability R-39(4), 409–418 (1990)
Hsueh, M.C., Tsai, T.K., Iyer, R.K.: Fault Injection Techniques and Tools. IEEE Computer 30(4), 75–82 (1997)
Carreira, J., Madeira, H., Silva, J.G.: Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers. IEEE Trans. on Software Engineering 24(2), 125–136 (1998)
Koopman, P., DeVale, J.: Comparing the Robustness of POSIX Operating Systems. In: 29th International Symposium on Fault-Tolerant Computing, FTCS-29, pp. 30–37
Arlat, J., Fabre, J.-C., Rodríguez, M., Salles, F.: Dependability of COTS Microkernel-based Systems. IEEE Transactions on Computers 51(2) (2002)
Vieira, M., Madeira, H.: A Dependability Benchmark for OLTP Application Environments. In: VLDB 2003 (2003)
Vieira, M.: Dependability Benchmarking for Transactional Systems, PhD Thesis, University of Coimbra, Portugal (2005)
Vieira, M., Madeira, H.: From Performance to Dependability Benchmarking: A Mandatory Path. In: Nambiar, R.O., Poess, M. (eds.) TPCTC 2009. LNCS, vol. 5895, pp. 67–83. Springer, Heidelberg (2009)
Poess, M., Nambiar, R.O., Vaid, K., Stephens, J.M., Huppler, K., Haines, E.: Energy Benchmarks: A Detailed Analysis. In: E-Energy 2010. ACM, New York (2010) ISBN: 978-1-4503-0042-1
Crolotte, A.: Issues in Benchmark Metric Selection. In: Nambiar, R.O., Poess, M. (eds.) TPCTC 2009. LNCS, vol. 5895, pp. 146–152. Springer, Heidelberg (2009)
Lee, I., Iyer, R.K.: Software Dependability in the Tandem GUARDIAN System. IEEE Transactions on Software Engineering 21(5) (1995)
Kalyanakrishnam, M., Kalbarczyk, Z., Iyer, R.: Failure Data Analysis of a LAN of Windows NT Based Computers. In: Symposium on Reliable Distributed Database Systems, SRDS, vol. 18. IEEE Press, Los Alamitos (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Almeida, R., Poess, M., Nambiar, R., Patil, I., Vieira, M. (2011). How to Advance TPC Benchmarks with Dependability Aspects. In: Nambiar, R., Poess, M. (eds) Performance Evaluation, Measurement and Characterization of Complex Systems. TPCTC 2010. Lecture Notes in Computer Science, vol 6417. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-18206-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-18206-8_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-18205-1
Online ISBN: 978-3-642-18206-8
eBook Packages: Computer ScienceComputer Science (R0)