Abstract
The purpose of much computer science research is to invent algorithms, and generate evidence to convince others that the new methods are worthwhile. All too often, however, the work has no impact, because the supporting evidence does not meet basic standards of rigor or persuasiveness. Here the notion of “experiment” is explored, with reference to our current investigation into distributed text query processing on a cluster of computers. We describe some of the issues that we encountered, and lessons that can be applied by researchers designing experiments in other areas of computing.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Badue, C., Baeza-Yates, R., Ribeiro-Neto, B., Ziviani, N.: Distributed query processing using partitioned inverted files. In: Navarro, G. (ed.) Proc. Symp. String Processing and Information Retrieval, Laguna de San Rafael, Chile, November 2001, pp. 10–20 (2001)
Bailey, P., Craswell, N., Hawking, D.: Engineering a multi-purpose test collection for web retrieval experiments. Information Processing & Management 39(6), 853–871 (2003)
Barroso, L.A., Dean, J., Hölzle, U.: Web search for a planet: The Google cluster architecture. IEEE Micro 23(2), 22–28 (2003)
Cacheda, F., Plachouras, V., Ounis, I.: Performance analysis of distributed architectures to index one terabyte of text. In: McDonald, S., Tait, J. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 394–408. Springer, Heidelberg (2004)
Cahoon, B., McKinley, K.S., Lu, Z.: Evaluating the performance of distributed architectures for information retrieval using a variety of workloads. ACM Transactions on Information Systems 18(1), 1–43 (2000)
Harman, D., McCoy, W., Toense, R., Candela, G.: Prototyping a distributed information retrieval system using statistical ranking. Information Processing & Management 27(5), 449–460 (1991)
Jeong, B.-S., Omiecinski, E.: Inverted file partitioning schemes in multiple disk systems. IEEE Transactions on Parallel and Distributed Systems 6(2), 142–153 (1995)
Johnson, D.S.: A theoretician’s guide to the experimental analysis of algorithms. In: Goldwasser, M., Johnson, D.S., McGeoch, C.C. (eds.) Proceedings of the 5th and 6th DIMACS Implementation Challenges, American Mathematical Society, Providence (2002)
Ribeiro-Neto, B.A., Barbosa, R.R.: Query performance for tightly coupled distributed digital libraries. In: Proc. 3rd ACM Conference on Digital Libraries, pp. 182–190. ACM Press, New York (1998)
Tichy, W.F.: Should computer scientists experiment more? IEEE Computer 31(5), 32–40 (1998)
Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A.: Experimental evaluation in computer science: a quantitative study. J. Systems and Software 28(1), 9–18 (1995)
Tomasic, A., García-Molina, H.: Performance of inverted indices in shared-nothing distributed text document information retrieval systems. In: Carey, M.J., Valduriez, P. (eds.) Proc. 2nd International Conference on Parallel and Distributed Information Systems, pp. 8–17. IEEE Computer Society Press, Los Alamitos (1993)
Williams, H.E., Zobel, J.: Compressing integers for fast file access. The Computer Journal 42(3), 193–201 (1999)
Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images, 2nd edn. Morgan Kaufmann, San Francisco (1999)
Zobel, J.: Writing for Computer Science, 2nd edn. Springer, London (2004)
Zobel, J., Moffat, A.: Adding compression to a full-text retrieval system. Software – Practice and Experience 25(8), 891–903 (1995)
Zobel, J., Moffat, A., Ramamohanarao, K.: Guidelines for presentation and comparison of indexing techniques. ACM SIGMOD Record 25(3), 10–15 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Moffat, A., Zobel, J. (2004). What Does It Mean to “Measure Performance”?. In: Zhou, X., Su, S., Papazoglou, M.P., Orlowska, M.E., Jeffery, K. (eds) Web Information Systems – WISE 2004. WISE 2004. Lecture Notes in Computer Science, vol 3306. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30480-7_1
Download citation
DOI: https://doi.org/10.1007/978-3-540-30480-7_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23894-2
Online ISBN: 978-3-540-30480-7
eBook Packages: Springer Book Archive