Skip to main content
SpringerLink
Log in
Book cover

Experimental Algorithmics pp 93–126Cite as

Using Finite Experiments to Study Asymptotic Performance

  • Chapter
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 2547))

Abstract

In the analysis of algorithms we are interested in obtaining closed form expressions for algorithmic complexity, or at least asymptotic expressions in O(·)-notation. It is often possible to use experimental results to make significant progress towards this goal, although there are fundamental reasons why we cannot guarantee to obtain such expressions from experiments alone. This paper investigates two approaches relating to problems of developing theoretical analyses based on experimental data.

We first consider the scientific method, which views experimentation as part of a cycle alternating with theoretical analysis. This approach has been very successful in the natural sciences. Besides supplying preliminary ideas for theoretical analysis, experiments can test falsifiable hypotheses obtained by incomplete theoretical analysis. Asymptotic behavior can also sometimes be deduced from stronger hypotheses which have been induced from experiments. As long as complete mathematical analyses remains elusive, well tested hypotheses may have to take their place. Several examples are given where average complexity can be tested experimentally so that support for hypotheses is quite strong.

A second question is how to approach systematically the problem of inferring asymptotic bounds from experimental data. Five heuristic rules for “empirical curve bounding” are presented, together with analytical results guaranteeing correctness for certain families of functions. Experimental evaluations of the correctness and tightness of bounds obtained by the rules for several constructed functions and real datasets are described.

Partially supported by the IST Programme of the EU under contract number IST-1999-14186 (ALCOM-FT).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. A. C. Atkinson. Plots, Transformations and Regression: an Introduction to Graphical Methods of Diagnostic Regression Analysis. Oxford University Press, U.K., 1987.

    Google Scholar 

  2. Y. Azar, A. Z. Broder, A. R. Karlin, and E. Upfal. Balanced allocations. In Proceedings of the 26th ACM Symposium on the Theory of Computation (STOC’94), pages 593–602, 1994.

    Google Scholar 

  3. D. Baldwin and J. A. G. M. Koomen. Using scientific experiments in early computer science laboratories. ACM SIGCSE Bulletin, 24(1):102–106, 1992.

    Article  Google Scholar 

  4. R. S. Barr, R. V. Helgaon, and J. L. Kennington. Minimal spanning trees: An empirical investigation of parallel algorithms. Parallel Computing, 12:45–52, 1989.

    Article  MATH  Google Scholar 

  5. R. A. Becker, J. A. Chambers, and A. R. Wilks. The New S Language: A Programming Enviornment for Data Analysis and Graphics. Wadsworth & Brooks/Cole, 1988.

    Google Scholar 

  6. J. L. Bentley, D. S. Johnson, F. T. Leighton, and C. C. McGeoch. An experimental study of bin packing. In Proceedings of the 21th Annual Allerton Conference on Communication, Control, and Computing, pages 51–60, 1983.

    Google Scholar 

  7. J. L. Bentley, D. S. Johnson, C. C. McGeoch, and L. A. McGeoch. Some unexpected expected behavior results for bin packing. In Proceedings of the 16th ACM Symposium on Theory of Computation (STOC’84), pages 279–298, 1984.

    Google Scholar 

  8. P. Berenbrink, A. Czumaj, A. Steger, and B. Vöcking. Balanced allocations: the heavily loaded case. In Proceedings of the 32nd ACM Symposium on the Theory of Computation (STOC’00), 2000.

    Google Scholar 

  9. R. D. Blumofe and C. E. Leiserson. Scheduling multithreaded computations by work stealing. In Proceedings of the 35th Symposium on Foundations of Computer Science (FOCS’94), pages 356–368, 1994.

    Google Scholar 

  10. G. P. Box, W. G. Hunter, and J. S. Hunter. Statistics for Experimenters. John Wiley & Sons, Inc., Chichester, 1978.

    MATH  Google Scholar 

  11. J. M. Chambers, W. S. Cleveland, B. Kleiner, and P. A. Tukey. Graphical Methods for Data Analysis. Duxbury Pres, Boston, 1983.

    MATH  Google Scholar 

  12. P. R. Cohen. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, MA, and London, England, 1995.

    MATH  Google Scholar 

  13. C. Cohen-Tannoudji, B. Diu, and F. Laloë. Quantum Mechanics, volume 2. John Wiley & Sons, Inc., Chichester, 1977.

    Google Scholar 

  14. P. J. Denning. What is experimental computer science? Communications of the ACM, 23(10):543–544, 1980.

    Article  Google Scholar 

  15. P. J. Denning. Performance analysis: Experimental computer science at its best. Communications of the ACM, 24(11):725–727, 1981.

    Article  Google Scholar 

  16. N. Fenton, S. L. Pfleger, and R. L. Glass. Science and substance: A challenge to software engineers. IEEE Software, 11(4):86–95, 1994.

    Article  Google Scholar 

  17. J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, 42(2):201–212, 1994.

    Article  MATH  Google Scholar 

  18. T. Jiang, M. Li, and P. Vitányi. Average-case complexity of Shellsort. In Proceedings of the 26th International Colloquium on Automata, Languages and Programming (ICALP’99). Springer Lecture Notes in Computer Science 1644, pages 453–462, 1999.

    Chapter  Google Scholar 

  19. D. S. Johnson. A theoretician’s guide to the experimental analysis of algorithms, 1996. Manuscript.

    Google Scholar 

  20. D. E. Knuth. The Art of Computer Programming, Volume 3: Sorting and Searching. Addison-Wesley, Reading, MA, 2nd edition, 1998.

    Google Scholar 

  21. V. Kumar, A. Grama, A. Gupta, and G. Karypis. Introduction to Parallel Computing. Design and Analysis of Algorithms. Benjamin/Cummings, 1994.

    Google Scholar 

  22. T. Leighton. Introduction to Parallel Algorithms and Architectures. Morgan Kaufmann, 1992.

    Google Scholar 

  23. P. Lukowicz, E. A. Heinz, L. Prechelt, and W. F. Tichy. Experimental evaluation in computer science: A quantitative case study. Journal of Systems and Software, 28(1):9–18, 1995.

    Article  Google Scholar 

  24. M. Matsumoto and T. Nishimura. Mersenne twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Simulation, 8:3–30, 1998. http://www.math.keio.ac.jp/~matumoto/emt.html.

    Article  MATH  Google Scholar 

  25. C. C. McGeoch. Analyzing algorithms by simulation: Variance reduction techniques and simulation speedups. ACM Computing Surveys, 245(2):195–212, 1992.

    Article  Google Scholar 

  26. C. C. McGeoch. All pairs shortest paths and the essential subgraph. Algorithmica, 13:426–441, 1995.

    Article  MATH  MathSciNet  Google Scholar 

  27. C. C. McGeoch. Toward an experimental method for algorithm simulation, 1996.

    Google Scholar 

  28. C. C. McGeoch and B. Moret. How to present a paper on experimental work with algorithms. SIGACT News, 30(4):85–90, 1999.

    Article  Google Scholar 

  29. B. M. E. Moret. Towards a discipline of experimental algorithmics. In 5th DIMACS Challenge, DIMACS Monograph Series, 2000. to appear.

    Google Scholar 

  30. R. Niedermeier, K. Reinhard, and P. Sanders. Towards optimal locality in mesh-indexings. In Proceedings of the 11th International Conference on Fundamentals of Computation Theory (FCT’97). Springer Lecture Notes in Computer Science 1279, pages 364–375, 1997.

    Chapter  Google Scholar 

  31. K. R. Popper. Logik der Forschung. Springer-Verlag, Heidelberg, 1934. English Translation: The Logic of Scientific Discovery, Hutchinson, 1959.

    Google Scholar 

  32. W. H. Press, S. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C. Cambridge University Press, Cambridge, England, 2. edition, 1992.

    MATH  Google Scholar 

  33. J. O. Rawlings. Applied Regression Analysis: A Research Tool. Wadsworth & Brooks/Cole, 1988.

    Google Scholar 

  34. P. Sanders. Lastverteilungsalgorithmen für parallele Tiefensuche. Number 463 in Fortschrittsberichte, Reihe 10. VDI Verlag, 1997.

    Google Scholar 

  35. P. Sanders, S. Egner, and J. Korst. Fast concurrent access to parallel disks. In Proceedings of the 11th ACM-SIAM Symposium on Discrete Algorithms (SODA’00), pages 849–858, 2000.

    Google Scholar 

  36. C. Schaffer. Domain-Independent Scientific Function Finding. Ph.D. thesis, Department of Computer Science, Rutgers University, 1990.

    Google Scholar 

  37. Computer Science and Telecommunications Board. Academic careers for experimental computer scientists and engineers. Communications of the ACM, 37(4):87–90, 1994.

    Article  Google Scholar 

  38. R. Sedgewick. Analysis of shellsort and related algorithms. In Proceedings of the 4th European Symposium on Algorithms (ESA’96). Springer Lecture Notes in Computer Science 1136, pages 1–11, 1996.

    Google Scholar 

  39. D. L. Shell. A high-speed sorting procedure. Communications of the ACM, 2(7):30–33, 1958.

    Article  Google Scholar 

  40. J. Stoer and R. Bulirsch. Introduction to Numerical Analysis. Springer-Verlag, Heidelberg, 1993.

    MATH  Google Scholar 

  41. W. F. Tichy. Should computer scientists experiment more? Computer, 31(5):32–40, 1998.

    Article  MathSciNet  Google Scholar 

  42. J. W. Tukey. Exploratory Data Analysis. Addison-Wesley, Reading, MA, 1977.

    MATH  Google Scholar 

  43. B. Vöcking. How asymmetry helps load balancing. In Proceedings of the 40th Symposium on Foundations of Computer Science (FOCS’99), pages 131–140, 1999.

    Google Scholar 

  44. L. Weisner. Introduction to the Theory of Equations. The MacMillan Press Ltd., London, 1938.

    MATH  Google Scholar 

  45. M. A. Weiss. Empirical study of the expected running time of Shellsort. The Computer Journal, 34(1):88–91, 1991.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Max-Planck-Institut für Informatik, Saarbrücken, Germany

    Catherine McGeoch

  2. Amherst College, Amherst, MA, USA

    Peter Sanders

  3. The Hong Kong University of Science and Technology, Hong Kong

    Rudolf Fleischer

  4. University of Massachussetts, Amherst, MA

    Paul R. Cohen & Doina Precup

Authors
  1. Catherine McGeoch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Sanders
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rudolf Fleischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paul R. Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Doina Precup
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong

    Rudolf Fleischer

  2. Department of Computer Science, University of New Mexico, Farris Engineering Bldg, 87131-1386, Albuquerque, NM, USA

    Bernard Moret

  3. Department of Computer Science, University of Aarhus, Bld. 540, Ny Munkegade, 8000, Aarhus C, Denmark

    Erik Meineche Schmidt

Rights and permissions

Reprints and permissions

Copyright information

© 2002 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

McGeoch, C., Sanders, P., Fleischer, R., Cohen, P.R., Precup, D. (2002). Using Finite Experiments to Study Asymptotic Performance. In: Fleischer, R., Moret, B., Schmidt, E.M. (eds) Experimental Algorithmics. Lecture Notes in Computer Science, vol 2547. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36383-1_5

Download citation

  • DOI: https://doi.org/10.1007/3-540-36383-1_5

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-00346-5

  • Online ISBN: 978-3-540-36383-5

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation