Skip to main content
SpringerLink
Log in

Five pitfalls of empirical scheduling research

  • Session 6
  • Conference paper
  • First Online:
Principles and Practice of Constraint Programming-CP97 (CP 1997)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1330))

Abstract

A number of pitfalls of empirical scheduling research are illustrated using real experimental data. These pitfalls, in general, serve to slow the progress of scheduling research by obsfucating results, blurring comparisons among scheduling algorithms and algorithm components, and complicating validation of work in the literature. In particular, we look at difficulties brought about by viewing algorithms in a monolithic fashion, by concentrating on CPU time as the only evaluation criteria, by failing to prepare for gathering of a variety of search statistics at the time of experimental design, by concentrating on benchmarks to the exclusion of other sources of experimental problems, and, more broadly, by a preoccupation with optimization of makespan as the sole goal of scheduling algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • AAAI Empirical Workshop (1994). Proceedings of the AAAI-94 Workshop on Experimental Evaluation of Reasoning and Search Methods.

    Google Scholar 

  • Allen, J. F. (1983). Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843.

    Google Scholar 

  • Beasley, J. E. (1990). OR-library: distributing test problems by electronic mail. Journal of the Operational Research Society, 41(11):1069–1072. Also available by ftp from ftp:// graph.ms.ic.ac.uk/pub/paper.txt.

    Google Scholar 

  • Beck, J. C. (1994). A schema for constraint relaxation with instantiations for partial constraint satisfaction and schedule optimization. Master's thesis, Department of Computer Science, University of Toronto.

    Google Scholar 

  • Beck, J. C. (1997). A generic framework for constraint-directed search and scheduling. Technical report, Department of Industrial Engineering, University of Toronto, 4 Taddle Creek Road, Toronto, Ontario M5S 3G9, Canada.

    Google Scholar 

  • Beck, J. C., Davenport, A. J., Sitarski, E. M., and Fox, M. S. (1997a). Beyond contention: extending texture-based scheduling heuristics. In Proceedings of AAAI-97. AAAI Press, Menlo Park, California.

    Google Scholar 

  • Beck, J. C., Davenport, A. J., Sitarski, E. M., and Fox, M. S. (1997b). Texture-based heuristics for scheduling revisited. In Proceedings of AAAI-97. AAAI Press, Menlo Park, California.

    Google Scholar 

  • Beck, J. C. and Jackson, K. (1997). Constrainedness and the phase transition in job shop scheduling. Technical report, School of Computing Science, Simon Fraser University.

    Google Scholar 

  • Brucker, P. and Thiele, O. (1996). A branch & bound method for the general-shop problems with sequence dependent set-up times. OR Spektrum, 18:145–161.

    Google Scholar 

  • Carlier, J. and Pinson, E. (1989). An algorithm for solving the job-shop problem. Management Science, 35(2):164–176.

    Google Scholar 

  • Caseau, Y. and Laburthe, F. (1996). Cumulative scheduling with task intervals. In Proceedings of the Joint International Conference and Symposium on Logic Programming. MIT Press.

    Google Scholar 

  • Cheeseman, P., Kanefsky, B., and Taylor, W. (1991). Where the really hard problems are. In yProceedings of IJCAI-91, volume 1, pages 331–337.

    Google Scholar 

  • Cheng, C. C. and Smith, S. F. (1996). Applying constraint satisfaction techniques to job shop scheduling. Annals of Operations Research, Special Volume on Scheduling: Theory and Practice, 1. Forthcoming.

    Google Scholar 

  • Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, Mass.

    Google Scholar 

  • Davenport, A. J., Beck, J. C., and Fox, M. S. (1997). Propagation over the meets temporal constraint. Technical report, Department of Industrial Engineering, University of Toronto.

    Google Scholar 

  • Drummond, M., Bresina, J., and Swanson, K. (1994). Just-in-case scheduling. In Proceedings of AAAI-94, pages 1098–1104, Menlo Park, CA. AAAI Press/MIT Press.

    Google Scholar 

  • ECAI Empirical Workshop (1996). Proceedings of the ECAI-96 Workshop on Empirical Artificial Intelligence.

    Google Scholar 

  • Erschler, J., Roubellat, F., and Vernhes, J. P. (1976). Finding some essential characteristics of the feasible solutions for a scheduling problem. Operations Research, 24:772–782.

    Google Scholar 

  • Erschler, J., Roubellat, F., and Vernhes, J. P. (1980). Characterising the set of feasible sequences for n jobs to be carried out on a single machine. European Journal of Operational Research, 4:189–194.

    Google Scholar 

  • Fox, M. S. (1983). Constraint-Directed Search: A Case Study of Job-Shop Scheduling. PhD thesis, Carnegie Mellon University, Intelligent Systems Laboratory, The Robotics Institute, Pittsburgh, PA. CMU-RI-TR-85-7.

    Google Scholar 

  • Fox, M. S. (1990). Constraint-guided scheduling-a short history of research at CMU. Computers in Industry, 14:79–88.

    Google Scholar 

  • Gent, I. P., Grant, S. A., MacIntyre, E., Prosser, P., Shaw, P., Smith, B. M., and Walsh, T. (1997). How not to do it. Technical Report 97.27, School of Computer Studies, University of Leeds.

    Google Scholar 

  • Gent, I. P., MacIntyre, E., Prosser, P., and Walsh, T. (1996). The constrainedness of search. In Proceedings ofAAAI-96, volume 1, pages 246–252.

    Google Scholar 

  • Harvey, W. D. (1995). Nonsystematic backtracking search. PhD thesis, Department of Computer Science, Stanford University.

    Google Scholar 

  • Harvey, W. D. and Ginsberg, M. L. (1995). Limited discrepancy search. In Proceedings of IJCAI-95, pages 607–613.

    Google Scholar 

  • Hildum, D. W. (1994). Flexibility in a knowledge-based system for solving dynamic resourceconstrained scheduling problems. PhD thesis, Department of Computer Science, University of Massachusetts, Amherst, MA. 01003-4610. UMass CMPSCI TR 94-77.

    Google Scholar 

  • Hooker, J. N. (1994). Needed: An empirical science of algorithms. Operations Research, 42:201–212.

    Google Scholar 

  • Hooker, J. N. (1996). Testing heuristics: We have it all wrong. Journal ofHeuristics, 1:33–42.

    Google Scholar 

  • Le Pape, C. (1994a). Implementation of resource constraints in ILOG Schedule: A library for the development of constraint-based scheduling systems. Intelligent Systems Engineering, 3(2):55–66.

    Google Scholar 

  • Le Pape, C. (1994b). Using a constraint-based scheduling library to solve a specific scheduling problem. In Proceedings of the AAAI-SIGMAN Workshop on Artificial Intelligence Approaches to Modelling and Scheduling Manufacturing Processes.

    Google Scholar 

  • Lhomme, O. (1993). Consistency techniques for numeric CSPs. In Proceedings of IJCAI-93, volume 1, pages 232–238.

    Google Scholar 

  • Muscettola, N. (1992). Scheduling by iterative partition of bottleneck conflicts. Technical Report CMU-RI-TR-92-05, The Robotics Institute, Carnegie Mellon University.

    Google Scholar 

  • Nuijten, W. and Aarts, E. (1997). A computational study of constraint satisfaction for multiple capacitated job shop scheduling. European Journal of Operational Research. To appear.

    Google Scholar 

  • Nuijten, W. P. M. (1994). Time and resource constrained scheduling: a constraint satisfaction approach. PhD thesis, Department of Mathematics and Computing Science, Eindhoven University of Technology.

    Google Scholar 

  • Ow, P. S., Smith, S. F., and Thiriez, A. (1988). Reactive plan revision. In Proceedings of AAAI-88, pages 77–82. AAAI.

    Google Scholar 

  • Sadeh, N. (1991). Lookahead techniques for micro-opportunistic job-shop scheduling. PhD thesis, Carnegie-Mellon University. CMU-CS-91-102.

    Google Scholar 

  • Saks, V. (1992). Distribution planner overview. Technical report, Carnegie Group, Inc., Pittsburgh, PA, 1522.

    Google Scholar 

  • Smith, S. F. and Cheng, C. C. (1993). Slack-based heuristics for constraint satisfaction scheduling. In Proceedings AAAI-93, pages 139–144.

    Google Scholar 

  • Taillard, E. (1993). Benchmarks for basic scheduling problems. European Journal of Operational Research, 64:278–285.

    Google Scholar 

  • Xiong, Y., Sadeh, N., and Sycara, K. (1992). Intelligent backtracking techniques for job-shop scheduling. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, Cambridge, MA.

    Google Scholar 

  • Zweben, M., Davis, E., Daun, B., and Deale, M. (1993). Informedness vs. computational cost of heuristics in iterative repair scheduling. In Proceedings of IJCAI-93, pages 1416–1422.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Toronto, M5S 3G9, Toronto, Ontario, Canada

    J. Christopher Beck & Mark S. Fox

  2. Department of Industrial Engineering, University of Toronto, M5S 3G9, Toronto, Ontario, Canada

    Andrew J. Davenport & Mark S. Fox

Authors
  1. J. Christopher Beck
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew J. Davenport
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark S. Fox
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Gert Smolka

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Beck, J.C., Davenport, A.J., Fox, M.S. (1997). Five pitfalls of empirical scheduling research. In: Smolka, G. (eds) Principles and Practice of Constraint Programming-CP97. CP 1997. Lecture Notes in Computer Science, vol 1330. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0017455

Download citation

  • DOI: https://doi.org/10.1007/BFb0017455

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-63753-0

  • Online ISBN: 978-3-540-69642-1

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation