John Woodward
Simon Martin
ABSTRACT
There have been several papers published relating to the practice of benchmarking in machine learning and Genetic Programming (GP) in particular. In addition, GP has been accused of targeting over-simplified 'toy' problems that do not reflect the complexity of real-world applications that GP is ultimately intended. There are also theoretical results that relate the performance of an algorithm with a probability distribution over problem instances, and so the current debate concerning benchmarks spans from the theoretical to the empirical.
The aim of this article is to consolidate an emerging theme arising from these papers and suggest that benchmarks should not be arbitrarily selected but should instead be drawn from an underlying probability distribution that reflects the problem instances which the algorithm is likely to be applied to in the real-world. These probability distributions are effectively dictated by the application domains themselves (essentially data-driven) and should thus re-engage the owners of the originating data.
A consequence of properly-founded benchmarking leads to the suggestion of meta-learning as a methodology for automatically designing algorithms rather than manually designing algorithms. A secondary motive is to reduce the number of research papers that propose new algorithms but do not state in advance what their purpose is (i.e. in what context should they be applied). To put the current practice of GP benchmarking in a particular harsh light, one might ask what the performance of an algorithm on Koza's lawnmower problem (a favourite toy-problem of the GP community) has to say about its performance on a very real-world cancer data set: the two are completely unrelated.
- Juergen Branke and Jawad Asem Elomari. Meta-optimization for parameter tuning with a flexible computing budget. In Proceedings of the fourteenth international conference on Genetic and evolutionary computation conference, GECCO '12, pages 1245--1252, New York, NY, USA, 2012. ACM. Google Scholar
Digital Library
- Gavin Brown. A new perspective for information theoretic feature selection. In International Conference on Artificial Intelligence and Statistics, pages 49--56, 2009.Google Scholar
- Edmund Burke, Graham Kendall, Jim Newall, Emma Hart, Peter Ross, and Sonia Schulenburg. Hyper-heuristics: An emerging direction in modern search technology. In Fred Glover, Gary Kochenberger, and Frederick S. Hillier, editors, Handbook of Metaheuristics, volume 57 of International Series in Operations Research and Management Science, pages 457--474. Springer New York, 2003.Google ScholarCross Ref
- Edmund K. Burke, Mathew R. Hyde, Graham Kendall, Gabriela Ochoa, Ender Ozcan, and John R. Woodward. Exploring hyper-heuristic methodologies with genetic programming. In Computational intelligence, pages 177--201. Springer Berlin Heidelberg, 2009.Google ScholarCross Ref
- Edmund K. Burke, Matthew R. Hyde, Graham Kendall, and John Woodward. Automatic heuristic generation with genetic programming: evolving a jack-of-all-trades or a master of one. In GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation, volume 2, pages 1559--1565, London, 7-11 July 2007. ACM Press. Google ScholarDigital Library
- Zehra Cataltepe, Yaser S. Abu-Mostafa, and Malik Magdon-Ismail. No free lunch for early stopping. Neural Comput., 11:995--1009, May 1999. Google ScholarDigital Library
- Stefan Droste, Thomas Jansen, and Ingo Wegener. Perhaps Not a Free Lunch But At Least a Free Appetizer. In Wolfgang Banzhaf, Jason Daida, Agoston E. Eiben, Max H. Garzon, Vasant Honavar, Mark Jakiela, and Robert E. Smith, editors, Proceedings of the Genetic and Evolutionary Computation Conference GECCO-1999, pages 833--839, San Francisco, CA, 1999. Morgan Kaufmann Publishers, Inc.Google Scholar
- Stefan Droste, Thomas Jansen, and Ingo Wegener. Optimization with randomized search heuristics - the (a)nfl theorem, realistic scenarios, and difficult functions. Theor. Comput. Sci., 287(1):131--144, 2002. Google ScholarDigital Library
- Edgar A. Duéñez Guzmán and Michael D. Vose. No Free Lunch and Benchmarks. Evolutionary Computation, pages 1--20, March 2012.Google Scholar
- Peter Flach. Machine Learning: The art and science of algorithms that make sense of data. Cambridge University Press, September 2012. Google ScholarCross Ref
- C. Giraud-Carrier and F. Provost. Toward a Justification of Meta-learning: Is the No Free Lunch Theorem a Show-stopper? In Proceedings of the ICML-2005 Workshop on Meta-learning, pages 12--19, 2005.Google Scholar
- Libin Hong, John Woodward, Jingpeng Li, and Ender Ozcan. Automated design of probability distributions as mutation operators for evolutionary programming using genetic programming. In Krzysztof Krawiec, Alberto Moraglio, Ting Hu, A. Sima Uyar, and Bin Hu, editors, Proceedings of the 16th European Conference on Genetic Programming, EuroGP 2013, volume 7831 of LNCS, pages 85--96, Vienna, Austria, 3-5 April 2013. Springer Verlag. Google ScholarDigital Library
- Marcus Hutter. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. EATCS. Springer, Berlin, 2005. 300 pages, http://www.hutter1.net/ai/uaibook.htm. Google ScholarCross Ref
- Marcus Hutter. A complete theory of everything (will be subjective). Algorithms, 3(4):329--350, 2010.Google ScholarCross Ref
- Christian Igel and Marc Toussaint. On classes of functions for which no free lunch results hold. Inf. Process. Lett., 86(6):317--321, 2003. Google ScholarDigital Library
- Christian Igel and Marc Toussaint. Recent results on no-free-lunch theorems for optimization. CoRR, cs.NE/0303032, 2003.Google Scholar
- Christian Igel and Marc Toussaint. A no-free-lunch theorem for nonuniform distributions of target functions. Journal of Mathematical Modeling and Algorithms, page 313, 2004.Google Scholar
- D. S. Johnson. A Theoretician's Guide to the Experimental Analysis of Algorithms. In 5th and 6th DIMACS Implementation Challenges. American Mathematical Society, 2002.Google Scholar
- Sean Luke. Essentials of Metaheuristics. Lulu, second edition, 2013. Available for free at http://cs.gmu.edu/~sean/book/metaheuristics/.Google Scholar
- James McDermott, David R. White, Sean Luke, Luca Manzoni, Mauro Castelli, Leonardo Vanneschi, Wojciech Jaskowski, Krzysztof Krawiec, Robin Harper, Kenneth De Jong, and Una-May O'Reilly. Genetic programming needs better benchmarks. In GECCO '12: Proceedings of the fourteenth international conference on Genetic and evolutionary computation conference, pages 791--798, Philadelphia, Pennsylvania, USA, 7-11 July 2012. ACM. Google ScholarDigital Library
- Tom M. Mitchell. The need for biases in learning generalizations. Technical report, Rutgers University, New Brunswick, NJ, 1980.Google Scholar
- Gisele L. Pappa and Alex A. Freitas. Automatically Evolving Data Mining Algorithms, volume XIII of Natural Computing Series. Springer, 2010.Google ScholarCross Ref
- Gisele L. Pappa, Gabriela Ochoa, Matthew R. Hyde, Alex A. Freitas, John Woodward, and Jerry Swan. Contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms. Genetic Programming and Evolvable Machines, pages 1--33. Google ScholarDigital Library
- Riccardo Poli, Leonardo Vanneschi, William B. Langdon, and Nicholas Freitag Mcphee. Theoretical results in genetic programming: The next ten years? Genetic Programming and Evolvable Machines, 11(3-4):285--320, September 2010. Google ScholarDigital Library
- J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993. Google ScholarDigital Library
- Jonathan E. Rowe and Michael D. Vose. Unbiased black box search algorithms. In GECCO, pages 2035--2042, 2011. Google ScholarDigital Library
- C. Schaffer. A conservation law for generalization performance. In Proceedings of the Eleventh International Conference on Machine Learning, pages 259--265. Morgan Kaufmann, 1994.Google ScholarDigital Library
- C. Schumacher, M. D. Vose, and L. D. Whitley. The no free lunch and problem description length. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pages 565--570. Morgan Kaufmann, 2001.Google Scholar
- Kenneth Sörensen. Metaheuristics - the metaphor exposed. International Transactions in Operational Research, 2013.Google Scholar
- Matthew J. Streeter. Two broad classes of functions for which a no free lunch result does not hold. In Proc. Genetic and Evolutionary Computation Conference GECCO-2003, pages 1418--1430. Morgan Kaufmann, 2003. Google ScholarDigital Library
- El-Ghazali Talbi. Metaheuristics - From Design to Implementation. Wiley, 2009. Google ScholarDigital Library
- Sebastian Thrun and Lorien Pratt, editors. Learning to learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. Google ScholarCross Ref
- Kiri Wagstaff. Machine learning that matters. CoRR, abs/1206.4656, 2012.Google Scholar
- David R White, James McDermott, Mauro Castelli, Luca Manzoni, Brian W Goldman, Gabriel Kronberger, Wojciech Jaśkowski, Una-May O' Reilly, and Sean Luke. Better gp benchmarks: community survey results and proposals. Genetic Programming and Evolvable Machines, 14(1):3--29, 2013. Google ScholarDigital Library
- D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Evolutionary Computation, IEEE Transactions on, 1(1):67--82, April 1997. Google ScholarDigital Library
- John R. Woodward. The necessity of meta bias in search algorithms. In Computational Intelligence and Software Engineering (CiSE), 2010 International Conference on, pages 1--4. IEEE, 2010.Google ScholarCross Ref
- John R. Woodward and Jerry Swan. The automatic generation of mutation operators for genetic algorithms. In Gisele L. Pappa, John Woodward, Matthew R. Hyde, and Jerry Swan, editors, GECCO 2012 2nd Workshop on Evolutionary Computation for the Automated Design of Algorithms, pages 67--74, Philadelphia, Pennsylvania, USA, 7-11 July 2012. ACM. Google ScholarDigital Library
- John Robert Woodward and Jerry Swan. Automatically selection heuristics. In Proceedings of the 13th annual conference companion on Genetic and evolutionary computation, GECCO '11, pages 583--590, New York, NY, USA, 2011. ACM.Google ScholarDigital Library
- Huaiyu Zhu and Richard Rohwer. No free lunch for cross-validation. Neural Comput., 8:1421--1426, October 1996. Google ScholarDigital Library
- Benchmarks that matter for genetic programming
Recommendations
Function optimization using cartesian genetic programming
GECCO '13 Companion: Proceedings of the 15th annual conference companion on Genetic and evolutionary computationIn function optimization one tries to find a vector of real numbers that optimizes a complex multi-modal fitness function. Although evolutionary algorithms have been used extensively to solve such problems, genetic programming has not. In this paper, we ...
Neural network crossover in genetic algorithms using genetic programming
AbstractThe use of genetic algorithms (GAs) to evolve neural network (NN) weights has risen in popularity in recent years, particularly when used together with gradient descent as a mutation operator. However, crossover operators are often omitted from ...
A Comparison of three evolutionary strategies for multiobjective genetic programming
We report what we believe to be the first comparative study of multi-objective genetic programming (GP) algorithms on benchmark symbolic regression and machine learning problems. We compare the Strength Pareto Evolutionary Algorithm (SPEA2), the Non-...
Comments