Skip to main content
Springer Nature Link
Log in

Meta-mining Evaluation Framework: A Large Scale Proof of Concept on Meta-learning

  • Conference paper
  • First Online:
AI 2016: Advances in Artificial Intelligence (AI 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9992))

Included in the following conference series:

  • 3287 Accesses

Abstract

This paper aims to provide a unified framework for the evaluation and comparison of the many emergent meta-mining techniques. This framework is illustrated on the case study of the meta-learning problem in a large scale experiment. The results of this experiment are then explored through hypothesis testing in order to provide insight regarding the performance of the different meta-learning schemes, advertising the potential of our approach regarding meta-level knowledge discovery.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Similar content being viewed by others

References

  1. Brazdil, P., Gama, J., Henery, B.: Characterizing the applicability of classification algorithms using meta-level learning. In: Bergadano, F., Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 83–102. Springer, Heidelberg (1994). doi:10.1007/3-540-57868-4_52

    Chapter  Google Scholar 

  2. Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)

    MathSciNet  MATH  Google Scholar 

  3. Dong, L., Frank, E., Kramer, S.: Ensembles of balanced nested dichotomies for multi-class problems. In: Jorge, A.M., Torgo, L., Brazdil, P., Camacho, R., Gama, J. (eds.) PKDD 2005. LNCS (LNAI), vol. 3721, pp. 84–95. Springer, Heidelberg (2005). doi:10.1007/11564126_13

    Chapter  Google Scholar 

  4. Giraud-Carrier, C., Vilalta, R., Brazdil, P.: Introduction to the special issue on meta-learning. Mach. Learn. 54(3), 187–193 (2004)

    Article  Google Scholar 

  5. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)

    Article  Google Scholar 

  6. Hilario, M., Nguyen, P., Do, H., Woznica, A., Kalousis, A.: Ontology-based meta-mining of knowledge discovery workflows. In: Jankowski, N., Duch, W., Gra̧bczewski, K. (eds.) Meta-Learning in Computational Intelligence. Studies in Computational Intelligence, vol. 358, pp. 273–315. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  7. Kalousis, A., Hilario, M.: Model selection via meta-learning: a comparative study. Int. J. Artif. Intell. Tools 10(04), 525–554 (2001)

    Article  MATH  Google Scholar 

  8. Kalousis, A., Hilario, M.: Feature selection for meta-learning. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, pp. 222–233. Springer, Heidelberg (2001). doi:10.1007/3-540-45357-1_26

    Chapter  MATH  Google Scholar 

  9. King, R.D., Feng, C., Sutherland, A.: StatLog: comparison of classification algorithms on large real-world problems. Int. J. Appl. Artif. Intell. 9(3), 289–333 (1995)

    Article  Google Scholar 

  10. Kononenko, I., Bratko, I.: Information-based evaluation criterion for classifier’s performance. Mach. Learn. 6(1), 67–80 (1991)

    Article  Google Scholar 

  11. Leite, R., Brazdil, P., Vanschoren, J.: Selecting classification algorithms with active testing. In: Perner, P. (ed.) MLDM 2012. LNCS (LNAI), vol. 7376, pp. 117–131. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31537-4_10

    Chapter  Google Scholar 

  12. Leyva, E., Gonzalez, A., Perez, R.: A set of complexity measures designed for applying meta-learning to instance selection. IEEE Trans. Knowl. Data Eng. 27(2), 354–367 (2015)

    Article  Google Scholar 

  13. Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine Learning, Neural and Statistical Classification. Ellis Horwood, Upper Saddle River (1994)

    MATH  Google Scholar 

  14. Misir, M., Sebag, M.: Algorithm selection as a collaborative filtering problem, p. 43 (2013). hal-00922840

  15. Nguyen, P., Hilario, M., Kalousis, A.: Using meta-mining to support data mining workflow planning and optimization. J. Artif. Intell. Res. 51, 605–644 (2014)

    Article  Google Scholar 

  16. Peng, Y., Flach, P.A., Brazdil, P., Soares, C.: Decision tree-based data characterization for meta-learning. In: IDDM-2002 p. 111 (2002)

    Google Scholar 

  17. Pfahringer, B., Bensusan, H., Giraud-Carrier, C.: Tell me who can learn you and i can tell you who you are: landmarking various learning algorithms. In: Proceedings of the 17th International Conference on Machine Learning, pp. 743–750 (2000)

    Google Scholar 

  18. Platt, J., et al.: Sequential minimal optimization: A fast algorithm for training support vector machines (1998)

    Google Scholar 

  19. Robnik-Šikonja, M., Kononenko, I.: Theoretical and empirical analysis of relieff and rrelieff. Mach. Learn. 53(1–2), 23–69 (2003)

    Article  MATH  Google Scholar 

  20. Serban, F., Vanschoren, J., Kietz, J.U., Bernstein, A.: A survey of intelligent assistants for data analysis. ACM Comput. Surv. (CSUR) 45(3), 31 (2013)

    Article  Google Scholar 

  21. Sun, Q., Pfahringer, B.: Pairwise meta-rules for better meta-learning-based algorithm ranking. Mach. Learn. 93(1), 141–161 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  22. Sun, Q., Pfahringer, B., Mayo, M.: Full model selection in the space of data mining operators. In: Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation, pp. 1503–1504. ACM (2012)

    Google Scholar 

  23. Todorovski, L., Brazdil, P., Soares, C.: Report on the experiments with feature selection in meta-level learning. In: Proceedings of the PKDD 2000 Workshop on Data Mining, Decision Support, Meta-learning and ILP: Forum For Practical Problem Presentation and Prospective Solutions, pp. 27–39. Citeseer (2000)

    Google Scholar 

  24. Uno, T., Asai, T., Uchida, Y., Arimura, H.: An efficient algorithm for enumerating closed patterns in transaction databases. In: Suzuki, E., Arikawa, S. (eds.) DS 2004. LNCS (LNAI), vol. 3245, pp. 16–31. Springer, Heidelberg (2004). doi:10.1007/978-3-540-30214-8_2

    Chapter  Google Scholar 

  25. Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. SIGKDD Explor. 15(2), 49–60 (2013). http://doi.acm.org/10.1145/2641190.2641198

    Article  Google Scholar 

  26. Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002). http://dx.doi.org/10.1023/A:1019956318069

    Article  Google Scholar 

  27. Xu, L., Hutter, F., Shen, J., Hoos, H.H., Leyton-Brown, K.: SATzilla2012: improved algorithm selection based on cost-sensitive classification models, pp. 57–58 (2012)

    Google Scholar 

  28. Zakova, M., Kremen, P., Zelezny, F., Lavrac, N.: Automating knowledge discovery workflow composition through ontology-based planning. IEEE Trans. Autom. Sci. Eng. 8(2), 253–264 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IRIT UMR 5505, UT1, UT3, Universite de Toulouse, Toulouse, France

    William Raynaut, Chantal Soule-Dupuy & Nathalie Valles-Parlangeau

Authors
  1. William Raynaut
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chantal Soule-Dupuy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nathalie Valles-Parlangeau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William Raynaut .

Editor information

Editors and Affiliations

  1. University of Tasmania, Hobart, Australia

    Byeong Ho Kang

  2. Auckland University of Technology, Auckland, New Zealand

    Quan Bai

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Raynaut, W., Soule-Dupuy, C., Valles-Parlangeau, N. (2016). Meta-mining Evaluation Framework: A Large Scale Proof of Concept on Meta-learning. In: Kang, B.H., Bai, Q. (eds) AI 2016: Advances in Artificial Intelligence. AI 2016. Lecture Notes in Computer Science(), vol 9992. Springer, Cham. https://doi.org/10.1007/978-3-319-50127-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-50127-7_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-50126-0

  • Online ISBN: 978-3-319-50127-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics