Abstract
Developing real-world Machine Learning-based Systems goes beyond algorithm development. ML algorithms are usually embedded in complex pre-processing steps and consider different stages like development, testing or deployment. Managing workflows poses several challenges, such as workflow versioning, sharing pipeline elements or optimizing individual workflow elements - tasks which are usually conducted manually by data scientists. A dataset containing 16 035 real-world Machine Learning and Data Science Workflows extracted from the ONE DATA platform (https://onelogic.de/en/one-data/) is explored and made available. Based on our analysis, we develop a representation learning algorithm using a graph-level Graph Convolutional Network with explicit residuals which exploits workflow versioning history. Moreover, this method can easily be adapted to supervised tasks and outperforms state-of-the-art approaches in NAS-bench-101 performance prediction. Another interesting application is the suggestion of component types, for which a classification baseline is presented. A slightly adapted GCN using both graph- and node-level information further improves upon this baseline. The used codebase as well as all experimental setups with results are available at https://github.com/wendli01/workflow_analysis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at https://zenodo.org/record/4633704.
References
Amershi, S., et al.: Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. IEEE (2019)
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
Basu, A., Blanning, R.W.: A formal approach to workflow analysis. Inf. Syst. Res. 11(1), 17–36 (2000)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Bresson, X., Laurent, T.: Residual gated graph convNets. arXiv preprint arXiv:1711.07553 (2017)
Du, J., Zhang, S., Wu, G., Moura, J.M., Kar, S.: Topology adaptive graph convolutional networks. arXiv preprint arXiv:1710.10370 (2017)
Friesen, N., Rüping, S.: Workflow analysis using graph kernels. In: LWA, pp. 59–66. Citeseer (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
Ward, J.H., Jr.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963). https://doi.org/10.1080/01621459.1963.10500845
Kaushik, G., Ivkovic, S., Simonovic, J., Tijanic, N., Davis-Dusenbery, B., Deniz, K.: Graph theory approaches for optimizing biomedical data analysis using reproducible workflows. bioRxiv, p. 074708 (2016)
Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Li, J., Fan, Y., Zhou, M.: Timing constraint workflow nets for workflow analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 33(2), 179–193 (2003)
Liben-Nowell, D., Kleinberg, J.: The link-prediction problem for social networks. J. Am. Soc. Inform. Sci. Technol. 58(7), 1019–1031 (2007)
Lukasik, J., Friede, D., Zela, A., Stuckenschmidt, H., Hutter, F., Keuper, M.: Smooth variational graph embeddings for efficient neural architecture search. arXiv preprint arXiv:2010.04683 (2020)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Adv. Neural. Inf. Process. Syst. 26, 3111–3119 (2013)
Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., Jaiswal, S.: graph2vec: learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 (2017)
Ning, X., Zheng, Y., Zhao, T., Wang, Y., Yang, H.: A generic graph-based neural architecture encoding scheme for predictor-based NAS (2020)
Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710 (2014)
Rosenberg, A., Hirschberg, J.: V-measure: a conditional entropy-based external cluster evaluation measure. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 410–420 (2007)
Rozemberczki, B., Sarkar, R.: Characteristic functions on graphs: birds of a feather, from statistical descriptors to parametric models (2020)
Shi, H., Pi, R., Xu, H., Li, Z., Kwok, J.T., Zhang, T.: Multi-objective neural architecture search via predictive network performance optimization (2019)
Stier, J., Granitzer, M.: Structural analysis of sparse neural networks. Procedia Comput. Sci. 159, 107–116 (2019)
Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 497–504 (2017)
Tang, Y., et al.: A semi-supervised assessor of neural architectures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1810–1819 (2020)
Balntas, V., Riba, E., Ponsa, D., Mikolajczyk, K.: Learning local feature descriptors with triplets and shallow convolutional neural networks. In: Wilson, R.C., Hancock, E.R., Smith, W.A.P. (eds.) Proceedings of the British Machine Vision Conference (BMVC), pp. 119.1–119.11. BMVA Press (September 2016). https://doi.org/10.5244/C.30.119
Weißgerber, T., Granitzer, M.: Mapping platforms into a new open science model for machine learning. Inf. Technol. 61(4), 197–208 (2019)
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2020)
Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-Bench-101: towards reproducible neural architecture search. In: International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)
Zhou, T., Lü, L., Zhang, Y.C.: Predicting missing links via local information. Eur. Phys. J. B 71(4), 623–630 (2009)
Acknowledgments
This work has been partially funded by the Bavarian Ministry of Economic Affairs, Regional Develoment and Energy under the grant ‘CrossAI’ (IUK593/002) as well as by BMK, BMDW, and the Province of Upper Austria in the frame of the COMET Programme managed by FFG. It was also supported by the FFG BRIDGE project KnoP-2D (grant no. 871299).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wendlinger, L., Berndl, E., Granitzer, M. (2021). Methods for Automatic Machine-Learning Workflow Analysis. In: Dong, Y., Kourtellis, N., Hammer, B., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12979. Springer, Cham. https://doi.org/10.1007/978-3-030-86517-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-86517-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86516-0
Online ISBN: 978-3-030-86517-7
eBook Packages: Computer ScienceComputer Science (R0)
