Abstract
The accuracy of many regression models suffers from inhomogeneous data coverage. Models loose accuracy because they are unable to locally adapt the model complexity. This article develops and evaluates an automated design process for the generation of hybrid regression models from arbitrary submodels. For the first time, these submodels are weighted by a One-Class Support Vector Machine, taking local data coverage into account. Compared to reference regression models, the newly developed hybrid models achieve significant better results in nine out of ten benchmark datasets. To enable straightforward usage in data science, an implementation is integrated in the open source MATLAB toolbox SciXMiner.
Zusammenfassung
Die Genauigkeit vieler Regressionsmodelle leidet unter inhomogener Datenabdeckung. Die meisten Regressionsmodelle sind nicht in der Lage, die Modellkomplexität lokal anzupassen. Dieser Beitrag entwickelt und bewertet ein automatisiertes Entwurfsverfahren zur Erzeugung hybrider Regressionsmodelle aus beliebigen Teilmodellen. Erstmalig wird eine Ein-Klassen-Support-Vektor-Maschine eingesetzt, die die lokale Datenabdeckung berücksichtigt. Im Vergleich mit Referenzmodellen erzielen die neu entwickelten hybriden Modelle auf neun von zehn Benchmark-Datensätzen signifikant bessere Genauigkeiten. Um eine unkomplizierte Nutzbarkeit in der Datenverarbeitung zu ermöglichen, wurde das hybride Entwurfsverfahren in die Open-Source-MATLAB-Toolbox SciXMiner integriert.
About the authors
Moritz Böhland works at the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology. Research areas: Machine learning, data mining, image processing.
Dr.-Ing. Wolfgang Doneit works at Dr. Wieselhuber & Partner GmbHResearch areas: Data quality evaluation, data modelling, parameter identification.
PD Dr.-Ing. Lutz Gröll is project manager at the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology. Research areas: Modelling of process plants, parameter identification, control theory.
apl. Prof. Dr.-Ing. Ralf Mikut is Head of the Research Area Automated Image and Data Analysis at the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology and Speaker of the Helmholtz Information and Data Science School for Health (HIDSS4Health). Research areas: Computational intelligence, data analytics, modelling and image processing with applications in biology, chemistry, medical engineering, energy systems and robotics.
PD Dr.-Ing. Markus Reischl is project manager at the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology. Research areas: Human-machine interfaces, image processing, machine learning, data mining.
Acknowledgment
With thanks to Katherine Quinlan-Flatter for proofreading this article.
References
1. Avnimelech, R.; Intrator, N.: Boosting Regression Estimators. Neural Computation 11 (1999) 2, pp. 499–520.10.1162/089976699300016746Search in Google Scholar PubMed
2. Babuška, R.: Fuzzy Modeling for Control. Boston: Kluwer, 1998.10.1007/978-94-011-4868-9Search in Google Scholar
3. Babyak, M. A.: What You See May Not Be What You Get: A Brief, Nontechnical Introduction to Overfitting in Regression-Type Models. Psychosomatic Medicine 66 (2004) p. 3.10.1097/00006842-200405000-00021Search in Google Scholar
4. Bänfer, O.; Hartmann, B.; Nelles, O.: POLYMOT versus HILOMOT – A Comparison of Two Different Training Algorithms for Local Model Networks. IFAC Proceedings Volumes 45 (2012) 16, pp. 1569–1574.10.3182/20120711-3-BE-2027.00354Search in Google Scholar
5. Bishop, C. M.: Novelty Detection and Neural Network Validation. IEE Proceedings – Vision, Image and Signal Processing 141 (1994) 4, pp. 217–222.10.1049/ip-vis:19941330Search in Google Scholar
6. Bishop, C. M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Secaucus, NJ, USA: Springer, 2006.Search in Google Scholar
7. Breiman, L.: Bagging Predictors. Machine Learning 24 (1996) 2, pp. 123–140.10.1007/BF00058655Search in Google Scholar
8. Bühlmann, P.; Hothorn, T.: Boosting Algorithms: Regularization, Prediction and Model Fitting. Statistical Science 22 (2007) 4, pp. 477–505.10.1214/07-STS242Search in Google Scholar
9. Chang, C.-C.; Lin, C.-J.: LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology 2 (2011), pp. 27:1–27:27, software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.Search in Google Scholar
10. Dannecker, L.: Energy Time Series Forecasting: Efficient and Accurate Forecasting of Evolving Time Series from the Energy Domain. Springer Vieweg, 1st ed. 2015 Aufl., 2015.10.1007/978-3-658-11039-0Search in Google Scholar
11. Dheeru, D.; Karra Taniskidou, E.: {UCI} Machine Learning Repository. 2017.Search in Google Scholar
12. Doneit, W.: Eine neue Methode zum robusten Entwurf von Regressionsmodellen bei beschränkter Rohdatenqualität. Dissertation, Karlsruher Institut für Technologie (KIT), 2018.Search in Google Scholar
13. Doneit, W.; Mikut, R.; Gröll, L.; Pychynski, T.; Reischl, M.: DaMoQ: Eine Open-Source-MATLAB-Toolbox zur Bewertung von Daten- und Modellqualität in Regressionen. at – Automatisierungstechnik 65 (2017) 3, pp. 207–218.10.1515/auto-2016-0128Search in Google Scholar
14. Ebert, T.; Belz, J.; Nelles, O.: Detektion von Extrapolation. In: Proceedings of the 24. Workshop on Computational Intelligence, Dortmund, Germany, Bd. 50, pp. 17–32, 2014.Search in Google Scholar
15. Falkner, S.; Klein, A.; Hutter, F.: BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In: Proceedings of the 35th International Conference on Machine Learning (Dy, J.; Krause, A., Hg.), Bd. 80 von Proceedings of Machine Learning Research, pp. 1437–1446, Stockholmsmässan, Stockholm Sweden: PMLR, 2018.Search in Google Scholar
16. Forrester, A.; Sobester, A.; Keane, A.: Engineering Design via Surrogate Modelling: A Practical Guide. John Wiley & Sons, 2008.10.1002/9780470770801Search in Google Scholar
17. González Ordiano, J. Á.; Waczowicz, S.; Hagenmeyer, V.; Mikut, R.: Energy Forecasting Tools and Services. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (2018) 2, p. e1235.10.1002/widm.1235Search in Google Scholar
18. Goodband, J.; Haas, O.; Mills, J.: A Mixture of Experts Committee Machine to Design Compensators for Intensity Modulated Radiation Therapy. Pattern Recognition 39 (2006) 9, pp. 1704–1714.10.1016/j.patcog.2006.03.018Search in Google Scholar
19. Hartmann, B.; Ebert, T.; Fischer, T.; Belz, J.; Kampmann, G.; Nelles, O.: LMNTOOL–Toolbox zum Automatischen Trainieren Lokaler Modellnetze. In: Proceedings of the 22. Workshop Computational Intelligence (Hoffmann, F.; Hüllermeier, E., Hg.), pp. 341–355, 2014.Search in Google Scholar
20. Hoffmann, F.; Bertram, T.; Mikut, R.; Reischl, M.; Nelles, O.: Benchmarking in classification and regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 9 (2019) 5, e1318. doi:10.1002/widm.1318.Search in Google Scholar
21. Khan, S. S.; Madden, M. G.: One-Class Classification: Taxonomy of Study and Review of Techniques. The Knowledge Engineering Review 29 (2014) 03, pp. 345–374.10.1017/S026988891300043XSearch in Google Scholar
22. Kroll, A.: Computational Intelligence: Eine Einführung in Probleme, Methoden und technische Anwendungen. Oldenbourg, München, Germany, 2013.10.1524/9783486737424Search in Google Scholar
23. Kroll, A.; Abel, D.: Modellbasierte Prädiktive Regelung (Model Predictive Control). at – Automatisierungstechnik 54 (2006) 12, pp. 587–589.10.1524/auto.2006.54.12.587Search in Google Scholar
24. Leonard, J.; Kramer, M. A.; Ungar, L.: A Neural Network Architecture that Computes its Own Reliability. Computers & Chemical Engineering 16 (1992) 9, pp. 819–835.10.1016/0098-1354(92)80035-8Search in Google Scholar
25. Leonard, J. A.; Kramer, M. A.; Ungar, L. H.: Using Radial Basis Functions to Approximate a Function and its Error Bounds. IEEE Transactions on Neural Networks 3 (1992) 4, pp. 624–627.10.1109/72.143377Search in Google Scholar PubMed
26. Matalas, N.; Bier, V.: B. Extremes, Extrapolation, And Surprise. Risk Analysis 19 (1999) 1, pp. 49–54.10.1111/j.1539-6924.1999.tb00387.xSearch in Google Scholar
27. Mikut, R.; Bartschat, A.; Doneit, W.; Ordiano, J. Á. G.; Schott, B.; Stegmaier, J.; Waczowicz, S.; Reischl, M.: The MATLAB Toolbox SciXMiner: User’s Manual and Programmer’s Guide. Techn. Ber., arXiv:1704.03298, 2017.Search in Google Scholar
28. Moretti, F.; Pizzuti, S.; Panzieri, S.; Annunziato, M.: Urban Traffic Flow Forecasting Through Statistical and Neural Network Bagging Ensemble Hybrid Modeling. Neurocomputing 167 (2015), pp. 3–7.Search in Google Scholar
29. Murray-Smith, R.: Local Model Networks and Local Learning. In: Fuzzy Duisburg, ’94, 1994.Search in Google Scholar
30. Murray-Smith, R.; Johansen, T.: Multiple Model Approaches to Nonlinear Modelling and Control. CRC press, 1997.Search in Google Scholar
31. Nelles, O.: LOLIMOT – Lokale, lineare Modelle zur Identifikation nichtlinearer, dynamischer Systeme. at – Automatisierungstechnik 45 (1997) 4, pp. 163–174.10.1524/auto.1997.45.4.163Search in Google Scholar
32. Nelles, O.; Fink, A.; Isermann, R.: Local Linear Model Trees (LOLIMOT) Toolbox for Nonlinear System Identification. IFAC Proceedings Volumes 33 (2000) 15, pp. 845–850.10.1016/S1474-6670(17)39858-0Search in Google Scholar
33. Nelles, O.; Fischer, M.: Local Linear Model Trees (LOLIMOT) for Nonlinear System Identification of a Cooling Blast. In: Proc., 4th European Congress on Intelligent Techniques and Soft Computing EUFIT’96, pp. 1187–1191, Aachen, 1996.Search in Google Scholar
34. Parzen, E.: On Estimation of a Probability Density Function and Mode. The Annals of Mathematical Statistics (1962), pp. 1065–1076.10.1214/aoms/1177704472Search in Google Scholar
35. Pimentel, M. A.; Clifton, D. A.; Clifton, L.; Tarassenko, L.: A Review of Novelty Detection. Signal Processing 99 (2014), pp. 215–249.Search in Google Scholar
36. Schölkopf, B.; Platt, J. C.; Shawe-Taylor, J.; Smola, A. J.; Williamson, R. C.: Estimating the Support of a High-Dimensional Distribution. Neural Computation 13 (2001) 7, pp. 1443–1471.10.1162/089976601750264965Search in Google Scholar
37. Shin, H. J.; Eom, D.-H.; Kim, S.-S.: One-Class Support Vector Machines – an Application in Machine Fault Detection and Classification. Computers & Industrial Engineering 48 (2005) 2, pp. 395–408.10.1016/j.cie.2005.01.009Search in Google Scholar
38. Tagscherer, D.-I. M.; Lewandowski, A.: Abschätzung der Vertrauenswürdigkeit von Neuronalen Netzprognosen bei der Prozessoptimierung. In: VDI-Berichte, Bd. 1626, pp. 335–339, 2000.Search in Google Scholar
39. Tax, D. M. J.: One-Class Classification. Dissertation, Technische Universität Delft, 2001.Search in Google Scholar
40. Tresp, V.: Committee Machines, pp. 1–18. Boca Raton, FL: CRC Press, 2001.Search in Google Scholar
41. Vapnik, V. N.: Statistical Learning Theory. John Wiley, 1998.Search in Google Scholar
42. Wang, W.; Xu, Z.; Lu, W.; Zhang, X.: Determination of the Spread Parameter in the Gaussian Kernel for Classification and Regression. Neurocomputing 55 (2003) 3–4, pp. 643–663.10.1016/S0925-2312(02)00632-XSearch in Google Scholar
43. Waterhouse, S. R.: Classification and Regression Using Mixtures of Experts. Dissertation, University of Cambridge, 1998.Search in Google Scholar
44. Yuksel, S. E.; Wilson, J. N.; Gader, P. D.: Twenty Years of Mixture of Experts. IEEE Transactions on Neural Networks and Learning Systems 23 (2012) 8, pp. 1177–1193.10.1109/TNNLS.2012.2200299Search in Google Scholar PubMed
45. Zhang, Y.; Haghani, A.: A Gradient Boosting Method to Improve Travel Time Prediction. Transportation Research Part C: Emerging Technologies 58 (2015), pp. 308–324.10.1016/j.trc.2015.02.019Search in Google Scholar
46. Zhou, Z.-H.: Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC, 2012.10.1201/b12207Search in Google Scholar
© 2019 Walter de Gruyter GmbH, Berlin/Boston
