Abstract
In many real-world applications concerning multi-objective optimization, the true objective functions are not observable. Instead, only noisy observations are available. In recent years, the interest in the effect of such noise in evolutionary multi-objective optimization (EMO) has increased and many specialized algorithms have been proposed. However, evolutionary algorithms are not suitable if the evaluation of the objectives is expensive and only a small budget is available. One popular solution is to use model-based multi-objective optimization (MBMO) techniques. In this paper, we present a first investigation on noisy MBMO. For this purpose we collect several noise handling strategies from the field of EMO and adapt them for MBMO algorithms. We compare the performance of those strategies in two benchmark situations: Firstly, we perform a purely artificial benchmark using homogeneous Gaussian noise. Secondly, we choose a setting from the field of machine learning, where the structure of the underlying noise is unknown.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
As per the 3\(\sigma \) rule, \(99.7\%\) of normal distributed random numbers are within \(\pm 3 \sigma \).
References
Aizawa, A.N., Wah, B.W.: Scheduling of genetic algorithms in a noisy environment. Evol. Comput. 2(2), 97–122 (1994)
Bischl, B., Mersmann, O., Trautmann, H., Weihs, C.: Resampling methods for meta-model validation with recommendations for evolutionary computation. Evol. Comput. 20(2), 249–275 (2012)
Bischl, B., Lang, M., Kotthoff, L., Schiffner, J., Richter, J., Studerus, E., Casalicchio, G., Jones, Z.M.: mlr: machine learning in R. J. Mach. Learn. Res. 17(170), 1–5 (2016)
Bischl, B., Lang, M., Mersmann, O., Rahnenführer, J., Weihs, C.: BatchJobs and BatchExperiments: abstraction mechanisms for using R in batch environments. J. Stat. Softw. 64(11), 1–25 (2015)
Bischl, B., Richter, J., Bossek, J., Horn, D., Lang, M.: mlrMBO: A Toolbox for Model-Based Optimization of Expensive Black-Box Functions. https://github.com/berndbischl/mlrMBO
Breiderhoff, B., Bartz-Beielstein, T., Naujoks, B., Zaefferer, M., Fischbach, A., Flasch, O., Friese, M., Mersmann, O., Stork, J.: Simulation and optimization of cyclone dust separators. In: Proceedings of the 23rd Workshop on Computational Intelligence, pp. 177–195 (2013)
Dem\(\breve{\rm {s}}\)ar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)
Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)
Fieldsend, J.E., Everson, R.M.: The rolling tide evolutionary algorithm: a multiobjective optimizer for noisy optimization problems. IEEE Trans. Evol. Comput. 19(1), 103–117 (2015)
Hernández-Lobato, D., Hernández-Lobato, J.M., Shah, A., Adams, R.P.: Predictive entropy search for multi-objective bayesian optimization. arXiv preprint arXiv:1511.05467 (2015)
Horn, D., Bischl, B.: Multi-objective parameter configuration of machine learning algorithms using model-based optimization. In: IEEE Symposium on Computational Intelligence in Multicriteria Decision-Making (2016, accepted)
Horn, D., Demircioğlu, A., Bischl, B., Glasmachers, T., Weihs, C.: A comparative study on large scale kernelized support vector machines. Adv. Data Anal. Classif., 1–17 (2016)
Horn, D., Wagner, T., Biermann, D., Weihs, C., Bischl, B.: Model-based multi-objective optimization: taxonomy, multi-point proposal, toolbox and benchmark. In: International Conference on Evolutionary Multi-criterion Optimization, pp. 64–78 (2015)
Jin, Y., Branke, J.: Evolutionary optimization in uncertain environments - a survey. IEEE Trans. Evol. Comput. 9(3), 303–317 (2005)
Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13(4), 455–492 (1998)
Kleijnen, J.P.C.: White noise assumptions revisited: regression metamodels and experimental designs in practice. In: Proceedings of the 38th Conference on Winter Simulation, pp. 107–117 (2006)
Knowles, J.: Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10(1), 50–66 (2006)
Knowles, J., Corne, D., Reynolds, A.: Noisy multiobjective optimization on a budget of 250 evaluations. In: International Conference on Evolutionary Multi-criterion Optimization, pp. 36–50 (2009)
Koch, P., Wagner, T., Emmerich, M.T., Bck, T., Konen, W.: Efficient multi-criteria optimization on noisy machine learning problems. Appl. Soft Comput. 29, 357–370 (2015)
Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective optimization on a limited amount of evaluations using model-assisted \(\cal{S}\)-metric selection. In: Proceedings of the 10th International Conference on Parallel Problem Solving from Nature (PPSN), pp. 784–794 (2008)
Schiffner, J., Bischl, B., Lang, M., Richter, J., Jones, Z.M., Probst, P., Pfisterer, F., Gallo, M., Kirchhoff, D., Kühn, T., Thomas, J., Kotthoff, L.: mlr tutorial. CoRR abs/1609.06146 (2016)
Schutze, O., Esquivel, X., Lara, A., Coello, C.A.C.: Using the averaged hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE Trans. Evol. Comput. 16(4), 504–522 (2012)
Syberfeldt, A., Ng, A., John, R.I., Moore, P.: Evolutionary optimisation of noisy multi-objective problems using confidence-based dynamic resampling. Eur. J. Oper. Res. 204(3), 533–544 (2010)
Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. SIGKDD Explor. Newsl. 15(2), 49–60 (2014)
Zhan, J., Ma, Y., Zhu, L.: Multiobjective simulation optimization using stochastic kriging. In: Proceedings of the 22nd International Conference on Industrial Engineering and Engineering Management 2015, pp. 81–91 (2016)
Zhang, Q., Zhou, A., Zhaoy, S., Suganthany, P.N., Liu, W., Tiwari, S.: Multiobjective optimization test instances for the cec 2009 special session and competition. University of Essex and Nanyang Technological University, Technical report (2008)
Zhou, A., Qu, B.Y., Li, H., Zhao, S.Z., Suganthan, P.N., Zhang, Q.: Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol. Comput. 1, 32–49 (2011)
Zuluaga, M., Krause, A., Püschel, M.: e-pal: an active learning approach to the multi-objective optimization problem. J. Mach. Learn. Res. 17(104), 1–32 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Horn, D., Dagge, M., Sun, X., Bischl, B. (2017). First Investigations on Noisy Model-Based Multi-objective Optimization. In: Trautmann, H., et al. Evolutionary Multi-Criterion Optimization. EMO 2017. Lecture Notes in Computer Science(), vol 10173. Springer, Cham. https://doi.org/10.1007/978-3-319-54157-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-54157-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54156-3
Online ISBN: 978-3-319-54157-0
eBook Packages: Computer ScienceComputer Science (R0)