Cope with diverse data structures in multi-fidelity modeling: A Gaussian process method

https://doi.org/10.1016/j.engappai.2017.10.008Get rights and content

Highlights

  • Multi-fidelity Gaussian process to handle diverse data structures is proposed.

  • Global trend term to capture common features of outputs.

  • Local term to capture shared and output-specific residual features of outputs.

  • The performance is fully assessed on four examples.

Abstract

Multi-fidelity modeling (MFM) frameworks, especially the Bayesian MFM, have gained popularity in simulation based modeling, uncertainty quantification and optimization, due to the potential for reducing computational budget. In the view of multi-output modeling, the MFM approximates the high-/low-fidelity outputs simultaneously by considering the output correlations, and particularly, it transfers knowledge from the inexpensive low-fidelity outputs that have many training points to enhance the modeling of the expensive high-fidelity output that has a few training points. This article presents a novel multi-fidelity Gaussian process for modeling with diverse data structures. The diverse data structures mainly refer to the diversity of high-fidelity sample distributions, i.e., the high-fidelity points may randomly fill the domain, or more challengingly, they may cluster in some subregions. The proposed multi-fidelity model is composed of a global trend term and a local residual term. Particularly, the flexible residual term extracts both the shared and output-specific residual information via a data-driven weight parameter. Numerical experiments on two synthetic examples, an aircraft example and a stochastic incompressible flow example reveal that this very promising Bayesian MFM approach is capable of effectively extracting the low-fidelity information for facilitating the modeling of the high-fidelity output using diverse data structures.

Introduction

In recent years, new advances in computers and computing science lead to the widespread use of computer simulation models, e.g., computational fluid dynamics (CFD) and finite element analysis (FEA), in engineering design and optimization. In simulation based engineering problems, surrogates are starting to play an important role, since they can approximate the expensive simulation model at some training points for alleviating computational burden. Gaussian process regression (Rasmussen and Williams, 2006), also known as Kriging (Lophaven et al., 2002), is a widely used surrogate model, since it can provide not only the prediction response but also the related prediction variance.

This article focuses on a multi-fidelity scenario where the simulator for the physics-based problem of interests can be run at multiple levels of fidelity. The high fidelity (HF) simulator yields the most accurate predictions but is most time-consuming; whereas the fast low fidelity (LF) simulators provide coarse predictions, which however include the main features of the problem and thus are useful for preliminary exploration. The LF simulators are usually simplified analysis models by using coarse finite element meshes, relaxed boundary or convergence conditions, etc. For example, it was reported by Benamara et al. (2016) that the HF simulation for a 1.5 stage booster has 5 million meshes and requires 2 h; but the LF simulation has only 0.7 million meshes and requires only 15 min. In practice, we cannot afford extensive HF simulations at many training points but many LF simulations are affordable. Suppose that the simulator has Q levels of fidelity, the multi-fidelity modeling (MFM), also known as variable-fidelity modeling or data fusion, attempts to utilize the knowledge from the correlated yet inexpensive Q1 LF simulators to enhance the modeling of the expensive HF simulator.

Considering Q levels of fidelity as Q correlated outputs, the information fusion can be achieved in the multi-output modeling framework. The multi-output GP (MOGP), also known as multi-variate Kriging (Kleijnen and Mehdad, 2014), has been developed and investigated with a long history. The MOGP attempts to model multiple correlated outputs simultaneously by sharing the information across them, with the aim of outperforming individual modeling. The key in MOGP is to construct a valid multi-output covariance function to transfer useful information across outputs. A pioneer and well-known model developed in the field of geostatistics is called linear model of coregionalization (LMC) (Journel and Huijbregts, 1978). This model constructs valid covariance functions by a linear combination of several Gaussian processes. Thereafter, various MOGPs have been developed and extended in the context of LMC Seeger et al. (2005), Bonilla et al. (2007), Hayashi et al. (2012), Osborne et al. (2012), Rakitsch et al. (2013), Dürichen et al. (2015), Hori et al. (2016). Another way to construct valid covariance functions is through process convolutions that convolve a base process, e.g., white Gaussian noise, with a smoothing kernel Ver Hoef and Barry (1998), Boyle and Frean (2004), Álvarez and Lawrence (2009), Álvarez and Lawrence (2011).1 The process convolutions can be regarded as a dynamic version of LMC Álvarez and Lawrence (2011), Álvarez et al. (2012).

In the multi-fidelity scenario, particularly, we attempt to use the inexpensive LF outputs to assist the modeling of the expensive HF output. Hence, compared to the typical MOGPs that share the information across the outputs, Kennedy and O’Hagan (2000) presented a Bayesian discrepancy-based MFM framework, which is an extension to the Co-Kriging model (Myers, 1982). In this framework, an auto-regressive model is proposed by expressing the HF output as the sum of the scaled LF outputs and an additive term that accounts for the discrepancy between the outputs, leading to not only the information sharing across outputs but also the asymmetric knowledge transfer from the LF outputs to the HF output. Later, Qian and Wu (2008) and Leen et al. (2012) provided an equivalent MFM framework in different views. Being a good multi-fidelity predictive model, Co-Kriging has been extended and improved, e.g., by reducing computational complexity Le Gratiet and Garnier (2014), Le Gratiet and Cannamela (2015), using space-dependent scaling factor (Perdikaris et al., 2017), and incorporating gradient information Han et al. (2013), Ulaganathan et al. (2015). Due to the remarkable performance, Co-Kriging has gained popularity in various fields, e.g., model inversion (Perdikaris and Karniadakis, 2016), uncertainty quantification Perdikaris et al. (2015), Kennedy and O’Hagan (2001), and multidisciplinary/robust/multi-objective optimization Forrester et al. (2007), Keane (2012), Han and Görtz (2012), Kontogiannis et al. (2017). For more information about Co-Kriging and MFM, one can refer to the recent reviews and comparison studies Fernández-Godino et al. (2016), Park et al. (2017), Toal (2015).

In the context of Co-Kriging, it is usually assumed that we can control the sampling process such that the HF and LF training points spread over the entire domain evenly by for example the nested sampling strategy (Qian, 2009) and the nearest neighbor sampling strategy (Le Gratiet and Garnier, 2014). The space-filling nested data structure, though being beneficial for Co-Kriging, cannot always be available in practice. In realistic scenarios, we need to handle diverse data structures. The diverse data structures here mainly refer to the diversity of HF sample distributions, while the inexpensive LF outputs are assumed to have sufficient training points that cover the entire domain. For example, as shown in Fig. 1,2 we have a set of uniformly distributed HF points, a set of randomly distributed HF points, and more challengingly, a set of partially distributed HF points clustered in a subregion. Besides, a practical example is that when using CFD solvers of different fidelities to simulate the flow around an aircraft, the inexpensive LF Euler simulation can be computed over the domain; while for saving computing time, the expensive HF Navier–Stokes simulation is only performed in flow regions with strong viscous effects. Hence, the diverse data structures, which contain different HF sample distributions, pose the demands of developing a particular multi-fidelity modeling approach that can effectively extract LF information to facilitate the HF modeling in different scenarios.

Therefore, this article presents a novel multi-fidelity GP model that is composed of a global trend term and a local residual term in order to tackle diverse data structures, e.g., full points that are available in the entire domain, or partial points that fill only some subregions. Particularly, in order to extract useful LF information effectively for facilitating the HF modeling, the local residual term contains a shared part and an output-specific part, the trade-off between which is dynamically determined by a data-driven weight parameter. The flexible model structure enables the approach to accomplish the multi-fidelity modeling well with diverse data structures.

The remainder of the article is organized as follows. Section 2 briefly introduces the single-output Gaussian process. Then, Section 3 presents the newly developed multi-fidelity Gaussian process in the MOGP framework. Thereafter, Section 4 comprehensively tests the new approach on two synthetic examples and two engineering examples with diverse characteristics and data structures. Finally, Section 5 offers some concluding remarks.

Section snippets

Single-output Gaussian process

Here we give a brief introduction to the single-output Gaussian process (SOGP). GP is a stochastic process wherein any finite subset of random variables follows a joint Gaussian distribution. As a non-parametric3 model, the GP interprets the target function f(x) where xRd as a probability distribution

General multi-output Gaussian process

Different from the single-output case, here we consider the multi-output case that is frequently encountered in practice. In the multi-output scenario, we attempt to learn the mapping between the input space DRd and the output space RQ where Q>1 is the number of outputs. The multi-output GP (MOGP) is to model Q outputs {ft}t=1Q simultaneously by considering the output correlations, with the aim of outperforming individual modeling.

The construction of MOGP is similar to that of SOGP. As stated

Numerical experiments

This section validates the performance of the proposed MFGP modeling approach on four examples with diverse characteristics and data structures. Among them, we consider two synthetic examples (a 1D pedagogical example and a 2D Branin example) with Q=2, and then study two real-world engineering examples (a 6D Airfoil dataset and a 2D stochastic incompressible flow dataset) with Q=2 and Q=3, respectively.

For the purpose of comparison, the testing involves four GP modeling approaches:

Conclusions

This article presents a novel multi-fidelity Gaussian process model for multi-fidelity regression problems with diverse data structures. Numerical results reveal that the proposed MFGP modeling approach has a promising performance in different scenarios. For different HF data structures, e.g., full HF points and partial HF points, the MFGP can effectively extract the LF information so as to enhance the modeling of the HF output. Future research efforts are needed to improve the modeling

Acknowledgments

This work was conducted within the Rolls-Royce@NTU Corporate Lab with support from the National Research Foundation (NRF) Singapore under the Corp Lab@University Scheme. It is also partially supported by the Data Science and Artificial Intelligence Research Center (DSAIR) and the School of Computer Science and Engineering at Nanyang Technological University .

References (48)

  • Bonilla, E.V., Chai, K.M.A., Williams, C.K., 2007. Multi-task Gaussian process prediction. In: NIPS, Vol. 20, pp....
  • Boyle, P., Frean, M.R., 2004. Dependent Gaussian processes. In: NIPS, Vol. 17, pp....
  • Burnaev, E., Zaytsev, A., 2016. Minimax error of interpolation and optimal design of experiments for variable fidelity...
  • DürichenR. et al.

    Multitask Gaussian processes for multivariate physiological time-series analysis

    IEEE Trans. Biomed. Eng.

    (2015)
  • Fernández-Godino, M.G., Park, C., Kim, N.-H., Haftka, R.T., 2016. Review of multi-fidelity models. arXiv preprint...
  • ForresterA.I. et al.

    Multi-fidelity optimization via surrogate modelling

    Proc. R. Soc. A: Math. Phys. Eng. Sci.

    (2007)
  • GramacyR.B. et al.

    Cases for the nugget in modeling computer experiments

    Stat. Comput.

    (2012)
  • HanZ.-H. et al.

    Hierarchical kriging model for variable-fidelity surrogate modeling

    AIAA J.

    (2012)
  • Han, Z.-H., Zimmermann, R., Goretz, S., 2010. A new cokriging method for variable-fidelity surrogate modeling of...
  • Hayashi, K., Takenouchi, T., Tomioka, R., Kashima, H., 2012. Self-measuring similarity for multi-task gaussian process....
  • HoriT. et al.

    Multi-task Gaussian process for imputing missing data in multi-trait and multi-environment trials

    Theor. Appl. Genet.

    (2016)
  • JournelA.G. et al.

    Mining Geostatistics

    (1978)
  • KarniadakisG. et al.

    Spectral/hp Element Methods for Computational Fluid Dynamics

    (2013)
  • KeaneA.J.

    Cokriging for robust design optimization

    AIAA J.

    (2012)
  • Cited by (62)

    View all citing articles on Scopus
    View full text