Class GP: Gaussian Process Modeling for Heterogeneous Functions

Abstract

Gaussian Processes (GP) are a powerful framework for modeling expensive black-box functions and have thus been adopted for various challenging modeling and optimization problems. In GP-based modeling, we typically default to a stationary covariance kernel to model the underlying function over the input domain, but many real-world applications, such as controls and cyber-physical system safety, often require modeling and optimization of functions that are locally stationary and globally non-stationary across the domain; using standard GPs with a stationary kernel often yields poor modeling performance in such scenarios. In this paper, we propose a novel modeling technique called Class-GP (Class Gaussian Process) to model a class of heterogeneous functions, i.e., non-stationary functions which can be divided into locally stationary functions over the partitions of input space with one active stationary function in each partition. We provide theoretical insights into the modeling power of Class-GP and demonstrate its benefits over standard modeling techniques via extensive empirical evaluations.

A Appendix

Proof sketch for the Theorem 1 follows along the lines of the proof of Theorem 3.1 in [11]. We get probabilistic uniform error bounds for GPs in each partitions $j \in [p]$ from [11] and we use per partition based bounds to bound the over all function and to derive bound on $L_1$ norm. The proof for the theorem and corollary given as follows:

Proof

1 Following bounds on each partition holds with probability $1-\delta _j$

$$\begin{aligned} \left| g_j(\textbf{x}) - \mu _{n_j}(\textbf{x})\right| \le \sqrt{\beta _j(r)}\sigma _{n_j}(\textbf{x}) + \gamma _j(r), \forall \textbf{x} \in \mathcal {X}_j \end{aligned}$$

(12)

where $\beta _j(r)$ and $\gamma _j(r)$ are given as follows

$$\begin{aligned} \beta _j(r) &= 2\log \left( \frac{M(r,\mathcal {X}_j)}{\delta _j}\right) \end{aligned}$$

(13)

$$\begin{aligned} \gamma _j(r) &= (L_{\mu _{n_j}} + L_{g_j})r + \sqrt{\beta (r)} \omega _{\sigma _{n_j}} \end{aligned}$$

(14)

Now to bound the entire function lets look at the difference $|f(\textbf{x}) - \mu _n(\textbf{x})|$.

$$\begin{aligned} \left| f(\textbf{x}) - \mu _n(\textbf{x})\right| &= \left| \sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\}(g_j(\textbf{x}) - \mu _{n_j}(\textbf{x})) \right| \end{aligned}$$

(15)

$$\begin{aligned} &= \sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\}\left| g_j(\textbf{x}) - \mu _{n_j}(\textbf{x}))\right| \end{aligned}$$

(16)

$$\begin{aligned} &\le \sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\} \left( \sqrt{\beta _j(r)}\sigma _{n_j}(\textbf{x}) + \gamma _j(r)\right) , \forall \textbf{x} \in \mathcal {X}_j \end{aligned}$$

(17)

The last inequality (17) follows from (12) and holds with probability $1-\delta$, where $\delta = \sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\} \delta _j$.

Now, redefining $\sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\} \left( \sqrt{\beta _j(r)}\sigma _{n_j}(\textbf{x})\right) = \sqrt{\beta (r)}\sigma _{n}(\textbf{x})$ and

$\sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\} \gamma _j(r) = \gamma _(r)$, we have the result.    $\square$

The proof for the Corollary 1 uses the high confidence bound 10 and is given as follows:

Proof

We know that $L_1$ norm is given by

$$\begin{aligned} \Vert f(\textbf{x}) - \mu _n(\textbf{x})\Vert _1 &= \textrm{E}[\left| f(\textbf{x}) - \mu _n(\textbf{x})\right| ] \end{aligned}$$

(18)

$$\begin{aligned} &= \int \left| f(\textbf{x}) - \mu _n(\textbf{x})\right| d\mu \end{aligned}$$

(19)

$$\begin{aligned} &= \int \left| \sum _{j=1}^{p} \mathbb {1}\{x\in \mathcal {X}_j\}(g_j(\textbf{x}) - \mu _{n_j}(\textbf{x})) \right| d\mu \end{aligned}$$

(20)

$$\begin{aligned} &= \sum _{j=1}^{p} \int \mathbb {1}\{x\in \mathcal {X}_j\} \left| (g_j(\textbf{x}) - \mu _{n_j}(\textbf{x})) \right| d\mu \end{aligned}$$

(21)

$$\begin{aligned} &= \sum _{j=1}^{p} \int _{\mathcal {X}_j} \left| (g_j(\textbf{x}) - \mu _{n_j}(\textbf{x})) \right| d\mu \end{aligned}$$

(22)

$$\begin{aligned} &\le \zeta r^{d} \sum _{j=1}^p M(r,\mathcal {X}_j)\left( \sqrt{\beta _j(r)}\sigma _{n_j}(\textbf{x}) + \gamma _j(r)\right) ~~~\text {holds w.p }1-\delta \end{aligned}$$

(23)

where $\delta = \sum _{j=1}^p \delta _j$ and $\delta _j = 1 - M(r,\mathcal {X}_j)\exp (-\beta _j(r)/2)$.    $\square$