Solution Existence and Compactness Analysis for Nonsmooth Optimization Problems

Abstract

This paper is concerned with the analysis of geometrical properties and behaviors of the optimal value and global optimal solutions for a class of nonsmooth optimization problems. We provide conditions under which the solution set of a nonsmooth and nonconvex optimization problem is non-empty and/or compact. We also examine related properties such as the compactness of the sublevel sets, the boundedness from below and the coercivity of the objective function to characterize the non-emptiness and the compactness of the solution set of the underlying optimization problem under the unboundedness of its associated feasible set.

1 Introduction

The existence of global optimal solutions and the compactness of solution set for mathematical optimization problems are among interesting research topics in optimization theory and have received remarkable attention from researchers; see, e.g., [2, 3, 5, 6, 10, 11, 13,14,15,16, 19,20,21, 25, 29] and the related references therein. An answer to a question on whether a function/optimization problem exists its global minimizers/optimal solutions is often non-trivial even in the case of verifying the existence of a solution for a polynomial function bounded from below [28]. The authors in [1] mentioned that checking if a polynomial problem with only degree 4 possesses its solution is strongly NP-hard, and moreover, they showed that justifying the coercivity of the objective function on the feasible set or verifying the non-empty compactness of the sublevel set of the underlying setting is strongly NP-hard.

Overall, the main approaches to the study of solution existence of optimization problems can be categorized into dual/variational analysis (see, e.g., [6, 14, 18, 19, 22,23,24]) and primal/asymptotic analysis (see, e.g., [3, 5, 13, 16, 21, 25, 26]). For an optimization problem, where the objective function is a quadratic polynomial bounded from below on the feasible set and the constraint functions are affine, the authors in [11] proved that an optimal solution of the underlying program exists. Based on the well-posed property, the solution existence for a sum-of-squares convex polynomial program was given in [6]. The existence of solution set for the class of convex problems was examined in [2, 3] by using asymptotic analysis, and some extension results were given in [4]. In [5], the existence of minimum points to an optimization problem involving a geometric set was investigated by focusing on the behavior of the objective and the recession cone of the constraint set. Subsequently, authors in [25] developed corresponding results for a problem where the constraint set was defined by functional inequalities.

Various general results on the existence of solutions and the compactness of the solution set for the class of quasiconvex problems involving the asymptotic direction were later developed by the researchers in [10, 12, 13, 20]. If the objective function is a polynomial and the constraint set is an unbounded closed semi-algebraic set, the author in [29] employed the concept of tangency variety to provide necessary and sufficient conditions for the existence of optimal solutions to the considered problem. These results were extended to a class of semi-algebraic optimization problems in [19]. In addition, the non-emptiness and compactness of the optimal solution set for a general optimization problem were examined in [27] by using some properties of the sublevel set and the coercivity property of its objective function. We refer the interested reader to [18, 22, 30, 31] for other approaches and related results on the existence of solutions and the boundedness of the solution set to some specific optimization models.

In this paper, we consider a nonsmooth optimization problem defined by

$$\begin{aligned} \inf _{x\in \mathbb {R}^n} \big \{ f(x) \mid \, x\in \Omega ,\, h_i(x) \le 0, \, i=1,...,m\big \}, \end{aligned}$$

(P)

where $\Omega \subset \mathbb {R}^n$ is a non-empty closed set and $f: \mathbb {R}^n \rightarrow \mathbb {R}$ and $h_i:\mathbb {R}^n \rightarrow \mathbb {R}, \, i=1,...,m,$ are locally Lipschitz functions. We assume unless otherwise stated that the feasible set of problem (P), denoted by

$$\begin{aligned} S:=\{ x\in \Omega \mid h_i(x)\le 0, \,i=1,...,m \} \end{aligned}$$

(1.1)

is unbounded.

The main purpose of this paper is to answer the following questions:

• Under what conditions is the optimal value of problem (P) finite?

• Under what conditions does the problem (P) have a global optimal solution?

• If the solution set of problem (P) is non-empty, how do we verify its compactness/boundedness?

More precisely, we provide geometrical conditions under which the solution set of (P) is non-empty and/or compact. We also examine other conditions that guarantee the finiteness of the optimal value of problem (P). In this way, we explore related properties including the boundedness from below or the coercivity of the objective function and the compactness of sublevel sets to characterize the non-emptiness and compactness for the solution set of problem (P).

It is worth noting that a recent paper [14] considered a robust optimization problem and provided related conditions that guarantee the existence of solutions by means of constraint qualifications and related conditions. The solution existence result obtained in [14] was based mainly on the constraint qualification conditions. Unlike the robust approach from that paper, in this paper, we examine the geometrical properties and behaviors of problem (P) such as the coercivity of the objective and the compactness of a sublevel set and provide geometrical conditions that ensure the existence of global solutions and boundedness of the solution set.

The structure of the paper is as follows. In Sect. 2, after giving some basic definitions, we employ the tangency properties to examine the compactness of the sublevel sets of (P). In Sect. 3, we utilize the coercivity and the compactness of the sublevel sets to investigate the optimal value finiteness and the solution existence for the problem (P). Section 4 is devoted to the study of the non-emptiness and compactness of the solution set of (P). Sect. 5 summarizes the obtained results.

2 Preliminaries and Compactness of the Sublevel Sets

In this section, we present some definitions and results on the compactness of sublevel sets of the objective function of problem (P) that are essential for the analysis in the sequel.

Let $\mathbb {R}^n$ be the Euclidean space with the usual scalar product $\langle \cdot , \cdot \rangle$ and the Euclidean norm $\Vert \cdot \Vert$, where $n\in \mathbb {N}:=\{1,2,...\}$. The symbol $I\!\!B_r$ stands for the closed ball centered at the origin with the radius $r>0$. The notation $\mathbb {R}_+^n$ signifies the nonnegative orthant of $\mathbb {R}^n$. For a non-empty subset $\Omega \subset \mathbb {R}^n,$ the symbol $x\xrightarrow {\Omega } \bar{x}$ means that $x\rightarrow \bar{x}$ with $x\in \Omega$.

Let us recall some concepts and a calculus rule from variational analysis (see, e.g., [23, 24]). The sequential Painlevé-Kuratowski upper/outer limit of a set-valued map $F: \mathbb {R}^n\rightrightarrows \mathbb {R}^m$ as $x \rightarrow \bar{x}$ is defined by

$$\begin{aligned} \mathop {\textrm{Lim}\,\textrm{sup}}_{x\rightarrow \bar{x}} F(x):= \left\{ \vartheta \in \mathbb {R}^m \mid \exists \, {x_k\rightarrow \bar{x}}, \,\exists \,\vartheta _k \in F(x_k),\,\vartheta _k \rightarrow \vartheta \right\} . \end{aligned}$$

The Fréchet/regular normal cone $\hat{N} (\bar{x}; \Omega )$ to $\Omega$ at $\bar{x}\in \Omega$ is defined by

$$\begin{aligned} \hat{N} (\bar{x}; \Omega ) := \Big \{ \vartheta \in \mathbb {R}^n \mid \limsup \limits _{x \overset{\Omega }{\rightarrow }\ \bar{x}} \frac{\langle \vartheta ,x-\bar{x}\rangle }{\Vert x-\bar{x}\Vert } \le 0\Big \}. \end{aligned}$$

The limiting/Mordukhovich normal cone $N(\bar{x}; \Omega )$ to $\Omega$ at $\bar{x}\in \Omega$ is given by

$$\begin{aligned} N(\bar{x}; \Omega ):=\mathop {\textrm{Lim}\,\textrm{sup}}\limits _{x \overset{\Omega }{\rightarrow }\ \bar{x}} {\widehat{N}}(x;\Omega ). \end{aligned}$$

As usual, we put $\hat{N}(\bar{x};\Omega ):=\emptyset$ and $N(\bar{x};\Omega ):=\emptyset$ for any $\bar{x}\notin \Omega .$

The limiting/Mordukhovich subdifferential of $\psi : \mathbb {R}^n \rightarrow \bar{\mathbb {R}}:=\mathbb {R}\cup \{\pm \infty \}$ at $\bar{x}\in \mathbb {R}^n$ with $|\psi (\bar{x})| < \infty$ is defined by

$$\begin{aligned} \partial \psi (\bar{x}):=\left\{ \vartheta \in \mathbb {R}^n \mid (\vartheta , -1)\in N((\bar{x};\psi (\bar{x}));\text{ epi }\,\psi )\right\} , \end{aligned}$$

where

$$\begin{aligned} \textrm{epi} \psi :=\{(x,y) \in \mathbb {R}^n \times \mathbb {R}\, \mid \, \psi (x) \le y\}. \end{aligned}$$

One puts $\partial \psi (\bar{x}):=\emptyset$ if $|\psi (\bar{x})| = \infty .$

The following optimality condition for a nonsmooth optimization problem is needed in the sequel.

Lemma 2.1

(See [24, Corollary 6.6]). Let the functions $\psi _i: \mathbb {R}^n \rightarrow \mathbb {R},\, i=0,...,m+p,$ be locally Lipschitz around $\bar{x}\in \Omega ,$ where $\Omega \subset \mathbb {R}^n$ is locally closed around this point. If $\psi _0$ attains its infimum value at $\bar{x}$ on the set

$$\begin{aligned} \{x \in \Omega \,\mid \, \psi _i (x) \le 0,\, i=1,...,m,\,\psi _i (x) = 0,\, i=m+1,...,m+p\}, \end{aligned}$$

then one can find $(\mu _0,...,\mu _{m+p}) \in \mathbb {R}^{m+p+1}\setminus \{0\}$ such that $\mu _i \ge 0,\, i=0,...,m,$ and

$$\begin{aligned}&0\in \sum \limits _{i=0}^{m}\mu _i \partial \psi _{i}(\bar{x})+\sum \limits _{i=m+1}^{m+p}\mu _i \big [\partial \psi _{i}(\bar{x}) \cup \big (-\partial (-\psi _{i})(\bar{x})\big )\big ]+ N(\bar{x};\Omega ),\\&\mu _i \psi _{i}(\bar{x})=0,\, i=1,...,m. \end{aligned}$$

To proceed, let us provide the concepts of coercivity and extended tangency variety for the problem (P).

Definition 2.1

• (i) We say that the problem (P) admits the coercivity if

  $$\begin{aligned} \big [\,\forall \{x_k\}_{k\in \mathbb {N}} \subset S,\,\Vert x_k\Vert \rightarrow \infty \big ] \Rightarrow \big [\,f(x_k) \rightarrow +\infty \,\text { as }\, k \rightarrow \infty \big ]. \end{aligned}$$

• (ii) The extended tangency variety at infinity with respect to $r>0$ for the feasible set S of (P) is defined by

  $$\begin{aligned} \Lambda ^\infty _r(P):= \big \{&x \in S \setminus I\!\!B_r\, \mid \,\exists \mu :=(\mu _0,...,\mu _m)\in \mathbb {R}_+^{m+1},\, \exists \lambda \in \mathbb {R}, \, (\mu ,\lambda )\ne 0, \\&0 \in \mu _0\partial f(x) + \sum \limits _{i=1}^m \mu _i \partial h_i(x)+ \lambda x \\ &+N(x;\Omega ), \, \mu _i\,h_i(x)=0,\,i=1,...,m\big \}. \end{aligned}$$

• (iii) The extended tangency variety for the feasible set S of (P) is given by

  $$\begin{aligned} \Lambda ^\infty (P):= \big \{&x \in S \, \mid \,\exists \mu :=(\mu _0,...,\mu _m)\in \mathbb {R}_+^{m+1},\, \exists \lambda \in \mathbb {R}, \, (\mu ,\lambda )\ne 0, \nonumber \\&0 \in \mu _0\partial f(x) + \sum \limits _{i=1}^m \mu _i \partial h_i(x)+ \lambda x \nonumber \\ &+N(x;\Omega ), \, \mu _i\,h_i(x)=0,\,i=1,...,m\big \}. \end{aligned}$$

  (2.1)

• (iv) The set of tangency values at infinity for the problem (P) is defined by

$$\begin{aligned} T^\infty (P) :=\{\lambda \in \mathbb {R} \mid \,\exists \{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty (P),\, \Vert x_k \Vert \rightarrow \infty , \, f(x_k) \rightarrow \lambda \, \,\textrm{as} \,\,k \rightarrow \infty \}. \end{aligned}$$

Note by definition that $\Lambda ^\infty _r(P) \subset \Lambda ^\infty (P)$ for any $r>0$. The concept in Definition 2.1(iii) was given in [14] for the setting of robust optimization, and if (P) is an unconstrained (i.e., $\Omega :={\mathbb {R}}^n$) polynomial program with a non-constant polynomial f, this concept agrees with the tangency variety of f in [32], and more particularly, if $n=2$, then it reduces to the curve of tangency in [9]. It is also worth mentioning that for an unconstrained polynomial program (P), the sets $\Lambda ^\infty _r (P)$ with $r>0$ and $T^\infty (P)$ can be effectively computed by using tractable/analogous formulas of the polynomial settings (cf. [7, 8, 17]).

In connection with the problem (P), we consider its optimal value, denoted by $\inf (P)$, and a sublevel set of the objective f at $\lambda \in \mathbb {R}$ on S given by

$$\begin{aligned} {S_{lev}}(\lambda ):=\{x \in S \mid f(x)\le \lambda \}. \end{aligned}$$

The following proposition shows under the finiteness of the optimal value that the compactness or non-compactness of the sublevel set of the objective of problem (P) can be justified through the set of tangency values at infinity and the extended tangency variety at infinity.

Proposition 2.1

For the problem (P), let $\inf (P)>-\infty$ and consider the set $S_{lev}(\lambda )$, where $\lambda \in \mathbb {R}$. Then, the following statements hold:

• (i)   If $T^\infty (P)\ne \emptyset$, then the problem $\inf \{\lambda \mid \lambda \in T^\infty (P)\}$ attains its infimum, denoted by $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\}$, and $\lambda _{\min }\ge \inf (P)$.

• (ii)    If $T^\infty (P)= \emptyset$, then $S_{lev}(\lambda )$ is compact.

• (iii)   If $T^\infty (P)\ne \emptyset$ and $\lambda > \lambda _{\min }$, then $S_{lev}(\lambda )$ is not compact.

• (iv)   If $T^\infty (P)\ne \emptyset$ and $\lambda < \lambda _{\min }$, then $S_{lev}(\lambda )$ is compact.

• (v)    Let $T^\infty (P)\ne \emptyset$ and $\lambda = \lambda _{\min }$. Then, $S_{lev}(\lambda )$ is compact if and only if there exists $r>0$ such that $f(x)>\lambda _{\min }$ for all $x \in \Lambda ^\infty _r (P)$.

Proof

Observe first that

$$\begin{aligned} \lambda \ge \inf (P) \text{ for } \text{ all } \lambda \in T^\infty (P). \end{aligned}$$

(2.2)

Indeed, if this is not the case, there exists $\lambda _0\in T^\infty (P)$ such that $\lambda _0<\inf (P).$ Then, there exists $\{x_k\}_{k\in \mathbb {N}} \subset S\setminus I\!\!B_r$ such that $f(x_k) \rightarrow \lambda _0$ as $k \rightarrow \infty .$ Since $f(x_k)\ge \inf (P)$ for all $k\in \mathbb {N}$ and hence $\lambda _0\ge \inf (P),$ which is absurd. So, our observation is valid.

(i)   Let $T^\infty (P)\ne \emptyset$. By the above observation, $\inf \{\lambda \mid \lambda \in T^\infty (P)\} \ge \inf (P)$ and so $\inf \{\lambda \mid \lambda \in T^\infty (P)\}:=\lambda _{\min }\in \mathbb {R}.$ It suffices to show that $\lambda _{\min } \in T^\infty (P).$ Since $\inf \{\lambda \mid \lambda \in T^\infty (P)\}=\lambda _{\min }$, there is a sequence $\{\lambda _k\}_{k \in \mathbb {N}} \subset T^\infty (P)$ such that $\lambda _k \rightarrow \lambda _{\min }$ as $k \rightarrow \infty$. If $\lambda _k = \lambda _{\min }$ for all sufficiently large $k\in \mathbb {N}$, then clearly, $\lambda _{\min } \in T^\infty (P).$

Now, there is no loss of generality in assuming that $\lambda _k \ne \lambda _{\min }$ for all $k \in \mathbb {N}$. Let $k \in \mathbb {N}$. We get by $\lambda _k\in T^\infty (P)$ that there exists $\{x_{k,m}\}_{m\in \mathbb {N}} \subset \Lambda ^\infty (P)$ satisfying

$$\begin{aligned} \Vert x_{k,m} \Vert \rightarrow \infty \text { and } f(x_{k,m}) \rightarrow \lambda _k \, \,\textrm{as} \,\,m \rightarrow \infty . \end{aligned}$$

Take $m_k\in \mathbb {N}$ such that $m_k>k$ and

$$\begin{aligned} |f(x_{k,m}) -\lambda _{k}|\le |\lambda _k-\lambda _{\min }| \text { for all } m \ge m_k. \end{aligned}$$

By letting $z_k:=x_{k,m_k},\, k\in \mathbb {N}$, we see that $\{z_{k}\}_{k\in \mathbb {N}} \subset \Lambda ^\infty (P)$ and $\Vert z_{k} \Vert \rightarrow \infty$ as $k\rightarrow \infty$ and that

$$\begin{aligned} |f(z_k) -\lambda _{\min }|\le |f(x_{k,m_k}) -\lambda _k|+|\lambda _k-\lambda _{\min }| \le 2|\lambda _k-\lambda _{\min }| \text { for all } k \in \mathbb {N}, \end{aligned}$$

which shows that $f(z_k) \rightarrow \lambda _{\min } \, \,\textrm{as} \,\,k \rightarrow \infty .$ Therefore, $\lambda _{\min } \in T^\infty (P)$ and consequently, the problem $\inf \{\lambda \mid \lambda \in T^\infty (P)\}$ attains its infimum with $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\}$ and $\lambda _{\min }\ge \inf (P)$ by virtue of (2.2).

(ii)   Let $T^\infty (P)= \emptyset .$ Assume on the contrary that $S_{lev}(\lambda )$ is not compact. Then, there exists a sequence $\{x_k\}_{k\in \mathbb {N}} \subset S$ such that $\Vert x_k \Vert \rightarrow \infty$ as $k \rightarrow \infty$ and $f(x_k) \le \lambda$ for all $k\in \mathbb {N}.$ Furthermore, by $\inf (P)>-\infty$, $\underset{x \in S,\,\Vert x\Vert \rightarrow \infty }{\liminf }\,\, f(x)$ is finite and so we denote

$$\begin{aligned} \omega :=\underset{x \in S,\,\Vert x\Vert \rightarrow \infty }{\liminf }\,\, f(x), \end{aligned}$$

(2.3)

where $\inf (P)\le \omega \le \lambda .$ By definition,

$$\begin{aligned} \underset{x \in S,\,\Vert x\Vert \rightarrow \infty }{\liminf }\,\, f(x)=\sup _{t>0}\Big (\inf _{x \in S,\,\Vert x\Vert \ge t}f(x)\Big ), \end{aligned}$$

and so for each $k \in \mathbb {N}$, there exists $t_k>0$ such that $\omega < \inf \limits _{x \in S,\,\Vert x\Vert \ge t_k}f(x)+\dfrac{1}{k},$ which ensures that

$$\begin{aligned} \omega -\dfrac{1}{k} <f(x) \text { for all } x \in S \text { with }\Vert x\Vert \ge t_k. \end{aligned}$$

(2.4)

Now, by the definition of $\inf \limits _{x \in S,\,\Vert x\Vert \ge \max \{k, t_k\}}f(x)$, for each $k\in \mathbb {N}$, there is $v_k \in S$ with $\Vert v_k\Vert \ge \max \{k, t_k\}$ such that $f(v_k)-\dfrac{1}{k}<\inf \limits _{x \in S,\,\Vert x\Vert \ge \max \{k, t_k\}}f(x)$ and hence

$$\begin{aligned} f(v_k)< \omega + \dfrac{1}{k}. \end{aligned}$$

This, together with (2.4), entails that

$$\begin{aligned} \omega -\dfrac{1}{k}<f(v_k)<\omega +\dfrac{1}{k}\; \text{ for } \text{ all } \; k\in \mathbb {N}. \end{aligned}$$

Consequently, we find a sequence $\{v_k\}_{k\in \mathbb {N}} \subset S$ such that

$$\begin{aligned} \Vert v_k\Vert \rightarrow \infty \text { and } f(v_k)\rightarrow \omega \text { as } k\rightarrow \infty . \end{aligned}$$

For each $k\in \mathbb {N}$, we let $S_k:= \{x\in S \mid \Vert x\Vert ^2=\Vert v_k\Vert ^2\}$. Note that $S_k$ is a non-empty set because of $v_k\in S_k.$ Moreover, since f is continuous on S and $S_k$ is compact, it allows us to find $x^*_k\in S_k$ such that

$$\begin{aligned} f(x^*_k)\le f(x)\, \text{ for } \text{ all } \, x\in S_k. \end{aligned}$$

This entails that $x^*_k$ is an optimal solution of the following problem

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert v_k\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\}. \end{aligned}$$

Using Lemma 2.1, we find $(\mu _0,...,\mu _{m+1}) \in \mathbb {R}^{m+2}\setminus \{0\}$ with $\mu _i \ge 0,\, i=0,...,m,$ such that

$$\begin{aligned}&0\in \mu _0 \partial f(x^*_k)+\sum \limits _{i=1}^m\mu _i \partial h_i(x^*_k) + 2\mu _{m+1} x^*_k + N(x^*_k;\Omega ), \\ \nonumber&\mu _i h_i(x^*_k)=0, \, i=1,...,m, \end{aligned}$$

(2.5)

which entails that $x^*_k \in \Lambda ^\infty (P)$, where $\Lambda ^\infty (P)$ is defined as in (2.1). Since $\Vert x^*_k\Vert =\Vert v_k\Vert$ for all $k\in \mathbb {N}$ and $\Vert v_k\Vert \rightarrow \infty$ as $k\rightarrow \infty ,$ we get a sequence $\{x^*_k\}_{k\in \mathbb {N}}\subset \Lambda ^\infty (P)\subset S$ satisfying $\Vert x^*_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$ and

$$\begin{aligned} f(x^*_k)\le f(v_k) \text{ for } \text{ all } k\in \mathbb {N}, \end{aligned}$$

which turns out that $\underset{k \rightarrow \infty }{\limsup }\,\, f(x^*_k)\le \omega$ due to $f(v_k)\rightarrow \omega$ as $k\rightarrow \infty$. Besides, by taking (2.3) into account, we see that $\omega \le \underset{k \rightarrow \infty }{\liminf }\,\, f(x^*_k)$. Then, we conclude that $f(x^*_k)\rightarrow \omega$ as $k\rightarrow \infty$ and so $\omega \in T^\infty (P)$. This contradicts our assumption that $T^\infty (P)=\emptyset$. In conclusion, $S_{lev}(\lambda )$ is compact.

(iii)   Let $T^\infty (P)\ne \emptyset$ and $\lambda > \lambda _{\min }$. By (i), $\lambda _{\min }\in T^\infty (P)$, and so there exists a sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty (P)$ such that

$$\begin{aligned} \Vert x_k\Vert \rightarrow \infty \text { and } f(x_k) \rightarrow \lambda _{\min } \text { as } k \rightarrow \infty . \end{aligned}$$

Because of $\lambda > \lambda _{\min }$ and $f(x_k) \rightarrow \lambda _{\min }$ as $k \rightarrow \infty$, we can find $n_0\in \mathbb {N}$ such that $f(x_k) < \lambda$ for all $k \ge n_0$. This shows that $S_{lev}(\lambda )$ contains the unbounded sequence $\{x_k\}_{k\ge n_0}$ and hence $S_{lev}(\lambda )$ is not compact.

(iv)   Let $T^\infty (P)\ne \emptyset$ and $\lambda < \lambda _{\min }$. Suppose on the contrary that $S_{lev}(\lambda )$ is not compact. Then, by similar arguments as in the proof of (ii), we arrive at

$$\begin{aligned} \omega \in T^\infty (P), \end{aligned}$$

where $\omega :=\underset{x \in S,\,\Vert x\Vert \rightarrow \infty }{\liminf }\,\, f(x)\le \lambda .$ Hence, $\omega \le \lambda <\lambda _{\min },$ which is impossible due to $\lambda _{\min } =\min \{\lambda \mid \lambda \in T^\infty (P)\}\le \omega .$ So, (iv) has been justified.

(v)  Let $T^\infty (P)\ne \emptyset$ and $\lambda = \lambda _{\min }$. We first assume that $S_{lev}(\lambda )$ is a compact set. Then, there exists $\bar{r}>0$ such that ${S_{lev}}(\lambda _{\min }) \subset I\!\!B_{\bar{r}}.$ For any $r\ge \bar{r}$, it holds that $S_{lev}(\lambda _{\min }) \cap (S\setminus I\!\!B_r) =\emptyset$, which implies that $S_{lev}(\lambda _{\min }) \cap \Lambda ^\infty _r (P) =\emptyset$ as $\Lambda ^\infty _r (P)\subset S\setminus I\!\!B_r$. Therefore, $f(x) > \lambda _{\min }$ for all $x \in \Lambda ^\infty _r (P)$.

Conversely, assume that there exists $r>0$ such that

$$\begin{aligned} f(x) > \lambda _{\min }, \,\,\,\forall x \in \Lambda ^\infty _r (P). \end{aligned}$$

(2.6)

We assert that $S_{lev}(\lambda _{\min })$ is compact. Indeed, if this is not the case, we can find a sequence $\{x_k\}_{k\in \mathbb {N}} \subset S$ satisfying $\Vert x_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$ and $f(x_k) \le \lambda _{\min }$ for all $k \in \mathbb {N}.$ Then, there is $n_0 \in \mathbb {N}$ satisfying $\Vert x_k\Vert > r$ for all $k \ge n_0$.

For $k\ge n_0$, we let $\tilde{S}_k:= \{x\in S \mid \Vert x\Vert ^2=\Vert x_k\Vert ^2\}$. By similar arguments as in the proof of (ii), we can find $\tilde{x}^*_k \in \tilde{S}_k$ such that $\tilde{x}^*_k$ is an optimal solution of the following problem

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert x_k\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\} \end{aligned}$$

and $\tilde{x}^*_k \in \Lambda ^\infty _r (P)$ for all $k \ge n_0$. By $x_k\in \tilde{S}_k$ for $k \ge n_0,$ we also arrive at

$$\begin{aligned} f(\tilde{x}^*_k) \le f(x_k)\le \lambda _{\min }, \end{aligned}$$

which contradicts (2.6). So, the proof is complete. $\square$

We finish this section with an example illustrating how we employ Proposition 2.1 to verify the compactness or non-compactness of sublevel sets by utilizing the tangency values at infinity.

Example 2.1

Let $f:\mathbb {R}\rightarrow \mathbb {R}$ and $h_1:\mathbb {R}\rightarrow \mathbb {R}$ be defined respectively by

$$\begin{aligned} f(x):={\left\{ \begin{array}{ll} -\dfrac{1}{x^2+1}\,\,\,\,& \text {if}\,\,x \ge 0, \\ -x-1\,\,\,\,& \text {if}\,\,x<0\end{array}\right. } \text { and } h_1(x):=-|x|+5 \text { for } x\in \mathbb {R}. \end{aligned}$$

Letting $\Omega :=(-\infty ,-2] \cup [6,+\infty )$, we consider the problem (P) with $m=1$ as follows:

$$\begin{aligned} \underset{x\in \mathbb {R}}{\inf }\big \{ f(x) \mid x\in \Omega ,\, h_1(x) \le 0 \big \}. \end{aligned}$$

(EP1)

In this setting, it is easy to see that the feasible set S of problem (EP1) is $S=(-\infty ,-5] \cup [6,+\infty )$. Moreover, the extended tangency variety $\Lambda ^\infty (EP1)$ for S of (EP1) is computed by

$$\begin{aligned} \Lambda ^\infty (EP1)=&\, \big \{x \in S \, \mid \, \exists (\mu _0,\mu _1)\in \mathbb {R}_+^2,\, \exists \lambda \in \mathbb {R},\, (\mu _0,\mu _1,\lambda )\ne 0, \\&\quad \quad \quad \quad \quad \quad \,\,\,0 \in \mu _0 \partial f(x) +\mu _1 \partial h_1(x)+ \lambda x +N(x;\Omega ),\, \mu _1h_1(x)=0\big \}\\ =&\,(-\infty ,-5] \cup [6,+\infty )=S. \end{aligned}$$

On the one hand, by choosing a sequence $x_k:=k$ for $k\ge 6$, we see that $\{x_k\}_{k\in \mathbb {N}}\subset \Lambda ^\infty (EP1)$ satisfying $|x_k|\rightarrow \infty$ and $f(x_k)=-\dfrac{1}{k^2+1}\rightarrow 0$ as $k\rightarrow \infty .$ Therefore, $0\in T^\infty (EP1)$. On the other hand, for any sequence $\{x_k\}_{k\in \mathbb {N}}\subset \Lambda ^\infty (EP1)$ satisfying $|x_k|\rightarrow \infty$ and $f(x_k)\rightarrow y\in \mathbb {R}$ as $k\rightarrow \infty$, we assert that $f(x_k)=-\dfrac{1}{x_k^2+1}$ for all k sufficiently large. Otherwise, existing a subsequence $\{x_{k_n}\}$ with $f(x_{k_n})=-x_{k_n}-1$ would result in that the sequence $f(x_k)$ is not convergent to y when $k\rightarrow \infty .$ Thus, $f(x_k)=-\dfrac{1}{x_k^2+1}\rightarrow 0$ as $k\rightarrow \infty$ and so $y=0$, which in turn guarantees that $T^\infty (EP1)=\{0\}.$

Now, we can justify that $\textrm{inf} (EP1) >-\infty$. By Proposition 2.1(i), the problem $\inf \{\lambda \mid \lambda \in T^\infty (EP1)\}$ attains its infimum and $\lambda _{\min }\ge \inf (P)$. In fact, we have

$$\begin{aligned} \lambda _{\min }=\inf \{\lambda \mid \lambda \in T^\infty (EP1)\}=0>-1= \inf (EP1). \end{aligned}$$

Moreover, we can see from Proposition 2.1(iii) that $S_{lev} (\lambda )$ is not compact for any $\lambda > 0$, i.e., $\lambda > \lambda _{\min }$, while it follows by Proposition 2.1(iv) that $S_{lev} (\lambda )$ is compact for any $\lambda < 0$, i.e., $\lambda < \lambda _{\min }$. Next, for $\lambda =\lambda _{\min }=0$, we consider arbitrary $r >0$. Then, by picking $x_k:=k \in \Lambda ^\infty _r (EP1)=\Lambda ^\infty (EP1) \setminus [-r,r]$ for $k >\max \{6, r\}$, we see that $f(x_k)=-\dfrac{1}{k^2+1}< \lambda _{\min }$. So, we cannot find $r>0$ such that $f(x)>\lambda _{\min }$ for all $x \in \Lambda ^\infty _r (EP1)$. From the assertion of Proposition 2.1(v), it entails that $S_{lev}(\lambda )$ is not compact. In fact, $S_{lev}(\lambda ):=\{x\in S\mid f(x)\le \lambda \}=[6,+\infty )$.

3 Optimal Value Finiteness and Solution Existence

In this section, we provide conditions in terms of compactness of the sublevel sets and the coercivity of the objective function that guarantee the finiteness of the optimal value or the existence of global optimal solutions for the problem (P).

The first theorem establishes characterizations for the finiteness of the optimal value of problem (P). It also shows the close relationships among the finiteness of the optimal value, the emptiness of the tangency value set, the compactness of the sublevel sets and the coercivity of problem (P).

Theorem 3.1

Consider the problem (P). Then, we have the following assertions.

• (i)   $\inf (P)>-\infty$ if and only if $\,\inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} >-\infty$ for any $r>0$.

• (ii)   The following three conditions are equivalent:

  • (a)  $\inf (P)>-\infty$ and $T^\infty (P)= \emptyset$.

  • (b)  $S_{lev} (\lambda )$ is a compact set for any $\lambda \in \mathbb {R}$.

  • (c)   The problem (P) is coercive.

Proof

(i)  Let $\inf (P)>-\infty$. Since $\Lambda ^\infty _r (P) \subset S$ for any $r>0$, it holds that

$$\begin{aligned} \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} \ge \inf \big \{f(x) \mid x \in S\big \}, \end{aligned}$$

and so

$$\begin{aligned} \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} \ge \inf (P) >-\infty . \end{aligned}$$

Conversely, for any fixed $r>0$, assume that $\inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} >-\infty$. To show $\inf (P)>-\infty$, we suppose on the contrary that $\inf (P)=-\infty$. Then, one can pick a sequence $\{x_k\}_{k\in \mathbb {N}} \subset S$ satisfying $f(x_k) \rightarrow -\infty \text { as } k \rightarrow \infty .$ We claim that $\{x_k\}_{k\in \mathbb {N}}$ is unbounded. Otherwise, we can find a convergent subsequence, say $\{x_{k_l}\}_{k_l\in \mathbb {N}}$, such that $x_{k_l} \rightarrow \bar{x}$ for some $\bar{x}\in S$. Since f is continuous, it entails that $f(x_{k_l}) \rightarrow f(\bar{x})$ as $k_l \rightarrow \infty$. This contradicts the fact that $f(x_{k}) \rightarrow -\infty$ as $k \rightarrow \infty$. So, we have $\Vert x_k\Vert \rightarrow \infty \text { as } k \rightarrow \infty .$

Pick $n_0 \in \mathbb {N}$ such that $\Vert x_k\Vert > r$ for all $k \ge n_0$. For $k\ge n_0$, we let $\tilde{S}_k:= \{x\in S \mid \Vert x\Vert ^2=\Vert x_k\Vert ^2\}$. By similar arguments as in the proof of Proposition 2.1(ii), we can find $x^*_k \in \tilde{S}_k$ such that $x^*_k$ is an optimal solution of the following problem

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert x_k\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\} \end{aligned}$$

and $x^*_k \in \Lambda ^\infty _r (P)$. By $x_k\in \tilde{S}_k$ for each $k \ge n_0,$ we also arrive at

$$\begin{aligned} f(x^*_k) \le f(x_k) \text { for all } k \ge n_0, \end{aligned}$$

which ensures that $f(x^*_k) \rightarrow -\infty \text { as } k \rightarrow \infty$ due to $f(x_k) \rightarrow -\infty \text { as } k \rightarrow \infty .$ Consequently, $\inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} =-\infty$, a contradiction. So, (i) holds.

(ii) Observe that [(b)$\Leftrightarrow$(c)]. To finish the proof, we need to show that [(a)$\Leftrightarrow$(c)].

[(a)$\Rightarrow$(c)]  Let $\inf (P)>-\infty$ and $T^\infty (P)= \emptyset$. It is followed by Proposition 2.1(ii) that $S_{lev} (\lambda )$ is compact for any $\lambda \in \mathbb {R}$. This means that the problem (P) is coercive.

[(c)$\Rightarrow$(a)] Let the problem (P) be coercive. By the coercivity, any sequence $\{x_k\}_{k\in \mathbb {N}}\subset \Lambda ^\infty (P) \subset S$ with $\Vert x_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$ ensures that $f(x_k)\rightarrow +\infty$ as $k\rightarrow \infty$. Therefore, by definition, $T^\infty (P)= \emptyset .$ It remains to show that $\inf (P)>-\infty$. We assume on the contrary that $\inf (P)=-\infty$. Then, there exists $\{x_k\}_{k\in \mathbb {N}} \subset S$ such that $f(x_k) \rightarrow -\infty \text { as } k \rightarrow \infty .$ If $\{x_k\}_{k\in \mathbb {N}}$ is bounded, we can pick a convergent subsequence $\{x_{k_l}\}_{k_l\in \mathbb {N}}$ of $\{x_k\}_{k\in \mathbb {N}}$ such that $x_{k_l} \rightarrow \bar{x}$ for some $\bar{x}\in S$ as $k_l \rightarrow \infty$. By the continuity of f, we arrive at $f(x_{k_l}) \rightarrow f(\bar{x})\in \mathbb {R}$ as $k_l \rightarrow \infty$, which is impossible. Otherwise, $\{x_k\}_{k\in \mathbb {N}}$ is unbounded, then, by the coercivity, $f(x_k)\rightarrow +\infty$ as $k\rightarrow \infty ,$ which is also a contradiction. So, $\inf (P)>-\infty$, which completes the proof of the theorem. $\square$

The next theorem presents characterizations by means of the tangency values at infinity or the extended tangency variety at infinity for the solution existence of problem (P). In what follows, the set of all global optimal solutions of problem (P) is denoted by ${\mathcal {S}}(P).$

Theorem 3.2

Consider the problem (P). The following conditions are equivalent:

• (i)   ${\mathcal {S}}(P)$ is a non-empty set.

• (ii)  One of the following statements holds but never both:

  • (a)   $\inf (P) >-\infty$ and $T^\infty (P) = \emptyset$.

  • (b)  $\inf (P) >-\infty$, $T^\infty (P) \ne \emptyset$ and there exists $\bar{x}\in S$ such that $\lambda _{\min } \ge f(\bar{x}),$ where $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\}$.

• (iii)  There exists $x^* \in S$ such that $f(x^*) \le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}$ for any $r>0.$

• (iv) There exists $r>0$ such that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le \underset{x \in \Lambda ^\infty _{ r} (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x), \end{aligned}$$

where “$\min$” indicates that the relaxation problem in the left-hand side attains its optimal solutions.

Proof

[(i) $\mathrm{\Rightarrow }$ (ii)] Let ${\mathcal {S}}(P) \ne \emptyset$. It is clear that $\inf (P) >-\infty$. If $T^\infty (P) = \emptyset$, then we arrive at the statement (a). Otherwise, $T^\infty (P) \ne \emptyset$. According to Proposition 2.1(i), $\lambda _{\min }$ is well-defined and $\lambda _{\min } \ge \inf (P).$ Now, picking an optimal solution $\bar{x}\in {\mathcal {S}}(P)$, we see that $\bar{x}\in S$ and $\lambda _{\min } \ge \inf (P)= f(\bar{x})$ and so the assertion (b) is verified.

[(ii) $\Rightarrow$ (iii)] Let (a) hold, we show that there is $x^* \in S$ such that

$$\begin{aligned} f(x^*) \le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} \end{aligned}$$

(3.1)

holds for all $r>0.$ Take $\bar{x} \in S$ and put $\bar{\lambda }:= f(\bar{x})$. We see that the non-empty sublevel set $S_{lev}(\bar{\lambda })$ is compact by Theorem 3.1(ii). Due to the continuity of f on $S_{lev}(\bar{\lambda })\subset S$, one can find $x^* \in S_{lev}(\bar{\lambda })$ such that

$$\begin{aligned} f(x^*) \le f(x) \text { for all } x \in S_{lev}(\bar{\lambda }), \end{aligned}$$

which entails that

$$\begin{aligned} f(x^*) \le f(x) \text { for all } x \in S. \end{aligned}$$

This shows that $x^*$ is an optimal solution to the problem (P), and therefore

$$\begin{aligned} f(x^*)=\inf (P) \le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}. \end{aligned}$$

This means that (3.1) holds for any $r>0$ and so does (iii).

Now, let (b) hold. This means that $\inf (P) >-\infty$, $T^\infty (P) \ne \emptyset$ and there exists $\bar{x}\in S$ satisfying $\lambda _{\min } \ge f(\bar{x}).$

If $\lambda _{\min } > f(\bar{x}):=\bar{\lambda },$ then, by $T^\infty (P) \ne \emptyset$ and Proposition 2.1(iv), the set $S_{lev}(\bar{\lambda })$ is compact. Furthermore, $\bar{x}\in S_{lev}(\bar{\lambda })$ and so $S_{lev}(\bar{\lambda })$ is a non-empty compact. Hence, the inequality (3.1) follows as above.

Next, let $\lambda _{\min } = f(\bar{x}).$ If $f(\bar{x})=\inf (P) >-\infty ,$ then $\bar{x}$ is an optimal solution of problem (P) and so the inequality (3.1) holds with $x^*:=\bar{x}.$ Otherwise, $f(\bar{x})>\inf (P) >-\infty .$ Pick a number $\lambda \in \mathbb {R}$ satisfying $\lambda _{\min }> \lambda >\inf (P)$ and consider a sublevel set of f at $\lambda$ as

$$\begin{aligned} S_{lev}(\lambda ):= \{ x \in S \mid f(x) \le \lambda \}. \end{aligned}$$

The compactness of $S_{lev}(\lambda )$ is implied by Proposition 2.1(iv). For $\epsilon :=\lambda -\inf (P)>0,$ we get by the definition of $\inf (P)$ that there exists $x_\epsilon \in S$ such that $f(x_\epsilon )<\lambda$ and so $x_\epsilon \in S_{lev}(\lambda )\ne \emptyset .$ Hence, the inequality (3.1) follows as above.

[(iii) $\Rightarrow$ (iv)] Let (iii) hold, i.e., there exists $x^* \in S$ satisfying (3.1) for any $r>0$. Choosing $\bar{r}>\Vert x^*\Vert$, we see that $x^* \in S \cap I\!\!B_{\bar{r}}$ and so $S \cap I\!\!B_{\bar{r}}$ is a non-empty compact set. This leads to the existence of optimal solutions for the following problem

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le \bar{r}^2,\, h_i(x)\le 0, \, i=1,...,m\}. \end{aligned}$$

In addition, it can be checked that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le \bar{r}^2,\, h_i(x)\le 0, \, i=1,...,m\}&\le f(x^*) \\&\le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}\\&\le \underset{x \in \Lambda ^\infty _{\bar{r}} (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x), \end{aligned}$$

which asserts that (iv) is valid.

[(iv) $\Rightarrow$ (i)] Assume that there exists ${r}>0$ such that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le \underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x), \end{aligned}$$

(3.2)

which also ensures that there exists $\bar{x}^*\in S$ such that

$$\begin{aligned} f(\bar{x}^*)=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}. \end{aligned}$$

(3.3)

By the definition of $\inf (P),$ there exists a sequence $\{x_k\}_{k\in \mathbb {N}}\subset S$ such that $f(x_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty .$ If $\{x_k\}_{k\in \mathbb {N}}$ is bounded, then there exists a subsequence $\{x_{k_l}\}_{k_l\in \mathbb {N}}$ of $\{x_k\}_{k\in \mathbb {N}}$ converging to $\bar{x}\in S$. By the continuity of f on S, we arrive at $f(x_{k_l}) \rightarrow f(\bar{x})$ as $k_l \rightarrow \infty$. We also have $f(x_{k_l}) \rightarrow \inf (P) \text { as } k_l \rightarrow \infty .$ So, $f(\bar{x})=\inf (P)$, which shows that $\bar{x}\in {\mathcal {S}}(P)$, i.e., ${\mathcal {S}}(P) \ne \emptyset$. Now, let $\{x_k\}$ be unbounded, i.e.,

$$\begin{aligned} \Vert x_k\Vert \rightarrow \infty \text { and } f(x_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty . \end{aligned}$$

Picking $n_0\in \mathbb {N}$ such that for all $k \ge n_0$, we have $\Vert x_k\Vert > r$. For each $k \ge n_0,$ put $\tilde{S}_k:=\{ x\in S \mid \, \Vert x\Vert ^2=\Vert x_k\Vert ^2\}$ and observe that $\tilde{S}_k$ is non-empty compact due to $x_k \in \tilde{S}_k.$ We argue similarly as in the proof Proposition 2.1(ii) to find $x^*_k \in \tilde{S}_k$ so that $x^*_k$ is an optimal solution of the following problem:

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert x_k\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\} \end{aligned}$$

and moreover $x^*_k \in \Lambda ^\infty _r (P)$ for all $k \ge n_0$.

Note that $\inf (P)\le f(x^*_k) \le f(x_k)$ for all $k \ge n_0$ because of $x_k \in \tilde{S}_k.$ As a result, we get a sequence $\{x^*_k\}_{k \ge n_0} \subset \Lambda ^\infty _r (P)$ such that

$$\begin{aligned} \Vert x^*_k\Vert \rightarrow \infty \text { and } f(x^*_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty \end{aligned}$$

and hence $\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)=\inf (P)$. Taking (3.2) into account, one has

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le \inf (P), \end{aligned}$$

which entails that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} = \inf (P). \end{aligned}$$

This together with (3.3) shows that $\bar{x}^*$ is an optimal solution of (P), i.e., ${\mathcal {S}}(P) \ne \emptyset$. So, the proof of the theorem is complete. $\square$

Let us provide an example which shows how to employ the equivalent conditions in the above theorem to determine the solution existence of a nonsmooth optimization problem.

Example 3.1

Let $f:\mathbb {R}\rightarrow \mathbb {R}$ and $h_1:\mathbb {R}\rightarrow \mathbb {R}$ be given respectively by

$$\begin{aligned} f(x):={\left\{ \begin{array}{ll} 1-x\,\,\,\,& \text {if}\,\,x \le 0, \\ \cos x\,\,\,\,& \text {if}\,\,x>0\end{array}\right. } \text { and } h_1(x):=-|x^{2}-5|+4 \text{ for } x\in \mathbb {R}. \end{aligned}$$

Let $\Omega :=(-\infty ,-2] \cup [2,+\infty )$ and consider the problem (P) with $m=1$ by

$$\begin{aligned} \underset{x\in \mathbb {R}}{\inf }\big \{ f(x) \, \mid \,x\in \Omega ,\,h_1(x) \le 0 \big \}. \end{aligned}$$

(EP2)

In this case, we see that the feasible set S of problem (EP2) is given by

$$\begin{aligned} S:=(-\infty ,-3] \cup [3,+\infty ) \end{aligned}$$

and that $f(x) \ge -1$ for every $x\in S$, which shows that $\textrm{inf} (EP2) =-1$. Moreover, the extended tangency variety at infinity $\Lambda ^\infty _r (EP2)$ with respect to $r>0$ for S of (EP2) is given by

$$\begin{aligned} \Lambda ^\infty _r (EP2):=&\big \{x \in S \setminus I\!\!B_r\, \mid \, \exists (\mu _0,\mu _1)\in \mathbb {R}_+^2,\, \exists \lambda \in \mathbb {R},\, (\mu _0,\mu _1,\lambda )\ne 0,\\&\quad \quad \quad \quad \quad \quad \,\,\,0 \in \mu _0 \partial f(x) +\mu _1 \partial h_1(x)+ \lambda x +N(x;\Omega ),\, \mu _1h_1(x)=0\big \}\\ =&((-\infty ,-3] \cup [3,+\infty ))\setminus [-r,r]. \end{aligned}$$

Moreover, it can be checked that $\Lambda ^\infty (EP2)=(-\infty ,-3] \cup [3,+\infty )$. Choosing any sequence $\{x_k\}_{k\in \mathbb {N}}\subset \Lambda ^\infty (EP2)$ with $x_k\rightarrow -\infty$ as $k\rightarrow \infty$, one gets $f(x_k)=1-x_k \rightarrow +\infty$ as $k\rightarrow \infty .$ For any $\alpha \in [-1,1]$, let $\cos a =\alpha$. We pick a sequence $\{x_k\}_{k \in \mathbb {N}}$ with $x_k:=a +k2 \pi +\dfrac{1}{k}$. Then, there is $n_0 \in \mathbb {N}$ satisfying $\{x_k\}_{k \ge n_0} \subset \Lambda ^\infty (EP2).$ It is clear that

$$\begin{aligned} x_k\rightarrow +\infty \text { and } f(x_k)= \cos x_k \rightarrow \alpha \text { as } k \rightarrow \infty . \end{aligned}$$

So, $T^\infty (EP2)=[-1,1]$, which shows that $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (EP2)\}=-1$. We can verify that the statements (ii)(b) and (iii) of Theorem 3.2 hold for $\bar{x}=x^*:=\pi ,$ and the statement (iv) holds for $r=3\pi$. Hence, we conclude by Theorem 3.2 that ${\mathcal {S}}(EP2)$ is non-empty. In fact, for this setting, we can verify directly that

$$\begin{aligned} {\mathcal {S}}(EP2)=\big \{ \pi + k2\pi \mid k=0,1,2,...\big \}. \end{aligned}$$

4 Compactness of the Solution Set

In this section, we provide geometrical conditions that ensure the non-emptiness and compactness of the solution set of problem (P).

We first present necessary and sufficient conditions based on the tangency value set, the coercivity of problem (P) and extended tangency variety at infinity for the non-emptiness and compactness of the solution set to the problem (P).

Theorem 4.1

Consider the problem (P). The following statements are valid.

• (i) The following conditions are equivalent:

  • (a) $-\infty < \inf (P) \notin T^\infty (P)$.

  • (b) There exists $\bar{x}\in S$ such that $f(\bar{x}) < \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}$ for some $r >\Vert \bar{x}\Vert$.

  • (c) There is $r>0$ satisfying $\inf (P)<\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)$.

• (ii) If one of conditions (a), (b) and (c) holds, then ${\mathcal {S}}(P)$ is a non-empty compact set.

• (iii) If ${\mathcal {S}}(P)$ is a non-empty compact set, then the problem (P) is coercive whenever $T^\infty (P) =\emptyset .$

Proof

(i) We will justify that [(a)$\Rightarrow$(b)$\Rightarrow$(c)$\Rightarrow$(a)].

[(a) $\Rightarrow$ (b)]  Let $-\infty < \inf (P) \notin T^\infty (P)$. If $T^\infty (P)=\emptyset ,$ then we imply from Theorem 3.1(ii) that the problem (P) is coercive, i.e., $f(x_k) \rightarrow +\infty$ for any sequence $\{x_k\}_{k \in \mathbb {N}} \subset S$ tending to infinity, or equivalently, the set $S_{lev}(\lambda )$ is compact for every $\lambda \in \mathbb {R}$. Taking arbitrarily $\bar{x}\in S$ and $\lambda \in \mathbb {R}$ with $f(\bar{x}) <\lambda$, we consider $r >\Vert \bar{x}\Vert$ satisfying $S_{lev}(\lambda ) \subset I\!\!B_r$. Then

$$\begin{aligned} f(\bar{x}) < f(x) \text { for all } x \in S \setminus I\!\!B_r \end{aligned}$$

and therefore $f(\bar{x}) < \inf \big \{f(x) \mid x \in S \setminus I\!\!B_r\big \}$. Moreover, because of $\Lambda ^\infty _r (P)\subset S \setminus I\!\!B_r,$ we have

$$\begin{aligned} f(\bar{x}) < \inf \big \{f(x) \mid x \in S \setminus I\!\!B_r\big \}\le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}. \end{aligned}$$

Next, consider the case of $T^\infty (P)\ne \emptyset .$ Due to Proposition 2.1(i), one has $\inf (P) \le \lambda _{\min }=\inf \big \{\lambda \mid \lambda \in T^\infty (P)\big \}$. By the assumption $\inf (P) \notin T^\infty (P)$, it holds that $\inf (P) <\lambda _{\min }$. Let $\lambda \in \mathbb {R}$ satisfy $\inf (P)<\lambda <\lambda _{\min }$. Certainly, there exist some $\tilde{x}\in S$ with $f(\tilde{x})=\lambda$ and moreover, $S_{lev}(\lambda )$ is a non-empty compact set by virtue of Proposition 2.1(iv). Then, there exists $r> \Vert \tilde{x}\Vert$ such that ${S_{lev}}(\lambda ) \subset I\!\!B_r.$ It follows from $S_{lev}(\lambda ) \cap (S\setminus I\!\!B_r) =\emptyset$ that $S_{lev}(\lambda ) \cap \Lambda ^\infty _r (P) =\emptyset$ as $\Lambda ^\infty _r (P)\subset S\setminus I\!\!B_r$. This yields $f(\tilde{x})=\lambda <f(x)$ for all $x \in \Lambda ^\infty _r (P)$. Thus, we conclude that

$$\begin{aligned} \lambda \le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}. \end{aligned}$$

Now, pick $\bar{\lambda }\in \mathbb {R}$ with $\inf (P)<\bar{\lambda }<\lambda$. There is $\bar{x}\in S$ satisfying $f(\bar{x})=\bar{\lambda }$ and $r >\Vert \bar{x}\Vert$. Clearly, $f(\bar{x})=\bar{\lambda }<\lambda \le \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}.$ This claims that the statement (b) is valid.

[(b) $\Rightarrow$ (c)]  Assume that $\bar{x}\in S$ such that $f(\bar{x}) < \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \}$ for some $r >\Vert \bar{x}\Vert$. It should be noted that

$$\begin{aligned} \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (P)\big \} \le \underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x), \end{aligned}$$

which implies that

$$\begin{aligned} \inf (P) \le f(\bar{x}) < \underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x). \end{aligned}$$

This asserts that (c) holds.

[(c) $\Rightarrow$ (a)]  Assume that there is $r>0$ satisfying $\inf (P)<\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x).$ By the definition of $\inf (P),$ there exists a sequence $\{x_k\}_{k\in \mathbb {N}}\subset S$ such that $f(x_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty .$ Assume that $\{x_k\}_{k\in \mathbb {N}}$ is unbounded, i.e.,

$$\begin{aligned} \Vert x_k\Vert \rightarrow \infty \text { and } f(x_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty . \end{aligned}$$

For each $k \in \mathbb {N}$, put $\tilde{S}_k:=\{ x\in S \mid \, \Vert x\Vert ^2=\Vert x_k\Vert ^2\}$ and observe that $\tilde{S}_k$ is a non-empty compact set due to $x_k \in \tilde{S}_k.$ We argue similarly as in the proof of Proposition 2.1(ii) to find $x^*_k \in \tilde{S}_k$ so that $x^*_k$ is an optimal solution of the following problem:

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert x_k\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\}, \end{aligned}$$

and moreover $x^*_k \in \Lambda ^\infty (P)$ for all $k \in \mathbb {N}$, where $\Lambda ^\infty (P)$ is defined as in (2.1).

Note that $\inf (P)\le f(x^*_k) \le f(x_k)$ for all $k \in \mathbb {N}$ because of $x_k \in \tilde{S}_k.$ Let $n_0 \in \mathbb {N}$ satisfy $\Vert x^*_k\Vert >r$ for all $k \ge n_0$. As a result, we get a sequence $\{x^*_k\}_{k \ge n_0} \subset \Lambda ^\infty _r (P)$ such that

$$\begin{aligned} \Vert x^*_k\Vert \rightarrow \infty \text { and } f(x^*_k) \rightarrow \inf (P) \text { as } k \rightarrow \infty , \end{aligned}$$

and hence $\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)=\inf (P)$. This contradicts the fact that $\inf (P)<\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x).$ Consequently, $\{x_k\}_{k\in \mathbb {N}}$ must be bounded. Then, there exists a subsequence $\{x_{k_l}\}_{k_l\in \mathbb {N}}$ of $\{x_k\}_{k\in \mathbb {N}}$ converging to $\bar{x}\in S$. By the continuity of f on S, we arrive at $f(x_{k_l}) \rightarrow f(\bar{x})$ as $k_l \rightarrow \infty$. We also have $f(x_{k_l}) \rightarrow \inf (P) \text { as } k_l \rightarrow \infty .$ So, $f(\bar{x})=\inf (P)$, which shows that $\inf (P) >-\infty$.

It remains to prove that $\inf (P) \notin T^\infty (P)$. Indeed, if $\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)=+\infty ,$ then it holds that for every sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty _r(P),$ we have

$$\begin{aligned} \Vert x_k \Vert \rightarrow \infty \text { and } f(x_k) \rightarrow +\infty \, \,\textrm{as} \,\,k \rightarrow \infty . \end{aligned}$$

(4.1)

Consider an arbitrary sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty (P)$ such that $\Vert x_k \Vert \rightarrow \infty$ as $k \rightarrow \infty$ and observe that $x_k \in \Lambda ^\infty _r(P)$ for all k large enough. Therefore, we conclude in this case that the sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty (P)$ satisfies (4.1), which in its turn entails by definition that $T^\infty (P)=\emptyset$. So, (a) is valid for this case. Now, if $\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)<+\infty ,$ then it is clear that $\lambda _{\min }=\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x).$ Hence, by $\inf (P)<\underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)$, one has $\inf (P) < \lambda _{\min }$ and so the assertion (a) is also valid. Consequently, [(a)$\Leftrightarrow$(b)$\Leftrightarrow$(c)] has been justified.

(ii)   By (i), it suffices to show that if (a) holds, then ${\mathcal {S}}(P)$ is a non-empty compact set. To see this, we let $\bar{\lambda }:= \inf (P)\in \mathbb {R}$ and justify that ${\mathcal {S}}(P)=S_{lev} (\bar{\lambda })$. Indeed, for each $x^* \in {\mathcal {S}}(P)$, we have $f(x^*) =\inf (P)$, and so

$$\begin{aligned} x^* \in S_{lev}(\bar{\lambda }):= \{ x \in S \mid f(x) \le \bar{\lambda }\}. \end{aligned}$$

Conversely, take $x^* \in S_{lev}(\bar{\lambda })$. Then, $x^* \in S$ and $f(x^*) \le \inf (P)$, which implies that $f(x^*) =\inf (P)$. This means that $x^* \in {\mathcal {S}}(P)$ and so ${\mathcal {S}}(P)=S_{lev} (\bar{\lambda })$.

If $T^\infty (P)= \emptyset$, then ${\mathcal {S}}(P) \ne \emptyset$ by Theorem 3.2. Moreover, by $\inf (P)>-\infty$ and $T^\infty (P)= \emptyset$, we get from Theorem 3.1(ii) that $S_{lev} (\bar{\lambda })$ is a compact set. Hence, ${\mathcal {S}}(P)$ is a non-empty compact set.

In the case of $T^\infty (P)\ne \emptyset$, we assert by Proposition 2.1(i) that $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\}\ge \inf (P)$. By (a), we conclude that $\inf (P) < \lambda _{\min }$ because of $\lambda _{\min } \in T^\infty (P).$ Choosing $\inf (P)<\lambda < \lambda _{\min }$, we consider the sublevel set of f at $\lambda$ by

$$\begin{aligned} S_{lev}(\lambda ):= \{ x \in S \mid f(x) \le \lambda \}. \end{aligned}$$

From Proposition 2.1(iv), it shows that $S_{lev}(\lambda )$ is compact. Moreover, we derive from the definition of $\inf (P)$ that there exists $x_\epsilon \in S$ such that $f(x_\epsilon )<\lambda$ for $\epsilon :=\lambda -\inf (P)>0.$ Therefore, $x_\epsilon \in S_{lev}(\lambda )\ne \emptyset ,$ which entails that the condition (ii)(b) of Theorem 3.2 is satisfied. Thus, ${\mathcal {S}}(P) \ne \emptyset$ by Theorem 3.2. Note that $S_{lev} (\bar{\lambda })$ is compact due to $S_{lev} (\bar{\lambda }) \subset S_{lev} (\lambda ).$ Consequently, ${\mathcal {S}}(P)$ is a non-empty compact set.

(iii)   Assume that ${\mathcal {S}}(P)$ is a non-empty compact set. Clearly, $\inf (P) >-\infty .$ If $T^\infty (P)= \emptyset$, then it is followed by Theorem 3.1(ii) that the problem (P) is coercive. The proof of the theorem is complete. $\square$

The following example shows how one can employ Theorem 4.1 to verify the non-emptiness and the compactness of the optimal solution set of a nonsmooth optimization problem.

Example 4.1

Let $f:\mathbb {R}^2 \rightarrow \mathbb {R}$ and $h_1:\mathbb {R}^2 \rightarrow \mathbb {R}$ be defined respectively by

$$\begin{aligned} f(x)&:={\left\{ \begin{array}{ll} e^{x_1+x_2}+1\,\,\,\,& \text {if}\,\,x_1+x_2 < 0, \\ |x_1+x_2-2|\,\,\,\,& \text {if}\,\,x_1+x_2\ge 0\end{array}\right. }\\ &\text { and } h_1(x):=x^2_1+x^3_2-2 \text { for } x:=(x_1,x_2) \in \mathbb {R}^2. \end{aligned}$$

Let $\Omega :=\{x \in \mathbb {R}^2 \mid 2|x_1|+x_2 \le 0\} \cup \mathbb {R}^2_+$. Consider the problem (P) with $m=1$ as follows:

$$\begin{aligned} \underset{x\in \mathbb {R}^2}{\inf }\big \{ f(x) \mid x\in \Omega ,\,h_1(x) \le 0\big \}. \end{aligned}$$

(EP3)

By direct calculation, we obtain the feasible set S of (EP3) (see Fig. 1a) as

$$\begin{aligned} S&:=\{x\in \Omega \mid \, h_1(x) \le 0\}=\{x \in \mathbb {R}^2 \mid 2|x_1|+x_2 \le 0\}\\ &\cup \big (\{x \in \mathbb {R}^2 \mid x^2_1+x^3_2-2 \le 0\} \cap \mathbb {R}^2_+\big ). \end{aligned}$$

Consider $r\ge \dfrac{3}{2}$ and take any $x \in S\setminus I\!\!B_r$. One can check that $x_1+x_2 <0$ and $h_1(x)<0.$ For any $x \in S\setminus I\!\!B_r$, we compute that

$$\begin{aligned}&\partial f(x)=(e^{x_1+x_2},e^{x_1+x_2}),\,\partial h_1(x)=(2x_1,3x_2^2),\\&N(x;\Omega )=\{(0,0)\} \text { for } x \in \textrm{int}(S \setminus I\!\!B_r),\\&N(x;\Omega )=\{(-2a,a) \mid a \ge 0\} \text { for } x=(x_1,2x_1) \in S\setminus I\!\!B_r \text { and }x_1 \le 0,\\&N(x;\Omega )=\{(2a,a) \mid a \ge 0\} \text { for } x=(x_1,-2x_1) \in S\setminus I\!\!B_r \text { and }x_1 \ge 0. \end{aligned}$$

We also calculate the extended tangency variety at infinity with respect to r for S (see Fig. 1b), which is given by

$$\begin{aligned} \Lambda ^\infty _r (EP3):=&\big \{x \in S \setminus I\!\!B_r\, \mid \, \exists (\mu _0,\mu _1)\in \mathbb {R}_+^2,\, \exists \lambda \in \mathbb {R},\, (\mu _0,\mu _1,\lambda )\ne 0,\\&\quad \quad \quad \quad \quad \quad \,\,\,0 \in \mu _0 \partial f(x) +\mu _1 \partial h_1(x)+ \lambda x +N(x;\Omega ),\, \mu _1h_1(x)=0\big \}\\ =&\{(x_1,-2|x_1|) \mid x_1 \in \mathbb {R}\} \setminus \{x\in \mathbb {R}^2 \mid x^2_1 +x^2_2 \le r^2\}. \end{aligned}$$

Fig. 1

(a) The feasible set S is shaded in blue, (b) The extended tangency set at infinity $\Lambda ^\infty _r(EP3)$ is in red

We observe by the definition of $T^\infty (EP3)$ that one can consider sequences in $\Lambda ^\infty _r (EP3)$ (instead of in $\Lambda ^\infty (EP3)$) for $r\ge \frac{3}{2}$. Pick any sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty _r (EP3)$ such that $\Vert x_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$. If $x_k:=(x_{1k},2x_{1k})$ and $x_{1k} < 0$, then we observe that $x_{1k} \rightarrow -\infty$ due to $\Vert x_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$. In this case, one has $f(x_k)=e^{3x_{1k}}+1 \rightarrow 1$ as $k \rightarrow \infty$. For the case of $x_k:=(x_{1k},-2x_{1k})$ and $x_{1k} > 0$, we obtain $f(x_k)=e^{-x_{1k}}+1 \rightarrow 1$ as $k \rightarrow \infty$. So, in both cases, we have

$$\begin{aligned} T^\infty (EP3)=\{1\}=\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (EP3)\}. \end{aligned}$$

Moreover, $f(x) \ge 0$ for all $x \in S$ and $f(x^*) = 0$ at $x^*:=(1,1)$. This tells us $\inf (EP3)=0$ and so $-\infty <\inf (EP3)=0 \notin T^\infty (EP3),$ which shows that the condition (a) of Theorem 4.1 is satisfied.

Now, take $r> \frac{3}{2}>\Vert \bar{x}\Vert$ for $\bar{x}:=(3-\sqrt{3},-1+\sqrt{3})$. Clearly, $f(\bar{x})=0$ and moreover, one has $x_1+x_2<0$ and $f(x)=e^{x_1+x_2}+1 >1$ for all $x \in S \setminus I\!\!B_r$. Therefore, $\inf \big \{f(x) \mid x \in \Lambda ^\infty _r (EP3)\big \}=1>0$ due to the fact that $\Lambda ^\infty _r (EP3) \subset S \setminus I\!\!B_r.$ Thus, $f(\bar{x}) < \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (EP3)\big \}$. Moreover, we get $\inf (EP3)<\underset{x \in \Lambda ^\infty _r (EP3),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)=1$. So, the conditions (b) and (c) of Theorem 4.1 hold and we conclude by Theorem 4.1 that the optimal solution set ${\mathcal {S}}(EP3)$ of problem (EP3) is a non-empty compact set. In fact, we can calculate directly that ${\mathcal {S}}(EP3)=\{(x_1,2-x_1) \mid 1\le x_1\le 3-\sqrt{3}\}$ for this setting.

We observe that the converse statements of Theorem 4.1(ii) are not necessarily valid as the next example illustrates.

Example 4.2

Let $f:\mathbb {R}\rightarrow \mathbb {R}$ and $h_1:\mathbb {R}\rightarrow \mathbb {R}$ be defined respectively by

$$\begin{aligned} f(x):={\left\{ \begin{array}{ll} e^{-x}\,\,\,\,& \text {if}\,\,x \ge 0, \\ |x+1|\,\,\,\,& \text {if}\,\,x<0\end{array}\right. }\quad \text { and } h_1(x):=-x-5 \text { for } x \in \mathbb {R}. \end{aligned}$$

Let $\Omega :=[-10,+\infty )$ and consider the problem (P) with $m=1$ as follows:

$$\begin{aligned} \underset{x\in \mathbb {R}}{\inf }\big \{ f(x) \, \mid \,x\in \Omega ,\,h_1(x) \le 0\big \}. \end{aligned}$$

(EP4)

In this setting, the constraint set S of (EP4) is calculated by

$$\begin{aligned} S:=\{x\in \Omega \mid \, h_1(x) \le 0\}=[-5,+\infty ) \end{aligned}$$

and the extended tangency variety for S is given by

$$\begin{aligned} \Lambda ^\infty (EP4):= \big \{&x \in S\, \mid \, \exists (\mu _0,\mu _1)\in \mathbb {R}_+^2,\, \exists \lambda \in \mathbb {R},\, (\mu _0,\mu _1,\lambda )\ne 0, \\&0 \in \mu _0 \partial f(x) + \mu _1\partial h_1(x) + \lambda x +N(x;\Omega ),\, \mu _1 h_1(x)=0\big \} =[-5,+\infty ). \end{aligned}$$

Pick any sequence $\{x_k\}_{k \in \mathbb {N}} \subset \Lambda ^\infty (EP4)$ such that $x_k \rightarrow +\infty$ as $k \rightarrow \infty$. Then, $f(x_k) =e^{-x_k}\rightarrow 0$ as $k \rightarrow \infty$. So, $T^\infty (EP4)=\{0\}.$ We observe that ${\mathcal {S}}(EP4)=\{-1\}$ and $\inf (EP4)=0$. By direct computation, we have

$$\begin{aligned} \inf \big \{f(x) \mid x \in \Lambda ^\infty _r (EP4)\big \}=\underset{x \in \Lambda ^\infty _r (EP4),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)=\inf (EP4) \end{aligned}$$

for any $r>0$. In conclusion, ${\mathcal {S}}(EP4)$ is a non-empty compact set, while none of the requirements (a), (b) and (c) of Theorem 4.1 holds.

We now provide characterizations by means of the sublevel sets, the coercivity and the tangency values at infinity for the non-emptiness and compactness of the solution set to the problem (P).

Theorem 4.2

Consider the problem (P). The following conditions are equivalent:

• (i) ${\mathcal {S}}(P)$ is a non-empty compact set.

• (ii) There exists $\bar{\lambda }\in \mathbb {R}$ satisfying $S_{lev}(\bar{\lambda })$ is a non-empty compact set.

• (iii) One of the following statements holds but never both:

  • (a) The problem (P) is coercive.

  • (b) $\inf (P)>-\infty$ and either $\lambda _{\min }>\inf (P)$ or there exist $r> 0$ and $\bar{x}\in S$ satisfying $f(\bar{x})= \lambda _{\min }<f(x)$ for all $x\in \Lambda ^\infty _r (P),$ where $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\}$.

• (iv) There exists $r>0$ such that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le \underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x) \end{aligned}$$

(4.2)

and there does not exist a sequence $\{u_k\}_{k \in \mathbb {N}} \subset \Lambda ^\infty _r (P)$ satisfying

$$\begin{aligned} f(u_k)&=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}\nonumber \\ &\text { and } \Vert u_k\Vert \rightarrow \infty \text { as } k \rightarrow \infty , \end{aligned}$$

(4.3)

where “$\min$” indicates that the corresponding relaxation problem attains its optimal solutions.

Proof

Observe first that [(i) $\Leftrightarrow$ (ii)].

[(i) $\Rightarrow$ (iii)] Suppose that ${\mathcal {S}}(P)$ is a non-empty compact set. Clearly, $\inf (P)>-\infty$. If the problem (P) is coercive, then (a) holds. We now assume that (P) is not coercive. Then, by Theorem 3.1(ii), we have $T^\infty (P)\ne \emptyset$. Therefore, $\lambda _{\min }:=\min \{\lambda \mid \lambda \in T^\infty (P)\} \ge \inf (P)$ because of Proposition 2.1(i).

If $\lambda _{\min }>\inf (P)$, then we have the first statement of (b). Otherwise, $\lambda _{\min }=\inf (P)$. Then, $S_{lev}(\lambda _{\min })={\mathcal {S}}(P)$ and so $S_{lev}(\lambda _{\min })$ is a non-empty compact set. Choosing $\bar{x}\in {\mathcal {S}}(P)$ and invoking Proposition 2.1(v), there exists $r>0$ such that $f(\bar{x})=\lambda _{\min }<f(x)$ for all $x\in \Lambda ^\infty _r (P)$, which concludes that (b) holds.

[(iii) $\Rightarrow$ (iv)] First, let (a) hold. Then, we get by Theorem 3.1(ii) that $S_{lev}(\lambda )$ is compact for any $\lambda \in \mathbb {R}$. Choosing any $\bar{x}\in S$ and setting $\bar{\lambda }:= f(\bar{x})$, we conclude that $S_{lev}(\bar{\lambda })$ is a non-empty compact set because of $\bar{x}\in S_{lev}(\bar{\lambda })$. According to the implication [(ii) $\Rightarrow$ (i)], it holds that ${\mathcal {S}}(P)$ is a non-empty compact set. By the compactness of $S_{lev}(\bar{\lambda })$, one can find $r >0$ such that $S_{lev}(\bar{\lambda }) \subset I\!\!B_r$. Take $x^* \in {\mathcal {S}}(P)$. By the continuity of f on the non-empty compact set $S \cap I\!\!B_r$, the following problem

$$\begin{aligned} \inf \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \end{aligned}$$

has an optimal solution. Furthermore, because of $x^* \in S \cap I\!\!B_r$, we obtain

$$\begin{aligned} \inf (P) \le \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le f(x^*)=\inf (P), \end{aligned}$$

which implies that

$$\begin{aligned} \min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}= \inf (P). \end{aligned}$$

(4.4)

So, the inequality (4.2) is satisfied by virtue of $\Lambda ^\infty _r(P)\subset S.$

To justify (4.3), assume on the contrary that there exists $\{u_k\}_{k \in \mathbb {N}} \subset \Lambda ^\infty _r (P)$ such that $f(u_k)=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}$ for all $k \in \mathbb {N}$ and $\Vert u_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$. However, we get by (4.4) that $f(u_k)= \inf (P)$ for $k\in \mathbb {N}$ and so $\{u_k\}_{k \in \mathbb {N}} \subset {\mathcal {S}}(P)$. This contradicts the compactness of ${\mathcal {S}}(P)$. Hence, (iv) has been justified under the validation of (a).

Now, let (b) hold. Then, we have $\inf (P)>-\infty$. Note under the assumption (iii) that the problem (P) is not coercive. Then, by Theorem 3.1(ii), we have $T^\infty (P)\ne \emptyset$. Consider the following two cases:

Case 1: Let $\lambda _{\min }>\inf (P)$. Pick a number $\bar{\lambda }\in \mathbb {R}$ satisfying $\lambda _{\min }> \bar{\lambda }>\inf (P)$ and consider the sublevel set $S_{lev}(\bar{\lambda })$. For $\epsilon :=\bar{\lambda }-\inf (P)>0,$ we can find $\bar{x}\in S$ such that $f(\bar{x})<\inf (P)+\epsilon =\bar{\lambda }$ and therefore $\bar{x}\in S_{lev}(\bar{\lambda }).$ We get by Proposition 2.1(iv) that $S_{lev}(\bar{\lambda })$ is compact and so it is a non-empty compact set. We proceed with the same arguments as in the proof of (a) and obtain the desired conclusion of (iv).

Case 2: Assume that there exist $r>0$ and $\bar{x}\in S$ satisfying $f(\bar{x})= \lambda _{\min }<f(x)$ for all $x\in \Lambda ^\infty _r (P).$ We claim that $\Vert \bar{x}\Vert \le r$. To see this, we suppose on the contrary that $\Vert \bar{x}\Vert > r$. Letting $\tilde{S}:= \{x\in S \mid \Vert x\Vert ^2=\Vert \bar{x}\Vert ^2\}$, we observe that $\tilde{S}$ is a non-empty set due to $\bar{x}\in \tilde{S}.$ As f is continuous on the compact set $\tilde{S}$, we can pick $\tilde{x}\in \tilde{S}$ such that $f(\tilde{x})\le f(x)\, \text{ for } \text{ all } \, x\in \tilde{S}$. Equivalently, $\tilde{x}$ is an optimal solution of the following problem

$$\begin{aligned} \inf \{ f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2-\Vert \bar{x}\Vert ^2=0,\, h_i(x)\le 0, \, i=1,...,m\}. \end{aligned}$$

By Lemma 2.1, there exists $(\mu _0,...,\mu _{m+1}) \in \mathbb {R}^{m+2}\setminus \{0\}$ with $\mu _i \ge 0,\, i=0,...,m,$ such that

$$\begin{aligned}&0\in \mu _0 \partial f(\tilde{x})+\sum \limits _{i=1}^m\mu _i \partial h_i(\tilde{x}) + 2\mu _{m+1} \tilde{x} + N(\tilde{x};\Omega ), \\&\mu _i h_i(\tilde{x})=0, \, i=1,...,m. \end{aligned}$$

Since $\Vert \tilde{x}\Vert =\Vert \bar{x}\Vert > r$, we get by Definition 2.1(ii) that $\tilde{x} \in \Lambda ^\infty _r(P)$. Clearly, $f(\tilde{x}) \le f(\bar{x})$, which results in a contradiction as $f(\bar{x})<f(x)$ for all $x\in \Lambda ^\infty _r (P).$

Because of $T^\infty (P)\ne \emptyset$, we get by Proposition 2.1(i) that $\lambda _{\min }\ge \inf (P)$ and by Proposition 2.1(v) that the sublevel set $S_{lev}( \lambda _{\min })$ is compact. This also entails that ${\mathcal {S}} (P)$ is compact inasmuch as ${\mathcal {S}} (P)\subset S_{lev}( \lambda _{\min }).$ Moreover, due to $\bar{x}\in S$ and $f(\bar{x})= \lambda _{\min }$, it implies that $\bar{x}\in S_{lev}( \lambda _{\min })$ and so $S_{lev}( \lambda _{\min })$ is a non-empty compact set. If $\lambda _{\min }>\inf (P)$, then we conclude as in Case 1 that the statement (iv) is satisfied.

Otherwise, i.e., $\lambda _{\min }=\inf (P)$. Then, one has $\bar{x}\in {\mathcal {S}}(P)$ due to $f(\bar{x})= \lambda _{\min }=\inf (P).$ Now, Theorem 3.2 shows that (4.2) holds. To justify (4.3), assume on the contrary that there exists $\{u_k\}_{k \in \mathbb {N}} \subset \Lambda ^\infty _r (P)$ such that $f(u_k)=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}$ for all $k \in \mathbb {N}$ and $\Vert u_k\Vert \rightarrow \infty$ as $k \rightarrow \infty$. As $\Vert \bar{x}\Vert \le r$ with $\bar{x}\in S$ and $f(\bar{x})=\lambda _{\min }$, we see that

$$\begin{aligned} f(u_k)=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}\le f(\bar{x})=\inf (P), \end{aligned}$$

which implies that $f(u_k)= \inf (P)$ for $k\in \mathbb {N}$ and so $\{u_k\}_{k \in \mathbb {N}} \subset {\mathcal {S}}(P)$. This contradicts the compactness of ${\mathcal {S}}(P)$. Consequently, (iv) has been justified.

[(iv) $\Rightarrow$ (i)] Let (iv) hold. Under the validation of (4.2), we assert by Theorem 3.2 that the set ${\mathcal {S}}(P)$ is non-empty and there exists $\bar{x}\in S$ with $\Vert \bar{x}\Vert \le r$ such that

$$\begin{aligned} f(\bar{x})=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}. \end{aligned}$$

We claim that ${\mathcal {S}}(P)$ is bounded. If this is not the case, we can find a sequence $\{u^*_k\}_{k \in \mathbb {N}} \subset {\mathcal {S}}(P)$ satisfying

$$\begin{aligned} f(u^*_k)=\inf (P) \text { and } \Vert u^*_k\Vert \rightarrow \infty \text { as } k \rightarrow \infty . \end{aligned}$$

For each $k \in \mathbb {N}$, employing Lemma 2.1, there exists $(\mu _0,...,\mu _m) \in \mathbb {R}^{m+1}_+\setminus \{0\}$ such that

$$\begin{aligned}&0\in \mu _0 \partial f(u^*_k)+\sum \limits _{i=1}^m\mu _i \partial h_i(u^*_k) + N(u^*_k;\Omega ), \\&\mu _i h_i(u^*_k)=0, \, i=1,...,m. \end{aligned}$$

When $k\in \mathbb {N}$ is large enough, we have $\Vert u^*_k\Vert > r$, which follows that $u^*_k \in \Lambda ^\infty _r (P)$ for such large k. Therefore,

$$\begin{aligned} \underset{x \in \Lambda ^\infty _r (P),\,\Vert x\Vert \rightarrow \infty }{\liminf }f(x)\le \liminf _{k\rightarrow \infty } f(u^*_k)=\inf (P). \end{aligned}$$

This together with (4.2) entails that

$$\begin{aligned} f(u^*_k)\le f(\bar{x})=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\} \le \inf (P), \end{aligned}$$

and so $f(u^*_k)=\min \{f(x) \mid x\in \Omega ,\, \Vert x\Vert ^2\le r^2,\, h_i(x)\le 0, \, i=1,...,m\}$ for such large k. Thus, we arrive at a contradiction to (4.3). So, ${\mathcal {S}}(P)$ is a non-empty compact set. The proof of the theorem is complete. $\square$

Finally in this section, we present an example to illustrate how we can employ different conditions in Theorem 4.2 to verify the non-empty compactness of a nonsmooth optimization problem.

Example 4.3

Let $f:\mathbb {R}^2 \rightarrow \mathbb {R}$ and $h_1:\mathbb {R}^2 \rightarrow \mathbb {R}$ be given respectively by

$$\begin{aligned} f(x)&:={\left\{ \begin{array}{ll} |x_1|+x^2_2\,\,\,\,& \text {if}\,\,x_1 \in [-2,2] , \\ \dfrac{4}{|x_1|}+x_2^2\,\,\,\,& \text {if}\,\,x_1 \notin [-2,2]\end{array}\right. }\\ &\text { and } h_1(x):=-x_1+e^{x_2}-1 \text { for } x:=(x_1,x_2) \in \mathbb {R}^2. \end{aligned}$$

Let $\Omega :=\mathbb {R}^2_+$ and consider the problem (P) with $m=1$ as follows:

$$\begin{aligned} \underset{x\in \mathbb {R}}{\inf }\big \{ f(x) \, \mid \,x\in \Omega ,\,h_1(x) \le 0 \big \}. \end{aligned}$$

(EP5)

In this case, the feasible set S of problem (EP5) is $S:=\{x \in \mathbb {R}^2_+ \mid -x_1+e^{x_2}-1 \le 0\}.$ Considering $r > 3$ and picking arbitrarily $x \in S\setminus I\!\!B_r$, we can verify that $x_1 \notin [-2,2]$ with $x_1>0$, and that

$$\begin{aligned}&\partial f(x)=(-\frac{4}{x_1^2},2x_2),\,\partial h_1(x)=(-1,e^{x_2}),\\&N(x;\Omega )=\{(0,0)\} \text { for } x=(x_1,x_2) \in S \setminus I\!\!B_r \text { and } x_2>0,\\&N(x;\Omega )=\{(0,a) \mid a \le 0\} \text { for } x=(x_1,0) \in S\setminus I\!\!B_r. \end{aligned}$$

Now, we compute the extended tangency variety at infinity $\Lambda ^\infty _r (EP5)$ with respect to r for S of (EP5), which is given as

$$\begin{aligned} \Lambda ^\infty _r (EP5):=&\big \{x \in S \setminus I\!\!B_r\, \mid \, \exists (\mu _0,\mu _1)\in \mathbb {R}_+^2,\, \exists \lambda \in \mathbb {R},\, (\mu _0,\mu _1,\lambda )\ne 0,\\&\quad \quad \quad \quad \quad \quad \,\,\,0 \in \mu _0 \partial f(x) +\mu _1 \partial h_1(x)+ \lambda x +N(x;\Omega ),\, \mu _1h_1(x)=0\big \}\\ =&\big \{(\root 3 \of {x_1},0) \mid x_1 > 0\big \}\setminus \{x \in \mathbb {R}^2 \mid x_1^2+x_2^2 \le r^2\}. \end{aligned}$$

   It can be justified that $f(\bar{x}) = 0$ if and only if $\bar{x}=(0,0)$ and $f(x) > 0$ for every $x \in S\setminus \{ \bar{x}\}$. For $\bar{\lambda }:=0$, it is obvious that the sublevel set $S_{lev}(\bar{\lambda })=\{(0,0)\}$ is a non-empty compact set. Therefore, the statement (ii) of Theorem 4.2 holds.

Note by the definition of $T^\infty (EP5)$ that we can consider sequences in $\Lambda ^\infty _r (EP5)$ (instead of in $\Lambda ^\infty (EP5)$) for $r>3$. For $r> 3$, we pick any sequence $\{x_k\}_{k\in \mathbb {N}} \subset \Lambda ^\infty _r (EP5)$, which implies that $x_k:=(\root 3 \of {x_{1k}},0)$ with $\root 3 \of {x_{1k}}> r$ for all $k \in \mathbb {N}$. If $x_k \rightarrow \infty$ as $k \rightarrow \infty$, then $f(x_k)=\dfrac{4}{\root 3 \of {x_{1k}}}\rightarrow 0$ as $k \rightarrow \infty$. So, $T^\infty (EP5)=\{0\}=\lambda _{\min }$. Since $f(\bar{x})= \lambda _{\min }<f(x)$ for all $x\in \Lambda ^\infty _r (P) \subset S$, it entails that (iii)(b) of Theorem 4.2 holds. Similarly, we can check that (iv) of Theorem 4.2 also holds. Now, we can employ Theorem 4.2 to conclude that the optimal solution set ${\mathcal {S}}(EP5)$ is a non-empty compact set. In fact, we can calculate that ${\mathcal {S}}(EP5)=\{(0,0)\} \text { and } \inf (EP5)=0$ for this setting.

5 Conclusions

In this paper, we have characterized the non-emptiness and compactness of the global optimal solution set for a nonconvex and nonsmooth optimization problem. We have examined geometrical conditions that ensure the finiteness of the optimal value of the underlying problem. We have also investigated related properties including the coercivity of the objective and the compactness of sublevel sets for ensuring the existence, non-emptiness and compactness of the solution set.

It would be worthwhile to examine the advantages and limitations of our conditions and results compared to those obtained by means of asymptotic analysis (e.g., [12, 26]).

It would also be interesting to see how we can provide numerical schemes that based on the obtained characterizations to verify the existence of the optimal solutions and the compactness of the optimal solution set to the underlying optimization problem. Moreover, analyzing and developing the obtained results of the solution existence and its related properties for a more general class of vector or set-valued optimization problems is worth further study.