A one-layer recurrent neural network for constrained nonconvex optimization

Abstract

In this paper, a one-layer recurrent neural network is proposed for solving nonconvex optimization problems subject to general inequality constraints, designed based on an exact penalty function method. It is proved herein that any neuron state of the proposed neural network is convergent to the feasible region in finite time and stays there thereafter, provided that the penalty parameter is sufficiently large. The lower bounds of the penalty parameter and convergence time are also estimated. In addition, any neural state of the proposed neural network is convergent to its equilibrium point set which satisfies the Karush–Kuhn–Tucker conditions of the optimization problem. Moreover, the equilibrium point set is equivalent to the optimal solution to the nonconvex optimization problem if the objective function and constraints satisfy given conditions. Four numerical examples are provided to illustrate the performances of the proposed neural network.

Introduction

In this paper, the following constrained nonconvex minimization problem is considered:

$$\begin{array}{cc}\text{minimize }\hfill & f\left(x\right)\hfill \\ \text{subject to }\hfill & g_i\left(x\right)\le 0\text{,}i\in I=\{1,2,\dots ,m\}\text{,}\hfill \end{array}$$

where $x\in R^n$ is the decision vector; $f$ and $g_i,:R^n\to R(i\in I)$ are continuously differentiable functions, but not necessarily convex. The feasible region

$$\mathcal{F}=\{x\in R^n:g_i\left(x\right)\le 0,i\in I\}$$

is assumed to be a nonempty set. We denote by $\mathcal{G}$ the set of global solutions of problem (1) as,

$$\mathcal{G}=\{x\in \mathcal{F}:f\left(y\right)\ge f\left(x\right),\forall y\in \mathcal{F}\}\text{.}$$

Many problems in engineering applications can be formulated as dynamic optimization problems such as kinematic control of redundant robot manipulators (Wang, Hu, & Jiang, 1999), nonlinear model predictive control (Piche et al., 2000, Yan and Wang, 2012), hierarchical control of interconnected dynamic systems (Hou, Gupta, Nikiforuk, Tan, & Cheng, 2007), compressed sensing in adaptive signal processing (Balavoine, Romberg, & Rozell, 2012), and so on. For example, real-time motion planning and control of redundant robot manipulators can be formulated as constrained dynamic optimization problems with nonconvex objective functions for simultaneously minimizing kinetic energy and maximizing manipulability. Similarly, in nonlinear and robust model predictive control, optimal control commands have to be computed with a moving time window by repetitively solving constrained optimization problems with nonconvex objective functions for error and control variation minimization, and robustness maximization. The difficulty of dynamic optimization is significantly amplified when the optimal solutions have to be obtained in real time, especially in the presence of uncertainty. In such applications, compared with traditional numerical optimization algorithms, neurodynamic optimization approaches based on recurrent neural networks have several unique advantages. Recurrent neural networks can be physically implemented in designated hardware/firmware, such as very-large-scale integration (VLSI) reconfigurable analog chips, optical chips, graphic processing units (GPU), field programmable gate array (FPGA), digit signal processor (DSP), and so on. Recent technological advances make the design and implementation of neural networks more feasible at a more reasonable cost (Asai, Kanazawa, & Amemiya, 2003).

Since the pioneering work of Hopfield neural networks (Hopfield and Tank, 1985, Tank and Hopfield, 1986), neurodynamic optimization has achieved great success in the past three decades. For example, a deterministic annealing neural network was proposed for solving convex programming problems (Wang, 1994), a Lagrangian network was developed for solving convex optimization problems with linear equality constraints based on the Lagrangian optimality conditions (Xia, 2003), the primal–dual network (Xia, 1996), the dual network (Xia, Feng, & Wang, 2004), and the simplified dual network (Liu & Wang, 2006) were developed for solving convex optimization problems based on the Karush–Kuhn–Tucker optimality conditions, projection neural networks were developed for constrained optimization problems based on the projection method (Gao, 2004, Hu and Wang, 2007, Liu et al., 2010, Xia et al., 2002). In recent years, neurodynamic optimization approaches have been extended to nonconvex and generalized convex optimization problems. For example, a Lagrangian neural network was proposed for nonsmooth convex optimization by using the Lagrangian saddle-point theorem (Cheng et al., 2011), a recurrent neural network with global attractivity was proposed for solving nonsmooth convex optimization problems (Bian & Xue, 2013), several neural networks were developed for nonsmooth pseudoconvex or quasiconvex optimization using the Clarke’s generalized gradient (Guo et al., 2011, Hosseini et al., 2013, Hu and Wang, 2006, Liu et al., 2012, Liu and Wang, 2013). In addition, various neural networks with finite-time convergence property were developed (Bian and Xue, 2009, Forti et al., 2004, Forti et al., 2006, Xue and Bian, 2008).

Despite the enormous success, neurodynamic optimization approaches would reach their solvability limits at constrained optimization problems with unimodal objective functions and are important for global optimization with general nonconvex objective functions. Little progress has been made on nonconvex optimization in the neural network community. Instead of seeking global optimal solutions, a more attainable and meaningful goal is to design neural networks for searching critical points (e.g., Karush–Kuhn–Tucker points) of nonconvex optimization problems. Xia, Feng, and Wang (2008) proposed a neural network for solving nonconvex optimization problems with inequality constraints, whose equilibrium points correspond to the KKT points. But the condition that the Hessian matrix of the associated Lagrangian function is positive semidefinite for the global convergence is too strong. In this paper, a one-layer recurrent neural network based on an exact penalty function method is proposed for searching KKT points of nonconvex optimization problems with inequality constraints. The contribution of this paper can be summarized as follows. (1) State of the proposed neural network is convergent to the feasible region in finite time and stays there thereafter, with a sufficiently large penalty parameter; (2) the proposed neural network is convergent to its equilibrium point set; (3) any equilibrium point $x^{\ast }$ of the proposed neural network corresponds to a KKT twofold $(x^{\ast },{\lambda }^{\ast })$ of the nonconvex problem and vice versa; (4) if the objective function and the constraint functions meet one of the following conditions: (a) the objective function and the constraint functions are convex; (b) the objective function is pseudoconvex and the constraint functions are quasiconvex, then the state of the proposed network converges to the global optimal solution. If the objective function and the constraint functions are invex with respect to the same kernel, then the state of the proposed network converges to optimal solution set. Hence, the results presented in Li, Yan, and Wang (2014) can be viewed as special cases of this paper.

The remainder of this paper is organized as follows. Section  2 introduces some definitions and preliminary results. Section  3 discusses an exact penalty function. Section  4 presented a neural network model and analyzed its convergent properties. Section  5 provides simulation results. Finally, Section  6 concludes this paper.