An analysis of the velocity updating rule of the particle swarm optimization algorithm

Abstract

The particle swarm optimization algorithm includes three vectors associated with each particle: inertia, personal, and social influence vectors. The personal and social influence vectors are typically multiplied by random diagonal matrices (often referred to as random vectors) resulting in changes in their lengths and directions. This multiplication, in turn, influences the variation of the particles in the swarm. In this paper we examine several issues associated with the multiplication of personal and social influence vectors by such random matrices, these include: (1) Uncontrollable changes in the length and direction of these vectors resulting in delay in convergence or attraction to locations far from quality solutions in some situations (2) Weak direction alternation for the vectors that are aligned closely to coordinate axes resulting in preventing the swarm from further improvement in some situations, and (3) limitation in particle movement to one orthant resulting in premature convergence in some situations. To overcome these issues, we use randomly generated rotation matrices (rather than the random diagonal matrices) in the velocity updating rule of the particle swarm optimizer. This approach makes it possible to control the impact of the random components (i.e. the random matrices) on the direction and length of personal and social influence vectors separately. As a result, all the above mentioned issues are effectively addressed. We propose to use the Euclidean rotation matrices for rotation because it preserves the length of the vectors during rotation, which makes it easier to control the effects of the randomness on the direction and length of vectors. The direction of the Euclidean matrices is generated randomly by a normal distribution. The mean and variance of the distribution are investigated in detail for different algorithms and different numbers of dimensions. Also, an adaptive approach for the variance of the normal distribution is proposed which is independent from the algorithm and the number of dimensions. The method is adjoined to several particle swarm optimization variants. It is tested on 18 standard optimization benchmark functions in 10, 30 and 60 dimensional spaces. Experimental results show that the proposed method can significantly improve the performance of several types of particle swarm optimization algorithms in terms of convergence speed and solution quality.

Appendices

Appendix I

Appendix I provides the formula for all used benchmark functions. The variable pd represents the number of dimensions.

$\varvec{f}_{1}$ : Rosenbrock’s function

$$\begin{aligned} f(x)=\sum _{i=1}^{pd-1} {(100(x_{i+1} -x_i ^{2})^{2}+(x_i -1)^{2})} \end{aligned}$$

$\varvec{f}_{2}$ : Rastrigin’s function

$$\begin{aligned} f(x)=\sum _{i=1}^{pd} {(x_i^2 -10\cos (2\pi x_i )+10)} \end{aligned}$$

$\varvec{f}_{3}$ : Ackley’s function

$$\begin{aligned} f(x)=20+e-20e^{-0.2\sqrt{\frac{\sum _{i=1}^{pd} {x_i^2 } }{pd}}}-e^{\frac{\sum _{i=1}^{pd} {\cos (2\pi x_i )} }{pd}} \end{aligned}$$

$\varvec{f}_{4}$ : Weierstrass function

$$\begin{aligned} \begin{array}{l} f(x)=\sum _{i=1}^{pd} {(\sum _{k=0}^{k\max } {[a^{k}\cos (2\pi b^{k}(x_i +0.5))]} )} \\ -pd\sum _{k=0}^{k\max } {[a^{k}\cos (2\pi b^{k}0.5)]} , \\ where_ a=0.5,b=0.3,k\max =20 \\ \end{array} \end{aligned}$$

$\varvec{f}_{5}$ : Griewank function

$$\begin{aligned} f(x)=1+\frac{\sum _{i=1}^{pd} {(x_i -100)^{2}} }{4000}-\prod _{i=1}^{pd} {\cos (\frac{x_i -100}{\sqrt{i}})} \end{aligned}$$

$\varvec{f}_{6}$ : Sphere function

$$\begin{aligned} f(x)=\sum _{i=1}^{pd} {x_i^2 } \end{aligned}$$

$\varvec{f}_{7}$ : Non-continuous Rastrigin’s function

$$\begin{aligned} \begin{array}{l} f(x)=\sum _{i=1}^{pd} {(y_i^2 -10\cos (2\pi y_i )+10)} \\ y_i =\left\{ {{\begin{array}{ll} {x_i }&{} {\left| {x_i } \right| <\frac{1}{2}} \\ {\frac{round(2x_i )}{2}}&{} {\left| {x_i } \right| \ge \frac{1}{2}} \\ \end{array} }} \right. ,\quad i=1,2,\ldots ,pd \\ \end{array} \end{aligned}$$

$\varvec{f}_{8}$ : Quadratic function

$$\begin{aligned} f(x)=\sum _{i=1}^{pd} {\left( {\sum _{j=1}^i {x_{j}} } \right) ^{2}} \end{aligned}$$

$\varvec{f}_{9}$ : Generalized penalized function

$$\begin{aligned} f(x)&= 0.1\left\{ {\begin{array}{ll} \sin ^{2}(3\pi x_1 )+\sum _{i=1}^{pd-1} {(x_i -1)^{2}[1+\sin ^{2}(3\pi x_{i+1} )]} + \\ (x_{pd} -1)^{2}[1+\sin ^{2}(2\pi x_{pd} )] \\ \end{array}} \right\} \\&+\sum _{i=1}^{pd} {u(x_i ,5,100,4)} \\ \end{aligned}$$

where

$$\begin{aligned} u(x_i a,k,m)=\left\{ {{\begin{array}{l} {{\begin{array}{ll} {k(x_i -a)^{m}}&{} {x_i >a} \\ \end{array} }} \\ {{\begin{array}{ll} 0&{} {-a\le x_i \le a} \\ \end{array} }} \\ {{\begin{array}{ll} {k(-x_i -a)^{m}}&{} {x_i <a} \\ \end{array} }} \\ \end{array} }} \right. \end{aligned}$$

Appendix II

Appendix II displays optimization curves for some of the benchmark functions ($f_{1}, f_{2}, f_{3}, f_{5}, f_{6}, f_{8})$ using theStdPSO2006, theStdPSO2006-Rotm,the CoPSO, the CoPSO-Rotm, the AIWPSO, and the AIWPSO-Rotm in 10, 30, and 60 dimensional spaces.

Appendix III

Appendix III presents the implementation details for calculating Rotm($\sigma )$ as given in Eq. (10). Assume we need to multiply $d$-dimensional vector $v$ by Rotm($\sigma )$. According to Eq. (10), we have:

$$\begin{aligned} v\times Rotm\left( \sigma \right) =v\times Rot_{1,2} \left( {\alpha _{1,2} } \right) \times Rot_{1,3} \left( {\alpha _{1,3} } \right) \times \cdots \times Rot_{d-1,d} \left( {\alpha _{d-1,d} } \right) \end{aligned}$$

The number of matrices on the right side of the equation is $d(d-1)/2$. However, according to Eq. (10), each matrix $Rot_{i,j} \left( {\alpha _{i,j} } \right)$ only contains 4 non-zero values that are in ($i$, $i)$, ($i$, $j)$, ($j$, $i)$, and ($j$, $j)$ positions of Rot. Consequently, multiplying $v$ by Rot can be performed as follows:

$$\begin{aligned} v_k \times Rot_{i,j} \left( {\alpha _{i,j} } \right) =\left\{ {{\begin{array}{ll} {v_k Rot_{i,i} \left( {\alpha _{i,i} } \right) +v_j Rot_{j,i} \left( {\alpha _{j,i} } \right) }&{}\quad {if k=i} \\ {v_k Rot_{j,j} \left( {\alpha _{j,j} } \right) +v_i Rot_{i,j} \left( {\alpha _{j,i} } \right) }&{}\quad {if k=j} \\ {v_k }&{}\quad {otherwise} \\ \end{array} }} \right. \end{aligned}$$

Clearly, multiplication of $v_{k}$ and $Rot_{i,j} \left( {\alpha _{i,j} } \right)$ can be done in O(1). Also, this multiplication alters only two indexes in $v$. This formulation can be repeatedly applied to $v$ for different matrices $Rot_{j,i} \left( {\alpha _{j,i} } \right)$. Because the number of these matrices is $d(d-1)/2$ and the multiplication is in O(1), this multiplication is done in $O(d^{2})$.