Q-learning-based simulated annealing algorithm for constrained engineering design problems

Abstract

Simulated annealing (SA) was recognized as an effective local search optimizer, and it showed a great success in many real-world optimization problems. However, it has slow convergence rate and its performance is widely affected by the settings of its parameters, namely the annealing factor and the mutation rate. To mitigate these limitations, this study presents an enhanced optimizer that integrates Q-learning algorithm with SA in a single optimization model, named QLSA. In particular, the Q-learning algorithm is embedded into SA to enhance its performances by controlling its parameters adaptively at run time. The main characteristics of Q-learning are that it applies reward/penalty technique to keep track of the best performing values of these parameters, i.e., annealing factor and the mutation rate. To evaluate the effectiveness of the proposed QLSA algorithm, a total of seven constrained engineering design problems were used in this study. The outcomes show that QLSA was able to report a mean fitness value of 1.33 on cantilever beam design, 263.60 on three-bar truss design, 1.72 on welded beam design, 5905.42 on pressure vessel design, 0.0126 on compression coil spring design, 0.25 on multiple disk clutch brake design, and 2994.47 on speed reducer design problem. Further analysis was conducted by comparing QLSA with the state-of-the-art population optimization algorithms including PSO, GWO, CLPSO, harmony, and ABC. The reported results show that QLSA significantly (i.e., 95% confidence level) outperforms other studied algorithms.

Appendix A: engineering design problems

1.1 A.1 Cantilever beam

$${\text{Minimize}} f\left( {{\vec{{x}}}} \right){ = 0} . 0 6 2 2 4 {x}_{ 1} { + x}_{ 2} { + x}_{ 3} { + x}_{ 4} { + x}_{ 5} ,\quad {\text{subject to }}g_{ 1} \left( {{\vec{x}}} \right){ = } \frac{ 6 1}{{{x}_{ 1}^{ 3} }}{ + }\frac{ 3 7}{{{x}_{ 2}^{ 3} }}{ + }\frac{ 1 9}{{{x}_{ 3}^{ 3} }}{ + }\frac{ 7}{{{x}_{ 4}^{ 3} }}{ + }\frac{ 1}{{{x}_{ 5}^{ 3} }} \le 1$$

where $0{\text{~}} \le {\text{~}}x_{{\text{i}}} \le 100,\quad {\text{i}} = 1,2, \ldots ,5$

1.2 A.2 Three-bar truss

$${\text{Minimize~}}f\left( {\vec{x}} \right) = 2{\text{~}}\sqrt 2 {\text{~}}x_{1} + x_{2}$$

$${\text{subject~to~}}g_{1} \left( {\vec{x}} \right) = {\text{~~}}2{\text{*}}\left( {\frac{{\sqrt 2 x_{1} + x_{2} }}{{\sqrt 2 {\text{x}}_{1}^{2} + 2x_{1} x_{2} }}} \right) - 2 \le 0,\quad {\text{g}}_{2} \left( {\vec{x}} \right) = {\text{~~}}2{\text{*}}\left( {\frac{1}{{x_{1} + \sqrt 2 x_{2} }}} \right) - 2 \le 0,\quad g_{3} \left( {\vec{x}} \right) = {\text{~~}}2{\text{*}}\left( {\frac{{x_{2} }}{{\sqrt 2 x_{1}^{2} + {\text{~}}2x_{1} x_{2} }}} \right) - 2 \le 0$$

where $0 \le {{x}}_{{i}} \le 100,\quad {{i}} = 1,2, \ldots ,5$, and A1 = A3.

1.3 A.3 Welded beam

$${\text{Minimize~}}f\left( {\vec{x}} \right) = 1.10471{\text{h}}^{2} {\text{l}} + 0.04811{\text{tb}}\left( {14 + {\text{l}}} \right)$$

$${\text{subject to }}$$

$$g_{1} \left( {\vec{x}} \right) = ~~\tau \left( x \right) - ~\tau _{{\max }} \le 0$$

$$g_{2} \left( {\vec{x}} \right) = {\text{~}}\sigma \left( x \right) - \sigma _{x} \le 0$$

$${\text{~}}g_{3} \left( {\vec{x}} \right) = {\text{~}}h - b \le 0$$

$$g_{4} \left( {\vec{x}} \right) = 0.125 - h \le 0$$

$$g_{5} \left( {\vec{x}} \right) = ~~L - 240 \le 0$$

1.4 A.4 Pressure vessel

$${\text{Minimize~}}f\left( {\vec{x}} \right) = 0.6224{\text{~}}T_{s} R~L + 1.7781{\text{~}}T_{h} R^{2} + 3.1661T_{s}^{2} {\text{~}}L + 19.84{\text{~}}T_{s}^{2} {\text{~}}R{\text{~}}$$

$${\text{subject to}}$$

$$g_{1} \left( {\vec{x}} \right) = 0.0193R - T_{s} \le 0$$

$$~g_{2} \left( {\vec{x}} \right) = ~~0.00954R - T_{h} \le 0$$

$${\text{~}}g_{3} \left( {\vec{x}} \right) = {\text{~~}}1296000 - \pi R^{2} L - \frac{4}{3}{\text{~}}\pi R^{3} \le 0$$

$$g_{4} \left( {\vec{x}} \right) = L - 240 \le 0$$

where

$$0~ \le ~T_{s} \le 99,\quad 0~ \le ~T_{h} \le 99,\quad 10 \le R \le 200,\quad 10 \le L \le 200.$$

1.5 A.5 Compression coil spring

$${\text{Minimize }}f\left( {\vec{x}} \right) = \left( {{x}_{3} + 2} \right)x_{2} x_{1}^{2}$$

Subject to

$$g_{1} \left( {\vec{x}} \right) = 1 - \frac{{x_{2}^{3} x_{3} }}{{7.1785 x_{1}^{4} }} \le 0$$

$$g_{2} (\vec{x}) = \frac{{4x_{2}^{3} - x_{1} x_{2} }}{{12,566(x_{2} x_{1}^{3} ) - x_{1}^{4} }} + \frac{1}{{5.108x_{1}^{2} }} - 1 \le 0$$

$$g_{3} \left( {\vec{x}} \right) = 1 - \frac{{140.45 x_{1} }}{{x_{2}^{2} x_{3} }} \le 0$$

$$g_{4} \left( {\vec{x}} \right) = \frac{{x_{2} + x_{1} }}{1.5} - 1 \le 0$$

where $0.05 \le x_{1} \le 2$, $0.25 \le x_{2} \le 1.3$, $2 \le x_{3} \le 15$

1.6 A.6 Multiple disk clutch brake

$${\text{Minimize}}~f\left( {\vec{x}} \right) = \pi ~\left( {x_{2}^{2} - x_{1}^{2} } \right)x_{3} \left( {x_{5} + 1} \right)\rho$$

$${\text{subject to}}$$

$$g_{1} \left( x \right) = ~x_{2} - x_{1} - \nabla R \ge 0$$

$$g_{2} \left( x \right) = ~L_{{\max }} - (x_{5} + 1~)\left( {x_{3} + ~\delta } \right) \ge ~0$$

$$g_{3} \left( x \right) = ~p_{{\max }} - p_{{{\text{rz}}}} \ge ~0$$

$$g_{4} \left( x \right) = ~p_{{\max }} *V{\text{sr}}_{{\max }} - p_{{{\text{rz}}}} *V{\text{sr}} \ge ~0$$

$$g_{5} \left( x \right) = {\text{Vsr}}_{ \hbox{max} } - {\text{Vsr}} \ge 0$$

$$g_{6} \left( x \right) = ~T_{{\max }} - T \ge ~0$$

$$g_{7} \left( x \right) = ~M_{h} - s~M_{s} \ge ~0$$

$$g_{8} \left( x \right) = T \ge 0$$

where

$$M_h = \frac{2}{3}\mu x_{4} x_{5} \frac{{x_{2}^{3} - x_{1}^{3} }}{{x_{2}^{2} - x_{1}^{2} }}\;{\text{N}}\cdot {\text{mm}},$$

$$w = \frac{\pi n}{30}\;{\text{rad}}/{\text{s}}$$

$$A = \pi \left( { x_{2}^{2} - x_{1}^{2} } \right)\;{\text{mm}}^{2}$$

$$P_{\text{rz}} = \frac{{x_{4} }}{A} \;{\text{N}}/{\text{mm}}^{2}$$

$$V_{\text{sr}} = \frac{{pi R_{\text{sr}} n}}{30} \;{\text{mm}}/{\text{s}}$$

$$R_{\text{sr}} = \frac{2}{3} \frac{{x_{2}^{3} - x_{1}^{3} }}{{x_{2}^{2} x_{1}^{2} }}\;{\text{mm}}$$

$$\Delta R = 20 \;{\text{mm}},\quad L_{ \hbox{max} } = 30\;{\text{mm}},\quad \mu = 0.6$$

$$p_{ \hbox{max} } = 1 \;{\text{M p}}_{\text{a}} ,\quad \rho = 0.0000078 \;{\text{kg}} /{\text{mm}}^{3}$$

$$V_{{{\text{sr}}_{ \hbox{max} } }} = 10\;{\text{m}}/{\text{s}},\quad \delta = 0.5 \;{\text{mm}},\quad s = 1.5$$

$$T_{ \hbox{max} } = 15 \;{\text{s}},\quad n = 250\;{\text{rpm}},\quad I_{z} = 55\;{\text{Kg}} \cdot {\text{m}}^{2}$$

$$M_{s} = 40 \;{\text{Nm}},\quad M f = 3\; {\text{N m}}$$

$$60 \le x_{1} \le 80,\quad 90 \le x_{2} \le 110, \quad 1 \le x_{3} \le 3,$$

$$0 \le x_{4} \le 1000,\quad 2 \le x_{5} \le 9,\quad i = 1, 2, 3, 4, 5$$

1.7 A.7 Speed reducer

$${\text{Minimize~}}f\left( {\vec{x}} \right) = 0.785{\text{~}}x_{1} x_{2}^{2} {\text{~}}\left( {{\text{~}}3.333{\text{~}}x_{3}^{2} + 14.9334~x_{3} - 42.0934} \right) - 1.508~x_{1} \left( {x_{6}^{2} + ~x_{7}^{2} ~} \right) + 7.4777x_{1} \left( {x_{6}^{3} + ~x_{7}^{3} ~} \right) + ~1.508~x_{1} \left( {x_{4} x_{6}^{2} + ~x_{5} x_{7}^{2} ~} \right)$$

(4)

$${\text{subject to}}$$

$$g_{1} \left( {\vec{x}} \right) = \frac{27}{{x_{1} x_{2}^{2} x_{3} }} - 1 \le 0$$

$$g_{2} \left( {\vec{x}} \right) = \frac{397.5}{{x_{1} x_{2} x_{3}^{2} }} - 1 \le 0$$

$$g_{3} \left( {\vec{x}} \right) = \frac{{1.93 x_{4}^{3} }}{{x_{1} x_{3} x_{6}^{4} }} - 1 \le 0$$

$$g_{4} \left( {\vec{x}} \right) = \frac{{1.93 x_{5}^{3} }}{{x_{1} x_{3} x_{7}^{4} }} - 1 \le 0$$

$$g_{5} \left( {\vec{x}} \right) = \frac{1}{{110x_{6}^{3} }} \sqrt {\left( {\frac{{745x_{4} }}{{x_{2} x_{3} }}} \right)^{2} + 16.9 \times 10^{6} } - 1 \le 0$$

$$g_{6} \left( {\vec{x}} \right) = \frac{1}{{85x_{7}^{3} }} \sqrt {\left( {\frac{{745x_{5} }}{{x_{2} x_{3} }}} \right)^{2} + 157.5 \times 10^{6} } - 1 \le 0$$

$$g_{7} \left( {\vec{x}} \right) = \frac{{x_{2} x_{3} }}{40} - 1 \le 0$$

$$g_{8} \left( {\vec{x}} \right) = \frac{{5 x_{2} }}{{x_{1} }} - 1 \le 0$$

$$g_{9} \left( {\vec{x}} \right) = \frac{{x_{1} }}{{12 x_{2} }} - 1 \le 0$$

$$g_{10} \left( {\vec{x}} \right) = \frac{{1.5 x_{6} + 1.9}}{{x_{4} }} - 1 \le 0$$

$$g_{11} \left( {\vec{x}} \right) = \frac{{1.1 x_{7} + 1.9}}{{x_{5} }} - 1 \le 0$$

where $2.6 \le x_{1} \le 3.6$, $0.7 \le x_{2} \le 0.8$, $17 \le x_{3} \le 28$, $7.3 \le x_{4} \le 8.3$, $7.8 \le x_{5} \le 8.3$, $2.9 \le x_{6} \le 3.9$, $x_{6} \le 3.9$, $5.0 \le x_{7} \le 5.5$