Skip to main content
SpringerLink
Log in

The optimal design and application of LSTM neural network based on the hybrid coding PSO algorithm

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Long short-term memory (LSTM) neural network has been widely studied and applied in the real world. To obtain the LSTM neural network with better accuracy and more appropriate structure, the hybrid coding particle swarm optimization (HCPSO) algorithm is proposed. Firstly, the hybrid coding scheme is developed to represent the weights and structure of LSTM neural network, simultaneously. Then, the novel update mechanism is proposed to adjust the position of particles. Meanwhile, the discrete update strategy  (DUS) and adaptive nonlinear moderate random search strategy (ANMRS) are proposed to enhance the convergence and global search capability of HCPSO, respectively. Finally, the effectiveness of HCPSO is demonstrated by multiple numerical examples. The experiment results show that the proposed HCPSO algorithm is more competitive in optimizing LSTM neural networks than other algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Greff K, Srivastava RK, Koutnik J, Steunebrink BR, Schmidhuber J (2017) LSTM: a search space odyssey. IEEE Transac Neural Netw Learn Syst 28(10):2222–2232. https://doi.org/10.1109/tnnls.2016.2582924

    Article  MathSciNet  Google Scholar 

  2. Wen S, Wang Y, Tang Y, Xu Y, Li P, Zhao T (2019) Real-time identification of power fluctuations based on LSTM recurrent neural network: a case study on singapore power system. IEEE Trans Industr Inf 15(9):5266–5275. https://doi.org/10.1109/TII.2019.2910416

    Article  Google Scholar 

  3. Dong Y, Zhang H, Wang C, Zhou X (2021a) Wind power forecasting based on stacking ensemble model, decomposition and intelligent optimization algorithm. Neurocomputing 462:169–184. https://doi.org/10.1016/j.neucom.2021.07.084

    Article  Google Scholar 

  4. Dong Y, Zhang H, Wang C, Zhou X (2021b) A novel hybrid model based on Bernstein polynomial with mixture of gaussians for wind power forecasting. Appl Energy . https://doi.org/10.1016/j.apenergy.2021.116545

  5. Zhao K, Li L, Cai S, Sun R (2019) An optimized time series combined forecasting method based on neural networks. J Supercomput 76(4):2986–3012. https://doi.org/10.1007/s11227-019-03064-5

    Article  Google Scholar 

  6. Paoletti ME, Haut JM, Plaza J, Plaza A (2020) Scalable recurrent neural network for hyperspectral image classification. J Supercomput 76(11):8866–8882. https://doi.org/10.1007/s11227-020-03187-0

    Article  Google Scholar 

  7. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–80. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  8. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166. https://doi.org/10.1109/72.279181

    Article  Google Scholar 

  9. Mazumdar J, Harley RG (2008) Recurrent neural networks trained with backpropagation through time algorithm to estimate nonlinear load harmonic currents. IEEE Trans Industr Electron 55(9):3484–3491. https://doi.org/10.1109/tie.2008.925315

    Article  Google Scholar 

  10. Puskorius GV, Feldkamp LA (1994) Neurocontrol of nonlinear dynamical-systems with kalman filter trained recurrent networks. IEEE Trans Neural Netw 5(2):279–297. https://doi.org/10.1109/72.279191

    Article  Google Scholar 

  11. Ergen T, Kozat SS (2018) Efficient online learning algorithms based on LSTM neural networks. IEEE Trans Neural Netw Learn Syst 29(8):3772–3783. https://doi.org/10.1109/tnnls.2017.2741598

    Article  MathSciNet  Google Scholar 

  12. Wang ZSLJMLJWZFMQ, Wang, (2019) E-LSTM: An Efficient Hardware Architecture for Long Short-Term. IEEE J Emerging Selected Topics Circuits Syst. https://doi.org/10.1109/jetcas.2019.2911739

  13. Sagheer A, Kotb M (2019) Time series forecasting of petroleum production using deep LSTM recurrent networks. Neurocomputing 323:203–213. https://doi.org/10.1016/j.neucom.2018.09.082

    Article  Google Scholar 

  14. ElSaid A, El Jamiy F, Higgins J, Wild B, Desell T (2018) Optimizing long short-term memory recurrent neural networks using ant colony optimization to predict turbine engine vibration. Appl Soft Comput 73:969–991. https://doi.org/10.1016/j.asoc.2018.09.013

    Article  Google Scholar 

  15. Gundu V, Simon SP (2021) PSO-LSTM for short term forecast of heterogeneous time series electricity price signals. J Ambient Intell Humaniz Comput 12(2):2375–2385. https://doi.org/10.1007/s12652-020-02353-9

    Article  Google Scholar 

  16. Yu J, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71(4–6):1054–1060. https://doi.org/10.1016/j.neucom.2007.10.013

    Article  Google Scholar 

  17. Zhao L, Qian F (2011) Tuning the structure and parameters of a neural network using cooperative binary-real particle swarm optimization. Expert Syst Appl 38(5):4972–4977. https://doi.org/10.1016/j.eswa.2010.09.154

    Article  Google Scholar 

  18. Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22(10):1448–62. https://doi.org/10.1016/j.neunet.2009.05.013

    Article  Google Scholar 

  19. Han HG, Lu W, Hou Y, Qiao JF (2018) An adaptive-PSO-based self-organizing rbf neural network. IEEE TransNeural Netw Learn Syst 29(1):104–117. https://doi.org/10.1109/tnnls.2016.2616413

    Article  MathSciNet  Google Scholar 

  20. Gao H, Xu WB (2011) A new particle swarm algorithm and its globally convergent modifications. IEEE Trans Syst Man and Cybernet Part B-Cybernet 41(5):1334–1351. https://doi.org/10.1109/tsmcb.2011.2144582

    Article  Google Scholar 

  21. Yaghini M, Khoshraftar MM, Fallahi M (2013) A hybrid algorithm for artificial neural network training. Eng Appl Artif Intell 26(1):293–301. https://doi.org/10.1016/j.engappai.2012.01.023

    Article  Google Scholar 

  22. Aljarah I, Faris H, Mirjalili S (2016) Optimizing connection weights in neural networks using the whale optimization algorithm. Soft Comput 22(1):1–15. https://doi.org/10.1007/s00500-016-2442-1

    Article  Google Scholar 

  23. Mirjalili S, Mohd Hashim SZ, Moradian Sardroudi H (2012) Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Appl Math Comput 218(22):11125–11137. https://doi.org/10.1016/j.amc.2012.04.069

    Article  MathSciNet  MATH  Google Scholar 

  24. Shao BL, Li ML, Zhao Y, Bian GQ (2019) Nickel Price Forecast Based on the LSTM Neural Network Optimized by the Improved PSO algorithm. Math Probl Eng 2019:15. https://doi.org/10.1155/2019/1934796

    Article  Google Scholar 

  25. Sheikhan M, Abbasnezhad Arabi M, Gharavian D (2015) Structure and weights optimisation of a modified elman network emotion classifier using hybrid computational intelligence algorithms: a comparative study. Connect Sci 27(4):340–357. https://doi.org/10.1080/09540091.2015.1080224

    Article  Google Scholar 

  26. Zhang L, Li H, Kong XG (2019) Evolving feedforward artificial neural networks using a two-stage approach. Neurocomputing 360:25–36. https://doi.org/10.1016/j.neucom.2019.03.097

    Article  Google Scholar 

  27. Kennedy J, Eberhart R, IEEE, IEEE, IEEE, IEEE (1995) Particle swarm optimization. 1995 IEEE Int Conf Neural Netw Proc. https://doi.org/10.1109/icnn.1995.488968

  28. Rai P, Hanmandlu M (2016) Gaussian Function-Based Particle Swarm Optimization. Springer Singapore, Proceedings of Fifth International Conference on Soft Computing for Problem Solving, pp 57–67

  29. Yang C, Qiao J, Wang L, Zhu X (2018) Dynamical regularized echo state network for time series prediction. Neural Comput Appl 31(10):6781–6794. https://doi.org/10.1007/s00521-018-3488-z

    Article  Google Scholar 

  30. Qiao J, Quan L, Yang C (2020) Design of modeling error PDF based fuzzy neural network for effluent ammonia nitrogen prediction. Appl Soft Comput . https://doi.org/10.1016/j.asoc.2020.106239

  31. Kennedy J, Eberhart RC, IEEE, (1997) A discrete binary version of the particle swarm algorithm, pp 4104–4108. IEEE Int Conf Syst Man, and Cybernatics, Conf Proc. https://doi.org/10.1109/icsmc.1997.637339

  32. Yang C, Qiao J, Ahmad Z, Nie K, Wang L (2019) Online sequential echo state network with sparse RLS algorithm for time series prediction. Neural Netw 118:32–42. https://doi.org/10.1016/j.neunet.2019.05.006

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (62021003, 61890930-5 and 61973010), National Natural Science Foundation of Beijing (4202006) and Major Science and Technology Program for Water Pollution Control and Treatment of China (2018ZX07111-005).

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing, 100124, People’s Republic of China

    Zhonglin Chen, Cuili Yang & Junfei Qiao

Authors
  1. Zhonglin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cuili Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junfei Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junfei Qiao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: The description of ODUS and MRS

For ODUS of HCPSO-LSTM-ODUS, the update of the particle position is as follows:

$$\begin{aligned} v_{j, d}(t+1)=\; & {} \omega v_{j, d}(t)+c_{1} r_{1}\left( p_{j, d}(t)-a_{j, d}(t)\right) +c_{2} r_{2}\left( g_{d}(t)-a_{j, d}(t)\right) \\ a_{j,d}(t+1)= & {} \left\{ \begin{array}{ll} 0, &{} \text{ if } r_{3}>{C}(v_{j,d}(t+1))\\ 1, &{} \text{ otherwise } \end{array}\right. \end{aligned}$$

with

$$\begin{aligned} C\left( v_{j, d}(t+1)\right) =\;\frac{1}{1+e^{-v_{j, d}(t+1)}} \end{aligned}$$

For MRS of HCPSO-LSTM-MRS, the update of the particle position is as follows:

$$\begin{aligned} {a}_{j,d}(t+1)= & {} {\hat{P}}_{d}(t)+\eta \beta \left( {M}_{d}(t)-{a}_{j,d}(t)\right) \\ {\hat{P}}_{d}(t)= \;& {} r_{4}{{p}_{j,d}(t)}+\left( 1-r_{4}\right) {g}_{d}(t) \\ {M}_{d}(t)= & {} \sum _{j=1}^{S} \frac{{p}_{j,d}(t)}{S} \\ \eta= & {} \frac{r_{5}-r_{6}}{r_{7}} \\ \beta= & {} \beta _{\max }-\frac{t}{t_{\max }}\left( \beta _{\max }-\beta _{\min }\right) \end{aligned}$$

where \(\beta _{\mathrm{max}}\) and \(\beta _{\mathrm{min}}\) are the upper and lower limits of \(\beta \), respectively, and they take values in the range of \([0.1,\,1.0]\).

Appendix B: Nomenclature and abbreviations

Table 8 Nomenclature and abbreviations
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Yang, C. & Qiao, J. The optimal design and application of LSTM neural network based on the hybrid coding PSO algorithm. J Supercomput 78, 7227–7259 (2022). https://doi.org/10.1007/s11227-021-04142-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-04142-3

Keywords

Search

Navigation