Skip to main content
SpringerLink
Log in

Numerical solution for high-dimensional partial differential equations based on deep learning with residual learning and data-driven learning

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Solving high-dimensional partial differential equations (PDEs) is a long-term computational challenge due to the fundamental obstacle known as the curse of dimensionality. This paper develops a novel method (DL4HPDE) based on residual neural network learning with data-driven learning elliptic PDEs on a box-shaped domain. However, to combine a strong mechanism with a weak mechanism, we reconstruct a trial solution to the equations in two parts: the first part satisfies the initial and boundary conditions, while the second part is the residual neural network algorithm, which is used to train the other part. In our proposed method, residual learning is adopted to make our model easier to optimize. Moreover, we propose a data-driven algorithm that can increase the training spatial points according to the regional error and improve the accuracy of the model. Finally, the numerical experiments show the efficiency of our proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Sirignano J, Spiliopoulos K (2018) DGM: a deep learning algorithm for solving partial differential equations. J Comput Phys 375:1339–1364

    Article  MathSciNet  Google Scholar 

  2. Han J, Jentzen A, Weinan E (2018) Solving high-dimensional partial differential equations using deep learning. Proc Natl Acad Sci 115(34):8505–8510

    Article  MathSciNet  Google Scholar 

  3. Bellman RE (2015) Adaptive control processes: a guided tour, vol 2045. Princeton University Press, Princeton

    Google Scholar 

  4. Bellman R (1966) Dynamic programming. Science 153(3731):34–37

    Article  Google Scholar 

  5. Mall S, Chakraverty S (2016) Application of legendre neural network for solving ordinary differential equations. Appl Soft Comput 43:347–356

    Article  Google Scholar 

  6. Sun H, Hou M, Yang Y, Zhang T, Weng F, Han F (2019) Solving partial differential equation based on Bernstein neural network and extreme learning machine algorithm. Neural Process Lett 50(2):1153–1172

    Article  Google Scholar 

  7. Mall S, Chakraverty S (2017) Single layer Chebyshev neural network model for solving elliptic partial differential equations. Neural Process Lett 45(3):825–840

    Article  Google Scholar 

  8. Wang Z, Tang Q, Guo W, Cheng Y (2016) Sparse grid discontinuous Galerkin methods for high-dimensional elliptic equations. J Comput Phys 314:244–263

    Article  MathSciNet  Google Scholar 

  9. Zhao Y, Zhang Q, Levesley J (2018) Multilevel sparse grids collocation for linear partial differential equations, with tensor product smooth basis functions. Comput Math Appl 75(3):883–899

    Article  MathSciNet  Google Scholar 

  10. Wu K, Xiu D (2020) Data-driven deep learning of partial differential equations in modal space. J Comput Phys 408:109307

    Article  MathSciNet  Google Scholar 

  11. Zhu Y, Zabaras N, Koutsourelakis P-S, Perdikaris P (2019) Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. J Comput Phys 394:56–81

    Article  MathSciNet  Google Scholar 

  12. Berner J, Grohs P, Jentzen A (2020) Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black–Scholes partial differential equations. SIAM J Math Data Sci 2:631–657

    Article  MathSciNet  Google Scholar 

  13. Zang Y, Bao G, Ye X, Zhou H (2020) Weak adversarial networks for high-dimensional partial differential equations. J Comput Phys 411:109409

    Article  MathSciNet  Google Scholar 

  14. Hornung F, Jentzen A, Salimova D (2020) Space-time deep neural network approximations for high-dimensional partial differential equations. arXiv:2006.02199

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: NIPS

  16. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  17. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  18. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser I, Julianand Antonoglou V, Panneershelvam ML et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  19. Pinkus A (1999) Approximation theory of the MLP model in neural networks. Acta Numer 8(1):143–195

    Article  MathSciNet  Google Scholar 

  20. Tao Z, Muzhou H, Chunhui L (2018) Forecasting stock index with multi-objective optimization model based on optimized neural network architecture avoiding overfitting. Comput Sci Inf Syst 15(1):211–236

    Article  Google Scholar 

  21. Yang Y, Hou M, Luo J (2018) A novel improved extreme learning machine algorithm in solving ordinary differential equations by Legendre neural network methods. Adv Differ Equations 2018(1):469

    Article  MathSciNet  Google Scholar 

  22. Zhou T, Liu X, Hou M, Liu C (2019) Numerical solution for ruin probability of continuous time model based on neural network algorithm. Neurocomputing 331:67–76

    Article  Google Scholar 

  23. Yang Y, Hou M, Luo J, Liu T (2018) Neural network method for lossless two-conductor transmission line equations based on the IELM algorithm. AIP Adv 8(6):065010

    Article  Google Scholar 

  24. T. Liu, M Hou (2017) A fast implicit finite difference method for fractional advection-dispersion equations with fractional derivative boundary conditions. Adv Math Phy 2017:1–8

  25. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  26. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  27. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), Boston, MA, pp 1–9

  28. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules, 31st Conference on Neural Information Processing Systems (NIPS 2017), pp 3856–3866

  29. Wang Z, Xiao Y, Li Y, Zhang J, Hou M, Liu X (2020) Automatically discriminating and localizing covid-19 from communityacquired pneumonia on chest X-rays. Pattern Recogn 110:107613

    Article  Google Scholar 

  30. Wang Z, Meng Y, Weng F, Chen Y, Hou M, Zhang J (2012) An effective CNN method for fully automated segmenting subcutaneous and visceral adipose tissue on CT scans. Ann Biomed Eng 48(1):312–328

    Article  Google Scholar 

  31. Raissi M, Perdikaris P, Karniadakis GE (2017) Physics informed deep learning (part I): data-driven solutions of nonlinear partial differential equations. arXiv:1711.10561

  32. Raissi M (2018) Deep hidden physics models: deep learning of nonlinear partial differential equations. J Mach Learn Res 19(1):932–955

    MathSciNet  MATH  Google Scholar 

  33. Raissi M (2018) Forward-backward stochastic neural networks: deep learning of high-dimensional partial differential equations. arXiv:1804.07010

  34. Becker S, Braunwarth R, Hutzenthaler M, Jentzen A, von Wurstemberger P (2020) Numerical simulations for full history recursive multilevel Picard approximations for systems of high-dimensional partial differential equations. arXiv:2005.10206

  35. Han J, Nica M, Stinchcombe AR (2020) A derivative-free method for solving elliptic partial differential equations with deep neural networks. J Comput Phys 419:109672

    Article  MathSciNet  Google Scholar 

  36. Chen J (2020) A comparison study of deep Galerkin method and deep Ritz method for elliptic problems with different boundary conditions. Commun Math Res 36(3):354–376

    Article  MathSciNet  Google Scholar 

  37. van der Meer R, Oosterlee C, Borovykh A (2020) Optimally weighted loss functions for solving PDEs with neural networks. arXiv:2002.06269

  38. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, pp 2921–2929

  39. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, pp 770–778

  40. Lagaris IE, Likas A, Fotiadis DI (1998) Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans Neural Netw 9(5):987–1000

    Article  Google Scholar 

  41. Zhang J, Xiao W, Li Y, Zhang S (2018) Residual compensation extreme learning machine for regression. Neurocomputing 311:126–136

    Article  Google Scholar 

  42. Vapnik V, Vapnik V (1998) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  43. Zhang C, Bengio S, Hardt M, Recht B, Vinyals O (2017) Understanding deep learning requires rethinking generalization. arXiv:1611.03530

  44. Guo Y, Cao X, Liu B, Gao M (2020) Solving partial differential equations using deep learning and physical constraints. Appl Sci 10(17):5917

    Article  Google Scholar 

  45. Zhang X (2020) Actor-critic algorithm for high-dimensional partial differential equations. arXiv:2010.03647

  46. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines Vinod Nair. Proceedings of ICML. 27. 807–814

  47. Bishop CM et al (1995) Neural networks for pattern recognition. Oxford University Press, Oxford

    MATH  Google Scholar 

  48. Ripley BD (2007) Pattern recognition and neural networks. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  49. Venables WN, Ripley BD (2013) Modern applied statistics with S-PLUS. Springer, New York

    MATH  Google Scholar 

  50. Kingma DP, Ba J (2017) Adam: a method for stochastic optimization. arXiv:1412.6980.

  51. Gu Y, Yang H, Zhou C (2020) SelectNet: self-paced learning for high-dimensional partial differential equations. arXiv:2001.04860

Download references

Funding

This work was supported by the Graduate Student Innovation Foundation of Central South University (2019zzts213) and supported by the Scientific Research Fund of Hunan Provincial Education Department (No. 18C0332).

Author information

Author notes

  1. Zheng Wang and Futian Weng contributed equally.

Authors and Affiliations

  1. School of Mathematics and Statistics, Central South University, Changsha, 410083, China

    Zheng Wang, Futian Weng, Jialin Liu, Kai Cao & Muzhou Hou

  2. Science and Engineering School, Hunan First Normal University, Changsha, 410205, China

    Zheng Wang

  3. College of Finance and Statistics, Hunan University, Changsha, 410082, China

    Juan Wang

Authors
  1. Zheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Futian Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jialin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muzhou Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Juan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muzhou Hou.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Weng, F., Liu, J. et al. Numerical solution for high-dimensional partial differential equations based on deep learning with residual learning and data-driven learning. Int. J. Mach. Learn. & Cyber. 12, 1839–1851 (2021). https://doi.org/10.1007/s13042-021-01277-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01277-w

Keywords

Search

Navigation