Abstract
Physics-based models are widely used to study dynamical systems in a variety of scientific and engineering problems. However, these models are necessarily approximations of reality due to incomplete knowledge or excessive complexity in modeling underlying processes. As a result, they often produce biased simulations due to inaccurate parameterizations or approximations used to represent the true physics. In this paper, we aim to build a new physics-guided machine learning framework to monitor dynamical systems. The idea is to use advanced machine learning model to extract complex spatio-temporal data patterns while also incorporating general scientific knowledge embodied in simulated data generated by the physics-based model. To handle the bias in simulated data caused by imperfect parameterization, we propose to extract general physical relations jointly from multiple sets of simulations generated by a physics-based model under different physical parameters. In particular, we develop a spatio-temporal network architecture that uses its gating variables to capture the variation of physical parameters. We initialize this model using a pre-training strategy that helps discover common physical patterns shared by different sets of simulated data. Then, we fine-tune it combining limited observations and adequate simulations. By leveraging the complementary strength of machine learning and domain knowledge, our method has been shown to produce accurate predictions, use less training samples and generalize to out-of-sample scenarios. We further show that the method can provide insights about the variation of physical parameters over space and time in two domain applications: predicting temperature in streams and predicting temperature in lakes.





Similar content being viewed by others
References
Hipsey MR et al (2019) A general lake model (GLM 3.0) for linking with high-frequency sensor data from the global lake ecological observatory network (GLEON). Geosci Model Dev 12:473–523
Cox P et al (1999) The impact of new land surface physics on the GCM simulation of climate and climate sensitivity. Clim Dyn 15:183–203
Tonks MR et al (2017) Mechanistic materials modeling for nuclear fuel performance. Ann Nuclear Energy 105:11–24
Gupta HV, Nearing GS (2014) Debates-The future of hydrological sciences: a (common) path forward? Using models and data to learn: a systems theoretic perspective on the future of hydrological science. Water Resour Res 50:5351–5359
Lall U (2014) Debates-The future of hydrological sciences: a (common) path forward? One water. One world. Many climes. Many souls Water Resour Res 50:5335–5341
McDonnell JJ, Beven K (2014) McDonnell JJ, Beven K (2014) Debates-The future of hydrological sciences: a (common) path forward? A call to action aimed at understanding velocities, celerities, and residence time distributions of the headwater hydrograph. Water Resour Res 50:5342–5350
Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Comput Intell Neurosci 2018:7068349. https://doi.org/10.1155/2018/7068349
Zhang L, Han J, Wang H, Car R, Weinan E (2018) Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Phys Rev Lett 120(14):143001
Otter DW, Medina JR, Kalita JK (2020) A survey of the usages of deep learning for natural language processing. IEEE Trans Neural Netw Learn Syst 32:604–624
Karpatne A, Atluri G, Faghmous JH, Steinbach M, Banerjee A, Ganguly A, Shekhar S, Samatova N, Kumar V (2017) Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans Knowl Data Eng 29(10):2318–2331
Willard J, Jia X, Xu S, Steinbach M, Kumar V (2021) Integrating scientific knowledge with machine learning for engineering and environmental systems. arXiv preprint arXiv:2003.04919
Xu T et al (2015) Data-driven methods to improve baseflow prediction of a regional groundwater model. Comput Geosci 85:124–136
Reichstein M, Camps-Valls G, Stevens B, Jung M, Denzler J, Carvalhais N (2019) Deep learning and process understanding for data-driven earth system science. Nature 566(7743):195–204
Krasnopolsky VM, Fox-Rabinovitz MS (2006) Complex hybrid models combining deterministic and machine learning components for numerical climate modeling and weather prediction. Neural Netw 19(2):122–134
Faghmous JH, Kumar V (2014) A big data guide to understanding climate change: the case for theory-guided data science. Big Data 2(3):155–163
O’Gorman PA, Dwyer JG (2018) Using machine learning to parameterize moist convection: potential for modeling of climate, climate change, and extreme events. J Adv Model Earth Syst 10(10):2548–2563
Graham-Rowe D, Goldston D, Doctorow C, Waldrop M, Lynch C, Frankel F, Reid R, Nelson S, Howe D, Rhee S (2008) Big data: science in the petabyte era. Nature 455(7209):8–9
Goh GB, Hodas NO, Vishnu A (2017) Deep learning for computational chemistry. J Comput Chem 38(16):1291–1307
Forssell U, Lindskog P (1997) Combining semi-physical and neural network modeling: An example of its usefulness. In: IFAC
Wan ZY et al (2018) Data-assisted reduced-order modeling of extreme events in complex dynamical systems. PloS One 13:e0197704
Jia X, Willard J, Karpatne A, Read JS, Zwart JA, Steinbach M, Kumar V (2021) Physics-guided machine learning for scientific discovery: an application in simulating lake temperature profiles. ACM/IMS Trans Data Sci 2:1–26
Fioretto F (2020) Predicting ac optimal power flows: Combining deep learning and lagrangian dual methods. In: AAAI
Karpatne A, Watkins W, Read J, Kumar V (2017) Physics-guided neural networks (pgnn): An application in lake temperature modeling. arXiv preprint arXiv:1710.11431
Read JS, Jia X, Willard J, Appling AP, Zwart JA, Oliver SK, Karpatne A, Hansen GJ, Hanson PC, Watkins W et al (2019) Process-guided deep learning predictions of lake water temperature. Water Resour Res 55(2019):9173–9190
Stewart R, Ermon S (2017) Label-free supervision of neural networks with physics and domain knowledge. In: AAAI
Hanson PC, Stillman AB, Jia X, Karpatne A, Dugan HA, Carey CC, Stachelek J, Ward NK, Zhang Y, Read JS et al (2020) Predicting lake surface water phosphorus dynamics using process-guided machine learning. Ecol Model 430:109136
Chen S, Sammak S, Givi P, Yurko JP, Jia X (2021) Reconstructing high-resolution turbulent flows using physics-guided neural networks. In: 2021 IEEE international conference on big data (Big Data), IEEE, pp 1369–1379
Hurtado DM et al (2018) Deep transfer learning in the assessment of the quality of protein models. arXiv preprint arXiv:1804.06281
Sultan MM, Wayment-Steele HK, Pande VS (2018) Transferable neural networks for enhanced sampling of protein dynamics. J Chem Theory Comput 14:1887–1894
Daw A, Thomas R, Carey C, Read J, Appling A, Karpatne A (2019) Physics-guided architecture (pga) of neural networks for quantifying uncertainty in lake temperature modeling. arXiv:1911.02682
Muralidhar N (2020) Phynet: Physics guided neural networks for particle drag force prediction in assembly. In: SDM
Ling J, Kurzawski A, Templeton J (2016) Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J Fluid Mech 807:155–166
Zhang L et al (2018) End-to-end symmetry preserving inter-atomic potential energy model for finite and extended systems. In: NeurIPS
Schütt KT (2017) et al.: Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In: NeurIPS
Bao T, Jia X, Zwart J, Sadler J, Appling A, Oliver S, Johnson TT (2021) Partial differential equation driven dynamic graph networks for predicting stream water temperature. In: 2021 IEEE international conference on data mining (ICDM), IEEE, pp 11–20
Bao T, Chen S, Johnson TT, Givi P, Sammak S, Jia X (2022) Physics guided neural networks for spatio-temporal super-resolution of turbulent flows. In: Uncertainty in artificial intelligence, PMLR, pp 118–128
Jia X, Willard J, Karpatne A, Read J, Zwart J, Steinbach M, Kumar V (2019) Physics guided rnns for modeling dynamical systems: a case study in simulating lake temperature profiles. In: SDM
Data release: Process-based predictions of lake water temperature in the Midwest US. https://www.usgs.gov/data/data-release-process-based-predictions-lake-water-temperature-midwest-us
Predicting water temperature in the Delaware River Basin. https://www.usgs.gov/data/predicting-water-temperature-delaware-river-basin
Data release: a large-scale database of modeled contemporary and future water temperature data for 10,774 Michigan, Minnesota and Wisconsin Lakes. https://www.usgs.gov/data/data-release-a-large-scale-database-modeled-contemporary-and-future-water-temperature-data
Anderson B, Hy TS, Kondor R (2019) Cormorant: Covariant molecular neural networks. In: NeurIPS
Muralidhar N (2018) Incorporating prior domain knowledge into deep neural networks. In: IEEE big data. IEEE
Ham Y-G, Kim J-H, Luo J-J (2019) Deep learning for multi-year ENSO forecasts. Nature 573(2019):568–572
Lu J, Yao K, Gao F (2009) Process similarity and developing new process models through migration. AIChE J 55(2009):2318–2328
Jia X, Zwart J, Sadler J, Appling A, Oliver S, Markstrom S, Willard J, Xu S, Steinbach M, Read J (2021) Physics-guided recurrent graph model for predicting flow and temperature in river networks. In: SDM. SIAM
Kratzert F, Klotz D, Shalev G, Klambauer G, Hochreiter S, Nearing G (2019) Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrol Earth Syst Sci 23(12):5089–5110
Lu J et al (2008) Model migration with inclusive similarity for development of a new process model. Ind Eng Chem Res 47(2008):9508–9516
Li Y, Gao J, Meng C, Li Q, Su L, Zhao B, Fan W, Han J (2016) A survey on truth discovery. ACM Sigkdd Explor Newsl 17:1–16
Van Engelen JE, Hoos HH (2020) A survey on semi-supervised learning. Mach Learn 109(2):373–440
Ouali Y, Hudelot C, Tami M (2020) An overview of deep semi-supervised learning. arXiv preprint arXiv:2006.05278
Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big Data 3(1):1–40
Wilson G, Cook DJ (2020) A survey of unsupervised deep domain adaptation. ACM Trans Intell Syst Technol (TIST) 11(5):1–46
Ngo BH, Park JH, Park SJ, Cho SI (2021) Semi-supervised domain adaptation using explicit class-wise matching for domain-invariant and class-discriminative feature learning. IEEE Access 9:128467–128480
Xu L, Hu C, Mei K (2022) Semi-supervised regression with manifold: a Bayesian deep kernel learning approach. Neurocomputing 497:76–85
Jaiswal A, Babu AR, Zadeh MZ, Banerjee D, Makedon F (2020) A survey on contrastive self-supervised learning. Technologies 9(1):2
Larsson G, Maire M, Shakhnarovich G (2017) Colorization as a proxy task for visual understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6874–6883
Appice A, Loglisci C, Malerba D (2018) Active learning via collective inference in network regression problems. Inf Sci 460:293–317
Markstrom SL et al. (2015) PRMS-IV, the precipitation-runoff modeling system, version 4. USGS Techniques and Methods (6-B7)
Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning, PMLR, pp 1126–1135
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, vol 30
Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025
Geological Survey US (2016) National water information system data available on the world wide web (USGS water data for the nation)
Read EK, Carr L, De Cicco L, Dugan HA, Hanson PC, Hart JA, Kreft J, Read JS, Winslow LA (2017) Water quality data for national-scale aquatic research: the water quality portal. Water Resour Res 53(2):1735–1745
Regan RS, Markstrom SL, Hay LE, Viger RJ, Norton PA, Driscoll JM, LaFontaine JH (2018) Description of the national hydrologic model for use with the precipitation-runoff modeling system (prms). Technical report, US Geological Survey
gridMET - Climatology Lab. http://www.climatologylab.org/gridmet.html
Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7167–7176
Burns DA, Murdoch PS, Lawrence GB, Michel RL (1998) Effect of groundwater springs on \(No_3^-\) concentrations during summer in Catskill mountain streams. Water Resour Res 34(8):1987–1996
Acknowledgements
This work was supported by the USGS awards G21AC10207, G21AC10564, and G22AC00266, the NSF awards 2147195, 2105133, and 2126474, the NASA award 80NSSC22K1164, Google’s AI for Social Good Impact Scholars program, the Pitt Momentum Award, and the DRI award at the University of Maryland. This research was supported in part by the University of Pittsburgh Center for Research Computing through the resources provided. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the US Government.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Chen, S., Kalanat, N., Xie, Y. et al. Physics-guided machine learning from simulated data with different physical parameters. Knowl Inf Syst 65, 3223–3250 (2023). https://doi.org/10.1007/s10115-023-01864-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-023-01864-z