Skip to main content
SpringerLink
Log in

GEO-WMS: an improved approach to geoscientific workflow management system on HPC

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

With the increasing complexity of scientific computing, it is imperative to enhance the efficiency and ease of High Performance Computing (HPC) utilization. Scientific workflow is introduced to that aim, but the current infrastructure still needs optimization. In this paper, we discuss the current problems based on scientific computing scenarios and design a more user-friendly workflow system solution targeting HPC services. In the proposed solution, we introduce a structured method to describe the workflow and employ a more user-friendly interface for scientific workflows to bring a better experience than traditional command line approaches. We have integrated a variety of methods to enhance the user experience during geoscience experiments. Data analytics are being used to make more intelligent recommendations to users. Runtime predictions help users to better plan their schedules for research. The statistics of the testing period and user feedback show that the proposed workflow management system can effectively save the operating time and complexity of the scientists, while saving computing resources. Our proposed system has a variety of advantageous features, including the ease of use, uniform specification with scalability, improved utilization of computing resources, and exemplary significance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Availability of data and materials

The data and materials covered in this article are available at https://doi.org/10.6084/m9.figshare.21521361.v1 and are under the GPL 3.0+ License.

References

  • Alper, P., Belhajjame, K., Goble, C.A.: Static analysis of Taverna workflows to predict provenance patterns. Future Gen. Comput. Syst. 75, 310–329 (2017). https://doi.org/10.1016/j.future.2017.01.004

    Article  Google Scholar 

  • Balis, B.: Hyperflow: a model of computation, programming approach and enactment engine for complex distributed workflows. Future Gen. Comput. Syst. 55, 147–162 (2016)

    Article  Google Scholar 

  • Brooks, C., Billings, J.J.: Introducing triquetrum, a possible future for Kepler and ptolemy II. Procedia Comput. Sci. 80, 2449–2454 (2016). https://doi.org/10.1016/j.procs.2016.05.546

    Article  Google Scholar 

  • Crawl, D., Singh, A., Altintas, I.: Kepler WebView: a lightweight, portable framework for constructing real-time web interfaces of scientific workflows. Procedia Comput. Sci. 80, 673–679 (2016). https://doi.org/10.1016/j.procs.2016.05.361

    Article  Google Scholar 

  • Danabasoglu, G., Lamarque, J., Bacmeister, J., Bailey, D.A., Duvivier, A.K., Edwards, J., Emmons, L.K., Fasullo, J., Garcia, R., Gettelman, A.: The community earth system model version 2 (CESM2). J. Adv. Model. Earth Syst. 12(2) (2020)

  • Deelman, E., Peterka, T., Altintas, I., Carothers, C.D., Dam, K., Moreland, K., Parashar, M., Ramakrishnan, L., Taufer, M., Vetter, J.: The future of scientific workflows. Exp. Mech. 32(1), 159–175 (2018)

    Google Scholar 

  • Eyring, V.: Earth system model evaluation with observations: Opportunities and challenges for CMIP6 (2017)

  • Fu, H., Yin, W., Yang, G., Chen, X., Liu, W.: 18.9-pflops nonlinear earthquake simulation on Sunway Taihulight: enabling depiction of 18-Hz and 8-meter scenarios. In: the International Conference for High Performance Computing, Networking, Storage and Analysis (2017)

  • Fu, H., Liao, J., Yang, J., Wang, L., Song, Z., Huang, X., Yang, C., Xue, W., Liu, F., Qiao, F., Zhao, W., Yin, X., Hou, C., Zhang, C., Ge, W., Zhang, J., Wang, Y., Zhou, C., Yang, G.: The Sunway TaihuLight supercomputer: system and applications. Sci. China Inf. Sci. 59(7), 1–16 (2016). https://doi.org/10.1007/s11432-016-5588-7

    Article  Google Scholar 

  • Garijo, D., Khider, D., Ratnakar, V., Gil, Y., Deelman, E., Da Silva, R.F., Knoblock, C., Chiang, Y.Y., Pham, M., Pujara, J., Vu, B., Feldman, D., Mayani, R., Cobourn, K., Duffy, C., Kemanian, A., Shu, L., Kumar, V., Khandelwal, A., Tayal, K., Peckham, S., Stoica, M., Dabrowski, A., Hardesty-Lewis, D., Pierce, S.: An intelligent interface for integrating climate, hydrology, agriculture, and socioeconomic models. In: International Conference on Intelligent User Interfaces, Proceedings IUI, pp. 111–112 (2019). https://doi.org/10.1145/3308557.3308711

  • Hu, J., Tao, L.: Visual modeling of xml constraints based on a new extensible constraint markup language. Eng. Lett. 13(3), 248–254 (2006)

    Google Scholar 

  • Hu, L., Che, X.L., Zheng, S.Q.: Online system for grid resource monitoring and machine learning-based prediction. IEEE Trans. Parallel Distrib. Syst. 23(1), 134–145 (2012). https://doi.org/10.1109/TPDS.2011.108

    Article  Google Scholar 

  • Ivie, P., Thain, D.: Reproducibility in scientific computing. ACM Comput. Surv. 51(3), 1–36 (2018)

    Article  Google Scholar 

  • Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gen. Comput. Syst. 29(3), 682–692 (2013). https://doi.org/10.1016/j.future.2012.08.015

    Article  Google Scholar 

  • Li, X., Song, J., Huang, B.: A scientific workflow management system architecture and its scheduling based on cloud service platform for manufacturing big data analytics. Int. J. Adv. Manuf. Technol. 84(1–4), 119–131 (2016)

    Article  Google Scholar 

  • Liang, X.Z., Min, X., Choi, H.I., Kunkel, K.E., Wang, J.: Development of the regional climate-weather research and forecasting model (CWRF): treatment of subgrid topography effects (2005)

  • Lin, Y., Huang, X., Liang, Y., Qin, Y., Xu, S., Huang, W., Xu, F., Liu, L., Wang, Y., Peng, Y., Wang, L., Xue, W., Fu, H., Zhang, G., Wang, B., Li, R., Zhang, C., Lu, H., Yang, K., Gong, P.: Community integrated earth system model (CIESM): description and evaluation. J. Adv. Model. Earth Syst. 12 (2020)

  • Ma, J., Xu, S., Wang, B.: Reducing numerical diffusion in dynamical coupling between atmosphere and ocean in community earth system model (CESM), version 1.2.1 (2020)

  • Mandal, N., Deelman, E., Mehta, G., Su, M.H., Vahi, K.: Integrating existing Scientific workflow systems: The kepler/pegasus example. In: WORKS 2007—Proceedings of the 2nd Workshop on Workflows in Support of Large-Scale Science—16th International Symposium on High Performance Distributed Computing, HPDC 2007, 21–28 (2007). https://doi.org/10.1145/1273360.1273365

  • Moreno, R., Pérez-Gil, F., Pardo, J.J., Navarro, A., Tapiador, F.J.: Science for everyone (ScifE): a proposed framework for science as a service using interactive web technologies. Comput. Geosci. 131, 70–79 (2019)

    Article  Google Scholar 

  • Pathak, R., Dasari, H.P., Mohtar, S.E., Subramanian, A.C., Hoteit, I.: Uncertainty quantification and Bayesian inference of cloud parameterization in the NCAR single column community atmosphere model (SCAM6). Front. Clim. 3, 670740 (2021)

  • Pathak, R., Sahany, S., Mishra, S.K.: Uncertainty quantification based cloud parameterization sensitivity analysis in the NCAR community atmosphere model. Sci. Rep. 10(1), 1–17 (2020)

    Article  Google Scholar 

  • Rew, R.K., Davis, G.P.: The unidata netCDF: software for scientific data access (1990)

  • Rotstein, M., Rostkier-Edelstein, D., Alpert, P.: Factor separation analysis of the diurnal temperature range using the WRF single column model. In: EGU 2018 (2018)

  • Sj, A., Sr, B., Ak, C., As, D., Rb, E., Tc, F., Pk, G., Pks, H.: Prediction of temperature for various pressure levels using ANN and multiple linear regression techniques: a case study (2022)

  • Sun, M., Zhang, J., Zhang, W.: Alternating traveltime tomography and waveform inversion for near-surface imaging. In: SEG 2017 Workshop: Full-Waveform Inversion and Beyond, Beijing, China, 20–22 November 2017 (2017)

  • Theil, H.: A rank-invariant method of linear and polynomial regression analysis. Nederl. Akad. Wetensch. Proc. 12(2), 345–381 (1992)

    Google Scholar 

  • Turuncoglu, U., Murphy, S., Deluca, C., Dalfes, N.: A scientific workflow environment for earth system related studies. Comput. Geosci. 37, 943–952 (2011)

    Article  Google Scholar 

  • UCAR: AMWG Diagnostics Package. (2014). https://www.cesm.ucar.edu/working_groups/Atmosphere/amwg-diagnostics-package/documentation.html

  • Xu, K., Chan, Y., Wang, S.: Refactoring and optimizing WRF model on Sunway TaihuLight (2019)

  • Zaknich, A.: Neural Networks for Intelligent Signal Processing || General Regression Neural Network. (2003)

  • Zhang, Y., Wu, L.: Stock market prediction of s&p 500 via combination of improved BCO approach and BP neural network. Expert Syst. Appl. 36(5), 8849–8854 (2009)

    Article  Google Scholar 

  • Zheng, J., Wu, W., Yuan, S., Fu, H., Yu, L.: Multisource-domain generalization-based oil palm tree detection using very-high-resolution (VHR) satellite images. IEEE Geosci. Remote Sens. Lett. PP(99), 1–5 (2021)

    Google Scholar 

Download references

Acknowledgements

This work is funded by: National Key R&D Plan of China under Grant No. 2017YFA0604500, and by National Sci-Tech Support Plan of China under Grant No. 2014BAH02F00, and by National Natural Science Foundation of China under Grant No. 61701190, and by Youth Science Foundation of Jilin Province of China under Grant No. 20160520011JH and 20180520021JH, and by Youth Sci-Tech Innovation Leader and Team Project of Jilin Province of China under Grant No. 20170519017JH, and by Key Technology Innovation Cooperation Project of Government and University for the whole Industry Demonstration under Grant No. SXGJSF2017-4, and by Key scientific and technological R&D Plan of Jilin Province of China under Grant No. 20180201103GX.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Jilin University, Changchun, Jilin, China

    Jiaxu Guo, Xianwei Wu, Liang Hu & Xilong Che

  2. Department of Earth System Science, Ministry of Education Key Laboratory for Earth System Modeling, Institute for Global Change Studies, Tsinghua University, Beijing, China

    Yidan Xu & Haohuan Fu

  3. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Wei Xue & Lin Gan

  4. National Supercomputing Center in Wuxi, Wuxi, Jiangsu, China

    Jiaxu Guo, Yidan Xu, Haohuan Fu, Wei Xue, Lin Gan, Mengxuan Tan, Tingye Wu, Yutong Shen & Xianwei Wu

Authors
  1. Jiaxu Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yidan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haohuan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lin Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mengxuan Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tingye Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yutong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xianwei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Liang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Xilong Che
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xilong Che.

Ethics declarations

conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, J., Xu, Y., Fu, H. et al. GEO-WMS: an improved approach to geoscientific workflow management system on HPC. CCF Trans. HPC 5, 360–373 (2023). https://doi.org/10.1007/s42514-022-00131-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-022-00131-x

Keywords

Search

Navigation