Skip to main content
Springer Nature Link
Log in

Framework for Development and Execution of Scientific WorkFlows: Designing Service-Oriented Applications

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

The development and use of scientific applications have become an integral part of conducting large-scale experiments in various fields of research that require high-performance computing and big data processing. In the context of developing such applications, non-trivial problems arise in the concerted description and further use of schemes, software, and computational resources to solve subject domain problems of a specific application. Research productivity has become highly dependent on the degree of automation in the preparation and execution of experiments in a computing environment whose resources may be distributed and heterogeneous. Many approaches to the experiment automation are based on workflows as a structure for formalizing and specifying data processing and high-performance computing using distributed applications. Within such approaches, developers and end-users work with workflow management systems for the collaborative development and use of distributed scientific applications. Nowadays, service-oriented applications are coming to the fore. However, there is a wide range spectrum of problems related to the support of modular scientific applications, the standardization of their components and interfaces, the use of heterogeneous information and computing resources, and organization of interdisciplinary research within service-oriented architecture. Known workflow management systems do not fully address the above problems. In this regards, we consider relevant aspects of organizing service-oriented computing in a heterogeneous distributed computing environment. We propose a new framework for creating service-oriented and workflow-based scientific applications. The paper shows that the proposed framework significantly extends and complements the capabilities of systems for such purposes. We also demonstrate the reduction in labour costs associated with the preparation and execution of experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

REFERENCES

  1. Erwin, D.W. and Snelling, D.F., UNICORE: A grid computing environment, Lect. Notes Comput. Sci., 2001, vol. 2150, pp. 825–834. https://doi.org/10.1007/3-540-44681-8_116

    Article  MATH  Google Scholar 

  2. Litzkow, M.J., Livny, M., and Mutka, M.W. Condor–A hunter of idle workstations, Proc. 8th Int. Conf. on Distributed Computing Systems, Institute of Electrical and Electronics Engineers, San Jose, 1988, pp. 104–111. https://doi.org/10.1109/DCS.1988.12507.

  3. Deelman, E., Vahi, K., and Juve, G., et al., Pegasus, a workflow management system for science automation, Future Gener. Comput. Syst., 2015, vol. 46, pp. 17–35. https://doi.org/10.1016/j.future.2014.10.008

    Article  Google Scholar 

  4. Talia, D., Workflow systems for science: Concepts and tools, ISRN Software Eng., 2013, no. 1, p. 404525. https://doi.org/10.1155/2013/404525

  5. Da Silva, R.F., Filgueira, R., Pietri, I., Jiang, M., Sakellariou, R., and Deelman, E., A characterization of workflow management systems for extreme-scale applications, Future Gener. Comput. Syst., 2017, vol. 75, pp. 228–238. https://doi.org/10.1016/j.future.2017.02.026

    Article  MATH  Google Scholar 

  6. Brown, A., Johnston, S., and Kelly, K., Using Service-Oriented Architecture and Component-Based Development to Build Web Service Applications, Rational Software Corp., 2002, vol. 6.

    MATH  Google Scholar 

  7. Afgan, E., Baker, D., Coraor, N., Chapman, B., Nekrutenko, A., and Taylor, J., Galaxy CloudMan: Delivering cloud compute clusters, BMC Bioinf., 2010, vol. 11, no. 12, pp. 1–6. https://doi.org/10.1186/1471-2105-11-S12-S4

    Article  Google Scholar 

  8. Balis, B., HyperFlow: A model of computation, programming approach and enactment engine for complex distributed workflows, Future Gener. Comput. Syst., 2016, vol. 55, pp. 147–162. https://doi.org/10.1016/j.future.2015.08.015

    Article  Google Scholar 

  9. Hilman, M.H., Rodriguez, M.A., and Buyya, R., Workflow-as-a-service cloud platform and deployment of bioinformatics workflow applications, in Knowledge Management in Development of Data-Intensive Systems, Mistrik, I., Galster, M., Maxim, B., and Tekiner-Dogan, B., Eds., Boca Raton, FL: CRC Press, 2021, pp. 205–226.

    MATH  Google Scholar 

  10. Papazoglou, M., Web Services: Principles and Technology, New York: Pearson Education, 2008.

    MATH  Google Scholar 

  11. Welke, R., Hirschheim, R., and Schwarz, A., Service-oriented architecture maturity, Computer, 2011, vol. 44, no. 2, pp. 61–67. https://doi.org/10.1109/MC.2011.56

    Article  Google Scholar 

  12. Tsalgatidou, A. and Pilioura, T., An overview of standards and related technology in web services, Distrib. Parallel Databases, 2002, vol. 12, pp. 135–162. https://doi.org/10.1023/A:1016599017660

    Article  MATH  Google Scholar 

  13. Ananthakrishnan, R., Chard, K., Foster, I., and Tuecke, S., Globus platform-as-a-service for collaborative science applications, Concurr. Comput.-Pract. E, 2015, vol. 27, no. 2, pp. 290–305. https://doi.org/10.1002/cpe.3262

    Article  MATH  Google Scholar 

  14. Foster, I., Globus Online: Accelerating and democratizing science through cloud-based services, IEEE Internet Comput., 2011, vol. 15, no. 3. pp. 70–73. https://doi.org/10.1109/MIC.2011.64

    Article  MATH  Google Scholar 

  15. Foster, I., Globus toolkit version 4: Software for service-oriented systems, J. Comput. Sci. Technol., 2006, vol. 21, pp. 513–520. https://doi.org/10.1007/s11390-006-0513-y

    Article  MATH  Google Scholar 

  16. Foster, I. And Kesselman, C., The Grid: Blueprint for a New Computing Infrastructure, Morgan-Kaufmann, 2002.

    MATH  Google Scholar 

  17. Foster, I. And Kesselman, C., Globus: A metacomputing infrastructure toolkit, Int. J. Supercomput. Appl., 1997, vol. 11, no. 2, pp. 115–128. https://doi.org/10.1177/109434209701100205

    Article  MATH  Google Scholar 

  18. Juric, M.B., Chandrasekaran, S., Frece, A., Hertis, M., and Srdic, G., WS-BPEL 2.0 for SOA Composite Applications with Oracle SOA Suite 11g, Packt Publishing Ltd., 2010.

    Google Scholar 

  19. Juric, M.B., Mathew, B., and Sarang, P.G., Business Process Execution Language for Web Services: an Architect and Developer’s Guide to Orchestrating Web Services Using BPEL4WS, Packt Publishing Ltd., 2006.

    MATH  Google Scholar 

  20. Kim, S.J., Foundation for composablemicroservices for rapid synthesis of highly reliable software systems, PhD Thesis, Dallas: Univ. of Texas, 2004.

  21. Kim, S., Bastani, F.B., Yen, I.L., and Chen, I.-R., High-assurance synthesis of security services from basic microservices, Proc. 14th IEEE Int. Symp. on Software Reliability Engineering (ISSRE 2003), Denver, 2003, pp. 154–165. https://doi.org/10.1109/ISSRE.2003.1251039.

  22. Fielding, R.T., Architectural styles and the design of network-based software architectures, Ph. D. Thesis, Irvine: Univ. of California, 2000.

  23. Rajasekar, A., iRODS Primer: Integrated Rule-Oriented Data System, Morgan & Claypool Publ., 2010.

  24. Sukhoroslov, O., Building web-based services for practical exercises in parallel and distributed computing, J. Parallel Distrib. Comput., 2018, vol. 118, pp. 177–188. https://doi.org/10.1016/j.jpdc.2018.02.024

    Article  MATH  Google Scholar 

  25. Savchenko, D.I., Radchenko, G.I., and Taipale, O., Microservices validation: Mjolnirr platform case study, Proc. 38th IEEE Int. Convention on Information and Communication Technology, Electronics and Microelectronics Conf. (MIPRO), Opatija, 2015, pp. 235–240. https://doi.org/10.1109/MIPRO.2015.7160271.

  26. Smirnov, P.A., Kovalchuk, S.V., and Boukhanovsky, A.V., Knowledge-based support for complex systems exploration in distributed problem solving environments, Commun. Comput. Inf. Sci., 2013, vol. 394, pp. 147–161. https://doi.org/10.1007/978-3-642-41360-5_12

    Article  MATH  Google Scholar 

  27. Knyazkov, K.V., Kovalchuk, S.V., Tchurov, T.N., Maryin, S.V., and Boukhanovsky, A.V., CLAVIRE: e-Science infrastructure for data-driven computing, J. Comput. Sci., J. Comput. Sci.-Neth., 2012, vol. 3, no. 6, pp. 504–510. https://doi.org/10.1016/j.jocs.2012.08.006

  28. Puzyrkov, D.V., Podryga, V.O., and Polyakov, S.V., Cloud service for HPC management: Ideas and appliance, Lobachevskii J. Math., 2018, vol. 39, no. 9, pp. 1251–1261. https://doi.org/10.1134/S1995080218090172

    Article  MathSciNet  MATH  Google Scholar 

  29. Kudryavtsev, A.O., Koshelev, V.K., Izbyshev, A.O., et al., HPC cloud system design and implementation, Proc. ISP RAS, 2013, vol. 24, pp. 13–34. https://ispranproceedings.elpub.ru/jour/article/download/948/673. Accessed June 17, 2024

  30. Sorokin, A.A., Makogonov, S.V., and Korolev, S.P., The information infrastructure for collective scientific work in the Far East of Russia, Sci. Tech. Inf. Process., 2017, vol. 44, no. 4, pp. 302–304. https://doi.org/10.3103/S0147688217040153

    Article  MATH  Google Scholar 

  31. Korolev, S.P., Sorokin, A.A., Verkhoturov, A.L., Konovalov, A.V., and Shestakov, N.V., Automated information system for instrument-data processing of the regional seismic observation network of FEB RAS, Seism, Instrum., 2015, vol. 51, no. 3, pp. 209–218. https://doi.org/10.3103/S0747923915030068

    Article  Google Scholar 

  32. Shokin, Y.I., Fedotov, A.M., and Zhizhimov, O.L., Technologies for designing of distributed information systems to support research, Comput. Technol., 2015, vol. 20, no. 5, pp. 251–274. http://www.ict.nsc.ru/jct/content/t20n5/Fedotov_Zhizhimov_n1.pdf. Accessed June 17, 2024.

    MATH  Google Scholar 

  33. Massel, L.V., Massel, A.G., and Tsybikov, A.R., Agent-service approach to building digital twins, Proc. IEEE Int. Russian Smart Industry Conf. (SmartIndustryCon), Sochi, 2024.

  34. Bychkov, I.V., Ruzhnikov, G.M., Paramonov, V.V., Shumilov, A.S., Fedorov, R.K., Levi, K.G., and Demberel, S., Infrastructural approach and geospatial data processing services in the tasks of territorial development management, IOP Conf. Ser.: Earth Environ. Sci., 2018, vol. 190, no. 1, p. 012048. https://doi.org/10.1088/1755-1315/190/1/012048

  35. Bychkov, I.V., Ruzhnikov, G.M., Fedorov, R.K., Khmelnov, A.E., and Popova, A.K., Organization of digital monitoring of the Baikal natural territory, IOP Conf. Ser.: Earth Environ. Sci., 2021, vol. 629, no. 1. p. 012067. https://doi.org/10.1088/1755-1315/629/1/012067

  36. Bychkov, I.V., Oparin, G.A., Feoktistov, A.G., Bogdanova, V.G., and Pashinin, A.A., Service-oriented multiagent control of distributed computations, Automat. Remote Control, 2015, vol. 76, no. 11, pp. 2000–2010. https://doi.org/10.1134/S0005117915110090

    Article  MATH  Google Scholar 

  37. Feoktistov, A., Gorsky, S., Kostromin, R., Fedorov, R., and Bychkov, I., Integration of web processing services with workflow-based scientific applications for solving environmental monitoring problems, ISPRS Int. J. Geo-Inf., 2022, vol. 11, no. 1, p. 8. https://doi.org/10.3390/ijgi11010008

    Article  Google Scholar 

  38. Kostromin, R., Basharina, O., Feoktistov, A., and Sidorov, I., Microservice-based approach to simulating environmentally-friendly equipment of infrastructure objects taking into account meteorological data, Atmosphere, 2021, vol. 12, no. 9, p. 1217. https://doi.org/10.3390/atmos12091217

    Article  Google Scholar 

  39. Yoo, T.J., State of the art in business process modeling and execution standard, Adv. Sci. Lett., 2016, vol. 22, no. 11, pp. 3650–3653. https://doi.org/10.1166/asl.2016.7904

    Article  MATH  Google Scholar 

  40. Wohlstadter, E., Tai, S., Mikalsen, T., Diament, J., and Rouvellou, I., A service-oriented middleware for runtime web services interoperability, Proc. IEEE Int. Conf. on Web Services, Chicago, 2006, pp. 1–8. https://doi.org/10.1109/ICWS.2006.13.

  41. Web Services Business Process Execution Language Version 2.0. https://docs.oasis-open.org/wsbpel/2.0/wsbpel-v2.0.pdf. Accessed June 17, 2024.

  42. Common Workflow Language (CWL). https://www.commonwl.org. Accessed June 17, 2024.

  43. Feoktistov, A., Edelev, A., Tchernykh, A., Gorsky, S., Basharina, O., and Fereferov, E., An approach to implementing high-performance computing for problem solving in workflow-based energy infrastructure resilience studies, Computation, 2023, vol. 11, no. 12, p. 243. https://doi.org/10.3390/computation11120243

    Article  Google Scholar 

  44. Tchernykh, A., Bychkov, I., Feoktistov, A., Gorsky, S., Sidorov, I., Kostromin, R., Edelev, A., Zorkalzev, V., and Avetisyan, A., Mitigating uncertainty in developing and applying scientific applications in an integrated computing environment, Program. Comput. Software, 2020, vol. 46, no. 8, pp. 483–502. https://doi.org/10.1134/S036176882008023X

    Article  Google Scholar 

  45. Apache Airflow. https://airflow.apache.org/. Accessed June 17, 2024.

  46. Feoktistov, A.G., Kostromin, R.O., Voskoboinikov, M.L., and Li-De, D.I., Implementation of computing environment implementation for developing and applying scientific workflows based on containerization, Comput. Technol., 2023, vol. 28, no. 6, pp. 151–164. https://doi.org/10.25743/ICT.2023.28.6.013

    Article  Google Scholar 

  47. Danilov, G. and Voskoboinikov, M., Testbed-based approach to testing a library for evaluating network reliability algorithms, in Proc. Int. Workshop on Critical Infrastructures in the Digital Worl (IWCI-2024), ESI SB RAS, 2024, pp. 3–4.

  48. Alaasam, A.B., Radchenko, G., and Tchernykh, A., Refactoring the monolith workflow into independent micro-workflows to support stream processing, Program. Comput. Software, 2021, vol. 47, no. 8, pp. 591–600. https://doi.org/10.1134/S0361768821080077

    Article  MathSciNet  Google Scholar 

Download references

Funding

The study was supported by the Ministry of Science and Higher Education of the Russian Federation, project no. FWEW-2021-0005 “Technologies for the development and analysis of subject-oriented intelligent group control systems in non-deterministic distributed environments” (reg. no. 121032400051-9).

Author information

Authors and Affiliations

  1. Matrosov Institute for System Dynamics and Control Theory, Siberian Branch, Russian Academy of Sciences, Irkutsk, Russia

    A. Feoktistov & M. Voskoboinikov

  2. CICESE Research Center, Ensenada, Mexico

    A. Tchernykh

  3. Ivannikov Institute for System Programming of RAS, Moscow, Russia

    A. Tchernykh

Authors
  1. A. Feoktistov
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. M. Voskoboinikov
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. A. Tchernykh
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to A. Feoktistov, M. Voskoboinikov or A. Tchernykh.

Ethics declarations

The authors of this work declare that they have no conflicts of interest.

Additional information

Publisher’s Note.

Pleiades Publishing remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

AI tools may have been used in the translation or editing of this article.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feoktistov, A., Voskoboinikov, M. & Tchernykh, A. Framework for Development and Execution of Scientific WorkFlows: Designing Service-Oriented Applications. Program Comput Soft 50, 900–913 (2024). https://doi.org/10.1134/S0361768824700828

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768824700828