Skip to main content
SpringerLink
Log in

Mitigating Uncertainty in Developing and Applying Scientific Applications in an Integrated Computing Environment

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

Effective solving complex mathematical modeling problems is based on the use of high-performance computing. Clouds, grids, and public access supercomputer centers are commonly used platforms. Their integration into a unified environment provides possibilities for carrying out mass large-scale scientific experiments and efficient scalable resource allocation at different stages of the application design and execution. However, end-users have to carefully select optimization criteria such as completion time, deadlines, reliability, cost, etc. It is a complicated problem due to integrated resources differ significantly in their computing capabilities, hardware and software platforms, system architectures, user interfaces, etc. The paper presents new features of the Orlando Tools framework for the development of distributed applied software packages (scalable scientific applications) that mitigates various types of uncertainties arising from the job distribution in the integrated computing environment. It provides continuous integration, delivery, and deployment of applied and system software to significantly mitigate the negative impact of uncertainty on problem-solving time, computation reliability, and resource efficiency. An experimental analysis of the sustainable design and development of the real energy sector clearly demonstrates the advantages of the tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.

Similar content being viewed by others

REFERENCES

  1. Inggs, G., Thomas, D.B., and Luk, W., A domain specific approach to high performance heterogeneous computing, IEEE Trans. Parallel Distrib. Syst., 2017, vol. 28, no. 1, pp. 2–15.

    Article  Google Scholar 

  2. Il’in, V., Artificial intelligence problems in mathematical modeling, Commun. Comput. Inf. Sci., 2019, vol. 1129, pp. 505–516.

    Google Scholar 

  3. Seinstra, F.J., Maassen, J., van Nieuwpoort, R.V., Drost, N., van Kessel, T., and van Werkhoven, B., Jungle computing: distributed supercomputing beyond clusters, grids, and clouds, in Grids, Clouds and Virtualization. Computer Communications and Networks, London: Springer, 2011, pp. 167–197.

    Google Scholar 

  4. Wang, L., Jie, W., and Chen, J., Grid Computing: Infrastructure, Service, and Applications, CRC Press, 2018.

    Book  Google Scholar 

  5. Varshney, S., Sandhu, R., and Gupta, P.K., QoS based resource provisioning in cloud computing environment: a technical survey, in Proc. Int. Conf. on Advances in Computing and Data Sciences, Singapore: Springer, 2019, pp. 711–723.

  6. Voevodin, Vl.V., Antonov, A.S., Nikitenko, D.A., Shvets, P.A., Sobolev, S.I., Sidorov, I.Yu., Stefanov, K.S., Voevodin, V.V., and Zhumatiy, S.A., Supercomputer Lomonosov-2: large scale, deep monitoring and fine analytics for the user community, Supercomput. Front. Innovations, 2019, vol. 6, no. 2, pp. 4–11.

    Google Scholar 

  7. Shabanov, B.M. and Samovarov, O.I., Building the software-defined data center, Program. Comput. Software, 2019, vol. 45, no. 8, pp. 458–466.

    Article  Google Scholar 

  8. Mateescu, G., Gentzsch, W., and Ribben, C.J., Hybrid computing – where HPC meets grid and cloud computing, Future Gener. Comput. Syst., 2011, vol. 27, no. 5, pp. 440–453.

    Article  Google Scholar 

  9. Feoktistov, A., Gorsky, S., Sidorov, I., Kostromin, R., Edelev, A., and Massel, L., Orlando tools: energy research application development through convergence of grid and cloud computing, Commun. Comput. Inf. Sci., 2019, vol. 965, pp. 289–300.

    Google Scholar 

  10. Feoktistov, A., Kostromin, R., Sidorov, I., and Gorsky, S., Development of distributed subject-oriented applications for cloud computing through the integration of conceptual and modular programming, in Proc. 41st Int. Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO-2018), Riejka: IEEE, 2018, pp. 256–261.

  11. Yu, J. and Buyya, R., A taxonomy of workflow management systems for grid computing, J. Grid Comput., 2005, vol. 3, no. 3–4, pp. 171–200.

    Article  Google Scholar 

  12. Feoktistov, A., Sidorov, I., Tchernykh, A., Edelev, A., Zorkalzev, V., Gorsky, S., Kostromin, R., Bychkov, I., and Avetisyan, A., Multi-agent approach for dynamic elasticity of virtual machines provisioning in heterogeneous distributed computing environment, Proc. IEEE Int. Conf. on High Performance Computing and Simulation (HPCS-2018), Orleans, 2018, pp. 909–916.

  13. Bychkov, I., Oparin, G., Feoktistov, A., Sidorov, I., Gorsky, S., Kostromin, R., and Edelev, E., Subject-oriented computing environment for solving large-scale problems of energy security research, J. Phys.: Conf. Ser., 2019, vol. 1368, pp. 052030-1–052030-12.

    Google Scholar 

  14. Burri, A., Dedner, A., Klofkorn, R., and Ohlberger, M., An efficient implementation of an adaptive and parallel grid in DUNE, Comput. Sci. High Perform. Comput. II: Notes Num. Fluid Mech. Multidiscipl. Des., 2006, vol. 91, pp. 67–82.

    Google Scholar 

  15. Radchenko, G. and Hudyakova, E., A service-oriented approach of integration of computer-aided engineering systems in distributed computing environments, Proc. UNICORE Summit, Dresden, 2012, pp. 57–66.

  16. Shamakina, A., Brokering service for supporting problem-oriented grid environments, Proc. UNICORE Summit, Dresden, 2012, pp. 67–75.

  17. Software for Exascale Computing-SPPEXA 2013-2015, Bungartz, H.J., Neumann, P., and Nagel, W.E., Eds., Cham: Springer, 2016, vol. 113.

    Google Scholar 

  18. Afgan, E., et al., The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update, Nucl. Acids Res., 2018, vol. 46, no. W1, pp. W537–W544.

    Article  Google Scholar 

  19. Ananthakrishnan, R., Blaiszik, B., Chard, K., and Chard, R., Globus platform services for data publication, Proc. ACM Conf. of the Practice and Experience on Advanced Research Computing, Pittsburgh, 2018, pp. 1–7.

  20. Sukhoroslov, O., Supporting efficient execution of workflows on Everest Platform, Commun. Comput. Inf., 2019, vol. 1129, pp. 713–724.

    Google Scholar 

  21. Gavvala, S.K., Chandrasheka, J., Gangadharan, G.R., and Buyya, R., QoS-aware cloud service composition using eagle strategy, Future Gener. Comput. Syst., 2019, vol. 90, pp. 273–290.

    Article  Google Scholar 

  22. Deelman, E., Peterka, T., Altintas, I., and Carothers, C.D., The future of scientific workflows, Int. J. High Perform. Comput. Appl., 2018, vol. 32, no. 1, pp. 159–175.

    Article  Google Scholar 

  23. Abramovici, A., et al., LIGO: the laser interferometer gravitational-wave observatory, Science, 1992, vol. 256, no. 5005, pp. 325–333.

    Article  Google Scholar 

  24. Berriman, G.B., et al., Montage: a grid enabled engine for delivering custom science-grade mosaics on demand, Proc. SPIE – Int. Soc. Opt. Eng., 2004, vol. 5493. https://doi.org/10.1117/12.550551

  25. Maechling, P., et al., SCEC CyberShake workflows-automating probabilistic seismic hazard analysis calculations, in Workflows for e–Science, Springer, 2006. https://doi.org/10.1007/978-1-84628-757-2_10

    Book  Google Scholar 

  26. Livny, J., Teonadi, H., Livny, M., and Waldor, M.K., High-throughput, kingdom-wide prediction and annotation of bacterial non-coding RNAs, PLoS One, 2008, vol. 3, no. 9, pp. e3197. https://doi.org/10.1371/journal.pone.0003197

    Article  Google Scholar 

  27. USC Epigenome Center. http://epigenome.usc.edu. Accessed 08.12.2019.

  28. Wangsom, P., Lavangnananda, K., and Bouvry, P., Multi-objective scientific-workflow scheduling with data movement awareness in cloud, IEEE Access, 2019, vol. 7, pp. 177063–177081.

    Article  Google Scholar 

  29. Feoktistov, A., Gorsky, S., Sidorov, I., and Tchernykh, A., Continuous integration in distributed applied software packages, Proc. 42st Int. Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO-2019), Riejka: IEEE, 2019, pp. 1775–1780.

  30. Gruver, G., Start and Scaling Devops in the Enterprise, BookBaby, 2016.

    Google Scholar 

  31. Talia, D., Workflow systems for science: concepts and tools, ISRN Software Eng., 2013, art. ID 404525. https://doi.org/10.1155/2013/404525

  32. Deelman, E., et al., Pegasus, a workflow management system for science automation, Future Gener. Comput. Syst., 2015, vol. 46, pp. 17–35.

    Article  Google Scholar 

  33. Bumgardner, V.K., OpenStack in Action, Shelter Island: Manning Publ., 2016.

    Google Scholar 

  34. Spruth, I.W.G., Discovering and classifying regions in workflow graphs, Diploma Thesis in Computer Science, Publ. of the University of Tubingen, 2005.

  35. Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Gaurang, S., and Mei-Hui, V.K., Characterization of scientific workflows, Proc. 3rd Workshop on Workflows in Support of Large-Scale Science (WORKS 2008), Austin, 2008, doi 1-10.https://doi.org/10.1109/WORKS.2008.4723958

  36. Hirales-Carbajal, A., González-García, J.L., and Tchernykh, A., Workload generation for trace based grid simulations, in Proc. 1st Int. Supercomputer Conf. in Mexico (ISUM–2010), Guadalajara University Publ., 2010, pp. 1–10.

  37. Bychkov, I., Oparin, G., Tchernykh, A., Feoktistov, A., Bogdanova, V., and Gorsky, S., Conceptual model of problem-oriented heterogeneous distributed computing environment with multi-agent managemen, Procedia Comput. Sci., 2017, vol. 103, pp. 162–167.

    Article  Google Scholar 

  38. Sokolinsky, L.B. and Shamakina, A.V., Methods of resource management in problem-oriented computing environment, Program. Comput. Software, 2016, vol. 42, no. 1, pp. 17–26.

    Article  MathSciNet  Google Scholar 

  39. Ramírez-Velarde, R., Tchernykh, A., Barba-Jimenez, C., Hirales-Carbajal, A., and Nolazco, J., Adaptive resource allocation with job runtime uncertainty, J. Grid Comput., 2017, vol. 15, no. 4, pp. 415–434.

    Article  Google Scholar 

  40. Tchernykh, A., Schwiegelshohn, U., Talbi, E.-g., and Babenko, M., Towards understanding uncertainty in cloud computing with risks of confidentiality, integrity, and availability, J. Comput. Sci., 2019, vol. 36, p. 100581. https://doi.org/10.1016/j.jocs.2016.11.011

    Article  Google Scholar 

  41. Babenko, M., Chervyakov, N., Tchernykh, A., Kucherov, N., Shabalina, M., Vashchenko, I., Radchenko, G., and Murga, D., Unfairness correction in P2P grids based on residue number system of a special form, Proc. 28th IEEE Int. Workshop on Database and Expert Systems Applications (DEXA), Lyon, 2017, pp. 147–151.

  42. Singh, A. and Malhotra, M., Agent based framework for scalability in cloud computing, Int. J. Comput. Sci. Eng., 2012, vol. 3, no. 4, pp. 41–45.

    Google Scholar 

  43. Kalyaev, A.I. and Kalyaev, I.A., Method of multiagent scheduling of resources in cloud computing environments, J. Comput. Syst. Sci. Int., 2016, vol. 55, no. 2, pp. 211–221.

    Article  Google Scholar 

  44. Prieto, A.G., Gillblad, D., Steinert, R., and Miron, A., Toward decentralized probabilistic management, IEEE Commun. Mag., 2011, vol. 49, no. 7, pp. 80–86.

    Article  Google Scholar 

  45. Walsh, A., UDDI, SOAP, and WSDL: the Web Services Specification Reference Book, Pearson Education, 2002.

    Google Scholar 

  46. Bychkov, I.V., Oparin, G.A., Feoktistov, A.G., Sidorov, I.A., Bogdanova, V.G., and Gorsky, S.A., Multiagent control of computational systems on the basis of meta-monitoring and imitational simulation, Optoelectron., Instrum. Data Process., 2016, vol. 52, no. 2, pp. 107–112.

    Article  Google Scholar 

  47. Java Agent DEvelopment Framework. https://jade.tilab.com. Accessed 08.12.2019.

  48. Herrera, J., Huedo, E., Montero, R., and Llorente, I., Porting of scientific applications to grid computing on GridWay, Sci. Program., 2005, vol. 13, no. 4, pp. 317–331.

    Google Scholar 

  49. Tannenbaum, T., Wright, D., Miller, K., and Livny, M., Condor – a Distributed Job Scheduler. Beowulf Cluster Computing with Linux, The MIT Press, 2002, pp. 307–350.

    Google Scholar 

  50. Feoktistov, A., Tchernych, A., Kostromin, R., and Gorsky, S., Knowledge elicitation in multi-agent system for distributed computing management, Proc. 40th Int. Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO-2017), Riejka: IEEE, 2017, pp. 1350–1355.

  51. Feoktistov, A., Kostromin, R., Sidorov, I., Gorsky, S., and Oparin, G., Multi-agent algorithm for re-allocating grid-resources and improving fault-tolerance of problem-solving processes, Procedia Comput. Sci., 2019, vol. 150, pp. 171–178.

    Article  Google Scholar 

  52. Vickrey, W., Counterspeculation, auctions, and competitive sealed tenders, J. Finance, 1961, vol. 16, no. 1, pp. 8–37.

    Article  MathSciNet  Google Scholar 

  53. Edelev, A., Zorkaltsev, V., Gorsky, S., Doan, V.B., and Nguyen, H. N., The combinatorial modelling approach to study sustainable energy development of Vietnam, Commun. Comput. Inf. Sci., 2017, vol. 793, pp. 207–218.

    Google Scholar 

  54. Irkutsk Supercomputer Centre of SB RAS. http://hpc.icc.ru. Accessed 08.12.2019.

  55. Tchernykh, A., Feoktistov, A., Gorsky, S., Sidorov, I., Kostromin, R., Bychkov, I., Basharina, O., Alexandrov, A., and Rivera-Rodriguez, R., Orlando tools: development, training, and use of scalable applications in heterogeneous distributed computing environments, Commun. Comput. Inf. Sci., 2019, vol. 979, pp. 265–279.

    Google Scholar 

  56. Bychkov, I.V., Oparin, G.A., Tchernykh, A.N., Feoktistov, A.G., Gorsky, S.A., and Rivera-Rodriguez, R., Scalable application for the search of global minima of multiextremal functions, Optoelectron., Instrum. Data Process., 2018, vol. 54, no. 1, pp. 83–89.

    Article  Google Scholar 

Download references

ACKNOWLEDGMENTS

The study is supported by the Russian Foundation of Basic Research, projects nos. 19-07-00097 and 18-07-01224. The development of meta-monitoring and resource allocation agents was supported in part by the Basic Research Program of SB RAS, project no. IV.38.1.1.

Author information

Authors and Affiliations

  1. CICESE Research Center, Carr Tijuana-Ensenada 3918, Zona Playitas, 22860, Ensenada, B.C., Mexico

    A. Tchernykh

  2. South Ural State University, Chelyabinsk, 454080, Chelyabinsk, Russia

    A. Tchernykh

  3. Ivannikov Institute for System Programming of RAS, 109004, Moscow, Russia

    A. Tchernykh & A. Avetisyan

  4. Matrosov Institute for System Dynamics and Control Theory, SB of RAS, 664033, Irkutsk, Russia

    I. Bychkov, A. Feoktistov, S. Gorsky, I. Sidorov & R. Kostromin

  5. Melentiev Energy Systems Institute, SB of RAS, 664033, Irkutsk, Russia

    A. Edelev

  6. Limnological Institute, SB of RAS, 664033, Irkutsk, Russia

    V. Zorkalzev

Authors
  1. A. Tchernykh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. I. Bychkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Feoktistov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. Gorsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. I. Sidorov
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. R. Kostromin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. A. Edelev
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. V. Zorkalzev
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. A. Avetisyan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to A. Tchernykh, I. Bychkov, A. Feoktistov, S. Gorsky, I. Sidorov, R. Kostromin, A. Edelev, V. Zorkalzev or A. Avetisyan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tchernykh, A., Bychkov, I., Feoktistov, A. et al. Mitigating Uncertainty in Developing and Applying Scientific Applications in an Integrated Computing Environment. Program Comput Soft 46, 483–502 (2020). https://doi.org/10.1134/S036176882008023X

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S036176882008023X

Search

Navigation