Skip to main content
SpringerLink
Log in

Enabling Organizations to Participate in the Ad Hoc Scheduling of a Cross-Organizational Data Pipeline

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

In case of an emergency situation, it is required to come up with solutions quickly. The supporting decision-making process should therefore be based on relevant data sources which are fed to data processing pipelines. These data sources may however be located in different domains of distinct organizations. Although the technical realisation of cross-organizational data pipelines is possible, for example using a federated Kubernetes cluster managed by a central operator, it is unclear how those pipelines should be scheduled among the participating organizations. Firstly, the cross-organizational infrastructure is unknown and highly heterogeneous, and secondly, there may be undiscovered scheduling preferences present. The first issue can be solved using software probes, while the second issue will be solved through an extension of this probing concept. During the proposed inter-domain scheduling process, organizations may specify monetary reward requirements and requirements on the maximum load they wish to bear. The former allows administrators to specify a lower limit with respect to payment they request for their contribution, the latter allows the specification of an upper limit on the workload an organization wishes to process. This way, mismatching expectations on contribution level, which may potentially cause harm from the hosting organization perspective, are prevented, increasing the trust level of contributors. These kinds of requirements have nothing to do with the technical assessment of a node, but they do impact scheduling decisions and performance of an application, as shown in our evaluation. The proposed scheduling flow not only allows organizations to steer scheduling decisions, but also to negotiate requirements among each other, giving rise to ad hoc conflict resolution all the while collaborating in solving the emergency situation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

No supplementary content available.

References

  1. Kubernetes. https://kubernetes.io

  2. Goethals, T., Kerkhove, S., Van Hoye, L., Sebrechts, M., De Turck, F., Volckaert, B.: FUSE: a microservice approach to cross-domain federation using docker containers. In: Proceedings of the 9th International Conference on Cloud Computing and Services Science (CLOSER), pp. 90–99. SciTePress, Heraklion, Greece (2019) https://doi.org/10.5220/0007706000900099

  3. Qi, S., Kulkarni, S.G., Ramakrishnan, K.K.: Assessing container network interface plugins: functionality, performance, and scalability. IEEE Trans. Netw. Serv. Manag. 18(1), 656–671 (2021). https://doi.org/10.1109/TNSM.2020.3047545

    Article  Google Scholar 

  4. Adeppady, M., Giaccone, P., Karl, H., Chiasserini, C.F.: Reducing microservices interference and deployment time in resource-constrained cloud systems. IEEE Trans. Netw. Serv. Manag (2023). https://doi.org/10.1109/TNSM.2023.3235710

    Article  Google Scholar 

  5. FUSE: Flexible federated Unified Service Environment. https://www.imec-int.com/en/what-we-offer/research-portfolio/fuse

  6. Van Hoye, L., Wauters, T., De Turck, F., Volckaert, B.: Trustful ad hoc cross-organizational data exchanges based on the Hyperledger Fabric framework. Int. J. Netw. Manag. 30(6), 2131 (2020). https://doi.org/10.1002/nem.2131

    Article  Google Scholar 

  7. Van Hoye, L., Wauters, T., De Turck, F., Volckaert, B.: A secure cross-organizational container deployment approach to enable ad hoc collaborations. Int. J. Netw. Manag. 32(4), 2194 (2022). https://doi.org/10.1002/nem.2194

    Article  Google Scholar 

  8. Van Hoye, L., Wauters, T., De Turck, F., Volckaert, B.: Enabling the rescheduling of containerized workloads in an ad hoc cross-organizational collaboration. J. Netw. Syst. Manag. 31(1), 10 (2023). https://doi.org/10.1007/s10922-022-09699-9

    Article  Google Scholar 

  9. Mubeen, S., Asadollah, S.A., Papadopoulos, A.V., Ashjaei, M., Pei-Breivold, H., Behnam, M.: Management of service level agreements for cloud services in IoT: a systematic mapping study. IEEE Access 6, 30184–30207 (2018). https://doi.org/10.1109/ACCESS.2017.2744677

    Article  Google Scholar 

  10. Macías, M., Guitart, J.: SLA negotiation and enforcement policies for revenue maximization and client classification in cloud providers. Future Gener. Comput. Syst. 41, 19–31 (2014). https://doi.org/10.1016/j.future.2014.03.004

    Article  Google Scholar 

  11. Shojaiemehr, B., Rahmani, A.M., Qader, N.N.: A three-phase process for SLA negotiation of composite cloud services. Comput. Stand. Interfaces 64, 85–95 (2019). https://doi.org/10.1016/j.csi.2019.01.001

    Article  Google Scholar 

  12. Yan, J., Kowalczyk, R., Lin, J., Chhetri, M.B., Goh, S.K., Zhang, J.: Autonomous service level agreement negotiation for service composition provision. Future Gener. Comput. Syst. 23(6), 748–759 (2007). https://doi.org/10.1016/j.future.2007.02.004

    Article  Google Scholar 

  13. Omezzine, A., Bellamine Ben Saoud, N., Tazi, S., Cooperman, G.: Towards a generic multilayer negotiation framework for efficient application provisioning in the cloud. Concurr. Comput. Pract. Exp. 32(1), 4182 (2020). https://doi.org/10.1002/cpe.4182

    Article  Google Scholar 

  14. Samimi, P., Teimouri, Y., Mukhtar, M.: A combinatorial double auction resource allocation model in cloud computing. Inf. Sci. 357, 201–216 (2016). https://doi.org/10.1016/j.ins.2014.02.008

    Article  MATH  Google Scholar 

  15. Mao, Y., Xu, X., Wang, L., Ping, P.: Priority combinatorial double auction based resource allocation in the cloud. In: Proceedings of the IEEE Sixth International Conference on Big Data Computing Service and Applications (BigDataService), pp. 224–228. IEEE, Oxford. https://doi.org/10.1109/BigDataService49289.2020.00043 (2020)

  16. Li, L., Liu, L., Huang, S., Lv, S., Lin, K., Zhu, S.: Agent-based multi-tier SLA negotiation for intercloud. J. Cloud Comput. 11, 16 (2022). https://doi.org/10.1186/s13677-022-00286-6

    Article  Google Scholar 

  17. Li, L., Yeo, C.S., Hsu, C.-Y., Yu, L.-C., Lai, K.R.: Agent-based fuzzy constraint-directed negotiation for service level agreements in cloud computing. Clust. Comput. 21(2), 1349–1363 (2018). https://doi.org/10.1007/s10586-017-1248-y

    Article  Google Scholar 

  18. Wickboldt, J.A., Guerreiro, M.Q., Granville, L.Z., Gaspary, L.P., Schwarz, M.F., Guok, C., Chaniotakis, V., Lake, A., MacAuley, J.: MEICAN: simplifying DCN life-cycle management from end-user and operator perspectives in inter-domain environments. IEEE Commun. Mag. 56(1), 179–187 (2018). https://doi.org/10.1109/MCOM.2017.1601205

    Article  Google Scholar 

Download references

Acknowledgements

The work described in this paper, was partly funded by the FUSE research project [5], in which a Flexible federated Unified Service Environment was investigated. The project was realized in collaboration with imec. Industry project partners were Barco, Axians and e-BO Enterprises, with project support from VLAIO (Flanders Innovation & Entrepreneurship).

Funding

MOZAIK strategic basic research project (FWO Grant Number S003321N).

Author information

Author notes

  1. Laurens Van Hoye, Tim Wauters, Filip De Turck and Bruno Volckaert have contributed equally to this work.

Authors and Affiliations

  1. IDLab, Ghent University - imec, Technologiepark-Zwijnaarde 126, 9052, Ghent, Belgium

    Laurens Van Hoye, Tim Wauters, Filip De Turck & Bruno Volckaert

Authors
  1. Laurens Van Hoye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tim Wauters
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Filip De Turck
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bruno Volckaert
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to this work.

Corresponding author

Correspondence to Laurens Van Hoye.

Ethics declarations

Conflict of interest

No competing interests present.

Ethical Approval

Not applicable to this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Van Hoye, L., Wauters, T., De Turck, F. et al. Enabling Organizations to Participate in the Ad Hoc Scheduling of a Cross-Organizational Data Pipeline. J Netw Syst Manage 31, 44 (2023). https://doi.org/10.1007/s10922-023-09733-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-023-09733-4

Keywords

Search

Navigation