Skip to main content
SpringerLink
Log in

HYAS: Hybrid Autoscaler Agent for Apache Flink

  • Conference paper
  • First Online:
Web Engineering (ICWE 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13893))

Included in the following conference series:

Abstract

Apache Flink is a distributed processing engine for stateful computations over unbounded and bounded data streams. Despite its versatility, Apache Flink cannot automatically and optimally adjust its computing resources to match the requirements of the incoming workload. HYAS agent monitors and models Flink’s responsiveness based on changes in operator idleness, backpressure and input record lag. The decision process instructs Flink to adjust its parallelism to keep up with the workload. The performance of HYAS has been assessed experimentally on an Apache Flink deployment on Kubernetes on the Google Cloud Platform using synthetic and real-life workloads. HYAS successfully maintains application performance and provides a better performance-to-cost ratio than existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/apache/flink-kubernetes-operator.

  2. 2.

    https://incubator.apache.org/clutch/heron.html.

  3. 3.

    https://cloud.google.com/dataflow.

  4. 4.

    https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/.

  5. 5.

    https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/.

  6. 6.

    https://prometheus.io.

  7. 7.

    https://docs.oracle.com/en/java/javase/15/jmx/.

  8. 8.

    https://zookeeper.apache.org.

  9. 9.

    https://github.com/nimamahmoudi/worldcup98-dataset.

  10. 10.

    https://cloud.google.com/compute/all-pricing.

References

  1. Apache Flink - Stateful Computations over Data Streams (2022). https://flink.apache.org, The Apache Software Foundation

  2. Bodik, P., Griffith, R., Sutton, C., Fox, A., Jordan, M., Patterson, D.: Statistical machine learning makes automatic control practical for internet datacenters. In: Hot Topics in Cloud Computing (HoTCloud 2009), pp. 195–203. USENIX Association, San Diego, California, USA (2009). https://doi.org/10.5555/1855533.1855545

  3. Celebi, U.: How Apache Flink Handles Backpressure (2015). https://www.ververica.com/blog/how-flink-handles-backpressure, ververica

  4. GCP: Kubernetes Scheduler (2022). https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/, kubernetes Documentation

  5. Giannakopoulos, P., Petrakis, E.G.M.: Smilax: statistical machine learning autoscaler agent for Apache Flink. In: International Conference on Advanced Information Networking and Applications (AINA-2021), pp. 433–444. Toronto, ON, Canada (2021). https://doi.org/10.1007/978-3-030-75075-6_35

  6. Flink Kubernetes Operator (2023). https://github.com/apache/flink-kubernetes-operator, Version: v1beta1

  7. Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D., Forshaw, M., Roscoe, T.: Three steps is all you need: Fast, accurate, automatic scaling decisions for distributed streaming dataflows. In: 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018), pp. 783–798. Carlsbad, CA (2018). https://www.usenix.org/conference/osdi18/presentation/kalavri

  8. Metzger, R.: Scaling Flink automatically with Reactive Mode (2021). https://flink.apache.org/2021/05/06/reactive-mode.html

  9. Narkhede, N., Shapira, G., Palino, T.: Kafka the Definitive Guide, Real Time Data and Stream Processing at Scale. O’Reilly Media (2017). https://kafka.apache.org/

  10. Paul, F.: Autoscaling Apache Flink with Ververica Platform Autopilot (2021). https://www.ververica.com/blog/autoscaling-apache-flink-with-ververica-platform-autopilot, Ververica Platform

  11. Phelps, J.: Backpressure Explained - The Resisted Flow of Data Through Software (2019). https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7

  12. Rampérez, V., Soriano, J., Lizcano, D., Lara, J.A.: FLAS: a combination of proactive and reactive auto-scaling architecture for distributed services. Futur. Gener. Comput. Syst. 118, 56–72 (2021). https://doi.org/10.1016/j.future.2020.12.025

    Article  Google Scholar 

  13. Varga, B., Balassi, M., Kiss, A.: Towards autoscaling of Apache Flink jobs. Acta Univ. Sapientiae Inform. 13(1), 39–59 (2021). https://doi.org/10.2478/ausi-2021-0003

    Article  MATH  Google Scholar 

  14. Autopilot (2022). https://docs.ververica.com/user_guide/application_operations/autopilot.html, Ververica Platform

Download references

Acknowledgment

We are grateful to Google for the Google Cloud Platform Education Grants program. The work has received funding from the European Union’s Horizon 2020 - Research and Innovation Framework Programme H2020-SU-SEC-2019, under Grant Agreement No 883272- BorderUAS.

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, Technical University of Crete (TUC), Chania, Crete, Greece

    Alexandros Nikolaos Zafeirakopoulos & Euripides G. M. Petrakis

Authors
  1. Alexandros Nikolaos Zafeirakopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Euripides G. M. Petrakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Euripides G. M. Petrakis .

Editor information

Editors and Affiliations

  1. University of Alicante, Alicante, Spain

    Irene Garrigós

  2. University of Extremadura, Cáceres, Spain

    Juan Manuel Murillo Rodríguez

  3. Johannes Kepler University Linz, Linz, Austria

    Manuel Wimmer

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zafeirakopoulos, A.N., Petrakis, E.G.M. (2023). HYAS: Hybrid Autoscaler Agent for Apache Flink. In: Garrigós, I., Murillo Rodríguez, J.M., Wimmer, M. (eds) Web Engineering. ICWE 2023. Lecture Notes in Computer Science, vol 13893. Springer, Cham. https://doi.org/10.1007/978-3-031-34444-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34444-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34443-5

  • Online ISBN: 978-3-031-34444-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation