Abstract
Recent trends in data collection and the decreasing prices of storage result in constantly growing amounts of analyzable data. These masses of data cannot easily be processed by traditional database systems as these do not allow for a sufficient degree of scalability. Programs especially designed for parallel data analysis on large-scale distributed systems are required. Developing such programs on clusters of commodity hardware is a complex challenge for even the most experienced system developers. Frameworks such as Apache Hadoop are scalable, but – when compared to SQL – extremely hard to program. The open-source platform Apache Flink is a link between conventional database systems and big data analysis frameworks. Flink is based on a fault tolerant runtime for data stream processing, which manages the distribution of data as well as communications within the cluster. A high diversity of use cases can be supported through various interfaces that allow for the implementation of data analysis processes. In this paper, we present an overview of Apache Flink as well as some current research activities on top of the Apache Flink ecosystem.
About the authors
Tilmann Rabl is a senior researcher at the Database Systems and Information Management (DIMA) group at TU Berlin and at the German Research Center for Artificial Intelligence (DFKI). At DIMA he is research director and technical coordinator of the Berlin Big Data Center (BBDC). Tilmann Rabl is also cofounder of the startup bankmark.
Technische Universität Berlin, FG DIMA, Einsteinufer 17, 10587 Berlin, Germany
Jonas Traub is a research associate and PhD student at the Database Systems and Information Management group at TU Berlin. In his research he focuses on real-time processing of sensor data in the Internet of Things. Prior to that, Jonas Traub worked several years at IBM and as an independent consultant.
Technische Universität Berlin, FG DIMA, Einsteinufer 17, 10587 Berlin, Germany
Asterios Katsifodimos is a senior researcher at the Database Systems and Information Management (DIMA) group at TU Berlin and at the German Research Center for Artificial Intelligence (DFKI). He received his PhD from INRIA Saclay and Universite Paris-Sud.
Technische Universität Berlin, FG DIMA, Einsteinufer 17, 10587 Berlin, Germany
Volker Markl is a Full Professor and Chair of the DIMA Group at TU Berlin and an Adjunct Full Professor at the University of Toronto. He is Director of the Intelligent Analytics for Massive Data Research Group at DFKI and Director of the Berlin Big Data Center.
Technische Universität Berlin, FG DIMA, Einsteinufer 17, 10587 Berlin, Germany
Acknowledgement
This work has been supported through grants by the German Ministry for Education and Research as Berlin Big Data Center BBDC (funding mark 01IS14013A) as well as by the DFG research group Stratosphere (FOR 1306) and also through grants by the European Union's Horizon 2020 research and innovation program under grant agreement 687691 for Proteus and 688191 for Streamline.
©2016 Walter de Gruyter Berlin/Boston
