Jörn Kuhlenkamp
ABSTRACT
Function-as-a-Service (FaaS), offers a new alternative to operate cloud-based applications. FaaS platforms enable developers to define their application only through a set of service functions, relieving them of infrastructure management tasks, which are executed automatically by the platform. Since its introduction, FaaS has grown to support workloads beyond the lightweight use-cases it was originally intended for, and now serves as a viable paradigm for big data processing. However, several questions regarding FaaS platform quality are still unanswered. Specifically, the impact of automatic infrastructure management on serverless big data applications remains unexplored.
In this paper, we propose a novel evaluation method (SIEM) to understand the impact of these tasks. For this purpose, we introduce new metrics to quantify quality in different big data application scenarios. We show an application of SIEM by evaluating the four major FaaS providers, and contribute results and new insights for FaaS-based big data processing.
- Chaitanya Baru, Milind Bhandarkar, Carlo Curino, Manuel Danisch, Michael Frank, Bhaskar Gowda, Hans-Arno Jacobsen, Huang Jie, Dileep Kumar, Raghunath Nambiar, et al. 2014. Discussion of Big- Bench: A Proposed Industry Standard Performance Benchmark for Big Data. In Technology Conference on Performance Evaluation and Benchmarking. Springer International Publishing, Cham, 44--63.Google Scholar
- David Bermbach, Jörn Kuhlenkamp, Akon Dey, Arunmoezhi Ramachandran, Alan Fekete, and Stefan Tai. 2017. BenchFoundry: A Benchmarking Framework for Cloud Storage Services. In Service- Oriented Computing (ICSOC'15). Springer International Publishing, Cham, 314--330.Google Scholar
- David Bermbach, Erik Wittern, and Stefan Tai. 2017. Cloud Service Benchmarking: Measuring Quality of Cloud Services from a Client Perspective. Springer International Publishing, Cham.Google Scholar
- Benjamin Congdon. 2018. GitHub Repository: Corral. http://github. com/bcongdon/corral. [Online; accessed 26.02.2019].Google Scholar
- Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC '10). ACM, New York, NY, USA, 143--154.Google ScholarDigital Library
- Matt Crane and Jimmy Lin. 2017. An Exploration of Serverless Architectures for Information Retrieval. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR'17). ACM, New York, NY, USA, 241--244.Google ScholarDigital Library
- Scott Hendrickson, Stephen Sturdevant, Tyler Harter, Venkateshwaran Venkataramani, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci- Dusseau. 2016. Serverless Computation with OpenLambda. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud'16). USENIX Association, Denver, CO, 14--19.Google Scholar
- Karl Huppler. 2009. The Art of Building a Good Benchmark. In Performance Evaluation and Benchmarking, Raghunath Nambiar and Meikel Poess (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 18--30.Google Scholar
- David Jackson and Gary Clynch. 2018. An Investigation of the Impact of Language Runtime on the Performance and Cost of Serverless Functions. In Proceedings of the 3rd International Workshop on Serverless Computing (WoSC'18). IEEE, 154--160.Google ScholarCross Ref
- Eric Jonas, Qifan Pu, Shivaram Venkataraman, Ion Stoica, and Benjamin Recht. 2017. Occupy the Cloud: Distributed Computing for the 99%. In Proceedings of the 2017 Symposium on Cloud Computing (SoCC'17). ACM, New York, NY, USA, 445--451.Google ScholarDigital Library
- Eric Jonas, Johann Schleier Smith, Vikram Sreekanti, Chia-Che Tsai, Anurag Khandelwal, Qifan Pu, Vaishaal Shankar, Joao Carreira, Karl Krauth, Neeraja Yadwadkar, Joseph E. Gonzales, Raluca A. Popa, Ion Stoica, and David A. Patterson. 2019. Cloud Programming Simplified: A Berkeley View on Serverless Computing. CoRR arXiv preprint arXiv:1902.03383 (9 Feb 2019), 1--33.Google Scholar
- Youngbin Kim and Jimmy Lin. 2018. Serverless Data Analytics with Flint. CoRR abs/1803.06354 (13 Aug 2018), 5.Google Scholar
- Markus Klems. 2016. Experiment-driven Evaluation of Cloud-based Distributed Systems. Ph.D. Dissertation. TU Berlin.Google Scholar
- Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018. Pocket: Elastic Ephemeral Storage for Serverless Analytics. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI'18). USENIX Association, Berkeley, CA, USA, 427--444.Google Scholar
- Jörn Kuhlenkamp and Markus Klems. 2017. Costradamus: A Cost- Tracing System for Cloud-Based Software Services. In Service-Oriented Computing (ICSOC'15). Springer International Publishing, Cham, 657-- 672.Google Scholar
- Jörn Kuhlenkamp, Markus Klems, and Oliver Röss. 2014. Benchmarking Scalability and Elasticity of Distributed Database Systems. Proceedings of the VLDB Endowment 7, 12 (2014), 1219--1230.Google ScholarDigital Library
- Jörn Kuhlenkamp and Sebastian Werner. 2018. Benchmarking FaaS Platforms: Call for Community Participation. In Proceedings of the 3rd International Workshop on Serverless Computing (WoSC'18). 189--194.Google ScholarCross Ref
- Aleksandr Kuntsevich, Pezhman Nasirifard, and Hans-Arno Jacobsen. 2018. A Distributed Analysis and Benchmarking Framework for Apache OpenWhisk Serverless Platform. In Proceedings of the 19th International Middleware Conference (Posters) (Middleware'18). ACM, New York, NY, USA, 3--4.Google ScholarDigital Library
- Hyungro Lee, Kumar Satyam, and Geoffrey Fox. 2018. Evaluation of Production Serverless Computing Environments. In Proceedings of the IEEE 11th International Conference on Cloud Computing (CLOUD'18). IEEE, 442--450.Google ScholarCross Ref
- Philipp Leitner, Erik Wittern, Josef Spillner, and Waldemar Hummer. 2018. A Mixed-Method Empirical Study of Function-As-A-Service Software Development in Industrial Practice. Journal of Systems and Software 149 (18 Dec 2018), 340 -- 359.Google Scholar
- Wes Lloyd, Shruti Ramesh, Swetha Chinthalapati, Lan Ly, and Shrideep Pallickara. 2018. Serverless Computing: An Investigation of Factors Influencing Microservice Performance. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E'18). IEEE, 159--169.Google ScholarCross Ref
- Pedro G. López, Marc Sánchez Artigas, Gerard París, Daniel B. Pons, Alvaro R. Ollobarren, and David A. Pinto. 2018. Comparison of FaaS Orchestration Systems. In Proceedings of the 3rd International Workshop on Serverless Computing (WoSC'18). IEEE, 148--153.Google Scholar
- Maciej Malawski, Kamil Figiela, Adam Gajek, and Adam Zima. 2018. Benchmarking Heterogeneous Cloud Functions. In Euro-Par 2017: Parallel Processing Workshops (Euro-Par'17). Springer International Publishing, Cham, 415--426.Google Scholar
- Maciej Malawski, Adam Gajek, Adam Zima, Bartosz Balis, and Kamil Figiela. 2017. Serverless Execution of Scientific Workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions. Future Generation Computer Systems (4 Nov 2017).Google Scholar
- Johannes Manner, Martin Endreß, Tobis Heckel, and Guido Wirtz. 2018. Cold Start Influencing Factors in Function as a Service. In Proceedings of the 3rd International Workshop on Serverless Computing (WoSC'18). IEEE, 181--188.Google ScholarCross Ref
- Josep Sampé, Marc Sánchez-Artigas, Pedro García-López, and Gerard París. 2017. Data-driven Serverless Functions for Object Storage. In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference (Middleware'17). ACM, New York, NY, USA, 121--133.Google ScholarDigital Library
- Bianca Schroeder, Adam Wierman, and Mor Harchol Balter. 2006. Open Versus Closed: A Cautionary Tale. In Proceedings of the 3rd Conference on Networked Systems Design & Implementation (NSDI'06). USENIX Association, Berkeley, CA, USA, 18--18.Google Scholar
- Jóakim v. Kistowski, Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. How to Build a Benchmark. In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE '15). ACM, New York, NY, USA, 333--336.Google ScholarDigital Library
- Erwin van Eyk, Alexandru Iosup, Cristina L. Abad, Johannes Grohmann, and Simon Eismann. 2018. A SPEC RG Cloud Group's Vision on the Performance Challenges of FaaS Cloud Architectures. In Companion of the 2018 ACM/SPEC International Conference on Performance Engineering (ICPE'18). ACM, New York, NY, USA, 21--24.Google ScholarDigital Library
- Sebastian Werner, Jörn Kuhlenkamp, Markus Klems, Johannes Müller, and Stefan Tai. 2018. Serverless Big Data Processing Using Matrix Multiplication as Example. In Proceedings of the IEEE International Conference on Big Data (Big Data'18). IEEE, Seattle, WA, USA, 358--365.Google ScholarCross Ref
Index Terms
- An Evaluation of FaaS Platforms as a Foundation for Serverless Big Data Processing
Recommendations
Benchmarking the Data Layer Across Serverless Platforms
HiPS '22: Proceedings of the 2nd Workshop on High Performance Serverless ComputingThe use of highly scalable serverless platforms for web microservices and IoT applications is well known. However, their use for data-intensive applications is restricted due to the stateless nature of serverless functions. Any data retrieval, storage, ...
Evaluating Serverless Architecture for Big Data Enterprise Applications
BDCAT '21: Proceedings of the 2021 IEEE/ACM 8th International Conference on Big Data Computing, Applications and TechnologiesMigration of enterprise applications to the cloud has been driven by a myriad of benefits ranging from availability of infinite computing resources to the elimination of upfront CapEx cost. However, many users still face the burden of complex framework ...
Benchmarking elasticity of FaaS platforms as a foundation for objective-driven design of serverless applications
SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied ComputingApplication providers have to solve the trade-off between performance and deployment costs by selecting the "right" amount of provisioned computing resources for their application. The high value of changing this trade-off decision at runtime fueled a ...
Comments