research-article

Performance and Dependability Evaluation of Distributed Event-based Systems: A Dynamic Code-injection Approach

Authors:

Matthew Forshaw,

Andrew DinnAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 349 - 352

https://doi.org/10.1145/3030207.3030245

Published: 17 April 2017 Publication History

Abstract

Distributed stream processing and event-based systems are an increasingly critical component in contemporary large-scale data processing applications, and are often subject to strict latency and reliability requirements. However, to achieve scalability demands, they are often deployed on distributed clusters of heterogeneous nodes, causing unpredictable runtime performance and complex fault characteristics.

The behaviour of these systems is poorly understood, and existing performance and dependability evaluation techniques are ill-equipped to handle the challenges introduced by the complex and distributed nature of event-based systems.

We develop a dynamic code-injection approach to evaluate the performance and dependability of stream processing and event-based systems. Our approach supports fine-grained instrumentation of applications and their runtime infrastructure, and the dynamic injection of code mutations and faults into a production system at runtime. We demonstrate the proposed approach by performing instrumentation and code injection on a distributed Apache Spark cluster.

References

[1]

Apache Spark. http://spark.apache.org/.

[2]

Thermostat. http://icedtea.classpath.org/thermostat/.

[3]

T. Cooper. Proactive Scaling of Distributed Stream Processing Work Flows Using Workload Modelling: Doctoral Symposium. In Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems, pages 410--413. ACM, 2016.

Digital Library

[4]

A. E. Dinn. Flexible, Dynamic Injection of Structured Advice Using Byteman. In Proceedings of the Tenth International Conference on Aspect-Oriented software Development companion, pages 41--50. ACM, 2011.

Digital Library

[5]

K. R. Dittrich, S. Gatziu, and A. Geppert. The Active Database Management System Manifesto: A Rulebase of ADBMS Features. In RIDS. Springer, 1995.

[6]

M. Forshaw, N. Thomas, and A. S. McGough. The Case for Energy-Aware Simulation and Modelling of Internet of Things (IoT). In ENERGY-SIM, 2016.

Digital Library

[7]

D. Gupta, L. Perronne, and S. Bouchenak. BFT-Bench: A Framework to Evaluate BFT Protocols. In ACM/SPEC ICPE '16, 2016.

Digital Library

[8]

V. Heorhiadi, S. Rajagopalan, H. Jamjoom, M. K. Reiter, and V. Sekar. Gremlin: Systematic Resilience Testing of Microservices. In IEEE ICDCS, 2016.

[9]

W. Hummer, C. Inzinger, P. Leitner, B. Satzger, and S. Dustdar. Deriving a Unified Fault Taxonomy for Event-Based Systems. In ACM DEBS, 2012.

Digital Library

[10]

G. Jacques-Silva, B. Gedik, H. Andrade, K.-L. Wu, and R. K. Iyer. Fault Injection-Based Assessment of Partial Fault Tolerance in Stream Processing Applications. In Proceedings of the 5th ACM International Conference on Distributed Event-Based system, pages 231--242. ACM, 2011.

Digital Library

[11]

A. Khoshkbarforoushha and R. Ranjan. Resource and Performance Distribution Prediction for Large Scale Analytics Queries. In ACM/SPEC ICPE. ACM, 2016.

Digital Library

[12]

M. A. Lopez, A. Lobato, and O. Duarte. A Performance Comparison of Open-Source Stream Processing Platforms. In IEEE Globecom, 2016.

[13]

P. Michalák, S. Heaps, M. Trenell, and P. Watson. Doctoral Symposium: Automating Computational Placement in IoT Environments. In DEBS'16, 2016.

Digital Library

[14]

S. Mohamed, M. Forshaw, and N. Thomas. Automatic Generation of Distributed Run-time Infrastructure for Internet of Things (IoT). In IEEE IoT-ASAP (Submitted), 2017.

[15]

R. Natella, D. Cotroneo, and H. S. Madeira. Assessing Dependability With Software Fault Injection: A Survey. ACM CSUR, 48(3):44, 2016.

Digital Library

[16]

R. Pietrantuono, S. Russo, and K. Trivedi. Emulating Environment-Dependent Software Faults: Position Paper. In Proceedings of the First International Workshop on Complex faUlts and Failures in LargE Software Systems, pages 34--40. IEEE Press, 2015.

Digital Library

[17]

M. Vögler, J. M. Schleicher, C. Inzinger, B. Nickel, and S. Dustdar. Non-Intrusive Monitoring of Stream Processing Applications. In 2016 IEEE SOSE'16, pages 162--171, March 2016.

Cited By

Jamieson SForshaw M(2024)Reasoning Over Streaming System Performance Using Response Surface Methodology2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825792(3702-3711)Online publication date: 15-Dec-2024
https://doi.org/10.1109/BigData62323.2024.10825792
Omoregbee PForshaw M(2023)Performability Requirements in Making a Rescaling Decision for Streaming ApplicationsComputer Performance Engineering10.1007/978-3-031-25049-1_9(133-147)Online publication date: 25-Jan-2023
https://doi.org/10.1007/978-3-031-25049-1_9
Jamieson SForshaw M(2023)Measuring Streaming System Robustness Using Non-parametric Goodness-of-Fit TestsComputer Performance Engineering10.1007/978-3-031-25049-1_1(3-18)Online publication date: 25-Jan-2023
https://doi.org/10.1007/978-3-031-25049-1_1
Show More Cited By

Index Terms

Performance and Dependability Evaluation of Distributed Event-based Systems: A Dynamic Code-injection Approach

Recommendations

Fault Injection and Dependability Evaluation of Fault-Tolerant Systems

The authors describe a dependability evaluation method based on fault injection that establishes the link between the experimental evaluation of the fault tolerance process and the fault occurrence process. The main characteristics of a fault injection ...
A Framework for Assessing Dependability in Distributed Systems with Lightweight Fault Injectors
IPDS '00: Proceedings of the 4th International Computer Performance and Dependability Symposium

Many fault injection tools are available for dependability assessment. Although these tools are good at injecting a single fault model into a single system, they suffer from two main limitations for use in distributed systems: (1) no single tool is ...
Experience with benchmarking dependability and performance of MapReduce systems

MapReduce provides a convenient means for distributed data processing and automatic parallel execution on clusters of machines. It has various applications and is used by several services featuring fault tolerance and scalability. Many studies ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
164
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jamieson SForshaw M(2024)Reasoning Over Streaming System Performance Using Response Surface Methodology2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825792(3702-3711)Online publication date: 15-Dec-2024
https://doi.org/10.1109/BigData62323.2024.10825792
Omoregbee PForshaw M(2023)Performability Requirements in Making a Rescaling Decision for Streaming ApplicationsComputer Performance Engineering10.1007/978-3-031-25049-1_9(133-147)Online publication date: 25-Jan-2023
https://doi.org/10.1007/978-3-031-25049-1_9
Jamieson SForshaw M(2023)Measuring Streaming System Robustness Using Non-parametric Goodness-of-Fit TestsComputer Performance Engineering10.1007/978-3-031-25049-1_1(3-18)Online publication date: 25-Jan-2023
https://doi.org/10.1007/978-3-031-25049-1_1
Karimov J(2022)Stream BenchmarksEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_299-2(1-6)Online publication date: 24-May-2022
https://doi.org/10.1007/978-3-319-63962-8_299-2
Lee YKim D(2020)A Taxonomy for Security Flaws in Event-Based SystemsApplied Sciences10.3390/app1020733810:20(7338)Online publication date: 20-Oct-2020
https://doi.org/10.3390/app10207338
Jamieson SGascon-Samson JZhang KDaudjee KKemme B(2020)Dynamic scaling of distributed data-flows under uncertaintyProceedings of the 14th ACM International Conference on Distributed and Event-based Systems10.1145/3401025.3406444(230-233)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.1145/3401025.3406444
Karimov J(2019)Stream BenchmarksEncyclopedia of Big Data Technologies10.1007/978-3-319-77525-8_299(1595-1600)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-319-77525-8_299
Karimov J(2018)Stream BenchmarksEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_299-1(1-6)Online publication date: 27-Mar-2018
https://doi.org/10.1007/978-3-319-63962-8_299-1
Mohamed SForshaw MThomas N(2017)Automatic Generation of Distributed Run-Time Infrastructure for Internet of Things2017 IEEE International Conference on Software Architecture Workshops (ICSAW)10.1109/ICSAW.2017.51(100-107)Online publication date: Apr-2017
https://doi.org/10.1109/ICSAW.2017.51

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten