Improving observability in Event Sourcing systems

Highlights

• Event Sourcing has the double purpose of keeping application state and providing decoupling between modules.

• Our proposal is improving the observability of the system.

• We make it possible to track all individual requests and their relation to events in the log.

• This combination reduces the need to replay the event log for debugging.

Abstract

Event Sourcing (ES) systems use an event log with the double purpose of keeping application state and providing decoupled communication. While ES systems keep track of all business events, other untracked events, either from internal components or from external systems may still cause failures. Determining the root cause of such failures usually involves complex procedures based on replaying the event log. Unlike this, in distributed systems, developers often instrument the source code, for the sake of improving observability and perform tracing on workflows and data.

Adding tracing to ES thus seems like an unexplored and powerful approach to improve the observability of the system. In this paper, we suggest possible implementations of the idea and discuss their merits. These include the adoption of well-known tracing-related tools and standards in ES systems, with the respective advantages for root-cause analysis, anomaly detection, profiling and others.

Introduction

Event Sourcing is a design approach where distributed applications keep their state as a sequence of state-changing operations. Instead of storing mutable objects, applications keep an immutable sequence of changes to such objects. Changing state and writing an event to the log is one single, therefore atomic, operation. The log becomes the authoritative source of truth, offering eventual consistency, as events propagate to different parts of the distributed application. This design entails a strongly decoupled architecture, typical of publish–subscribe systems (Clayman et al., 2010), while providing reliable auditing and logging functionality. For example, developers may bring the application back to a previous execution state, by replaying the events in the log. This feature is particularly powerful, not only for the sake of failure recovery and state management (most services can be stateless), but also for debugging and to experiment alternative what-if scenarios. Furthermore, it makes the system’s data schema more flexible, as it becomes possible to recover field values or even calculate additional ones from the log.

The literature (Fowler, 2005a) mentions ease of debugging as one of the advantages of having the event log and the ability to do partial and branched replays. In this context, ES is often presented as a way to deal with some of the complications created by distribution. However, this approach preserves no metadata about the system, which is necessary to observe failures and assert the correctness of the ES mechanisms themselves. To investigate and correct these situations, developers normally have to resort to unstructured logging. It is worth noting that the cost of branching or replaying the log is very high, both in source code complexity and resources, as a portion of the events will essentially have to be reprocessed.

At the same time, the rising trend of fine-grained distributed systems using microservices and the Function-as-a-Service (FaaS) paradigm made applications more fragmented than ever. Essentially, while breaking the monolith simplifies programming and provides horizontal scalability, complexity goes into the distributed system, which become difficult to observe and comprehend (Pautasso et al., 2017). Getting a consistent picture of state involves causality relationships. End-to-end tracing (Xiang et al., 2016) is one such method, preserving a portion of causality relationships at the cost of requiring code instrumentation, for the sake of system analysis and debugging.

As we discuss in Section 2, tracing and ES are usually taken as separate paradigms. To some degree, ES is presented as a complete solution for debugging any issue, as it can theoretically recreate state at any point in time for analysis. State-changing events, however, lack information of the system’s internal variables and code execution path, not to mention interaction with external components. Log events lack structured metadata, thus failing to provide end-to-end tracing information. Operators thus lack good means to recover the workflows taking place inside the distributed system. On the other hand, tracing is an observability tool that uses source code instrumentation to pass specific data, known as baggage, between different parts of the application at run-time. Baggage relates and stitches the different services of the application together, for the sake of a posteriori debugging of their interactions. Tracing has a meta-functionality; it is not meant to be the source of truth of business data. Tools like  zipkin.io (2021) or  Jaeger (2021) demonstrate the acceptance of tracing in the industry.

In Section 3, we review other work that explores or enriches ES in ways that have similarities to our own, e.g., to recover the state of the distributed system or for simulation purposes. Interestingly, our work also displays similarities to the topic of process mining, where the goal is to retrieve operational processes from system logs. We, therefore, briefly review the topic in this section.

In Section 4 we formally reason about ES and tracing, their differences and complementarities. We propose to use them together in Section 5. We do this by keeping internal events that, while not changing the business state of the application, are either in the causation or consequence chain of such changes. Processing and storage restrictions aside, this would enable a complete recovery of the execution state of the entire system.

Our goal is to increase observability, by recovering causality inside the entire system, across separate components. Our contribution is an approach to leverage distributed tracing, to preserve relationships between events, as well as connecting these to software metadata, such as versions, ultimate event publishers and subscribers, and other runtime data, such as exceptions and logging. This will eventually make the identification of failure root causes much cheaper, by resorting to proven standards, tools and approaches.

To better illustrate our idea, in Section 6, we explore a Guitar Store as an ES use case and analyze scenarios where an event may be missing from the log, duplicated, or incorrect. We then address these scenarios with and without tracing and evaluate the differences. We extend this analysis to a case where system developers and operators can take advantage of tracing to eliminate unused code.

We discuss possible implementations for adding tracing to ES and elaborate on the advantages of the idea in Section 7. In Section 8, we conclude the paper and enumerate some consequences of our proposal for developers and operators.