Dmitrii Fedoseev
ABSTRACT
Mnesia, an Erlang distributed database, serves as an embedded storage and replication layer for OTP applications requiring low read latency and high availability. EMQX is a publish-subscribe message broker supporting the MQTT protocol that uses Mnesia to replicate its internal state across the cluster. We analyze the limitations of Mnesia's replication protocol scalability in large clusters under high load. To address these limitations, we developed Mria, an extension to the Mnesia database that provides eventual consistency within a cluster and achieves better horizontal scalability. We validated Mria using a variety of testing techniques, including model checking, chaos engineering, and formal verification. Replacing Mnesia with Mria allowed us to scale the EMQX cluster to 23 nodes, handle 100 million simultaneous client sessions and achieve a higher sustained load.
- Håkan Mattsson, Hans Nilsson, and Claes Wikström. Mnesia—a distributed robust dbms for telecommunications applications. In Practical Aspects of Declarative Languages: First International Workshop, PADL’99 San Antonio, Texas, USA, January 18–19, 1999 Proceedings 1, pages 152–163. Springer, 1998.
Google Scholar
- EMQX: The most scalable open-source MQTT broker for IoT, IIoT, and connected vehicles . https://github.com/emqx/emqx/, 2023. [Online; accessed 26-5-2023].
Google Scholar
- Andrew Banks, Ed Briggs, Ken Borgendale, and Rahul Gupta. Mqtt version 5.0. Technical report, OASIS Open, March 2019.
Google Scholar
- Biswajeeban Mishra and Attila Kertesz. The use of mqtt in m2m and iot systems: A survey. IEEE Access, 8:201071–201086, 2020.
Google ScholarCross Ref
- Eric C Cooper. Analysis of distributed commit protocols. In Proceedings of the 1982 ACM SIGMOD international conference on Management of data, pages 175–183, 1982.
Google ScholarDigital Library
- Leslie Lamport. Time, Clocks, and the Ordering of Events in a Distributed System, page 179–196. Association for Computing Machinery, New York, NY, USA, 2019.
Google Scholar
- Mria: Asynchronously replicated Mnesia-like database for Erlang/Elixir . https://github.com/emqx/mria/, 2023. [Online; accessed 26-5-2023].
Google Scholar
- Mnesia source code . https://github.com/erlang/otp/blob/maint-25/lib/mnesia/src/mnesia_tm.erl#L2130, 2023. [Online; accessed 26-5-2023].
Google Scholar
- Diego Ongaro and John Ousterhout. In search of an understandable consensus algorithm (extended version). In Proceeding of USENIX annual technical conference, USENIX ATC, pages 19–20, 2014.
Google Scholar
- Alkis Gotovos, Maria Christakis, and Konstantinos Sagonas. Test-driven development of concurrent programs using concuerror. In Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, pages 51–61, 2011.
Google ScholarDigital Library
- Parosh Abdulla, Stavros Aronis, Bengt Jonsson, and Konstantinos Sagonas. Optimal dynamic partial order reduction. ACM SIGPLAN Notices, 49(1):373–384, 2014.
Google ScholarDigital Library
- snabbkaffe: Distributed trace-based test framework with fault injection . https://github.com/kafka4beam/snabbkaffe/, 2023. [Online; accessed 26-5-2023].
Google Scholar
- Axiomatic abstraction of container . https://github.com/libtx/Storage/blob/main/theories/Storage/Properties.v, 2023. [Online; accessed 26-5-2023].
Google Scholar
- emqtt-bench: Lightweight MQTT benchmark tool written in Erlang . https://github.com/emqx/emqtt-bench/, 2023. [Online; accessed 26-5-2023].
Google Scholar
Index Terms
- Mria: An Eventually Consistent Mnesia
Recommendations
Invariant Control in Eventually Consistent Databases
WebDB'17: Proceedings of the 20th International Workshop on the Web and DatabasesDue to the requirements imposed by data-intensive applications, NoSQL and NewSQL databases are becoming more present in the IT Market. These products provide alternative data models to the relational databases, and most of them are intrinsically ...
Flexible Integration of Eventually Consistent Distributed Storage with Strongly Consistent Databases
NCCA '12: Proceedings of the 2012 Second Symposium on Network Cloud Computing and ApplicationsIn order to design distributed business applications or services, the common practice consists in setting up a multi-tier architecture on top of a relational database. Due to the recent evolution of the needs in terms of scalability and availability in ...
Towards verifying eventually consistent applications
PaPEC '14: Proceedings of the First Workshop on Principles and Practice of Eventual ConsistencyModern cloud and distributed systems depend heavily on replication of large-scale databases to guarantee properties like high availability, scalability and fault tolerance. These replicas are maintained in geographically distant locations to be able to ...
Comments