Abstract
Distributed database services are an increasingly important part of cloud computing. They are required to satisfy several key properties, including consensus and fault tolerance. Given the highly concurrent nature of these systems, subtle errors can arise that are difficult to discover through traditional testing methods. Formal verification can help in discovering bugs and ensuring correctness of these systems. In this paper, we apply formal methods to specify and verify an industrial distributed database, Taurus, which uses a combination of several fundamental protocols, including Multi-Version Concurrency Control and Raft-based Cluster Management. TLA\(^{+}\) is used to model an abstraction of the system and specify its properties. The properties are verified using the TLC model checker, as well as by theorem proving using the TLA proof system (TLAPS). We show that model checking is able to reproduce a bug in Taurus that was found during testing. But our most significant result is twofold: we successfully verified an abstract model of Taurus, and convinced our industrial partners of the usefulness of formal methods to industrial systems.
This work is based on a joint project with Huawei, together with colleagues in Huawei. Zhi currently works at TEG, Tencent.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The specifications are available at https://iscasmc.ios.ac.cn/?page_id=2148.
- 2.
The configurations are available at https://iscasmc.ios.ac.cn/?page_id=2148.
References
Depoutovitch, A., et al.: Taurus database: how to be fast, available, and frugal in the cloud. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1463–1478. ACM (2020)
Drăgoi, C., Henzinger, T., Zufferey, D.: PSYNC: a partially synchronous language for fault-tolerant distributed algorithms. In: POPL, pp. 400–415. ACM (2016)
Hawblitzel, C., et al.: IronFleet: proving practical distributed systems correct. In: SOSP, pp. 1–17. ACM (2015)
Lamport, L.: The temporal logic of actions. TOPLAS 94 16(3), 872–923 (1994)
Lamport, L.: Specifying Systems, vol. 388. Addison-Wesley, Boston (2002)
Lamport, L.: Byzantizing Paxos by refinement. In: Peleg, D. (ed.) DISC 2011. LNCS, vol. 6950, pp. 211–224. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24100-0_22
Lamport, L.: Industrial use of TLA\(^+\) (2018). https://lamport.azurewebsites.net/tla/industrial-use.html. Accessed 1 May 2021
McMillan, K.L., Padon, O.: Ivy: a multi-modal verification tool for distributed algorithms. In: Lahiri, S.K., Wang, C. (eds.) CAV 2020. LNCS, vol. 12225, pp. 190–202. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53291-8_12
Newcombe, C.: Why Amazon chose TLA+. In: Ait Ameur, Y., Schewe, K.D. (eds.) Abstract State Machines, Alloy, B, TLA, VDM, and Z. ABZ 2014. Lecture Notes in Computer Science, vol. 8477, pp. 25–39. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-43652-3_3
Newcombe, C., Rath, T., Fan, Z., Munteanu, B., Brooker, M., Deardeuff, M.: How Amazon web services uses formal methods. Commun. ACM 58(4), 66–73 (2015)
Ongaro, D., Ousterhout, J.: In search of an understandable consensus algorithm. In: USENIX Annual Technical Conference, pp. 305–319. USENIX Association (2014)
Shukla, D.: High-level TLA+ specifications for the five consistency levels offered by Azure CosmosDB (2018). https://github.com/Azure/azure-cosmos-tla. Accessed 1 May 2021
Taube, M., et al.: Modularity for decidability of deductive verification with applications to distributed systems. In: PLDI, pp. 190–202. ACM (2018)
Wilcox, J.R., et al.: Verdi: a framework for implementing and formally verifying distributed systems. In: PLDI, pp. 357–368. ACM (2015)
Acknowledgements
We would like to thank the anonymous reviewers for their valuable suggestions and comments about this paper. Work supported in part by the Guangdong Science and Technology Department (Grant No. 2018B010107004).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Gao, S. et al. (2021). Formal Verification of Consensus in the Taurus Distributed Database. In: Huisman, M., Păsăreanu, C., Zhan, N. (eds) Formal Methods. FM 2021. Lecture Notes in Computer Science(), vol 13047. Springer, Cham. https://doi.org/10.1007/978-3-030-90870-6_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-90870-6_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90869-0
Online ISBN: 978-3-030-90870-6
eBook Packages: Computer ScienceComputer Science (R0)