Skip to main content
SpringerLink
Log in

VLL: a lock manager redesign for main memory database systems

  • Special Issue Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Lock managers are increasingly becoming a bottleneck in database systems that use pessimistic concurrency control. In this paper, we introduce very lightweight locking (VLL), an alternative approach to pessimistic concurrency control for main memory database systems, which avoids almost all overhead associated with traditional lock manager operations. We also propose a protocol called selective contention analysis (SCA), which enables systems implementing VLL to achieve high transactional throughput under high-contention workloads. We implement these protocols both in a traditional single-machine multi-core database server setting and in a distributed database where data are partitioned across many commodity machines in a shared-nothing cluster. Furthermore, we show how VLL and SCA can be extended to enable range locking. Our experiments show that VLL dramatically reduces locking overhead and thereby increases transactional throughput in both settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. The transaction is not required to be at the front of the TxnQueue when it is removed. In this sense, TxnQueue is not, strictly speaking, a queue.

  2. Although there may be some false negatives (in which an “actually” runnable transaction is still perceived as blocked) due to the need to hash the entire key space into a 100 kB bitstring, this algorithm gives no false positives.

  3. This can sometimes result in locking a much larger part of the key space than needed. For example, in an 8-bit key space, locking the range \(R=\) [00111100,01000010] with this method results in locking all keys with prefix 0—half the key space rather than the necessary 7/256 of the key space. In this particular case, \(P\) could instead consist of the prefixes 001111xx, 0100000x, and 01000010, thus locking exactly the rows in \(R\). Or \(P\) could consist of 001111xx and 010000xx, locking the rows in \(R\), plus one additional row (010000011). Under some workloads, locking larger ranges than necessary may incur expensive and unnecessary contention costs, justifying the costs of finer-grained \(R \rightarrow P\) transformations. Under other workloads, coarser-grained prefix locking may be acceptable and users may profit from the reduced locking overhead.

  4. Although we refer to this system as “H-Store” in the discussion that follows, we actually implemented the H-Store protocol within the Calvin framework in order to provide as fair a comparison as possible.

  5. The exception is that deadlock is possible in this workload if transactions conflict on cold items. This was rare enough, however, that we observed no deadlocks in our experiments.

  6. Subsequent versions of H-Store proposed to speculatively execute transactions during this idle time [16], but this can lead to cascading aborts and wasted work if there would have been a conflict. We do not implement speculative execution in our H-Store prototype since H-Store is designed for workloads with small percentages of multi-partition transactions (when speculative execution is not necessary), and our purpose for including it as a comparison point is only to analyze how it compares to VLL on these target single-partition workloads.

  7. For example, distributed deadlock detection mechanisms differ for Standard Range Locking than for the other mechanisms, because partially overlapping ranges must be split, so the number of lock queues that a transaction’s requests are in may increase over time without the transaction requesting new locks.

References

  1. Agrawal, D., Sengupta, S.: Modular synchronization in distributed, multiversion databases: version control and concurrency control. IEEE TKDE 5, 126–137 (1993)

  2. Agrawal, R., Livny, M.J.C.M.: Concurrency control performance modeling: alternatives and implications. ACM Trans. Database Syst. 12, 609–654 (1987)

    Article  Google Scholar 

  3. Mahmoud, H.A., Arora, V., Nawab, F., Agrawal, D., Abbadi, A.E.: Maat: effective and scalable coordination of distributed transactions in the cloud. In: Proceedings of PVLDB, vol. 7, no. 5, (2014)

  4. Bernstein, P.A., Goodman, N.: Concurrency control in distributed database systems. ACM Comput. Surv. 13(2), 185–221 (1981)

    Article  MathSciNet  Google Scholar 

  5. Bernstein, P.A., Hadzilacos, V., Goodman, N.: Concurrency Control and Recovery in Database Systems. Addison-Wesley, Reading (1987)

    Google Scholar 

  6. Diaconu, C., Freedman, C., Ismert, E., ke Larson, P., Mittal, P., Stonecipher, R., Verma, N., Zwilling, M.: Hekaton: Sql server’s memory-optimized oltp engine. In: SIGMOD (2013)

  7. Corbett, J.C., et al.: Spanner: Googles globally-distributed database. In: Proceedings of OSDI 2012 (2012)

  8. Gottemukkala, V., Lehman, T.: Locking and latching in a memory-resident database system. In: VLDB (1992)

  9. Gray, J.: Notes on Database Operating Systems. Operating System, An Advanced Course. Springer, Berlin (1979)

    Google Scholar 

  10. Gray, J.N., Lorie, R.A., Putzolu, G.R., Traiger, I.L.: Granularity of locks and degrees of consistency in a shared database. In: Proceedings of IFIP Working Conference on Modelling of Database Management Systems (1975)

  11. Gray, J., Reuter.: Transaction Processing: Concepts and Techniques. Morgan Kaufmann, New York (1993)

  12. Harizopoulos, S., Abadi, D.J., Madden, S.R., Stonebraker, M.: OLTP through the looking glass, and what we found there. In: SIGMOD (2008)

  13. Hellerstein, J.M., Stonebraker, M., Hamilton, J.: Architecture of a database system. Found. Trends Databases 1(2), 141–259 (2007)

    Article  MATH  Google Scholar 

  14. Johnson, R., Pandis, I., Ailamaki, A.: Improving oltp scalability using speculative lock inheritance. PVLDB 2(1), 479–489 (2009)

    Google Scholar 

  15. Johnson, R., Pandis, I., Hardavellas, N., Ailamaki, A., Falsafi, B.: Shore-mt:a scalable storage manager for the multicore era. In: Proceedings of EDBT (2009)

  16. Jones, E.P.C., Abadi, D.J., Madden, S.R.: Concurrency control for partitioned databases. In: SIGMOD (2010)

  17. Joshi, A.: Adaptive locking strategies in a multi-node data sharing environment. In: VLDB (1991)

  18. Jung, H., Han, H., Fekete, A.D., Heiser, G., Yeom, H.Y.: A scalable lock manager for multicores. In: Proceedings of SIGMOD (2013)

  19. Kemper, A., Neumann, T., Finis, J., Funke, F., Leis, V., Muhe, H., Muhlbauer, T., Rodiger, W.: Transaction processing in the hybrid OLTP/OLAP main-memory database system hyper. IEEE Data Eng. Bull. 36(2), 41–47 (2013)

  20. Kimura, H., Graefe, G., Kuno, H.: Efficient locking techniques for databases on modern hardware. In: Workshop on Accelerating Data Management Systems Using Modern Processor and Storage Architectures (2012)

  21. Kung, H.T., Robinson, J.T.: On optimistic methods for concurrency control. TODS 6(2), 213–226 (1981)

  22. Larson, P., Blanas, S., Diaconu, C., Freedman, C., Patel, J., Zwilling, M.: High-performance concurrency control mechanisms for main-memory database. In: PVLDB (2011)

  23. Lee, J., Muehle, M., May, N., Faerber, F., Sikka1, V., Plattner, H., Krueger, J., Grund, M.: High-performance transaction processing in sap hana. IEEE Data Eng. Bull. 36(2), 28–33 (2013)

  24. Lehman, T.: Design and performance evaluation of a main memory relational database system. Ph.D. thesis, University of Wisconsin-Madison (1986)

  25. Lehman, T.J., Gottemukkala, V.: The Design and Performance Evaluation of a Lock Manager for a Memory-Resident Database System. Performance of Concurrency Control Mechanisms in Centralised Database System. Addison-Wesley, Reading (1996)

    Google Scholar 

  26. Lomet, D.: Key range locking strategies for improved concurrency. In: Proceedings of VLDB (1993)

  27. Mohan, C., Lindsay, B.G., Obermarck, R.: Transaction management in the r* distributed database management system. ACM Trans. Database Syst. 11(4), 378–396 (1986)

  28. Mohan, C., Narang, I.: Recovery and coherency-control protocols for fast inter-system page transfer and fine-granularity locking in shared disks transaction environment. In: VLDB (1991)

  29. Pandis, I., Johnson, R., Hardavellas, N., Ailamaki, A.: Data-oriented transaction execution. PVLDB 3(1), 928–939 (2010)

    Google Scholar 

  30. Ren, K., Thomson, A., Abadi, D.J.: Lightweight locking for main memory database systems. In: PVLDB (2013)

  31. Shute, J., Vingralek, R., Samwel, B., Handy, B., Whipkey, C., Rollins, E., Oancea, M., Littleeld, K., Menestrina, D., Ellner, S., Cieslewicz, J., Rae, I., Stancescu, T., Apte, H.: F1: a distributed sql database that scales. In: VLDB (2013)

  32. Stonebraker, M.: Concurrency control and consistency of multiple copies of data in distributed ingres. IEEE Trans. Softw. Eng. SE-5, 188–194 (1979)

  33. Stonebraker, M., Madden, S.R., Abadi, D.J., Harizopoulos, S., Hachem, N., Helland, P.: The end of an architectural era (it’s time for a complete rewrite). In: VLDB, Vienna, Austria (2007)

  34. Thomasian, A.: Two-phase locking performance and its thrashing behavior. TODS 18(4), 579–625 (1993)

    Article  Google Scholar 

  35. Thomson, A., Abadi, D.J.: The case for determinism in database systems. In: VLDB (2010)

  36. Thomson, A., Abadi, D.J.: Modularity and scalability in calvin. IEEE Data Eng. Bull. 36(2), 48–55 (2013)

  37. Thomson, A., Diamond, T., Shao, P., Ren, K., Weng, S.-C., Abadi, D.J.: Calvin: Fast distributed transactions for partitioned database systems. In: SIGMOD (2012)

  38. Horikawa, T.: Latch-free data structures for dbms: design, implementation, and evaluation. In: Proceedings of SIGMOD (2013)

  39. Tu, S., Zheng, W., Kohler, E., Liskov, B., Madden, S.: Speedy transactions in multicore in-memory databases. In: Proceedings of SOSP, SOSP ’13, pp. 18–32 (2013)

  40. Whitney, A., Shasha, D., Apter, S.: High volume transaction processing without concurrency control, two phase commit, SQL or C++. In: HPTS (1997)

Download references

Acknowledgments

We would like to thank the anonymous reviewers for their detailed and insightful comments. This work was sponsored by the NSF under grants IIS-0845643 and IIS-1249722, and by a Sloan Research Fellowship.

Author information

Authors and Affiliations

  1. Yale University, New Haven, CT, USA

    Kun Ren & Daniel J. Abadi

  2. Google, New York, NY, USA

    Alexander Thomson

Authors
  1. Kun Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Thomson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel J. Abadi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Ren.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ren, K., Thomson, A. & Abadi, D.J. VLL: a lock manager redesign for main memory database systems. The VLDB Journal 24, 681–705 (2015). https://doi.org/10.1007/s00778-014-0377-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-014-0377-7

Keywords

Search

Navigation