Skip to main content
SpringerLink
Log in

Benchmarking in-memory database

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

We have witnessed exciting development of RAM technology in the past decade. The memory size grows rapidly and the price continues to decrease, so that it is feasible to deploy large amounts of RAM in a computer system. Several companies and research institutions have devoted a lot of resources to develop in-memory databases (IMDB) that implement queries after loading data into (virtual) memory in advance. The bloom of various in-memory databases pursues us to test and evaluate their performance objectively and fairly. Although the existing database benchmarks like Wisconsin benchmark and TPC-X series have achieved great success, they cannot suit for in-memory databases due to the lack of consideration of unique characteristics of an IMDB. In this study, we propose MemTest, a novel benchmark that concerns some major characteristics of an in-memory database. This benchmark constructs particular metrics, which cover processing time, compression ratio, minimal memory space and column strength of an in-memory database. We design a data model based on inter-bank transaction applications, and a data generator to support uniform and skew data distributions. The MemTest workload includes a set of queries and transactions against the metrics and data model. Finally, we illustrate the efficacy of MemTest through the implementations on two different in-memory databases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Gray J. Benchmark handbook: for database and transaction processing systems. San Francisco: Morgan Kaufmann Publishers Inc., 1992

    MATH  Google Scholar 

  2. Carey M J, De Witt D J, Naughton J F. The 007 benchmark. In: Buneman P, Jajodia S, eds. SIGMOD Conference. ACM Press, 1993, 12–21

    Google Scholar 

  3. Carey M J, De Witt D J, Naughton J F, Asgarian M, Brown P, Gehrke J E, Shah D N. The buckyobject-relational benchmark (experience paper). In: Proceedings of the 1997 ACM SIGMOD International Conference on Management of Data. 1997, 135–146

    Chapter  Google Scholar 

  4. Schmidt A, Waas F, Kersten M, Carey M J, Manolescu I, Busse R. Xmark: a benchmark for XML data management. In: Proceedings of the 28th International Conference on Very Large Data Bases. 2002, 974–985

    Chapter  Google Scholar 

  5. Carey M J, Ling L, Nicola M, Shao L. EXRT: towards a simple benchmark for XML readiness testing. In: Nambiar R, Poess M, eds. Performance Evaluation, Measurement and Characterization of Complex Systems. Springer Berlin Heidelberg, 2010, 93–109

    Google Scholar 

  6. Cooper B F, Silberstein A, Tam E, Ramakrishnan R, Sears R. Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 143–154

    Chapter  Google Scholar 

  7. Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen H A. BigBench: towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 2013, 1197–1208

    Chapter  Google Scholar 

  8. Rabl T, Poess M, Jacobsen H A, O’Neil P, O’ Neil E. Variations of the star schema benchmark to test the effects of data skew on query performance. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering. 2013, 361–372

    Chapter  Google Scholar 

  9. Färber F, Cha S K, Primsch J, Bornhövd C, Sigg S, Lehner W. SAPHANA database: data management for modern business applications. ACM Sigmod Record, 2012, 40(4): 45–51

    Article  Google Scholar 

  10. Lahiri T, Neimat M A, Folkman S. Oracle TimesTen: an in-memory database for enterprise applications. IEEE Data Engineering Bulletin, 2013, 36(2): 6–13

    Google Scholar 

  11. Diaconu C, Freedman C, Ismert E, Larson P A, Mittal P, Stonecipher R, Verma N, Zwilling M. Hekaton: SQL server’s memory-optimized OLTP engine. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 2013, 1243–1254

    Chapter  Google Scholar 

  12. Gupta M K, Verma V, Verma M S. In-memory database systems — a paradigm shift. International Journal of Engineering Trends and Technology, 2013, 6(6): 333–336

    Google Scholar 

  13. Cole R, Funke F, Giakoumakis L, Guy W, Kemper A, Krompass S, Kuno H, Nambiar R, Neumann T, PoessM, Sattler K U, Seibold M, Simon E, Waas F. The mixed workload CH-benCHmark. In: Proceedings of the 4th ACM International Workshop on Testing Database Systems. 2011

    Google Scholar 

  14. Kang Q, Jin C Q, Zhang Z, Zhou A Y. MemTest: a novel benchmark for in-memory database. In: Zhan J F, Han R, Weng C L, eds. Big Data Benchmarks, Performance Optimization, and Emerging Hardware. Springer International Publishing, 2014, 34–46

    Google Scholar 

  15. Jin C Q, Qian WN, Zhou MQ, Zhou A Y. Benchmarking data management systems: from traditional database to emergent big data. Chinese Journal of Computers, 2015, 38(1): 18–34 (in Chinese)

    MathSciNet  Google Scholar 

  16. Cattell R G G, Skeen J. Object operations benchmark. ACM Transactions on Database Systems (TODS), 1992, 17(1): 1–31

    Article  Google Scholar 

  17. Runapongsa K, Patel J M, Jagadish H V, Chen Y, Al-Khalifa S. The Michigan benchmark: towards XML query performance diagnostics. Information Systems, 2006, 31(2): 73–97

    Article  Google Scholar 

  18. Nicola M, Kogan I, Schiefer B. An XML transaction processing benchmark. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data. 2007, 937–948

    Chapter  Google Scholar 

  19. Werstein P. A performance benchmark for spatiotemporal databases. In: Proceedings of the 10th Annual Colloquium of the Spatial Information Research Centre. 1998, 1365–1374

    Google Scholar 

  20. Düntgen C, Behr T, Güting R H. BerlinMOD: a benchmark for moving object databases. The VLDB Journal, 2009, 18(6): 1335–1368

    Article  Google Scholar 

  21. Arasu A, Cherniack M, Galvez E, Maier D, Maskey A S, Ryvkina E, Ryvkina E, Stonebraker M, Tibbetts R. Linear road: a stream data management benchmark. In: Proceedings of the 30th International Conference on Very Large Data Bases. 2004, 480–491

    Google Scholar 

  22. Tözün P, Pandis I, Kaynak C, Jevdjic D, Ailamaki A. From A to E: analyzing TPC’s OLTP benchmarks: the obsolete, the ubiquitous, the unexplored. In: Proceedings of the 16th ACM International Conference on Extending Database Technology. 2013, 17–28

    Google Scholar 

  23. Liu D W, Luan H, Wang S, Qin B. Main memory database TPC-H workload characterization on modern processor. Journal of Software, 2008, 19(10): 2573–2584

    Article  Google Scholar 

  24. Müller I, Ratsch C, Faerber F. Adaptive string dictionary compression in in-memory column-store database systems. In: Proceedings of the International Conference on Extending Database Technology (EDBT). 2014, 283–294

    Google Scholar 

  25. Abadi D J, Madden S R, Hachem N. Column-storesvs. row-stores: how different are they really? In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. 2008, 967–980

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Data Science and Engineering, School of Computer Science and Software Engineering, East China Normal University, Shanghai, 200062, China

    Cheqing Jin, Yangxin Kong, Qiangqiang Kang, Weining Qian & Aoying Zhou

Authors
  1. Cheqing Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yangxin Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiangqiang Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weining Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aoying Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cheqing Jin.

Additional information

Cheqing Jin is a professor at East China Normal University, China. He received his master and bachelor degrees from Zhejiang University, China in 1999 and 2002 respectively, and PhD degree from Fudan University, China in 2005, all in computer science. He worked as an assistant professor in East China University of Science and Technology, China from 2005 to 2008, afterwards he joined ECNU on October 2008. In 2003 and 2007, he visited the HongKong University and the Chinese University of Hongkong respectively. He has acted as the PCmembers for more than ten conferences. His main research interests include streaming data management, location-based services, uncertain data management, data quality, and database benchmarking.

Yangxin Kong received the bachelor degree from Nantong University, China. He is currently working toward the master degree in the Software Engineering Institute at East China Normal University, China. His research interests include technology and application on location based services, data mining, etc.

Qiangqiang Kang received the bachelor and master degrees from Software Engineering Institute at East China Normal University, China. He is now working in China Merchants Bank after graduation. His research interests mainly include database benchmark and reverse query.

Weining Qian is currently a professor in computer science at East China Normal University, China. He received his MS and PhD in computer science from Fudan University, China in 2001 and 2004, respectively. He served as the co-chair of WISE 2012 Challenge, and program committee member of several international conferences, including ICDE 2009/2010/2012 and KDD 2013. His research interests include Web data management and mining of massive data sets.

Aoying Zhou is a professor on computer science at East China Normal University (ECNU), China where he is heading the Institute for Data Science and Engineering. He got his master and bachelor degree in computer science from Sichuan University, China in 1988 and 1985 respectively, and won his PhD degree from Fudan University, China in 1993. Before joining ECNU in 2008, he worked for Fudan University at the Computer Science Department from 1993 to 2007, where he served as the department chair from 1999 to 2002. He worked as a visiting scholar under the Berkeley Scholar Program in UC Berkeley in 2005. He is the winner of the National Science Fund for Distinguished Young Scholars supported by NSFC and the professorship appointment under Changjiang Scholars Program of Ministry of Education. He is now acting as the vice-director of ACM SIGMOD China and Technology Committee on Database of China Computer Federation. He is serving as a member of the editorial boards of some prestigious academic journals, such as VLDB Journal, and WWW Journal. His research interests include Web data management, data management for data-intensive computing, and in-memory data analytics.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, C., Kong, Y., Kang, Q. et al. Benchmarking in-memory database. Front. Comput. Sci. 10, 1067–1081 (2016). https://doi.org/10.1007/s11704-016-5366-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-016-5366-0

Keywords

Search

Navigation