Vincent Reniers
ABSTRACT
The proliferation of Big Data systems and namely NoSQL databases has resulted in a tremendous heterogeneity in its offerings. It has become increasingly difficult to compare and select the most optimal NoSQL storage technology. Current benchmark efforts, such as the Yahoo! Cloud Serving Benchmark (YCSB), evaluate simple read and write operations on a primary key. However, while YCSB has become the de-facto benchmark solution for practitioners and NoSQL vendors, it is lacking in capabilities to extensively evaluate specific NoSQL solutions.
In this paper, we present a systematic survey of current NoSQL benchmarks, in which we identify a clear gap in benchmarking more advanced workloads (e.g. nested document search) for features specific to NoSQL database families (e.g. document stores). Secondly, based on our survey, we discuss the strengths and weaknesses of the different benchmark design approaches, and argue in favor of a benchmark suite that targets specific families of NoSQL databases yet still allows overall comparison of databases in terms of their commonalities.
- Ahmad Ghazal et al. BigBench: towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data. ACM, 2013. Google Scholar
Digital Library
- T. G. Armstrong, V. Ponnekanti, D. Borthakur, and M. Callaghan. LinkBench: a database benchmark based on the Facebook social graph. In ACM SIGMOD '13. ACM, 2013. Google ScholarDigital Library
- C. B\uaz\uar, C. S. Iosif, et al. The Transition from RDBMS to NoSQL. A Comparative Analysis of Three Popular Non-Relational Solutions: Cassandra, MongoDB and Couchbase. Database Systems Journal, 5(2):49--59, 2014.Google Scholar
- S. Beis, S. Papadopoulos, and Y. Kompatsiaris. Benchmarking graph databases on the problem of community detection. In New Trends in Database and Information Systems II. Springer, 2015. Google ScholarCross Ref
- BigDataBench. Bigdatabench. http://prof.ict.ac.cn/BigDataBench/.Google Scholar
- R. Cattell. Scalable SQL and NoSQL data stores. ACM SIGMOD Record, 39(4):12--27, 2011. Google ScholarDigital Library
- M. Ciglan, A. Averbuch, and L. Hluchy. Benchmarking traversal operations over graph databases. In Data Engineering Workshops (ICDEW). IEEE, 2012. Google ScholarDigital Library
- B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM symposium on Cloud computing. ACM, 2010. Google ScholarDigital Library
- M. Dayarathna and T. Suzumura. XGDBench: A benchmarking platform for graph stores in exascale clouds. In Cloud Computing Technology and Science (CloudCom). IEEE, 2012. Google ScholarDigital Library
- D. J. DeWitt. The Wisconsin Benchmark: Past, Present, and Future., 1993.Google Scholar
- A. Dey, A. Fekete, R. Nambiar, and U. Röhm. YCSBGoogle Scholar
- T: Benchmarking web-scale transactional databases. In ICDEW '14. IEEE, 2014.Google Scholar
- Giuseppe DeCandia et al. Dynamo: amazon's highly available key-value store. ACM SIGOPS Operating Systems Review, 41(6):205--220, 2007. Google ScholarDigital Library
- S. Gokhale, N. Agrawal, S. Noonan, and C. Ungureanu. Kvzone and the search for a write-optimized key-value store. In HotStorage, 2010.Google Scholar
- K. Grolinger, W. A. Higashino, A. Tiwari, and M. A. Capretz. Data management in cloud environments: Nosql and newsql data stores. Journal of Cloud Computing: Advances, Systems and Applications, 2(1):1, 2013.Google ScholarDigital Library
- IBM. IBM: The FOUR V's of Big Data. http://www-01.ibm.com/software/data/bigdata/.Google Scholar
- S. Jouili and V. Vansteenberghe. An empirical comparison of graph databases. In Social Computing (SocialCom). IEEE, 2013. Google ScholarDigital Library
- A. Lakshman and P. Malik. Cassandra: a decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2):35--40, 2010. Google ScholarDigital Library
- I. Lungu, B. G. Tudorica, et al. The Development of a Benchmark Tool for NoSQL Databases. Database Systems Journal BOARD, 13, 2013.Google Scholar
- Max Chevalier et al. Benchmark for OLAP on NoSQL technologies. In RCIS '15. IEEE, 2015.Google Scholar
- Michael Stonebraker et al. The End of an Architectural Era: (It's Time for a Complete Rewrite). VLDB '07. VLDB Endowment, 2007.Google Scholar
- NoSQL databases. NoSQL databases. http://www.nosql-database.org.Google Scholar
- Parinaz Ameri et al. NoWog: A Workload Generator for Database Performance Benchmarking. In (DASC/PiCom/DataCom/CyberSciTech). IEEE, 2016.Google Scholar
- Pouria Pirzadeh et al. Performance evaluation of range queries in key value stores. Journal of Grid Computing, 10(1):109--132, 2012. Google ScholarDigital Library
- D. Pritchett. BASE: An Acid Alternative. Queue, 6(3):48--55, 2008. Google ScholarDigital Library
- V. Reniers, A. Rafique, D. Van Landuyt, and W. Joosen. Object-NoSQL Database Mappers: a benchmark study on the performance overhead. Journal of Internet Services and Applications, 8(1):1, 2017. Google ScholarCross Ref
- Rui Han et al. Benchmarking big data systems: State-of-the-art and future directions. arXiv preprint arXiv:1506.01494, 2015.Google Scholar
- M. Stonebraker. Sql databases v. nosql databases. Communications of the ACM, 53(4):10--11, 2010. Google ScholarDigital Library
- Swapnil Patil et al. YCSBGoogle Scholar
- : benchmarking and performance debugging advanced features in scalable table stores. In Proceedings of the 2nd ACM Symposium on Cloud Computing. ACM, 2011.Google Scholar
- TPC. Transaction Processing Performance Council. http://www.tpc.org/.Google Scholar
Index Terms
- On the State of NoSQL Benchmarks
Recommendations
Comparing NoSQL MongoDB to an SQL DB
ACMSE '13: Proceedings of the 51st ACM Southeast ConferenceNoSQL database solutions are becoming more and more prevalent in a world currently dominated by SQL relational databases. NoSQL databases were designed to provide database solutions for large volumes of data that is not structured. However, the ...
NoSQL databases: MongoDB vs cassandra
C3S2E '13: Proceedings of the International C* Conference on Computer Science and Software EngineeringIn the past, relational databases were used in a large scope of applications due to their rich set of features, query capabilities and transaction management. However, they are not able to store and process big data effectively and are not very ...
NewSQL Through the Looking Glass
iiWAS2019: Proceedings of the 21st International Conference on Information Integration and Web-based Applications & ServicesSeveral applications require to handle large and heterogeneous data volumes as well as thousands of OLTP transactions per second. Traditional relational databases are not suitable for these requirements. On the other hand, NoSQL databases are able to ...
Comments