Dhathri Purohith
ABSTRACT
Benchmarking systems in a repeatable fashion is complex and error-prone. The systems community has repeatedly discussed the complexities of benchmarking and how to properly report benchmarking results. Using the example of SQLite, we examine the current state of benchmarking in industry and academia. We show that changing just one parameter in SQLite can change the performance by 11.8X, and that changing multiple parameters can lead up to a 28X difference in performance. We find that these configuration parameters are often not set or reported in academic research, leading to incomplete and misleading evaluations. Running different off-the-shelf SQLite benchmarking tools such as Mobibench and Androbench in their default configuration shows upto 50% difference in performance. We hope this paper incites discussion in the systems community and among SQLite developers. We hope that our detailed analysis will help application developers to choose optimal SQLite parameters to achieve better performance.
- Brian F Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM symposium on Cloud computing, pages 143--154. ACM, 2010. Google Scholar
Digital Library
- Google. Leveldb. https://github.com/google/leveldb.Google Scholar
- RocksDB | A persistent key-value store. http://rocksdb.org.Google Scholar
- Transaction Processing Performance Council. Tpc benchmark c, standard specification version 5, 2001.Google Scholar
- Transaction Processing Performance Council. 'tpc benchmark b. Standard Specification, Waterside Associates, Fremont, CA, 1990.Google Scholar
- Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney. Producing wrong data without doing anything obviously wrong! In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, pages 265--276, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- Vasily Tarasov, Saumitra Bhanage, Erez Zadok, and Margo Seltzer. Benchmarking file system benchmarking: It* is* rocket science. In HotOS, 2011.Google Scholar
- SQLite. SQLite transactional SQL database engine. http://www.sqlite.org/.Google Scholar
- Why to use SQLite in Android. https://www.quora.com/Why-should-we-use-SQLite-in-Android-development, January 2017.Google Scholar
- SQLite in Android. http://www.grokkingandroid.com/sqlite-in-android, January 2017.Google Scholar
- SQLite. Most Widely Deployed SQL Database Engine. https://www.sqlite.org/mostdeployed.html.Google Scholar
- Suli Yang, Tyler Harter, Nishant Agrawal, Salini Selvaraj Kowsalya, Anand Krishnamurthy, Samer Al-Kiswany, Rini T Kaushik, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. Split-level i/o scheduling. In Proceedings of the 25th Symposium on Operating Systems Principles, pages 474--489. ACM, 2015.Google ScholarDigital Library
- Pierre Olivier, Jalil Boukhobza, and Eric Senn. Revisiting read-ahead efficiency for raw nand flash storage in embedded linux. ACM SIGBED Review, 11(4):43--48, 2015.Google ScholarDigital Library
- Wook-Hee Kim, Jinwoong Kim, Woongki Baek, Beomseok Nam, and Youjip Won. Nvwal: Exploiting nvram in write-ahead logging. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, pages 385--398. ACM, 2016. Google ScholarDigital Library
- Dmitrii Kuvaiskii, Rasha Faqeh, Pramod Bhatotia, Pascal Felber, and Christof Fetzer. Haft: hardware-assisted fault tolerance. In Proceedings of the Eleventh European Conference on Computer Systems, page 25. ACM, 2016.Google Scholar
- Publications involving SQLite. http://dl.acm.org/results.cfm?query=(sqlite)&within=owners.owner=HOSTED&filtered=&dte=2008&bfr=, January 2017.Google Scholar
- Nikolaos Ch. Kasmeridis and Michael Gr. Vassilakopoulos. A diet-guide mobile application for diabetes mellitus management. In Proceedings of the 19th Pan-hellenic Conference on Informatics, PCI '15, pages 377--381, New York, NY, USA, 2015. ACM.Google Scholar
- Simone Mutti, Enrico Bacis, and Stefano Paraboschi. Sesqlite: Security enhanced sqlite: Mandatory access control for android databases. In Proceedings of the 31st Annual Computer Security Applications Conference, ACSAC 2015, pages 411--420, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- Eunryoung Lim, Seongjin Lee, and Youjip Won. Androtrace: Framework for tracing and analyzing ios on android. In Proceedings of the 3rd Workshop on Interactions of NVM/FLASH with Operating Systems and Workloads, INFLOW '15, pages 3:1--3:8, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- Suli Yang, Tyler Harter, Nishant Agrawal, Salini Selvaraj Kowsalya, Anand Krishnamurthy, Samer Al-Kiswany, Rini T. Kaushik, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Split-level i/o scheduling. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP '15, pages 474--489, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- Dmitrii Kuvaiskii, Rasha Faqeh, Pramod Bhatotia, Pascal Felber, and Christof Fetzer. Haft: Hardware-assisted fault tolerance. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys '16, pages 25:1--25:17, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- Charlie Curtsinger and Emery D. Berger. Coz: Finding code that counts with causal profiling. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP '15, pages 184--197, New York, NY, USA, 2015. ACM. Google ScholarDigital Library
- Yan Wang and Atanas Rountev. Profiling the responsiveness of android applications via automated resource amplification. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft '16, pages 48--58, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- Georgios Chatzopoulos, Aleksandar Dragojević, and Rachid Guerraoui. Estima: Extrapolating scalability of in-memory applications. In Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP '16, pages 27:1--27:11, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- Pierre Olivier, Jalil Boukhobza, and Eric Senn. Revisiting read-ahead efficiency for raw nand flash storage in embedded linux. SIGBED Rev., 11(4):43--48, January 2015. Google ScholarDigital Library
- Phil Mcminn, Chris J. Wright, and Gregory M. Kapfhammer. The effectiveness of test coverage criteria for relational database schema integrity constraints. ACM Trans. Softw. Eng. Methodol., 25(1):8:1--8:49, December 2015.Google ScholarDigital Library
- Lei Li, Kai Qian, Qian Chen, Ragib Hasan, and Guifeng Shao. Developing hands-on labware for emerging database security. In Proceedings of the 17th Annual Conference on Information Technology Education, SIGITE '16, pages 60--64, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- Gihwan Oh, Sangchul Kim, Sang-Won Lee, and Bongki Moon. Sqlite optimization with phase change memory for mobile applications. Proc. VLDB Endow., 8(12):1454--1465, August 2015. Google ScholarDigital Library
- Wook-Hee Kim, Jinwoong Kim, Woongki Baek, Beomseok Nam, and Youjip Won. Nvwal: Exploiting nvram in write-ahead logging. SIGOPS Oper. Syst. Rev., 50(2):385--398, March 2016. Google ScholarCross Ref
- Changman Lee, Dongho Sim, Joo-Young Hwang, and Sangyeun Cho. F2fs: A new file system for flash storage. In Proceedings of the 13th USENIX Conference on File and Storage Technologies, FAST'15, pages 273--286, Berkeley, CA, USA, 2015. USENIX Association.Google Scholar
- Eric Koskinen and Junfeng Yang. Reducing crash recoverability to reachability. SIGPLAN Not., 51(1):97--108, January 2016. Google ScholarDigital Library
- Dam Quang Tuan, Seungyong Cheon, and Youjip Won. On the io characteristics of the sqlite transactions. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft '16, pages 214--224, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- Sooman Jeong, Kisung Lee, Jungwoo Hwang, Seongjin Lee, and Youjip Won. Androstep: Android storage performance analysis tool. In Software Engineering (Workshops), volume 13, pages 327--340, 2013.Google Scholar
- RL Benchmark. https://play.google.com/store/apps/details?id=com.redlicense.benchmark.sqlite&hl=en, December 2016.Google Scholar
- Je-Min Kim and Jin-Soo Kim. Androbench: Benchmarking the storage performance of android-based mobile devices. In Frontiers in Computer Education, pages 667--674. Springer, 2012.Google ScholarCross Ref
- Hyojun Kim, Nitin Agrawal, and Cristian Ungureanu. Revisiting storage for smartphones. Trans. Storage, 8(4):14:1--14:25, December 2012.Google ScholarDigital Library
- WAL mode in SQLite. https://www.sqlite.org/wal.html, January 2017.Google Scholar
- No journal in SQLite. https://www.sqlite.org/pragma.html#pragma_journal_mode, January 2017.Google Scholar
- Vijayan Prabhakaran, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Analysis and Evolution of Journaling File Systems. In The Proceedings of the USENIX Annual Technical Conference (USENIX '05), pages 105--120, Anaheim, CA, April 2005.Google Scholar
Recommendations
SQLite: past, present, and future
In the two decades following its initial release, SQLite has become the most widely deployed database engine in existence. Today, SQLite is found in nearly every smartphone, computer, web browser, television, and automobile. Several factors are likely ...
Comments