Skip to main content
SpringerLink
Log in
Book cover

Technology Conference on Performance Evaluation and Benchmarking

TPCTC 2019: Performance Evaluation and Benchmarking for the Era of Cloud(s) pp 99–115Cite as

Benchmarking Pocket-Scale Databases

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12257))

Abstract

Embedded database libraries provide developers with a common and convenient data persistence layer. They are a key component of major mobile operating systems, and are used extensively on interactive devices like smartphones. Database performance affects the response times and resource consumption of millions of smartphone apps and billions of smartphone users. Given their wide use and impact, it is critical that we understand how embedded databases operate in realistic mobile settings, and how they interact with mobile environments. We argue that traditional database benchmarking methods produce misleading results when applied to mobile devices, due to evaluating performance only at saturation. To rectify this, we present PocketData, a new benchmark for mobile device database evaluation that uses typical workloads to produce representative performance results. We explain the performance measurement methodology behind PocketData, and address specific challenges. We analyze the results obtained, and show how different classes of workload interact with database performance. Notably, our study of mobile databases at non-saturated levels uncovers significant latency and energy variation in database workloads resulting from CPU frequency scaling policies called governors—variation that we show is hidden by typical benchmark measurement techniques.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Available for download at http://pocketdata.info.

  2. 2.

    Here, we follow [18], which observes lognormal delay with a 6ms mean in typical use.

  3. 3.

    Complete benchmark results are available at http://www.pocketdata.info/.

  4. 4.

    http://www.msoon.com/LabEquipment/.

References

  1. Ahmed, M., Uddin, M.M., Azad, M.S., Haseeb, S.: Mysql performance analysis on a limited resource server: fedora vs. ubuntu linux. In: SpringSim (2010)

    Google Scholar 

  2. Atikoglu, B., Xu, Y., Frachtenberg, E., Jiang, S., Paleczny, M.: Workload analysis of a large-scale key-value store. SIGMETRICS Perform. Eval. Rev. 40(1), 53–64 (2012)

    Article  Google Scholar 

  3. Baumgärtel, P., Endler, G., Lenz, R.: A benchmark for multidimensional statistical data. In: ADBIS (2013)

    Google Scholar 

  4. Bitton, D., DeWitt, D.J., Turbyfill, C.: Benchmarking database systems a systematic approach. In: VLDB, pp. 8–19. Morgan Kaufmann (1983)

    Google Scholar 

  5. Cao, Y., Nejati, J., Wajahat, M., Balasubramanian, A., Gandhi, A.: Deconstructingthe energy consumption of the mobile page load. PMACS 1(1), 6:1–6:25 (2017)

    Google Scholar 

  6. Carroll, A., Heiser, G.: An analysis of power consumption in a smartphone. In: USENIXATC, p. 21 (2010)

    Google Scholar 

  7. Chen, X., Chen, Y., Dong, M., Zhang, C.: Demystifying energy usage in smartphones. In: DAC (2014)

    Google Scholar 

  8. Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: SOCC (2010)

    Google Scholar 

  9. Curino, C.A., Difallah, D.E., Pavlo, A., Cudre-Mauroux, P.: Benchmarking OLTP/web databases in the cloud: the OLTP-bench framework. In: CloudDB (2012)

    Google Scholar 

  10. Dietrich, B., Chakraborty, S.: Power management using game state detection on android smartphones. In: MobiSys, pp. 493–494 (2013)

    Google Scholar 

  11. Egilmez, B., Memik, G., Ogrenci-Memik, S., Ergin, O.: User-specific skin temperature-aware dvfs for smartphones. In: DATE, pp. 1217–1220 (2015)

    Google Scholar 

  12. Erling, O., et al.: The LDBC social network benchmark: interactive workload. In: SIGMOD (2015)

    Google Scholar 

  13. Frank, M., Poess, M., Rabl, T.: Efficient update data generation for DBMS benchmarks. In: ICPE (2012)

    Google Scholar 

  14. Google: Android open source project (2018). https://source.android.com/

  15. Gupta, A., Davis, K.C., Grommon-Litton, J.: Performance comparison of property map and bitmap indexing. In: DOLAP, pp. 65–71. ACM (2002)

    Google Scholar 

  16. Hussein, A., Payer, M., Hosking, A., Vick, C.A.: Impact of GC design on power and performance for android. In: SYSTOR. pp, 13:1–13:12 (2015)

    Google Scholar 

  17. Kambadur, M., Kim, M.A.: An experimental survey of energy management across the stack. In: OOPSLA, pp. 329–344 (2014)

    Google Scholar 

  18. Kennedy, O., Ajay, J.A., Challen, G., Ziarek, L.: Pocket data: the need for TPC-MOBILE. In: TPC-TC (2015)

    Google Scholar 

  19. Kim, J., Kim, J.: Androbench: Benchmarking the storage performance of android-based mobile devices. In: Sambath, S., Zhu, E. (eds.) Advances in Intelligent and Soft Computing, ICFCE, vol. 133, pp. 667–674. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-27552-4_89

  20. Klein, J., Gorton, I., Ernst, N.A., Donohoe, P., Pham, K., Matser, C.: Performance evaluation of NoSql databases: a case study. In: PABS@ICPE, pp. 5–10. ACM (2015)

    Google Scholar 

  21. Kuhlenkamp, J., Klems, M., Röss, O.: Benchmarking scalability and elasticity of distributed database systems. PVLDB 7(12), 1219–1230 (2014)

    Google Scholar 

  22. Lee, K.: Mobile benchmark tool (mobibench)

    Google Scholar 

  23. Madden, S.R., Franklin, M.J., Hellerstein, J.M., Hong, W.: TinyDB: an acquisitional query processing system for sensor networks. ACM TODS 30(1), 122–173 (2005)

    Article  Google Scholar 

  24. Malkowski, S., Jayasinghe, D., Hedwig, M., Park, J., Kanemasa, Y., Pu, C.: Empirical analysis of database server scalability using an n-tier benchmark with read-intensive workload. In: SAC (2010)

    Google Scholar 

  25. Mercati, P., Bartolini, A., Paterna, F., Rosing, T.S., Benini, L.: A linux-governor based dynamic reliability manager for android mobile devices. In: DATE, pp. 104:1–104:4 (2014)

    Google Scholar 

  26. Nandi, A., Jiang, L., Mandel, M.: Gestural query specification. PVLDB 7(4), 289–300 (2013)

    Google Scholar 

  27. Niemann, R.: Towards the prediction of the performance and energy efficiency of distributed data management systems. In: ICPE Companion, pp. 23–28 (2016)

    Google Scholar 

  28. Olson, M.A., Bostic, K., Seltzer, M.I.: Berkeley DB. In: USENIX Annual Technical Conference, FREENIX Track, pp. 183–191. USENIX (1999)

    Google Scholar 

  29. O’Neil, P.E., O’Neil, E.J., Chen, X.: The star schema benchmark (SSB) (2007)

    Google Scholar 

  30. Owens, M., Allen, G.: SQLite. Springer, Cham (2010). https://doi.org/10.1007/978-1-4302-0136-6_22

  31. Poess, M., Nambiar, R.O.: Energy cost, the key challenge of today’s data centers: a power consumption analysis of TPC-C results. PVLDB 1(2), 1229–1240 (2008)

    Google Scholar 

  32. Poess, M., Nambiar, R.O., Vaid, K.: Optimizing benchmark configurations for energy efficiency. In: ICPE, pp. 217–226. ACM (2011)

    Google Scholar 

  33. Poess, M., Nambiar, R.O., Vaid, K., Stephens, J.M., Huppler, K., Haines, E.: Energy benchmarks: a detailed analysis. In: e-Energy, pp. 131–140. ACM (2010)

    Google Scholar 

  34. Srinivasa, G.P., Begum, R., Haseley, S., Hempstead, M., Challen, G.: Separated by birth: hidden differences between seemingly-identical smartphone Cpus. In: HotMobile, pp. 103–108. ACM (2017)

    Google Scholar 

  35. Tomás, G., et al.: Fmke: a real-world benchmark for key-value data stores. In: PaPoC (2017)

    Google Scholar 

  36. Transaction Processing Performance Council: TPC-H, TPC-C, and TPC-DS specifications. http://www.tpc.org/

  37. Wang, Y., Rountev, A.: Profiling the responsiveness of android applications via automated resource amplification. In: MOBILESoft, pp. 48–58. ACM (2016)

    Google Scholar 

  38. Wilke, C., Piechnick, C., Richly, S., Püschel, G., Götz, S., Aßmann, U.: Comparing mobile applications’ energy consumption. In: SAC, pp. 1177–1179. ACM (2013)

    Google Scholar 

  39. Yang, S., Yan, D., Rountev, A.: Testing for poor responsiveness in Android applications. In: Workshop on Engineering Mobile-Enabled Systems, pp. 1–6 (2013)

    Google Scholar 

  40. Yarger, R.J., Reese, G., King, T.: MySQL and mSQL - databases formoderate-sized organizations and websites. O’Reilly (1999)

    Google Scholar 

Download references

Acknowledgments

This work is supported by NSF Awards IIS-1617586, CNS-1629791 and CCF 1749539.

Author information

Authors and Affiliations

  1. University at Buffalo, Buffalo, NY, 14260, USA

    Carl Nuessle, Oliver Kennedy & Lukasz Ziarek

Authors
  1. Carl Nuessle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver Kennedy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lukasz Ziarek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carl Nuessle .

Editor information

Editors and Affiliations

  1. Advanced Micro Devices (United States), Santa Clara, CA, USA

    Raghunath Nambiar

  2. Oracle Corporation, Redwood Shores, CA, USA

    Meikel Poess

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nuessle, C., Kennedy, O., Ziarek, L. (2020). Benchmarking Pocket-Scale Databases. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking for the Era of Cloud(s). TPCTC 2019. Lecture Notes in Computer Science(), vol 12257. Springer, Cham. https://doi.org/10.1007/978-3-030-55024-0_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-55024-0_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-55023-3

  • Online ISBN: 978-3-030-55024-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation