Abstract
We propose four methods for improving the accuracy of aggregation query-result estimation using histograms and/or kernel density estimation and the efficiency of query processing on a distributed key-value store (D-KVS). Recently, aggregation queries have played a key role in analyzing a large amount of multidimensional data generated from sensors, Internet-of-Things devices, etc. A D-KVS is a platform to manage and process such large-scale multidimensional data. However, querying large-scale multidimensional data on a D-KVS sometimes requires a costly data scan owing to its insufficient support for indexes. Since aggregation-query results do not always need to be accurate, our four methods are not only for estimating accurate query results rather than obtaining accurate results by scanning all data, but also improving query-processing performance. We first propose two kernel density estimation-based methods. To further improve query-result estimation accuracy, we combined each of these two methods with a histogram-based scheme so that we can dynamically select an optimal estimation method based on the relationship between a query and the data distribution. We evaluated the efficiency and accuracy of the proposed methods by comparing them with a current method and showed that the proposed methods perform better.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brinkhoff, T.: A framework for generating network-based moving objects. GeoInform. 6(2), 153–180 (2002)
Chakrabarti, K., Garofalakis, M.N., Rastogi, R., Shim, K.: Approximate query processing using wavelets. Proc. VLDB 2000, 111–122 (2000)
Eldawy, A., Mokbel, M.F.: Spatialhadoop: a mapreduce framework for spatial data. Proc. of IEEE ICDE 2015, 1352–1363 (2015)
Garcia-Molina, H., Ullman, J.D., Widom, J.: Database Systems: The Complete Book. Prentice Hall, New Jersey (2002)
Han, X., Wang, B., Li, J., Gao, H.: Efficiently processing deterministic approximate aggregation query on massive data. Knowl. Inf. Syst. 57(2), 437–473 (2018)
Heule, S., Nunkesser, M., Hall, A.: Hyperloglog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In: Proceedings of EDBT 2013. pp. 683–692. ACM (2013)
Ioannidis, Y.: The history of histograms (abridged). Proc. VLDB 2003, 19–30 (2003)
Jagadish, H.V., Koudas, N., Muthukrishnan, S., Poosala, V., Sevcik, K.C., Suel, T.: Optimal histograms with quality guarantees. In: Proceedings of VLDB 1998, pp. 275–286 (1998)
Kooi, R.P.: The Optimization of Queries in Relational Databases. Ph.D. thesis (1980)
Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. ACM SIGOPS Operating Syst. Rev. 44(2), 35–40 (2010)
Muralikrishna, M., DeWitt, D.J.: Equi-depth multidimensional histograms. In: Proceedings of ACM SIGMOD 1988. pp. 28–36 (1988)
Nishimura, S., Agrawal, S.D.D., Abbadi, A.E.: MD-HBase: design and implementation of an elastic data infrastructure for cloud-scale location services. Distrib. Parallel Databases 31(2), 289–319 (2013)
Papadias, D., Kalnis, P., Zhang, J., Tao, Y.: Efficient OLAP operations in spatial data warehouses. In: Jensen, C.S., Schneider, M., Seeger, B., Tsotras, V.J. (eds.) SSTD 2001. LNCS, vol. 2121, pp. 443–459. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-47724-1_23
Piatetsky-Shapiro, G., Connell, C.: Accurate estimation of the number of tuples satisfying a condition. In: Proceedings of ACM SIGMOD 1984, pp. 256–276 (1984)
Poosala, V., Haas, P.J., Ioannidis, Y.E., Shekita, E.J.: Improved histograms for selectivity estimation of range predicates. In: Proceedings of ACM SIGMOD 1996, pp. 294–305 (1996)
Poosala, V., Ioannidis, Y.E.: Selectivity estimation without the attribute value independence assumption. In: Proceedings of VLDB 1997, pp. 486–495 (1997)
Silverman, B.W.: Density Estimation for Statistics and Data Analysis. No. 26 in Monographs on Statistics and Applied Probability. CRC Press (1986)
Wang, J., Wu, S., Gao, H., Li, J., Ooi, B.C.: Indexing multi-dimensional data in a cloud system. Proc. ACM SIGMOD 2010, 591–602 (2010)
Watari, Y., Keyaki, A., Miyazaki, J., Nakamura, M.: Efficient Aggregation Query Processing for Large-Scale Multidimensional Data by Combining RDB and KVS. In: Hartmann, S., Ma, H., Hameurlain, A., Pernul, G., Wagner, R.R. (eds.) DEXA 2018. LNCS, vol. 11029, pp. 134–149. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98809-2_9
Zhang, X., Ai, J., Wang, Z., Lu, J., Meng, X.: An efficient multi-dimensional index for cloud data management. In: Proceedings of CloudDB 2009, pp. 17–24. ACM (2009)
Acknowledgments
This work was partly supported by JSPS KAKENHI Grant Numbers 18H03242, 18H03342, and 19H01138.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yuki, K., Keyaki, A., Miyazaki, J., Nakamura, M. (2019). Accurate Aggregation Query-Result Estimation and Its Efficient Processing on Distributed Key-Value Store. In: Ordonez, C., Song, IY., Anderst-Kotsis, G., Tjoa, A., Khalil, I. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2019. Lecture Notes in Computer Science(), vol 11708. Springer, Cham. https://doi.org/10.1007/978-3-030-27520-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-27520-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-27519-8
Online ISBN: 978-3-030-27520-4
eBook Packages: Computer ScienceComputer Science (R0)