Skip to main content
SpringerLink
Log in

Efficient query processing framework for big data warehouse: an almost join-free approach

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

The rapidly increasing scale of data warehouses is challenging today’s data analytical technologies. A conventional data analytical platform processes data warehouse queries using a star schema — it normalizes the data into a fact table and a number of dimension tables, and during query processing it selectively joins the tables according to users’ demands. This model is space economical. However, it faces two problems when applied to big data. First, join is an expensive operation, which prohibits a parallel database or a MapReduce-based system from achieving efficiency and scalability simultaneously. Second, join operations have to be executed repeatedly, while numerous join results can actually be reused by different queries.

In this paper, we propose a new query processing framework for data warehouses. It pushes the join operations partially to the pre-processing phase and partially to the post-processing phase, so that data warehouse queries can be transformed into massive parallelized filter-aggregation operations on the fact table. In contrast to the conventional query processing models, our approach is efficient, scalable and stable despite of the large number of tables involved in the join. It is especially suitable for a large-scale parallel data warehouse. Our empirical evaluation on Hadoop shows that our framework exhibits linear scalability and outperforms some existing approaches by an order of magnitude.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chaudhuri S, Dayal U. An overview of data warehousing and olap technology. SIGMOD Record, 1997, 26(1): 65–74

    Article  Google Scholar 

  2. Dean J, Ghemawat S. Mapreduce: Simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating Systems Design and Implementation. 2004, 137–150

    Google Scholar 

  3. Apache hadoop. http://hadoop.apache.org

  4. Pavlo A, Paulson E, Rasin A, Abadi D J, DeWitt D J, Madden S, Stonebraker M. A comparison of approaches to large-scale data analysis. In: Proceedings of the 35th SIGMOD International Conference on Management of Data. 2009, 165–178

    Chapter  Google Scholar 

  5. Afrati F N, Ullman J D. Optimizing joins in a map-reduce environment. In: Proceedings of the 2010 International Conference on Extending Databas Technology. 2010, 99–110

    Google Scholar 

  6. Dawei Jiang G CA. K. H. Map-join-reduce: Towards scalable and efficient data analysis on large clusters. IEEE Transactions on Knowledge and Data Engineering, 2011, 23(9): 1299–1311

    Article  Google Scholar 

  7. Olston C, Reed B, Srivastava U, Kumar R, Tomkins A. Pig latin: a notsoforeign language for data processing. In: Proceedings of the 2008 SIGMOD International Conference on Management of Data. 2008, 1099–1110

    Chapter  Google Scholar 

  8. Dittrich J, Quiané-Ruiz J A, Jindal A, Kargin Y, Setty V, Schad J. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). Proceedings of the VLDB Endowment, 2010, 3(1): 518–529

    Google Scholar 

  9. Floratou A, Patel J M, Shekita E J, Tata S. Column-oriented storage techniques for mapreduce. Proceedings of the VLDB Endowent, 2011, 4(7): 419–429

    Article  Google Scholar 

  10. Lin Y, Agrawal D, Chen C, Ooi B C, Wu S. LLAMA: leveraging columnar storage for scalable join processing in the mapreduce framework. In: Proceedings of the 2011 SIGMOD International Conference on Management of Data. 2011, 961–972

    Chapter  Google Scholar 

  11. Xu Y, Kostamaa P, Gao L. Integrating hadoop and parallel DBMS. In: Proceedings of the 2010 SIGMOD Conference on Management of Data. 2010, 969–974

    Chapter  Google Scholar 

  12. Abouzeid A, Bajda-Pawlikowski K, Abadi D J, Rasin A, Silberschatz A. Hadoopdb: An architectural hybrid of mapreduce and DBMS technologies for analytical workloads. Proceedings of the VLDB Endowment, 2009, 2(1): 922–933

    Article  Google Scholar 

  13. Swami A, Gupta A. Optimization of large join queries. SIGMOD Record, 1988, 17(3): 8–17

    Article  Google Scholar 

  14. Raman V, Swart G, Qiao L, Reiss F, Dialani V, Kossmann D, Narang I, Sidle R. Constant-time query processing. In: Proceedings of the 2008 International Conference of Data Engineering. 2008, 60–69

    Chapter  Google Scholar 

  15. Valduriez P. Join indices. ACM Transactions on Database Systems, 1987, 12: 218–246

    Article  Google Scholar 

  16. Markl V, Ramsak F, Bayer R. Improving OLAP performance by multidimensional hierarchical clustering. In: Proceedings of the 1999 International Symposium on Database Engineering and Applications. 1999, 165–177

    Google Scholar 

  17. Karayannidis N, Tsois A, Sellis T K, Pieringer R, Markl V, Ramsak F, Fenk R, Elhardt K, Bayer R. Processing star queries on hierarchicallyclustered fact tables. In: Proceedings of the 28th VLDB Conference. 2002, 730–741

    Google Scholar 

  18. Bayer R. The universal b-tree for multidimensional indexing: general concepts. In: Proceedings of the 1997 International Conference on Worldwide Computing and Its Applications. 1997, 198–209

    Google Scholar 

  19. Theodoratos D, Tsois A. Heuristic optimization of olap queries in multidimensionally hierarchically clustered databases. In: Proceedings of ACM 4th International Workshop on Data Warehousing and OLAP. 2001, 48–55

    Google Scholar 

  20. Korth H F, Kuper G M, Feigenbaum J, Gelder A V, Ullman J D. System/u: A database system based on the universal relation assumption. ACM Transactions on Database Systems, 1984, 9(3): 331–347

    Article  Google Scholar 

  21. Floratou A, Patel J M, Shekita E J, Tata S. Column-oriented storage techniques for mapreduce. Proceedings of the VLDB Endowent, 2011, 4(7): 419–429

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DEKE Lab (Renmin University of China), Beijing, 100872, China

    Huiju Wang, Xiongpai Qin, Xuan Zhou, Furong Li, Zuoyan Qin, Qing Zhu & Shan Wang

  2. School of Information, Renmin University of China, Beijing, 100872, China

    Huiju Wang, Xiongpai Qin, Furong Li, Qing Zhu & Shan Wang

  3. School of Computing, National University of Singapore, Singapore, 117417, Singapore

    Huiju Wang

Authors
  1. Huiju Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiongpai Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Furong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zuoyan Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huiju Wang.

Additional information

Huiju Wang graduated from Renmin University of China in 2012 and works as postdoctoral research fellow at the school of computing of National University of Singapore. His research spans the areas of big data, clouding computing, databases and data management, with emphasis on graph database, graph index, graph data exploration.

Xiongpai Qin received his MS and PhD degree in computer science from Renmin University of China in 1998 and 2009 respectively, and works as a lecturer at Information School of Renmin University of China. His research interests include semantic based information retrieval, high performance database and big data.

Xuan Zhou is an associate professor at the Renmin University of China. He obtained his BS in computer science from Fudan University, China in 2001, and his PhD from the National University of Singapore in 2005. His research interests include database and information management. He has published his work in the top conferences and journals on data management.

Furong Li is a PhD candidate at National University of Singapore. She obtained her BS from Renmin University of China in 2012. Her research interests include data integration, social networks and big data management.

Zuoyan Qin received his BS and MS from Renmin University of China in 2008 and 2011 respectively. He is one senior engineer in Baidu company one. Before joining Baidu, he worked in Tencent. His main focus is big data processing and cloud computing.

Qing Zhu is an associate professor of School of Information, Renmin University of China. She completed her Phd in 2005 in Renmin University and MS in 1991 in Beijing University of Technology, China. Her research interests include Grid computing, distributed algorithms, Semantic Web service and high performance Database.

Professor Shan Wang finished her undergraduate studies at the Peking University, China in 1968, and completed her Master study at Renmin University of China in 1981. Her research interests include high performance database, data warehouse and knowledge engineering, information retrieval, etc.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Qin, X., Zhou, X. et al. Efficient query processing framework for big data warehouse: an almost join-free approach. Front. Comput. Sci. 9, 224–236 (2015). https://doi.org/10.1007/s11704-014-4025-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-014-4025-6

Keywords

Search

Navigation