Skip to main content
SpringerLink
Log in

Multidimensional Similarity Join Using MapReduce

  • Conference paper
  • First Online:
Web-Age Information Management (WAIM 2016)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9659))

Included in the following conference series:

Abstract

Similarity join is arguably one of the most important operators in multidimensional data analysis tasks. However, processing a similarity join is costly especially for large volume and high dimensional data. In this work, we attempt to process the similarity join on MapReduce such that the join computation can be scaled horizontally. In order to make the workload balancing among all MapReduce nodes, we systemically select the most profitable feature based on a novel data selectivity approach. Given the selected feature, we develop the partitioning scheme for MapReduce processing based on two different optimization goals. Our proposed techniques are extensively evaluated on real datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Gunopulos, D., Kollios, G., Tsotras, J., Domeniconi, C.: Selectivity estimators for multidimensional range queries over real attributes. VLDB J.–Int. J. Very Large Data Bases 14(2), 137–154 (2005)

    Article  Google Scholar 

  2. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 5(1), 107–113 (2008)

    Article  Google Scholar 

  3. Yang, H., Dasdan, A., Hsiao, R.L., Parker, D.S.: Map-reduce-merge: simplified relational data processing on large clusters. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, pp. 1029–1040. ACM (2007)

    Google Scholar 

  4. Blanas, S., Patel, J.M., Ercegovac, V., Rao, J., Shekita, E.J., Tian, Y.: A comparison of join algorithms for log processing in mapreduce. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 975–986. ACM (2010)

    Google Scholar 

  5. Wu, F., Wu, Q., Tan, Y.: Comparison and performance analysis of join approach in mapreduce. In: Yuan, Y., Wu, X., Lu, Y. (eds.) Trustworthy Computing and Services. CCIS, vol. 320, pp. 629–636. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  6. Atta, F., Viglas, S.D., Niazi, S.: Sand joina skew handling join algorithm for google’s mapreduce framework. In: 2011 IEEE 14th International Multitopic Conference (INMIC), pp. 170–175. IEEE (2011)

    Google Scholar 

  7. Lin, Z., Cai, M., Huang, Z., Lai, Y.: SALA: A skew-avoiding and locality-aware algorithm for mapreduce-based join. In: Dong, X.L., Yu, X., Dong, X.L., Li, J., Sun, Y., Sun, Y. (eds.) WAIM 2015. LNCS, vol. 9098, pp. 311–323. Springer, Heidelberg (2015). doi:10.1007/978-3-319-21042-1_25

    Chapter  Google Scholar 

  8. Zhang, S., Han, J., Liu, Z., Wang, K., Xu, Z.: SJMR: Parallelizing spatial join with mapreduce on clusters. In: 2009 IEEE International Conference on Cluster Computing and Workshops, CLUSTER 2009, pp. 1–8. IEEE (2009)

    Google Scholar 

  9. Vernica, R., Carey, M.J., Li, C.: Efficient parallel set-similarity joins using mapreduce. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 495–506. ACM (2010)

    Google Scholar 

  10. Böhm, C., Braunmüller, B., Krebs, F., Kriegel, H.P.: Epsilon grid order: an algorithm for the similarity join on massive high-dimensional data. In: ACM SIGMOD Record, vol. 30, pp. 379–388. ACM (2001)

    Google Scholar 

  11. Kalashnikov, D.V.: Super-ego: fast multi-dimensional similarity join. VLDB J. Int. J. Very Large Data Bases 22(4), 561–585 (2013)

    Article  MathSciNet  Google Scholar 

  12. Chaudhuri, S., Motwani, R., Narasayya, V.: On random sampling over joins. In: ACM SIGMOD Record, vol. 28, pp. 263–274. ACM (1999)

    Google Scholar 

  13. Olken, F., Rotem, D.: Simple random sampling from relational databases. In: VLDB, vol. 86, pp. 25–28 (1986)

    Google Scholar 

  14. Das Sarma, A., He, Y., Chaudhuri, S.: Clusterjoin: a similarity joins framework using map-reduce. Proc. VLDB Endowment 7(12), 1059–1070 (2014)

    Article  Google Scholar 

  15. Wang, Y., Metwally, A., Parthasarathy, S.: Scalable all-pairs similarity search in metric spaces. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 829–837. ACM (2013)

    Google Scholar 

Download references

Acknowledgements

This work was supported by grant MYRG109(Y1-L3)-FST12-ULH from UMAC Research Committee and grant NSFC 61502548 from National Natural Science Foundation of China.

Author information

Authors and Affiliations

  1. Zhuhai Research Institute, University of Macau, Macau, China

    Ye Li, Jian Wang & Leong Hou U

Authors
  1. Ye Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leong Hou U
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leong Hou U .

Editor information

Editors and Affiliations

  1. Peking University , Beijing, China

    Bin Cui

  2. The George Washington University , Washington, D.C., USA

    Nan Zhang

  3. Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

    Jianliang Xu

  4. University of Texas Rio Grande Valley, Edinburg, Texas, USA

    Xiang Lian

  5. Jiangxi University of Finance and Economics, Nanchang, Jiangxi, China

    Dexi Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Li, Y., Wang, J., U, L.H. (2016). Multidimensional Similarity Join Using MapReduce. In: Cui, B., Zhang, N., Xu, J., Lian, X., Liu, D. (eds) Web-Age Information Management. WAIM 2016. Lecture Notes in Computer Science(), vol 9659. Springer, Cham. https://doi.org/10.1007/978-3-319-39958-4_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-39958-4_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-39957-7

  • Online ISBN: 978-3-319-39958-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation