Skip to main content
Springer Nature Link
Log in

Negotiation Strategy of Divisible Tasks for Large Dataset Processing

  • Conference paper
  • First Online:
Multi-Agent Systems and Agreement Technologies (EUMAS 2017, AT 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10767))

  • 738 Accesses

Abstract

MapReduce is a design pattern for processing large datasets on a cluster. Its performances depend on some data skews and on the runtime environment. In order to tackle these problems, we propose an adaptive multiagent system. The agents interact during the data processing and the dynamic task allocation is the outcome of negotiations. These negotiations aim at improving the workload partition among the nodes within a cluster and so decrease the runtime of the whole process. Moreover, since the negotiations are iterative the system is responsive in case of node performance variations. In this paper, we show how, when a task is divisible, an agent may split it in order to negotiate its subtasks.

This project is supported by the CNRS Challenge Mastodons.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Fault tolerance is out of the scope of our study.

  2. 2.

    http://webscope.sandbox.yahoo.com/.

  3. 3.

    It is worth noticing that the negotiations and the data processing are not sequential but concurrent.

References

  1. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: SOSDI, pp. 137–150 (2004)

    Google Scholar 

  2. Kwon, Y., Balazinska, M., Howe, B., Rolia, J.: Skewtune: mitigating skew in MapReduce applications. In: ACM SIGMOD ICMD, pp. 25–36 (2012)

    Google Scholar 

  3. Kwon, Y., Ren, K., Balazinska, M., Howe, B.: Managing skew in Hadoop. IEEE Data Eng. Bull. 36(1), 24–33 (2013)

    Google Scholar 

  4. Baert, Q., Caron, A.C., Morge, M., Routier, J.C.: Fair multi-agent task allocation for large datasets analysis. KAIS (2017). https://doi.org/10.1007/s10115-017-1087-4

    Article  Google Scholar 

  5. Lama, P., Zhou, X.: Aroma: automated resource allocation and configuration of MapReduce environment in the cloud. In: ICAC, pp. 63–72 (2012)

    Google Scholar 

  6. Verma, A., Cherkasova, L., Campbell, R.: Aria: automatic resource inference and allocation for MapReduce environments. In: ICAC, pp. 235–244 (2011)

    Google Scholar 

  7. Chen, Q., Zhang, D., Guo, M., Deng, Q., Guo, S.: SAMR: a self-adaptive MapReduce scheduling algorithm in heterogeneous environment. In: ICCIT, pp. 2736–2743. IEEE (2010)

    Google Scholar 

  8. Liroz-Gistau, M., Akbarinia, R., Valduriez, P.: FP-Hadoop: efficient execution of parallel jobs over skewed data. VLDB Endow. 8(12), 1856–1859 (2015)

    Article  Google Scholar 

  9. Li, W.: Random texts exhibit Zipf’s-law-like word frequency distribution. IEEE Trans. Inf. Theory 38(6), 1842–1845 (1992)

    Article  Google Scholar 

  10. Lin, J.: The curse of Zipf and limits to parallelization: a look at the stragglers problem in MapReduce. In: Workshop on Large-Scale Distributed Systems for Information Retrieval (2009)

    Google Scholar 

  11. Gufler, B., Augsten, N., Reiser, A., Kemper, A.: Handling data skew in MapReduce. In: ICCCSS, pp. 574–583 (2011)

    Google Scholar 

  12. Vinyals, M., Macarthur, K.S., Farinelli, A., Ramchurn, S.D., Jennings, N.R.: A message-passing approach to decentralized parallel machine scheduling. Comput. J. 57(6), 856–874 (2014)

    Article  Google Scholar 

  13. Nongaillard, A., Mathieu, P.: Egalitarian negotiations in agent societies. AAI 25(9), 799–821 (2011)

    Google Scholar 

  14. Essa, Y.M., Attiya, G., El-Sayed, A.: Mobile agent based new framework for improving big data analysis. IJACSA 5(3), 25–32 (2014)

    Google Scholar 

  15. Gray, J., et al.: Data cube: a relational aggregation operator generalizing group-by, cross-tab, and sub-totals. Data Mining Knowl. Discov. 1(1), 29–53 (1997)

    Article  Google Scholar 

  16. Baert, Q., Caron, A.C., Morge, M., Routier, J.C.: Stratégie de découpe de tâche pour le traitement de données massives. In: Garbay, C., Bonnet, G., (eds.) Journées Francophones sur les Systèmes Multi-Agents. Cohésion: fondement ou propriété émergente, Caen, France, Cépaudès édition, pp. 65–75, July 2017

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Univ. Lille, CNRS, Centrale Lille, UMR 9189 - CRIStAL - Centre de Recherche en Informatique Signal et Automatique de Lille, 59000, Lille, France

    Quentin Baert, Anne-Cécile Caron, Maxime Morge & Jean-Christophe Routier

Authors
  1. Quentin Baert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anne-Cécile Caron
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maxime Morge
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jean-Christophe Routier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quentin Baert .

Editor information

Editors and Affiliations

  1. University of Évry Val d'Essonne, Paris Saclay, France

    Francesco Belardinelli

  2. Polytechnic University of Valencia, Valencia, Spain

    Estefanía Argente

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Baert, Q., Caron, AC., Morge, M., Routier, JC. (2018). Negotiation Strategy of Divisible Tasks for Large Dataset Processing. In: Belardinelli, F., Argente, E. (eds) Multi-Agent Systems and Agreement Technologies. EUMAS AT 2017 2017. Lecture Notes in Computer Science(), vol 10767. Springer, Cham. https://doi.org/10.1007/978-3-030-01713-2_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01713-2_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01712-5

  • Online ISBN: 978-3-030-01713-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics