Abstract
The map-reduce paradigm has shown to be a simple and feasible way of filtering and analyzing large data sets in cloud and cluster systems. Algorithms designed for the paradigm must implement regular data distribution patterns so that appropriate use of resources is ensured. Good scalability and performance on Map-Reduce applications greatly depend on the design of regular intermediate data generation-consumption patterns at the map and reduce phases. We describe the data distribution patterns found in current Map-Reduce read mapping bioinformatics applications and show some data decomposition principles to greatly improve their scalability and performance
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Dean J et al (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51:107–113
Bialecki A, Cafarella M, Cutting D, O’Malley O (2005) Hadoop: a framework for running applications on large clusters built of commodity hardware. Wiki at http://hadoop.apache.org/
Shi X (2009) Evaluating MapReduce on virtual machines: the Hadoop case. In: CloudCom 2009. LNCS, vol 5931. Springer, Berlin, pp 519–528
Schatz M (2009) CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics 25(11):1363–1369
Langmead B, Schatz MC, Lin J, Pop M, Salzberg SL (2009) Searching for SNPs with cloud computing. Genome Biol 10:R134
Matthews SJ, Williams TL (2010) MrsRF: an efficient MapReduce algorithm for analyzing large collections of evolutionary trees. BMC Bioinform 11:S15
Ranger C, Raghurama R, Penmetsa A, Bradski G, Kozykaris C (2007) Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of the 13th international symposium on high-performance computer architecture (HPCA), Phoenix, AZ
Mao Y, Morris R, Kaashoek MF (2010) Optimizing MapReduce for multicore architectures. Tech Rep, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
Li H, Ruan J, Durbin R (2008) Mapping short DNA sequencing reads and calling variants using mapping quality scores. Genome Res 18(11):1851–1858
Baeza-Yates RA et al (1992) Fast and practical approximate string matching. In: Proceedings of the combinatorial pattern matching. Third annual symposium, Tucson, pp 185–192
Li R, Li Y, Kristiansen K, Wang J (2008) SOAP: short oligonucleotide alignment program. Bioinformatics 24(5):713–714
Smith AD et al (2008) Using quality scores and longer reads improves accuracy of Solexa read mapping. BMC Bioinform 9:128
Babu S (2010) Towards automatic optimization of MapReduce programs. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, New York
Palla K (2009) A comparative analysis of join algorithms using the Hadoop Map/Reduce framework. Master of science thesis. School of informatics, University of Edinburgh
Acknowledgements
We want to thank Eduard Ayguade, David Carrera and the staff at Barcelona Supercomputing Center (BSC) for their help and support to the usage of the IBM Blade computer cluster.
This paper was supported by Consolider Project CSD2007-00050 of the Spanish Ministerio de Ciencia y Tecnologia.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Espinosa, A., Hernandez, P., Moure, J.C. et al. Analysis and improvement of map-reduce data distribution in read mapping applications. J Supercomput 62, 1305–1317 (2012). https://doi.org/10.1007/s11227-012-0792-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-012-0792-8