Kristi Morton
ABSTRACT
Time-oriented progress estimation for parallel queries is a challenging problem that has received only limited attention. In this paper, we present ParaTimer, a new type of time-remaining indicator for parallel queries. Several parallel data processing systems exist. ParaTimer targets environments where declarative queries are translated into ensembles of MapReduce jobs. ParaTimer builds on previous techniques and makes two key contributions. First, it estimates the progress of queries that translate into directed acyclic graphs of MapReduce jobs, where jobs on different paths can execute concurrently (unlike prior work that looked at sequences only). For such queries, we use a new type of critical-path-based progress-estimation approach. Second, ParaTimer handles a variety of real systems challenges such as failures and data skew. To handle unexpected changes in query execution times due to runtime condition changes, ParaTimer provides users with not only one but with a set of time-remaining estimates, each one corresponding to a different carefully selected scenario. We implement our estimator in the Pig system and demonstrate its performance on experiments running on a real, small-scale cluster.
- C. Ballinger. Born to be parallel: Why parallel origins give Teradata database an enduring performance edge. http://www.teradata.com/t/page/87083/index.html.Google Scholar
- S. Chaudhuri, R. Kaushik, and R. Ramamurthy. When can we trust progress estimators for SQL queries. In Proc. of the SIGMOD Conf., Jun 2005. Google ScholarDigital Library
- S. Chaudhuri, V. Narassaya, and R. Ramamurthy. Estimating progress of execution for SQL queries. In Proc. of the SIGMOD Conf., Jun 2004. Google ScholarDigital Library
- DB2. SQL/monitoring facility. http://www.sprdb2.com/SQLMFVSE.PDF, 2000.Google Scholar
- DB2. DB2 Basics: The whys and how-tos of DB2 UDB monitoring. http://www.ibm.com/developerworks/db2/library/techarticle/dm-0408hubel/index.html, 2004.Google Scholar
- J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. In Proc. of the 6th OSDI Symp., 2004. Google ScholarDigital Library
- M. Dempsey. Monitoring active queries with Teradata Manager 5.0. http://www.teradataforum.com/attachments/a030318c.doc, 2001.Google Scholar
- D. J. DeWitt, E. Paulson, E. Robinson, J. Naughton, J. Royalty, S. Shankar, and A. Krioukov. Clustera: an integrated computation and data management system. In Proc. of the 34th VLDB Conf., pages 28--41, 2008.Google ScholarDigital Library
- A. Ganapathi, H. Kuno, U. Dayal, J. L. Wiener, A. Fox, M. Jordan, and D. Patterson. Predicting multiple metrics for queries: Better decisions enabled by machine learning. In Proc. of the 25th ICDE Conf., pages 592--603, 2009. Google ScholarDigital Library
- Greenplum. Database performance monitor datasheet (Greenplum Database 3.2.1). http://www.greenplum.com/pdf/Greenplum-Performance-Monitor.pdf.Google Scholar
- Greenplum database. http://www.greenplum.com/.Google Scholar
- Hadoop. http://hadoop.apache.org/.Google Scholar
- J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In Proc. of the SIGMOD Conf., 1997. Google ScholarDigital Library
- IBM zSeries SYSPLEX. http://publib.boulder.ibm.com/infocenter/\\dzichelp/v2r2/index.jsp?topic=/com.ibm.db2.doc.admin/xf6495.htm.Google Scholar
- M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In Proc. of the European Conference on Computer Systems (EuroSys), pages 59--72, 2007. Google ScholarDigital Library
- C. Jermaine, A. Dobra, S. Arumugam, S. Joshi, and A. Pol. A disk-based join with probabilistic guarantees. In Proc. of the SIGMOD Conf., pages 563--574, 2005. Google ScholarDigital Library
- Large Synoptic Survey Telescope. http://www.lsst.org/.Google Scholar
- G. Luo, J. F. Naughton, C. J. Ellman, and M. Watzke, Increasing the accuracy and coverage of SQL progress indicators. In Proc. of the 20th ICDE Conf., 2004. Google ScholarDigital Library
- G. Luo, J. F. Naughton, C. J. Ellman, and M. Watzke. Toward a progress indicator for database queries. In Proc. of the SIGMOD Conf., Jun 2004. Google ScholarDigital Library
- G. Luo, J. F. Naughton, and P. S. Yu. Multi-query SQL progress indicators. In Proc. of the 10th EDBT Conf., 2006. Google ScholarDigital Library
- C. Mishra and N. Koudas. A lightweight online framework for query progress indicators. In Proc. of the 23rd ICDE Conf., 2007.Google ScholarCross Ref
- C. Mishra and M. Volkovs. ConEx: A system for monitoring queries (demonstration). In Proc. of the SIGMOD Conf., Jun 2007. Google ScholarDigital Library
- K. Morton, A. Friesen, M. Balazinska, and D. Grossman. Estimating the progress of MapReduce pipelines. In Proc. of the 26th ICDE Conf., 2010.Google ScholarCross Ref
- C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In Proc. of the SIGMOD Conf., pages 1099--1110, 2008. Google ScholarDigital Library
- Pig Progress Indicator. http://hadoop.apache.org/pig/.Google Scholar
- G. Plivna. Long running operations in Oracle. http://www.gplivna.eu/papers/v\$session_longops.htm, 2007.Google Scholar
- A. Pruscino. Oracle RAC: Architecture and performance. In Proc. of the SIGMOD Conf., page 635, 2003.Google Scholar
- R. Ramakrishnan and J. Gehrke. Database Management Systems. McGraw-Hill Science Engineering, third edition, 2002. Google ScholarDigital Library
- Vertica, inc. http://www.vertica.com/.Google Scholar
- M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. Proc. of the 8th OSDI Symp., 2008. Google ScholarDigital Library
Index Terms
- ParaTimer: a progress indicator for MapReduce DAGs
Recommendations
HadoopDB in action: building real world applications
SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of dataHadoopDB is a hybrid of MapReduce and DBMS technologies, designed to meet the growing demand of analyzing massive datasets on very large clusters of machines. Our previous work has shown that HadoopDB approaches parallel databases in performance and ...
Towards interactive analytics and visualization on one billion tweets
SIGSPACIAL '16: Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information SystemsWe present a system called "Cloudberry" that allows users to interactively query, analyze, and visualize large amounts of data with temporal, spatial, and textual dimensions. As a general-purpose full-stack solution, it has a friendly UI, intelligent ...
GISQAF: MapReduce guided spatial query processing and analytics system
The Global Database of Event, Language, and Tone GDELT is the only global political georeferenced event dataset with more than 250 million observations covering all countries in the world since January 1, 1979. TABARI and CAMEO are the tools that are ...
Comments