Online MapReduce scheduling problem of minimizing the makespan

Abstract

MapReduce system is a popular big data processing framework, and the performance of it is closely related to the efficiency of the centralized scheduler. In practice, the centralized scheduler often has little information in advance, which means each job may be known only after being released. In this paper, hence, we consider the online MapReduce scheduling problem of minimizing the makespan, where jobs are released over time. Both preemptive and non-preemptive version of the problem are considered. In addition, we assume that reduce tasks cannot be parallelized because they are often complex and hard to be decomposed. For the non-preemptive version, we prove the lower bound is $\frac{m+m(\Psi (m)-\Psi (k))}{k+m(\Psi (m)-\Psi (k))}$, higher than the basic online machine scheduling problem, where k is the root of the equation $k=\big \lfloor {\frac{m-k}{1+\Psi (m)-\Psi (k)}+1 }\big \rfloor$ and m is the quantity of machines. Then we devise an $(2-\frac{1}{m})$-competitive online algorithm called MF-LPT (Map First-Longest Processing Time) based on the LPT. For the preemptive version, we present a 1-competitive algorithm for two machines.

Appendices

Appendix 1: The solution of $\rho ^*$

To make (3.5) easier to solve, we transform it into:

$$\begin{aligned} \min \quad \rho =\max \{b_1,b_2,\ldots ,b_m\} \end{aligned}$$

Substituting (7.1b) and (7.1c) into (7.1a), we have

$$\begin{aligned} \sum _{i=1}^m x_i = \frac{b_1}{m}+\sum _{i=1}^{m-1}\frac{(b_{m-i+1}-1)(1-i\cdot s_1)}{m-i} +(m-1)s_1 = \sum _{i=1}^m s_i+1 \end{aligned}$$

(7.2)

Noticing that, in (7.2), $\frac{\partial b_1}{\partial s_i}=m>0 ~ (\forall i\in \{2,3,\ldots ,m\})$ and $\frac{\partial b_j}{\partial s_i}=\frac{j-1}{1-(m-j+1)s_1}>0 ~ (\forall i,j\in \{2,3,\ldots ,m\})$, we have that $\forall j\in \{1,2,\ldots ,m\}$, $b_j$ is monotonically increasing in $s_i$ for any $i\in \{2,3,\ldots ,m\}$. Thus, to minimize the objective function $\rho =\max \{b_1,b_2,\ldots ,b_m\}$, the $s_i$ must be as small as possible. As well as (7.1d), we conclude that $s_1=s_2=\cdots =s_m$. So we can simplify (7.2) into the following equation

$$\begin{aligned} \frac{b_1}{m}+\sum _{i=1}^{m-1}\frac{(b_{m-i+1}-1)(1-i\cdot s_1)}{m-i}-s_1 = 1 \end{aligned}$$

(7.3)

For (7.3), we have that

$$\begin{aligned} \left\{ \! \begin{aligned} \frac{{\partial {b_1}}}{{\partial {s_1}}}&= m\left( {1 + \sum \limits _{i = 1}^{m - 1} {\frac{{\left( {{b_{m + 1 - i}} - 1} \right) \cdot i}}{{\left( {m - i} \right) }}} } \right) > 0\\ \frac{{\partial {b_j}}}{{\partial {s_1}}}&= \frac{{\left( {j - 1} \right) \left( {1 + \sum \nolimits _{i = 1}^{m - 1} {\frac{{\left( {{b_{m + 1 - i}} - 1} \right) \cdot i}}{{\left( {m - i} \right) }}} } \right) }}{{1 - \left( {m - j + 1} \right) {s_1}}} > 0,\quad j = 2,3,\ldots ,m \end{aligned} \right. \end{aligned}$$

Since $\forall j\in \{1,2,\ldots ,m\}$, $b_j$ is monotonically increasing in $s_1$, to minimize the objective function $\rho =\max \{b_1,b_2,\ldots ,b_m\}$, the $s_1$ must be as small as possible, i.e. $s_1=0$. As well as (7.1e), we can simplify (7.3) into the following

$$\begin{aligned}&\frac{{{b_1}}}{m} + \frac{{{b_2} - 1}}{1} + \frac{{{b_3} - 1}}{2} + \dots + \frac{{{b_m} - 1}}{{m - 1}} = 1 \end{aligned}$$

(7.4)

$$\begin{aligned}&\frac{{{b_1}}}{m} \ge \frac{{{b_2} - 1}}{1} \ge \frac{{{b_3} - 1}}{2} \ge \dots \ge \frac{{{b_m} - 1}}{{m - 1}} \end{aligned}$$

(7.5)

Suppose the number $p\in \{1,2,\ldots ,m\}$ such that $b_p=\max \{b_1,b_2,\ldots ,b_m\}$. For (7.4), $\forall j \in \{1,2,\ldots ,p-1,p+1,\ldots ,m\}$, $\frac{{\partial {b_p}}}{{\partial {b_j}}} < 0$ obviously. Hence, $\forall j\in \{1,2,\ldots ,m\}$, $b_p$ is monotonically increasing in $b_j$. To minimizie $b_p$, $b_j$ must be as large as possible with the constraints $b_j\le b_p$ and (7.5).

Firstly, For $b_1$, it should be equal to $b_p$. Then, if we temporarily ignore the first inequation in (7.5), $\forall j\in \{2,3,\ldots ,m\}$, $b_j$ can freely increase to $b_p$. But taking the first inequation into account may lead to some $b_j$ can not reach $b_p$. We denote the last $b_j$ who can not reach $b_p$ by $b_k$, i.e. $\{b_2,b_3,\ldots ,b_k\}$ cannot increase to $b_p$. To minimizie $b_p$, $\forall i\in \{2,3,\ldots ,k\}$, $b_i$ can increase to $\frac{i-1}{m}b_p+1$ so that $\frac{b_p}{m}=\frac{b_i-1}{i-1}$. Lastly, $\{b_{k+1},b_{k+2},\ldots ,b_m\}$ can increase to $b_p$. Eventually, after each $b_j$ become as large as possible, (7.4) turn into

$$\begin{aligned} \begin{array}{l} \underbrace{\frac{{{b_p}}}{m} + \frac{{{b_p}}}{m} + \cdots +\frac{{{b_p}}}{m}}_k + \underbrace{\frac{{{b_p} - 1}}{{{k}}} + \frac{{{b_p} - 1}}{{{k} + 1}} + \cdots + \frac{{{b_p} - 1}}{{m - 1}}}_{m - k} \\ = \displaystyle {k} \cdot \frac{{{b_p}}}{m} + \left( {{b_p} - 1} \right) {\sum _{{k}}^{m - 1} {\frac{1}{i}} } = 1 \end{array} \end{aligned}$$

(7.6)

where k satisfy

$$\begin{aligned} \left\{ \! \begin{aligned} \frac{{{b_p} - 1}}{{k - 1}}&\ge \textstyle \frac{{{b_p}}}{m}\\ \frac{{{b_p} - 1}}{k}&< \textstyle \frac{{{b_p}}}{m}\\ \end{aligned} \right. \end{aligned}$$

(7.7)

From (7.6) and (7.7), we have:

$$\begin{aligned} \rho ^* =b_p = \frac{m+m(\Psi (m)-\Psi (k))}{k+m(\Psi (m)-\Psi (k))} \end{aligned}$$

where $\Psi (n+1)=\sum _{i=1}^n \frac{1}{i}-\gamma$, and k is the root of $k =\Big \lfloor {\frac{m-k}{1+\Psi (m)-\Psi (k)}+1 }\Big \rfloor$.

Appendix 2: The proof of $\rho ^*<\frac{3}{2}$

To prove $\rho ^*=\frac{m+m(\Psi (m)-\Psi (k))}{k+m(\Psi (m)-\Psi (k))}<\frac{3}{2}$, we just need to prove

$$\begin{aligned} \frac{{3k}}{m} + \Psi (m)-\Psi (k) > 2 \end{aligned}$$

(8.1)

Since we can hardly know the exact value of $\frac{{3k}}{m} + \Psi (m)-\Psi (k)$ in (8.1), we have to use inequation to bound it. DeTemple (1993) propose that

$$\begin{aligned} \frac{1}{24n^2} < \Psi (n)-\ln \left( n-\frac{1}{2}\right) < \frac{1}{24(n-1)^2} \end{aligned}$$

(8.2)

Substituting (8.2) into (8.1), we have

$$\begin{aligned} \frac{3k}{m} + \Psi (m)-\Psi (k) > \frac{{3k}}{m} + \frac{1}{{24{m^2}}} - \frac{1}{{24{{\left( {k - 1} \right) }^2}}} + \ln \left( \frac{m - \frac{1}{2}}{k - \frac{1}{2}} \right) \end{aligned}$$

After calculation, for $k \ge 2$, we have

$$\begin{aligned} \frac{{3k}}{m} + \frac{1}{{24{m^2}}} - \frac{1}{{24{{\left( {k - 1} \right) }^2}}} + \ln \left( \frac{m - \frac{1}{2}}{k - \frac{1}{2}} \right) > 2 \end{aligned}$$

For $k=1$, $\frac{{3k}}{m} + \Psi (m)-\Psi (k)=\frac{{3}}{m} + \Psi (m) + \gamma$, which is increasing in m when $m\ge 2$. Hence, $\frac{{3}}{m} + \Psi (m) + \gamma \ge \frac{3}{2}+\Psi (2)+ \gamma =\frac{5}{2}>2$. From the above we can draw a conclusion that $\rho ^*<\frac{3}{2}$.