Boosting adaptivity of fault-tolerant scheduling for real-time tasks with service requirements on clusters

https://doi.org/10.1016/j.jss.2011.04.067Get rights and content

Abstract

Thank to the excellent extensibility and usability, computer clusters have become the dominating platform for parallel computing. Fault-tolerance is mandatory for safety-critical applications running on clusters. In this paper we propose a service-aware and adaptive fault-tolerant scheduling algorithm using overlapping technologies (SAO in short) that can tolerate a node’s permanent failure at any time instant for real-time tasks with service requirements in heterogeneous clusters. SAO adopts the primary/backup model and considers the timing constraints, service requirements, and system resource utilization. To improve system resource utilization, we employ backup-backup (BB in short) and primary-backup (PB in short) overlapping technologies and analyze the overlapping constraints. In addition, SAO has high system adaptivity by dynamically adjusting the service levels of tasks based on system load. Furthermore, to improve resource utilization and schedulability, SAO makes backup copies adopt passive execution scheme or decrease the overlapping execution time of the primary copy and backup copy of a task as much as possible. Compared with a baseline algorithm SAWO (a service-aware and adaptive fault-tolerant scheduling algorithm without using overlapping technologies) and an existing algorithm DYFARS with simulation experiments, SAO achieves an average of 51.25% improvement in performability.

Highlights

► A service-aware and adaptive fault-tolerant scheduling algorithm SAO was proposed. ► SAO adopts the primary/backup model and overlapping technologies. ► SAO can tolerate a node’s permanent failure at any time instant. ► SAO improves the adaptivity of real-time fault-tolerant systems.

Introduction

Clusters are widely adopted platforms that provide high performance computing solutions for large-scale scientific and engineering applications. According to the TOP 500 report in November 2010, cluster computers account for 82.8% of the fastest 500 computers (Anon., 2010), which indicates traditional high-performance computers have given way to clusters consisting of a great deal of low-cost and general-purpose computers interconnected by high-speed networks. Meantime, clusters are also employed to run real-time applications. As far as real-time applications are concerned, the correctness of real-time tasks in these applications depends not only on the computation results, but also on the time instances at which these results are produced (Krishna and Shin, 2001). For some security-critical real-time applications such as flight control and signal processing in a battle field, each task must be executed within its deadline even in the presence of hardware and software faults. Consequently, providing a fault-tolerant mechanism for such real-time applications is mandatory due to the critical nature of tasks in the applications.

Fault-tolerance technologies can generally be classified into two categories: software fault-tolerance and hardware fault-tolerance. The advantage of adopting software fault-tolerance lies in it can efficiently improve the system reliability without requiring extra hardware overhead. In practice, fault-tolerant scheduling algorithms that belong to the software fault-tolerance technology draw a good deal of attention (Stavrinides and Karatza, 2009, Stavrinides and Karatza, 2010, Han et al., 2003). Since recent decades, some research has been conducted to develop fault-tolerance scheduling algorithms for clusters to support real-time applications (Qin and Jiang, 2006).

In fault-tolerant scheduling, several copies of a task are allocated to different nodes to realize fault-tolerance. The primary/backup mode is the most popular strategy. In primary/backup model, two copies of one task are scheduled on two different nodes and an acceptance test is used to check the correctness of schedules (Tsuchiya et al., 1995). If the node with the primary copy of a real-time task is faulty, the backup copy of the task can still be executed on a different node to guarantee the successful completion of the task within timing constraint.

Ghosh et al. proposed two techniques: deallocation and overloading, to improve schedulability while providing fault-tolerance with low overhead. The overloading means multiple backup copies may overlap in the same time slot on the same processor, and the deallocation is the reclamation of resources reserved for backup copies when the corresponding primary copies complete successfully (Ghosh et al., 1997). Manimaran et al. extended the strategy in Ghosh et al. (1997) by taking the resource constraints among tasks into consideration, and partitioned processors into multiple groups to tolerate more than one fault at a time (Manimaran and Murthy, 1998). Al-Omari et al. studied a PB overloading technique that allows the primary copy of one task to overlap with the backup copy of another task for the purpose of high schedulability (Al-Omari et al., 2004). One of the common features in the above algorithms is passive backup copy scheme, i.e., the backup copy of a task is permitted to execute only if a fault is detected in the primary copy. Hence, tasks must have enough laxity (refers to the time interval between deadline and finish time of the primary copy of a task) to restart the backup copies. That is, the laxity must be larger than the computation time. All the above papers assumed that the laxity was at least twice as large as the computation time. However, this assumption is unrealistic in practice, particularly when systems are under heavy load. The drawback of passive backup copy scheme is that the laxity must be larger than the execution time of backup copies. Otherwise, the backup copies cannot be successfully finished if the node on which their corresponding primary copies are allocated is faulty.

Another scheme named active backup copy scheme refers that the primary copies and their corresponding backup copies can execute simultaneously under some constraints, i.e., the backup copy of a task can begin to execute even though no fault is detected in the primary copy. This scheme results in waste of system resource when a task has enough laxity. As a result, active backup copy scheme is usually used if a task has tight laxity. Tsuchiya et al. proposed a technique in which two copies of each task are executed concurrently, and these copies are allowed to start running at different times (Tsuchiya et al., 1995). Yang et al. studied a fault-tolerant scheduling algorithm that also allows the two copies of a task executed simultaneously to improve schedulability (Yang et al., 2004). Al-Omari et al. investigated an adaptive scheme that controls the overlap interval between the primary copy and backup copy of one task based on the primary fault probability and a task’s laxity (Al-Omari et al., 2005). Although these methods overcome the drawbacks of timing constraints on backup copies to some extent, they only support homogeneous computing environment and are unable to support heterogeneous systems.

Recently, Qin and Jiang developed a scheduling algorithm in real-time heterogeneous systems providing fault-tolerance for tasks with precedence constraints and taking reliability as its main scheduling objective (Qin and Jiang, 2006). However, the algorithm is static in nature and not suitable for dynamic scheduling. Later, Luo et al. proposed a dynamic and reliability-driven real-time fault-tolerant scheduling algorithm called DYFARS for heterogeneous systems considering the backup copies in both active and passive schemes, which improves the flexibility of systems (Luo et al., 2007).

Motivation. In addition to the timeliness and fault-tolerance requirements, many real-time applications have service requirements such as security service and data accuracy. For example, an automated flight control system described in Atdelzater et al. (2000) is utilized to fly a F-16 flight aircraft. Since the system can be used in the military battle field, it must ensure that each task is executed within its deadline even in the presence of hardware and software faults. In this system, all the fight control tasks including Guidance, Controller, Slow Navigation, Fast Navigation and Missile Control need to be completed within timing constraints. In order to improve the stability of the system, each task can select different service levels by varying its period or execution time. Different service levels provide different flight qualities. Another example from Zhu and Lu (2008) is a real-time signal processing application. Although the miss of deadlines by some tasks will not result in a disaster, the outdated or half-baked processed data may be useless for users, especially in the field of modern information battle. It should be noted that the signal data can be processed using different algorithms. As for decoding block turbo codes, several algorithms can be employed (Pyndiah et al., 1994, Chi et al., 2001). High-complexity algorithms can guarantee that signal processing has higher service level (higher data accuracy) at the expense of processing time, while low-complexity algorithms are the opposite.

Some scheduling algorithms have been proposed to consider service requirements for real-time applications in distributed systems (Harada et al., 2007, He et al., 2004, Doğan and Özgüner, 2006). Unfortunately, to the best of our knowledge, little work has been done on fault-tolerant scheduling for real-time tasks with service requirements, which presents the challenge of devising novel fault-tolerant scheduling algorithms for real-time tasks with service requirements to fill the gap.

In our prior work (Zhu and Lu, 2008, Zhu and Lu, 2009a, Zhu and Lu, 2009b, Zhu et al., 2010, Ge et al., 2010), we investigated some scheduling strategies for real-time tasks on heterogeneous clusters. However, these strategies do not consider the fault-tolerance problem. In this work, we concentrate on the non-preemptive fault-tolerant scheduling algorithm of real-time aperiodic tasks with service needs on heterogeneous clusters. The algorithm belongs to dynamic sort and tolerates one node’s failure at a time instant.

Contributions. The major contributions of this paper are:

  • (1)

    We develop a novel service-aware fault-tolerant scheduling model on heterogeneous clusters.

  • (2)

    We investigate a service-aware and adaptive fault-tolerant scheduling algorithm using overlapping technology (SAO in short). SAO sufficiently considers the system resource utilization, timing constraints of real-time tasks, and service requirements. Moreover, by adapting service level and overlapping time according to the system load, SAO provides high service quality for real-time tasks within timing constraints.

  • (3)

    We analyze the BB and PB overlapping constraints and combine them into our scheduling model to decrease the resource redundancy derived from primary/backup fault-tolerance model.

The rest of this paper is organized as follows: The system model with fault-tolerance and service requirements is given in Section 2. In Section 3, the BB and PB overlapping technologies are described and the constraints are analyzed. Section 4 describes the SAO algorithm and the main principles behind it. Simulation experiments and performance analysis are provided in Section 5. Section 6 concludes the paper with a summary and future work.

Section snippets

System models

In this section, we describe models that are built to represent a task allocation framework, real-time applications with service requirements, and fault model. For future reference, we summarize notation used in this study in Table 1.

BB overlapping and PB overlapping

Now we study the backup-backup (BB in short) and primary-backup (PB in short) overlapping policies that could efficiently boost the the system resource utilization. Provided that a majority of backup copies only occupy resource but not execute as the probability of a node’s fault is small, considering the BB and PB overlapping policies are reasonable (Al-Omari et al., 2004).

Fault-tolerant scheduling algorithm SAO

In this section, we present an efficient s ervice-aware and a daptive fault-tolerant scheduling algorithm using o verlapping technologies (SAO in short) for real-time aperiodic tasks with service requirements on heterogeneous clusters. We first introduce some algorithm properties to facilitate the algorithm presentation.

Property 1

The primary copy tiP and backup copy tiB are mutually exclusive in space, i.e., tiP and tiB cannot be allocated to the same node, n(tiP)n(tiB).

Property 2

The start time of primary copy tiP

Performance evaluation

In this section, we present several groups of experimental results obtained from extensive simulations to evaluate the performance of SAO. A competitive advantage of simulation is that it does not incur additional hardware cost to evaluate performance on a large-scale distributed system. To demonstrate the performance improvements gained by SAO, we compare it with DYFARS (a dynamic and reliability-driven real-time fault-tolerant scheduling algorithm) lately proposed in the literature Luo et al.

Conclusions and future work

In this paper, we presented a service-aware fault-tolerant scheduling algorithm SAO, in which independent real-time tasks with service requirements can tolerate one node’s failures in a heterogeneous cluster. The fault-tolerant capability is incorporated in the algorithm by using a primary/backup model. In order to sufficiently utilize system resource, SAO employs the BB and PB overlapping technologies, as well as strives to advance the start time of primary copies and to delay the start time

Acknowledgements

This research was supported by the National Basic Research Program of China under Grant No. 6136101, and the National High-Tech Research and Development Plan of China under Grant No. 2008AA7070412.

Xiaomin Zhu received the B.S. and M.S. degrees in computer science from Liaoning Technical University, Liaoning, China, in 2001 and 2004, respectively, and Ph.D. degree in computer science from Fudan University, Shanghai, China, in 2009. He is currently an assistant professor in the College of Information System and Management at National University of Defense Technology, Changsha, China. His research interests are cluster computing, fault-tolerant computing, green computing, and performance

References (28)

  • R. Ge et al.

    PowerPack: energy profiling and analysis of high-performance systems and applications

    IEEE Transactions on Parallel and Distributed Systems

    (2010)
  • S. Ghosh et al.

    Fault-tolerance through scheduling of aperiodic tasks in hard real-time multiprocessor systems

    IEEE Transactions on Parallel and Distributed Systems

    (1997)
  • C.C. Han et al.

    A fault-tolerant scheduling algorithm for real-time periodic tasks with possible software faults

    IEEE Transactions on Computers

    (2003)
  • F. Harada et al.

    Adaptive resource allocation control for fair QoS management

    IEEE Transactions on Computers

    (2007)
  • Cited by (15)

    • A novel fault-tolerant scheduling approach for collaborative workflows in an edge-IoT environment

      2022, Digital Communications and Networks
      Citation Excerpt :

      If the primary copy of the edge server fails, the replica copy is executed for compensation. Recently, extensive research efforts were paid to developing fault-tolerant scheduling strategies by using basic or extended PB for workflows executed over distributed computing platforms, e.g., clouds, grids, and edge clouds [12–15]. However, how to enforce fault tolerance for workflows executed upon edge computing platforms remains a challenge and related contributions are rare.

    • DEFT: Dynamic Fault-Tolerant Elastic scheduling for tasks with uncertain runtime in cloud

      2019, Information Sciences
      Citation Excerpt :

      Zheng et al. developed two dynamic algorithms, MRC-ECT and MCT-LRC, which analyzed the constraints of overlapping for independent and dependent tasks’ backups, respectively [32]. Zhu et al. proposed a hybrid overlapping in task scheduling for QoS and reliability [33,34]. Al Omari et al. studied the PB overlapping strategy for real-time tasks based on the tasks’ time laxity [1].

    • Fault-tolerant elastic scheduling algorithm for workflow in Cloud systems

      2017, Information Sciences
      Citation Excerpt :

      Al-Omari et al. designed a PB overloading technique that allowed the primary copy of one task to overlap with the backup copy of another task for the purpose of high schedulability [3]. Zhu et al. proposed a QoS-aware fault-tolerant scheduling algorithm and an adaptive service-aware fault-tolerant scheduling algorithm based on overlapping technology for independent tasks within heterogeneous clusters [37,38]. By juxtaposing the strong points of neighbor-based and cluster-based load-balancing methods in Grid systems, Balasangameshwara and Raju developed a load-balancing approach and integrated it with the fault-tolerant scheduling namely MinRC, as such proposed a performance-driven fault-tolerant load-balancing algorithm for independent tasks [6].

    • An immune system-inspired rescheduling algorithm for workflow in Cloud systems

      2016, Knowledge-Based Systems
      Citation Excerpt :

      The two fundamental and widely recognized techniques to support the fault in distributed environments are replication and rescheduling. And they are complementary and can be used in cooperation: the replication is a method suited to the tasks scheduling phase, while the rescheduling is the one mostly applicable during execution [6]. In this paper, we focus on the rescheduling method.

    • Failure-Aware Elastic Cloud Workflow Scheduling

      2023, IEEE Transactions on Services Computing
    • Fault tolerant dynamic task offloading with cloudlets in mobile cloud computing

      2019, International Journal of Innovative Technology and Exploring Engineering
    View all citing articles on Scopus

    Xiaomin Zhu received the B.S. and M.S. degrees in computer science from Liaoning Technical University, Liaoning, China, in 2001 and 2004, respectively, and Ph.D. degree in computer science from Fudan University, Shanghai, China, in 2009. He is currently an assistant professor in the College of Information System and Management at National University of Defense Technology, Changsha, China. His research interests are cluster computing, fault-tolerant computing, green computing, and performance evaluation. He is a member of the IEEE, the IEEE Communication Society, and the ACM.

    Chuan He received the B. S. and M. S. degrees in military operations research from Air Defense Force Command Academic, Zhengzhou, China, in 2007 and 2010, respectively. He is currently a Ph. D. candidate in the College of Information System and Management at National Universtiy of Defense Technology, Changsha, China. His research interests include parallel and distributed computing, real-time systems and scheduling design.

    Rong Ge received the PhD degree in computer science from Virginia Tech. She is currently an assistant professor in the Department of Mathematics, Statistics and Computer Science at Marquette University. Her research interests include performance modeling and analysis, energy-efficient computing, parallel and distributed systems, and computational science. She is a member of the IEEE and the IEEE Computer Society, and a member of the ACM and Upsilon Pi Epsilon.More details about her research and background can be found at http://www.mscs.mu.edu/~rge.

    Peizhong Lu received the B.S. and M.S. degrees in applied mathematics from Institute of Information Engineering, Zhengzhou, China, in 1982 and 1987, respectively. He received the Ph.D. degree from Institute of Systems Science, AMSS, China, in 1998. He is currently a professor in the School of Computer Science at Fudan University. His research interests are information security, algebra coding, parallel and distributed systems, real-time computing and performance evaluation. He has published more than 50 technical papers in reputable journals and conference proceedings. He received the Excellent Doctoral Dissertation Award of China in 2000. He is a member of IEEE.

    View full text