VSA: An offline scheduling analyzer for Xen virtual machine monitor

https://doi.org/10.1016/j.future.2012.12.004Get rights and content

Abstract

Nowadays, it is an important trend in the system domain to use the software-based virtualization technology to build the execution environments (e.g., the Clouds). After introducing the virtualization layer, there exist two schedulers: One in the hypervisor and the other inside the Guest Operating System (GOS). To fully understand the virtualized system and identify the possible reasons for performance problems incurred by the virtualization technology, it is very important for the system administrators and engineers to know the scheduling behavior of the hypervisor, in addition to understanding the scheduler inside the GOS. In this paper, we develop a virtualization scheduling analyzer, called VSA, to analyze the trace data of the Xen virtual machine monitor. With VSA, one can easily obtain the scheduling data associated with virtual processors (i.e., VCPUs) and physical processors (i.e., PCPUs), and further conduct the scheduling analysis for a group of interacting VCPUs running in the same domain.

Highlights

► Demonstrate the lack of support for advanced scheduling analyses in the existing tools. ► Develop a tool to conduct advance scheduling analyses in virtualized systems. ► Conduct the case studies to demonstrate the effectiveness of the developed tool.

Introduction

It is an important trend nowadays to use software-based virtualization technologies to build the private data centers and public cloud systems, such as Amazon EC2  [1] and GoGrid  [2]. Among these virtualization technologies, the Xen Virtual Machine Monitor (VMM)  [3], which allow users to execute up to hundreds of virtual machines on a single physical machine with low extra overhead, is widely adopted for such purposes  [4].

With the introduction of the Xen virtualization layer, traditional operating systems (e.g., Linux) are para-virtualized (i.e., the machine sensitive and privileged operations are replaced with hypercalls) to run on top of the Xen hypervisor. The operating systems in the virtualized environments are called Guest Operating Systems (GOS). In this new architecture, there exist two schedulers working together: one works inside the Xen hypervisor and the other in the GOS. The examples of the former include the Simple Earliest Deadline First Scheduler (SEDF)  [5] and the Credit Scheduler  [6], which schedule the VCPUs to run on top of physical processors, while the examples of the latter include Completely Fair Scheduler (i.e., CFS)  [7], [8] in Linux, which schedules the application processes to run in the VCPUs. Although there is long history of research in GOS schedulers and the GOS schedulers have been relatively mature, the schedulers inside the hypervisor are still under active research and development. To help understand the scheduling behaviors inside the hypervisor, Xen provides a powerful tool, named xentrace, to collect system trace data (including scheduling-related data), and a related format tool, xentrace_format, to convert the trace data into a human readable form. However, it is still very difficult to analyze the scheduling behavior of each individual VCPU by using the trace data alone.2

In this work, we design and implement VSA,3 an offline scheduling analyzer for the Xen VMM. The system can reconstruct the scheduling history of each VCPU based on the offline trace data generated by xentrace and xentrace_format. By using the scheduling history data of each VCPU, VSA can render detailed analysis data on VCPUs, including:

  • block-to-wake time: the duration between the time when a VCPU is blocked and the time it is woken up from the blocking state

  • wake-to-schedule-in time: the duration between the time when a VCPU is woken up and the time when it is scheduled to run

  • preemption: whether a VCPU is preempted or gives up the PCPU voluntarily when it is scheduled out by the hypervisor

  • migration: the time point at which a VCPU is migrated and the migration frequency of a VCPU.

Moreover, with the rich data about the scheduling history of VCPUs, VSA can conduct further advanced analysis, including:

  • utilization of each PCPU

  • domain-based analysis: the scheduling analysis for a group of interacting VCPUs running in the same guest domain.

This paper presents the key design issues of VSA, and also a number of case studies using VSA to analyze the scheduling behaviors of a Xen-based virtualized system. The design strategies and the case studies explain the merits of VSA and demonstrate how the analysis data generated by VSA can help understand the behaviors of Xen-based virtualized systems, and therefore help further system optimization.

The rest of the paper is organized as follows: Section  2 introduces the background knowledge for understanding this paper. Section  3 discusses the key design issues of VSA. In Section  4, we present three case studies to demonstrate and explain how the analysis data generated by VSA can be used to reveal the scheduling behavior of VCPUs, PCPUs and domains. Section  5 presents related works. Section  6 concludes the paper and discusses future works of VSA.

Section snippets

Background

This section briefly discusses the background knowledge required to understand this work, including the scheduling architecture in Xen VMM, the split driver model, three schedulers used in the Xen hypervisor, and how trace data are generated and processed in Xen.

System design

In this section, we first explain in Section  3.1 how VSA uses the records in the text trace file to compute the time attributes that can be used to conduct the scheduling analysis, and also the method provided by VSA to extract interested time durations from the entire trace period. Then, we discuss the key functionalities of VSA: conducting the scheduling analysis for individual VCPUs (Section  3.2) and conducting a group of interacting VCPUs running in the same domain (Section  3.3).

Case studies of using VSA to conduct analyses

In this section, we present three case studies to demonstrate how VSA can be used to analyze the behavior of the schedulers in the Xen hypervisor, and to analyze different scheduling characteristics for different types of applications. In the first case study, by using the VSA data, we compare the behaviors of the SEDF and the Credit scheduler when they handle the identical parallel applications. In the second case study, we use VSA to compare the scheduling characteristics of different types

Related works

Xentop and XenMon are two widely used tools to analyze the performance of Xen VMM.

  • Xentop: Xentop is the built-in performance monitor tool of Xen. It provides the real time data for the usage rate of system resources (such as memory, virtual network interfaces, virtual block devices, and so on), the domains and their constituent VCPUs. However, as a general tool, the information provided by Xentop on VCPU scheduling is not sufficient (only the usage rates are provided). Therefore, it is very

Conclusions and future work

An offline scheduling analyzer for the Xen VMM, called VSA, is developed in this paper. By using the trace data, the system can reconstruct the scheduling history of each VCPU, and conduct advanced analyses, such as the block-to-wakeup, wake-to-schedule-in, preemption analysis and so on. This paper discusses the design and functionalities of VSA, and also presents three case studies. The design strategies and the case studies explain the merits of VSA and demonstrate how the analysis data

Acknowledgments

This work is supported by National Natural Science Foundation of China under grant No. 60903022, 612320081, Research Fund for the Doctoral Program of Higher Education of China under grant No. 20100142120027, and the Leverhulme Trust in UK (grant number RPG-101).

Zhiyuan Shao received Ph.D. in computer science and engineering from Huazhong University of Science and Technology (HUST), China, in 2005. He is now an Associate Professor in school of Computer Science and Engineering at HUST. He has served as a reviewer for many conferences and journal papers. His research interests are in the areas of operating systems, virtualization technology for computing system and computer networks. He is a member of the IEEE and the IEEE Computer Society.

References (21)

  • X. Liao et al.

    ESPM: an optimized resource distribution policy in virtual user environment

    Future Generation Computer Systems

    (2010)
  • Amazon EC2 Cloud. URL...
  • Gogrid Cloud. URL...
  • B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt, A. Warfield, P. Barham, R. Neugebauer, Xen and the art of...
  • I.M. Leslie et al.

    The design and implementation of an operating system to support distributed multimedia applications

    IEEE Journal of Selected Areas in Communications

    (1996)
  • Credit Scheduler. URL...
  • C.S. Pabla

    Completely fair scheduler

    Linux Journal

    (2009)
  • C.S. Wong et al.

    Towards achieving fairness in the Linux scheduler

    ACM SIGOPS Operating Systems Review

    (2008)
  • L. Cherkasova et al.

    Comparison of the three CPU schedulers in Xen

    ACM SIGMETRICS Performance Evaluation Review

    (2007)
  • K.J. Duda, D.R. Cheriton, Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a...
There are more references available in the full text version of this article.

Cited by (7)

View all citing articles on Scopus

Zhiyuan Shao received Ph.D. in computer science and engineering from Huazhong University of Science and Technology (HUST), China, in 2005. He is now an Associate Professor in school of Computer Science and Engineering at HUST. He has served as a reviewer for many conferences and journal papers. His research interests are in the areas of operating systems, virtualization technology for computing system and computer networks. He is a member of the IEEE and the IEEE Computer Society.

Ligang He is an Associate Professor in the Department of Computer Science at the University of Warwick. He studied for a Ph.D. in Computer Science at the University of Warwick, UK, from 2002 to 2005, and then worked as a post-doctor in the University of Cambridge, UK. In 2006, he joined the Department of Computer Science at the University of Warwick as an Assistant Professor. His research interests focus on parallel and distributed processing, Cluster, Grid and Cloud computing. He has published more than 40 papers in international conferences and journals, such as IEEE Transactions on Parallel and Distributed Systems, IPDPS, CCGrid, MASCOTS. He has been a member of the program committee for many international conferences, and has been the reviewer for a number of international journals, including IEEE Transactions on Parallel and Distributed Systems, IEEE Transactions on Computers, etc. He is a member of the IEEE.

Zhiqiang Lu is a Master’s student in school of Computer Science and Engineering at Huazhong University of Science and Technology. His research area is Cloud computing and virtualization.

Hai Jin is a professor of computer science and engineering at the Huazhong University of Science and Technology (HUST) in China. He is now Dean of the School of Computer Science and Technology at HUST. Jin received his Ph.D. in computer engineering from HUST in 1994. In 1996, he was awarded a German Academic Exchange Service fellowship to visit the Technical University of Chemnitz in Germany. Jin worked at the University of Hong Kong between 1998 and 2000, and as a visiting scholar at the University of Southern California between 1999 and 2000. He was awarded Excellent Youth Award from the National Science Foundation of China in 2001. Jin is the chief scientist of ChinaGrid, the largest grid computing project in China. Jin is a senior member of the IEEE and a member of the ACM. Jin is the member of Grid Forum Steering Group (GFSG). He has co-authored 15 books and published over 400 research papers. His research interests include computer architecture, virtualization technology, cluster computing and grid computing, peer-to-peer computing, network storage, and network security. Jin is the steering committee chair of International Conference on Grid and Pervasive Computing (GPC), Asia-Pacific Services Computing Conference (APSCC), International Conference on Frontier of Computer Science and Technology (FCST), and Annual ChinaGrid Conference. Jin is a member of the steering committee of the IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid), the IFIP International Conference on Network and Parallel Computing (NPC), and the International Conference on Grid and Cooperative Computing (GCC), International Conference on Autonomic and Trusted Computing (ATC), International Conference on Ubiquitous Intelligence and Computing (UIC).

1

Tel.: +44 2476573802.

View full text