VSA: An offline scheduling analyzer for Xen virtual machine monitor
Highlights
► Demonstrate the lack of support for advanced scheduling analyses in the existing tools. ► Develop a tool to conduct advance scheduling analyses in virtualized systems. ► Conduct the case studies to demonstrate the effectiveness of the developed tool.
Introduction
It is an important trend nowadays to use software-based virtualization technologies to build the private data centers and public cloud systems, such as Amazon EC2 [1] and GoGrid [2]. Among these virtualization technologies, the Xen Virtual Machine Monitor (VMM) [3], which allow users to execute up to hundreds of virtual machines on a single physical machine with low extra overhead, is widely adopted for such purposes [4].
With the introduction of the Xen virtualization layer, traditional operating systems (e.g., Linux) are para-virtualized (i.e., the machine sensitive and privileged operations are replaced with hypercalls) to run on top of the Xen hypervisor. The operating systems in the virtualized environments are called Guest Operating Systems (GOS). In this new architecture, there exist two schedulers working together: one works inside the Xen hypervisor and the other in the GOS. The examples of the former include the Simple Earliest Deadline First Scheduler (SEDF) [5] and the Credit Scheduler [6], which schedule the VCPUs to run on top of physical processors, while the examples of the latter include Completely Fair Scheduler (i.e., CFS) [7], [8] in Linux, which schedules the application processes to run in the VCPUs. Although there is long history of research in GOS schedulers and the GOS schedulers have been relatively mature, the schedulers inside the hypervisor are still under active research and development. To help understand the scheduling behaviors inside the hypervisor, Xen provides a powerful tool, named xentrace, to collect system trace data (including scheduling-related data), and a related format tool, xentrace_format, to convert the trace data into a human readable form. However, it is still very difficult to analyze the scheduling behavior of each individual VCPU by using the trace data alone.2
In this work, we design and implement VSA,3 an offline scheduling analyzer for the Xen VMM. The system can reconstruct the scheduling history of each VCPU based on the offline trace data generated by xentrace and xentrace_format. By using the scheduling history data of each VCPU, VSA can render detailed analysis data on VCPUs, including:
- •
block-to-wake time: the duration between the time when a VCPU is blocked and the time it is woken up from the blocking state
- •
wake-to-schedule-in time: the duration between the time when a VCPU is woken up and the time when it is scheduled to run
- •
preemption: whether a VCPU is preempted or gives up the PCPU voluntarily when it is scheduled out by the hypervisor
- •
migration: the time point at which a VCPU is migrated and the migration frequency of a VCPU.
Moreover, with the rich data about the scheduling history of VCPUs, VSA can conduct further advanced analysis, including:
- •
utilization of each PCPU
- •
domain-based analysis: the scheduling analysis for a group of interacting VCPUs running in the same guest domain.
This paper presents the key design issues of VSA, and also a number of case studies using VSA to analyze the scheduling behaviors of a Xen-based virtualized system. The design strategies and the case studies explain the merits of VSA and demonstrate how the analysis data generated by VSA can help understand the behaviors of Xen-based virtualized systems, and therefore help further system optimization.
The rest of the paper is organized as follows: Section 2 introduces the background knowledge for understanding this paper. Section 3 discusses the key design issues of VSA. In Section 4, we present three case studies to demonstrate and explain how the analysis data generated by VSA can be used to reveal the scheduling behavior of VCPUs, PCPUs and domains. Section 5 presents related works. Section 6 concludes the paper and discusses future works of VSA.
Section snippets
Background
This section briefly discusses the background knowledge required to understand this work, including the scheduling architecture in Xen VMM, the split driver model, three schedulers used in the Xen hypervisor, and how trace data are generated and processed in Xen.
System design
In this section, we first explain in Section 3.1 how VSA uses the records in the text trace file to compute the time attributes that can be used to conduct the scheduling analysis, and also the method provided by VSA to extract interested time durations from the entire trace period. Then, we discuss the key functionalities of VSA: conducting the scheduling analysis for individual VCPUs (Section 3.2) and conducting a group of interacting VCPUs running in the same domain (Section 3.3).
Case studies of using VSA to conduct analyses
In this section, we present three case studies to demonstrate how VSA can be used to analyze the behavior of the schedulers in the Xen hypervisor, and to analyze different scheduling characteristics for different types of applications. In the first case study, by using the VSA data, we compare the behaviors of the SEDF and the Credit scheduler when they handle the identical parallel applications. In the second case study, we use VSA to compare the scheduling characteristics of different types
Related works
Xentop and XenMon are two widely used tools to analyze the performance of Xen VMM.
- •
Xentop: Xentop is the built-in performance monitor tool of Xen. It provides the real time data for the usage rate of system resources (such as memory, virtual network interfaces, virtual block devices, and so on), the domains and their constituent VCPUs. However, as a general tool, the information provided by Xentop on VCPU scheduling is not sufficient (only the usage rates are provided). Therefore, it is very
Conclusions and future work
An offline scheduling analyzer for the Xen VMM, called VSA, is developed in this paper. By using the trace data, the system can reconstruct the scheduling history of each VCPU, and conduct advanced analyses, such as the block-to-wakeup, wake-to-schedule-in, preemption analysis and so on. This paper discusses the design and functionalities of VSA, and also presents three case studies. The design strategies and the case studies explain the merits of VSA and demonstrate how the analysis data
Acknowledgments
This work is supported by National Natural Science Foundation of China under grant No. 60903022, 612320081, Research Fund for the Doctoral Program of Higher Education of China under grant No. 20100142120027, and the Leverhulme Trust in UK (grant number RPG-101).
Zhiyuan Shao received Ph.D. in computer science and engineering from Huazhong University of Science and Technology (HUST), China, in 2005. He is now an Associate Professor in school of Computer Science and Engineering at HUST. He has served as a reviewer for many conferences and journal papers. His research interests are in the areas of operating systems, virtualization technology for computing system and computer networks. He is a member of the IEEE and the IEEE Computer Society.
References (21)
- et al.
ESPM: an optimized resource distribution policy in virtual user environment
Future Generation Computer Systems
(2010) - Amazon EC2 Cloud. URL...
- Gogrid Cloud. URL...
- B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt, A. Warfield, P. Barham, R. Neugebauer, Xen and the art of...
- et al.
The design and implementation of an operating system to support distributed multimedia applications
IEEE Journal of Selected Areas in Communications
(1996) - Credit Scheduler. URL...
Completely fair scheduler
Linux Journal
(2009)- et al.
Towards achieving fairness in the Linux scheduler
ACM SIGOPS Operating Systems Review
(2008) - et al.
Comparison of the three CPU schedulers in Xen
ACM SIGMETRICS Performance Evaluation Review
(2007) - K.J. Duda, D.R. Cheriton, Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a...
Cited by (7)
A novel virtual machine scheduling policy based on performance prediction model
2018, International Journal of Networking and Virtual OrganisationsFine-grained multilayer virtualized systems analysis
2016, Journal of Cloud ComputingMultilayer virtualized systems analysis with kernel tracing
2016, Proceedings - 2016 4th International Conference on Future Internet of Things and Cloud Workshops, W-FiCloud 2016A scheme of memory privacy protection in cloud computing environment
2015, Wuhan University Journal of Natural SciencesAnomaly detection of virtual machines based on multi-attribute entropy in cloud computing environments
2015, Huazhong Keji Daxue Xuebao (Ziran Kexue Ban)/Journal of Huazhong University of Science and Technology (Natural Science Edition)Virtual machines CPU monitoring with kernel tracing
2014, Canadian Conference on Electrical and Computer Engineering
Zhiyuan Shao received Ph.D. in computer science and engineering from Huazhong University of Science and Technology (HUST), China, in 2005. He is now an Associate Professor in school of Computer Science and Engineering at HUST. He has served as a reviewer for many conferences and journal papers. His research interests are in the areas of operating systems, virtualization technology for computing system and computer networks. He is a member of the IEEE and the IEEE Computer Society.
Ligang He is an Associate Professor in the Department of Computer Science at the University of Warwick. He studied for a Ph.D. in Computer Science at the University of Warwick, UK, from 2002 to 2005, and then worked as a post-doctor in the University of Cambridge, UK. In 2006, he joined the Department of Computer Science at the University of Warwick as an Assistant Professor. His research interests focus on parallel and distributed processing, Cluster, Grid and Cloud computing. He has published more than 40 papers in international conferences and journals, such as IEEE Transactions on Parallel and Distributed Systems, IPDPS, CCGrid, MASCOTS. He has been a member of the program committee for many international conferences, and has been the reviewer for a number of international journals, including IEEE Transactions on Parallel and Distributed Systems, IEEE Transactions on Computers, etc. He is a member of the IEEE.
Zhiqiang Lu is a Master’s student in school of Computer Science and Engineering at Huazhong University of Science and Technology. His research area is Cloud computing and virtualization.
Hai Jin is a professor of computer science and engineering at the Huazhong University of Science and Technology (HUST) in China. He is now Dean of the School of Computer Science and Technology at HUST. Jin received his Ph.D. in computer engineering from HUST in 1994. In 1996, he was awarded a German Academic Exchange Service fellowship to visit the Technical University of Chemnitz in Germany. Jin worked at the University of Hong Kong between 1998 and 2000, and as a visiting scholar at the University of Southern California between 1999 and 2000. He was awarded Excellent Youth Award from the National Science Foundation of China in 2001. Jin is the chief scientist of ChinaGrid, the largest grid computing project in China. Jin is a senior member of the IEEE and a member of the ACM. Jin is the member of Grid Forum Steering Group (GFSG). He has co-authored 15 books and published over 400 research papers. His research interests include computer architecture, virtualization technology, cluster computing and grid computing, peer-to-peer computing, network storage, and network security. Jin is the steering committee chair of International Conference on Grid and Pervasive Computing (GPC), Asia-Pacific Services Computing Conference (APSCC), International Conference on Frontier of Computer Science and Technology (FCST), and Annual ChinaGrid Conference. Jin is a member of the steering committee of the IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid), the IFIP International Conference on Network and Parallel Computing (NPC), and the International Conference on Grid and Cooperative Computing (GCC), International Conference on Autonomic and Trusted Computing (ATC), International Conference on Ubiquitous Intelligence and Computing (UIC).
- 1
Tel.: +44 2476573802.