research-article

KVM/ARM: the design and implementation of the linux ARM hypervisor

Authors:

Christoffer Dall,

Jason NiehAuthors Info & Claims

ASPLOS '14: Proceedings of the 19th international conference on Architectural support for programming languages and operating systems

Pages 333 - 348

https://doi.org/10.1145/2541940.2541946

Published: 24 February 2014 Publication History

Abstract

As ARM CPUs become increasingly common in mobile devices and servers, there is a growing demand for providing the benefits of virtualization for ARM-based devices. We present our experiences building the Linux ARM hypervisor, KVM/ARM, the first full system ARM virtualization solution that can run unmodified guest operating systems on ARM multicore hardware. KVM/ARM introduces split-mode virtualization, allowing a hypervisor to split its execution across CPU modes and be integrated into the Linux kernel. This allows KVM/ARM to leverage existing Linux hardware support and functionality to simplify hypervisor development and maintainability while utilizing recent ARM hardware virtualization extensions to run virtual machines with comparable performance to native execution. KVM/ARM has been successfully merged into the mainline Linux kernel, ensuring that it will gain wide adoption as the virtualization platform of choice for ARM. We provide the first measurements on real hardware of a complete hypervisor using ARM hardware virtualization support. Our results demonstrate that KVM/ARM has modest virtualization performance and power costs, and can achieve lower performance and power costs compared to x86-based Linux virtualization on multicore hardware.

References

[1]

K. Adams and O. Agesen. A Comparison of Software and Hardware Techniques for x86 Virtualization. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, pages 2--13, Oct. 2006.

Digital Library

[2]

O. Agesen, J. Mattson, R. Rugina, and J. Sheldon. Software Techniques for Avoiding Hardware Virtualization Exits. In Proceedings of the 2012 USENIX Annual Technical Conference, pages 373--385, June 2012.

Digital Library

[3]

ARM Ltd. ARM Energy Probe. http://www.arm.com/products/tools/arm-energy-probe.php.

[4]

ARMLtd. ARMCortex-A15 Technical ReferenceManual ARM DDI 0438C, Sept. 2011.

[5]

ARMLtd. ARMGeneric Interrupt Controller Architecture version 2.0 ARMIHI 0048B, June 2011.

[6]

ARM Ltd. ARM Architecture Reference Manual ARMv7-A DDI0406C.b, July 2012.

[7]

ARM Ltd. ARM Architecture Reference Manual ARMv8-A DDI0487A.a, Sept. 2013.

[8]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the Art of Virtualization. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, pages 164--177, Oct. 2003.

Digital Library

[9]

K. Barr, P. Bungale, S. Deasy, V. Gyuris, P. Hung, C. Newell, H. Tuch, and B. Zoppis. The VMware Mobile Virtualization Platform: is that a hypervisor in your pocket? SIGOPS Operating Systems Review, 44(4):124--135, Dec. 2010.

Digital Library

[10]

E. Bugnion, S. Devine, M. Rosenblum, J. Sugerman, and E. Y. Wang. Bringing Virtualization to the x86 Architecture with the Original VMwareWorkstation. ACM Transactions on Computer Systems, 30(4):12:1--12:51, Nov. 2012.

Digital Library

[11]

C. Dall and A. Jones. KVM/ARM Unit Tests. https://github.com/columbia/kvm-unit-tests.

[12]

C. Dall and J. Nieh. KVM for ARM. In Proceedings of the Ottawa Linux Symposium, pages 45--56, July 2010.

[13]

C. Dall and J. Nieh. Supporting KVMon the ARMarchitecture. LWN.net, July 2013. http://lwn.net/Articles/557132/.

[14]

David Brash, Architecture Program Manager, ARM Ltd. Personal communication, Nov. 2012.

[15]

J.-H. Ding, C.-J. Lin, P.-H. Chang, C.-H. Tsang, W.-C. Hsu, and Y.-C. Chung. ARMvisor: System Virtualization for ARM. In Proceedings of the Ottawa Linux Symposium (OLS), pages 93--107, July 2012.

[16]

General Dynamics. OKL4 Microvisor. http://www.ok-labs.com/products/okl4-microvisor.

[17]

Green Hills Software. INTEGRITY Secure Virtualization. http://www.ghs.com/products/rtos/integrity_virtualization.html.

[18]

J. Hwang, S. Suh, S. Heo, C. Park, J. Ryu, S. Park, and C. Kim. Xen on ARM: System Virtualization using Xen Hypervisor for ARM-based Secure Mobile Phones. In Proceedings of the 5th Consumer Communications and Newtork Conference, Jan. 2008.

[19]

InSignal Co. ArndaleBoard.org. http://arndaleboard.org.

[20]

Intel Corporation. Intel 64 and IA-32 Architectures Software DevelopersManual, 325462-044US, Aug. 2012.

[21]

A. Kivity. KVMUnit Tests. https://git.kernel.org/cgit/virt/kvm/kvm-unit-tests.git.

[22]

A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: The Linux VirtualMachineMonitor. In Proceedings of the Ottawa Linux Symposium (OLS), volume 1, pages 225--230, June 2007.

[23]

KVM/ARM Mailing List. https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm.

[24]

Linux ARM Kernel Mailing List. A15 H/W Virtualization Support, Apr. 2011. http://archive.arm.linux.org.uk/lurker/message/20110412.204714.a36702d9.en.html.

[25]

L. McVoy and C. Staelin. lmbench: Portable Tools for Performance Analysis. In Proceedings of the 1996 USENIX Annual Technical Conference, pages 279--294, Jan. 1996.

Digital Library

[26]

I. Molnar. Hackbench. http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c.

[27]

G. J. Popek and R. P. Goldberg. Formal Requirements for Virtualizable Third Generation Architectures. Communications of the ACM, 17(7):412--421, July 1974.

Digital Library

[28]

Red Bend Software. vLogixMobile. http://www.redbend.com/en/mobile-virtualization.

[29]

R. Russell. virtio: Towards a De-Facto Standard for Virtual I/O Devices. SIGOPS Operating Systems Review, 42(5):95--103, July 2008.

Digital Library

[30]

U. Steinberg and B. Kauer. Nova: A Microhypervisor-Based Secure Virtualization Architecture. In Proceedings of the 5th European Conference on Computer Systems, pages 209--222, Apr. 2010.

Digital Library

[31]

P. Varanasi and G. Heiser. Hardware-Supported Virtualization on ARM. In Proceedings of the Second Asia-Pacific Workshop on Systems, pages 11:1--11:5, July 2011.

Digital Library

[32]

Xen.org. Xen ARM. http://xen.org/products/xen_arm.html.

Cited By

Chiang YChang WLi STu J(2024)Securing a Multiprocessor KVM Hypervisor with RustProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698562(650-667)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698562
Röckl JBernsdorf NMüller TQuek TGao DZhou JCardenas A(2024)TeeFilter: High-Assurance Network Filtering Engine for High-End IoT and Edge Devices based on TEEsProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3637643(1568-1583)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3637643
Sousa JMartins JGomes TPinto S(2024)HSP-V: Hypervisor-Less Static Partitioning for RISC-V COTS PlatformsIEEE Access10.1109/ACCESS.2024.339960112(71131-71144)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3399601
Show More Cited By

Index Terms

KVM/ARM: the design and implementation of the linux ARM hypervisor

Recommendations

NEVE: Nested Virtualization Extensions for ARM
SOSP '17: Proceedings of the 26th Symposium on Operating Systems Principles

Nested virtualization, the ability to run a virtual machine inside another virtual machine, is increasingly important because of the need to deploy virtual machines running software stacks on top of virtualized cloud infrastructure. As ARM servers make ...
KVM/ARM: the design and implementation of the linux ARM hypervisor
ASPLOS '14

As ARM CPUs become increasingly common in mobile devices and servers, there is a growing demand for providing the benefits of virtualization for ARM-based devices. We present our experiences building the Linux ARM hypervisor, KVM/ARM, the first full ...
KVM/ARM: the design and implementation of the linux ARM hypervisor
ASPLOS '14

As ARM CPUs become increasingly common in mobile devices and servers, there is a growing demand for providing the benefits of virtualization for ARM-based devices. We present our experiences building the Linux ARM hypervisor, KVM/ARM, the first full ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASPLOS '14: Proceedings of the 19th international conference on Architectural support for programming languages and operating systems

February 2014

780 pages

ISBN:9781450323055

DOI:10.1145/2541940

General Chairs:
Rajeev Balasubramonian
University of Utah
,
Al Davis
University of Utah
,
Program Chair:
Sarita Adve
University of Illinois at Urbana-Champ

ACM SIGPLAN Notices Volume 49, Issue 4
ASPLOS '14
April 2014
729 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2644865
Editors:
Mark W. Bailey
Hamilton College, Clinton, NY
,
Rajeev Balasubramonian
University of Utah
,
Al Davis
University of Utah
,
Sarita Adve
University of Illinois at Urbana-Champ
Issue’s Table of Contents
ACM SIGARCH Computer Architecture News Volume 42, Issue 1
ASPLOS '14
March 2014
729 pages
ISSN:0163-5964
DOI:10.1145/2654822
Editor:
Doug DeGroot
acm dot org
Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 February 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASPLOS '14

Sponsor:

ASPLOS '14: Architectural Support for Programming Languages and Operating Systems

March 1 - 5, 2014

Utah, Salt Lake City, USA

Acceptance Rates

ASPLOS '14 Paper Acceptance Rate 49 of 217 submissions, 23%;

Overall Acceptance Rate 535 of 2,713 submissions, 20%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

156
Total Citations
View Citations
2,478
Total Downloads

Downloads (Last 12 months)178
Downloads (Last 6 weeks)17

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chiang YChang WLi STu J(2024)Securing a Multiprocessor KVM Hypervisor with RustProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698562(650-667)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698562
Röckl JBernsdorf NMüller TQuek TGao DZhou JCardenas A(2024)TeeFilter: High-Assurance Network Filtering Engine for High-End IoT and Edge Devices based on TEEsProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3637643(1568-1583)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3637643
Sousa JMartins JGomes TPinto S(2024)HSP-V: Hypervisor-Less Static Partitioning for RISC-V COTS PlatformsIEEE Access10.1109/ACCESS.2024.339960112(71131-71144)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3399601
Zhao KXue KWang ZSchatzberg DYang LManousis AWeiner JVan Riel RSharma BTang CSkarlatos DSolihin YHeinrich M(2023)Contiguitas: The Pursuit of Physical Memory Contiguity in DatacentersProceedings of the 50th Annual International Symposium on Computer Architecture10.1145/3579371.3589079(1-15)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3579371.3589079
Doran MKandalaft N(2023)Embedded Virtualization on RISC-V with seL42023 IEEE 14th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON)10.1109/UEMCON59035.2023.10316016(0736-0740)Online publication date: 12-Oct-2023
https://doi.org/10.1109/UEMCON59035.2023.10316016
Ning ZWang CChen YZhang FCao J(2023)Revisiting ARM Debugging Features: Nailgun and its DefenseIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2021.313984020:1(574-589)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TDSC.2021.3139840
Stojkovic JSkarlatos DKokolis AXu TTorrellas JFalsafi BFerdman MLu SWenisch T(2022)Parallel virtualized memory translation with nested elastic cuckoo page tablesProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3503222.3507720(84-97)Online publication date: 28-Feb-2022
https://dl.acm.org/doi/10.1145/3503222.3507720
Gong LLi ZWang HLin HMa XLiu Y(2022)Overlay-Based Android Malware Detection at Market Scales: Systematically Adapting to the New Technological LandscapeIEEE Transactions on Mobile Computing10.1109/TMC.2021.307943321:12(4488-4501)Online publication date: 1-Dec-2022
https://doi.org/10.1109/TMC.2021.3079433
Wei NSani A(2022)SchrodinText: Strong Protection of Sensitive Textual Content of Mobile ApplicationsIEEE Transactions on Mobile Computing10.1109/TMC.2020.302511921:4(1402-1419)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TMC.2020.3025119
Poovizhi JDevi R(2022)Performance Analysis of Cloud Hypervisor using Different Workloads in Virtualization2022 11th International Conference on System Modeling & Advancement in Research Trends (SMART)10.1109/SMART55829.2022.10047415(414-418)Online publication date: 16-Dec-2022
https://doi.org/10.1109/SMART55829.2022.10047415
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten