KVM/ARM: The Design and
Implementation of the Linux
ARM Hypervisor
Christoffer Dall Department of Computer Science Columbia University [email protected] Jason Nieh
Department of Compouter Science Columbia University [email protected]
32091700 김해천
ARM
~1.2 billion
~300 million
ARM
ARM Server
ARM Network infrasturcture
http://www.v3.co.uk/v3-uk/news/2397366/arm-aims-at-networkinfrastructure-with-many-core-socs-and-open-software-stack
Key Challenges
No PC-standard on ARM
Hypervisor Layering in software stack

Above part

HyperOne, Xen, PikeOS, OKL4,
Hyper V, Vmware ESX

Lower part

KVM, VirtyalBox, Virtual PC,
Parallels, BlueStacks
ARM Virtualization Extensions

Provides virtualization in 4 key areas:

CPU Virtualization

Memory Virtualization

Interrupt Virtualization

Timer Virtualization
ARM Virtualization Extensions
CPU Virtualization

Hyp mode was introduced as a trap and emulate mechanism to support
virtualization
User
User
Kernel
Kernel
Hyp
H/w
System call, page fault
To reduce virtualization overhead
ARM Virtualization Extensions
memory virtualization

Hardware support to virtualize physical memory: stage 2 Page Tables
ARM Virtualization Extensions
Interrupt virtualization

One distributor in a system, but each CPU core has a cpu Interface

Distributor is used to configure the GIC

CPU interface is used to acknowledge(ACK) and to signal End-OfInterrupt(EOI)


Interrupt can be configured to trap to either Hyp or Kernel mode

Trap to kernel : avoiding the overhead of going through Hyp mode

Trap to Hypervisor : hypervisor retain control, but big cost
VGIC
GIC v2.0 include H/W virtualization (VGIC)

Virtual CPU interface, List Register
PPI: Private Peripheral Interrupts
SPI: Shared Peripheral Interrupts
SGI: Soft Generated Interrupt
GIC : Generic Interrupt Controller
ARM Virtualization Extensions
Interrupt virtualization

Generic Interrupt Controller : Trapping Interrupt in Hyp Mode
Vm
3) Emulate Virtual Interrupt By signal
Cumbersome & Expensive
Hypervisor
2) trap
1) interrupt
H/W
ARM Virtualization Extensions
Interrupt virtualization

Generic Interrupt Controller (V2.0) , Virtual GIC : Trapping Interrupt in Kernel Mode
Vm
Hypervisor
2) trap
1) interrupt
H/W
Good
ARM Virtualization Extensions
Timer virtualization

ARM define the Generic Timer Architecture

The timers used by the hypervisor cannot be directly configured and manipulated by guest OSes.

Such timer accesses from guest OS would need to trap to Hyp mode, incurring additional overhead
counter
ARM provides ☞
Timer 0
CPU 0
Timer 1
CPU 1
Timer 2
CPU 2
Timer 3
CPU 3
Virtual counter
Virtual Timer 0
Virtual CPU 0
Virtual counter
Virtual Timer 1
Virtual CPU 1
Virtual counter
Virtual Timer 2
Virtual CPU 2
Virtual counter
Virtual Timer 3
Virtual CPU 3
Accessible from Hyp mode
Accessible from VMs
Hypervisor Architecture

KVM/ARM builds on KVM and leverages existing infrastructure in the Linux kernel

Bare metal hypervisor(xen) vs KVM/ARM

ARM platform designs are non-standard ways by different manufactures


Samsung exynos, qualcomm snapdragon, Apple A series
But, Linux is supported across almost all ARM platform ☞ by integrating KVM/ARM with Linux
PL0 User
PL1 Kernel
PL2 Hyp
Linux kernel KVM
Hypervisor Architecture
Split-mode Virtualization

Running KVM/ARM in Hyp mode implies running the Linux kernel in Hyp mode
This is problematic

Low-level architecture dependent code in Linux is written to work in kernel mode

Running the entire kernel in Hyp mode would adversely affect native performance
Kernel mode
Hyp mode
Kernel
Kernel
Hypervisor Architecture
Split-mode Virtualization



KVM/ARM introduces split-mode virtualization

It runs across different privileged CPU mode to take advantage offered by each CPU mode

Two components, the lowvisor and the highvisor
Lowvisor takes advantage of the H/W virtualization support available in Hyp mode

Set up the correct execution context by configuration of the H/W

Enforce protection and isolation between different execution context

Switch from a VM execution context to the host execution, vice versa

Provide a virtualization trap handler
Highvisor can directly leverage existing Linux functionality

Scheduler, kernel data structure, locking, memory allocation functions
Kernel
OS
mode Kernel
Hyp
mode
Hypervisor
High
visor
Handles High level
Functionality
Low
visor
Handles Low level
Functionality
Hypervisor Architecture
Split-mode Virtualization

Switching between a VM and the highvisor
OS Kernel
Kernel mode
Run VM
Hypervisor
Highvisor
Trap
Hyp mode
VM
Lowvisor
Trap
Hypervisor Architecture
Split-mode Virtualization

Switching between a VM and the highvisor
OS Kernel
Kernel mode
Function call
Hypervisor
Highvisor
Trap
Hyp mode
VM
Trap
Lowvisor
As a result, split mode virtualization incurs a double trap cost in switching to and from the highvisor
Hypervisor Architecture
CPU Virtualization
Context switch register
during world-switch
physical H/W state associated
with the hypervisor and its host
kernel is persistent across
running VMs
Performs trap and
emulate on sensitive
instruction and when
accessing H/W state
Hypervisor
VM
ARM
Virtualized cpu
trap
Controlled by the Hypervisor
S/W in the VM must have persistent
access to same register state as
S/W running on the physical CPU
Hypervisor Architecture
Memory Virtualization

KVM/ARM provides memory virtualization by enabling Stage-2 translation

When running in a VM

Completely transparent to the VM

The highvisor manages the Stage-2 translation page tables to only allow access to memory
allocated for a VM

Other accesses will cause stage-2 page faults which trap to the hypervisor

Stage-2 translation is disabled when running in the highvisor and lowvisor
Hypervisor Architecture
Memory Virtualization

Configuring page tables is a high level Functionality
OS Kernel
Kernel mode
Hyp mode
Configures Stage-2
Page Tables
Hypervisor
Highvisor
Lowvisor
VM
Hypervisor Architecture
Memory Virtualization

LowVisor has hardware access as it runs in Hyp Mode
OS Kernel
Kernel mode
Hyp mode
Hypervisor
VM
Highvisor
Lowvisor
Enables Stage-2
Translation
Hypervisor Architecture
Memory Virtualization
get_user_pages()
OS Kernel
Hypervisor
VM
Highvisor
Kernel mode
Hyp mode
Lowvisor
Disables Stage-2
Translation
Page fault
Hypervisor Architecture
Interrupt Virtualization

When running in a VM or Host & highvisor


However, VM must receive notifications in the form of virtual interrupt from emulated
devices


All, H/W interrupt processing is done in the host by using Linux’s existing interrupt handling func
KVM/ARM uses the VGIC
Multicore guest Oses musts be able to send virtual IPIs
Kernel
mode
OS
Kernel
to others
VM
High
visor
Hypervisor
Hyp
Trap
mode
Low
visor
Trap
H/W
Hypervisor Architecture
Timer Virtualization

KVM/ARM leverage ARM’s H/W virtualization features of the generic timer

Unfortunately, due to architectural limitations, the virtual timers cannot directly
raise virtual interrupts, but always raise hardware interrupt, which trap to the
hypervisor

KVM/ARM detects when a Virtual timer expires Injects
a corresponding virtual interrupt to the VM

Kernel
mode
KVM/ARM performs all hardware ACK and EOI operations
OS
Kernel
VM
VM
High
visor
Prepare
Low
visor
Hyp
mode
Hypervisor
trap
timer
Enable virtual
timer
Experimental Setup
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
Experimental Setup
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
Experimental Results

Table 3 presents costs of virtualization using KVM/ARM on ARM and KVM x86 on x86

Measured in cycle units
Hypercall
- Cost of two
world switch
Trap
- Cost of switching the
h/w mode from the
into the cpu mode
bg
bg
bg
- Saving & restore VGIC
state is quite expensive on
ARM.
- x86 provides H/W support
Experimental Results

Figure 3,4 show normalized performance for running lmbench in a VM versus Host
Caused by updating the
run-queue clock
KVM/ARM has less overhead
than KVM x86 fork & exec
UP: uni-processor
SMP:symmetrical multi-processing
Repeatedly sending an IPI Cost of KVM is higher
than KVM/ARM
Because this require tapping to the hypervisor on
x86 but not on ARM
Experimental Results

Figure 5,6 show normalized performance for running application workloads
More mature KVM x86 system has significantly higher virtualization overheads,
KVM/ARM’s split-mode virtualization design allows it to leverage ARM H/W support
with comparable performance to traditional hypervisor
Experimental Results

Figure 7 shows normalized power consumption of using virtualization

As well as ARM, mac air’s i7 is one of Intel’s more power optimized processors
Both workloads are not CPU bound & the power consumption
is not significantly affected by the virtualization layer

http://www.linux-kongress.org/2010/slides/KVM-Architecture-LK2010.pdf

http://gigglehd.com/zbxe/4440629

http://www.arm.com/

http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf

http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf

esc2014chhypforarmv7-140829040738.pdf

kvm/arm experiences building the linux arm hypervisor
Virtual CPU
Interface
Virtual CPU
Interface
VIRQ
ACK