Xen 3.0.3 with
VMX
Presented by psboy
(Dec 7th,2006)
Scheme
－ Introduction to Xen 3.0.3 with VMX
－ Install/Build Domain 0 (Privileged
Domain)
－ Install Domain U (Unprivileged Domain)
－ Operate Xen Virtual Machine / Xen VMX
Server
－ Q&A
Introduction to Xen 3.0.3 with VMX
What is Xen?
 Xen is an open-source hypervisor that enables increased
server utilization and server consolidation by enabling
multiple operating system images to simultaneously run on a
single physical server. Xen provides resource guarantees to
virtual servers to ensure that application layer SLAs are met,
including CPU, memory and I/O guarantees.
 Xen is the industry’s fastest and most secure infrastructure
virtualization software technology, and has been endorsed
and adopted by over 20 of the industry’s major vendors,
including AMD, Dell, Egenera, Hewlett-Packard, IBM, Intel,
Mellanox Technologies, Network Appliance, Novell, Red Hat,
Sun Microsystems, TopSpin, Unisys and Voltaire.
Prepared by
psboy
Page 3
Introduction to Xen 3.0.3 with VMX
 Xen is licensed under the GNU General Public License (GPL2),
and is available free for download. It is offered by XenSource
and other vendors as a supported enterprise software product.
Who create Xen?
 Xen was created in 2003 at the University of Cambridge
Computer Laboratory in what’s known as the Xen Hypervisor
project led by Ian Pratt. Additional team members include Keir
Fraser, Steven Hand, and Christian Limpach. This same team
founded XenSource with experienced Silicon Valley
technology entrepreneurs Nick Gault and Simon Crosby.
Prepared by
psboy
Page 4
Introduction to Xen 3.0.3 with VMX
What are the business benefits of adopting
Xen?

Increased server utilization: Data centers can achieve dramatic
improvements in server utilization and enable server consolidation,
reducing wasted capital costs and personnel management expenses.
 Reduced burden on IT: By enabling server consolidation, Xen
reduces the cost, complexity, and personnel time required for data
center server management. XenSource’s virtualization infrastructure
software solutions provide both Xen based virtualization for servers
and management, and control and automation tools that allow IT
organizations to “operationalize” virtualization in a seamless, low cost
fashion.
 Lowest TCO: Today virtualization has yet to enter the heart of the data
center. There are two reasons: poor performance of proprietary
hypervisors and their tremendous cost. Xen is the best performing
hypervisor, and it’s free. That radically changes the economics of the
adoption of virtualization.
Prepared by
psboy
Page 5
Introduction to Xen 3.0.3 with VMX
Xen 3.0 features
 Support for up to 32-way SMP guest
 Intel® VT-x and AMD Pacifica hardware virtualization support
 PAE support for 32 bit servers with over 4 GB memory
 x86/64 support for both AMD64 and EM64T
Xen 3.0.3 features
 New easy-to-use CPU scheduler including weights, caps and
automatic load balancing
 Much enhanced support for unmodified ('hvm') guests including
windows and legacy linux systems
 Support for sparse and copy-on-write disks
 High performance networking using segmentation off-load
Prepared by
psboy
Page 6
Introduction to Xen 3.0.3 with VMX
Xen Networking
Prepared by
psboy
Page 7
Introduction to Xen 3.0.3 with VMX
Logical network cards connected between dom0
and dom1
Prepared by
psboy
Page 8
Introduction to Xen 3.0.3 with VMX
Illustration on network-bridge and vif-bridge:
Prepared by
psboy
Page 9
Introduction to Xen 3.0.3 with VMX
Intel Virtualization Technology
Prepared by
psboy
Xen 3.0 architecture
Page 10
Introduction to Xen 3.0.3 with VMX
VMCS
Shadow Page Table
Prepared by
psboy
Page 11
Introduction to Xen 3.0.3 with VMX
For the IA-32 architecture, a VMCS structure is created
for each CPU in a HVM domain
 Instructions such as CPUID, MOV from/to CR3, MOV to
CR0/CR4, RDMSR, WRMSR, HLT, INVLPG, MOV from
CR8, MOV DR, and MWAIT are intercepted as VM exits.
 Exceptions/faults, such as page fault, are intercepted as VM
exits, and virtualized exceptions/faults are injected on VM
entry to guests.
 External interrupts unrelated to guests are intercepted as VM
exits, and virtualized interrupts are injected on VM entry to
the guests.
 Read shadows are created for the guest CR0, CR4, and time
stamp counter (TSC). Read accesses to such registers will
not cause VM exit, but will return the shadow values.
Prepared by
psboy
Page 12
Introduction to Xen 3.0.3 with VMX
The virtual MMU module supports all page table
formats that can be used by the guest OS
 For IA-32
a. it supports 2-level page tables with 4 KB page size for 32-bit guests.
 For IA-32 Physical Address Extension (PAE)
a. it supports 2-level page tables with 4 KB page sizes for 32-bit guests.
b. it supports 3-level page tables with 4 KB and 2 MB page sizes for
32-bit PAE guests.
 For Intel® EM64T
a. it supports 2-level page tables with 4 KB page size for 32-bit guests.
b. it supports 3-level page tables with 4 KB and 2 MB page sizes for
32-bit PAE guests.
c. it supports 4-level page tables with 4 KB and 2 MB page sizes for
64-bit guests.
Prepared by
psboy
Page 13
Introduction to Xen 3.0.3 with VMX
Device IO Virtualization
The primary function of the device
model is to wait for an I/O event
from the HVM guest and dispatch it
to the appropriate device emulation
model. Once the device emulation
I/O Device virtualization model
model completes the I/O request, it
will respond back with the result.
A shared memory between the device model and the Xen
hypervisor is used for communication of I/O request and
response.
Prepared by
psboy
Page 14
Install Domain 0 (Privileged Domain)
Before install Domain 0
 Unmodified operating systems are supported. Support is
provided for running Windows on VT or AMD-V hardware.
 Xen supports 32-bit processors with and without Physical
Address Extension (PAE), 64-bit processors, and Symmetric
Multiprocessing (SMP) guest operating systems.
 Enterprise Linux guest operating systems supported: Red
Hat Enterprise Linux 3.5, 4.1, SUSE Linux Enterprise
Server 9.2; Also Fedora Core 4, CentOS 4, SUSE Open
Linux 10, Debian and others.
 Current test with Xen 3.03 for example , linux kernel
2.6.16.29.
Prepared by
psboy
Page 15
Install Domain 0 (Privileged Domain)
System requirement
1. Cpu need support VMX /SVM
 You can use command “cat /proc/cpuinfo”to
see if your cpu have vmx/svm flag
2. Chipset need support VMX
3. Bios need to enable VMX /SVM
 Intel VMX default is disable.
 AMD SVM default is enable.
4. You can try to use Xen Test CD first before
test Xen HVM.
Prepared by
psboy
Page 16
Install Domain 0 (Privileged Domain)
Several ways to install Domain 0
 Install from binary tarball
 Quick for installation but no preferred modules contain in
official release
 Install from RPMs
 Quick for installation but no preferred modules contain in
official release
 Install from source
 Kernel and module can customize and more flexible
 Dom0 kernel and hypervisor tuning is available
Prepared by
psboy
Page 17
Install Domain 0 (Privileged Domain)
Step 0.1
 Download Xen 3.03 source from
http://bits.xensource.com/oss-xen/release/3.0.30/src.tgz/xen-3.0.3_0-src.tgz
Download linux-2.6.16.29.tar.bz2 from
http://www.kernel.org
Step 0.2
 Install RHEL4 AS U4 32bit.(Full install)
 Recommend set your system resolution to 1024x768 for
later install Domain U.
 Don’t use whole disk as “/” (left some space for guest os )
Prepared by
psboy
Page 18
Install Domain 0 (Privileged Domain)
Step 1
 Extract xen-3.0.3_0-src.tgz.
 #gunzip xen-3.0.3_0-src.tgz.
 #tar zxvf xen-3.0.3_0-src.tar.
 #cd xen-3.0.3_0-src
 Copy linux-2.6.16.29.tar.bz2 to xen-3.0.3_0-src
directory.
 Don’t extract linux-2.6.16.29.tar.bz2 !!
Prepared by
psboy
Page 19
Install Domain 0 (Privileged Domain)
Step 2 Compile and install Xen and Dom0
kernel
 Check if your system need PAE support.When system
memory over 4G,you will need PAE support for high
address memory.
 Use “XEN_TARGET_X86_PAE=y” to make parameter.
 #XEN_TARGET_X86_PAE=y make world install
 Use the following command if you don’t need PAE
support.
 #make world install
Prepared by
psboy
Page 20
Install Domain 0 (Privileged Domain)
 If you want to rebuild or clean source directory before
compile , use the following command.
 #make clean world install
 If you want to customize your dom0 kernel config ,
use the following command.
 #make linux-2.6-xen-config CONFIGMODE=menuconfig
 #make linux-2.6-xen-build
 #make linux-2.6-xen-install
Prepared by
psboy
Page 21
Install Domain 0 (Privileged Domain)
 You can build dom0 and domU by KERNELS=“linux-2.6xen0 linux-2.6-xenU” parameter.
 #KERNELS=“linux-2.6-xen0 linux-2.6-xenU” make world
install
 To rebuild without change and clean source tree ,
use the following command.
 #make dist
 #make install
Prepared by
psboy
Page 22
Install Domain 0 (Privileged Domain)
Step 3 Make initial ram disk for domain 0
 If you want to use initial ram disk , your kernel
configuration should contain BLK_DEV_RAM=y and
BLK_DEV_INITRD=y setting , and BLK_DEV_RAM
can not be compile as module.
 #depmod 2.6.16.29-xen
 #mkinitrd –f –with=ata_piix –with=scsi_mod /boot/initrd2.6.16.29-xen 2.6.16.29-xen
 -f: overwrite existing image file
 --with=module_name : load module in the initial ram disk
Prepared by
psboy
Page 23
Install Domain 0 (Privileged Domain)
Step 4 Modify grub.conf
 Grub config need to modify for apply compiled xen
kernel and dom0 kernel
 Add entry in grub.conf such like below:
 title Xen 3.0.3
 root (hd0,0)
 kernel /xen-3.0.3-0.gz console=vga dom0_mem=262144
 module /vmlinuz-2.6.16.29-xen ro
root=/dev/VolGroup00/LogVol01 rhgb console=tty0
 module /initrd-2.6.16.29-xen
Prepared by
psboy
Page 24
Install Domain 0 (Privileged Domain)
Step 5 Disable tls library
#mv /lib/tls /lib/tls.disabled
Step 6 Reboot system and choose Xen 3.03
from grub boot menu
Step 7 Check your Domain 0 Xend service is
running
#chkconfig xend on
#service xend start
Prepared by
psboy
Page 25
Install Domain U (Unprivileged Domain)
Before install Domain U
 Check your hardware (bios) support VMX
 #xm info|grep xen_cap
xen_caps : xen-3.0-x86-32p hvm-3.0-x86-32 hvm-3.0-x86_32p
 #xm dmesg|grep –i vmxon
(XEN) VMXON is done
 Each time you start a guest domain , Xen will random
select a Mac 00:16:3E:XX:XX:XX if you didn’t set Mac
in your guest domain config file.
Prepared by
psboy
Page 26
Install Domain U (Unprivileged Domain)
Install windows based guest OS
(Install windows2003 RC2 as example)
Step 1 Create lvm partition for guest OS
 #lvcreate –L 10000 –n w2k3 VolGroup00
Step 2 Use xmexample.hvm as sample and
modify it.
 #cp /etc/xen/xmexample.hvm /etc/xen/w2k3.hvm
Modify below entries in w2k3.hvm





by
Prepared
psboy
memory = 512
name = w2k3
disk = [‘phy:/dev/VolGroup00/w2k3,hda,w’ ,‘phy:/dev/hdb,hdb:cdrom,r’]
cdrom = ‘/dev/hdb’
boot= ‘d’
sdl=1
Page 27
Install Domain U (Unprivileged Domain)
Step 3 Put windows 2003 RC2 cd into cdrom
 If /dev/hdb does not exist,use command “mknod /dev/hdb b 3 64” to
create /dev/hdb device node.
Step 4 Use xm command power on guest os
 #xm create /etc/xen/w2k3.hvm
Step 5 Follow the popup window screen and
continue install guest os
 Press any key when boot from windows 2003 cdrom.
 After first time install complete and reboot , change boot = ‘d’ to boot = ‘c’.
You can use iso file emulate cdrom
Prepared by
psboy
Page 28
Install Domain U (Unprivileged Domain)
Install linux based guest OS
(Install sles10 as example)
Step 1 Create lvm partition for guest OS
 #lvcreate –L 10000 –n sl10 VolGroup00
Step 2 Use xmexample.hvm as sample and
modify it.
 #cp /etc/xen/xmexample.hvm /etc/xen/sl10.hvm
Modify below entries in sl10.hvm






Prepared by
psboy

memory = 512
name = sl10
disk = [‘phy:/dev/VolGroup00/sl10,hda,w’ ,‘phy:/dev/hdb,hdb:cdrom,r’]
cdrom = ‘/dev/hdb’
boot= ‘d’
vnc=1
Page 29
vncviewer=1
Install Domain U (Unprivileged Domain)
If you have multiple nic devices , you need to modify
xend-config.sxp , sl10.hvm , script/mynetwork-script .
Modify sl10.hvm
 vif= [‘type=ioemu , bridge=xenbr0’,’type=ioemu ,
bridge=xenbr1’]
Add mynetwork-script to /etc/xen/script
#!/bin/sh mynetwork-script
dir=$(dirname “$0”)
“$dir/network-bridge” “$@” vifnum=0 netdev=eth0
“$dir/network-bridge” “$@” vifnum=1 netdev=eth1
Modify xend-config.sxp (search network-script)
(network-script mynetwork-script)
Prepared by
psboy
Page 30
Install Domain U (Unprivileged Domain)
Step 3 Put SLES10 cd into cdrom
 If /dev/hdb does not exist,use command “mknod /dev/hdb b 3 64” to
create /dev/hdb device node.
Step 4 Use xm command power on guest os
 #xm create /etc/xen/sl10.hvm
Step 5 Follow the popup window screen and
continue install guest os
 Type linux when boot from SLES10 cdrom.
 After first time install complete and reboot , change boot = ‘d’ to boot = ‘c’.
 Suggest to adjust resolution to 800x600 , use “vga=0x314”
You can use iso file emulate cdrom
Prepared by
psboy
Page 31
Operate Xen Virtual Machine / Xen VMX Server
Booting domain 0
 #service xend start
Booting guest domain
 #xm create /etc/xen/xmexample.hvm
You need modify your guest domain config file before start it
Stop guest domain
 Shutdown by init 0 command in guest OS
 #xm destroy domain-name (Or xm shutdown domain-name)
Prepared by
psboy
Page 32
Operate Xen Virtual Machine / Xen VMX Server
Booting domain 0
 #service xend start
Booting guest domain
 #xm create /etc/xen/xmexample.hvm
You need modify your guest domain config file before start it
Stop guest domain
 Shutdown by init 0 command in guest OS
 #xm destroy domain-name
XM tool (use this tool to operate guest domain)
 #xm help
Prepared by
psboy
Page 33
Operate Xen Virtual Machine / XenEnterprise Server
Basic Management Commands
# xm list which lists all domains running in rows of the following
format
The meaning of each field is as follows:
name
The descriptive name of the virtual machine.
domid
The number of the domain ID this virtual machine is running in.
memory
Memory size in megabytes.
vcpus
Prepared by
The number of virtual CPUs this domain has.
psboy
Page 34
Operate Xen Virtual Machine / XenEnterprise Server
state
Domain state consists of 5 fields:
r
running
b
blocked
p
paused
s
shutdown
c
crashed
cputime
How much CPU time (in seconds) the domain has used so far.
Prepared by
psboy
Page 35
Operate Xen Virtual Machine / XenEnterprise Server
# xm top
Monitor a host and domains in real time
# xm dmesg
# xm info
# xm uptime
Read and/or clear Xend’s message buffer
Get information about Xen host
Print uptime for a domain
Prepared by
psboy
Page 36
Operate Xen Virtual Machine / XenEnterprise Server
More detail guest domain config file entry
vif
List of MAC addresses (random addresses are assigned if not given)
and bridges to use for the domain's network interfaces, e.g. vif =
[ 'mac=00:16:3E:00:00:11, bridge=xen-br0', 'bridge=xen-br1' ] to
assign a MAC address and bridge to the first interface and assign a
different bridge to the second interface, leaving xend to choose the
MAC address.
disk
List of block devices to export to the domain e.g. disk =
[ 'phy:hda1,sda1,r' ] exports physical device /dev/hda1 to the
domain as /dev/sda1 with read-only access. Exporting a disk readwrite which is currently mounted is dangerous - if you are certain
you wish to do this, you can specify w as the mode.
memory
Memory size in megabytes.
extra
Extra string to append to the kernel command line (if any)
Prepared by
psboy
Page 37
Question and
Answer
Thank You