How to boot Xen on FVP(fast) model
This wiki page explains how to boot the Xen on FVP(fast) model. I have used the FVP_Base_AEMv8A-AEMv8A_0.8_5502 model. Note that all the FVP models are licensed models.
Build Xen
Clone the following repository and build the Xen hypervisor
host$ git clone git://xenbits.xen.org/xen.git host$ cd xen xen$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CONFIG_EARLY_PRINTK=fastmodel
Build Linux
host$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git host$ cd linux
Create a new kernel config:
linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig linux$ sed -e 's/.*CONFIG_XEN is not set/CONFIG_XEN=y/g' -i .config linux$ sed -e 's/.*CONFIG_BLK_DEV_LOOP is not set/CONFIG_BLK_DEV_LOOP=y/g' -i .config linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- oldconfig
Make sure to select Y to all Xen config options
linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image linux$ cd ..
Build Bootwrapper and device tree
host$ git://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git host$ cd boot-wrapper-aarch64 boot-wrapper-aarch64$ git branch xen-psci-support origin/xen-psci-support boot-wrapper-aarch64$ git checkout xen-psci-support boot-wrapper-aarch64$ ln -s <pathtoxen>/xen/xen/xen Xen boot-wrapper-aarch64$ ln -s <patchtolinux>/linux/arch/arm64/boot/Image Image
Clone the git repo and copy the DT files into bootwrapper directory
host$ git clone https://github.com/ARM-software/arm-trusted-firmware.git host$ cd arm-trusted-firmware arm-trusted-firmware$ cp ./fdts/fvp-base-gicv2-psci.dts ../boot-wrapper-aarch64/ arm-trusted-firmware$ cp ./fdts/rtsm_ve-motherboard.dtsi ../boot-wrapper-aarch64/ arm-trusted-firmware$ cd ../boot-wrapper-aarch64/ boot-wrapper-aarch64$ dtc -I dts -O dtb -o fvp-base-gicv2-psci.dtb fvp-base-gicv2-psci.dts
Note that everytime when you change the dts file; you need to re-compile the DT file using above command. Next configure and compile the boot-wrapper-aarch64
boot-wrapper-aarch64$ autoreconf -i boot-wrapper-aarch64$ ./configure --host=aarch64-linux-gnu --with-kernel-dir=../../linux --with-dtb=fvp-base-gicv2-psci.dtb --enable-psci --with-cmdline="earlyprintk=pl011,0x1c090000 console=hvc0 root=/dev/vda2 debug rw" --with-xen=Xen --with-xen-bootargs="console=dtuart dtuart=serial0 dom0_mem=512M dom0_max_vcpus=4 maxcpus=4 conswitch=x loglvl=all guest_loglvl=all no-bootscrub" boot-wrapper-aarch64$ make
Run the license and model. Below is a sample script to run the model.
model_bin=<PathTo>/FVP_Base_AEMv8A-AEMv8A/models/Linux64_GCC-4.1/FVP_Base_AEMv8A-AEMv8A
mmc=<PathTo>/vexpress64-leg-sd.img
bootwrapper_dir=<PathTo>/boot-wrapper-aarch64
app=$bootwrapper_dir/xen-system.axf
CORES=4
$model_bin \
-a "cluster0.*=$app" \
-C cache_state_modelled=0 \
-C bp.virtioblockdevice.image_path=$mmc \
-C bp.pl011_uart0.untimed_fifos=1 \
-C bp.secure_memory=0 \
-C pctl.startup=0.0.0.* \
-C cluster0.NUM_CORES=$CORES \
-C gicv3.gicv2-only=1 \
-C gic_distributor.wakeup-on-reset=1 \
-C gic_distributor.GICD-alias=0x2F000000 \
-C gic_distributor.IIDR=0x43b \
-C bp.refcounter.non_arch_start_at_default=1 \
$@Important note on SMP boot: When you are booting SMP on Xen you need to set as below. One could argue that Xen could be a little nicer here and time out if secondary CPUs never come up, but on the other hand, one shouldn't put CPUs in the DT that aren't there...!!!
-C pctl.startup=0.0.0.*
for booting cluster 0 only, and similarly
-C pctl.startup=0.0.*.*
for starting more than one clusters etc.
Download any latest filesystem from http://snapshots.linaro.org/kernel-hwpack/linux-leg-vexpress64/latest or http://snapshots.linaro.org/openembedded/pre-built/vexpress64/latest
You can download latest vanilla kernel, compile and copy the Image on the FS using kpartx
Xen hyp and DOM0 Boot logs on FVP base model
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000080000000 - 00000000feffffff
(XEN) RAM: 0000000880000000 - 00000008ffffffff
(XEN)
(XEN) MODULE[0]: 0000000088000000 - 00000000880025f4 Device Tree
(XEN) MODULE[1]: 0000000080080000 - 0000000080880000 Kernel
(XEN) RESVD[0]: 0000000080000000 - 0000000080010000
(XEN)
(XEN) Command line: console=dtuart dtuart=serial0 dom0_mem=512M dom0_max_vcpus=4 maxcpus=4 conswitch=x loglvl=all guest_loglvl=all no-bootscrub
(XEN) Placing Xen at 0x00000000fee00000-0x00000000ff000000
(XEN) Update BOOTMOD_XEN from 0000000080a00000-0000000080afad81 => 00000000fee00000-00000000feefad81
(XEN) WARNING: only using 1 out of 2 memory banks
(XEN) Domain heap initialised
(XEN) Looking for UART console serial0
Xen 4.5-unstable
(XEN) Xen version 4.5-unstable (nareshbhat@) (aarch64-linux-gnu-gcc (crosstool-NG linaro-1.13.1-4.8-2014.03 - Linaro GCC 2014.03) 4.8.3 20140303 (prerelease)) debug=y Thu Aug 28 12:13:33 IST 2014
(XEN) Latest ChangeSet: Mon Aug 11 15:13:04 2014 +0200 git:f093fcf
(XEN) Processor: 410fd0f0: "ARM Limited", variant: 0x0, part 0xd0f, rev 0x0
(XEN) 64-bit Execution:
(XEN) Processor Features: 0000000001002222 0000000000000000
(XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD
(XEN) Debug Features: 00000000f0f0f106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001122 0000000000000000
(XEN) ISA Features: 0000000000000000 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 00000131:10011011
(XEN) Instruction Sets: AArch32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer Security
(XEN) Debug Features: 03010066
(XEN) Auxiliary Features: 00000000
(XEN) Memory Model Features: 10101105 20000000 01260000 02102111
(XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00000001
(XEN) Platform: VERSATILE EXPRESS
(XEN) Using PSCI for SMP bringup
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27
(XEN) Using generic timer at 100000 KHz
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=000000002f000000
(XEN) gic_cpu_addr=000000002c000000
(XEN) gic_hyp_addr=000000002c010000
(XEN) gic_vcpu_addr=000000002c02f000
(XEN) gic_maintenance_irq=25
(XEN) GICv2: 256 lines, 8 cpus, secure (IID 0000043b).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) I/O virtualisation disabled
(XEN) Allocated console ring of 32 KiB.
(XEN) Bringing up CPU1
- CPU 00000001 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000002 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
- CPU 00000003 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 3 booted.
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000000080080000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x000000a0000000-0x000000c0000000 (512MB)
(XEN) Loading zImage from 0000000080080000 to 00000000a0080000-00000000a0880000
(XEN) Loading dom0 DTB to 0x00000000a8000000-0x00000000a80020de
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-x' three times to switch input to Xen)
(XEN) Freed 256kB init memory.
Initializing cgroup subsys cpu
Linux version 3.17.0-rc1+ (nareshbhat@Latitude-E6430) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.03 - Linaro GCC 2014.03) ) #1 SMP PREEMPT Thu Aug 21 13:59:14 IST 2014
CPU: AArch64 Processor [410fd0f0] revision 0
Detected PIPT I-cache on CPU0
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
cma: Reserved 16 MiB at bf000000
On node 0 totalpages: 131072
DMA zone: 1792 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 131072 pages, LIFO batch:31
psci: probing for conduit method from DT.
psci: Using PSCI v0.1 Function IDs from DT
PERCPU: Embedded 11 pages/cpu @ffffffc01efc0000 s13952 r8192 d22912 u45056
pcpu-alloc: s13952 r8192 d22912 u45056 alloc=11*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129280
Kernel command line: earlyprintk=pl011,0x1c090000 console=hvc0 root=/dev/vda2 debug rw
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 12288 bytes
log_buf_len min size: 16384 bytes
log_buf_len: 32768 bytes
early log buf free: 14944(91%)
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 485504K/524288K available (4580K kernel code, 310K rwdata, 1628K rodata, 225K init, 210K bss, 38784K reserved)
Virtual kernel memory layout:
vmalloc : 0xffffff8000000000 - 0xffffffbdffff0000 ( 247 GB)
vmemmap : 0xffffffbe00000000 - 0xffffffbfc0000000 ( 7 GB maximum)
0xffffffbe02300000 - 0xffffffbe02a00000 ( 7 MB actual)
PCI I/O : 0xffffffbffa000000 - 0xffffffbffb000000 ( 16 MB)
fixed : 0xffffffbffbdfe000 - 0xffffffbffbdff000 ( 4 KB)
modules : 0xffffffbffc000000 - 0xffffffc000000000 ( 64 MB)
memory : 0xffffffc000000000 - 0xffffffc020000000 ( 512 MB)
.init : 0xffffffc000692000 - 0xffffffc0006ca680 ( 226 KB)
.text : 0xffffffc000080000 - 0xffffffc000691114 ( 6213 KB)
.data : 0xffffffc0006cb000 - 0xffffffc0007189a0 ( 311 KB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Preemptible hierarchical RCU implementation.
RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS:64 nr_irqs:64 0
Architected cp15 and mmio timer(s) running at 100.00MHz (virt/phys).
sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 2748779069440ns
Console: colour dummy device 80x25
allocated 4194304 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=1000000)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
Initializing cgroup subsys memory
Initializing cgroup subsys hugetlb
hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 1 counters available
EFI services will not be available.
Xen 4.5 support found, events_irq=31 gnttab_frame=0x00000000b0000000
xen:grant_table: Grant tables using version 1 layout
Grant table initialized
xen:events: Using FIFO-based ABI
Xen: initializing cpu0
CPU1: Booted secondary processor
Detected PIPT I-cache on CPU1
Xen: initializing cpu1
CPU2: Booted secondary processor
Detected PIPT I-cache on CPU2
Xen: initializing cpu2
CPU3: Booted secondary processor
Detected PIPT I-cache on CPU3
Xen: initializing cpu3
Brought up 4 CPUs
SMP: Total of 4 processors activated.
devtmpfs: initialized
regulator-dummy: no parameters
NET: Registered protocol family 16
vdso: 2 pages (1 code @ ffffffc0006d1000, 1 data @ ffffffc0006d0000)
hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
software IO TLB [mem 0xbd000000-0xbd400000] (4MB) mapped at [ffffffc01d000000-ffffffc01d3fffff]
software IO TLB [mem 0xbcc00000-0xbd000000] (4MB) mapped at [ffffffc01cc00000-ffffffc01cffffff]
Serial: AMBA PL011 UART driver
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@3,00000000/sysctl@020000
1c0a0000.uart: ttyAMA1 at MMIO 0x1c0a0000 (irq = 38, base_baud = 0) is a PL011 rev2
1c0b0000.uart: ttyAMA2 at MMIO 0x1c0b0000 (irq = 39, base_baud = 0) is a PL011 rev2
1c0c0000.uart: ttyAMA3 at MMIO 0x1c0c0000 (irq = 40, base_baud = 0) is a PL011 rev2
xen:balloon: Initialising balloon driver
xen_balloon: Initialising balloon driver
3V3: 3300 mV
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switched to clocksource arch_sys_counter
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
kvm [1]: HYP mode not available
futex hash table entries: 1024 (order: 4, 65536 bytes)
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(0.470:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
fuse init (API version 7.23)
9p: Installing v9fs 9p2000 file system support
msgmni has been set to 980
io scheduler noop registered
io scheduler cfq registered (default)
xen:xen_evtchn: Event-channel device installed
console [hvc0] enabled
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
loop: module loaded
vda: vda1 vda2
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
smc91x: not found (-19).
xen_netfront: Initialising Xen virtual ethernet driver
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
mmci-pl18x 1c050000.mmci: Got CD GPIO #254.
mmci-pl18x 1c050000.mmci: Got WP GPIO #255.
mmci-pl18x 1c050000.mmci: No vqmmc regulator found
mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 41,42 (pio)
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
TCP: cubic registered
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
registered taskstats version 1
EXT3-fs (vda2): error: couldn't mount because of unsupported optional features (240)
EXT2-fs (vda2): error: couldn't mount because of unsupported optional features (240)
EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 254:2.
devtmpfs: mounted
Freeing unused kernel memory: 224K (ffffffc000692000 - ffffffc0006ca000)
INIT: version 2.88 booting
Mounting local filesystems...
Starting udev
udev: using cache file /etc/dev.tar
udevd[644]: starting version 182
Activating swap
EXT4-fs (vda2): re-mounted. Opts: (null)
Starting Bootlog daemon: bootlogd.
Initializing random number generator...
random: dd urandom read with 14 bits of entropy available
Populating volatile Filesystems.
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
System time was Thu Jan 1 00:00:03 UTC 1970.
Setting the System Clock using the Hardware Clock as reference...
hwclock: can't open '/dev/misc/rtc': No such file or directory
System Clock set. System local time is now Thu Jan 1 00:00:03 UTC 1970.
Sun Jul 27 13:22:09 UTC 2014
Saving the System Clock time to the Hardware Clock...
hwclock: can't open '/dev/misc/rtc': No such file or directory
Hardware Clock updated to Sun Jul 27 13:22:09 UTC 2014.
INIT: Entering runlevel: 5
Configuring network interfaces... ifconfig: SIOCGIFFLAGS: No such device
Starting system message bus: dbus.
Starting OpenBSD Secure Shell server: sshd
done.
Starting rpcbind daemon...rpcbind: cannot create socket for udp6
rpcbind: cannot create socket for tcp6
done.
creating NFS state directory: done
starting statd: done
System time was Sun Jul 27 13:22:10 UTC 2014.
Setting the System Clock using the Hardware Clock as reference...
hwclock: can't open '/dev/misc/rtc': No such file or directory
System Clock set. System local time is now Sun Jul 27 13:22:10 UTC 2014.
NFS daemon support not enabled in kernel
Starting syslogd/klogd: done
* Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ]
Starting php-fpm done
140727 13:22:12 mysqld_safe Logging to '/var/log/mysqld.err'.
Starting Gator: gatord... depmod: ERROR: could not open directory /lib/modules/3.17.0-rc1+: No such file or directory
depmod: FATAL: could not search modules: No such file or directory
done.
Starting web server: apache2140727 13:22:12 mysqld_safe Starting mysqld daemon with databases from /var/mysql
.
Starting auto-serial-console: done
Stopping Bootlog daemon:
bootlogd.
Last login: Sun Jul 27 13:22:13 UTC 2014 on tty1
INIT: no more processes left in this runlevel
root@genericarmv8:~# cat /proc/cpuinfo
features : fp asimd evtstrm
processor : 0
implementer : 0x41
variant : 0x0
partnum : 0xd0f
revision : 0x0
processor : 1
implementer : 0x41
variant : 0x0
partnum : 0xd0f
revision : 0x0
processor : 2
implementer : 0x41
variant : 0x0
partnum : 0xd0f
revision : 0x0
processor : 3
implementer : 0x41
variant : 0x0
partnum : 0xd0f
revision : 0x0
root@genericarmv8:~#
root@genericarmv8:~# halt
Broadcast message from root@genericarmv8 (hvc0) (Sun Jul 27 13:41:54 2014):
The system is going down for system halt NOW!
INIT: Switching to runlevel: 0
root@genericarmv8:~# Stopping OpenBSD Secure Shell server: sshdstopped /usr/sbin/sshd (pid 853)
.
* Stopping Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ]
.
Stopping system message bus: dbus.
Saving the System Clock time to the Hardware Clock...
hwclock: can't open '/dev/misc/rtc': No such file or directory
Hardware Clock updated to Sun Jul 27 13:41:55 UTC 2014.
stopping mountd: done
stopping nfsd: done
Stopping syslogd/klogd: stopped syslogd (pid 890)
stopped klogd (pid 894)
done
stopping statd: done
Gracefully shutting down php-fpm .140727 13:41:56 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
done
Stopping rpcbind daemon...
done.
Stopping Gator: gatord... stopped /usr/sbin/gatord (pid 992)
done.
Deconfiguring network interfaces... ifdown: interface eth0 not configured
done.
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed...
Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
EXT4-fs (vda2): re-mounted. Opts: (null)
reboot: Power down
(XEN) Domain 0 halted: halting machine.LEG/Engineering/Xen_boot_on_FVP (last modified 2014-08-28 07:28:53)