How to boot Xen on FVP(fast) model
This wiki page explains how to boot the Xen on FVP(fast) model. I have used the FVP_Base_AEMv8A-AEMv8A_0.8_5502 model. Note that all the FVP models are licensed models.
Build Xen
Clone the following repository and build the Xen hypervisor
host$ git clone git://xenbits.xen.org/xen.git host$ cd xen xen$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CONFIG_EARLY_PRINTK=fastmodel
Build Linux
host$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git host$ cd linux
Create a new kernel config:
linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig linux$ sed -e 's/.*CONFIG_XEN is not set/CONFIG_XEN=y/g' -i .config linux$ sed -e 's/.*CONFIG_BLK_DEV_LOOP is not set/CONFIG_BLK_DEV_LOOP=y/g' -i .config linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- oldconfig
Make sure to select Y to all Xen config options
linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image linux$ cd ..
Build Bootwrapper and device tree
host$ git://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git host$ cd boot-wrapper-aarch64 boot-wrapper-aarch64$ git branch xen-psci-support origin/xen-psci-support boot-wrapper-aarch64$ git checkout xen-psci-support boot-wrapper-aarch64$ ln -s <pathtoxen>/xen/xen/xen Xen boot-wrapper-aarch64$ ln -s <patchtolinux>/linux/arch/arm64/boot/Image Image
Clone the git repo and copy the DT files into bootwrapper directory
host$ git clone https://github.com/ARM-software/arm-trusted-firmware.git host$ cd arm-trusted-firmware arm-trusted-firmware$ cp ./fdts/fvp-base-gicv2-psci.dts ../boot-wrapper-aarch64/ arm-trusted-firmware$ cp ./fdts/rtsm_ve-motherboard.dtsi ../boot-wrapper-aarch64/ arm-trusted-firmware$ cd ../boot-wrapper-aarch64/ boot-wrapper-aarch64$ dtc -I dts -O dtb -o fvp-base-gicv2-psci.dtb fvp-base-gicv2-psci.dts
Note that everytime when you change the dts file; you need to re-compile the DT file using above command. Next configure and compile the boot-wrapper-aarch64
boot-wrapper-aarch64$ autoreconf -i boot-wrapper-aarch64$ ./configure --host=aarch64-linux-gnu --with-kernel-dir=../../linux --with-dtb=fvp-base-gicv2-psci.dtb --enable-psci --with-cmdline="earlyprintk=pl011,0x1c090000 console=hvc0 root=/dev/vda2 debug rw" --with-xen=Xen --with-xen-bootargs="console=dtuart dtuart=serial0 dom0_mem=512M dom0_max_vcpus=4 maxcpus=4 conswitch=x loglvl=all guest_loglvl=all no-bootscrub" boot-wrapper-aarch64$ make
Run the license and model. Below is a sample script to run the model.
model_bin=<PathTo>/FVP_Base_AEMv8A-AEMv8A/models/Linux64_GCC-4.1/FVP_Base_AEMv8A-AEMv8A mmc=<PathTo>/vexpress64-leg-sd.img bootwrapper_dir=<PathTo>/boot-wrapper-aarch64 app=$bootwrapper_dir/xen-system.axf CORES=4 $model_bin \ -a "cluster0.*=$app" \ -C cache_state_modelled=0 \ -C bp.virtioblockdevice.image_path=$mmc \ -C bp.pl011_uart0.untimed_fifos=1 \ -C bp.secure_memory=0 \ -C pctl.startup=0.0.0.* \ -C cluster0.NUM_CORES=$CORES \ -C gicv3.gicv2-only=1 \ -C gic_distributor.wakeup-on-reset=1 \ -C gic_distributor.GICD-alias=0x2F000000 \ -C gic_distributor.IIDR=0x43b \ -C bp.refcounter.non_arch_start_at_default=1 \ $@
Important note on SMP boot: When you are booting SMP on Xen you need to set as below. One could argue that Xen could be a little nicer here and time out if secondary CPUs never come up, but on the other hand, one shouldn't put CPUs in the DT that aren't there...!!!
-C pctl.startup=0.0.0.*
for booting cluster 0 only, and similarly
-C pctl.startup=0.0.*.*
for starting more than one clusters etc.
Download any latest filesystem from http://snapshots.linaro.org/kernel-hwpack/linux-leg-vexpress64/latest or http://snapshots.linaro.org/openembedded/pre-built/vexpress64/latest
You can download latest vanilla kernel, compile and copy the Image on the FS using kpartx
Xen hyp and DOM0 Boot logs on FVP base model
Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. - UART enabled - - CPU 00000000 booting - - Current EL 00000008 - - Xen starting at EL2 - - Zero BSS - - Setting up control registers - - Turning on paging - - Ready - (XEN) Checking for initrd in /chosen (XEN) RAM: 0000000080000000 - 00000000feffffff (XEN) RAM: 0000000880000000 - 00000008ffffffff (XEN) (XEN) MODULE[0]: 0000000088000000 - 00000000880025f4 Device Tree (XEN) MODULE[1]: 0000000080080000 - 0000000080880000 Kernel (XEN) RESVD[0]: 0000000080000000 - 0000000080010000 (XEN) (XEN) Command line: console=dtuart dtuart=serial0 dom0_mem=512M dom0_max_vcpus=4 maxcpus=4 conswitch=x loglvl=all guest_loglvl=all no-bootscrub (XEN) Placing Xen at 0x00000000fee00000-0x00000000ff000000 (XEN) Update BOOTMOD_XEN from 0000000080a00000-0000000080afad81 => 00000000fee00000-00000000feefad81 (XEN) WARNING: only using 1 out of 2 memory banks (XEN) Domain heap initialised (XEN) Looking for UART console serial0 Xen 4.5-unstable (XEN) Xen version 4.5-unstable (nareshbhat@) (aarch64-linux-gnu-gcc (crosstool-NG linaro-1.13.1-4.8-2014.03 - Linaro GCC 2014.03) 4.8.3 20140303 (prerelease)) debug=y Thu Aug 28 12:13:33 IST 2014 (XEN) Latest ChangeSet: Mon Aug 11 15:13:04 2014 +0200 git:f093fcf (XEN) Processor: 410fd0f0: "ARM Limited", variant: 0x0, part 0xd0f, rev 0x0 (XEN) 64-bit Execution: (XEN) Processor Features: 0000000001002222 0000000000000000 (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD (XEN) Debug Features: 00000000f0f0f106 0000000000000000 (XEN) Auxiliary Features: 0000000000000000 0000000000000000 (XEN) Memory Model Features: 0000000000001122 0000000000000000 (XEN) ISA Features: 0000000000000000 0000000000000000 (XEN) 32-bit Execution: (XEN) Processor Features: 00000131:10011011 (XEN) Instruction Sets: AArch32 Thumb Thumb-2 Jazelle (XEN) Extensions: GenericTimer Security (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: 00000000 (XEN) Memory Model Features: 10101105 20000000 01260000 02102111 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00000001 (XEN) Platform: VERSATILE EXPRESS (XEN) Using PSCI for SMP bringup (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 (XEN) Using generic timer at 100000 KHz (XEN) GICv2 initialization: (XEN) gic_dist_addr=000000002f000000 (XEN) gic_cpu_addr=000000002c000000 (XEN) gic_hyp_addr=000000002c010000 (XEN) gic_vcpu_addr=000000002c02f000 (XEN) gic_maintenance_irq=25 (XEN) GICv2: 256 lines, 8 cpus, secure (IID 0000043b). (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) I/O virtualisation disabled (XEN) Allocated console ring of 32 KiB. (XEN) Bringing up CPU1 - CPU 00000001 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) CPU 1 booted. (XEN) Bringing up CPU2 - CPU 00000002 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) CPU 2 booted. (XEN) Bringing up CPU3 - CPU 00000003 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) CPU 3 booted. (XEN) Brought up 4 CPUs (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0000000080080000 (XEN) Allocating 1:1 mappings totalling 512MB for dom0: (XEN) BANK[0] 0x000000a0000000-0x000000c0000000 (512MB) (XEN) Loading zImage from 0000000080080000 to 00000000a0080000-00000000a0880000 (XEN) Loading dom0 DTB to 0x00000000a8000000-0x00000000a80020de (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) *** Serial input -> DOM0 (type 'CTRL-x' three times to switch input to Xen) (XEN) Freed 256kB init memory. Initializing cgroup subsys cpu Linux version 3.17.0-rc1+ (nareshbhat@Latitude-E6430) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.03 - Linaro GCC 2014.03) ) #1 SMP PREEMPT Thu Aug 21 13:59:14 IST 2014 CPU: AArch64 Processor [410fd0f0] revision 0 Detected PIPT I-cache on CPU0 efi: Getting EFI parameters from FDT: efi: UEFI not found. cma: Reserved 16 MiB at bf000000 On node 0 totalpages: 131072 DMA zone: 1792 pages used for memmap DMA zone: 0 pages reserved DMA zone: 131072 pages, LIFO batch:31 psci: probing for conduit method from DT. psci: Using PSCI v0.1 Function IDs from DT PERCPU: Embedded 11 pages/cpu @ffffffc01efc0000 s13952 r8192 d22912 u45056 pcpu-alloc: s13952 r8192 d22912 u45056 alloc=11*4096 pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129280 Kernel command line: earlyprintk=pl011,0x1c090000 console=hvc0 root=/dev/vda2 debug rw log_buf_len individual max cpu contribution: 4096 bytes log_buf_len total cpu_extra contributions: 12288 bytes log_buf_len min size: 16384 bytes log_buf_len: 32768 bytes early log buf free: 14944(91%) PID hash table entries: 2048 (order: 2, 16384 bytes) Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) Inode-cache hash table entries: 32768 (order: 6, 262144 bytes) Memory: 485504K/524288K available (4580K kernel code, 310K rwdata, 1628K rodata, 225K init, 210K bss, 38784K reserved) Virtual kernel memory layout: vmalloc : 0xffffff8000000000 - 0xffffffbdffff0000 ( 247 GB) vmemmap : 0xffffffbe00000000 - 0xffffffbfc0000000 ( 7 GB maximum) 0xffffffbe02300000 - 0xffffffbe02a00000 ( 7 MB actual) PCI I/O : 0xffffffbffa000000 - 0xffffffbffb000000 ( 16 MB) fixed : 0xffffffbffbdfe000 - 0xffffffbffbdff000 ( 4 KB) modules : 0xffffffbffc000000 - 0xffffffc000000000 ( 64 MB) memory : 0xffffffc000000000 - 0xffffffc020000000 ( 512 MB) .init : 0xffffffc000692000 - 0xffffffc0006ca680 ( 226 KB) .text : 0xffffffc000080000 - 0xffffffc000691114 ( 6213 KB) .data : 0xffffffc0006cb000 - 0xffffffc0007189a0 ( 311 KB) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 Preemptible hierarchical RCU implementation. RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4. RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 NR_IRQS:64 nr_irqs:64 0 Architected cp15 and mmio timer(s) running at 100.00MHz (virt/phys). sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 2748779069440ns Console: colour dummy device 80x25 allocated 4194304 bytes of page_cgroup please try 'cgroup_disable=memory' option if you don't want memory cgroups Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=1000000) pid_max: default: 32768 minimum: 301 Security Framework initialized Mount-cache hash table entries: 1024 (order: 1, 8192 bytes) Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes) Initializing cgroup subsys memory Initializing cgroup subsys hugetlb hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 1 counters available EFI services will not be available. Xen 4.5 support found, events_irq=31 gnttab_frame=0x00000000b0000000 xen:grant_table: Grant tables using version 1 layout Grant table initialized xen:events: Using FIFO-based ABI Xen: initializing cpu0 CPU1: Booted secondary processor Detected PIPT I-cache on CPU1 Xen: initializing cpu1 CPU2: Booted secondary processor Detected PIPT I-cache on CPU2 Xen: initializing cpu2 CPU3: Booted secondary processor Detected PIPT I-cache on CPU3 Xen: initializing cpu3 Brought up 4 CPUs SMP: Total of 4 processors activated. devtmpfs: initialized regulator-dummy: no parameters NET: Registered protocol family 16 vdso: 2 pages (1 code @ ffffffc0006d1000, 1 data @ ffffffc0006d0000) hw-breakpoint: found 16 breakpoint and 16 watchpoint registers. software IO TLB [mem 0xbd000000-0xbd400000] (4MB) mapped at [ffffffc01d000000-ffffffc01d3fffff] software IO TLB [mem 0xbcc00000-0xbd000000] (4MB) mapped at [ffffffc01cc00000-ffffffc01cffffff] Serial: AMBA PL011 UART driver of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@3,00000000/sysctl@020000 1c0a0000.uart: ttyAMA1 at MMIO 0x1c0a0000 (irq = 38, base_baud = 0) is a PL011 rev2 1c0b0000.uart: ttyAMA2 at MMIO 0x1c0b0000 (irq = 39, base_baud = 0) is a PL011 rev2 1c0c0000.uart: ttyAMA3 at MMIO 0x1c0c0000 (irq = 40, base_baud = 0) is a PL011 rev2 xen:balloon: Initialising balloon driver xen_balloon: Initialising balloon driver 3V3: 3300 mV SCSI subsystem initialized libata version 3.00 loaded. usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switched to clocksource arch_sys_counter NET: Registered protocol family 2 TCP established hash table entries: 4096 (order: 3, 32768 bytes) TCP bind hash table entries: 4096 (order: 4, 65536 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP: reno registered UDP hash table entries: 256 (order: 1, 8192 bytes) UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) NET: Registered protocol family 1 RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. kvm [1]: HYP mode not available futex hash table entries: 1024 (order: 4, 65536 bytes) audit: initializing netlink subsys (disabled) audit: type=2000 audit(0.470:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages fuse init (API version 7.23) 9p: Installing v9fs 9p2000 file system support msgmni has been set to 980 io scheduler noop registered io scheduler cfq registered (default) xen:xen_evtchn: Event-channel device installed console [hvc0] enabled Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled loop: module loaded vda: vda1 vda2 tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> smc91x: not found (-19). xen_netfront: Initialising Xen virtual ethernet driver usbcore: registered new interface driver usb-storage mousedev: PS/2 mouse device common for all mice mmci-pl18x 1c050000.mmci: Got CD GPIO #254. mmci-pl18x 1c050000.mmci: Got WP GPIO #255. mmci-pl18x 1c050000.mmci: No vqmmc regulator found mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 41,42 (pio) usbcore: registered new interface driver usbhid usbhid: USB HID core driver TCP: cubic registered NET: Registered protocol family 17 9pnet: Installing 9P2000 support registered taskstats version 1 EXT3-fs (vda2): error: couldn't mount because of unsupported optional features (240) EXT2-fs (vda2): error: couldn't mount because of unsupported optional features (240) EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null) VFS: Mounted root (ext4 filesystem) on device 254:2. devtmpfs: mounted Freeing unused kernel memory: 224K (ffffffc000692000 - ffffffc0006ca000) INIT: version 2.88 booting Mounting local filesystems... Starting udev udev: using cache file /etc/dev.tar udevd[644]: starting version 182 Activating swap EXT4-fs (vda2): re-mounted. Opts: (null) Starting Bootlog daemon: bootlogd. Initializing random number generator... random: dd urandom read with 14 bits of entropy available Populating volatile Filesystems. net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 System time was Thu Jan 1 00:00:03 UTC 1970. Setting the System Clock using the Hardware Clock as reference... hwclock: can't open '/dev/misc/rtc': No such file or directory System Clock set. System local time is now Thu Jan 1 00:00:03 UTC 1970. Sun Jul 27 13:22:09 UTC 2014 Saving the System Clock time to the Hardware Clock... hwclock: can't open '/dev/misc/rtc': No such file or directory Hardware Clock updated to Sun Jul 27 13:22:09 UTC 2014. INIT: Entering runlevel: 5 Configuring network interfaces... ifconfig: SIOCGIFFLAGS: No such device Starting system message bus: dbus. Starting OpenBSD Secure Shell server: sshd done. Starting rpcbind daemon...rpcbind: cannot create socket for udp6 rpcbind: cannot create socket for tcp6 done. creating NFS state directory: done starting statd: done System time was Sun Jul 27 13:22:10 UTC 2014. Setting the System Clock using the Hardware Clock as reference... hwclock: can't open '/dev/misc/rtc': No such file or directory System Clock set. System local time is now Sun Jul 27 13:22:10 UTC 2014. NFS daemon support not enabled in kernel Starting syslogd/klogd: done * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ] Starting php-fpm done 140727 13:22:12 mysqld_safe Logging to '/var/log/mysqld.err'. Starting Gator: gatord... depmod: ERROR: could not open directory /lib/modules/3.17.0-rc1+: No such file or directory depmod: FATAL: could not search modules: No such file or directory done. Starting web server: apache2140727 13:22:12 mysqld_safe Starting mysqld daemon with databases from /var/mysql . Starting auto-serial-console: done Stopping Bootlog daemon: bootlogd. Last login: Sun Jul 27 13:22:13 UTC 2014 on tty1 INIT: no more processes left in this runlevel root@genericarmv8:~# cat /proc/cpuinfo features : fp asimd evtstrm processor : 0 implementer : 0x41 variant : 0x0 partnum : 0xd0f revision : 0x0 processor : 1 implementer : 0x41 variant : 0x0 partnum : 0xd0f revision : 0x0 processor : 2 implementer : 0x41 variant : 0x0 partnum : 0xd0f revision : 0x0 processor : 3 implementer : 0x41 variant : 0x0 partnum : 0xd0f revision : 0x0 root@genericarmv8:~# root@genericarmv8:~# halt Broadcast message from root@genericarmv8 (hvc0) (Sun Jul 27 13:41:54 2014): The system is going down for system halt NOW! INIT: Switching to runlevel: 0 root@genericarmv8:~# Stopping OpenBSD Secure Shell server: sshdstopped /usr/sbin/sshd (pid 853) . * Stopping Avahi mDNS/DNS-SD Daemon: avahi-daemon [ ok ] . Stopping system message bus: dbus. Saving the System Clock time to the Hardware Clock... hwclock: can't open '/dev/misc/rtc': No such file or directory Hardware Clock updated to Sun Jul 27 13:41:55 UTC 2014. stopping mountd: done stopping nfsd: done Stopping syslogd/klogd: stopped syslogd (pid 890) stopped klogd (pid 894) done stopping statd: done Gracefully shutting down php-fpm .140727 13:41:56 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended done Stopping rpcbind daemon... done. Stopping Gator: gatord... stopped /usr/sbin/gatord (pid 992) done. Deconfiguring network interfaces... ifdown: interface eth0 not configured done. Sending all processes the TERM signal... Sending all processes the KILL signal... Saving random seed... Unmounting remote filesystems... Deactivating swap... Unmounting local filesystems... EXT4-fs (vda2): re-mounted. Opts: (null) reboot: Power down (XEN) Domain 0 halted: halting machine.
LEG/Engineering/Xen_boot_on_FVP (last modified 2014-08-28 07:28:53)