ninjaARM-cortex-chip

{i} This page is under construction

How to boot Xen with GRUB on Foundation_v8 model

The main purpose of this Wiki page is describing

  • How to build some essential binaries for Foundation_v8
  • How to boot Xen/Dom0 linux kernel (with initramfs/XSM) by GRUB2 on Foundation_v8
  • Where are all the pre-build binaries/Image/files for GRUB2 multiboot Xen on Foundation_v8

Boot Foundation_v8 model with LEG Image and Firmware/UEFI

For recompiling all the binaries below natively, you may need to boot Foundation_v8 model with LEG Image, please following the steps here : How to boot Foundation_v8 model with LEG Image and Firmware/UEFI

Or you can use other AArch64 platform instead.

Option : Add XSM and Initrd into the original LEG Image manually

If you want to test XSM and Initrd at the same time, you can add XSM and Initrd into the original LEG Image manually.

The modification steps for boot partition:

tekkamanninja@Super-MAGI:~/images$ sudo kpartx -av vexpress64-leg-sd.img 
[sudo] password for tekkamanninja: 
add map loop0p1 (252:0): 0 139200 linear /dev/loop0 63
add map loop0p2 (252:1): 0 4055040 linear /dev/loop0 139264
tekkamanninja@Super-MAGI:~/images$ sudo mount /dev/mapper/loop0p1  boot/
tekkamanninja@Super-MAGI:~/images$ cp xenpolicy.24 boot/xsm
tekkamanninja@Super-MAGI:~/images$ cp initramfs_leg_armv8.cpio.gz boot/initramfs_leg_armv8.cpio.gz
tekkamanninja@Super-MAGI:~/images$ sudo umount /dev/mapper/loop0p1 
tekkamanninja@Super-MAGI:~/images$ sync
tekkamanninja@Super-MAGI:~/images$ sudo kpartx -d vexpress64-leg-sd.img 
loop deleted : /dev/loop0

What is XSM

Xen offers a security framework called Xen Security Modules, or XSM, to enable an administrator or developer to exert fine-grained control over a Xen domain and its capabilities. Specifically, XSM makes it possible to define permissible interactions between domains, the hypervisor itself, and related resources such as memory and devices.

For more info, please read this wiki page


All the operations below are on an AArch64 platform

Bootloader--GRUB2

UEFI

Get GRUB2 source code(with multiboot support)

root@genericarmv8:~# git clone http://git.linaro.org/people/fu.wei/grub.git
root@genericarmv8:~# cd grub
root@genericarmv8:~/grub# git checkout origin/multiboot_xen_support_upstream

Build

In GRUB2 source code dir :

root@genericarmv8:~/grub# ./autogen.sh
root@genericarmv8:~/grub# ./configure --with-platform=efi --prefix=/usr
root@genericarmv8:~/grub# make

Install grub utilities

root@genericarmv8:~/grub# make install

Install grub into boot partition

root@genericarmv8:~/grub# mkdir -p /boot/efi
root@genericarmv8:~/grub# mount /dev/vda1 /boot/efi
root@genericarmv8:~/grub# grub-install
root@genericarmv8:~/grub# umount /dev/vda1
root@genericarmv8:~/grub# sync

Xen

Xen-Virtual-Servers

We can download the pre-build Xen binary (here) or you can build the binary by following the steps below

Get upstream Xen source code

root@genericarmv8:~/grub# cd ~
root@genericarmv8:~# git clone git://xenbits.xen.org/xen.git

Build

root@genericarmv8:~# cd xen
root@genericarmv8:~/xen# make XSM_ENABLE=y xen debug=y

{i} If you don't want to boot with XSM, you can delete "XSM_ENABLE=y" in the command line.

{i} If you don't need debug info, you can delete "debug=y" in the command line.

Install xen binary into boot partition

root@genericarmv8:~/xen# mount /dev/vda1 /boot
root@genericarmv8:~/xen# cp xen/xen /boot/
root@genericarmv8:~/xen# umount /dev/vda1
root@genericarmv8:~/xen# sync

Dom0 Linux kernel

Tux

We can download the pre-build Dom0 Linux kernel Image (here) or you can build the Image by following the steps below

Get upstream Linux source code

root@genericarmv8:~/xen# cd ~
root@genericarmv8:~# git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Build the Image

In Linux source code dir :

root@genericarmv8:~# cd linux
root@genericarmv8:~/linux# make defconfig
root@genericarmv8:~/linux# make menuconfig

Kernel Features --->

  • [*] Xen guest support on ARM64

More detail of configuration for DOM0 linux kernel. please check XEN Wiki for Mainline_Linux_Kernel_Configs

root@genericarmv8:~/linux# make Image

Install DOM0 linux Image into boot partition

root@genericarmv8:~/linux# mount /dev/vda1 /boot
root@genericarmv8:~/linux# cp arch/arm64/boot/Image /boot/Image_xen_dom0
root@genericarmv8:~/linux# umount /dev/vda1
root@genericarmv8:~/linux# sync

'Now we have installed all the components we need. Then we can reboot the model, and test booting Xen by GRUB'

Foundation_v8 reboot

After 'reboot' command, because we run a 'grub-install' in Linux, you can see a new boot entry in UEFI shell '[1] grub'

{i} If you kill the "Foundation_v8" model's process, you need to #Install grub into boot partition on "Foundation_v8" again, then 'reboot'

root@genericarmv8:~/linux# reboot

.......

Unmounting remote filesystems...
Deactivating swap...
Unmounting local filesystems...
[ 4481.308907] EXT4-fs (vda2): re-mounted. Opts: (null)
Rebooting... 
[ 4483.508727] reboot: Restarting system
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.0(release):14b6608
NOTICE:  BL1: Built : 17:07:57, Dec  3 2014
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v1.0(release):14b6608
NOTICE:  BL2: Built : 17:07:58, Dec  3 2014
NOTICE:  BL1: Booting BL3-1
NOTICE:  BL3-1: v1.0(release):14b6608
NOTICE:  BL3-1: Built : 17:08:00, Dec  3 2014
UEFI firmware (version linaro-edk2-2014.12-prep built at 17:07:43 on Dec  3 2014)
The default boot selection will start in  10 seconds
[1] grub
[2] Linaro disk image on virtio
[3] Shell
[4] Boot Manager

Start: 1

Welcome to GRUB!


                          GNU GRUB  version 2.02~beta2

   Minimal BASH-like line editing is supported. For the first word, TAB   
   lists possible command completions. Anywhere else TAB lists possible   
   device or file completions.                                                                         

Boot Xen using GRUB multiboot support

grub> set root=(hd1,msdos1)

grub> xen_hypervisor /xen no-bootscrub loglvl=all guest_loglvl=all conswitch=x dtuart=serial0 dom0_mem=512M

If you have added XSM and Initrd into the original LEG Image

If you did NOT add XSM and Initrd into the original LEG Image

grub> xen_module /Image_xen_dom0 console=hvc0 root=/dev/ram0 debug ro

grub> xen_module /Image_xen_dom0 console=hvc0 root=/dev/vda2 rw

grub> xen_module --type "xen,xsm-policy" --type "multiboot,module" /xsm

grub> xen_module /initramfs_leg_armv8.cpio.gz

grub> boot

Xen 4.5.0-rc (c/s Mon Dec 8 14:45:46 2014 +0100 git:2a549b9) EFI loader
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000080000000 - 0000000087ffffff
(XEN) RAM: 000000008c000000 - 00000000febe2fff
(XEN) RAM: 00000000febeb000 - 00000000feffffff
(XEN) RAM: 0000000880000000 - 00000008bb27efff

...

(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-x' three times to switch input to Xen)
(XEN) Freed 264kB init memory.
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpu
Linux version 3.18.0+ (wefu@mustang-02.farm.hsv.redhat.com) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #1 SMP PREEMPT Fri Dec 12 09:54:40 EST 2014
CPU: AArch64 Processor [410fd000] revision 0
Detected PIPT I-cache on CPU0
efi: Getting EFI parameters from FDT:
efi: UEFI not found.

...

TCP: cubic registered
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
registered taskstats version 1
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Freeing unused kernel memory: 256K (ffffffc000847000 - ffffffc000887000)
Freeing alternatives memory: 8K (ffffffc000887000 - ffffffc000889000)
+ for arg in '$CMDLINE'
++ expr xro : 'x[^=]*=\(.*\)'
+ optarg=
+ case $arg in
+ setsid sh
linaro-test [rc=0]#



Reference: Cross-compiling GRUB2/XEN/DOM0_Linux_kernel

Bootloader--GRUB2

Get GRUB2 source code(with multiboot support)

git clone http://git.linaro.org/people/fu.wei/grub.git
cd grub
git checkout origin/multiboot_xen_support_upstream

Build

In GRUB2 source code dir :

CROSS_TOOL_DIR_ARMv8=<the-path-of-toolchain-dir>
CROSS_COMPILE_ARMv8=aarch64-linux-gnu-

GRUB_INSTALL_DIR_V8=./install/grub-efi_installed_v8
GRUB_BUILD_CONFIG_V8=--target=aarch64-linux-gnu --with-platform=efi --prefix=${GRUB_INSTALL_DIR_V8}
GRUB_BIN_NAME=grub_v8.efi

export PATH=${PATH}:${CROSS_TOOL_DIR_V8}
export CROSS_COMPILE=${CROSS_COMPILE_V8}

rm -r ${GRUB_INSTALL_DIR_V8}

./autogen.sh
./configure ${GRUB_BUILD_CONFIG_V8}
make
make install

cd ${GRUB_INSTALL_DIR_V8}
./bin/grub-mkstandalone -v -o ${GRUB_BIN_NAME} -O arm64-efi

The Components

Location

GRUB

${GRUB_INSTALL_DIR_V8}/${GRUB_BIN_NAME}

Xen

Get upstream Xen source code

git clone git://xenbits.xen.org/xen.git
cd xen
git checkout master

Build

CROSS_TOOL_DIR_ARMv8=<the-path-of-toolchain-dir>
CROSS_COMPILE_ARMv8=aarch64-linux-gnu-
export PATH=${PATH}:${CROSS_TOOL_DIR_V8}

make XEN_TARGET_ARCH=arm64 XSM_ENABLE=y CROSS_COMPILE=${CROSS_COMPILE_V8} xen debug=y

The Components

Location

XEN

xen/xen

Dom0 Linux kernel

Get upstream Linux source code

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout master

Build the Image

In Linux source code dir :

CROSS_TOOL_DIR_ARMv8=<the-path-of-toolchain-dir>
CROSS_COMPILE_ARMv8=aarch64-linux-gnu-

export PATH=${PATH}:${CROSS_TOOL_DIR_V8}
export CROSS_COMPILE=${CROSS_COMPILE_V8}

export ARCH=arm64

make defconfig
make menuconfig

Kernel Features --->

  • [*] Xen guest support on ARM64

make Image

The Components

Location

DOM0 Linux kernel Image

arch/arm64/boot/Image

Modify the original LEG Image

Now we have got all the components we need, so we can modify the SD image and boot the model.

The modification steps for boot partition:

tekkamanninja@Super-MAGI:~/images$ sudo kpartx -av vexpress64-leg-sd.img 
[sudo] password for tekkamanninja: 
add map loop0p1 (252:0): 0 139200 linear /dev/loop0 63
add map loop0p2 (252:1): 0 4055040 linear /dev/loop0 139264
tekkamanninja@Super-MAGI:~/images$ sudo mount /dev/mapper/loop0p1  boot/

#create EFI/BOOT dir in boot partition for grub_v8.efi binary
tekkamanninja@Super-MAGI:~/images$ sudo mkdir -p boot/EFI/BOOT

#copy grub_v8.efi binary
tekkamanninja@Super-MAGI:~/images$ sudo cp grub_v8.efi boot/EFI/BOOT/grub.efi

#copy xen binary to boot partition
tekkamanninja@Super-MAGI:~/images$ sudo cp xen boot/

#copy Dom0 kernel image file to boot partition
tekkamanninja@Super-MAGI:~/images$ sudo cp Image boot/Image_xen_dom0

Foundation_v8 boot log

$ ./Foundation_v8.sh

terminal_0: Listening for serial connection on port 5000
terminal_1: Listening for serial connection on port 5001
terminal_2: Listening for serial connection on port 5002
terminal_3: Listening for serial connection on port 5003
Simulation is started
xterm: Xt error: Can't open display: 
xterm: DISPLAY is not set

$ telnet localhost 5000

Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Secure firmware (version  built at 17:50:35 on Nov  3 2014)
PROGRESS CODE: V03020003 I0
PROGRESS CODE: V03020002 I0
PROGRESS CODE: V03020003 I0
PROGRESS CODE: V03021001 I0

The default boot selection will start in   7 seconds
[1] Linux from SemiHosting
[2] Shell
[3] Boot Manager

Start: 2

UEFI Interactive Shell v2.0
EDK II
UEFI v2.40 (ARM Fixed Virtual Platform EFI Nov  5 2014 16:55:22, 0x00000000)
Mapping table
      FS2: Alias(s):F8:
          VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
      FS0: Alias(s):F3:
          MemoryMapped(0xB,0x88000000,0x8827FFFF)
      FS1: Alias(s):F4:
          MemoryMapped(0xB,0xFEC10000,0xFEDF6E3F)
      FS3: Alias(s):HD12b:;BLK5:
          VenHw(FE61BB5F-1B67-4C24-B346-73DB42E873E5)/HD(1,MBR,0x00000000,0x3F,0x21FC0)
     BLK3: Alias(s):
          VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
     BLK0: Alias(s):
          VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
     BLK1: Alias(s):
          VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
     BLK2: Alias(s):
          VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
     BLK4: Alias(s):
          VenHw(FE61BB5F-1B67-4C24-B346-73DB42E873E5)
     BLK6: Alias(s):
          VenHw(FE61BB5F-1B67-4C24-B346-73DB42E873E5)/HD(2,MBR,0x00000000,0x22000,0x3DE000)
Shell> grub

                          GNU GRUB  version 2.02~beta2

   Minimal BASH-like line editing is supported. For the first word, TAB   
   lists possible command completions. Anywhere else TAB lists possible   
   device or file completions.                                            

The rest of part is the same with #Boot Xen using GRUB multiboot support

{i} For 'Debug' purpose, you can add "set debug=xen_boot_loader,linux" into grub,cfg(or use that directly in grub shell), you will get debug info when you run any xen_*/linux command.

loader/arm64/xen_boot.c:550: Unknown option --nounzip, skip.
loader/arm64/linux.c:122: UEFI stub kernel:
loader/arm64/linux.c:124: text_offset = 0x000000000000
loader/arm64/linux.c:125: PE/COFF header @ 00000040
loader/arm64/xen_boot.c:408: Xen_boot xen_hypervisor file size: 0xc8798
loader/arm64/xen_boot.c:418: Xen_boot xen_hypervisor numpages: 0xca
loader/arm64/xen_boot.c:445: Xen_boot xen_hypervisor cmdline @ 0x8fa60ece0 --
no-bootscrub console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M
dom0_max_vcpus=2, size: 89
loader/arm64/xen_boot.c:98: Unknown option --fake, skip.
loader/arm64/xen_boot.c:510: Init multiboot,kernel module and node info:
compatible multiboot,kernel
compat_string_size 0x22
loader/arm64/xen_boot.c:408: Xen_boot multiboot,kernel file size: 0x6becc8
loader/arm64/xen_boot.c:418: Xen_boot multiboot,kernel numpages: 0x6bf
loader/arm64/xen_boot.c:445: Xen_boot multiboot,kernel cmdline @ 0x8fa5369c0
console=hvc0 root=/dev/ram0 ro, size: 31
loader/arm64/xen_boot.c:98: Unknown option --fake, skip.
loader/arm64/xen_boot.c:510: Init xen,xsm-policy module and node info:
compatible xen,xsm-policy
compat_string_size 0x20
loader/arm64/xen_boot.c:408: Xen_boot xen,xsm-policy file size: 0x25e7
loader/arm64/xen_boot.c:418: Xen_boot xen,xsm-policy numpages: 0x3
loader/arm64/xen_boot.c:510: Init multiboot,ramdisk module and node info:
compatible multiboot,ramdisk
compat_string_size 0x23
loader/arm64/xen_boot.c:408: Xen_boot multiboot,ramdisk file size: 0xb9de00
loader/arm64/xen_boot.c:418: Xen_boot multiboot,ramdisk numpages: 0xb9e
loader/arm64/linux.c:95: allocating 10101 bytes for fdt
loader/arm64/xen_boot.c:177: Xen Hypervisor cmdline : -- no-bootscrub
console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2 @
0x8fa60ece0 size:89
loader/arm64/xen_boot.c:310: Module multiboot,ramdisk @ 0x8bbd56000
size:0xb9de00
loader/arm64/xen_boot.c:197: Module node name module@8bbd56000 
loader/arm64/xen_boot.c:223: Module multiboot,ramdisk compatible =
multiboot,ramdisk size = 0x23
loader/arm64/xen_boot.c:246: Module multiboot,ramdisk has not bootargs!
loader/arm64/xen_boot.c:310: Module xen,xsm-policy @ 0x8faba2000 size:0x25e7
loader/arm64/xen_boot.c:197: Module node name module@8faba2000 
loader/arm64/xen_boot.c:223: Module xen,xsm-policy compatible = xen,xsm-policy
size = 0x20
loader/arm64/xen_boot.c:246: Module xen,xsm-policy has not bootargs!
loader/arm64/xen_boot.c:310: Module multiboot,kernel @ 0x8bc8f4000
size:0x6becc8
loader/arm64/xen_boot.c:197: Module node name module@8bc8f4000 
loader/arm64/xen_boot.c:223: Module multiboot,kernel compatible =
multiboot,kernel size = 0x22
loader/arm64/xen_boot.c:236: Module multiboot,kernel cmdline : console=hvc0
root=/dev/ram0 ro @ 0x8fa5369c0 size:31
loader/arm64/xen_boot.c:266: Installed/updated FDT configuration table @
0x8fab9f000
loader/arm64/linux.c:272: linux command line: '-- no-bootscrub console=dtuart
conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2'
loader/arm64/linux.c:288: starting image 0x8fc560f98

Some rootfs problem's solution

all the components

Classification

The Components

Pre-build Download

Git repo

Emulator

Fixed Virtual Platforms:Foundation_v8 Model

Foundation_v8 Model(need Register and Login)

--

SD Image

Linaro LEG image

The latest Linaro LEG Image

--

Toolchain

Linaro ToolChain
(Optional, if you want to recompile any component)

The latest Linaro ToolChain tarball
gcc-linaro-*-*-x86_64_aarch64-linux-gnu.tar.xz

Linaro GCC

Bootloader

BL1 (ARM Trusted Firmware)

Pre-build binaries
For Foundation_v8 model, they are in release/fvp_minimal

ARM Trusted Firmware

FIP (ARM Trusted Firmware + UEFI)

Linaro-EDK2
uefi-tools

GRUB (multiboot support)

Pre-build binary and files

GRUB(with multiboot support)

Hypervisor

Xen

XEN

XSM(Optional)

--

Dom0

Linux Kernel

Linux(vanilla tree)

Initrd(Optional)

--

LEG/Engineering/Grub2/Xen_booting_on_Foundation_FVP_model_by_GRUB (last modified 2015-07-13 09:26:49)