Xen ARM64 on APM XGENE Mustang


A. Setting up U-Boot environment

  • The current approach requires tftp to get the images.
  • Basic IP address settings :-
    # setenv ipaddr 10.y.y.y

    • This is an IP address of your board.
    # setenv serverip 10.x.x.x
    • This is an IP address of your tftp server.
    # setenv ethaddr 00:zz:zz:zz:zz:zz
    • This is the MAC of your Ethernet.

    # setenv user_dir <TFTP DIRECTORY FOR GETTING XEN IMAGES>

    • e.g. If your images are inside /tftpboot/xen then "setenv user_dir xen"
    # setenv xen_addr_r 0x4000080000
    • This is the address where Xen kernel will be copied using tftp.
    # setenv gatewayip 10.x.x.x # setenv netmask 255.x.x.x # ping $serverip
    • This should work before proceeding further.

    #setenv kern_addr_r 0x4002000000
    • This is the address where DOM0 kernel will be copied using tftp.
    # setenv fdt_addr_r 0x4003000000
    • This is the address where DOM0 dtb will be copied using tftp.

    # saveenv

  • Setting up Xen booting environment :-

    # setenv xen_load 'tftp ${xen_addr_r} ${user_dir}/uXen'

    # setenv xen_dtb 'tftp ${fdt_addr_r} ${user_dir}/apm-mustang-xen.dtb;fdt addr ${fdt_addr_r};fdt resize;'

    # setenv xen_boot 'setenv bootargs ${bootargs_xen};bootm ${xen_addr_r} - ${fdt_addr_r}'

    # setenv dom0_node 'fdt set /chosen/module@0 compatible xen,linux-zimage xen,multiboot-module'

    # setenv xen_dom0 'dom0=tftp ${kern_addr_r} ${user_dir}/Image; run dom0_node;fdt set /chosen/module@0 reg < 0x40 ${kern_addr_r} 0x${filesize} >'

    # setenv xen_run 'run xen_dtb xen_dom0 xen_load;fdt print /chosen;run xen_boot'

    # setenv bootargs_xen 'conswitch=x sync_console dom0_mem=1G dom0_max_vcpus=1 console=dtuart dtuart=/soc/serial@1c020000'
    # saveenv

B. Preparing all the Images

  • Building Xen Kernel :-

  • Pull latest xen code from branch http://xenbits.xen.org/gitweb/?p=xen.git;a=summary
    b. #make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

    • TIP: To get early prints

    • #make dist-xen XEN_TARGET_ARCH=arm64 debug=y CONFIG_EARLY_PRINTK=xgene-storm CROSS_COMPILE=aarch64-linux-gnu-

    c. #mkimage -A arm -C none -T kernel -a 0x0200000 -e 0x00200000 -n Xen -d xen/xen uXen
    d. Copy "uXen" to your tftp directory.

  • Building DOM0 :-
    For DOM0 rootfs we need SATA support to be enabled in DOM0 kernel.

    Till today (17/02/2014) SATA, CLOCK and PCIe Linux driver patches for Mustang are not yet merged and are under review. Hence please use Ian Campbell's linux tree which has un-merged SATA CLOCK and PCIe drivers for mustang. Once above mentioned drivers are made it to mainline then DOM0 can be build directly from mainline Linux.

Thanks to Ian for having this tree!! -

  • DOM0 Linux Tree:
    • git://xenbits.xen.org/people/ianc/linux.git
      Branch: for-apm/mustang-v3.12-v3

  • Compiling DOM0 Kernel :

    1. #make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
      b. #make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
      c. #make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image

  • This will build DOM0 and DomU kernel.
    • Copy arch/arm64/boot/Image to your tftp directory.

  • NOTE: Please make sure you have following CONFIG options selected to "y" while compiling DOM0 kernel.
    • CONFIG_XEN=y
      CONFIG_XEN_DOM0=y
      CONFIG_XEN_BLKDEV_FRONTEND=y
      CONFIG_XEN_BLKDEV_BACKEND=y
      CONFIG_XEN_NETDEV_FRONTEND=y
      CONFIG_XEN_NETDEV_BACKEND=y
      CONFIG_HVC_DRIVER=y
      CONFIG_HVC_IRQ=y
      CONFIG_HVC_XEN=y
      CONFIG_HVC_XEN_FRONTEND=y
      CONFIG_XEN_BALLOON=y
      CONFIG_XEN_SCRUB_PAGES=y
      CONFIG_XEN_DEV_EVTCHN=y
      CONFIG_XEN_BACKEND=y
      CONFIG_XENFS=y
      CONFIG_XEN_COMPAT_XENFS=y
      CONFIG_XEN_SYS_HYPERVISOR=y
      CONFIG_XEN_XENBUS_FRONTEND=y
      CONFIG_XEN_GNTDEV=y
      CONFIG_SWIOTLB_XEN=y
      CONFIG_XEN_PRIVCMD=y
      CONFIG_CLKDEV_LOOKUP=y
      CONFIG_HAVE_CLK_PREPARE=y
      CONFIG_COMMON_CLK=y
      CONFIG_BLK_DEV_LOOP=y
      CONFIG_BLK_DEV_LOOP_MIN_COUNT=8

      CONFIG_ARCH_XGENE=y
      CONFIG_GENERIC_PHY=y
      CONFIG_PCI_XGENE=y
      CONFIG_SATA_AHCI=y
      CONFIG_SATA_AHCI_PLATFORM=y
      CONFIG_SATA_XGENE_PHY=y
      CONFIG_SATA_XGENE=y
      CONFIG_POWER_RESET_XGENE=y

  • DOM0 DTB -

Apply following Patch:
diff --git a/arch/arm64/boot/dts/apm-mustang-xen.dts b/arch/arm64/boot/dts /apm-mustang-xen.dts
index 848b34a..d6ce7ee 100644
--- a/arch/arm64/boot/dts/apm-mustang-xen.dts
+++ b/arch/arm64/boot/dts/apm-mustang-xen.dts
@@ -18,6 +18,8 @@

  • compatible = "apm,mustang", "apm,xgene-storm";

    chosen {

+ #address-cells=<2>;
+ #size-cells=<1>;

  • module@0 {

    • bootargs = "console=hvc0 root=/dev/sda2 rw earlyprintk=uart8250-32bit,0x1c020000 debug maxcpus=8";

    };

C. Booting DOM0 Kernel

  • Check your tftp directory has following files: uXen, Image and apm-mustang-xen.dts

  • Make sure SATA disk (with above mantioned rootfs) is attached to mustang board.

  • Make sure PCIe NiC (like Intel's e1000) is attached to your board and it's driver is enabled in DOM0 config (for e1000 CONFIG_E1000=y CONFIG_E1000E=y should be enabled).

  • On u-boot prompt execute "run xen_run:

    • #run xen_run

  • This should reach to DOM0 login prompt!!

D. DomU

* Preparing Tools:

  • Make sure your e1000 network card is up in DOM0 and has internet connectivity.

  • Installing build packages like gcc, libfdt etc... (This is one time configuration after first boot):

  • Build xen tools:

    • # git clone git://xenbits.xen.org/xen.git
      # cd xen
      # ./configure
      # make tools
      #make install

  • DomU Images:

    • Image : Kernel Image (same DOM0 kenel Image can be used here)

    • rootfs.ext3 : Create ext3 rootdisk using opensuse or linaro arm64 rootfs.

  • Booting DomU: <BR>>

    • setup the harddisk -

      • #losetup /dev/loop0 /root/domU/rootfs.ext3
        # cat d
        name = "d"
        uuid = "3fb78ba6-8182-484c-acf7-8faba9773f68"
        disk = ["phy:/dev/loop0,xvda,rw"]
        memory = 512
        kernel = "./Image"
        extra = "root=/dev/xvda rw console=hvc0 earlyprintk=xen"

        # /etc/init.d/xencommons start || true
        # echo 9 > /proc/sysrq-trigger
        # export LIBXL_DEBUG_DUMP_DTB=guest.dtb
        # xl -vvv create d
        # xl console d

    • Enjoy !!!!

LEG/Engineering/Virtualization/Xen_on_XGene_Mustang (last modified 2014-02-17 12:27:34)