Uboot + Xen + Linux Kernel + BusyBox Build from Source

Setup environment:

Create workspace folder:

mkdir ~/Workspace/xenonarm
export WORK_DIR=~/Workspace/TCC/xenonarm

Install qemu for arm64 arch:

apt install qemu-system-aarch64

download busybox + linux kernel + xen + uboot:

cd $WORK_DIR
 
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-x.x.x.tar.xz
wget http://busybox.net/downloads/busybox-x.x.x.tar.bz2
wget https://downloads.xenproject.org/release/xen/x.xx.x/xen-x.xx.x.tar.gz
wget -c ftp://ftp.denx.de/pub/u-boot/u-boot-yyyy.mm.tar.bz2

extract:

tar xf linux-x.x.x.tar.xz
tara xf busybox-x.x.x.tar.bz2

Build ARM Linux Kernel:

cd $WORK_DIR
cd linux-x.x.x
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

open .config file with the text editor and check lines:

CONFIG_XEN_DOM0=y
CONFIG_XEN=y

build the kernel:

make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

Build BusyBox:

cd $WORK_DIR
cd busybox-x.x.x
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- defconfig
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- menuconfig

Navigating to Busybox Settings → Build Options and checking “Build BusyBox as a static binary (no shared libs)” option. Now we can proceed with compilation:

make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- install

Create root filesystem (rootfs) on your working directory:

cd $WORK_DIR
mkdir rootfs
touch rootfs/init

open rootfs/init with the text editor and paste the following bash script:

#!/bin/sh
 
mount -t proc none /proc
mount -t sysfs none /sys
mknod -m 660 /dev/mem c 1 1
mount -n -t tmpfs none /dev
mknod -m 622 /dev/console c 5 1
mknod -m 666 /dev/null c 1 3
mknod -m 666 /dev/zero c 1 5
mknod -m 666 /dev/ptmx c 5 2
mknod -m 666 /dev/tty c 5 0
mknod -m 444 /dev/random c 1 8
mknod -m 444 /dev/urandom c 1 9
chown root:tty /dev/{console,ptmx,tty}
 
echo -e "\nHello!\n"
 
exec /bin/sh

Make it executable running:

chmod +x rootfs/init

Copy busybox stuff to rootfs:

cp -av busybox-x.x.x/_install/* rootfs/

Create standar directory layout:

mkdir -pv rootfs/{bin,sbin,etc,proc,sys,usr/{bin,sbin}}

Create etc/init.d/rcS script:

mkdir rootfs/etc/init.d
touch rootfs/etc/init.d/rcS

open etc/init.d/rcS with the text editor and paste the following bash script:

#! /bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
/sbin/mdev -s

Make it executable running:

chmod +x rootfs/etc/init.d/rcS

cd rootfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../rootfs.cpio.gz

Running Linux + BusyBox:

Go to the work directory:

cd $WORK_DIR

Run:

qemu-system-aarch64 -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -kernel linux-4.20.11/arch/arm64/boot/Image -nographic -no-reboot -initrd rootfs.cpio.gz -append "rw root=/dev/ram rdinit=/sbin/init  earlyprintk=serial,ttyAMA0 console=ttyAMA0"

U-BOOT

cd u-boot-yyyy.mm
make CROSS_COMPILE=aarch64-linux-gnu- qemu_arm64_defconfig

open .config file with the text editor and add, if not exist, the following lines:

CONFIG_ARCH_QEMU=y
CONFIG_TARGET_QEMU_ARM_64BIT=y

build the u-boot:

make CROSS_COMPILE=aarch64-linux-gnu- -j4

all images must be on the same folder, so move them:

cd $WORK_DIR/u-boot-yyyy.mm
cp u-boot.bin $WORK_DIR/.
cd $WORK_DIR
cp $WORK_DIR/linux-4.20.11/arch/arm64/boot/Image $WORK_DIR/.

should be able to run u-boot with Linux:

qemu-system-aarch64 -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 512M -bios u-boot.bin -device loader,file=Image,addr=0x45000000 -nographic -no-reboot -chardev socket,id=qemu-monitor,host=localhost,port=7777,server,nowait,telnet -mon qemu-monitor,mode=readline

and we can start Linux with this command:

booti 0x45000000 - 0x40000000

where 0x45000000 is the address of our kernel image and 0x40000000 is the default address of the qemu’s device tree blob.

After execution of the command we should have the kernel booting and failing on trying to load a rootfs (since we haven’t provided it).

Building XEN:

cd $WORK_DIR
tar xf xen-x.xx.x.tar.gz

 
cd xen-x.xx.x
make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

cp xen/xen $WORK_DIR/.

Let’s generate a device tree blob:

qemu-system-aarch64  -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -display none -machine dumpdtb=virt-gicv3.dtb
 
dtc -I dtb -O dts virt-gicv3.dtb > virt-gicv3.dts
 
sed 's/compatible = "arm,pl061.*/status = "disabled";/g' virt-gicv3.dts > virt-gicv3-edited.dts
 
dtc -I dts -O dtb virt-gicv3-edited.dts > virt-gicv3.dtb

disable ARM PL061. PL061 is an Advanced Microcontroller Bus Architecture (AMBA) compliant controller.

Get (and remember) the size of the linux kernel image, you will need to provide it during u-boot stage (substitute <> with the names) :

printf "%x\n" `stat -c "%s" <linux_image_filename>`
printf "%x\n" `stat -c "%s" <rootfs_image_size>`

qemu-system-aarch64  -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -bios u-boot.bin -device loader,file=xen,force-raw=on,addr=0x49000000 -device loader,file=Image,addr=0x47000000 -device loader,file=virt-gicv3.dtb,addr=0x44000000 -nographic -no-reboot -chardev socket,id=qemu-monitor,host=localhost,port=7777,server,nowait,telnet -mon qemu-monitor,mode=readline

After intterupting “u-boot autoboot” run this commands to set the Flatten Device Tree (FDT) config:

fdt addr 0x44000000
 
fdt resizefdt set /chosen \#address-cells <1>
 
fdt set /chosen \#size-cells <1>fdt mknod /chosen module@0
 
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
 
fdt set /chosen/module@0 reg <0x47000000 linux_image_size>
 
fdt set /chosen/module@0 bootargs "earlyprintk=serial,ttyAMA0
console=ttyAMA0,115200n8 earlycon=xenboot"
 
booti 0x49000000 - 0x44000000

(here it will fail because we didn’t provide any rootfs)

Running everything on Qemu:

qemu-system-aarch64  -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -bios u-boot.bin -device loader,file=xen,force-raw=on,addr=0x49000000 -device loader,file=Image.gz,addr=0x47000000 -device loader,file=virt-gicv3.dtb,addr=0x44000000 -device loader,file=rootfs.img.gz,addr=0x42000000 -nographic -no-reboot -chardev socket,id=qemu-monitor,host=localhost,port=7777,server,nowait,telnet -mon qemu-monitor,mode=readline

After intterupting “u-boot autoboot” run this commands to set the Flatten Device Tree (FDT) config:

fdt addr 0x44000000
 
fdt resize
 
fdt set /chosen \#address-cells <1>
 
fdt set /chosen \#size-cells <1>
 
fdt mknod /chosen module@0
 
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
 
fdt set /chosen/module@0 reg <0x47000000 linux_image_size>
 
fdt set /chosen/module@0 bootargs "rw root=/dev/ram rdinit=/sbin/init   earlyprintk=serial,ttyAMA0 console=hvc0 earlycon=xenboot"
 
fdt resize
 
fdt mknod /chosen module@1
 
fdt set /chosen/module@1 compatible "xen,linux-initrd" "xen,multiboot-module"
 
fdt set /chosen/module@1 reg <0x42000000 rootfs_image_size>
 
booti 0x49000000 - 0x44000000

(if you face some FDT_ERR_NOSPACE it just means that FDT doesn’t have enough space for further allocations. Use the command fdt resize to provide extra space)

Running Dom0 and DomU simultaneously on Qemu-arm64:

qemu-system-aarch64 -machine virt,gic_version=3 -machine virtualization=true -cpu cortex-a57 -machine type=virt -m 4096 -smp 4 -bios u-boot.bin -device loader,file=xen,force-raw=on,addr=0x49000000 -device loader,file=Image,addr=0x47000000 -device loader,file=Image,addr=0x53000000 -device loader,file=virt-gicv3.dtb,addr=0x44000000 -device loader,file=rootfs.cpio.gz,addr=0x42000000  -device loader,file=rootfs.cpio.gz,addr=0x58000000 -nographic -no-reboot -chardev socket,id=qemu-monitor,host=localhost,port=7777,server,nowait,telnet -mon qemu-monitor,mode=readline

After intterupting “u-boot autoboot” run this commands to set the Flatten Device Tree (FDT) config:

fdt addr 0x44000000
 
fdt resize
 
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
 
fdt mknod /chosen module@0
 
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <0x47000000 linux_image_size>
fdt set /chosen/module@0 bootargs "rw root=/dev/ram rdinit=/sbin/init   earlyprintk=serial,ttyAMA0 console=hvc0 earlycon=xenboot"
 
fdt resize
 
fdt mknod /chosen module@1
 
fdt set /chosen/module@1 compatible "xen,linux-initrd" "xen,multiboot-module"
fdt set /chosen/module@1 reg <0x42000000 rootfs_image_size>
 
fdt mknod /chosen domU1
 
fdt set /chosen/domU1 compatible "xen,domain"
fdt set /chosen/domU1 \#address-cells <1>
fdt set /chosen/domU1 \#size-cells <1>
fdt set /chosen/domU1 \cpus <1>
fdt set /chosen/domU1 \memory <0 548576>
fdt set /chosen/domU1 vpl011
 
fdt mknod /chosen/domU1 module@0
 
fdt set /chosen/domU1/module@0 compatible "multiboot,kernel" "multiboot,module"
fdt set /chosen/domU1/module@0 reg <0x53000000 linux_image_size>
fdt set /chosen/domU1/module@0 bootargs "rw root=/dev/ram rdinit=/sbin/init console=ttyAMA0"
 
fdt mknod /chosen/domU1 module@1
 
fdt set /chosen/domU1/module@1 compatible "multiboot,ramdisk" "multiboot,module"
fdt set /chosen/domU1/module@1 reg <0x58000000 rootfs_image_size>
 
booti 0x49000000 - 0x44000000