Linaro runs a growing number of CI tests that track the QEMU master branch. Initially the only tests were basic and packaging tests running directly from our Jenkins instance. Increasingly we are adding more tests using the LAVA framework.

Current Tests

In Jenkins

Name

Host Platform

Target Platforms

Link

Notes

Build Source Package

docker-utopic

Source

qemu-docker

Root build for all Jenkins jobs

Build Binary Package

qemu-docker-builder

All

qemu-docker-builder

Builds armhf, arm64, x86_64 packages

LTP Tests

x86, 12.04

armhf

qemu-ltp

Limited linux-user support, some tests fail

In addition to the tests being run on the CI infrastructure we also launch a number of jobs in LAVA.

In LAVA

The LAVA jobs typically do a full build and run several sets of related tests with the resultant binary.

Name

CI Config

Job Spec

Host Platform

Target Platforms

LTP

Config Jenkins

aarch64 armhf

x86_64 KVM

armhf, aarch64

Not yet on par with Jenkins LTP

RISU

Config Jenkins

aarch64

x86_64 KVM

aarch64

Random Instruction Tests

TCG System

Config Jenkins

job spec

x86_64 KVM

aarch64

boot and stress tests, more can be added

Monitoring

Currently monitoring is done manually so there is no automatic dashboard type functionality to indicate when things have regressed. All results run in LAVA are dumped into the qemu-master bundle stream. There are also some time-series graphs which can help visualise progress. Although the versions being reported in this stream should increase monotonically there is nothing that enforces that requirement. Generally when re-running tests the user should send results to their own bundle stream to avoid confusion.

Adding New Tests

Tests are added and controlled through a number of files

Firstly you create a LAVA test definition. This is is a standard LAVA yaml description (usually tweak-able with job parameters). Often there will be a helper script to actually invoke the tests and report results to LAVA. There may additionally be an expect like wrapper to control input into the system and monitor the results.

There are already some general helper "tests" which deal with the compiling of QEMU and setting up of a chroot or system image. These are configurable by passing parameters in the job description to control exactly what you want built.

You then need to create a LAVA Job Description that calls these tests. Usually in sequence as you'll need to build QEMU, install it and setup. Currently all these job descriptions live in a repo controlled by Alex here. When adding new tests it may be worth considering expanding an existing job as the setup cost to build QEMU is quite high.

Finally the jobs need to be triggered from Jenkins. This is done by adding a job to the [https://git.linaro.org/ci/job/configs.git|CI Job Configs] and submitting a change request to our gerrit server.

Example: qemu-system-test

The qemu-system-test currently runs two tests. A simple boot test that check a supplied kernel boots and a stress test the boots into the image and then runs the stress tool until it exits. To take the stress test as an example the following files are involved:

File

Notes

qemu-system-test.yaml

Main test file, configurable with IMAGE, TESTS, QEMU_BIN, QEMU_ARGS, KERNEL_ARGS

qemu-system-tests.sh

Helper script to invoke QEMU, report results to LAVA

qemu-stress.expect

Expect script for the stress test

For the CI system to run the test there is a generic system test job defined in test runners. If you look at the testdef_repos of the job you'll see there are actually 3 separate steps. These are:

  1. Build and install the QEMU binary
  2. Run the boot and stress test with a 64 bit kernel + userspace

  3. Run the boot and stress test with a 64 bit kernel and 32 bit userspace

Finally there is a CI config which triggers this LAVA job each time the QEMU build is updated.

Core/Virtualization/Testing/QEMU (last modified 2015-02-23 19:25:55)