The goal of this spec is to ensure that all opensource software which ever built on ARM continues building on ARM.
Building does not indicate that the software works, but it's an important first step and a prerequisite for development or bug fixes. Also, some builds include a testsuite which does exercize at least part of the software.
It's important to keep an eye on buildability over time as every piece of software will eventually get updated, particularly the toolchain.
Linaro targets multiple software platforms / distributions. Some of the testing needs to be repeated on each target platform, while it might be enough to run some tests on a single platform.
Choice of software
To test the toolchain, we should exercize it in all the contexts it will be usedin, so the more target platforms we can test it on, the better.
We don't necessarily need to rebuild every piece of software under the sun though; priority should be given to software used in production today, or software which is likely to be shipped soon.
For instance in the case of Ubuntu, if rebuilding all of Ubuntu is simply too much, it might make sense to only rebuild "Ubuntu main".
The Linaro Toolchain Working Group releases new toolchains at a steady pace. We want to continue integrating these into Ubuntu and other platforms, but we need to do proactive testing of the toolchain for this to be a pleasant experience.
To integrate a new Linaro Toolchain release in a platform we should:
- do a rebuild of a representative set of software before updating the toolchain
- do a rebuild of the same set of software with the new toolchain
- compare the rebuilds to create a list of regressions/progressions
- investigate all regressions if any (see build regression tracking)
- reach an agreement with the toolchain maintainer(s) for this platform that the progressions/regressions are an acceptable tradeoff
- update the toolchain in the target platform to the new version
If the target platform doesn't provide a rebuild facility to test new toolchains, the Linaro Infrastructure team should develop and host such a service.
Rebuilds will be setup by the Linaro Toolchain team and the toolchain maintainer(s) for the platform and rebuild results will be reviewed by Linaro platform developers and the janitors for this target platform.
Build regression tracking
To chase build regressions, either exposed by a new toolchain version or updates in a platform, the first step is checking whether it's a toolchain issue or a bug in the affected software. This needs to be checked by a human; for instance, if the compiler is crashing, it is clearly a toolchain bug. However sometimes a toolchain update will expose bugs in the source code of some pieces of software which need to be fixed, for instance missing includes or missing assembly decorations.
XXX who does the investigation
we should provide baseline builds to track down which toolchains are affected.
The first step is checking whether it's a toolchain issue or a bug in the software; this
Usually, there are three providers of toolchains:
- FSF provides source-only toolchains
- Linaro provides source-only toolchains derived from FSF
- Each platform has its own binary toolchain, with additional patches on top of Linaro (or FSF when they don't yet use a Linaro toolchain)
So for each platform, there should be a quick way to test:
- is this bug introduced
One target platform is Ubuntu; here is how things could work.
Rebuilds of Ubuntu main could be run by Matthias Klose using the test-rebuild infrastructure and results reviewed by the Linaro Foundations and Ubuntu ARM teams.
Each problem is identified as being in the toolchain or in a package.
For problems in the toolchain, the person doing the initial investigation would try reproducing the problem with a vanilla Linaro toolchain (one without the Ubuntu patches and default flags). This issue would then either be picked by the Ubuntu toolchain maintainer(s) or by the Linaro Toolchain WG depending on which codebase is affected. The Linaro Toolchain WG would try reproducing with the latest development version of the Linaro Toolchain as built from tip, and with FSF toolchains.
For problems in Ubuntu packages, the Linaro Foundations and Ubuntu ARM teams would work on a fix.
Specs/ArmRebuilds (last modified 2010-11-26 11:13:12)