Setting up jobs in CI

Linaro teams are maintaining various build jobs inside the Linaro CI system. AOSP related builds are co-maintained by B&B/LMG teams.

CI configuration files build jobs relies on Jenkins Job Builder (JJB) and are controlled by a set of YAML files describing the build jobs and maintained in a git repository. To check out the git repository. Use git clone ssh://

A README file is available in the repository and provide more information. JJB has its own documentation available online here.

Setting up an AOSP build

There are already a number of AOSP builds in Linaro CI - all are pretty similar. The best thing to do is to copy an existing AOSP build job (e.g. android-lcr-member-hikey-m) and just replace a few parameters.

AOSP builds in Linaro CI use another repository to store AOSP build configurations. Use git clone ssh://

A presentation covering the files there in more detail is available here.

TODO: copy the information from the slides into this doc?

Building something other than AOSP

Building something other than AOSP follows the same principles - look for an existing build job to start from. If you don't find a close enough build job, you'll have to write a new YAML file (or edit one much more extensively).

If you aren't familiar with YAML, the key thing to keep in mind is that whitespace is extremely important. A single missing space in an indentation can cause obscure build/validation errors. You can use some online tools to validate the syntax (e.g.

To see what parameters go into the YAML file, let's go over the android-gcc-toolchain file as an example:

- job:
    name: android-gcc-toolchain
    project-type: freestyle
    defaults: global

This describes the basics of the job -- its name, project type ("freestyle" means we're using a custom build script instead of something ready-made), and we pull in global defaults. Per convention, the job name should match YAML file name.

        - authorization:
                - job-read
                - job-extended-read
                - job-read
                - job-extended-read
                - job-build
                - job-cancel

This sets what permissions various people have -- everyone (anonymous -- someone who isn't logged in) has permissions to look at the job (job-read) and to look at some additional details (job-extended-read), members of the group linaro (defined by the system admins to be every Linaro employee/assignee) has the same rights, and can run a build (job-build) and cancel a running build (job-cancel).

TODO: are there other important groups and/or permissions that should be covered?

            - build-discarder:
                days-to-keep: 60
                num-to-keep: 60
                artifact-num-to-keep: 1

This controls how long builds of the job are kept -- in this example, a build will be kept for at most 60 days (days-to-keep), and at most 60 builds will be kept (num-to-keep - so a build will be automatically deleted when 60 more recent builds have been run). Build logs (artifact-num-to-keep) will be kept only for the most recent build.

        - string:
            name: GCCVER
            default: '6.1.0'
            description: 'GCC version (e.g. x.y.z or x.y.z-YYYY.MM for TCWG releases)'

A user trying to build this job will have to enter any parameters listed here when starting the build. In this example, the user is prompted for one parameter. The user will be prompted for whatever is in the description field (Here: "GCC version (e.g. x.y.z or x.y.z-YYYY.MM for TCWG releases)"), and the field will be pre-filled with the value "6.1.0" (default). The user's input into the field will be passed to the build script (below, in the builders: section) as a shell variable with the name given in name -- here: GCCVER

    disabled: false

The build isn't disabled.

    node: docker-utopic-aosp

This controls on which build node(s) the job will be run. docker-utopic-aosp is a docker container starting out with an install of Ubuntu 14.10 (Utopic Unicorn) on an x86 build machine. Other nodes are available, including Aarch64 build machines, but "docker-utopic-aosp" is the node commonly used by AOSP builds.

TODO: What other nodes are available? How to run a job on multiple nodes? (e.g. build a toolchain for both x86 and aarch64 hosts?)

    display-name: 'GCC based AOSP toolchain'

The name that will be displayed for this job.


The section that follows the triggers: keyword describes triggers that will automatically trigger a build of this job. (Despite the automatic build triggers, a build job can always be started manually.)

        - gerrit:
            server-name: ___ANY___
                - change-merged-event
                - project-compare-type: 'PLAIN'
                  project-pattern: 'people/bernhard.rosenkranzer/android-ndk-build'
                      - branch-compare-type: 'PLAIN'
                        branch-pattern: 'master'
                      - compare-type: 'PLAIN'
                        pattern: ''

This trigger tells Jenkins to automatically trigger a build whenever a change to the branch master of file in the repository people/bernhard.rosenkranzer/android-ndk-build is submitted in gerrit.

Triggers can also be time based, e.g.

        - timed: 'H H * * 6'

would run the build every day at 6.00. This is especially useful when tracking an upstream master branch.

        - timestamps
        - timeout:
            timeout: 500

This tells Jenkins there's a problem if the job takes longer than 500 minutes - the job will be cancelled if it takes longer than that.

        - git:
            refspec: +refs/heads/master:refs/remotes/origin/master
            name: origin
                - refs/heads/master
            basedir: android-ndk-build
            skip-tag: true
            shallow-clone: true
                after: true
            wipe-workspace: false

This tells Jenkins where the files related to this job live. shallow-clone makes sure the scripts are checked out with git --depth 1 (see the git documentation for details).

Essentially, the entire section above makes sure the job's workspace is initialized with

git clone --depth 1 android-ndk-build
cd android-ndk-build
git checkout -b master origin/master


This is the most "interesting" section - it describes how the project needs to be built.

        - linaro-publish-token

Gets a one-time token needed to publish anything on server.

        - shell: |

Tells Jenkins to launch a shell and run the commands given below the line. As usual in YAML, the end of the script to be run is determined by the next line with less indentation.


Boilerplate to make sure we get the shell we're expecting

            # Install build dependencies
            sudo sed -i -e 's/\|' /etc/apt/sources.list
            sudo apt-get update
            sudo apt-get install -y \
              autoconf automake bash bison build-essential bzip2 chrpath \
              coreutils cpio flex g++ gawk gcc git gzip libffi-dev libncurses5-dev \
              make realpath sed tar texi2html texinfo unzip wget python-pycurl

            test -d ${HOME}/bin || mkdir -p ${HOME}/bin
            curl > ${HOME}/bin/repo
            chmod a+x ${HOME}/bin/repo
            export PATH=${HOME}/bin:${PATH}

This installs the extra packages we need to run the job - compilers, repo to do AOSP checkouts, etc. - and sets PATH so repo can be found.

            # Set some reasonable defaults
            git config --global ""
            git config --global "Linaro CI"

Some defaults to make sure certain git servers don't barf if they can't determine our identity

            cd ${WORKSPACE}/android-ndk-build
            GCCVER=${GCCVER} OURDIR="${WORKSPACE}/android-ndk-build" ./ 2>&1 |tee ${WORKSPACE}/build.log

This goes into the directory that was checked out earlier (in the "scm:" section), and runs the script inside the android-ndk-build repository.

The variable GCCVER (from the parameter defined early on in the "parameters:" section of the YAML file) is passed to the script, and OURDIR is passed to point at our local directory. Those variables are used by the script.

The addition of 2>&1 |tee ${WORKSPACE}/build.log" makes sure we get a decent log of the build, containing both the build script's stdout and stderr (2>&1 redirects stderr to stdout) while also keeping Jenkins aware of any output from the scripts. (tee prints everything to stdout for the sake of Jenkins and also dumps stdout to ${WORKSPACE}/build.loog, which we'll archive in one of the next steps).

It would also be possible to just put all the commands from that script into the yaml file itself. Some users prefer to keep it separate as it can be used for local builds (outside of Jenkins). The downside to keep it externally to the build jobs configurations repository is that you don't have the history in a single place and can't restore the build jobs in a previous state if needed. A third approach is to commit the build script next to the build job configuration.

            if [ ! -e ${WORKSPACE}/android-ndk-build/ndk/out/dist/gcc-arm-linux-x86_64.tar.bz2 ]; then
                echo "ARMv7 crosscompiler failed to build"
                exit 1
            if [ ! -e ${WORKSPACE}/android-ndk-build/ndk/out/dist/gcc-arm64-linux-x86_64.tar.bz2 ]; then
                echo "ARMv8 crosscompiler failed to build"
                exit 1

This is sanity checks to make sure the Jenkins job fails if the output files we need weren't generated.

            # Let's save some space and assign better filenames
            cd ${WORKSPACE}/android-ndk-build/ndk/out/dist
            bunzip2 gcc-arm-linux-x86_64.tar.bz2
            mv gcc-arm-linux-x86_64.tar gcc-${GCCVER}-arm-linux-x86_64.tar
            bunzip2 gcc-arm64-linux-x86_64.tar.bz2
            mv gcc-arm64-linux-x86_64.tar gcc-${GCCVER}-arm64-linux-x86_64.tar
            xz -9ef *.tar
            # And remove stuff we don't need
            rm -rf *.tar.bz2 *.zip logs

The next step will publish everything in the ${WORKSPACE}/android-ndk-build/ndk/out/dist directory -- so we're cleaning it up and recompressing to xz to save some bandwidth

            # Publish
            wget -q -O ${HOME}/bin/
            time python ${HOME}/bin/ \
                --api_version 3 \
                --link-latest \
                ${WORKSPACE}/android-ndk-build/ndk/out/dist android/${JOB_NAME}/${BUILD_NUMBER}

This is used to publish the build results (all files in ${WORKSPACE}/android-ndk-build/ndk/out/dist - first filename on the last line) to the server. The build results will go to the directory android/${JOB_NAME}/${BUILD_NUMBER} (second filename on the last line) -- where ${JOB_NAME} is the job name as set in the beginning of the YAML file, and BUILD_NUMBER is the current build number. will also create a symlink "latest" pointing to the latest ${BUILD_NUMBER} directory.

        - archive:
            artifacts: '*.log'
        - email:
            recipients: ''

This tells Jenkins to archive the build logs -- any file matching the "*.log" glob in the ${WORKSPACE} directory -- and mail the build results to me (you'll want to put your email address there).

It is possible to specify multiple recipients for those notification emails by using a space separated list of email addresses instead of just one email address.

Committing the YAML file

That's it -- hopefully we have a working YAML file by now.

Add the YAML file to ssh:// and push it to gerrit for review - please add Vishal Bhoj as a reviewer and also some additional reviewers so the relevant people get notified and can take a look at the file you just created.


LMG/Engineering/SettingUpJobsInCI (last modified 2016-07-22 07:24:56)