Running benchmarks with the auto builders

This section describes how to run benchmark using the auto builders.

Spawn a job

http://ex.seabright.co.nz/helpers/scheduler/spawn

Merge requests are automatically built.Otherwise, drop arbitrary tarballs into cbuild@orion:~/snapshots and spawn <tarball name minus extension>. For example:

scp gcc-4.6.3.tar.gz cbuild@orion:~/snapshots; spawn gcc-4.6.3 into a9-builder

After starting a job, it will show up in the scheduler: http://ex.seabright.co.nz/helpers/scheduler

Jobs

  • gcc-version - build and test GCC
  • benchmarks-gcc-version - run coremark, denbench, eembc against the already built version
  • benchmarks-spec2000-gcc-version - run spec2000

Queues

  • a9-builders: anything that can naive build a A9 compiler
  • a9-ref: reference A9 boards
  • a8-ref: reference A8 boards

Variables

  • BENCHMARKS = list, such as coremark spec2000 pybench - run these benchmarks instead of the defaults

Variants:

Running benchmarks manually

This section describes how to run benchmarks manually using the cbuild scripts. (Some of this is specific for running on the toolchain validation boards, but most of the instructions applies to the general case.)

Log in to a board

ssh ursa2.ex
# longer version...
ssh -p 7022 asa-san@ex.seabright.co.nz
# screen makes sure your work continues even if the connection is lost
screen

Set up a few things. (One time stuff.)

apt-get install ccrypt bzr time ...
mkdir -p ~/.config/cbuild
echo secretPassword > ~/.config/cbuild/password 

# preferably create your build folder on /scratch
mkdir bench; cd bench
bzr branch lp:cbuild

# put the test files in the cbuild folder
tar -xvf ../../files.tar

# Create a slave directory for this host
mkdir -p slaves/$(hostname); cd slaves/$(hostname)

# Create a local.mk and add some basic stuff. Add more configuration later if needed
echo "TOPDIR=../.." > local.mk
echo "CONFIG=cortexa9" >> local.mk
echo "PUBLISH_URL=some-local-publish-dir" >> local.mk

Commands for running the benchmarks

cd cbuild/slaves/$(hostname)

# EEMBC with default toolchain and configuration
make -f ../../lib/eembc.mk

# EEMBC with a specific toolchain
make -f ../../lib/build.mk gcc-linaro-4.6-2011.10/benchmarks.stamp BENCHMARKS=eembc

# SPEC2000 with default configuration, executing a set of tests with full workload
make -f ../../lib/spec2000.mk  TESTS="int 177 179 183 188" WORKLOAD=ref

# Use the gcc in the system for coremark, default variants.
make -f ../../lib/build.mk gcc-native/benchmarks.stamp BENCHMARKS=coremark;

Variants

  • For running more than one variant in a batch, create a file containing the variants you like and set the DEFAULT_VARIANTS variable to point to that file.

cp ../../lib/mlh1-variants.mk ../../lib/asa-variants.mk
# chose variants
vi ../../lib/asa-variants.mk
echo "DEFAULT_VARIANTS=asa-variants" >> local.mk
  • The variants in DEFAULT_VARIANTS are picked automatically from build.mk.

Results

# Install dependencies if needed
sudo apt-get install python-numpy

# Get the results from the board
scp -r asa-san@ursa4.ex://home/asa-san/bench/cbuild/slaves/ursa4/some-local-publish-dir results

# Tabulate
cd results
find -name *run* -execdir cp '{}' ../.. ';'
python ~/linaro/benchs/linaro-toolchain-benchmarks/scripts/tabulate.py  gcc*/*run.txt> tmp.csv

4.4.x compilers

  • To avoid build errors for c++ tests when using 4.4.x compilers, remove the LD_LIBRARY_PATH from build.mk (the gcc-%/benchmarks.stamp target) and run the C only tests.

https:/wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/RunningBenchmarksWithCbuild (last modified 2012-03-07 12:43:04)