Currently the subject of a blueprint

Example build job definition on android-build.linaro.org

   1 MANIFEST_REPO=git://android.git.linaro.org/platform/manifests.git
   2 MANIFEST_BRANCH=linaro_android_2.3.4
   3 TARGET_PRODUCT=pandaboard
   4 TOOLCHAIN_URL=https://android-build.linaro.org/jenkins/job/linaro-android_toolchain-4.5-2011.06-0/2/artifact/build/out/android-toolchain-eabi-linaro-4.5-2011.06-0-2-2011-06-21_00-59-57-linux-x86.tar.bz2
   5 LAVA_SUBMIT=1
   6 DEFAULT_TIMEOUT=3600
   7 
   8 LAVA_TEST_1="monkey 100"
   9 LAVA_TEST_1_TIMEOUT=60
  10 LAVA_TEST_2="monkey 200"
  11 LAVA_TEST_2_PATTERN="## Network stats: elapsed time=(?P<measurement>\\d+)ms"
  12 LAVA_TEST_2_TIMEOUT=120
  13 LAVA_TEST_4="monkey 400"
  14 LAVA_TEST_3="monkey 300"
  15 
  16 LAVA_TEST_PLAN="busybox,skia,v8,monkey,bluetooth"
  17 BLUETOOTH_OPTION="00:15:83:15:A3:10"
  18 BLUETOOTH_TIMEOUT=500
  19 MONKEY_TIMEOUT=600
  20 
  21 MONKEY_RUNNER_URL_1="git://android.git.linaro.org/test/linaro/android/system.git"
  22 MONKEY_RUNNER_URL_1_TIMEOUT=800
  23 MONKEY_RUNNER_URL_2="file:///git-url"

PaulSokolovsky says: That's too complicated, not readable, not writable. And general idea is that build config should stay lean and clean, so anything not related to build per se, should go to separate file(s) (hosted on HTTP), referenced by a single var from build config. So, alternative proposal:

New LAVA_CUSTOM_TEST_URL var is added, it would point to the file as presented here: Platform/Android/AndroidBuild-LavaIntegration/pfalcon, which should be equivalent to the above.

Parameters

Required:

  • To enable submission to LAVA, the android build job needs to set LAVA_SUBMIT, e.g. LAVA_SUBMIT=1.

Optional:

  • To specify the tests, the android build needs to set LAVA_TEST_PLAN or LAVA_TEST_XXX or MONKEY_RUNNER_URL_X

  • The MONKEY_RUNNER_URL_X variable:
    Specify the git url that monkeyrunner scripts are located. All '*.py' file in that repository will be execute in the order of their path. And the generated '.png' files will be collected.

  • The LAVA_TEST_PLAN variable:
    define the tests that already defined by lava-android-test.
    e.g. LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,cts,monkey"
    By default, the test plan will be "0xbench, glmark2, monkey".

  • The ${TEST}_OPTION variable
    define the option that needed by the test defined in LAVA_TEST_PLANe.g.
    BLUETOOTH_OPTION="00:15:83:15:A3:10"
    defined the pair address that will be passed to the bluetooth test in lava.
    Note: the ${TEST} is the upper case format of the defined test name.

  • The LAVA_TEST_xxx variable:
    specify the android commands that can be run on the android system.

    • The LAVA_TEST_xxx variable can not be LAVA_TEST_PLAN, and can not end with _PATTERN too.

    • Each variable specified in LAVA_TEST_xxx will be treated as one test action in the lava job. That is to say each variable will be treated as one test case by LAVA.

  • Specify the parser for LAVA_TEST_xxx test:
    With _PATTERN appended after the test variable name, we can define the parser for each test.
    Like for test "LAVA_TEST_2", we can define the parser with "LAVA_TEST_1_PATTERN" variable:
    e.g. LAVA_TEST_2_PATTERN="## Network stats: elapsed time=(?P<measurement>\\d+)ms"

    • Note here, the '\' in parser regular expression should be escaped to '\\', because the regular expression will be written to the job file as a json string.
  • XXX_TIMEOUT variable: Specifies the timeout in seconds for the test.

    • DEFAULT_TIMEOUT: Specifies the default timeout in seconds for all tests that do not specify the timeout option.

    • MONKEY_RUNNER_URL_X_TIMEOUT: Specifies the timeout in seconds for the monkeyrunner test.

    • LAVA_TEST_X_TIMEOUT: Specifies the timeout in seconds for the custom test.

    • ${TESTNAME_UPPERCASE}_TIMEOUT: Specifies the timeout in seconds for the test listed in LAVA_TEST_PLAN. Note that ${TESTNAME_UPPERCASE} is the upper case name of the test listed in LAVA_TEST_PLAN (i.e. BLUETOOTH_TIMEOUT and MONKEY_TIMEOUT).

  • Execution order:
    All the tests are executed after the android has booted up.

    1. The LAVA_TEST_xxx tests will be executed in the dictionary order of their variable names first
      e.g. LAVA_TEST_1 will be run before LAVA_TEST_2, and LAVA_TEST_2 will be run before test busybox

    2. Then the tests specified in LAVA_TEST_PLAN will be run

    3. The MONKEY_RUNNER_URL_ tests will be run in the dictionary order of their variable names after LAVA_TEST_PLAN tests

Implementation

All jobs are currently submitted as one user: "android-build-system".

This means:

  • Test results can only be sent to certain types of streams:
    • /anonymous/*
    • /{private,public}/personal/android-build-system/*
    • Anything owned by a team that the android-build-system user is in.
  • No-one can log in as the user to generate a new token, or revoke an old one, if needed.
    • If this happens contact the LAVA admins, who can reset the password for the user temporarily to allow someone to log in to generate a new token and revoke the old one.


CategoryAndroidTips

Platform/Android/AndroidBuild-LavaIntegration (last modified 2013-08-22 09:10:32)