Browser Benchmarking

Background

An effort to add browser benchmarking using ci.linaro.org and Lava was started in late 2012. The main requirement was it had to run in a CI loop so it had to be totally automatic. Certain versions of Firefox had a command line option to render a page and then exit but that would only allow taking the total time of a benchmark suite and not of individual components.

Selenium webdriver seemed to provide the best framework for a browser independent solution for running benchmarks and scraping the results.

Benchmarks

A google search showed that the most popular benchmarks seemed to be V8 (from chrome) Kraken (from Firefox) and Octane (newer replacement for V8). These benchmarks are really javascript benchmarks and not browser benchmarks.

Selenium

Selenium requires an extension for firefox which it is able to install as needed. For Chrome it needs a gateway to translate from selenium wire protocol to Chrome wire protocol. This is a binary called chromedriver which can be built as part of chrome but is not included in the Ubuntu build of the Chrome browser. Chrome browser packaging was modified to include chromedriver and an armhf build of chromedriver has been added to the browser benchmark source here: http://bazaar.launchpad.net/~linaro-foundations/linaro-ubuntu/lava-test-browser-benchmarks/files/head:/bin/

The changes to Ubuntu Chrome packaging were also submitted in a bug but that went nowhere: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1069930

Using the chromedriver, an out of tree Lava test was developed and it worked initially on both Chrome and Firefox. Later it stopped working on Firefox. https://bugs.launchpad.net/linaro-ubuntu/+bug/1084527

This could be worked around or may be fixed with a new Firefox selenium extension.

Issues

These benchmarks are really javascript benchmarks not general browser benchmarks. Benchmarking of Firefox is currently broken. Benchmarking of Chrome requires a chromedriver binary that may not work with future versions of Chrome.

Further investigation may prove that there is a better framework for this than Selenium.

Future

If this task gets taken up again these things should happen.

  • Determine what exactly we are supposed to be benchmarking. As stated above most existing benchmarks are measuring javascript performance not anything else.
  • Revisit if Selenium is the right approach. If it works for Chrome and Firefox then it is great. There may not be another alternative.

JohnRigby/BrowserBenchmarking (last modified 2013-01-31 15:59:24)