Release Review: 2012.08

  • Created: David Zinman <david.zinman AT linaro DOT org>

Past reports

Key Points for wider discussion

One issue of concern was the deployment of an infrastructure project on the last week before release. A the new django application was deployed on snapshots.linaro.org that disrupted several builds on the Friday before the release. The issues took one week to resolve. The issues are:

  • The deployment was not communicated and broadcast properly
  • Build maintainers were not given proper warning
  • deploying new functional software on a Friday with no one available on the weekend for emergency
  • deploying during a release week

A big "thank you" is in order for Georgy Redkozubov (gesha) for his help and availability in troubleshooting and getting it fixed.

An incident report has been filed here:

Release Highlights

Blueprint Bump List

https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AjEaTwrvj1bidGhNZHF0V2NjY2lOdEh6OG1OQk5oZEE#gid=5

Android

Highlights

Fairly uneventful cycle for Android Team

Issues

Bero, no one knew who was going to spin the final builds Confusion with cross-platform interaction: QA not sure what builds to test.

Virtual Connect was disruptive, but was good.

Lessons Learned

Dzin, never do an infrastructure deployment the friday before a release

Boot testing should be done by the development teams.

The card cycle and release cycle need to get in sync

We need to tell the QA team which build they should run through QA.

Developer Platform

Highlights

  • Kernel CI is working on LAVA
  • Got most of the kernel updates this month
  • Delivered on time even though the infrastructure was broken much of the time.

Issues

  • Infrastructure implementation making the build platform unstable on the last week of the cycle.
    • no documentation given at the time for new tools or build infrastructure
  • Sometimes LAVA is broken without any knowledge of it
    • Sometimes it is unknown what the problem is
  • Origen: lack of ability to test bootloader created respins
  • Lack of device tree support on LAVA

Lessons Learned

  • Commicate closer with infrastructure
  • Never update critical infrastructure components on release week

Graphics

Highlights

Issues

Lessons Learned

Kernel

Highlights

Issues

Lessons Learned

Multimedia

Highlights

Issues

Lessons Learned

OCTO

Highlights

Issues

Lessons Learned

Power Management

Highlights

Issues

Lessons Learned

Toolchain

Highlights

Issues

Lessons Learned

LAVA

Highlights

  • Made the release date.

Issues

  • Incoming requests should have a process
  • Resources are scarce.

Lessons Learned

Infrastructure

Highlights

  • snapshots.l.o/releases.l.o are running live code, and we have a fully working staging env for both
  • we are getting ci-dashboard to actually do useful stuff, thanks to the whole team for that!
  • Worked on the Kernel ci loop, it was a good experience working with Paul, we were able to achieve a good amount of things in short time.
  • Staging for snapshots and releases are live and were already usefull. Worked with Milo on initial implementation of android-build ci dashboard.
  • pairing was a good experience, we did also some pair programming
  • dashboard, lot of code to study, lot of good discussion on further approaches and refactorings, some already done, more to do
  • First time on maintenance. Helped with license protection staging deployment, which was new XP for me. Work on Git hosting re-evaluation. Did a lot of code reviews on CI dashboard.

Issues

  • there's a lack of clarity regarding our schedule and timing, we need to fix this; hopefully our cycle time email helps but will need to work with that
  • once the snapshots + releases servers went live a lot of strong opinions came out of the woodwork
  • Need an API for license protection stuff. More testing should be done but unfortunately it's hard to cover all items with so wide usage area
  • Half cycle on vacation, didn't have chance to feel the maintenance tasks in full. Will make up next time.

Lessons Learned

  • communicate openly about the scheduling requirements, and communicate often (i.e. it's not enough to mention it once, but you have to repeat it)
  • Since the dashboard dev is spread across the team, we had some good discussions before implementing it as a whole team. That was a good thing to establish what would be good to implement and avoid rework.
  • more scripts interacting with the web interface than expected
    • so we need to try and stop that and there is talk of an API (well, there has been for a while) to allow scripts to work better with those servers.
  • More discussion before implementation as with ci dashboard being worked in parallel
  • keep the discussions coming, but we also need to move things forward (we will refactor later)
  • with a bit "open" approach to dashboard devel, we'll probably need to do a lot of refactoring going forward, but well, a development approach too.

Samsung Landing Team

Highlights

Issues

Lessons Learned

Release

Highlights

Issues

Lessons Learned

Cycles/1208/Release/Review (last modified 2012-09-11 17:31:46)