Luis Villa is absolutely right in his castigation of our X update on Wednesday this week. As a team we made a series of errors, and the result was a desktop that was broken for thousands of users, for several hours. It has been a severe lesson in QA, something Luis knows plenty about.
An incident report is being compiled by the team and we will publish that for our broader community and users as soon as it is complete. My apologies to those who have been affected, I know that a blue screen of death is the very last thing anybody ever wants to see on Linux desktops and that any downtime caused by mistakes on our part, even measured in minutes, is unacceptable.
In addition to the incident report, we are also putting into production a long-discussed mechanism for widespread testing of non-essential updates (support for new hardware, for example) by users who want advanced access to that code, or those who are part of our more sophisticated user community. We know now that no amount of internal testing will find certain issues, even issues which could have a widespread footprint and obvious failures, and the only way to get certainty on the potential impact of a change is to put it out to a wider, but controlled, audience.
If there is a silver lining to the error, it is that it happened during the one week in six months when we have the core distribution development team together in one place. This gave us the opportunity not just to analyse and fix the issue, and to talk about the sequence of events that led to the problem, but also to discuss the processes we must improve to further reduce the likelihood of a repeat. The team is now more aware than ever of the responsibility we assume given extraordinary rate of adoption of Ubuntu.
My goal is for the team to grow and learn from this experience without becoming paralyzed on future updates. We can’t afford to take risks with our user’s trust, but I balance that with the need to continue to improve the desktop. We WANT to certify new hardware and keep Dapper usable for as much of its five year lifespan as possible. That said, Edgy is the right place to make exciting changes, not Dapper!
Luis, well said.