Mark Shuttleworth | here be dragons

Luis Villa is absolutely right in his castigation of our X update on Wednesday this week. As a team we made a series of errors, and the result was a desktop that was broken for thousands of users, for several hours. It has been a severe lesson in QA, something Luis knows plenty about.

An incident report is being compiled by the team and we will publish that for our broader community and users as soon as it is complete. My apologies to those who have been affected, I know that a blue screen of death is the very last thing anybody ever wants to see on Linux desktops and that any downtime caused by mistakes on our part, even measured in minutes, is unacceptable.

In addition to the incident report, we are also putting into production a long-discussed mechanism for widespread testing of non-essential updates (support for new hardware, for example) by users who want advanced access to that code, or those who are part of our more sophisticated user community. We know now that no amount of internal testing will find certain issues, even issues which could have a widespread footprint and obvious failures, and the only way to get certainty on the potential impact of a change is to put it out to a wider, but controlled, audience.

If there is a silver lining to the error, it is that it happened during the one week in six months when we have the core distribution development team together in one place. This gave us the opportunity not just to analyse and fix the issue, and to talk about the sequence of events that led to the problem, but also to discuss the processes we must improve to further reduce the likelihood of a repeat. The team is now more aware than ever of the responsibility we assume given extraordinary rate of adoption of Ubuntu.

My goal is for the team to grow and learn from this experience without becoming paralyzed on future updates. We can’t afford to take risks with our user’s trust, but I balance that with the need to continue to improve the desktop. We WANT to certify new hardware and keep Dapper usable for as much of its five year lifespan as possible. That said, Edgy is the right place to make exciting changes, not Dapper!

Luis, well said.

One of the Blueprint hacks I got around to this weekend was to add the idea of “essential” subscribers to a feature specification in Launchpad. Since we use Launchpad to plan our meetings, its useful to know when we are arranging the discussion sessions who has to be present when a topic is discussed, and who is simply interested in the discussion.

There’s a brief doc describing the feature further on help.launchpad.net and I think it will first be used for Edgy+1 planning in November.

PS – thanks to Google for the star image, artistic improvements welcome!

A short while ago I blogged about what I think is one of the most interesting and challenging positions at Canonical – the Ubuntu community manager. We had several fantastic folks in the shortlist and I’m pleased to say that Jono Bacon (a.k.a. jono on IRC.freenode.net, pictured here playing his own interpretation of Hamlet) will be stepping up to the plate.

Jono – welcome aboard!

We have one of the world’s best technology communities in Ubuntu – from the UbuntuForums to the MOTU with LoCo teams, Art, Doc, Marketing, and specialist interest groups all collaborating to make Ubuntu rock. I’m excited to have someone working across the project to help them all rock even harder!

Dapper has now been out two months, and in general the feedback from the release has been wonderful. I feel somewhat relieved – it was a challenging set of goals we set ourselves, but it seems that (a) most users prefer Dapper to any previous release and (b) most users are very happy with the combination of features and stability. Phew.

However, there’s always room for improvement, so we have been evaluating the release and figuring out what we can do next time to have a closer alignment of expectations and reality.

Matt Zimmerman put it best in a private mail which he’s given me permission to republish:

Matt Zimmerman wrote:

I've been thinking some lately about a disconnect in terms of the
community's expectations for Dapper.  We've had more negative reviews (and
forums comments) of Dapper than for previous releases, and while it's not
clear to me whether this is proportional to the apparent increase in user
base, I'm concerned by what I find when analyzing the criticism.

Many are criticizing shortcomings in Ubuntu which have existed for years
now, or deplore the lack of eye candy and other superficial features, as
justification for an overall negative impression of the release.  In
particular, I see repeated mentions of:

- Lack of 3D support out of the box on nVidia chipsets: [0]

- Need to use the command line for certain administrative tasks (including
the above) [0] [5]

- Lack of out-of-the-box support for Java, Flash, MP3, DVD, etc.
(RestrictedFormats) [0] [4]

- Lack of 3D accelerated desktop effects and other eye candy (e.g.
Xgl/AIGLX, prettier usplash) [1] [5]

- Lack of support for a particular hardware component (e.g., wireless card
or printer) [1] [2] [4] [6]

- Ubuntu not being easy enough for the typical user [0]

- Lack of availability of development tools in the default install [0] [2]

- Manual partitioning is clunky [0] [3]

None of these are new problems, but they are pointed out as examples of
major shortcomings by these reviews.  It's notable that in some cases, we're
being compared with Windows, rather than other Linux distributions, which is
a much higher bar, but overall my impression is that there has been a
disconnect between the expectations of the community and what we delivered
with Dapper.  In particular, I see indications that users expected Dapper:

- to be better-looking ("polished")
- to have more long-standing feature wishes implemented ("polished")
- to have no regressions from Breezy ("polished")
- to have fewer bugs than a typical Ubuntu release ("polished")

There have been a number of comments on bugs in the less-frequently-used
portions of Ubiquity, but we expected this when rolling a new installer for
Dapper.  The only other genuine quality issue I've ascertained is a
collection of printing-related regressions, which are still being analyzed
and will be fixed via updates as we are able.  Apart from those, the harsh
criticisms mostly fall into the category of unmet expectations.

Another disconcerting trend is that a number of the articles mention having
a negative experience interacting with the Ubuntu community for support.
It's possible that we're starting to see a decline in overall coherence
resulting from such rapid growth of the community: new people are getting
involved faster than they can learn proper behaviour from the existing
members.  I'm not sure how to address this, but perhaps it should be
discussed at the next CC meeting?

I had a dialogue with a representative from the forums where he expressed
surprise at the Edgy feature list, because he thought the features which
went into Dapper were more "edgy" in comparison.  He thought that things
like the revamped boot process infrastructure and 3D eye candy were "polish"
that he would have expected in Dapper, whereas from a developer perspective,
they are risky and disruptive features.  It is this illustrative example
which led me to notice this disconnect, and do this analysis of the reviews.

So, what have we learned?  Perhaps that even if we meet our goals in our own
eyes, we may be considered a failure by some if they have a different
interpretation of our intentions.

Should we be less high-level and more explicit about our goals for a
release?  That's risky, too.  Our Edgy announcement[7] went into detail
about specific features which fit the theme, but of the five specific
examples, it's quite likely that no more than one or two will actually be
implemented in Edgy.  Users who latched onto the examples and built
excitement around them will be disappointed.  The announcement explained
that the development team would be given the freedom to bring in risky new
technologies, but also foretold an increase in "bling" as a result, while it
turns out that our development team isn't very interested in that at all.
Users who expect Edgy to be MovieOS will feel misled.

My conclusion is that we should be rather more careful about how we announce
our plans, because these predictions can have disproportionate repercussions
on how our release is received by the world.  We should be specific, but
only where we can be accurate.  We should express the high-level vision for
the release, but guide the user on its interpretation as well.

We gained our position by meeting and exceeding the expectations of users,
not merely our own, and I think it's important that we continue to do so.

[0] http://www.ourtweaks.com/articles/linux.html
[1] http://linuxguy.bur.st/?p=12
[2] http://www.tectonic.co.za/view.php?id=1026
[3] http://www.nuxified.org/the_ubuntu_dapper_experience
[4] http://www.linuxforums.org/reviews/ubuntu_6.06_review.html
[5] http://www.tuxmachines.org/node/7300
[6] http://distrowatch.com/weekly.php?issue=20060612#opinion
[7] https://lists.ubuntu.com/archives/ubuntu-announce/2006-April/000064.html

So, what can we do better next time?

First, I think we need to realise that any heightened publicity will create impossible expectactions. Dapper was by far the “biggest” release the project has produced to date, we were all excited about it, there was a lot of buzz, and as a result there’s a temptation for each person to hope that this release will have the set of things that THEY think is essentia.

Second, I think the word “polish” is misleading, so I will try to stop using it. The problem is that it says different things to different people. To “polish” a release means to fix lots of little things that are irritating but not essential, and of course, everyone has their own list of bugbears. So saying the release will be “more polished” is about as bad as saying it will be “more better” :-). In future, I think we should articulate specifically where we will be investing the effort. In the case of Dapper, it was (a) the kernel, (b) the installer, (c) the server, and (d) the graphics theme and icons. Of course there was a ton of other work done, but that was what *I* meant by “polish”. So we need to communicate much more clearly EXACTLY where the investment and the results will be found.

Third, we should be honoured to be in head-to-head comparisons with Windows. Some of the toughest criticism of Dapper I have seen comes from people who have never used Linux before. And so they quite rightly stack us up against the OS they use every day. That is something of a success already – we can at least claim to be in the ballpark. Now we need to look PAST where Windows is today, and deliver an experience to those users that is so compelling in some places that they are willing to forgive a few of the warts.

The inevitable Web 2.0-ism has landed in Launchpad… TAGS. Congrats to Brad Bollenbach and Bjorn Tillenius on the landing. In this first cut, only the bug tracker supports them, but if you can suggest interesting other applications for tags in Launchpad I’m sure the guys on #launchpad would be keen to listen.

There’s a demo server with a full list of Python’s bugs from Sourceforge imported into a Launchpad instance, with tags, that you can check out for a taste of bug tagging goodness. That’s not the usual, production LP, it’s a demo setup we use to show new features or to import sample data.

Personally, I’m something of a sceptic. Tags are certainly a useful way to do short term organisation of things in a project, but they can’t compete with rigorous structure for ongoing management. Essentially, a wiki is FULL of tags, but organising things in wiki’s is fraught with challenges. We used to organise the feature planning for Ubuntu in the wiki, but it drove me nuts – to the point that I had to sit down and write the feature planning system in Launchpad. Folksonomies are powerful in the right places, but they are no magic bullet.
Still, I’m intrigued to see how projects use them. You can see how the LP team themselves are coordinating their tags-onomy – I like the idea of having agreed, and proposed, tags. Of course, you can tag anything with anything, but *some* structure or convention makes the whole endeavour much more useful.

Hmm… time to add that to the Malone Highlights doc I wrote up over the w/e.

I woke up this morning to a flood of mail regarding Tony Mobily’s commentary on Ubuntu vs Red Hat. This would be a good time to remind all of us in the Ubuntu community that Red Hat’s demise is not our goal, and not how we should measure our success.

My own view is that Red Hat will continue to do well in the specific areas that they have targeted – they are extremely well established in the high-availability enterprise Linux server market, and it will take some years before Ubuntu can make the same claim.

Our focus is different to that of Red Hat – we want to ensure that there are free (in both FSF and economic senses) platforms for commodity requirements, like desktops and typical web or email of HPC servers, where the existing free software stack does everything that people typically want. And we want to ensure that this free offering is sustainable, so that it is independent of the whims of any large corporate (and frankly independent of my own whims, too).

Essentially, we want to create a sustainable, practical home for the very best Linux engineers and architects (initially drawing primarily from the pool of Debian developers but now we’re starting to bring in upstream folks too). If that can be sustainable, and charged with delivering a free platform that anyone can use for a desktop or a standard server, then I think we will have accomplished something remarkable and unique. Much like the FSG, and OSDL, Ubuntu will have a role to play in making Linux widely and freely available and keeping it at the cutting edge. Our job is to make the amazing, cutting edge work of thousands of free software developers available in a neat, elegant package that anybody can deploy free of charge, with easy access to the whole universe of free software, and for which they can get commercial grade support if they want it.

That’s a much less grand vision than Red Hat’s goal to challenge the establishment – SUN, Microsoft et al – and create a new enterprise serving other enterprises.

It’s true that competition with Red Hat and Novell is part of what energises the Ubuntu team – we strive to produce something that is cleaner, crisper, faster and better engineered than the alternatives that are out there, and we value the bar that they set. But our primary competition is ourselves – we set extremely high standards for our own team, and we aim to beat that with every release.

So, Red Hat will I think retain a place in the world. Given that their balance sheet is almost as strong as (ours), I would expect them to be around for some time ;-).