Mark Shuttleworth | here be dragons

It’s been two weeks since Rick Spencer made the case for a rolling release approach in Ubuntu. Having a rolling release is one of the very top suggestions from the hardcore Ubuntu user community, and after years of it being mooted by all and sundry I thought it deserved the deep consideration that Rick and his team, who represent most of Canonical’s direct contributions to Ubuntu, brought to the analysis.

It’s obviously not helpful to have mass hysteria break out when ideas like this get floated, so I would like to thank everyone who calmly provided feedback on the proposal, and blow a fat raspberry at those of you who felt obliged to mount soapboxes and opine on The End Of the World As We Know It. Sensible people the world over will appreciate the dilemma at being asked to take user feedback seriously, and being accused of unilateralism when exploring options.

Change is warranted. If we want to deliver on our mission, we have to be willing to stare controversy in the face and do the right thing anyway, recognising that we won’t know if it’s the right thing until much later, and for most of the intervening time, friends and enemies alike will go various degrees of apoplectic. Our best defense against getting it wrong is to have a strong meritocracy, which I think we do. That means letting people like Rick, who have earned their leadership roles, explore controversial territory.

So, where do we stand? And where do I stand? What’s the next step?

What makes this conversation hard is the sheer scale of the Ubuntu ecosystem, all of which is profoundly affected by any change. Here are the things I think we need to optimise for, and the observations that I think we should structure our thinking around:

Releases are good discipline, cadence is valuable.

Releases, even interim releases, create value for parts of the Ubuntu ecosystem that are important. They allow us to get more widespread feedback on decisions made in that cycle – what’s working, what’s not working. Interestingly, in the analysis that played into Rick’s proposal, we found that very few institutional users depend on extended support of the interim releases. Those who care about support tend to use the LTS releases and LTS point releases.

Release management detracts from development time, and should be balanced against the amount of use that release gets.

While reaffirming our interest in releases, I think we established that the amount of time spend developing in a cycle versus spent doing release management is currently out of whack with the amount to which people actually DEPEND on that release management, for interim releases, on the desktop. On the server, we found that the interim releases are quite heavily used in the cloud, less so on physical metal.

Daily quality has raised the game dramatically for tip / trunk / devel users, and addresses the Rolling Release need.

There’s widespread support for the statement that ‘developers can and should use the daily development release’. The processes that have been put in place make it much more reliable for folks who want to track development, either as a contributor to Ubuntu or as someone who ships software for Ubuntu and wants to know what’s happening on the latest release, to use Ubuntu throughout the development cycle. For those of you not aware, uploads to the edge get published in a special ‘pocket’, and only moved into the edge if they don’t generate any alarms from people who are on the VERY BLEEDING EDGE. So you can use Raring (without that bleeding edge pocket) and get daily updates that are almost certain not to bork you.  There is a real community that WANTS a rolling release, and the daily development release of Ubuntu satisfies this need already.

LTS point releases are a great new enhancement to the LTS concept.

On a regular basis, the LTS release gets a point update which includes access to a new, current kernel (supporting new hardware without regressing the old hardware on the previous kernel, which remains supported), new OpenStack (via the Cloud Archive), and various other elements. I think we could build on this to enhance the LTS with newer and better versions of the core UX (Unity) as long as we don’t push those users through a major transition in the process (Unity/Qt, anybody? ;-)).

Separating platform from apps would enhance agility.

Currently, we make one giant release of the platform and ALL APPS. That means an enormous amount of interdependence, and an enormous bottleneck that depends largely on a single community to line everything up at once. If we narrowed the scope of the platform, we would raise the quality of the platform. Quite possibly, we could place the responsibility for apps on the developers that love them, giving users access to newer versions of those apps if (and only if) the development communities behind them want to do that and believe it is supportable.

Phew.

That’s what I observed from all the discussion that ensued from Rick’s proposal.

Here’s a new straw man proposal. Note – this is still just a proposal. I will ask the TB to respond to this one, since it incorporates both elements of Rick’s team’s analysis and feedback from wider circles.

Updated Ubuntu Release Management proposal

In order to go even faster as the leading free software platform, meet the needs of both our external users and internal communities (Unity, Canonical, Kubuntu, Xubuntu and many many others) and prepare for a wider role in personal computing, Ubuntu is considering:

1. Strengthening the LTS point releases.

Our end-user community will be better served by higher-quality LTS releases that get additional, contained update during the first two years of their existence (i.e. as long as they are the latest LTS). Updates to the LTS in each point release might include:

• addition of newer kernels as options (not invalidating prior kernels). The original LTS kernel would be supported for the full duration of the LTS, interim kernels would be supported until the subsequent LTS, and the next LTS kernel would be supported on the prior LTS for teh length of that LTS too. The kernel team should provide a more detailed updated straw man proposal to the TB along these lines.
• optional newer versions of major, fast-moving and important platform components. For example, during the life of 12.04 LTS we are providing as optional updates newer versions of OpenStack, so it is always possible to deploy 12.04 LTS with the latest OpenStack in a supported configuration, and upgrade to newer versions of OpenStack in existing clouds without upgrading from 12.04 LTS itself.
• required upgrades to newer versions of platform components, as long as those do not break key APIs. For example, we know that the 13.04 Unity is much faster than the 12.04 Unity, and it might be possible and valuable to backport it as an update.

2. Reducing the amount of release management, and duration of support, for interim releases.

Very few end users depend on 18 months support for interim releases. The proposal is to reduce the support for interim releases to 7 months, thereby providing constant support for those who stay on the latest interim release, or any supported LTS releases. Our working assumption is that the latest interim release is used by folks who will be involved, even if tangentially, in the making of Ubuntu, and LTS releases will be used by those who purely consume it.

3. Designating the tip of development as a Rolling Release.

Building on current Daily Quality practices, to make the tip of the development release generally useful as a ‘daily driver’ for developers who want to track Ubuntu progress without taking significant risk with their primary laptop. We would ask the TB to evaluate whether it’s worth changing our archive naming and management conventions so that one release, say ‘raring’, stays the tip release so that there is no need to ‘upgrade’ when releases are actually published. We would encourage PPA developers to target the edge release, so that we don’t fragment the ‘extras’ collection across interim releases.

That is all.

Harald,

Of course what Kubuntu and Xubuntu and Ubuntu GNOME Remix et al do matters. If it didn’t, we wouldn’t invest a ton of time and energy in finding ways to share the archives effectively. And I consider it one of the lovely things about Ubuntu that there is room for all of us here. As long as there are people willing to make it happen, there’s room for a new face.

You all make the broad Ubuntu family more diverse and more interesting. For which I’m grateful.

In return, you get the benefit of an enormous and concentrated investment in making a core platform that can be widely consumed (on top of the already enormous efforts of the open source community, Debian, and any number of other groups). That investment brings with it a pace of change, and a willingness to be focused on specific outcomes. Mir, which is a fantastic piece of engineering by a very talented team that has looked hard at the problem and is motivated to do something that will work well, is just one example. Every week, we’re figuring out how to coordinate changes. Why blow a gasket over this one? I’ve absolutely no doubt that Kwin will work just fine on top of Mir. And I’m pretty confident Mir will be on a lot more devices than Wayland. Which would be good for KDE and Kubuntu and Plasma Active.

So, before you storm off, have a cup of tea and think about the gives and gets of our relationship. Seriously.

Mark

Jonathan says that Canonical is not taking care of the Ubuntu community.

Consider for a minute, Jonathan, the difference between our actions.

Canonical, as one stakeholder in the Ubuntu community, is spending a large amount of energy to evaluate how its actions might impact on all the other stakeholders, and offering to do chunks of work in support of those other stakeholder needs.

You, as one stakeholder in the Ubuntu community, are inviting people to contribute less to the broader project, and more to one stakeholder.

Hmm. Just because you may not get what you want is no basis for divisive leadership.

Yes, you should figure out what’s important to Kubuntu, and yes, you should motivate folks to help you achieve those goals. But it’s simply wrong to suggest that Canonical isn’t hugely accommodating to the needs of others, or that it’s not possible to contribute or participate in the parts of Ubuntu which Canonical has a particularly strong interest in. Witness the fantastic work being done on both the system and the apps to bring Ubuntu to the phone and tablet. That may not be your cup of tea, but it’s tremendously motivating and exciting and energetic.

The ‘rolling release’ meme has been a popular one for years in Ubuntu. It’s one of the top requests from members of our user community. And it’s popular with Canonical team members too (who, largely, come from the community and share its values).

The problem for me is straightforward: a rolling release isn’t actually a release at all. It offers little certainty for those who need certainty. And we essentially accommodate the need for daily crack with our development releases, which have become highly usable (for developers) because of the strong commitment the Canonical and community teams made to daily quality throughout the release cycle.

So I haven’t personally given any air time to the topic of rolling releases over the years.

This year, the topic bubbled up again, and given the level of interest I supported that the core Canonical engineering team do a deep assessment of what it would actually mean, in hard pro’s and con’s, and how we might implement it, so that a straw man proposal (‘one you can poke holes in’) could be presented. Rick put forward that proposal last week. It should be clear that Rick is a strong and sincere proponent of the idea, hence the passion with which the case is made, but he is not the sole decision maker.

It’s nonsense to portray Rick’s position as a final position for Ubuntu. The TB have not weighed in,  the CC (who were briefed that the assessment was being made and that a straw man would be proposed) are still considering their perspective, and I’m not convinced either. So, for those inclined to melodrama, you may want to calm down and join the conversation.

Some unexpected findings

In the course of Rick’s team’s assessment, several interesting and (to me) unexpected findings emerged.

First, there’s real confusion around interim releases. Between 12.04 LTS and 14.04 LTS there will be three interim releases on our current approach, and lots of people will find that confusing. Should ISVs target quantal AND raring AND ssssss? In practice, we have lots of data to say they can’t and won’t. PPAs are often inconsistent between interim releases. That suggests that having an ‘edge’ release (for which PPAs would over time build up a rich source of extra software) and LTS releases may be easier on that segment of the community.

Second, we have proven the LTS point release mechanism, which brings new hardware support and new software to the LTS releases. The cloud archive, for example, brings the latest OpenStack release to 12.04 LTS, and is by far the most popular way to deploy OpenStack. Point releases have brought fresh kernels, fresh OpenStack, and fresh Unity to 12.04 LTS, and there is no reason why we could not broaden that commitment. It’s worth discussing whether that doesn’t become a better mechanism to meet the needs of people who care about a stable release.

Third, the daily quality story really has been impressive. The amazing work of a sizable quality team has transformed the widespread expectations of participants and contributors in Ubuntu – raring is really useful, every day, with little risk of unproductive hours when things go wonky. That’s grown the number of *developers* running raring, and boosted Ubuntu in other ways as a result. I’m not convinced it’s good enough for end-users, but it’s worth digging in to see how it could get there.

Some unrealistic expectations

In the commentary I’ve seen during the course of the discussion, some of the expectations expressed by stakeholders strike me as unrealistic.

Ben Collins’ perspective, which addresses the need of a PowerPC OEM, is an example. Ben is a friend and former colleague, I’d like to be supportive, but the real cost of supporting an architecture is way outside the scope of Ubuntu’s non-commercial commitments. IBM and Canonical discussed bringing Ubuntu to the PowerPC architecture some years ago and chose not to; the gap is not something Canonical will close alone. I’m delighted if Ubuntu is useful for Ben, and pretty certain it will remain the best platform for his work regardless, but we should not spend millions of dollars on that rather than cloud computing or mobile, which have a much broader impact on both society and our commercial prospects.

Some unwarranted melodrama

The sky is not falling in.

Really.

Ubuntu is a group of people who get together with common purpose. How we achieve that purpose is up to us, and everyone has a say in what they can and will contribute. Canonical’s contribution is massive. It’s simply nonsense to say that Canonical gets ‘what it wants’ more than anybody else. Hell, half the time *I* don’t get exactly what I want. It just doesn’t work that way: lots of people work hard to the best of their abilities, the result is Ubuntu.

The combination of Canonical and community is what makes that amazing. There are lots of pure community distro’s. And wow, they are full of politics, spite, frustration, venality and disappointment. Why? Because people are people, and work is hard, and collaboration is even harder. That’s nothing to do with Canonical, and everything to do with life. In fact, in most of the pure-community projects I’ve watched and participated in, the biggest meme is ‘if only we had someone that could do the heavy lifting’. Ubuntu has that in Canonical – and the combination of our joint efforts has become the most popular platform for Linux fans.

If you’ve done what you want for Ubuntu, then move on. That’s normal – there’s no need to poison the well behind you just because you want to try something else.

It’s also the case that we’ve shifted gear to leadership rather than integration.

When we started, we said we wanted to deliver the best of open source on a cadence. It was up to KDE, GNOME, XFCE to define what that was going to look like, we would just integrate and deliver (a hard problem in itself). By 2009 I was convinced that none of the existing free software communities could create an experience that could challenge the existing proprietary leaders, and so, if we were serious about the dream of a free software norm, we would have to lead.

The result is Unity, which is an experience that could become widely adopted across phones, tablets, PCs and other devices. Of course, that is a disruptive change, and has caused some members of existing communities to resent our work. I respect that others may prefer different experiences, so we remain willing to do a large (but not unlimited) amount of work to enable KDE, GNOME, and other DEs to thrive inside the broader Ubuntu umbrella. We also take steps to accommodate developers who want to support both Unity and another DE. But if we want to get beyond being a platform for hobbyists, we need to accelerate the work on Unity to keep up with Android, Chrome, Windows and Apple. And that’s more important than taking care of the needs of those who don’t share our goal of a free software norm.

A once-in-a-lifetime opportunity

Everyone that I care about in open source has a shared dream: they want free software to become the norm, not the exception. And Ubuntu is the only way I can see for that to happen, which is why I spend all my time on it, and why so many other people spend huge amounts of time on it too.

I simply have zero interest in the crowd who wants to be different. Leet. ‘Linux is supposed to be hard so it’s exclusive’ is just the dumbest thing that a smart person could say. People being people, there are of course smart people who hold that view.

What I’m really interested in is this once-in-a-lifetime opportunity to create a free and open platform that is THE LEADER across both consumer and enterprise computing.

With Ubuntu (and Unity) we have that. It’s amazing. Think about it – unlike years gone by, a free software platform is actually winning awards for innovative leadership in the categories that count: mobile, cloud. Investing your time and energy here might have a truly profound impact on the world. That’s worth digging into. Just roll your eyeballs at the 1337 crowd, roll up your sleeves, find something interesting to improve, and join in. To the extent that you can master a piece, you will get what you want. If you think the grand vision should follow your whims, you won’t.

If we work hard, and work together, Ubuntu will become a widespread platform for phones, tablets and PCs. You’ll have the satisfaction of designing, building and fixing tools that are used every day by millions of people. That’s meaningful. And it’s worth looking hard at our practices to ask the question: how best to achieve that goal? Of those practices, interim releases are just as subject to evaluation and revision as any other.

Going faster

So, rolling releases are not real releases.

But cadence is good, releases are good discipline even if they are hard. In LEAN software engineering, we have an interesting maxim: when something is hard, DO IT MORE OFTEN. Because that way you concentrate your efforts on the hard problem, master it, automate and make it easy. That’s the philosophy that underpins agile development, devops, juju and loads of other goodness.

In the web-lead world, software is moving faster than ever before. Is six months fast enough?

So I think it IS worth asking the question: can we go even faster? Can we make even MORE releases in a year? And can we automate that process to make it bulletproof for end-users?

That’s where I think we should steer the conversation on rolling releases:

• Can we make the update process from point to point really bulletproof? Upgrading today is possible, but to keep the system clean over multiple successive upgrades requires an uncommonly high level of skill with APT.
• Can we strengthen the definition of point releases in the LTS so that interim releases are obviously less relevant?
• Can we do a reasonable amount of release management on, say, MONTHLY releases that they are actual releases rather than just snapshots?

Daily quality has made the Ubuntu development release perfectly usable for developers. That’s a huge accomplishment. Now let’s think carefully about the promises we’re making end-users, and see if it isn’t time to innovate again, just as we innovated when we created Ubuntu on a six month cadence.