Mark Shuttleworth | here be dragons

It’s been two weeks since Rick Spencer made the case for a rolling release approach in Ubuntu. Having a rolling release is one of the very top suggestions from the hardcore Ubuntu user community, and after years of it being mooted by all and sundry I thought it deserved the deep consideration that Rick and his team, who represent most of Canonical’s direct contributions to Ubuntu, brought to the analysis.

It’s obviously not helpful to have mass hysteria break out when ideas like this get floated, so I would like to thank everyone who calmly provided feedback on the proposal, and blow a fat raspberry at those of you who felt obliged to mount soapboxes and opine on The End Of the World As We Know It. Sensible people the world over will appreciate the dilemma at being asked to take user feedback seriously, and being accused of unilateralism when exploring options.

Change is warranted. If we want to deliver on our mission, we have to be willing to stare controversy in the face and do the right thing anyway, recognising that we won’t know if it’s the right thing until much later, and for most of the intervening time, friends and enemies alike will go various degrees of apoplectic. Our best defense against getting it wrong is to have a strong meritocracy, which I think we do. That means letting people like Rick, who have earned their leadership roles, explore controversial territory.

So, where do we stand? And where do I stand? What’s the next step?

What makes this conversation hard is the sheer scale of the Ubuntu ecosystem, all of which is profoundly affected by any change. Here are the things I think we need to optimise for, and the observations that I think we should structure our thinking around:

Releases are good discipline, cadence is valuable.

Releases, even interim releases, create value for parts of the Ubuntu ecosystem that are important. They allow us to get more widespread feedback on decisions made in that cycle – what’s working, what’s not working. Interestingly, in the analysis that played into Rick’s proposal, we found that very few institutional users depend on extended support of the interim releases. Those who care about support tend to use the LTS releases and LTS point releases.

Release management detracts from development time, and should be balanced against the amount of use that release gets.

While reaffirming our interest in releases, I think we established that the amount of time spend developing in a cycle versus spent doing release management is currently out of whack with the amount to which people actually DEPEND on that release management, for interim releases, on the desktop. On the server, we found that the interim releases are quite heavily used in the cloud, less so on physical metal.

Daily quality has raised the game dramatically for tip / trunk / devel users, and addresses the Rolling Release need.

There’s widespread support for the statement that ‘developers can and should use the daily development release’. The processes that have been put in place make it much more reliable for folks who want to track development, either as a contributor to Ubuntu or as someone who ships software for Ubuntu and wants to know what’s happening on the latest release, to use Ubuntu throughout the development cycle. For those of you not aware, uploads to the edge get published in a special ‘pocket’, and only moved into the edge if they don’t generate any alarms from people who are on the VERY BLEEDING EDGE. So you can use Raring (without that bleeding edge pocket) and get daily updates that are almost certain not to bork you.  There is a real community that WANTS a rolling release, and the daily development release of Ubuntu satisfies this need already.

LTS point releases are a great new enhancement to the LTS concept.

On a regular basis, the LTS release gets a point update which includes access to a new, current kernel (supporting new hardware without regressing the old hardware on the previous kernel, which remains supported), new OpenStack (via the Cloud Archive), and various other elements. I think we could build on this to enhance the LTS with newer and better versions of the core UX (Unity) as long as we don’t push those users through a major transition in the process (Unity/Qt, anybody? ;-)).

Separating platform from apps would enhance agility.

Currently, we make one giant release of the platform and ALL APPS. That means an enormous amount of interdependence, and an enormous bottleneck that depends largely on a single community to line everything up at once. If we narrowed the scope of the platform, we would raise the quality of the platform. Quite possibly, we could place the responsibility for apps on the developers that love them, giving users access to newer versions of those apps if (and only if) the development communities behind them want to do that and believe it is supportable.

Phew.

That’s what I observed from all the discussion that ensued from Rick’s proposal.

Here’s a new straw man proposal. Note – this is still just a proposal. I will ask the TB to respond to this one, since it incorporates both elements of Rick’s team’s analysis and feedback from wider circles.

Updated Ubuntu Release Management proposal

In order to go even faster as the leading free software platform, meet the needs of both our external users and internal communities (Unity, Canonical, Kubuntu, Xubuntu and many many others) and prepare for a wider role in personal computing, Ubuntu is considering:

1. Strengthening the LTS point releases.

Our end-user community will be better served by higher-quality LTS releases that get additional, contained update during the first two years of their existence (i.e. as long as they are the latest LTS). Updates to the LTS in each point release might include:

• addition of newer kernels as options (not invalidating prior kernels). The original LTS kernel would be supported for the full duration of the LTS, interim kernels would be supported until the subsequent LTS, and the next LTS kernel would be supported on the prior LTS for teh length of that LTS too. The kernel team should provide a more detailed updated straw man proposal to the TB along these lines.
• optional newer versions of major, fast-moving and important platform components. For example, during the life of 12.04 LTS we are providing as optional updates newer versions of OpenStack, so it is always possible to deploy 12.04 LTS with the latest OpenStack in a supported configuration, and upgrade to newer versions of OpenStack in existing clouds without upgrading from 12.04 LTS itself.
• required upgrades to newer versions of platform components, as long as those do not break key APIs. For example, we know that the 13.04 Unity is much faster than the 12.04 Unity, and it might be possible and valuable to backport it as an update.

2. Reducing the amount of release management, and duration of support, for interim releases.

Very few end users depend on 18 months support for interim releases. The proposal is to reduce the support for interim releases to 7 months, thereby providing constant support for those who stay on the latest interim release, or any supported LTS releases. Our working assumption is that the latest interim release is used by folks who will be involved, even if tangentially, in the making of Ubuntu, and LTS releases will be used by those who purely consume it.

3. Designating the tip of development as a Rolling Release.

Building on current Daily Quality practices, to make the tip of the development release generally useful as a ‘daily driver’ for developers who want to track Ubuntu progress without taking significant risk with their primary laptop. We would ask the TB to evaluate whether it’s worth changing our archive naming and management conventions so that one release, say ‘raring’, stays the tip release so that there is no need to ‘upgrade’ when releases are actually published. We would encourage PPA developers to target the edge release, so that we don’t fragment the ‘extras’ collection across interim releases.

That is all.

The ‘rolling release’ meme has been a popular one for years in Ubuntu. It’s one of the top requests from members of our user community. And it’s popular with Canonical team members too (who, largely, come from the community and share its values).

The problem for me is straightforward: a rolling release isn’t actually a release at all. It offers little certainty for those who need certainty. And we essentially accommodate the need for daily crack with our development releases, which have become highly usable (for developers) because of the strong commitment the Canonical and community teams made to daily quality throughout the release cycle.

So I haven’t personally given any air time to the topic of rolling releases over the years.

This year, the topic bubbled up again, and given the level of interest I supported that the core Canonical engineering team do a deep assessment of what it would actually mean, in hard pro’s and con’s, and how we might implement it, so that a straw man proposal (‘one you can poke holes in’) could be presented. Rick put forward that proposal last week. It should be clear that Rick is a strong and sincere proponent of the idea, hence the passion with which the case is made, but he is not the sole decision maker.

It’s nonsense to portray Rick’s position as a final position for Ubuntu. The TB have not weighed in,  the CC (who were briefed that the assessment was being made and that a straw man would be proposed) are still considering their perspective, and I’m not convinced either. So, for those inclined to melodrama, you may want to calm down and join the conversation.

Some unexpected findings

In the course of Rick’s team’s assessment, several interesting and (to me) unexpected findings emerged.

First, there’s real confusion around interim releases. Between 12.04 LTS and 14.04 LTS there will be three interim releases on our current approach, and lots of people will find that confusing. Should ISVs target quantal AND raring AND ssssss? In practice, we have lots of data to say they can’t and won’t. PPAs are often inconsistent between interim releases. That suggests that having an ‘edge’ release (for which PPAs would over time build up a rich source of extra software) and LTS releases may be easier on that segment of the community.

Second, we have proven the LTS point release mechanism, which brings new hardware support and new software to the LTS releases. The cloud archive, for example, brings the latest OpenStack release to 12.04 LTS, and is by far the most popular way to deploy OpenStack. Point releases have brought fresh kernels, fresh OpenStack, and fresh Unity to 12.04 LTS, and there is no reason why we could not broaden that commitment. It’s worth discussing whether that doesn’t become a better mechanism to meet the needs of people who care about a stable release.

Third, the daily quality story really has been impressive. The amazing work of a sizable quality team has transformed the widespread expectations of participants and contributors in Ubuntu – raring is really useful, every day, with little risk of unproductive hours when things go wonky. That’s grown the number of *developers* running raring, and boosted Ubuntu in other ways as a result. I’m not convinced it’s good enough for end-users, but it’s worth digging in to see how it could get there.

Some unrealistic expectations

In the commentary I’ve seen during the course of the discussion, some of the expectations expressed by stakeholders strike me as unrealistic.

Ben Collins’ perspective, which addresses the need of a PowerPC OEM, is an example. Ben is a friend and former colleague, I’d like to be supportive, but the real cost of supporting an architecture is way outside the scope of Ubuntu’s non-commercial commitments. IBM and Canonical discussed bringing Ubuntu to the PowerPC architecture some years ago and chose not to; the gap is not something Canonical will close alone. I’m delighted if Ubuntu is useful for Ben, and pretty certain it will remain the best platform for his work regardless, but we should not spend millions of dollars on that rather than cloud computing or mobile, which have a much broader impact on both society and our commercial prospects.

Some unwarranted melodrama

The sky is not falling in.

Really.

Ubuntu is a group of people who get together with common purpose. How we achieve that purpose is up to us, and everyone has a say in what they can and will contribute. Canonical’s contribution is massive. It’s simply nonsense to say that Canonical gets ‘what it wants’ more than anybody else. Hell, half the time *I* don’t get exactly what I want. It just doesn’t work that way: lots of people work hard to the best of their abilities, the result is Ubuntu.

The combination of Canonical and community is what makes that amazing. There are lots of pure community distro’s. And wow, they are full of politics, spite, frustration, venality and disappointment. Why? Because people are people, and work is hard, and collaboration is even harder. That’s nothing to do with Canonical, and everything to do with life. In fact, in most of the pure-community projects I’ve watched and participated in, the biggest meme is ‘if only we had someone that could do the heavy lifting’. Ubuntu has that in Canonical – and the combination of our joint efforts has become the most popular platform for Linux fans.

If you’ve done what you want for Ubuntu, then move on. That’s normal – there’s no need to poison the well behind you just because you want to try something else.

It’s also the case that we’ve shifted gear to leadership rather than integration.

When we started, we said we wanted to deliver the best of open source on a cadence. It was up to KDE, GNOME, XFCE to define what that was going to look like, we would just integrate and deliver (a hard problem in itself). By 2009 I was convinced that none of the existing free software communities could create an experience that could challenge the existing proprietary leaders, and so, if we were serious about the dream of a free software norm, we would have to lead.

The result is Unity, which is an experience that could become widely adopted across phones, tablets, PCs and other devices. Of course, that is a disruptive change, and has caused some members of existing communities to resent our work. I respect that others may prefer different experiences, so we remain willing to do a large (but not unlimited) amount of work to enable KDE, GNOME, and other DEs to thrive inside the broader Ubuntu umbrella. We also take steps to accommodate developers who want to support both Unity and another DE. But if we want to get beyond being a platform for hobbyists, we need to accelerate the work on Unity to keep up with Android, Chrome, Windows and Apple. And that’s more important than taking care of the needs of those who don’t share our goal of a free software norm.

A once-in-a-lifetime opportunity

Everyone that I care about in open source has a shared dream: they want free software to become the norm, not the exception. And Ubuntu is the only way I can see for that to happen, which is why I spend all my time on it, and why so many other people spend huge amounts of time on it too.

I simply have zero interest in the crowd who wants to be different. Leet. ‘Linux is supposed to be hard so it’s exclusive’ is just the dumbest thing that a smart person could say. People being people, there are of course smart people who hold that view.

What I’m really interested in is this once-in-a-lifetime opportunity to create a free and open platform that is THE LEADER across both consumer and enterprise computing.

With Ubuntu (and Unity) we have that. It’s amazing. Think about it – unlike years gone by, a free software platform is actually winning awards for innovative leadership in the categories that count: mobile, cloud. Investing your time and energy here might have a truly profound impact on the world. That’s worth digging into. Just roll your eyeballs at the 1337 crowd, roll up your sleeves, find something interesting to improve, and join in. To the extent that you can master a piece, you will get what you want. If you think the grand vision should follow your whims, you won’t.

If we work hard, and work together, Ubuntu will become a widespread platform for phones, tablets and PCs. You’ll have the satisfaction of designing, building and fixing tools that are used every day by millions of people. That’s meaningful. And it’s worth looking hard at our practices to ask the question: how best to achieve that goal? Of those practices, interim releases are just as subject to evaluation and revision as any other.

Going faster

So, rolling releases are not real releases.

But cadence is good, releases are good discipline even if they are hard. In LEAN software engineering, we have an interesting maxim: when something is hard, DO IT MORE OFTEN. Because that way you concentrate your efforts on the hard problem, master it, automate and make it easy. That’s the philosophy that underpins agile development, devops, juju and loads of other goodness.

In the web-lead world, software is moving faster than ever before. Is six months fast enough?

So I think it IS worth asking the question: can we go even faster? Can we make even MORE releases in a year? And can we automate that process to make it bulletproof for end-users?

That’s where I think we should steer the conversation on rolling releases:

• Can we make the update process from point to point really bulletproof? Upgrading today is possible, but to keep the system clean over multiple successive upgrades requires an uncommonly high level of skill with APT.
• Can we strengthen the definition of point releases in the LTS so that interim releases are obviously less relevant?
• Can we do a reasonable amount of release management on, say, MONTHLY releases that they are actual releases rather than just snapshots?

Daily quality has made the Ubuntu development release perfectly usable for developers. That’s a huge accomplishment. Now let’s think carefully about the promises we’re making end-users, and see if it isn’t time to innovate again, just as we innovated when we created Ubuntu on a six month cadence.

Even though the idea of formal alignment between the freezes of Debian and Ubuntu didn’t hold, there has been some good practical collaboration between the maintainers of key subsystems. There are real benefits to this, because maintainers have a much more fruitful basis for sharing patches when they are looking at the same underlying version.

Harmonization for Ubuntu 10.04 LTS and Debian Squeeze

I think this is where we stand now:

I’m sure there are inaccuracies, please help me keep this up to date, sabdfl on freenode is the best way to reach me. The RHEL and SLES numbers are third-hand, so up-to-date information would be appreciated.

The actual release dates of Ubuntu LTS and Debian will vary of course, because of different priorities. And there’s no requirement that the same base version be used for every major component – there may well be differences allowing for different approaches. But where we do have it, we’ll be able to collaborate much more effectively on bug fixes for key upstream pieces. If a lot of distributions pick the same base upstream version, it greatly increases the value of extended shared maintenance and point releases of that upstream.

Why every two years?

Two years is a compromise between those who want 1 year releases for better support of cutting-edge hardware and those who want 7 year releases so their software stack doesn’t change before their job description does ;-).

A whole-year multiple has several advantages. It means we can schedule the processes that are needed for collaboration at the same time of year whenever we need them – unlike 1.5 or 2.5 year cycles. Three years was felt to be too long for hardware support. Two years is perceived to be the Goldilocks Cadence – just right.

What are the criteria for choosing a common base version?

In both the Ubuntu and Debian cases, we’ll be making a release that we support for many years. So be looked for versions of key upstreams that will pass the test of time. Sometimes, that means they can’t be too old, because they’ll be completely obsolete or unmaintainable in the life of the release. And sometimes that means they can’t be too young. In general, it would be better to be reviewing code that is already out there. But there are also lots of upstreams that do a credible job of release management, so we could commit to shipping a version that is not yet released, based on the reputation of the community it’s coming from.

What if there’s no agreement on a particular kernel, or X or component-foo?

We will almost certainly diverge on some components, and that’s quite OK. This is about finding opportunities to do a better job for upstreams and for users, not about forcing any distro to make a particular choice. If anyone feels its more important to them to use a particular version than another, they’ll do that.

Open invitations

It’s really helpful to have upstreams and other distributions participate in this process.

If you’re an upstream, kick off a thread in your mailing list or forums about this. Upstreams don’t need to do anything different if they don’t want to, we’ll still just make the best choices we can. But embracing a two year cadence is the best way you have to be sure which versions of your software are going to be in millions of hands in the future – it’s a great opportunity to influence how your users will experience your work.

Of course, we’d also like to have more distributions at the table. There’s no binding commitment needed – collaboration is opportunistic. But without participating in the conversation one can’t spot those opportunities! If you represent a distribution and are interested, then please feel free to contact me, or Matt Zimmerman, or anyone on the Debian release management team about it.

I think this is a big win for the free software community. Many upstreams have said “we’d really like to help deliver a great stable release, but which distro should we arrange that around?” Upstreams should not have to play favourites with distributions, and it should be no more work to support 10 distributions as to support one. If we can grow the number of distributions that embrace this cadence, the question becomes moot – upstreams can plan around that cycle knowing that many distributions will deliver their work straight to users.

Six-month cycles are great. Now let’s talk about meta-cycles: broader release cycles for major work. I’m very interested in a cross-community conversation about this, so will sketch out some ideas and then encourage people from as many different free software communities as possible to comment here. I’ll summarise those comments in a follow-up post, which will no doubt be a lot wiser and more insightful than this one 🙂

Background: building on the best practice of cadence

The practice of regular releases, and now time-based releases, is becoming widespread within the free software community. From the kernel, to GNOME and KDE, to X, and distributions like Ubuntu, Fedora, the idea of a regular, predictable cycle is now better understood and widely embraced. Many smarter folks than me have articulated the benefits of such a cadence: energising the whole community, REALLY releasing early and often, shaking out good and bad code, rapid course correction.

There has been some experimentation with different cycles. I’m involved in projects that have 1 month, 3 month and 6 month cycles, for different reasons. They all work well.

..but addressing the needs of the longer term

But there are also weaknesses to the six-month cycle:

• It’s hard to communicate to your users that you have made some definitive, significant change,
• It’s hard to know what to support for how long, you obviously can’t support every release indefinitely.

I think there is growing insight into this, on both sides of the original “cadence” debate.

A tale of two philosophies, perhaps with a unifying theory

A few years back, at AKademy in Glasgow, I was in the middle of a great discussion about six month cycles. I was a passionate advocate of the six month cycle, and interested in the arguments against it. The strongest one was the challenge of making “big bold moves”.

“You just can’t do some things in six months” was the common refrain. “You need to be able to take a longer view, and you need a plan for the big change.” There was a lot of criticism of GNOME for having “stagnated” due to the inability to make tough choices inside a six month cycle (and with perpetual backward compatibility guarantees). Such discussions often become ideological, with folks on one side saying “you can evolve anything incrementally” and others saying “you need to make a clean break”.

At the time of course, KDE was gearing up for KDE 4.0, a significant and bold move indeed. And GNOME was quite happily making its regular releases. When the KDE release arrived, it was beautiful, but it had real issues. Somewhat predictably, the regular-release crowd said “see, we told you, BIG releases don’t work”. But since then KDE has knuckled down with regular, well managed, incremental improvements, and KDE is looking fantastic. Suddenly, the big bold move comes into focus, and the benefits become clear. Well done KDE 🙂

On the other side of the fence, GNOME is now more aware of the limitations of indefinite regular releases. I’m very excited by the zest and spirit with which the “user experience MATTERS” campaign is being taken up in Gnome, there’s a real desire to deliver breakthrough changes. This kicked off at the excellent Gnome usability summit last year, which I enjoyed and which quite a few of the Canonical usability and design folks participated in, and the fruits of that are shaping up in things like the new Activities shell.

But it’s become clear that a change like this represents a definitive break with the past, and might take more than a single six month release to achieve. And most important of all, that this is an opportunity to make other, significant, distinctive changes. A break with the past. A big bold move. And so there’s been a series of conversations about how to “do a 3.0”, in effect, how to break with the tradition of incremental change, in order to make this vision possible.

It strikes me that both projects are converging on a common set of ideas:

• Rapid, predictable releases are super for keeping energy high and code evolving cleanly and efficiently, they keep people out of a deathmarch scenario, they tighten things up and they allow for a shakeout of good and bad ideas in a coordinated, managed fashion.

• Big releases are energising too. They are motivational, they make people feel like it’s possible to change anything, they release a lot of creative energy and generate a lot of healthy discussion. But they can be a bit messy, things can break on the way, and that’s a healthy thing.

Anecdotally, there are other interesting stories that feed into this.

Recently, the Python community decided that Python 3.0 will be a shorter cycle than the usual Python release. The 3.0 release is serving to shake out the ideas and code for 3.x, but it won’t be heavily adopted itself so it doesn’t really make sense to put a lot of effort into maintaining it – get it out there, have a short cycle, and then invest in quality for the next cycle because 3.x will be much more heavily used than 3.0. This reminds me a lot of KDE 4.0.

So, I’m interesting in gathering opinions, challenges, ideas, commitments, hypotheses etc about the idea of meta-cycles and how we could organise ourselves to make the most of this. I suspect that we can define a best practice, which includes regular releases for continuous improvement on a predictable schedule, and ALSO defines a good practice for how MAJOR releases fit into that cadence, in a well structured and manageable fashion. I think we can draw on the experiences in both GNOME and KDE, and other projects, to shape that thinking.

This is important for distributions, too

The major distributions tend to have big releases, as well as more frequent releases. RHEL has Fedora, Ubuntu makes LTS releases, Debian takes cadence to its logical continuous integration extreme with Sid and Testing :-).

When we did Ubuntu 6.06 LTS we said we’d do another LTS in “2 to 3 years”. When we did 8.04 LTS we said that the benefits of predictability for LTS’s are such that it would be good to say in advance when the next LTS would be. I said I would like that to be 10.04 LTS, a major cycle of 2 years, unless the opportunity came up to coordinate major releases with one or two other major distributions – Debian, Suse or Red Hat.

I’ve spoken with folks at Novell, and it doesn’t look like there’s an opportunity to coordinate for the moment. In conversations with Steve McIntyre, the current Debian Project Leader, we’ve identified an interesting opportunity to collaborate. Debian is aiming for an 18 month cycle, which would put their next release around October 2010, which would be the same time as the Ubuntu 10.10 release. Potentially, then, we could defer the Ubuntu LTS till 10.10, coordinating and collaborating with the Debian project for a release with very similar choices of core infrastructure. That would make sharing patches a lot easier, a benefit both ways. Since there will be a lot of folks from Ubuntu at Debconf, and hopefully a number of Debian developers at UDS in Barcelona in May, we will have good opportunities to examine this opportunity in detail. If there is goodwill, excitement and broad commitment to such an idea from Debian, I would be willing to promote the idea of deferring the LTS from 10.04 to 10.10 LTS.

Questions and options

So, what would the “best practices” of a meta-cycle be? What sorts of things should be considered in planning for these meta-cycles? What problems do they cause, and how are those best addressed? How do short term (3 month, 6 month) cycles fit into a broader meta-cycle? Asking these questions across multiple communities will help test the ideas and generate better ones.

What’s a good name for such a meta-cycle? Meta-cycle seems…. very meta.

Is it true that the “first release of the major cycle” (KDE 4.0, Python 3.0) is best done as a short cycle that does not get long term attention? Are there counter-examples, or better examples, of this?

Which release in the major cycle is best for long term support? Is it the last of the releases before major new changes begin (Python 2.6? GNOME 2.28?) or is it the result of a couple of quick iterations on the X.0 release (KDE 4.2? GNOME 3.2?) Does it matter? I do believe that it’s worthwhile for upstreams to support an occasional release for a longer time than usual, because that’s what large organisations want.

Is a whole-year cycle beneficial? For example, is 2.5 years a good idea? Personally, I think not. I think conferences and holidays tend to happen at the same time of the year every year and it’s much, much easier to think in terms of whole number of year cycles. But in informal conversations about this, some people have said 18 months, others have said 30 months (2.5 years) might suit them. I think they’re craaaazy, what do you think?

If it’s 2 years or 3 years, which is better for you? Hardware guys tend to say “2 years!” to get the benefit of new hardware, sooner. Software guys say “3 years!” so that they have less change to deal with. Personally, I am in the 2 years camp, but I think it’s more important to be aligned with the pulse of the community, and if GNOME / KDE / Kernel wanted 3 years, I’d be happy to go with it.

How do the meta-cycles of different projects come together? Does it make sense to have low-level, hardware-related things on a different cycle to high-level, user visible things? Or does it make more sense to have a rhythm of life that’s shared from the top to the bottom of the stack?

Would it make more sense to stagger long term releases based on how they depend on one another, like GCC then X then OpenOffice? Or would it make more sense to have them all follow the same meta-cycle, so that we get big breakage across the stack at times, and big stability across the stack at others?

Are any projects out there already doing this?

Is there any established theory or practice for this?

A cross-community conversation

If you’ve read this far, thank you! Please do comment, and if you are interested then please do take up these questions in the communities that you care about, and bring the results of those discussions back here as comments. I’m pretty sure that we can take the art of software to a whole new level if we take advantage of the fact that we are NOT proprietary, and this is one of the key ways we can do it.

I had the opportunity to present at the Linux Symposium on Friday, and talked further about my hope that we can improve the coordination and cadence of the entire free software stack. I tried to present both the obvious benefits and the controversies the idea has thrown up.

Afterwards, a number of people came up to talk about it further, with generally positive feedback.

Christopher Curtis, for example, emailed to say that the idea of economic clustering in the motor car industry goes far further than the location of car dealerships. He writes:

Firstly, every car maker releases their new models at about the same time. Each car maker has similar products – economy, sedan, light truck. They copy each other prolifically. Eventually, they all adopt a certain baseline – seatbelts, bumpers, airbags, anti-lock brakes. Yet they compete fiercely (OnStar from GM; Microsoft Sync from Ford) and people remain brand loyal. This isn’t going to change in the Linux world. Even better, relations like Debian->Ubuntu match car maker relations like Toyota->Lexus.

I agree with him wholeheartedly. Linux distributions and car manufacturers are very similar: we’re selling products that reach the same basic audience (there are niche specialists in real-time or embedded or regional markets) with a similar range (desktop, workstation, server, mobile), and we use many of the same components just as the motor industry uses common suppliers. That commonality and coordination benefits the motor industry, and yet individual brands and products retain their identity.

Let’s do a small thought experiment. Can you name, for the last major enterprise release of your favourite distribution, the specific major versions of kernel, gcc, X, GNOME, KDE, OpenOffice.org or Mozilla that were shipped? And can you say whether those major versions were the same or different to any of the enterprise releases of Ubuntu, SLES, Debian, or RHEL which shipped at roughly the same time? I’m willing to bet that any particular customer would say that they can’t remember either which versions were involved, or how those stacked up against the competition, and don’t care either. So looking backwards, differences in versions weren’t a customer-differentiating item.  We can do the same thought experiment looking forwards. WHAT IF you knew that the next long-term supported releases of Ubuntu, Debian, Red Hat and Novell Linux would all have the same major versions of kernel, GCC, X, GNOME, KDE, OO.o and Mozilla. Would that make a major difference for you? I’m willing to bet not – that from a customer view, folks who prefer X will still prefer X. A person who prefers Red Hat will stick with Red Hat. But from a developer view, would that make it easier to collaborate? Dramatically so.

Another member of the audience came up to talk about the fashion industry. That’s also converged on a highly coordinated model – fabrics and technologies “release” first, then designers introduce their work simultaneously at fashion shows around the world. “Spring 2009” sees new collections from all the major houses, many re-using similar ideas or components. That hasn’t hurt their industry, rather it helps to build awareness amongst the potential audience.

The ultimate laboratory, nature, has also adopted release coordination. Anil Somayaji, who was in the audience for the keynote, subsequently emailed this:

Basically, trees of a given species will synchronize their seed releases in time and in amount, potentially to overwhelm predators and to coordinate with environmental conditions. In effect, synchronized seed releases is a strategy for competitors to work together to ensure that they all have the best chance of succeeding. In a similar fashion, if free software were to “release its seeds” in a synchronized fashion (with similar types of software or distributions having coordinated schedules, but software in different niches having different schedules), it might maximize the chances of all of their survival and prosperity.

There’s no doubt in my mind that the stronger the “pulse” we are able to create, by coordinating the freezes and releases of major pieces of the free software stack, the stronger our impact on the global software market will be, and the better for all companies – from MySQL to Alfresco, from Zimbra to OBM, from Red Hat to Ubuntu.

An update on the long term plans for Ubuntu release management. 8.04 LTS represented a very significant step forward in our release management thinking. To the best of my knowledge there has never been an “enterprise platform” release delivered exactly on schedule, to the day, in any proprietary or Linux OS. Not only did it prove that we could execute an LTS release in the standard 6-month timeframe, but it showed that we could commit to such an LTS the cycle beforehand. Kudos to the technical decision-makers, the release managers, and the whole community who aligned our efforts with that goal.

As a result, we can commit that the next LTS release of Ubuntu will be 10.04 LTS, in April 2010.

This represents one of the most extraordinary, and to me somewhat unexpected, benefits of free software to those who deploy it. Most people would assume that precise release management would depend on having total control of all the moving parts – and hence only be possible in a proprietary setting. Microsoft writes (almost) every line of code in Windows, so you would think they would be able to set, and hit, a precise target date for delivery. But in fact the reverse is true –  free software distributions or OSV’s can provide much better assurances with regard to delivery dates than proprietary OSV’s, because we can focus on the critical role of component selection, integration, testing, patch management and distribution rather than the pieces which upstream projects are better able to handle – core component feature development. This is in my mind a very compelling reason for distributions to focus on distribution – that’s the one thing they do which the upstreams don’t, so they need to invest heavily in that in order to serve as the most efficient conduit of upstream’s work.

We also committed, for the first time, to a regular set of point releases for 8.04 LTS. These will start three months after the LTS, and be repeated every six months until the next LTS is out. These point releases will include support for new hardware as well as rolling up all the updates published in that series to date. So a fresh install of a point release will work on newer hardware and will also not require a big download of additional updates.

Gerry Carr at Canonical put together this diagram which describes the release management plan very nicely:

The Ubuntu team does an amazing job of ensuring that one can update from release to release, and from LTS release to LTS release directly, too. I’m very proud to be part of this community! With the addition of some capability to support newer hardware in LTS releases, I think we are doing our part in the free software community – helping to deliver the excellent work of thousands of other teams, from kernel.org to GNOME and KDE, safely to a huge audience.

There’s one thing that could convince me to change the date of the next Ubuntu LTS: the opportunity to collaborate with the other, large distributions on a coordinated major / minor release cycle. If two out of three of Red Hat (RHEL), Novell (SLES) and Debian are willing to agree in advance on a date to the nearest month, and thereby on a combination of kernel, compiler toolchain, GNOME/KDE, X and OpenOffice versions, and agree to a six-month and 2-3 year long term cycle, then I would happily realign Ubuntu’s short and long-term cycles around that. I think the benefits of this sort of alignment to users, upstreams and the distributions themselves would be enormous. I’ll write more about this idea in due course, for now let’s just call it my dream of true free software syncronicity.