Rigor and its results

Friday, December 20th, 2013

Perhaps the biggest change in Ubuntu since 12.04 LTS has been our shift, under Rick’s leadership, towards rigorous, highly automated, test-based QA across all of Ubuntu – server, desktop and mobile.

And what I love about the process is:

  • it’s completely transparent – check out http://ci.ubuntu.com/ and drill down to see the individual test runs
  • it spans every product – server, desktop and mobile, across a wide and growing range of hardware
  • the team takes it absolutely seriously and “stops the line” to fix issues on a regular basis

This is the result of two years hard work by an amazing team – at the package testing and system test level – to help us raise the bar for free software platforms.

This is what the 64-bit x86 server test run looks like today:

Automated testing of the Ubuntu Server platform gives us quick feedback on breaking changes.

Automated testing of the Ubuntu Server platform gives us quick feedback on breaking changes.

 

Over time, thanks to contributions of tests by community, partners and Canonical folks, the number of tests has grown substantially. In the mobile environment we run over 400 smoke tests on every build of every image:

 

Automated test results on a Nexus device.

Automated test results on a Nexus device.

 

In addition to the system image testing you see here, there is a growing portfolio of package-level tests, and processes that test changes both for problems inside the modified package AND for packages that depend on it. So increasingly, we are able to pick up on a problem before it spreads to any developer desktops that are tracking the tip of development.

Testing makes us smarter

It’s significantly more challenging to create test harnesses than code itself. Building this capability exercised our best contributors for the better part of two years; every team has had to figure out how to “get meta” on the parts of Ubuntu they care about. And in the process, we come to a deeper understanding of what it is that users care about, how our platform fits together, and the magic that lives inside the kernel that enables much of this work to happen at scale and in an automated fashion. Grappling with hard problems is like training; the more you do it, the better you understand what’s possible.

Testing helps us go faster

It’s a curious phenomenon that taking time to work on the stuff around the code helps to get the code done faster. But in something as large and complex as a free software platform, change is both your friend and your enemy. Every week thousands of changes flow into Ubuntu from a huge range of sources, and it’s impossible for any on person to anticipate the consequences of every change in advance. Having a rigorous, automated test framework tells us immediately when a change causes a problem somewhere else. Greater confidence that problems will be caught lets us move faster with changes, knowing we can either revert them quickly or stop the line to concentrate everyone’s attention on the issue when it touches a broad swath of the platform.

Testing brings more eyeballs

I run the tip of development – Trusty today – because I can trust the team to spot a problem before it affects me almost every time. That gives me a better view on how Ubuntu is evolving day by day and the ability to ask questions when they are relevant, not right before release. For developers at upstreams or in companies where Trusty is going to be a key platform, the ability to exercise it personally is a huge advantage; you can directly influence the trajectory best if you know where things stand at any given moment. So more rigour translates into more eyeballs which translates into a better result for everybody down the line.

 

We expect to ship our first Ubuntu mobile devices in 2014, and this initiative gives me confidence that we can bring new features and capabilities and improvements to those users fast. And that’s one of the things that makes Ubuntu great.

Two weeks with Mir

Tuesday, July 9th, 2013

Mir has been running smoothly on my laptop for two weeks now. It’s an all-Intel Dell XPS, so the driver stack on Ubuntu is very clean, but I’m nonetheless surprised that the system feels *smoother* than it did pre-Mir. It might be coincidence, Saucy is changing pretty fast and new versions of X and Compiz have both landed while I’ve had Mir running. But watching top suggests that both Xorg and Compiz are using less memory and fewer CPU cycles under Mir than they were with X handling the hardware directly.

Talking with the Mir team, they say others have seen the same thing, and they attribute it to more efficient buffering of requests on the way to the hardware. YMMV but it’s definitely worth trying. I have one glitch which catches me out – Chromium triggers an issue in the graphics stack which freezes the display. Pressing Alt-F1 unfreezes it (it causes Compiz to invoke something which twiddles the right bits to bring the GPU back from it’s daze). I’m told that will get sorted trivially in a coming update to the PPA.

The overall impression I have is that Mir has delivered what we hoped. Perhaps it had the advantage of being able to study what went before – SurfaceFlinger, Wayland, X – and perhaps also the advantage of looking at things through the perspective of a mobile lens, where performance and efficiency are a primary concern, but regardless, it’s lean, efficient, high quality and brings benefits even when running a legacy X stack.

We take a lot of flack for every decision we make in Ubuntu, because so many people are affected. But I remind the team – failure to act when action is needed is as much a failure as taking the wrong kind of action might be. We have a responsibility to our users to explore difficult territory. Many difficult choices in the past are the bedrock of our usefulness to a very wide audience today.

Building a graphics stack is not a decision made lightly – it’s not an afternoon’s hacking. The decision was taken based on a careful consideration of technical factors. We need a graphics stack that works reliably across a very wide range of hardware, that performs predictably, that provides a consistent quality of user experience on many different desktop environments.

Of course, there is competition out there, which we think is healthy. I believe Mir will be able to evolve faster than the competition, in part because of the key differences and choices made now. For example, rather than a rigid protocol that can only be extended, Mir provides an API. The implementation of that API can evolve over time for better performance, while it’s difficult to do the same if you are speaking a fixed protocol. We saw with X how awkward life becomes when you have a fixed legacy protocol and negotiate over extensions which themselves might be versioned. Others have articulated the technical rationale for the Mir approach better than I can, read what they have to say if you’re interested in the ways in which Mir is different, the lessons learned from other stacks, and the benefits we see from the architecture of Mir.

Providing Mir as an option is easy. Mir is a very focused part of the stack, it has far fewer tentacles and knock-on consequences for app developers than, say, the init system, which means we should be able to work with a very tight group of communities to get great performance. It’s much easier for a distro to engage with Mir than to move to SystemD, for example; instead of an impact on every package, there is a need to coordinate in just a few packages for great results. We’ve had a very positive experience working with the Qt and WebKit communities, for example, so we know those apps will absolutely fly and talk Mir natively. Good upstreams want their code widely useful, so I’ve no doubt that the relevant toolkits will take patches that provide enhanced capabilities on Mir when available. And we also know that we can deliver a high-performance X stack on Mir, which means any application that talks X, or any desktop environment that talks X, will perform just as well with Mir, and have smoother transitions in and out thanks to the system compositor capabilities that Mir provides.

On Ubuntu, we’re committed that every desktop environment perform well with Mir, either under X or directly. We didn’t press the ‘GO’ button on Mir until we were satisfied that the whole Ubuntu community, and other distributions, could easily benefit from the advantages of a leaner, cleaner graphics stack. We’re busy optimising performance for X now so that every app and every desktop environment will work really well in 13.10 under Mir, without having to make any changes. And we’re taking patches from people who want Mir to support capabilities they need for native, super-fast Mir access. Distributions should be able to provide Mir as an option for their users to experiment with very easily – the patch to X is very small (less than 500 lines). For now, if you want to try it, the easiest way to do so is via the Ubuntu PPA. It will land in 13.10 just as soon as our QA and release teams are happy that its ready for very widespread testing.

It’s been two weeks since Rick Spencer made the case for a rolling release approach in Ubuntu. Having a rolling release is one of the very top suggestions from the hardcore Ubuntu user community, and after years of it being mooted by all and sundry I thought it deserved the deep consideration that Rick and his team, who represent most of Canonical’s direct contributions to Ubuntu, brought to the analysis.

It’s obviously not helpful to have mass hysteria break out when ideas like this get floated, so I would like to thank everyone who calmly provided feedback on the proposal, and blow a fat raspberry at those of you who felt obliged to mount soapboxes and opine on The End Of the World As We Know It. Sensible people the world over will appreciate the dilemma at being asked to take user feedback seriously, and being accused of unilateralism when exploring options.

Change is warranted. If we want to deliver on our mission, we have to be willing to stare controversy in the face and do the right thing anyway, recognising that we won’t know if it’s the right thing until much later, and for most of the intervening time, friends and enemies alike will go various degrees of apoplectic. Our best defense against getting it wrong is to have a strong meritocracy, which I think we do. That means letting people like Rick, who have earned their leadership roles, explore controversial territory.

So, where do we stand? And where do I stand? What’s the next step?

What makes this conversation hard is the sheer scale of the Ubuntu ecosystem, all of which is profoundly affected by any change. Here are the things I think we need to optimise for, and the observations that I think we should structure our thinking around:

Releases are good discipline, cadence is valuable.

Releases, even interim releases, create value for parts of the Ubuntu ecosystem that are important. They allow us to get more widespread feedback on decisions made in that cycle – what’s working, what’s not working. Interestingly, in the analysis that played into Rick’s proposal, we found that very few institutional users depend on extended support of the interim releases. Those who care about support tend to use the LTS releases and LTS point releases.

Release management detracts from development time, and should be balanced against the amount of use that release gets.

While reaffirming our interest in releases, I think we established that the amount of time spend developing in a cycle versus spent doing release management is currently out of whack with the amount to which people actually DEPEND on that release management, for interim releases, on the desktop. On the server, we found that the interim releases are quite heavily used in the cloud, less so on physical metal.

Daily quality has raised the game dramatically for tip / trunk / devel users, and addresses the Rolling Release need.

There’s widespread support for the statement that ‘developers can and should use the daily development release’. The processes that have been put in place make it much more reliable for folks who want to track development, either as a contributor to Ubuntu or as someone who ships software for Ubuntu and wants to know what’s happening on the latest release, to use Ubuntu throughout the development cycle. For those of you not aware, uploads to the edge get published in a special ‘pocket’, and only moved into the edge if they don’t generate any alarms from people who are on the VERY BLEEDING EDGE. So you can use Raring (without that bleeding edge pocket) and get daily updates that are almost certain not to bork you.  There is a real community that WANTS a rolling release, and the daily development release of Ubuntu satisfies this need already.

LTS point releases are a great new enhancement to the LTS concept.

On a regular basis, the LTS release gets a point update which includes access to a new, current kernel (supporting new hardware without regressing the old hardware on the previous kernel, which remains supported), new OpenStack (via the Cloud Archive), and various other elements. I think we could build on this to enhance the LTS with newer and better versions of the core UX (Unity) as long as we don’t push those users through a major transition in the process (Unity/Qt, anybody? ;-)).

Separating platform from apps would enhance agility.

Currently, we make one giant release of the platform and ALL APPS. That means an enormous amount of interdependence, and an enormous bottleneck that depends largely on a single community to line everything up at once. If we narrowed the scope of the platform, we would raise the quality of the platform. Quite possibly, we could place the responsibility for apps on the developers that love them, giving users access to newer versions of those apps if (and only if) the development communities behind them want to do that and believe it is supportable.

Phew.

That’s what I observed from all the discussion that ensued from Rick’s proposal.

Here’s a new straw man proposal. Note – this is still just a proposal. I will ask the TB to respond to this one, since it incorporates both elements of Rick’s team’s analysis and feedback from wider circles.

Updated Ubuntu Release Management proposal

In order to go even faster as the leading free software platform, meet the needs of both our external users and internal communities (Unity, Canonical, Kubuntu, Xubuntu and many many others) and prepare for a wider role in personal computing, Ubuntu is considering:

1. Strengthening the LTS point releases.

Our end-user community will be better served by higher-quality LTS releases that get additional, contained update during the first two years of their existence (i.e. as long as they are the latest LTS). Updates to the LTS in each point release might include:

  • addition of newer kernels as options (not invalidating prior kernels). The original LTS kernel would be supported for the full duration of the LTS, interim kernels would be supported until the subsequent LTS, and the next LTS kernel would be supported on the prior LTS for teh length of that LTS too. The kernel team should provide a more detailed updated straw man proposal to the TB along these lines.
  • optional newer versions of major, fast-moving and important platform components. For example, during the life of 12.04 LTS we are providing as optional updates newer versions of OpenStack, so it is always possible to deploy 12.04 LTS with the latest OpenStack in a supported configuration, and upgrade to newer versions of OpenStack in existing clouds without upgrading from 12.04 LTS itself.
  • required upgrades to newer versions of platform components, as long as those do not break key APIs. For example, we know that the 13.04 Unity is much faster than the 12.04 Unity, and it might be possible and valuable to backport it as an update.

2. Reducing the amount of release management, and duration of support, for interim releases.

Very few end users depend on 18 months support for interim releases. The proposal is to reduce the support for interim releases to 7 months, thereby providing constant support for those who stay on the latest interim release, or any supported LTS releases. Our working assumption is that the latest interim release is used by folks who will be involved, even if tangentially, in the making of Ubuntu, and LTS releases will be used by those who purely consume it.

3. Designating the tip of development as a Rolling Release.

Building on current Daily Quality practices, to make the tip of the development release generally useful as a ‘daily driver’ for developers who want to track Ubuntu progress without taking significant risk with their primary laptop. We would ask the TB to evaluate whether it’s worth changing our archive naming and management conventions so that one release, say ‘raring’, stays the tip release so that there is no need to ‘upgrade’ when releases are actually published. We would encourage PPA developers to target the edge release, so that we don’t fragment the ‘extras’ collection across interim releases.

 

That is all.