Mark Shuttleworth | here be dragons

Perhaps the biggest change in Ubuntu since 12.04 LTS has been our shift, under Rick’s leadership, towards rigorous, highly automated, test-based QA across all of Ubuntu – server, desktop and mobile.

And what I love about the process is:

• it’s completely transparent – check out http://ci.ubuntu.com/ and drill down to see the individual test runs
• it spans every product – server, desktop and mobile, across a wide and growing range of hardware
• the team takes it absolutely seriously and “stops the line” to fix issues on a regular basis

This is the result of two years hard work by an amazing team – at the package testing and system test level – to help us raise the bar for free software platforms.

This is what the 64-bit x86 server test run looks like today:

Automated testing of the Ubuntu Server platform gives us quick feedback on breaking changes.

Over time, thanks to contributions of tests by community, partners and Canonical folks, the number of tests has grown substantially. In the mobile environment we run over 400 smoke tests on every build of every image:

Automated test results on a Nexus device.

In addition to the system image testing you see here, there is a growing portfolio of package-level tests, and processes that test changes both for problems inside the modified package AND for packages that depend on it. So increasingly, we are able to pick up on a problem before it spreads to any developer desktops that are tracking the tip of development.

Testing makes us smarter

It’s significantly more challenging to create test harnesses than code itself. Building this capability exercised our best contributors for the better part of two years; every team has had to figure out how to “get meta” on the parts of Ubuntu they care about. And in the process, we come to a deeper understanding of what it is that users care about, how our platform fits together, and the magic that lives inside the kernel that enables much of this work to happen at scale and in an automated fashion. Grappling with hard problems is like training; the more you do it, the better you understand what’s possible.

Testing helps us go faster

It’s a curious phenomenon that taking time to work on the stuff around the code helps to get the code done faster. But in something as large and complex as a free software platform, change is both your friend and your enemy. Every week thousands of changes flow into Ubuntu from a huge range of sources, and it’s impossible for any on person to anticipate the consequences of every change in advance. Having a rigorous, automated test framework tells us immediately when a change causes a problem somewhere else. Greater confidence that problems will be caught lets us move faster with changes, knowing we can either revert them quickly or stop the line to concentrate everyone’s attention on the issue when it touches a broad swath of the platform.

Testing brings more eyeballs

I run the tip of development – Trusty today – because I can trust the team to spot a problem before it affects me almost every time. That gives me a better view on how Ubuntu is evolving day by day and the ability to ask questions when they are relevant, not right before release. For developers at upstreams or in companies where Trusty is going to be a key platform, the ability to exercise it personally is a huge advantage; you can directly influence the trajectory best if you know where things stand at any given moment. So more rigour translates into more eyeballs which translates into a better result for everybody down the line.

We expect to ship our first Ubuntu mobile devices in 2014, and this initiative gives me confidence that we can bring new features and capabilities and improvements to those users fast. And that’s one of the things that makes Ubuntu great.