Precise Quality, not just for Precise

Wednesday, January 4th, 2012

I upgraded my primary laptop to Precise yesterday. Very smoooooth! Kudos to the Ubuntu team for the way they are running this cycle; their commitment to keeping the Precise Pangolin usable from opening to release as 12.04 LTS is very evident.

The three legs of our engineering practice are cadence, quality and design. For those teams which maintain their own codebases (unity, juju, bzr, lp and many more) the quality position is a easier to define, because we can make test coverage and continuous tested integration standard practices. It’s more challenging for the platform team and Ubuntu community, who integrate thousands of packages from all sorts of places into one product: Ubuntu. We’ve traditionally focused on items like security, where participation in a global security process helps us ensure Ubuntu gets world-class security support and has established a world-leading track record of security patches and proactive security.

Nevertheless, the last year has seen some amazing leaps forward in our ability to manage quality across the entire platform. In large part, that’s thanks to the leadership of Rick Spencer and Pete Graner, who made smoke-testing and benchmarking a rigorous part of the process for every change to the platform, and lead the work to make that commitment sane in practice across all the hundreds of people, inside and outside Canonical, who needed to be on board with it. And it’s thanks to tools like Jenkins and LAVA which automate the testing and reporting across a vast array of problem spaces, architectures and packages.

So we have a daily weather report for Precise, which gives you a feeling for where things stand right now, as well as tighter integration of the test suites being run by Canonical upstreams on code destined for Precise with the test harness used by the platform team integrating that work into the distribution.  I’ll take the liberty of repeating some of Rick’s core points here:

For upstreams, it boils down to “treat your trunk as sacred”. Practically, it requires:

  • There is a trunk of code bound for Ubuntu.
  • This trunk always builds automatically.
  • This trunk has tests that are always passing automatically.
  • All branches are properly reviewed for having both good tests and good implementation before merged into trunk.
  • Any branch that breaks trunk by causing automated tests to fail or causes trunk to stop building, are immediately reverted.

For Ubuntu Engineering, the responsibilities include:

  • Every maintainer in Ubuntu must have a test plan for upstream trunks that are run before uploading to the development release.
  • Tests in the test plan that are automated can be run with the help of the QA team.
  • Tests in the test plan that are manual can be run with the help of Nicholas, the new community QA Lead
  • Refrain from uploading a trunk into Ubuntu if there are serious bugs found in testing that will slow down people using the development release for testing and development.
  • Revert uploads that break Ubuntu, as there is no point in having the latest of a trunk in Ubuntu if it’s broken and just slowing everyone down.
  • Add tests to upstream projects for the Ubuntu test plan if serious bugs do get through that cause a revert.
Now that the harnesses are in place, we’re going to crank up the sensitivity of the test suite, by adding more tests and flagging more of them as critical issues for immediate resolution when they break. Key items to add next are daily tests on software center changes, and tests of the multi-monitor work that is under way for 12.04 in Unity (using some pretty magical hardware setups).
There are a variety of additional practices and processes in place too, such as testing of the dialy ISO’s, reversion of changes that cross specific thresholds of stability for specific types of users, pro-active smoke testing of archive sanity throughout the cycle, and a dedicated vanguard quality team that aim to keep velocity high for everyone despite these additional gates and checks.

This isn’t limited to Canonical team members; didrocks and the French Musketeers have built a Unity SRU testing process which should let us crowdsource perspectives on the quality improvements or regressions of changes in Unity. Ara’s ongoing work around component and system testing is giving us a very useful database of known issues at the hardware level. Work on Checkbox and related tools continues to ensure that people can contribute data and help prioritise the issues which will have the widest benefit for millions of community adopters.

Upstream quality

Where upstreams have test suites, we’re integrating those into the automated QA framework. In an ideal world, whenever a package is changed, we’d have an upstream test suite to run for that package AND for every package which depends on it. That way, we’d catch breakage in the package itself, but more importantly, we’d catch consequential damage elsewhere, which is much harder for upstreams to catch themselves.

We’re already running that program, and as upstreams start to take testing more seriously, coverage across the whole platform will improve significantly. It’s been Canonical practice to have test suites for several years, and it’s very encouraging to see other upstreams adopting TDD and at the least rigorous unit and functional testing, one at a time. Open source projects love to talk about quality – but it’s important to back that with measurable practices and data. As an example in a complex case, we run the LTP against every kernel SRU, in addition to our own kernel and hardware cert tests.

In future, it should be possible to link this to the existing daily builds of tip (we have over 500 upstreams running daily builds on Launchpad, which is fantastic). THAT would give upstreams the ability to know when commits to their tip break tests in dependent packages. It would suck a large amount of compute, but it would provide a fantastic early warning system of collisions between independent changes in diverse but related projects.

There’s a lot more we will do, by integrating Apport for crash data collection, and routing those reports through a big data sieve we should be able to identify the issues which are having the biggest impact on the most users. But that’s a blog for another day. For now, well done, team Ubuntu!

30 comments:

  1. razvi_me says: (permalink)
    January 4th, 2012 at 6:18 pm

    You are a great human being and Ubuntu/Unity is awesome!

  2. Jo-Erlend Schinstad says: (permalink)
    January 4th, 2012 at 6:30 pm

    I upgraded my primary laptop about a week ago. The news about changes in the development process hadn’t reached me. I’m very happy with Precise Pangolin so far. It’s faster and much cooler — obviously, in more sense than one. Over the years, I’ve lost some of the interest in the lower level stuff that goes on with an operating system, focusing more on what you can do with it. Nonetheless, I do want to get started as early as possible so that I can at least report bugs, etc. The problem has always been when to dare upgrading. In that regard, it’s very important that as many as possible feel comfortable upgrading their primary devices early. After all, those are the ones you spend hours with every day, which allows you to catch the glitches more easily. So this is really exciting.

    After I upgraded my laptop, I’ve had a few small issues with the upgrade manager and some other minor things. But all in all, it’s been a very positive surprise and I’m actually considering upgrading my primary workbox, even if I’m usually very conservative on that one. I am extremely impressed.

  3. Omer says: (permalink)
    January 4th, 2012 at 8:12 pm

    Hey Mark,
    You’ve mentioned in the past that you wanted a new sound theme. I know there are many more important things that you deal with on a daily basis so it has slipped through the cracks. Maybe it would be best if Canonical hired a consultant on this like how the ubuntu font was developed?

    Thanks!

  4. Effenberg0x0 says: (permalink)
    January 4th, 2012 at 9:23 pm

    That’s great. Not sure if you know it, all updates to Ubuntu pre-release software is tested by the group of regulars / testers at Ubuntu+1 section of ubuntuforums. We test, discuss, report bugs, find workarounds and now help the newly created/recognised alpha version users that are directed to us by other forum sections (that don’t know the specifics of PP). When a new version is released, most of us jump to other forum section areas to help at the usual chaos of release week, teaching other helpers about the release specifics. Although this group seems to be invisible, as it is unfortunately not recognised in any way at the forums and not even mentioned in any “official” documentation or communication, they keep on going daily, for every update, release, cycle, etc.

  5. Joey Wilson says: (permalink)
    January 5th, 2012 at 12:52 am

    I really hope Ubuntu 12.04 LTS solves some of the recent stability issues. I have been a huge Ubuntu evangelist, but 11.04 and 11.10 were so problematic for me I stopped recommending it. 10.04 and 10.10 ran great for me, but the new Unity changes now cause many problems with my graphics. Unity is also extremely slow for me, and the menu system has been a bit frustrating – too many clicks to reach the destination. I documented my frustrations in more detail on my G+ page: https://plus.google.com/101773630805931921626/posts/E8CxYqbDxUu

    My fingers are crossed that these issues are resolved in this next release so that I can go back to my old ways of recommending Ubuntu to everyone.

  6. Patrick Dickey says: (permalink)
    January 5th, 2012 at 1:33 am

    I realize that my question isn’t “platform related”, but could you have the Ubuntu One team look into some of the issues surrounding that application? Issues such as it failing to sign in with an existing account, or getting an infinite “Loading” message regardless of which tab in the Control Panel you choose. I ended up having to use man u1sdtool in order to learn how to subscribe my folders via the command line (this was in 11.10, as I couldn’t get any part of Ubuntu One to work in 12.04 so I reverted back). So, the issues have been around for at least one or two distributions–and are not due to it being in an Alpha status.

    Have a great day:)
    Patrick.

  7. adred says: (permalink)
    January 5th, 2012 at 1:46 am

    I don’t know if I tried upgrading before but the upgrade to Precise was almost flawless. There was a single firmware error which didn’t cause any problem at all and now it’s gone. Enjoying Precise so far. Thanks you!

  8. mark says: (permalink)
    January 5th, 2012 at 11:28 am

    Thanks Effenberg, I hadn’t heard of that work and it’s awesome!

  9. mark says: (permalink)
    January 5th, 2012 at 11:29 am

    Omer, I do believe sound is currently getting revamped, ask around on #ubuntu-design on freenode.

  10. Desktop_Guy says: (permalink)
    January 5th, 2012 at 12:00 pm

    Linux Marketshare are raising: http://www.omgubuntu.co.uk/2012/01/is-linux-marketshare-on-the-rise-it-seems-so/

    Ubuntu is best Linux desktop distro! go go go go go go go go!!!!!!! Linux Ubuntu is the future OS!!!

  11. Links 5/1/2012: Nginx Beats Microsoft, Alpine 2.3.3, and Hadoop 1.0 | Techrights says: (permalink)
    January 5th, 2012 at 12:20 pm

    [...] Precise Quality, not just for Precise I upgraded my primary laptop to Precise yesterday. Very smoooooth! Kudos to the Ubuntu team for the way they are running this cycle; their commitment to keeping the Precise Pangolin usable from opening to release as 12.04 LTS is very evident. [...]

  12. Adam Williamson says: (permalink)
    January 5th, 2012 at 3:16 pm

    Have you looked at our abrt tool for assisted crash reporting and analysis? It’s pretty mature these days and has neat features like the retrace server (send a partial backtrace to the retrace server and it gets filled out there, no need to install debuginfo packages locally). I expect it wouldn’t be too much work to ‘port’ to Ubuntu either. https://fedorahosted.org/abrt/

  13. Bob says: (permalink)
    January 5th, 2012 at 3:44 pm

    In your blog you say “Open source projects love to talk about quality – but it’s important to back that with measurable practices and data.” I’d like to know more about the measures that are used. Could someone point me to a website describing them? Thanks.

  14. katzenrs says: (permalink)
    January 5th, 2012 at 6:51 pm

    I read about multi-monitor enhancements?
    Well hopefully that includes the proprietary Nvidia driver also, as it is the only driver that hast the performance to run all of my lovely games but on the other hands has some serious flaws when it comes to fullscreen games now working correctly on a dual screen setup.
    Just saying there are a LOT of bugs in there…

  15. behzad says: (permalink)
    January 5th, 2012 at 7:35 pm

    thanks for all the effort, enthusiasm, energy and cash that you put into linux and open source.
    you are doing a fantastic job, keep it up!

  16. Ryan says: (permalink)
    January 5th, 2012 at 8:44 pm

    I am looking forward to the release of Precise. Oneiric was great. I thought it could never get better. But I think since 12.04 is an LTS release, you will make it happen. You talked earlier about polishing Ubuntu for Precise. I think Precise will be one of the greatest releases of Ubuntu, certainly the greatest release of the Unity desktop.

  17. Bryce Harrington says: (permalink)
    January 5th, 2012 at 9:08 pm

    @Adam, Ubuntu uses a tool called apport for doing those things. I believe Fedora looked at using it before making ABRT (https://fedoraproject.org/wiki/Releases/FeatureApport).

  18. no music, just silence says: (permalink)
    January 6th, 2012 at 2:07 am

    I saw the Ubuntu one files app for ios today. I think I understand. It’s a lucrative market, I understand you’re a businessman first, you’ve always said there would be a time limit, and no one should have to remain in a situation that frustrates them unnecessarily. May you find whatever you’re looking for by taking this new direction. I just ask that you consider doing one thing: Please, at this point as there is no longer any doubt as to the lack of GNU philosophy, stop referring to Canonicals products as Free software or GNU. I’ve tried to illustrate the differences between Free software and Open Source in the past (I’m assuming you got the book I sent you), but all I can do now is ask. It really is important to a lot of people that the concepts behind Free software are preserved. I expect I won’t be reading your blog anymore, but before I go I would like to thank you for Ubuntu as it was my very first introduction to GNU/Linux years ago. **hugs**

  19. Liam says: (permalink)
    January 6th, 2012 at 3:32 am

    Ubuntu has been my first Linux distro and I think I will stick to it forever – never change a winning team. Tested out Precise Pangolin on LiveCD and must say it’s really an over-the-top distro! Thanks Mark and all members of the Ubuntu team!

  20. Benjamin Kerensa says: (permalink)
    January 6th, 2012 at 7:39 am

    I have been running Precise as my default install on my laptop since before Alpha 1 release and I continue to see improvements as uploads occur. There are still quite a handful of “devious” bugs that I hope will get sorted out before Alpha 2 such as Software Sources not allowing selection of different mirror and software-center crashing periodically.

    I think 12.04 LTS is going to shine and be precisely the kind of stability that will quell the Unity naysayers.

  21. Marcos Rey says: (permalink)
    January 6th, 2012 at 8:32 am

    My only hope is that Unity 3d desktop, could be faster in 12.04 than in ubuntu 11.10 and 11.04. Because until now it’s one of the slowlest and heavy desktops, and now i use lxde on mi laptop, also the unity 2d is buggy.

  22. Marian Vasile says: (permalink)
    January 6th, 2012 at 11:01 am

    I just wait to happen, because I am very tired because of the orange square bug in unity 11.10, maybe the LTS will fix this annoing issue; in fact I had to move to kubuntu to avoid this bug (https://bugs.launchpad.net/compiz-plugins-main/+bug/875557), but I miss unity interface.

  23. Timo says: (permalink)
    January 6th, 2012 at 1:57 pm

    Adam: Ubuntu has had a tool for that for over five years (apport), that said I don’t know the history behind creating abrt instead of using apport in Fedora..

  24. silian says: (permalink)
    January 6th, 2012 at 6:21 pm

    Ryan wrote, …thought it could never get better… ROFL, this is what I thought when 8.04 was released, but Mark, Canonical team, developers and community surprised me each upgrade again. Ubuntu is a miracle because of the people building it. Thank you! I also very much appreciate the transparency of communicating background processes.

    @Marian Vasile, the bug is fixed; besides, in the meantime you could’ve only disabled grid-plugin.

  25. faycel says: (permalink)
    January 6th, 2012 at 10:40 pm

    Thanks MARK and all members of the Ubuntu team y did very good job :-)

  26. Skype in Ubuntu Precise alpha | Callum Macdonald says: (permalink)
    January 8th, 2012 at 4:38 pm

    [...] night I installed Ubuntu Precise Pangolin 12.04. I was inspired by Mark Shuttleworth’s post and figured because this is an LTS release, it might be ok to upgrade this early, instead of going [...]

  27. Duncan Murray says: (permalink)
    January 8th, 2012 at 7:48 pm

    Hi Mark, I’m looking forward to 12.04 release, as currently using your last LTS with much satisfaction for my main computer.

    I was just wondering if you’ve had any problems at all getting unity accepted over gnome 2 in the corporate environment? Do you find the hardware is generally capable to run it, and that prospective customers aren’t put off by the new interface?

    I don’t think I’m personally sold yet on Unity or Gnome 3 (although I’ve been testing both) – and am happier with gnome 2, but then I’m someone who uses workspaces a lot, and I find that is the main failing point of both of the newer desktops.

    Wishing you all the best for 2012 and 12.04 – and will be excited to see what ubuntu will be doing this year.

    All best,

    Duncan

  28. Stefano says: (permalink)
    January 9th, 2012 at 3:02 pm

    @ Marian Vasile: the bug you mention is solved already, look at the end of the launchpad page you linked..

  29. Testing Required: Unity PPA | jonobacon@home says: (permalink)
    January 12th, 2012 at 8:53 am

    [...] from the ppa. Which means that YOU can help too! I’m bootstrapping this process with the French Musketeers to ensure everything is correct and ready for the next release. How to help there will be [...]

  30. Christian says: (permalink)
    January 13th, 2012 at 5:36 pm

    Mark,

    at first I want to thank you for bringing a free and easy to use operating system like Ubuntu to the masses. Ubuntu has great potencial but the main reason many people won’t use it, is all the unsupported hardware. Unfortunately, Dell is the only manufacturer I know of, trying to make their computers compatible with Linux. I think the next big thing after Unity should be to support as much of todays hardware as possible. This will be the key to really bring Ubuntu to a wider audience.

    So you said your primary laptop runs very well with Precise, my question is: What is your primary laptop? A Dell?

    All the best,
    Christian