All the other guys are not wrong

Sunday, March 13th, 2011

The discussion in blogs and comments on collaboration and standards is really important. It’s also easy to cast as “GNOME vs Canonical vs KDE”, and that would be incorrect. My critique here is not of the body of GNOME developers, it’s of specific decisions and processes, which in my view have let GNOME down.

The reason I care is, to state the obvious, a well-functioning GNOME is important to Ubuntu and Canonical. And I don’t think we’re there. Alternatively, a well-functioning FreeDesktop.org is important, and we’re not there either.

Dave Neary, who to his credit has been trying to understand how matters reached this point, blogged his conclusions to date. They warrant a response.

He summarises:

FreeDesktop.org is broken as a standards body

That may be true today, but I think we should fix it. With Meego around, and other efforts that are lower profile, there are now even more reasons to have well specified standards for desktop interoperability. They won’t all work, but they deserve better respect and quality than they have today. So my response to Dave is: let’s fix that, rather than pretending “it’s broke so it don’t matter”.

Mark Shuttleworth doesn’t understand how GNOME works

Fortunately I’m apparently in good company because his next conclusion is GNOME is not easy to understand. Perhaps a more accurate summation would be “Gnome is not self-consistent, or deterministic, so it can often come to two quite contradictory conclusions at the same time, and Mr Shuttleworth didn’t understand the one we’d now like to promote.”

Dave mailed me to say that he’d got a “definitive” perspective on how the appindicator API’s came to be rejected. It includes pointers as to the requirements for submitting external dependencies. These are “make the case for the dependency, which should be a few sentences or so, and wait a short while for people to check it out (e.g. making sure it builds)”. Now, a reading of the correspondence around the API’s suggests that Ted and others did a lot more than the “few sentences and make sure it builds”, yet the proposal was rejected.

In addition, Dave got commentary from two members of the Gnome release team, who make these decisions. The two views disagree with one another.

I’m not sure what to think, then, about Dave’s assertion that I don’t understand Gnome. Seems the follow-on conclusion is closer to the truth.

Dave says:

[…] to get things done in GNOME, you need to talk to the right people. That means, defining your problem, and identifying the stakeholders who are also interested in that problem, and working out a solution with them (am I repeating myself?). Mark seems to want GNOME to behave like a company, so that he can get “his people” to talk to “our people” and make it happen. I think that this misunderstanding of how to wield influence within the GNOME project is a key problem.

But then again, over the years I have heard similar feedback from GNOME Mobile participants, and people in Nokia – so it’s not all Mark’s fault. As Jono says here: GNOME does have a reputation of being hard to work with for companies – no point in denying it (then again, so does the kernel, and they seem to get along fine).

Hold on a sec. There’s been ample documentation of conversations. Dave can’t even point to two stakeholder who agree with each other in the release team!

I also understand that there is an interest in putting on a good face, and not airing your dirty laundry in public (ironic, eh?) – for the past few years, the party line in Canonical has been “We love GNOME, we’re a GNOME shop” while behind the scenes there have been heartfelt conversations about the various problems which exist in GNOME & how to address them. The problem is, because these discussioons happen behind the scenes, they stay there. We never get beyond discussions, agreeing there is a problem, but never working together on a solution.

Yes, the stated line from Canonical has been exactly what Dave describes – we’ve worked hard to stay within the Gnome umbrella, even where we’ve felt that the deck was stacked against us. It’s tiring. After a year or so of being the public whipping boy for cutting commentary from competitors under the Gnome banner, a franker line is needed.

Owen Taylor, desktop lead at Red Hat and the person to whom Jon McCann referred the app indicators API decision, then weighed in. He suggests several things:


Mark argues that GNOME should be a place where we have internal competition. But his idea of internal competition seems to be competition between different end-user experiences. His entrant into the competition is Unity, an environment with a user experience designed completely in isolation from GNOME. The other entrant would, I suppose, be the GNOME 3 desktop that GNOME has created.

This competition doesn’t make sense to me: what would be left of GNOME if Unity “won” that competition? Not even the libraries are left, because every decision that is made about what goes into library should be driven by that same question “what does the user see?” No widget should go into GTK+ unless it makes sense in a GNOME application. GNOME cannot cede the user experience and still work as a project.

This is exactly why we proposed the app indicator API’s as *external* dependencies. They are capabilities which GNOME apps can take advantage of if they are around, but which are not essential if they are not there. Unity could quite easily move to the fore in GNOME, if it won this competition, just like lots of other ideas and pieces of code have started outside the core of GNOME but become essential to it.

Owen’s argument reinforces the idea (which is in my view broken) that the only idea that matter are the ones generated internally to the GNOME project (defined very tightly by folks who maintain core modules or have core responsibilities). It’s precisely this inward view that I think is leading GNOME astray.

Owen’s point that “no widget should go into Gtk if it is not needed by a GNOME application” is unlikely to be comforting to the XFCE folk, or other desktop environments which build on GNOME. If anything, it will make them feel that things in “core GNOME” are likely to be difficult to adopt and collaborate with, because their needs, apparently don’t matter.

He also says “But I’ve never seen Canonical make the leap and realize that they could actually dive in and make GNOME itself better.”.. which is rather insulting to all the people from Canonical who spend a lot of their day doing exactly that.

He talks about App Indicators, saying that “They didn’t even propose changes to core GNOME components to support application indicators.” Actually, we did, and those changes required App Indicators to be an external dependency. So we proposed that, and it was rejected. Repeat ad absurdum.

In the end, in comments, Owen says that “[It] is a common misperception [that Gnome Shell and Gnome 3 can be separated]. That somehow the work we’ve done on GNOME Shell is somehow separable from the rest of GNOME 3. The work on the GTK+ theme and the window manager theme is done on together. The work on GNOME Shell is done together work work on System Settings and on the internal gnome-settings-daemon and gnome-session components.” Well, that’s convenient. Define one piece – your piece – as critical, therefor making it above competition. I expect that Ubuntu will ship Gnome 3 perfectly well with Unity.

Also in comments, Owen points out that the work Shell developers did around messaging was done as an update to an FD.o spec. An update that AFAICT was not discussed, just amended and pushed. He says, in a triumph of understatement, “We certainly haven’t done a good job discussing the small additions to the specification we needed”.

Finally, Owen concludes that having Unity and Gnome Shell as separate desktops may be the only way forward. Seems like he’s worked hard to ensure that’s the case.

Next up, Jeff Waugh is writing a set of related blog posts. One of them walks through the app indicators timeline, and is relatively comprehensive in doing so.

Jeff draws a conclusion that we’re mistaken in feeling that App Indicators were deliberately blocked because Unity presented a risk of competition with Gnome Shell; but he draws that purely based on the timing of the conversation proposing App Indicators as an external dependency, which was four days *before* the name Unity was introduced.

Yes, that’s true. But Unity was simply the new name for work which has been ongoing since 2007: The Ubuntu Netbook Remix interface. That work was very much in the frame throughout, and it’s been suggested that it was that work which catalysed Gnome Shell in the first place.

So even though Jeff is right on the claim that app indicators were discussed *before* the Unity name was revealed, that’s not in any way material to a discussion of the motives of those who rejected app indicators. This was an API from Canonical, related to work in the Ubuntu Netbook interface, and it got rejected for reasons which are unlike any other response to a proposed external dependency.

Perhaps, in the light of changed circumstances, Jeff will change his opinion. Good commentators do.

Jeff also goes on to talk about Ted and Aurelien, who were proposing the app indicators work in GNOME and KDE respectively. KDE apps worked smoothly, Gnome rejected Ted’s proposal. Jeff says “I don’t believe the key difference here is between GNOME and KDE, it is in Canonical’s approach to engagement, and its commitment of developers to the task.” It’s worth pointing out that Ted and Aurelien were both working for the same manager under the same guidance with the same goals. Jeff draws the conclusion that Canonical could have done things differently. I would have drawn the conclusion that Gnome was less open to collaboration than KDE.

Finally, Jeff looks at the requirements for dependencies, and notes that Canonical didn’t need to engage at all, though he (and we recognised the same) says that would have caused other problems. He concludes:

Canonical barely made an effort to have libappindicator “accepted into GNOME” (which, in the context of his criticism, Mark believed was important or necessary)

As I’ve shown above, the stated requirements are a very low bar. We did that, and more, yet the App Indicator API’s were rejected. As I’ve said before, it’s bizarre that such a different standard was applied to this API, and not other API’s. The only rational explanation is that the decision is nothing to do with the API’s, and everything to do with politics. Those politics are bad for Gnome in the long run. I want Gnome to be healthy in the long run, ergo, my critique.

It did not even need to go through this process anyway, because it did not need to be an “external dependency” in order to achieve Mark’s stated goals (ie. application adoption of the API)

Unfortunately for Jeff, we’d been told in no uncertain terms that module owners and core apps were under pressure about these API’s. They wanted to see the external dependency blessed. So we proposed it. Owen says we “didn’t try to propose changes to core apps”… we did, and the external dependency was the blocker.

So the rejection of libappindicator, for all the sturm und drang, is essentially meaningless — it had no bearing on the opportunity for collaboration between Canonical and GNOME

In fact, it’s what’s left that collaboration in limbo. What to do with all the patches produced for GNOME apps that make them work with app indicators? Hmm… that would be collaboration, but the uncertainty created by the rejection as an external dependency creates a barrier to that collaboration. As Jeff says, those patches can land without any problems. But to land such a patch, after the refusal, takes some guts on the part of the maintainer. Lots have done it (thank you!) but some have said they are concerned they will be criticised for doing so.

Unity did not exist in the public sphere when libappindicator was declared irrelevant to GNOME Shell, and was not ever the reason why: (a) there wasn’t much interest in libappindicator, or (b) GNOME Shell developers decided they were on a different path

Correct, *Unity* was not the public name of the work at the time, Ubuntu Netbook Remix was.

Not proven in this part of the series, but worth noting: the person Mark specifically chose to attack, Jon McCann, did not decide to exclude libappindicator because — being a design contributor to GNOME Shell — he felt threatened by Unity. In fact, he had no part in the decision, and didn’t know Unity existed!

Jon certainly knew a great deal of work on interfaces was being done. That became branded Unity later, but the timing of the change in name is irrelevant.

Phew.

There are good faith efforts being made to bridge divides all over the show, for which I’m grateful and to which we’re contributing. My comments here are to address what I see as convenient papering over, which will not stand the test of time. It’s important – to me, to the members of the community working on Unity and Ubuntu (and there are substantial communities in both) that simplistic accusations against us are not left to stand unchallenged.

The goal – for everyone, I think – is great free software. I know we’re committed to that, and doing what we think is needed to achieve it.

Linaro at work: porting, testing, and Android

Thursday, November 11th, 2010

Congratulations to Team Linaro on their first full release yesterday. For those not yet in the know, Linaro is a collaborative forum with dedicated engineers making sure that Linux rocks on ARM (and potentially other architectures). Staffed by a combination of Canonical and new Linaro engineers, together with secondees from the major ARM silicon vendors, it’s solving the problems of fragmentation in Linux across that ecosystem and reducing the time to market for ARM devices.

Linaro uses the same cadence as Ubuntu and we’re able to collaborate on the selection, integration and debugging of key components like the kernel, toolchain, X.org (still ;-)), and hundreds of small-but-important libraries and tools in between. Team Linaro was @UDS and it was very cool to see the extent to which their sessions drew attendance from the wider Ubuntu community – I think there’s a growing interest in efficient computing across the Ubuntu landscape.

The Linaro team is pleased to announce the release of Linaro 10.11.
10.11 is the first public release that brings together the huge amount
of engineering effort that has occurred within Linaro over the past 6
months. In addition to officially supporting the TI OMAP3 (Beagle
Board and Beagle Board XM) and ARM Versatile Express platforms, the
images have been tested and verified on a total of 7 different platforms
including TI OMAP4 Panda Board, IGEPv2, Freescale iMX51 and ST-E U8500.

The advances that have happened in this cycle are numerous but include a
completely rebuilt archive using GCC 4.4.4 and the latest ARM optimised
tool chain, the Linux kernel version 2.6.35, support for
cross-compiling, a new hardware pack way of building images, 3D
acceleration improvements, u-boot enhancements and initial device tree
support, a new QA tracking structure, the list goes on.

Android in the house

The road ahead looks even more interesting. For the next cycle, the Linaro team is going to build an Android environment on the same kernel and toolchain that we collaborate on with Ubuntu. For folks building devices, picking a board that’s part of the Linaro process means you’ll be able to get either an Ubuntu-style or Android-style core environment up and running at Day 1, which should reduce time to market for everyone.

If the Linaro team pulls this off, it will mean that Linaro provides an intersection point for the majority of the consumer electronics x86 and ARM ecosystem, regardless of the end OS. I’m sure over time we’ll find more groups that are interested in joining the process, and I see no reason why they couldn’t be accommodated in this cadence-driven model.

More players, more diversity in services

It was also good to see folks from Montavista and Mentor at Linaro@UDS this year. Whether the Linaro kernel and toolchain plug into their own distros, or they start to offer their services around the Linaro/Ubuntu/Android BSP’s, the result is a healthier ecosystem with fewer snags and gotchas for device makers.

One group asked me explicitly if Linaro was a Canonical show, and I was glad to say it isn’t. Canonical can’t possibly do everything that embedded Linux needs done, but our competence in cadence and release management makes us good custodians of a public project, which is what we do with Ubuntu itself. Participation and collaboration are at the heart, and they benefit from being partnered with a commitment to delivery and deadlines. We can’t do everything in a single cycle, but we can provide a roadmap for things like kernel defragmentation, the device-tree work, enablement of an ever-increasing cross-section of the ARM ecosystem, and transitions between versions of GCC or Python or X or even Wayland. So Canonical makes a good anchor, but Linaro has room for lots of other service-providers. Having multiple companies participate in Linaro means that the products we’re all shipping get better, faster.

Testing

The Linaro team is also going to focus on repeatable, rigorous testing of the core platform in the next cycle. That harmonises nicely with our growing focus on quality in Ubuntu, and the need for better quality and testing in open source in general. I’m interested to see what tools and results the Linaro team can produce in the next six months. Open source *can* be bulletproof, but it can also degrade in quality if we don’t put the right processes in place upstream and downstream, so this is a very welcome initiative.

A distribution occupies a very specific niche in the free software ecosystem. Among other things, we need to accept some responsibility for ALL the software defects (“bugs”) that users actually experience across the entire stack. Most users don’t install their apps from upstream source tarballs, they install them from the packages provided by their distribution. So when they experience a bug, they don’t know if it’s a bug introduced by that distribution, or a bug in the underlying upstream code. They don’t know, they don’t care, and they shouldn’t have to. More often than not they will report the issue to their distribution, and the way we respond to it is important, because it represents an opportunity to make the whole ecosystem more robust.

I had a lecturer who was very opposed to the use of the term “bugs”. He said that the term “bug” was a cute-sification for “nasty biting insect”, and similarly, software defects have potentially serious consequences, so we shouldn’t treat them lightly. Bug work is serious work, and it’s one of the most important forms of contribution to the digital commons that Ubuntu can make, so I’d like to salute the extraordinary efforts of the Ubuntu Quality Assurance Team and Bug Squad. Initiatives like five-a-day are already making a huge difference to our users. As Henrik Omma says, effective bug reporting requires a diligent and professional approach, and I’ve noticed a real improvement in our community. Hopefully, we can bring the benefits of that competence to the broader free software ecosystem.

Ubuntu gets as many bugs reported against it as OpenOffice, Mozilla, Gnome, and KDE combined.The vast majority of those bugs are issues that exist in upstream tarball releases, or in Debian. Our primary goals should be to ensure that fixes we produce, and information we generate in the QA process, make their way upstream where they will benefit the broadest cross-section of the community. Separately, we want to ensure that each Ubuntu release ships without major issues, regardless of where those issues originated. We are responsible for the user experience of every line of code, even though we don’t produce every line of code.

In the month of April 2008, I found the following bug counts for large FLOSS projects:

Upstreams:Mozilla5,334

OpenOffice 1,076
Gnome 5,364
KDE 1,335
Total: 13,109
Distributions:
Ubuntu 13,064
Debian 5,103

With hindsight, April was possibly a bad choice, because it was an Ubuntu release month so there’s usually a small spike in the number of bugs filed. It would be interesting to see the stats for other distributions, and projects, over a full year. But the general picture is clear – within our family of distributions, Ubuntu carries the brunt of the load w.r.t. bug tracking, triage and patch management – not only for our users, but for a broad cross-section of the open source stack.

When I delved into the data to see how we do with pushing bugs upstream, I found a somewhat mixed picture. In many cases, we do very well indeed. We have a very good relationship with GNOME, for example, with a very high percentage of bugs appropriately forwarded to the relevant upstream bug tracker. In other projects, it’s harder to make a definitive statement. The percentage varies based on whether the Ubuntu team members have good relationships upstream, or whether there’s a person acting as an ambassador from Ubuntu to upstream (this is a great way to make a difference if you care about a specific application in Ubuntu!) or whether upstream themselves have taken an interest.

We need to improve the tools that support these kinds of cross-project conversations. Launchpad does currently allow us to track the status of a bug in many different bug trackers, and there are quite a few distributions and upstreams that are now either using Launchpad directly or exchanging data efficiently. We’ll keep working to improve the quality of exchange across the whole ecosystem, including those projects that don’t use Launchpad themselves