Mark Shuttleworth | here be dragons

One of the tough choices VCS designers make is “what do we REALLY care about”. If you can eliminate some use cases, you can make the tool better for the other use cases. So, for example, the Git guys choose not to care too much about annotate. By design, annotate is slow on Git, because by letting go of that they get it to be super-fast in the use cases they care about. And that’s a very reasonable position to take.

My focus today is lossiness, and I’m making the case for starting out a project using tools which are lossless, rather than tools which discard useful information in the name of achieving performance that’s only necessary for the very largest projects.

It’s a bit like saying “shoot your pictures in RAW format, because you can always convert to JPEG and downscale resolution for Flickr, but you can’t always get your top-quality images back from a low-res JPEG”.

When you choose a starting VCS, know that you are not making your final choice of tools. Projects who started with CVS have moved to SVN and then to Bitkeeper and then to something else. Converting is often a painful process, sometimes so painful that people opt to throw away history rather than try and convert properly. We’ll see new generations of tools over the next decade, and the capability of machines and the network will change, so of course your optimal choice of tools will change accordingly.

Initially, projects do best if they choose a tool which makes it as easy to migrate to another tool, as possible. Migrating is a little bit like converting from JPEG to PNG, or PNG to GIF. Or PNG to JPEG2000. You really want to be in the situation where your current format has as much of the detail as possible, so that your conversion can be as clean and as comprehensive as possible. Of course, that comes at a price, typically in performance. If you shoot in RAW, you get fewer frames on a memory stick. So you have to ask yourself “will this bite me?”. And it turns out, that for 99% of photographers, you can get SO MANY photos on a 1GB memory stick, even in RAW mode, that the slower performance is worth trading for the higher quality. The only professional photographers I know who shoot in JPEG are the guys who shoot 3-4000 pictures in an event, and publish them instantly to the web, with no emphasis on image quality because they are not to sort of pics anyone will blow up as a poster.

What’s the coding equivalent?

Well, you are starting a free software project. You will have somewhere between 50 and 500 files in your project initially, it will take a while before you have more than 5,000 files. During that time, you need performance to be good enough. And you want to make sure that, if you need to migrate, you have captured as much of your history in detail so that your conversion can be as easy, and as rich and complete, as possible.

I’ve watched people try to convert CVS to SVN, and it’s a nightmare, because CVS never recorded details that SVN needs, such as which file-specific changes are a consistent set. It’s all interpolation, guesswork, voodoo and ultimately painful work that results often enough in people capitulating, throwing history away and just doing a fresh start in SVN. What a shame.

The Bazaar guys, I think, thought about this a lot. It’s another reason the perfect rename tracking is so important. You can convert a Bazaar tree to Git trivially, whenever you want to, if you need to scale past 10,000 files up to 100,000 files with blazing performance. In the process, you’ll lose the renaming information. But going the other way is not so simple, because Git never recorded that information in the first place. You need interpolation and an unfortunate goat under a full moon, and even then there’s no guarantee. You chose a lossy tool, you lost the renaming data as you used it, you can’t get that data back.

Now, performance is important, but “good enough performance” is the threshold we should aim for in order to get as much out of other use cases as possible. If my tool is lossless, and still gives me a “status” in less than a heartbeat, which Bazaar does up to about 7,000 files, then I have perfectly adequate performance and perfectly lossless recording. If my project grows to the point where Bazaar’s performance is not good enough, I can convert to any of the other systems and lose ONLY the data that I choose to lose in my selection of new tool. And perhaps, by then, Git has gained perfect renaming support, so I can get perfect renaming AND blazing performance. But I made the smart choice by starting in RAW mode.

Now, there are projects out there for which the optimisations and tradeoffs made for Git are necessary. If you want to see what those tradeoffs are, watch Linus describe Git here. But the projects which immediately need to make those tradeoffs are quite unusual – they are not multiplatform, they need extraordinary performance from the beginning, and they are willing to lose renaming data and have slow annotate in order to achieve that. X, OpenSolaris, the Linux kernel… those are hardly representative of the typical free software project.

Those projects, though are also the folks who’ve spoken loudest about version control, because they have the scale and resources to do detailed assessments. But we should recognise that their findings are filtered through the unique lenses of their own constraints, and don’t let that perspective colour the decision for a project that does not operate under those constraints.

What’s good enough performance? Well, I like to think in terms of “heartbeat time”. If the major operations which I have to do regularly (several times in an hour) take less than a heartbeat, then I don’t ever feel like I’m waiting. Things which happen 3-5 times in a day can take a bit longer, up to a minute, and those fit with regular workbreaks that I would take anyhow to clear my head for the next phase of work, or rest my aching fingers.
In summary – I think new and smaller (<10,000 files) projects should care more about correctness, completeness and experience in their choice of VCS tools. Performance is important, but perfectly adequate if it takes less than a heartbeat to do the things you do regularly while working on your code. Until you really have to lose them, don’t discard the ability to work across multiple platforms (lots of free software projects have more users on Windows than on Linux), don’t discard perfect renames, don’t opt for “lossy over lossless” just because another project which might be awesomely cool but has totally different requirements from yours, did so.

I’ve had quite a lot of positive email feedback on my posting about on renaming as the killer app of distributed version control. So I thought it would be interesting to delve into this subject in more detail. I’ll blog over the next couple of months, starting tomorrow, about the things I think we need from this set of tools – whether they be Git, Darcs, Mercurial, Monotone or Bazaar.

First, to clear something up, Ubuntu selected Bazaar based on our assessment of what’s needed to build a great VCS for the free software community. Because of our work with Ubuntu, we know that what is important is the full spectrum of projects, not just the kernel, or X, or OpenOffice. It’s big and large projects, Linux and Windows projects, C and Python projects, Perl and Scheme projects… the best tools for us are the ones that work well across a broad range of projects, even if those are not the ones that are optimal for a particular project (in the way that Git works brilliantly for the kernel, because its optimisations suit that use case well, it’s a single-platform single-workflow super-optimised approach).

I’ve reviewed our choice of Bazaar in Ubuntu a couple of times, when projects like OpenSolaris and X made other choices, and in each case been satisfied that it’s still the best project for our needs. But we’re not tied to it, we could move to a different one. Canonical has no commercial interest in Bazaar (it’s ALL GPL software) and no cunning secret plans to launch a proprietary VCS based on it. We integrated Bazaar into Launchpad because Bazaar was our preferred VCS, but Bazaar could just as well be integrated into SourceForge and Collab since it’s free code.

So, what I’m articulating here is a set of values and principles – the things we find important and the rationale for our decisions – rather than a ra-ra for a particular tool. Bazaar itself doesn’t meet all of my requirements, but right now it’s the closest tool for the full spectrum of work we do.

Tomorrow, I’ll start with some commentary on why “lossless” tools are a better starting point than lossy tools, for projects that have that luxury.

This sort of discovery, I guess, is why I wanted to be a physicist. Alas, after two days at the amazing CERN when I was 18 I was pretty sure I wasn’t clever enough to do that, and so pursued other interests into IT and business and space. But I still get a thrill out of living vicariously in a life that involves some sort of particle accelerator. What an incredible rush for the scientists involved.

Also very glad to have exited the “prime number products are hard to factor so it helps if you generated the beasties in the first place” business.

Much has been written about Microsoft’s allegation of patent infringements in Linux (by which I’m sure they mean GNU/Linux ). I don’t think Microsoft is the real threat, and in fact, I think Microsoft and the Linux community will actually end up fighting on the same side of this issue.

I’m in favour of patents in general, but not software or business method patents. I’ll blog separately some day about why that’s the case, but for the moment I’ll just state for the record my view that software patents hinder, rather than help, innovation in the software industry.

And I’m pretty certain that, within a few years, Microsoft themselves will be strong advocates against software patents. Why? Because Microsoft is irrevocably committed to shipping new software every year, and software patents represent landmines in their roadmap which they are going to step on, like it or not, with increasing regularity. They can’t sit on the sidelines of the software game – they actually have to ship new products. And every time they do that, they risk stepping on a patent landmine.

They are a perfect target – they have deep pockets, and they have no option but to negotiate a settlement, or go to court, when confronted with a patent suit.

Microsoft already spends a huge amount of money on patent settlements (far, far more than they could hope to realise through patent licensing of their own portfolio). That number will creep upwards until it’s abundantly clear to them that they would be better off if software patents were history.

In short, Microsoft will lose a patent trench war if they start one, and I’m sure that cooler heads in Redmond know that.

But let’s step back from the coal-face for a second. I have high regard for Microsoft. They produce some amazing software, and they made software much cheaper than it ever was before they were around. Many people at Microsoft are motivated by a similar ideal to one we have in Ubuntu: to empower people for the digital era. Of course, we differ widely on many aspects of the implementation of that ideal, but my point is that Microsoft is actually committed to the same game that we free software people are committed to: building things which people use every day.

So, Microsoft is not the real patent threat to Linux. The real threat to Linux is the same as the real threat to Microsoft, and that is a patent suit from a person or company that is NOT actually building software, but has filed patents on ideas that the GNU project and Microsoft are equally likely to be implementing.

Yes, Nathan, I’m looking at you!

As they say in Hollywood, where there’s a hit there’s a writ. And Linux is a hit. We should expect a patent lawsuit against Linux, some time in the next decade.

There are three legs to IP law: copyright, trademark and patents. I expect a definitive suit associated with each of them. SCO stepped up on the copyright front, and that’s nearly dealt with now. A trademark-based suit is harder to envisage, because Linus and others did the smart thing and established clear ownership of the “Linux” trademark a while ago. The best-practice trademark framework for free software is still evolving, and there will probably be a suit or two, but none that could threaten the continued development of free software. And the third leg is patent law. I’m certain someone will sue somebody else about Linux on patent grounds, but it’s less likely to be Microsoft (starting a trench war) and more likely to be a litigant who only holds IP and doesn’t actually get involved in the business of software.

It will be a small company, possibly just a holding company, that has a single patent or small portfolio, and goes after people selling Linux-based devices.

Now, the wrong response to this problem is to label pure IP holders as “patent trolls”. While I dislike software patents, I deeply dislike the characterisation of pure IP holders as “patent trolls”. They are only following the rules laid out in law, and making the most of a bad system; they are not intrinsically bad themselves. Yes, Nathan, all is forgiven . One of the high ideals of the patent system is to provide a way for eccentric genius inventors to have brilliant insights in industries where they don’t have any market power, but where their outsider-perspective leads them to some important innovation that escaped the insiders. Ask anyone on the street if they think patents are good, and they will say, in pretty much any language, “yes, inventors should be compensated for their insights”. The so-called “trolls” are nothing more than inventors with VC funding. Good for them. The people who call them trolls are usually large, incumbent players who cross-license their patent portfolios with other incumbents to form a nice, cosy oligopoly. “Trolling” is the practice of interrupting that comfortable and predictably profitable arrangement. It’s hard to feel any sympathy for the incumbents at all when you look at it that way.

So it’s not the patent-holders who are the problem, it’s the patent system.

What to do about it?

Well, there are lots of groups that are actively engaged in education and policy discussion around patent reform. Get involved! I recently joined the FFII: Foundation for a Free Information Infrastructure, which is doing excellent work in Europe in this regard. Canonical sponsored the EUPACO II conference, which brought together folks from across the spectrum to discuss patent reform. And Canonical also recently joined the Open Invention Network, which establishes a Linux patent pool as a defensive measure against an attack from an incumbent player. You can find a way to become part of the conversation, too. Help to build better understanding about the real dynamics of software innovation and competition. We need to get consensus from the industry – including Microsoft, though it may be a bit soon for them – that software patents are a bad thing for society.

My message of support for Ms Machado has touched a nerve, most strongly amongst free software advocates who live in Venezuela.

Every country will have its own culture and way of doing things, and we should pay great respect to the choices and decisions of that country. It is a tragic thing to impose ones own cultural, religious or political views on people who see things differently. That tragedy has played out far too many times – from Apartheid, to the Holocaust, to the invasion of Iraq in recent history, to the acts of the Conquistadors centuries ago. It shows up when a new government renames the streets and cities of the old government, which renamed them from the previous government. We lose our own identity when we lose the voice of history, even if it is a history of which we are ashamed. It also shows up in the homogenization of global culture, with McDonalds and Disney turning the rich culture of the world into large swathes of barren desert. I am very sensitive to the beauty of the cultures that I’ve been privileged to experience in depth – South Africa, Russia, England, America. And I find it sad when one culture arrogantly suppresses another. I believe in letting people make their own choices. The future belongs to those who embrace global thinking without losing their identity and their culture.

At its largest, grandest level, “making choices” is what democracy is all about. However, sometimes the illusion of democracy is used to give legitimacy to choices that were not, at all, democratic.

In Zimbabwe, for example, we have a government that is in power “democratically” because of the systematic culture of fear that was created every time people expressed an interest in making a different choice. I cannot therefor pay much respect to the idea that the government of Zimbabwe is a true reflection of the cultural choices of Zimbabweans.

In such cases, we are obliged to question the decisions made by governments who claim to hold power by democratic mandate, when in fact they hold it by brute force. They may make some good claims and have some noble ideals, but the foundation of their authority is rotten, and it’s highly unlikely that much good will come of it for the long term.

I’m not going to comment directly on the policies of Mr Chavez. Frankly, I’m not qualified to speak on the details of his administration. But I will say that my experience of countries and governance, across continents and decades, has taught me the value of certain key principles:

First, that human nature is unchanging across the world and across time. This, as they say, is why history rhymes with itself. We make the same mistakes, we inspire ourselves to fix them, rinse and repeat. It’s human nature that makes absolute power corrupt absolutely. And its human nature to seek additional power. It’s rare to find someone who will create checks and balances on themselves. This is most eloquently described in the early writings of the American constitutional authors, who sought to “pit ambition against ambition”, and create checks and balances in society, so that neither the authorities, nor the judges, nor the media, could dominate the decisions we make for ourselves.

Second, that the presumption of innocence until the proof of guilt is a vital choice in the maintenance of a free society. In a world where even good countries can elect bad governments, we cannot let the unchallenged word of a government, any government, be sufficient to silence and stifle the lives of their citizens. I find it equally disturbing that American citizens can be locked up without access to attorneys in confidence, and that Zimbabwean opposition members can be arrested and held without charge for long periods. I also find it equally disturbing that residents of the United Kingdom can find themselves in Guantanamo Bay, on what is clearly flimsy or false evidence, without the UK fighting for their release or impartial trial. I am neither for Mr Bush, nor Mr Mugabe, nor Mr Blair, I am simply for the presumption of innocence until an impartial trial finds one guilty.

Third, that freedom of speech is essential for a healthy society. This is a freedom which we cannot take for granted. There is constantly a desire on the part of those in power to reduce the volume of criticism they must face. We have to constantly remind ourselves that those in authority have chosen to play a public role, and they must accept a level of public accountability and criticism, even from people who may have a personal agenda. Of course, not all speech is truth, and conspiracies often arise which seek to use the media to spread misinformation. But we are all better off when multiple viewpoints can be expressed. I’m no believe in media infallibility – we’ve seen very bad journalism from the biggest media networks in the world, for example when they get “embedded” in a controlled fashion into armies of war. But I’m a big believer in allowing calm voices to be heard, globally.

These principles are not written in the laws of physics – we create them in society, and we must defend them. They cannot be taken for granted, even in countries like the USA, which have them written into their constitutional DNA. Since they are a choice that society makes, and since society is reborn in each generation, they are a choice that society must make, and remake, constantly. Sometimes, we fail. Usually, we fail for fear when we are confronted by a perceived threat to security, or for greed when we are presented with the opportunity to benefit ourselves at great cost to others. And it as at times like that, when there is great stress, noise, fear, anger and shouting, that it is most important for calm voices to be heard.

At times like these, we are our own worst enemy. We hear what we want to hear. It is painful to hear that one might be wrong, that one’s hero might have flaws, that one’s leaders might not be all that we wished them to be. The awful truth of the media is that it pays to tell people what they want to hear, much more than it pays to tell people what they need to hear, and so society can whip itself into a frenzy of mistaken greed or fear or anger, and make poor decisions.

It takes great courage to speak out, when these basic principles are at risk. In a free society, there is nevertheless pressure to conform, to stay with the herd. In a society that is not free, one speaks out at some considerable personal cost to life and liberty. I salute those who do.

I read today of the renewed efforts of the Venezuelan authorities to clamp down on Sumate and their leaders, in particular Maria Corina Machado. Most recently they prevented her from attending a World Economic Forum event.

One of the privileges of working in the free software community is the interaction between different groups trying to bring together social and economic change. People like Maria are inspiring leaders, because they devote themselves to a cause much greater than any one person’s life, but in the process they sacrifice many of the comforts that many of us take for granted. It would be much easier to watch from the sidelines, emigrate, or simply ignore the situation.

I know that the Ubuntu community is very active in Venezuela and I hope they will not also some day face repression. It seems the country is on a knife-edge, facing tough decisions that will have a major impact on the quality of life of citizens there for decades.

Well, that didn’t take long. Ars Technica is reporting that further vulnerabilities in the HD DVD content protection system have been uncovered. As I noted previously, any DRM system that depends on offline key distribution will be cracked. This latest vulnerability is one step closer to the complete dismantling of the HD DVD protection system.

How long before these guys ask the question: “what do our customers want”? From experience, 5-7 years.

One of the very interesting issues-du-jour is the interaction between the three “legs” of “intellectual property”. Traditionally, those three are copyrights, patents and trademarks, and they have quite different laws and contractual precedents that are associated with them.

Recently, however, I’ve observed an increase in the cross-talk between them.

Classically, “software freedom” was about the copyright license associated with the code. But patents and trademarks are now being brought into the mix. For example, the discussion around Mozilla’s trademark policy was directly linking the concept of “freedom” to trademark policy as much as code copyright license. And much of the very hard debate in the GPLv3 process is about linkages between copyright license and relevant patents. And like it or not, the GPL is widely considered the reference implementation of freedom so GPLv3′s approach will be, for many, definitive on the subject.

In the Ubuntu community we’ve recently gone through a process to agree a trademark policy. This was recently approved by the Community Council, and the final draft is here:

http://www.ubuntu.com/aboutus/trademarkpolicy

We’ve tried to strike a balance that keeps the trademarks of Ubuntu meaningful (i.e. if it says Ubuntu, it really is Ubuntu) but also recognizes the fact that Ubuntu is a shared work, in which many different participants of our community make a personal investment, and which they should have the right to share. So we’ve made explicit the idea of a remix – a reworking of Ubuntu that addresses the needs of a specific community (could be national, could be an industry like medical or educational) but preserves the key things that people would expect from Ubuntu, like hardware support and certification.

I’m sure this isn’t the last word on the subject, but I hope it’s a useful contribution to the debate, and would welcome other projects adopting similar licenses. For that reason, our trademark license is published under the Creative Commons Sharealike with Attribution license (CC-BY-SA).

There are some ideas that are broken, but attractive enough to some people that they are doomed to be tried again and again.

DRM is one of them.

I was thrilled to see recently that the processing key for *all* HD discs produced to date has been discovered and published. I expect this to lead to the complete unraveling of the Blu-Ray and HD-DVD content protection schemes before even 1% of the potential market for those players has been reached. Good news indeed, because it may inspire the people who setup such schemes to reconsider.

We’ve been here before. The DVD-CSS encryption system was cracked very quickly – stylishly and legally so. Content owners – Hollywood Inc – were outraged and pursued anybody who even referred to the free software which could perform the trivial decryption process. They used the DMCA as a way to extend the laws of copyright well beyond their original intent. They behaved like a deer in the headlights – blinded by the perceived oncoming doom of a world where their content flows quickly and efficiently, unable to see potential routes to safety while those headlights approach. Their market was changing, facing new opportunities and new threats, and they wanted to slow down the pace of change.
Content owners think that DRM can slow down the natural evolution of a marketplace.

In the case of movies, a big driver of DRM adoption was the unwillingness of the industry to get out of the analog era. Movies are typically distributed to theaters on celluloid film, great big reels of it. It costs a lot to print and distribute those films to the cinemas who will display it. So the realities of real-world distribution have come to define the release strategy of most movies. Companies print a certain number of films, and ship those to cinemas in a few countries. When the movie run is finished there, those same films are shipped to new countries. This is why a movie is typically released at different times in different countries. It’s purely a physical constraint on the logistics of moving chunks of celluloid, and has no place in today’s era of instant, global, digital distribution.

Of course, when DVD’s came along, content owners did not want people to buy the DVD in the USA, then ship that to Australia before the film was showing in cinemas there. Hence the brain damage that we call region encoding – the content owners designed DVD-CSS so that it was not only encrypted, but contained a region marker that is supposed to prevent it from being played anywhere other than the market for which it was released. If you live outside the US, and have ever tried to buy a small-run por^W documentary movie from the US you’ll know what I mean by brain damage: it doesn’t play outside the US, and the demand in your region is not sufficient to justify a print run in your region-coding, so sorry for you.

The truth is that survival in any market depends on your ability to keep up with what is possible. The movie owners need to push hard for global digital distribution – that will let them get movies out on cinema globally on the same day (modulo translation), the same way that you and I can see everything on YouTube the day it is uploaded.

The truth is also that, as the landscape changes, different business models come and go in their viability. Those folks who try to impose analog rules on digital content will find themselves on the wrong side of the tidal wave. Sorry for you. It’s necessary to innovate (again, sometimes!) and stay ahead of the curve, perhaps even being willing to cannibalize your own existing business – though to be honest cannibalizing someone else’s is so much more appealing.

Right now the content owners need to be thinking about how they turn this networked world to their advantage, not fight the tide, and also how to restructure the costs inherent in their own businesses to make them more in line with the sorts of revenues that are possible in a totally digital world.

Here are some reality bites:

• Any DRM that involves offline key storage will be broken. It doesn’t matter if that key is mostly stored on protected hardware, either, because sooner or later one of those gets broken too. And if you want your content to be viewable on most PC’s you will have software viewers. They get broken even faster. So, even if you try to protect every single analog pathway (my favourite is the push for encrypted channels between the hifi and the speakers!) someone, somewhere will get raw access to your content. All you are doing is driving up the cost of your infrastructure – I wonder what the cost of all the crypto associated with HD DVD/BluRay is, when you factor in the complexity, the design, and the incremental cost of IP, hardware and software for every single HD-capable device out there.
• The alternative to offline key storage is streaming-only access, and that is equally unprotectable. The classic streaming system, TV broadcast, was hacked when the VCR came out, and that was blessed as fair use. Today we see one of the digital satellite radio companies (Sirius or XM, I think) being sued by content owners for their support of a device which records their CD-quality broadcasts to MP3 players. Web content streaming services that don’t allow you to save the content locally are a very useless form of protection, easily and regularly subverted. And of course not everyone wants to be online when they are watching your content.
• It only takes one crack. For any given piece of content, all it takes is one unprotected copy, and you have to assume that anyone who wants it will get it. Whether it is software off a warez site, or music from an MP3 download service in Russia, or a file sharing system, you cannot plug all the holes. Face it, people either want to pay you for your content, or they don’t, and your best strategy is to make it as easy as possible for people who want to comply with the law to do so. That does not translate into suing grannies and schoolkids, it translates into effective delivery systems that allow everyone to do the right thing, easily.
• Someone will find a business model that doesn’t depend on the old way of thinking, and if it is not you, then they will eat you alive. You will probably sue them, but this will be nothing but a defensive action as the industry reforms around their new business model, without you. And by the industry I don’t mean your competitors – they will likely be in the same hole – but your suppliers and your customers. The distributors of content are the ones at risk here, not the creators or the consumers.

The music industry’s fear of Napster led them down the DRM rabbit-hole. Microsoft, Apple, SONY and others all developed DRM systems and pitched those to the music industry as a “sane ” approach to online music distribution. It was a nice pitch: “All the distribution benefits of download, all the economic benefits of vinyl”, in a nutshell.

Of these contenders, SONY was clearly ruled out because they are a content owner and there’s no way the rest of the industry would pay a technology tax to a competitor (much as Nokia’s Symbian never gained much traction with the other biggies, because it was too tied to Nokia). Microsoft was a non-starter, because they are too obviously powerful and the music industry could see a hostile takeover coming a mile away. But cute, cuddly Apple wouldn’t harm anyone! So iTunes and AAC were roundly and widely embraced, and Apple succeeded in turning the distribution and playing of legal digital music into a virtual monopoly. Apple played a masterful game, and took full advantage of the music industry’s fear.

The joyful irony in this of course is Steve Jobs recent call for the music industry to adopt DRM-free distribution, giving Apple the moral high ground. Very, very nicely played indeed!

A few years back I was in Davos, at the World Economic Forum. It was perhaps 2002 or 2003, a few years after the dot-com bust. It was the early days of the iPaq, everyone at the conference had been loaned one. I remember clearly sitting in on a session that was more or less a CEO confessional, a sort of absolution-by-admission-of-stupidity gig. One by one, some well known figures stood up and told horror stories about how they’d let the inmates run the asylum, and allowed twenty-something year olds to tell them how to spend their shareholder capital on dot-com projects. This was really interesting to me, as I’d spent the dot-com period telling big companies NOT to over-invest, and to focus on improving their relationships with existing customers and partners using the net, not taking over the world overnight.

But the real kicker came at the very end, when the head of SONY USA, also responsible for its music division, Sir Stringer, stood up to make his peace. He gloated on at length about how SONY had NOT invested in the dot-com, and thus how he felt he must be the only person in the room who had not been taken in by the kids. It was a very funny, very witty speech that earned a round of applause and laughter. I was left wondering whether he had any clue whatsoever how many songs would fit on the iPaq in his pocket, or how long it would take to download them. I suspected not. Of all the CEO’s who had spoken that day, I thought he was the one most likely to be hit hard, and soon, by the digital train.

Sir Stringer is now CEO of SONY worldwide. Funny, then, that the SONY PS3 should have been delayed so that work could be completed on its DRM system.

Some bad ideas are just too attractive to die, once and for all.

Matthew Garrett expressed frustration with Debian recently, in a blog post that’s become rather famous.

I’m of the opinion that Ubuntu could not exist without Debian. So it’s absolutely my intention to see that Ubuntu is a constructive part of the broader Debian landscape. It’s vital that Ubuntu help to sustain and grow Debian, because it’s the breadth and strength of Debian which make up the “shoulders of greatness” on which we in the Ubuntu community stand when we reach for the stars. That doesn’t mean I’m naive enough to think this relationship will ever be an easy one, but I would hope that the discussion doesn’t turn into one of “Ubuntu versus Debian”. Because it isn’t the case that one of them will succeed and the other fail. You could only perceive that as an outcome if you assumed that the two have exactly the same goals.

And that’s where I think a lot of tension is created: it’s hard to know what Debian’s goals are. Those goals are technically articulated in some founding documents but I don’t believe the specific, detailed nature of those goals is actually matched by the personal goals of many members of the community, or users. Debian, in many senses, is at that dangerous stage where it’s a victim of its own success. Its infrastructure and developer recruitment model are for many people what define Debian, and they have been so successful that a community has been created of people who, drawn together by the same things, nonetheless have subtly different personal agendas and goals.

Those differences are a cause of tremendous stress.

When a flamewar erupts, the notional topic of the flameage is often less relevant than the underlying tension between people’s true goals. It’s hard to come to agreement on how to address a specific issue, if there’s no agreement on the very high-level goals that everyone is working towards. Arguments go on forever because one person REALLY wants to see Debian get even more stable on the server, and another person wants to see it get even more cutting edge on the desktop. One person wants more translation of stable versions of applications, another wants newer versions which are by definition not as well translated. One person wants fewer architectures, another wants the full power of Debian on a small embedded architecture.

And all of them have every right to BE RIGHT. All of them, ARE right.

The problem comes if anybody believes that one institution, one product, one single leadership team can synthesise all of that into something which is optimal for EVERYBODY. It’s just not possible to deliver one thing which is optimal for two sets of conflicting requirements, let alone those of a thousand or so of the smartest, most passionate, and lets face it most eclectic of the world’s free software developers. Debian has almost unlimited capacity for some things, by virtue of its openness and democratic governance. That is a wonderful thing. At a time when we all must play to our strengths, many organisations out there would love to have a strength as potent as that. But openness and democracy come at a price if you have narrow goals. No one person or institution can bend that democratic forum to its own specific goals, whether they be desktop, server, embedded, global, local or whatever. Debian, like any institution or product, cannot be all things to all people. It can also not be perfect for one group at the expense of another.

To me, this is the real joy of Debian – it can provide a forum for almost every part of the free software world to come together to hammer out differences and find common ground to the extent that common ground exists. It’s a level playing field – independent of company agendas or technical historical baggage. Debian is the Tibetan Plateau of the free software landscape – elevated through the grinding efforts of conflicting passions to the point of forcing those who visit to get along in a somewhat rarified atmosphere. It can be difficult to breathe up there, sometimes . It’s a bit like the Linux kernel itself: show up, with code, and take your place at the table. And the results are spectacular – Debian as a community creates what I believe is one of the great digital artistic works of the era, and frankly comes as close as I can think possible to actually delivering something that does meet all those conflicting agendas and goals.

Consider Sid. Yes, it breaks your toys now and then, but by and large it represents an extraordinary achievement – pretty much the latest releases of the upstream communities, packaged and categorised. Nothing else, from Ubuntu or Red Hat or Novell (or Microsoft) comes anywhere close. Debian Developers are at their happiest running and working on Sid – a recent survey found that something like 76% of Debian users run Sid, while only something like 6% of Ubuntu users run the equivalent beta code. And remember, Ubuntu only has an Edgy or an Edgy+1 because of Sid. When I look at the ebb and flow of discussions on the Debian mailing lists, I see that Sid is in fact where the very best of Debian comes forth. It’s forward looking, it’s focused on the next generation, it requires exceptional skill and up to date technical knowledge to participate, and it’s not subject to the same political tradeoffs that are inevitable when dealing with releases, architectures, dates, deliverables, translation, documentation and so on. There are very few flamewars about Sid.

If Debian were a business, now would be the time for a careful review of strengths and weaknesses, and perhaps for a plan to focus the resources of the organisation on the things it does best. There’s nothing wrong with cutting goals. Jane, the COO at Canonical, keeps me on the straight and narrow with a fairly regular pruning of Canonical’s focus points too . Every conflicting goal sucks resources from the overall cohesiveness and strength of the group. If there is no consensus in the community, and the leadership don’t think they can get consensus, then it might be better to cut out those conflicting goals altogether. To my mind, the two things that Debian developers absolutely agree on are first, the uncompromising emphasis on free software, and second, the joy of Sid. If I was to try to resolve the bickering and frustration that I see evident in the community, that’s where I would direct the focus of my efforts. Of course, that’s a tough approach, and leaves many other goals for other people and other communities. But it’s where I think Debian, and DD’s, would be most productive and ultimately happiest. There are many things that Debian does brilliantly – celebrate that, focus on it, and trust that others will fill in the gaps.

By contrast with Debian’s Plateau, Ubuntu is a cluster of peaks. By narrowing the focus and allowing the KDE, Gnome and server communities to leverage the base of Debian without treading on one another’s toes, we can create a K2, and a Kangchenjunga and a Lhotse. Ubuntu’s peaks depend on the plateau for their initial start, and their strong base. Ubuntu needs to be humble about its achievements, because much of its elevation comes from Debian. At the same time, Ubuntu can be proud of the way it has lifted beyond the plateau, drawing together people with specific goals to raise the bar and deliver specific releases that meet ambitious, but narrow, goals.

Many people have asked why I decided to build Ubuntu alongside, or on top of, Debian, rather than trying to get Debian to turn into a peak in its own right. The reason is simple – I believe that Debian’s breadth is too precious to compromise just because one person with resources cares a lot about a few specific use cases. We should not narrow the scope of Debian. The breadth of Debian, its diversity of packages and architectures, together with the social equality of all DD’s, is its greatest asset.

So, what’s to be done about the current furore?

A little introspection is healthy, and Debian will benefit from the discussion. Matt is to be credited for his open commentary – a lesser person would simply have disengaged, quietly. I hope that Matt will in fact stay involved in Debian, either directly or through Ubuntu, because his talent and humour are both of enormous benefit to the project. I also hope that Debian developers will make better use of the work we do in Ubuntu, integrating relevant bits of it back into Debian so as to help uplift some of those other peaks – Xandros, Linspire, Maemo, Skolelinux and of course Etch. And most of all, I hope that Debian will start to appreciate its strengths even more, and to play to them, rather than dividing itself along the lines of its weaknesses. Debian/rules, remember?