Mark Shuttleworth | here be dragons

I’ve had quite a lot of positive email feedback on my posting about on renaming as the killer app of distributed version control. So I thought it would be interesting to delve into this subject in more detail. I’ll blog over the next couple of months, starting tomorrow, about the things I think we need from this set of tools – whether they be Git, Darcs, Mercurial, Monotone or Bazaar.

First, to clear something up, Ubuntu selected Bazaar based on our assessment of what’s needed to build a great VCS for the free software community. Because of our work with Ubuntu, we know that what is important is the full spectrum of projects, not just the kernel, or X, or OpenOffice. It’s big and large projects, Linux and Windows projects, C and Python projects, Perl and Scheme projects… the best tools for us are the ones that work well across a broad range of projects, even if those are not the ones that are optimal for a particular project (in the way that Git works brilliantly for the kernel, because its optimisations suit that use case well, it’s a single-platform single-workflow super-optimised approach).

I’ve reviewed our choice of Bazaar in Ubuntu a couple of times, when projects like OpenSolaris and X made other choices, and in each case been satisfied that it’s still the best project for our needs. But we’re not tied to it, we could move to a different one. Canonical has no commercial interest in Bazaar (it’s ALL GPL software) and no cunning secret plans to launch a proprietary VCS based on it. We integrated Bazaar into Launchpad because Bazaar was our preferred VCS, but Bazaar could just as well be integrated into SourceForge and Collab since it’s free code.

So, what I’m articulating here is a set of values and principles – the things we find important and the rationale for our decisions – rather than a ra-ra for a particular tool. Bazaar itself doesn’t meet all of my requirements, but right now it’s the closest tool for the full spectrum of work we do.

Tomorrow, I’ll start with some commentary on why “lossless” tools are a better starting point than lossy tools, for projects that have that luxury.

The number one thing I want from a distributed version control system is robust renaming. Why is that? Because without a rigorous approach to renaming that guarantees perfect results, I’m nervous to merge from someone I don’t know. And merging from “people you don’t know” is the real thing that distributed version control gives which you cannot get from centralized systems like CVS and Subversion.

Distributed version control is all about empowering your community, and the people who might join your community. You want newcomers to get stuck in and make the changes they think make sense. It’s the difference between having blessed editors for an encyclopedia (in the source code sense we call them “committers”) and the wiki approach, which welcomes new contributors who might just have a very small fix or suggestion. And perhaps more importantly, who might be willing to spend time on cleaning up and reshaping the layout of your wiki so that it’s more accessible and understandable for other hackers.

The key is to lower the barrier to entry. You don’t want to have to dump a whole lot of rules to new contributors like “never rename directories a, b and c because you will break other people and we will be upset”. You want those new contributors to have complete freedom, and then you want to be able to merge, review changes, and commit if you like them. If merging from someone might drop you into a nightmare of renaming fixups, you will be resistant to it, and your community will not be as widely empowered.

So, try this in your favorite distributed VCS:

1. Make two branches of your favorite upstream. In Bzr, you can find some projects to branch in the project cloud.
2. In one branch, pretend to be a new contributor, cleaning up the build system. Rearrange some directories to make better sense (and almost every large free software project can benefit from this, there’s a LOT of cruft that’s crept in over the years… the bigger the project, the bigger the need).
3. Now, in the second branch, merge from the branch where you did that renaming. Some systems will fail, but most will actually handle this easy case cleanly.
4. Go back to the first branch. Add a bunch of useful files to the repo in the directories you renamed. Or make a third branch, and the files to the directories there.
5. Now, merge in from that branch.
6. Keep playing with this. Sooner or later, if you are not using a system like Bzr which treats renames as a first class operation… Oops.

Now, this is not a contrived example, it’s actually a perfect study of what we HOPE will happen as distributed version control is more widely adopted. If I look at the biggest free software projects, the thing they all have in common is crufty tree structures (directory layouts) and build systems. This is partly a result of never having had tools which really supported renaming, in a way which Would Not Break. And this is one of the major reasons why it takes 8 hours to build something like OpenOffice, and why so few people have the stomach to step up and contribute to a project like that.

The exact details of what it takes to break the renaming support of many DVCS’s vary from implementation to implementation. But by far the most robust of them is Bzr at the moment, which is why we make such heavy use of it at Ubuntu. Many of the other systems have just waved past the renaming problem, saying it’s “not essential” and that heuristics and guesstimates are sufficient. I disagree. And I think the more projects really start to play with these tools, the more they will appreciate renaming is the critical feature that needs to Just Work. I’ll gladly accept the extra 0.3 seconds it takes Bzr to give me a tree status in my 5,100 file project, for the security of knowing I never ever have to spend long periods of time sorting out a merge by hand when stuff got renamed. It still comes back in less than a second. Which is plenty fast enough for me. Even though I know it will get faster, that extra performance is not nearly as important to me as the overall time saved by the robustness of the tool in the face of a constant barrage of improvements by new contributors.

Anybody else frustrated with the state of fonts in Linux today?

It seems there are two distinct issues: the availability of high quality fonts under Free licenses, and the infrastructure for installing, managing and accessing those fonts.

There has been some progress on both fronts. Bitstream’s Vera, and the new Liberation font work (kudos to Red Hat for driving that effort) are steps to provide us with a clean, crisp set of high quality fonts with good hinting that can be installed by default. There is also good work being done by, amongst others, SIL International on a free font license framework, and fonts to go with it. I hope the community can build on these efforts to expand the font coverage to the full Unicode glyphset, preserving their essential character and metrics.

The second problem, the infrastructure and API’s to manage fonts on Linux systems, is more complicated. Here’s a mail to the ubuntu-devel list describing the situation and calling for leadership from the community in helping to address it.

We need a clean, clear way of:

1. Packaging fonts, and knowing which packages to install to get which fonts.
2. Cataloguing fonts, and allowing people to manage the fonts that are immediately accessible to them or loaded by default, everywhere.
3. Making all of this sane in a world where you MIGHT want to read a document in Korean using a French desktop. In other words, where there need to be a lot of fonts available, even if most of those fonts are not used all the time.

Most of the long list of fonts I see in OpenOffice are lost on me, I don’t know when I would choose any of them.

Sounds like a mess, but then again it also sounds like the sort of Gordian knot that the flaming sword of free software can slice straight through, given strong leadership and a forum for the work. Who will step up?

Much has been written about Microsoft’s allegation of patent infringements in Linux (by which I’m sure they mean GNU/Linux ). I don’t think Microsoft is the real threat, and in fact, I think Microsoft and the Linux community will actually end up fighting on the same side of this issue.

I’m in favour of patents in general, but not software or business method patents. I’ll blog separately some day about why that’s the case, but for the moment I’ll just state for the record my view that software patents hinder, rather than help, innovation in the software industry.

And I’m pretty certain that, within a few years, Microsoft themselves will be strong advocates against software patents. Why? Because Microsoft is irrevocably committed to shipping new software every year, and software patents represent landmines in their roadmap which they are going to step on, like it or not, with increasing regularity. They can’t sit on the sidelines of the software game – they actually have to ship new products. And every time they do that, they risk stepping on a patent landmine.

They are a perfect target – they have deep pockets, and they have no option but to negotiate a settlement, or go to court, when confronted with a patent suit.

Microsoft already spends a huge amount of money on patent settlements (far, far more than they could hope to realise through patent licensing of their own portfolio). That number will creep upwards until it’s abundantly clear to them that they would be better off if software patents were history.

In short, Microsoft will lose a patent trench war if they start one, and I’m sure that cooler heads in Redmond know that.

But let’s step back from the coal-face for a second. I have high regard for Microsoft. They produce some amazing software, and they made software much cheaper than it ever was before they were around. Many people at Microsoft are motivated by a similar ideal to one we have in Ubuntu: to empower people for the digital era. Of course, we differ widely on many aspects of the implementation of that ideal, but my point is that Microsoft is actually committed to the same game that we free software people are committed to: building things which people use every day.

So, Microsoft is not the real patent threat to Linux. The real threat to Linux is the same as the real threat to Microsoft, and that is a patent suit from a person or company that is NOT actually building software, but has filed patents on ideas that the GNU project and Microsoft are equally likely to be implementing.

Yes, Nathan, I’m looking at you!

As they say in Hollywood, where there’s a hit there’s a writ. And Linux is a hit. We should expect a patent lawsuit against Linux, some time in the next decade.

There are three legs to IP law: copyright, trademark and patents. I expect a definitive suit associated with each of them. SCO stepped up on the copyright front, and that’s nearly dealt with now. A trademark-based suit is harder to envisage, because Linus and others did the smart thing and established clear ownership of the “Linux” trademark a while ago. The best-practice trademark framework for free software is still evolving, and there will probably be a suit or two, but none that could threaten the continued development of free software. And the third leg is patent law. I’m certain someone will sue somebody else about Linux on patent grounds, but it’s less likely to be Microsoft (starting a trench war) and more likely to be a litigant who only holds IP and doesn’t actually get involved in the business of software.

It will be a small company, possibly just a holding company, that has a single patent or small portfolio, and goes after people selling Linux-based devices.

Now, the wrong response to this problem is to label pure IP holders as “patent trolls”. While I dislike software patents, I deeply dislike the characterisation of pure IP holders as “patent trolls”. They are only following the rules laid out in law, and making the most of a bad system; they are not intrinsically bad themselves. Yes, Nathan, all is forgiven . One of the high ideals of the patent system is to provide a way for eccentric genius inventors to have brilliant insights in industries where they don’t have any market power, but where their outsider-perspective leads them to some important innovation that escaped the insiders. Ask anyone on the street if they think patents are good, and they will say, in pretty much any language, “yes, inventors should be compensated for their insights”. The so-called “trolls” are nothing more than inventors with VC funding. Good for them. The people who call them trolls are usually large, incumbent players who cross-license their patent portfolios with other incumbents to form a nice, cosy oligopoly. “Trolling” is the practice of interrupting that comfortable and predictably profitable arrangement. It’s hard to feel any sympathy for the incumbents at all when you look at it that way.

So it’s not the patent-holders who are the problem, it’s the patent system.

What to do about it?

Well, there are lots of groups that are actively engaged in education and policy discussion around patent reform. Get involved! I recently joined the FFII: Foundation for a Free Information Infrastructure, which is doing excellent work in Europe in this regard. Canonical sponsored the EUPACO II conference, which brought together folks from across the spectrum to discuss patent reform. And Canonical also recently joined the Open Invention Network, which establishes a Linux patent pool as a defensive measure against an attack from an incumbent player. You can find a way to become part of the conversation, too. Help to build better understanding about the real dynamics of software innovation and competition. We need to get consensus from the industry – including Microsoft, though it may be a bit soon for them – that software patents are a bad thing for society.

I’ve been on the road solidly for the past 10 days but itching to write about Dell’s announcement of pre-installed Linux for consumers.

This is a significant milestone, not just for Ubuntu but for every flavour of Linux and the free software community as a whole. While there are already a number of excellent companies like System76 offering Linux pre-installed, Dell represents “the industry”, and it’s very important for all of us that the industry sees a future for Linux on the desktop.

Device compatibility is the top issue people raise as a blocker of broad Linux adoption. Many hardware manufacturers don’t yet provide zero-day Linux drivers for their components, because of the perceived lack of market demand for those drivers. The Dell announcement is already changing that. Those manufacturers who are Linux-aware will have a significant advantage selling their components to global PC vendors who are shipping Linux, because those PC vendors can offer the same components across both Linux and Windows PC’s. That commonality reduces cost, and cost is everything in the volume PC market.

I believe that the free software approach is a better device driver development model for component and peripheral manufacturers, and that once they have learned how to work with the Linux community they will quickly ensure that their devices work with Linux as soon as, or before, they work with proprietary platforms. It will take some time to help those vendors understand the full process of working in a collaborative forum with the upstream kernel community, to ensure the widest possible benefit from their efforts. I’ve no doubt that vendors who start out thinking in proprietary terms will, over time, shift towards providing free drivers in partnership with the Linux community. I would credit companies like Intel for their leadership in that regard, it’s great to be able to show how their free drivers make it possible to reach the widest possible audience with their hardware.

The most important thing for all of us is the commercial success of Dell’s offering. A sustainable business in pre-installed Linux in Western markets will give credibility to the Linux desktop as well as providing an opportunity to build relationships with the rest of the consumer PC ecosystem. We don’t have to fix Bug #1 in order to make Linux a top-tier target for hardware vendors – we just need to show that there’s an economic incentive for them to engage with our community.

One of the very interesting issues-du-jour is the interaction between the three “legs” of “intellectual property”. Traditionally, those three are copyrights, patents and trademarks, and they have quite different laws and contractual precedents that are associated with them.

Recently, however, I’ve observed an increase in the cross-talk between them.

Classically, “software freedom” was about the copyright license associated with the code. But patents and trademarks are now being brought into the mix. For example, the discussion around Mozilla’s trademark policy was directly linking the concept of “freedom” to trademark policy as much as code copyright license. And much of the very hard debate in the GPLv3 process is about linkages between copyright license and relevant patents. And like it or not, the GPL is widely considered the reference implementation of freedom so GPLv3’s approach will be, for many, definitive on the subject.

In the Ubuntu community we’ve recently gone through a process to agree a trademark policy. This was recently approved by the Community Council, and the final draft is here:

http://www.ubuntu.com/aboutus/trademarkpolicy

We’ve tried to strike a balance that keeps the trademarks of Ubuntu meaningful (i.e. if it says Ubuntu, it really is Ubuntu) but also recognizes the fact that Ubuntu is a shared work, in which many different participants of our community make a personal investment, and which they should have the right to share. So we’ve made explicit the idea of a remix – a reworking of Ubuntu that addresses the needs of a specific community (could be national, could be an industry like medical or educational) but preserves the key things that people would expect from Ubuntu, like hardware support and certification.

I’m sure this isn’t the last word on the subject, but I hope it’s a useful contribution to the debate, and would welcome other projects adopting similar licenses. For that reason, our trademark license is published under the Creative Commons Sharealike with Attribution license (CC-BY-SA).

I’ve long believed there’s a general phenomenon that underlies the free software movement. It’s “volunteer-driven, internet-powered collaboration”. I think it will ultimately touch every industry that has any digital workflow. Lets face it, that’s pretty much every industry.

The phenomenon has three key elements:

1. Freedom-driven licensing. If you want the magic, you have to set it free, because it’s the possibility of doing things for themselves that motivates people to build on your work. Just exposing the “source” (whether that’s code or other content) isn’t as interesting. Microsoft will show you the source to Windows these days, but they won’t give you the freedom to remix it.
2. Community. The net allows us to build a community of eyeballs and fingers based on personal interest rather than personal geography. It used to be that companies always had to do the best they could with local talent – or fly people in and deal with visa issues (that’s why Microsoft is a big proponent of greater H1-B visa allocations). Today we can find the best talent wherever it is, talent that is really personally interested in the underlying issue. And we call that talent pool “community”.
3. Revision control. I’m much happier to give you read AND write access to my stuff, if I can know who changed what, when, and easily revert it. And if that revision control allows cheap branching, then there can be multiple, parallel efforts to solve a particular problem.

Consider wikipedia in this light: it clearly meets all three criteria. Its content has a license that gives you genuine freedom. There is a big community that takes a personal interest in that content (actually, multiple communities, one which I call “the librarians” wants to make sure the institution itself is healthy, the others are communities that form around specific content, given the nature of wikipedia as a repository of knowledge). And of course every change is logged with some level of identity associated with it.

The linux kernel is the same, as are most of the components we associate with a GNU OS.

But why stop at just code and knowledge? I’m a big fan of the work of the Creative Commons, because they have taken to heart the idea of generalizing the licensing problem. And conferences like the Digital Freedom Expo in South Africa this week, which TSF has agreed to sponsor, are forums for discussing the ways in which these principles can apply to other domains. I would love to be part of the exploration of this phenomenon at all levels but Ubuntu is plenty of work for one lifetime. Nevertheless I think there are real opportunities, both social and commercial, in this idea.

Incidentally, one of the reasons I picked the Bazaar revision control project for use in our infrastructure, and why I sponsor it, is because I think it will be great to have a revision control system which can be adapted to manage LOTS of different kinds of content, not just code. And the Bazaar guys abstract things to an appropriate level to be able to do just this. I’d like to be able to see a house I like, and “bzr branch” the plans to that house, then share my modifications together with my experiences of living in that house so that others can merge the ideas they think worked best. All we need is bzr embedded in an architectural drawing application

A number of folks have asked about the new “radical freedom” flavour of Ubuntu that I hinted at in the announcement of work on Gutsy Gibbon.

Part of that initiative is focused on code freedom – going further than anybody else, though, beyond the CPU down to the level of the code running in firmware on your peripherals. We want to highlight the good work of hardware vendors who have completely embraced that idea. Of course – if you REALLY want freedom then you need to run that flavour on a SUN SPARC chip in an FPGA, in which case you would have freedom to modify even the CPU itself, and everything running on it. Raising the profile of genuinely free hardware is one way I hope we can reach the point where we no longer choose to include any binary drivers in vanilla Ubuntu.

But a broader part of this “radical freedom” thrust is to explore freedom in other domains. If we ship a PDF, do we ship the source document? If we ship a JPG, do we ship the source artwork? If we ship a nicely edited video, do we ship the original, unedited recording so you can really remix it? If we ship music, do we ship the samples and the separated tracks?

Potent medicine indeed. I’m looking forward to seeing how far we can push the concept, just inside the Ubuntu project.

Isn’t StumbleUpon wonderful?

…to Debian on the release of Etch!

There are some ideas that are broken, but attractive enough to some people that they are doomed to be tried again and again.

DRM is one of them.

I was thrilled to see recently that the processing key for *all* HD discs produced to date has been discovered and published. I expect this to lead to the complete unraveling of the Blu-Ray and HD-DVD content protection schemes before even 1% of the potential market for those players has been reached. Good news indeed, because it may inspire the people who setup such schemes to reconsider.

We’ve been here before. The DVD-CSS encryption system was cracked very quickly – stylishly and legally so. Content owners – Hollywood Inc – were outraged and pursued anybody who even referred to the free software which could perform the trivial decryption process. They used the DMCA as a way to extend the laws of copyright well beyond their original intent. They behaved like a deer in the headlights – blinded by the perceived oncoming doom of a world where their content flows quickly and efficiently, unable to see potential routes to safety while those headlights approach. Their market was changing, facing new opportunities and new threats, and they wanted to slow down the pace of change.
Content owners think that DRM can slow down the natural evolution of a marketplace.

In the case of movies, a big driver of DRM adoption was the unwillingness of the industry to get out of the analog era. Movies are typically distributed to theaters on celluloid film, great big reels of it. It costs a lot to print and distribute those films to the cinemas who will display it. So the realities of real-world distribution have come to define the release strategy of most movies. Companies print a certain number of films, and ship those to cinemas in a few countries. When the movie run is finished there, those same films are shipped to new countries. This is why a movie is typically released at different times in different countries. It’s purely a physical constraint on the logistics of moving chunks of celluloid, and has no place in today’s era of instant, global, digital distribution.

Of course, when DVD’s came along, content owners did not want people to buy the DVD in the USA, then ship that to Australia before the film was showing in cinemas there. Hence the brain damage that we call region encoding – the content owners designed DVD-CSS so that it was not only encrypted, but contained a region marker that is supposed to prevent it from being played anywhere other than the market for which it was released. If you live outside the US, and have ever tried to buy a small-run por^W documentary movie from the US you’ll know what I mean by brain damage: it doesn’t play outside the US, and the demand in your region is not sufficient to justify a print run in your region-coding, so sorry for you.

The truth is that survival in any market depends on your ability to keep up with what is possible. The movie owners need to push hard for global digital distribution – that will let them get movies out on cinema globally on the same day (modulo translation), the same way that you and I can see everything on YouTube the day it is uploaded.

The truth is also that, as the landscape changes, different business models come and go in their viability. Those folks who try to impose analog rules on digital content will find themselves on the wrong side of the tidal wave. Sorry for you. It’s necessary to innovate (again, sometimes!) and stay ahead of the curve, perhaps even being willing to cannibalize your own existing business – though to be honest cannibalizing someone else’s is so much more appealing.

Right now the content owners need to be thinking about how they turn this networked world to their advantage, not fight the tide, and also how to restructure the costs inherent in their own businesses to make them more in line with the sorts of revenues that are possible in a totally digital world.

Here are some reality bites:

• Any DRM that involves offline key storage will be broken. It doesn’t matter if that key is mostly stored on protected hardware, either, because sooner or later one of those gets broken too. And if you want your content to be viewable on most PC’s you will have software viewers. They get broken even faster. So, even if you try to protect every single analog pathway (my favourite is the push for encrypted channels between the hifi and the speakers!) someone, somewhere will get raw access to your content. All you are doing is driving up the cost of your infrastructure – I wonder what the cost of all the crypto associated with HD DVD/BluRay is, when you factor in the complexity, the design, and the incremental cost of IP, hardware and software for every single HD-capable device out there.
• The alternative to offline key storage is streaming-only access, and that is equally unprotectable. The classic streaming system, TV broadcast, was hacked when the VCR came out, and that was blessed as fair use. Today we see one of the digital satellite radio companies (Sirius or XM, I think) being sued by content owners for their support of a device which records their CD-quality broadcasts to MP3 players. Web content streaming services that don’t allow you to save the content locally are a very useless form of protection, easily and regularly subverted. And of course not everyone wants to be online when they are watching your content.
• It only takes one crack. For any given piece of content, all it takes is one unprotected copy, and you have to assume that anyone who wants it will get it. Whether it is software off a warez site, or music from an MP3 download service in Russia, or a file sharing system, you cannot plug all the holes. Face it, people either want to pay you for your content, or they don’t, and your best strategy is to make it as easy as possible for people who want to comply with the law to do so. That does not translate into suing grannies and schoolkids, it translates into effective delivery systems that allow everyone to do the right thing, easily.
• Someone will find a business model that doesn’t depend on the old way of thinking, and if it is not you, then they will eat you alive. You will probably sue them, but this will be nothing but a defensive action as the industry reforms around their new business model, without you. And by the industry I don’t mean your competitors – they will likely be in the same hole – but your suppliers and your customers. The distributors of content are the ones at risk here, not the creators or the consumers.

The music industry’s fear of Napster led them down the DRM rabbit-hole. Microsoft, Apple, SONY and others all developed DRM systems and pitched those to the music industry as a “sane ” approach to online music distribution. It was a nice pitch: “All the distribution benefits of download, all the economic benefits of vinyl”, in a nutshell.

Of these contenders, SONY was clearly ruled out because they are a content owner and there’s no way the rest of the industry would pay a technology tax to a competitor (much as Nokia’s Symbian never gained much traction with the other biggies, because it was too tied to Nokia). Microsoft was a non-starter, because they are too obviously powerful and the music industry could see a hostile takeover coming a mile away. But cute, cuddly Apple wouldn’t harm anyone! So iTunes and AAC were roundly and widely embraced, and Apple succeeded in turning the distribution and playing of legal digital music into a virtual monopoly. Apple played a masterful game, and took full advantage of the music industry’s fear.

The joyful irony in this of course is Steve Jobs recent call for the music industry to adopt DRM-free distribution, giving Apple the moral high ground. Very, very nicely played indeed!

A few years back I was in Davos, at the World Economic Forum. It was perhaps 2002 or 2003, a few years after the dot-com bust. It was the early days of the iPaq, everyone at the conference had been loaned one. I remember clearly sitting in on a session that was more or less a CEO confessional, a sort of absolution-by-admission-of-stupidity gig. One by one, some well known figures stood up and told horror stories about how they’d let the inmates run the asylum, and allowed twenty-something year olds to tell them how to spend their shareholder capital on dot-com projects. This was really interesting to me, as I’d spent the dot-com period telling big companies NOT to over-invest, and to focus on improving their relationships with existing customers and partners using the net, not taking over the world overnight.

But the real kicker came at the very end, when the head of SONY USA, also responsible for its music division, Sir Stringer, stood up to make his peace. He gloated on at length about how SONY had NOT invested in the dot-com, and thus how he felt he must be the only person in the room who had not been taken in by the kids. It was a very funny, very witty speech that earned a round of applause and laughter. I was left wondering whether he had any clue whatsoever how many songs would fit on the iPaq in his pocket, or how long it would take to download them. I suspected not. Of all the CEO’s who had spoken that day, I thought he was the one most likely to be hit hard, and soon, by the digital train.

Sir Stringer is now CEO of SONY worldwide. Funny, then, that the SONY PS3 should have been delayed so that work could be completed on its DRM system.

Some bad ideas are just too attractive to die, once and for all.