“in addition to”

Friday, October 19th, 2012

Nothing in yesterday’s post about inviting members of the community into some of the projects we work on in confidence implied that the Ubuntu development process is becoming less open.

Ubuntu set the standard for transparency as a company producing a distribution a long time ago, when we invited anybody who showed a passion and competence to have commit and upload rights, a strong contrast with the Fedora policy of the time, which required you to be a Red Hat employee. We continue that tradition with a leadership Community Council that has no requirement of Canonical employment, unlike our competitors. And we invite everyone to participate in the design and development of Ubuntu, which happens in public at UDS (week after next, as it happens) and online on IRC and Launchpad.

Every member of a community works on personal projects. Our competitors do so too. There are any number of changes thrust upon Gnome by Red Hat for example, that then get whitewashed as “maintainers discretion” or “designers design”. There are any number of reveals, prototypes, patents and other decisions that are taken in private, by all members of all communities. Even amongst volunteers its normal to see someone saying “I’ve been hacking on this for a while, now I want some feedback”.

What I offered to do, yesterday, spontaneously, is to invite members of the community in to the things we are working on as personal projects, before we are ready to share them. This would mean that there was even less of Ubuntu that was NOT shaped and polished by folk other than Canonical – a move that one would think would be well received. This would make Canonical even more transparent.

So please disregard the commentary by folk who assumed that the public discussion of Ubuntu development would somehow change. Instead, I hope you will welcome the idea that even Canonical’s most exciting initiatives will now be open to participation by members of the community. And I challenge you to find another place where you can participate at EVERY level in the design and construction of a free platform that is used by millions of people.

Raring community skunkworks

Thursday, October 18th, 2012

Mapping out the road to 13.04, there are a few items with high “tada!” value that would be great candidates for folk who want to work on something that will get attention when unveiled. While we won’t talk about them until we think they are ready to celebrate, we’re happy to engage with contributing community members that have established credibility (membership, or close to it) in Ubuntu, who want to be part of the action.

This would provide early community input and review, without spoiling the surprise when we think the piece is ready. It would allow community members to work on something that will be widely covered at release (at least, on OMG ;-))

The skunkworks approach has its detractors. We’ve tried it both ways, and in the end, figured out that critics will be critics whether you discuss an idea with them in advance or not. Working on something in a way that lets you refine it till it feels ready to go has advantages: you can take time to craft something, you can be judged when you’re ready, you get a lot more punch when you tell your story, and you get your name in lights (though not every headline is one you necessarily want ;)).

So, we thought we would extend the invitation to people who trust us and in whom we have reason to trust, to work together on some sexy 13.04 surprises. The projects range from webby (javascript, css, html5) to artistic (do you obsess about kerning and banding) to scientific (are you a framerate addict) to glitzy (pixel shader sherpas wanted) to privacy-enhancing (how is your crypto?) to analytical (big daddy, big brother, pick your pejorative). But they all make the Ubuntu experience better for millions of users, they are all groundbreaking in free software, they will all result in code under the GPL (or an existing upstream license if they are extensions to existing projects). No NDA’s needed but we will need to trust you not to talk in your sleep ;). We’ll also need to trust you to write code that is thorough and tested, stuff you’ll be as proud of as we are of the rest of the Ubuntu experience. Of course.

There’s also plenty going on that doesn’t warrant the magician’s reveal. But if you are game for a bit of the spotlight, bring some teflon and ping Michael Hall at mhall119 on Freenode.

The dash to Copenhagen combined with a dash across the Atlantic has me righteously ramfeezled, but the roisterous reception we got at the OpenStack summit (congrats, stackers, on a respectable razzmatazz of rugible cloud enthusiasm) made it worthwhile. A quick shout out to the team behind the Juju gooooey, that puts a whole new face on cloud agility – rousing stuff.

Nevertheless, it’s way past time to root our next rhythmic release in some appropriate adjective.

The challenge, of course, has been the number of entirely inappropriate adjectives that presented themselves along the way. Go read the dictionary. R is just loaded with juicy stuff we can’t use without invoking the radge wrath of the rinky-dink chorus. Sigh.

Nevertheless, somewhere between the risibly rambunctious and the reboantly ran-tan, the regnally rakish and the reciprocornously rorty, there was bound to be a good fit. Something radious or rident, something to rouse our rowthy rabble.

So what will we be up to in the next six months? We have two short cycles before we’re into the LTS, and by then we want to have the phone, tablet and TV all lined up. So I think it’s time to look at the core of Ubuntu and review it through a mobile lens: let’s measure our core platform by mobile metrics, things like battery life, number of running processes, memory footprint, and polish the rough edges that we find when we do that. The tighter we can get the core, the better we will do on laptops and the cloud, too.

So bring along a Nexus 7 if you’re coming to Copenhagen, because it makes a rumpty reference for our rootin’ tootin’ radionic razoring. The raving Rick and his merry (wo)men will lead us to a much leaner, sharper, more mobile world. We’ll make something… wonderful, and call it the Raring Ringtail. See you there soon.

Update: for clarity, this ringtail is no laconic lemur, it’s a ringtail raccoon. However, for the sake of sanity, it’s not a raring ringtail raccoon, just a raring ringtail. There.

Amazon search results in the Dash

Sunday, September 23rd, 2012

It makes perfect sense to integrate Amazon search results in the Dash, because the Home Lens of the Dash should let you find *anything* anywhere. Over time, we’ll make the Dash smarter and smarter, so you can just ask for whatever you want, and it will Just Work.

The Home Lens of the Dash is a “give me X” experience. You hit the Super key, and say what you want, and we do our best to figure out what you mean, and give you that. Of course, you can narrow the scope of that search if you want. For example, you can hit Super-A and just search applications. But if you throw your query out to the Dash, we need to be a smart as possible about where we go looking for answers for you.

In 12.10 we’ll take the first step of looking both online and locally for possible results. The Home lens will show you local things like apps and music, as it always has, as well as results from Amazon. Note – these are not ads, they are results to your search. We don’t promote any product or service speculatively, these are not banners or spyware. These are results from underlying scopes, surfaced to the Home lens, because you didn’t narrow the scope to a specific, well scope.

What we have in 12.10 isn’t the full experience, so those who leap to judgement are at maximum risk of having to eat their words later. Chill out. If the first cut doesn’t work for you, remove it, or just search the specific scope you want (there are hotkeys for all the local scopes).

Here’s a quick Q&A on the main FUD-points.

Why are you putting ads in Ubuntu?

We’re not putting ads in Ubuntu. We’re integrating online scope results into the home lens of the dash. This is to enable you to hit “Super” and then ask for anything you like, and over time, with all of the fantastic search scopes that people are creating, we should be able to give you the right answer.

These are not ads because they are not paid placement, they are straightforward Amazon search results for your search. So the Dash becomes a super-search of any number of different kinds of data. Right now, it’s not dynamically choosing what to search, it’s just searching local scopes and Amazon, but it will get smarter over time.

I don’t wan’t to search Amazon for the Terminal

Use Super-A. You can tell Unity exactly what you want to search. And in future you’ll be able to do that from the home lens, too, more easily than the current Lens Bar at the bottom of the Dash.

I want to control what is searched on the Home Lens

So do I! Designs and patches welcome in the usual places. I’m pretty sure by 14.04 LTS we’ll have the kinks unkinked. Till then, come along for the ride, or stick with 12.04 LTS. We can’t wait till it’s perfect before landing everything, because the only way to learn what’s not perfect is to have other people – real people – use it.

I can’t believe you just changed Ubuntu. I liked it the way it was.

Looks like those six months are nearly up again ;)

This is just a moneymaking scheme.

We picked Amazon as a first place to start because most of our users are also regular users of Amazon, and it pays us to make your Amazon journey  get off to a faster start. Typing Super “queen marking cage”  Just Worked for me this morning. I am now looking forward to my game of Ultimate Where’s Waldo hunting down the queens in my bee colonies, Ubuntu will benefit from the fact that I chose to search Amazon that way, Amazon benefits from being more accessible to a very discerning, time-conscious and hotkey-friendly audience.

But there are many more kinds of things you can search through with Unity scopes. Most of them won’t pay Ubuntu a cent, but we’ll still integrate them into the coolest just-ask-and-you’ll-receive experience. I want us to do this because I think we can make the desktop better.

Why are you telling Amazon what I am searching for?

We are not telling Amazon what you are searching for. Your anonymity is preserved because we handle the query on your behalf. Don’t trust us? Erm, we have root. You do trust us with your data already. You trust us not to screw up on your machine with every update. You trust Debian, and you trust a large swathe of the open source community. And most importantly, you trust us to address it when, being human, we err.

The query should be encrypted!

It will be in the release version.

I want to control this in the settings!

Yes, we agree, and designs and work are under way to make that possible. That should land in 12.10 too, or as an update, or in 13.04.

In summary – please don’t feed the trolls. We’re interested in feedback in what sorts of things would be useful to search straight from the home lens, and how to improve the search results, as well as provide better control of the process to you.

Here is the key question, as I see it:

Can Canonical and the Ubuntu community handle the responsibility associated with this sort of service?

Well, if we want to run a modern platform, that updates automatically and provides users with the full benefit of living in a connected world, then we have to be able to do that. If we can’t we won’t be relevant. So we should talk about the appropriate kinds of privacy policy, appropriate encryption, appropriate settings and preferences, to make this all world class.

Microsoft has built an impressive new entrant to the Infrastructure-as-a-Service market, and Ubuntu is there for customers who want to run workloads on Azure that are best suited to Linux. Windows Azure was built for the enterprise market, an audience which is increasingly comfortable with Ubuntu as a workhorse for scale-out workloads; in short, it’s a good fit for both of us, and it’s been interesting to do the work to bring Ubuntu to the platform.

Given that it’s normal for us to spin up 2,000-node Hadoop clusters with Juju, it will be very valuable to have a new enterprise-oriented cloud with which to evaluate performance, latency, reliability, scalability and many other key metrics for production deployment scenarios.

As IAAS grows in recognition as a standard part of the enterprise toolkit, it will be important to have a wide range of infrastructures that are addressable, with diverse strengths. In the case of Windows Azure, there is clearly a deep connection between Windows-based IT and the new IAAS. But I think Microsoft has set their sights on a bigger story, which is high-quality enterprise-oriented infrastructure that is generally useful. That’s why Ubuntu is important to them, and why it was worthwhile for us to work together despite our differences. Just as we need to ensure that customers can run Ubuntu and Windows together inside their data centre and on the LAN, we want to ensure that cloud workloads play nicely.

The team leading Azure has a sophisticated understanding of Ubuntu and Linux in general. They are taking a pragmatic approach that will raise eyebrows around the Redmond campus, but is exactly what customers want to see. We have taken a similar view. I know there will be members of the free software community that will leap at the chance to berate Microsoft for its very existence, but it’s not very Ubuntu to do so: let’s argue our perspective, work towards our goals, be open to those who are open to us, and build great stuff. There is nothing proprietary in Ubuntu-for-Azure, and no about-turn from us on long-held values. This is us making sure our audience, and especially the enterprise audience, can benefit from the work our community and Canonical do no matter where they want to do it.

Windows Azure IAAS is in beta. If you are using the cloud today, or interested in it, I highly recommend you try it out. There’s no better way to make yourself heard over there.

UDS & Cloud Day in Oakland – yeah!

Monday, May 7th, 2012

Had a blast this morning welcoming everyone to Oakland, California, kicking off UDS and unveiling the first rack-ready 48-node 192- core ARM server from Calxeda. Looking forward to the new Cloud Day at UDS tomorrow, great speakers including Richard Kaufmann, CTO of HP Cloud, Randy Bias of Cloud Scaling, Mark Collier of Rackspace. We’ll have VMWare, Scality, 10gen, Engine Yard, Iron.IO, Scalr, and Enstratus all under one roof for the day talking open source cloud greatness. Come and say hi, or dive in with them to see how the smart money is building clouds, then stick around for anything on the UDS schedule that takes your fancy!

Quality has a new name

Monday, April 23rd, 2012

This quirky scheme of adjectives and animals presents a pretty puzzle every six months. What mix of characteristics do we want to celebrate in the next release? Here we are, busily finalizing the precise pangolin (which was a rather perfect product placement for a scaly anteater, all things considered) and before one realises it’s time to talk turkey, so to speak, about Q! Our code names may raise a quizzical eyebrow here and there, but they capture the zeitgeist of a cycle and shape our discussions in surprising ways. The quest for a name has no quick answer unless, of course, you jump to the last paragraph ;)

12.04 being an LTS we’ve been minding our P’s and Q’s, but many of our quality-oriented practices from 12.04 LTS will continue into Q territory. We’ll keep the platform usable throughout the cycle, because that helped hugely to encourage daily use of the release, which in turn gives us much better feedback on questions of quality. And we’ll ratchet up the continuous integration, smoke testing and automated benchmarking of the release, since we can do it all in the cloud. We have, so to speak, stacks and stacks of cloud to use. So quality is quotidian rather than quarterly. And it is both qualitative and quantitative, with user research and testing continuing to shape our design decisions. The effort we put into polishing Unity and the rest of the platform in 12.04 seem to have paid off handsomely, with many quondam quarrelsome suddenly quiescent in the face of a surge in support for the work.

But the finest quality is that without a name, so support for “quality” as a codename would at best be qualified. Every release has quality first these days – they all get used, on the server, on devices, and while the term of maintenance might vary, our commitment to interim releases is just as important as that to an LTS.

Our focus on quality permeates from the platform up to the code we write upstream, and our choices of upstream components too. We require tests and gated trunks for all Canonical codebases, and prefer upstreams that share the same values. Quality starts at the source, it’s not something that can be patched in after the fact. And I’m delighted that we have many upstreams using our tools to improve their quality too! We have awesome tools for daily builds from branches, continuous integration support in Launchpad, the ability to provide a gated trunk with tests run in the cloud for projects that really care about quality. Rumours and allegations of a move from Upstart to systemd are unfounded: Upstart has a huge battery of tests, the competition has virtually none. Upstart knows everything it wants to be, the competition wants to be everything. Quality comes from focus and clarity of purpose, it comes from careful design and rigorous practices. After a review by the Ubuntu Foundations team our course is clear: we’re committed to Upstart, it’s the better choice for a modern init, innit. For our future on cloud and client, Upstart is crisp, clean and correct. It will be a pleasure to share all the Upstart-enablement patches we carry with other family friends as soon as their release is ready and they can take a breath, so to speak.

From a styling point of view, we think in terms of quadruples: this next release starts a cycle of four, which will culminate in 14.04 LTS. So there’s an opportunity to refresh the look. That will kick off with a project on typography to make sure we are expressing ourselves with crystal clarity – making the most of Ubuntu’s Light and Medium font weights for a start. And a project on iconography, with the University of Reading, to refine the look of apps and interfaces throughout the platform. It’s amazing how quaint the early releases of Ubuntu look compared to the current style. And we’re only just getting started! In our artistic explorations we want to embrace tessellation as an expression of the part-digital, part-organic nature of Ubuntu. We love the way tessellated art expresses both the precision and reliability of our foundations, and the freedom and collaboration of a project driven by people making stuff for people. There’s nothing quixotic in our desire to make Ubuntu the easiest, steadiest, and most beautiful way to live digitally.

On the fauna front, the quotable campaign for the Queer Quokka is quorate but, it must sometimes be said, this is not a democracy. One man’s favourite furball is another’s mangy marsupial. No, the quintessential stories of Q will be all about style on the client, with a refresh of our theme and typography, a start on new iconography and perhaps even a new form factor taking flight. So brown is out and something colourful and light is called for. On the cloud front, the new virtualized network madness called Quantum will make its appearance. Being a first cut, it’s more likely to be Folsom than wholesome, but it’s going to be worth calling out, and the name is reminiscent of our package-oriented practices, where goodness is delivered one piece at a time. And so the stage is set for a decision:  I give you the Quantal Quetzal, soon to be dressed in tessellated technicolour, now open for toolchains, kernels and other pressing preparatory packages.

It may not be riding rockets, but bee-keeping has a certain edge to it when you’re allergic to their stings.

Last autumn we set up three hives on a corner of the garden. Andrew and the botanical team have since given the hives proper plinths, we’ve registered them with the local authorities, and it’s become a rite of passage for willing guests to join a hive inspection or feeding.

Claire agreed to join, if I could find a pink bee suit. Turns out, white ones dye really well:

Claire in her new bee suit

The bees just call her "Blossom" with a capital B

For those of you who, like me, know virtually nothing about bees or beekeeping, here’s the short short version.

Beehives are cleverly designed as stacks of rectangular boxes without a floor or a ceiling. You combine different types of boxes to get different things done. For example, the main body of the hive lives in the base box, called a brood box. That’s where the queen hangs out, the comb there has both honey cells and hatching cells (think honey-eating bee-maggots, more charmingly called brood). The worker bees move freely up and down the hive, from box to box. You put a filter above the brood box to stop the queen, who is too big to fit through the grille, from laying eggs upstairs.

You get boxes for feeding the bees, and boxes for collecting honey. The honey-collecting boxes are called “supers”, and the others, which are more like spacing boxes with room inside for gadgets and gizmos are called ekes (possibly Norse, for “augment”?). Since our colonies are young, they haven’t yet filled out their brood boxes, so I’ve focused on feeding them. I accidentally drowned half the bees in each of the colonies in the first attempt by failing to install the feeders properly so, all in all, I’ve come to appreciate how fragile the colonies can be.

Throughout the course of the year, there are different things to watch out for, or get done. Early in spring you give the hives a dose of syrup to get them started. We started feeding them in March.

This Easter weekend, with my brother Bradley & guest visiting, and spring springing, seemed like a perfect occasion to see how the colonies were doing. We made some feeding syrup last night, woke up at sparrowfart this morning and suited up. Our goal was to check out the hives, make sure they seemed healthy & happy, and replenish the food to help the colonies grow quickly.

Things got a little adventurous while we were inspecting the brood frames of the first and most vigorous of the hives. Two of us ended up with bees inside our veils, so we closed up quickly and beat a hasty retreat for a brush-off and tea. I got stung on the neck, but I think the sting failed to set properly so the epi-pens are all intact. Some disappointment in the household that nobody got to jab me in the heart with one.

Restored, we went back to finish off the other hives, and Bradley spotted a queen – first time I’ve seen one of them. Once we all relaxed it was a pleasure to work through the frames together one by one. As Bradley said, there are millions of years of evolution telling you that you urgently have to be somewhere else every time the hive buzzes. We’ve got to figure out the best way to clear each frame without ending up surrounded by a cloud of upset bees. But it all went smoothly.

So, all things being well, next year we’ll have honey that is genuinely local. And we’ll get to test that old story about local honey being good for hayfever. All in the name of science, of course.

As we move from “tens” to “hundreds” to “thousands” of nodes in a typical data centre we need new tools and practices. This hyperscale story – of hyper-dense racks with wimpy nodes – is the big shift in the physical world which matches the equally big shift to cloud computing in the virtualised world. Ubuntu’s popularity in the cloud comes in part from being leaner, faster, more agile. And MAAS – Metal as a Service – is bringing that agility back to the physical world for hyperscale deployments.

Servers used to aspire to being expensive. Powerful. Big. We gave them names like “Hercules” or “Atlas”. The bigger your business, or the bigger your data problem, the bigger the servers you bought. It was all about being beefy – with brands designed to impress, like POWER and Itanium.

Things are changing.

Today, server capacity can be bought as a commodity, based on the total cost of compute: the cost per teraflop, factoring in space, time, electricity. We can get more power by adding more nodes to our clusters, rather than buying beefier nodes. We can increase reliability by doubling up, so services keep running when individual nodes fail. Much as RAID changed the storage game, this scale-out philosophy, pioneered by Google, is changing the server landscape.

In this hyperscale era, each individual node is cheap, wimpy and, by historical standards for critical computing, unreliable. But together, they’re unstoppable. The horsepower now resides in the cluster, not the node. Likewise, the reliability of the infrastructure now depends on redundancy, rather than heroic performances from specific machines. There is, as they say, safety in numbers.

We don’t even give hyperscale nodes proper names any more – ask “node-0025904ce794”. Of course, you can still go big with the cluster name. I’m considering “Mark’s Magnificent Mountain of Metal” – significantly more impressive than “Mark’s Noisy Collection of Fans in the Garage”, which is what Claire will probably call it. And that’s not the knicker-throwing kind of fan, either.

The catch to this massive multiplication in node density, however, is in the cost of provisioning. Hyperscale won’t work economically if every server has to be provisioned, configured  and managed as if it were a Hercules or an Atlas. To reap the benefits, we need leaner provisioning processes. We need deployment tools to match the scale of the new physical reality.

That’s where Metal as a Service (MAAS) comes in. MAAS makes it easy to set up the hardware on which to deploy any service that needs to scale up and down dynamically – a cloud being just one example. It lets you provision your servers dynamically, just like cloud instances – only in this case, they’re whole physical nodes. “Add another node to the Hadoop cluster, and make sure it has at least 16GB RAM” is as easy as asking for it.

With a simple web interface, you can  add, commission, update and recycle your servers at will.  As your needs change, you can respond rapidly, by adding new nodes and dynamically re-deploying them between services. When the time comes, nodes can be retired for use outside the MAAS.

As we enter an era in which ATOM is as important in the data centre as XEON, an operating system like Ubuntu makes even more sense. Its freedom from licensing restrictions, together with the labour saving power of tools like MAAS, make it cost-effective, finally, to deploy and manage hundreds of nodes at a time

Here’s another way to look at it: Ubuntu is bringing cloud semantics to the bare metal world. What a great foundation for your IAAS.

Holistic UI is smarter UX

Tuesday, March 27th, 2012

In the open source community, we celebrate having pieces that “do one thing well”, with lots of orthogonal tools compounding to give great flexibility. But that same philosophy leads to shortcomings on the GUI / UX front, where we want all the pieces to be aware of each other in a deeper way.

For example, we consciously place the notifications in the top right of the screen, avoiding space that is particularly precious (like new tab titles, and search boxes). But the indicators are also in the top right, and they make menus, which drop down into the same space a notification might occupy.

Since we know that notifications are queued, no notification is guaranteed to be displayed instantly, so a smarter notification experience would stay out of the way while you were using indicator menus, or get out of the way when you invoke them. The design story of focusayatana, where we balance the need for focus with the need for awareness, would suggest that we should suppress awareness-oriented things in favour of focus things. So when you’re interacting with an indicator menu, we shouldn’t pop up the notification. Since the notification system, and the indicator menu system, are separate parts, the UNIX philosophy sells us short in designing a smart, smooth experience because it says they should each do their thing individually.

Going further, it’s silly that the sound menu next/previous track buttons pop up a notification, because the same menu shows the new track immediately anyway. So the notification, which is purely for background awareness, is distracting from your focus, which is conveying exactly the same information!

But it’s not just the system menus. Apps can play in that space too, and we could be better about shaping the relationship between them. For example, if I’m moving the mouse around in the area of a notification, we should be willing to defer it a few seconds to stay out of the focus. When I stop moving the mouse, or typing in a window in that region, then it’s OK to pop up the notification.

It’s only by looking at the whole, that we can design great experiences. And only by building a community of both system and application developers that care about the whole, that we can make those designs real. So, thank you to all of you who approach things this way, we’ve made huge progress, and hopefully there are some ideas here for low-hanging improvements too :)