Mark Shuttleworth | here be dragons

With the release of LXC 2.0 and LXD, we now have a pure-container hypervisor that delivers bare-metal performance with a standard Linux guest OS experience. Very low latency, very high density, and very high control of specific in-guest application processes compared to KVM and ESX make it worth checking out for large-scale Linux virtualisation operations.

Even better, the drivers to enable LXD as a hypervisor in  OpenStack, are maturing upstream.

That means you get bare metal performance on OpenStack for Linux workloads, without actually giving people the whole physical server. LXD supports live migration so you can migrate those users to a different physical server with no downtime, which is great for maintenance. And you can have all the nice Openstack semantics for virtual networks etc without having to try very hard.

By contrast, Ironic has the problem that the user can now modify any aspect of the machine as if you gave them physical access to it. In most cases, that’s not desirable, and in public clouds it’s a fun way to let the NSA (and other agencies) install firmware for your users to enjoy later.

NSA-as-a-Service does have a certain ring to it though.

Canonical just announced a new, free, and very cool way to provide thousands of IP addresses to each of your VMs on AWS. Check out the fan networking on Ubuntu wiki page to get started, or read Dustin’s excellent fan walkthrough. Carry on here for a simple description of this happy little dose of awesome.

Containers are transforming the way people think about virtual machines (LXD) and apps (Docker). They give us much better performance and much better density for virtualisation in LXD, and with Docker, they enable new ways to move applications between dev, test and production. These two aspects of containers – the whole machine container and the process container, are perfectly complementary. You can launch Docker process containers inside LXD machine containers very easily. LXD feels like KVM only faster, Docker feels like the core unit of a PAAS.

The density numbers are pretty staggering. It’s *normal* to run hundreds of containers on a laptop.

And that is what creates one of the real frustrations of the container generation, which is a shortage of easily accessible IP addresses.

It seems weird that in this era of virtual everything that a number is hard to come by. The restrictions are real, however, because AWS restricts artificially the number of IP addresses you can bind to an interface on your VM. You have to buy a bigger VM to get more IP addresses, even if you don’t need extra compute. Also, IPv6 is nowehre to be seen on the clouds, so addresses are more scarce than they need to be in the first place.

So the key problem is that you want to find a way to get tens or hundreds of IP addresses allocated to each VM.

Most workarounds to date have involved “overlay networking”. You make a database in the cloud to track which IP address is attached to which container on each host VM. You then create tunnels between all the hosts so that everything can talk to everything. This works, kinda. It results in a mess of tunnels and much more complex routing than you would otherwise need. It also ruins performance for things like multicast and broadcast, because those are now exploding off through a myriad twisty tunnels, all looking the same.

The Fan is Canonical’s answer to the container networking challenge.

We recognised that container networking is unusual, and quite unlike true software-defined networking, in that the number of containers you want on each host is probably roughly the same. You want to run a couple hundred containers on each VM. You also don’t (in the docker case) want to live migrate them around, you just kill them and start them again elsewhere. Essentially, what you need is an address multiplier – anywhere you have one interface, it would be handy to have 250 of them instead.

So we came up with the “fan”. It’s called that because you can picture it as a fan behind each of your existing IP addresses, with another 250 IP addresses available. Anywhere you have an IP you can make a fan, and every fan gives you 250x the IP addresses. More than that, you can run multiple fans, so each IP address could stand in front of thousands of container IP addresses.

We use standard IPv4 addresses, just like overlays. What we do that’s new is allocate those addresses mathematically, with an algorithmic projection from your existing subnet / network range to the expanded range. That results in a very flat address structure – you get exactly the same number of overlay addresses for each IP address on your network, perfect for a dense container setup.

Because we’re mapping addresses algorithmically, we avoid any need for a database of overlay addresses per host. We can calculate instantly, with no database lookup, the host address for any given container address.

More importantly, we can route to these addresses much more simply, with a single route to the “fan” network on each host, instead of the maze of twisty network tunnels you might have seen with other overlays.

You can expand any network range with any other network range. The main idea, though, is that people will expand a class B range in their VPC with a class A range. Who has a class A range lying about? You do! It turns out that there are a couple of class A networks that are allocated and which publish no routes on the Internet.

We also plan to submit an IETF RFC for the fan, for address expansion. It turns out that “Class E” networking was reserved but never defined, and we’d like to think of that as a new “Expansion” class. There are several class A network addresses reserved for Class E, which won’t work on the Internet itself. While you can use the fan with unused class A addresses (and there are several good candidates for use!) it would be much nicer to do this as part of a standard.

The fan is available on Ubuntu on AWS and soon on other clouds, for your testing and container experiments! Feedback is most welcome while we refine the user experience.

Configuration on Ubuntu is super-simple. Here’s an example:

In /etc/network/fan:

# fan 241
241.0.0.0/8 172.16.3.0/16 dhcp

In /etc/network/interfaces:

iface eth0 static
address 172.16.3.4
netmask 255.255.0.0
up fanctl up 241.0.0.0/8 172.16.3.4/16
down fanctl down 241.0.0.0/8 172.16.3.4/16

This will map 250 addresses on 241.0.0.0/8 to your 172.16.0.0/16 hosts.

Docker, LXD and Juju integration is just as easy. For docker, edit /etc/default/docker.io, adding:

DOCKER_OPTS=”-d -b fan-10-3-4 –mtu=1480 –iptables=false”

You must then restart docker.io:

sudo service docker.io restart

At this point, a Docker instance started via, e.g.,

docker run -it ubuntu:latest

will be run within the specified fan overlay network.

Enjoy!

Subhu writes that OpenStack’s blossoming project list comes at a cost to quality. I’d like to follow up with an even leaner approach based on an outline drafted during the OpenStack Core discussions after ODS Hong Kong, a year ago.

The key ideas in that draft are:

Only call services “core” if the user can detect them.

How the cloud is deployed or operated makes no difference to a user. We want app developers to

Define both “core” and “common” services, but require only “core” services for a cloud that calls itself OpenStack compatible.

Separation of core and common lets us recognise common practice today, while also acknowledging that many ideas we’ve had in the past year or three are just 1.0 iterations, we don’t know which of them will stick any more than one could predict which services on any major public cloud will thrive and which will vanish over time. Signalling that something is “core” means it is something we commit to keeping around a long time. Signalling something is “common” means it’s widespread practice for it to be available in an OpenStack environment, but not a requirement.

Require that “common” services can be self-deployed.

Just as you can install a library or a binary in your home directory, you can run services for yourself in a cloud. Services do not have to be provided by the cloud infrastructure provider, they can usually be run by a user themselves, under their own account, as a series of VMs providing network services. Making it a requirement that users can self-provide a service before designating it common means that users can build on it; if a particular cloud doesn’t offer it, their users can self-provide it. All this means is that the common service itself builds on core services, though it might also depend on other common services which could be self-deployed in advance of it.

Require that “common” services have a public integration test suite that can be run by any user of a cloud to evaluate conformance of a particular implementation of the service.

For example, a user might point the test suite at HP Cloud to verify that the common service there actually conforms to the service test standard. Alternatively, the user who self-provides a common service in a cloud which does not provide it can verify that their self-deployed common service is functioning correctly. This also serves to expand the test suite for the core: we can self-deploy common services and run their test suites to exercise the core more thoroughly than Tempest could.

Keep the whole set as small as possible.

We know that small is beautiful; small is cleaner, leaner, more comprehensible, more secure, easier to test, likely to be more efficiently implemented, easier to attract developer participation. In general, if something can be cut from the core specification it should. “Common” should reflect common practice and can be arbitrarily large, and also arbitrarily changed.

In the light of those ideas, I would designate the following items from Subhu’s list as core OpenStack services:

• Keystone (without identity, nothing)
• Nova (the basis for any other service is the ability to run processes somewhere)
  • Glance (hard to use Nova without it)
• Neutron (where those services run)
  • Designate (DNS is a core aspect of the network)
• Cinder (where they persist data)

I would consider these to be common OpenStack services:

• SWIFT (widely deployed, can be self-provisioned with Cinder block backends)
• Ceph RADOS-GW object storage (widely deployed as an implementation choice, common because it could be self-provided on Cinder block)
• Horizon (widely deployed, but we want to encourage innovation in the dashboard)

And these I would consider neither core nor common, though some of them are clearly on track there:

• Barbican (not widely implemented)
• Ceilometer (internal implementation detail, can’t be common because it requires access to other parts)
• Juju (not widely implemented)
• Kite (not widely implemented)
• HEAT (on track to become common if it can be self-deployed, besides, I eat controversy for breakfast)
• MAAS (who cares how the cloud was built?)
• Manila (not widely implemented, possibly core once solid, otherwise common once, err, common)
• Sahara (not widely implemented, weird that we would want to hardcode one way of doing this in the project)
• Triple-O (user doesn’t care how the cloud was deployed)
• Trove (not widely implemented, might make it to “common” if widely deployed)
• Tuskar (see Ironic)
• Zaqar (not widely implemented)

In the current DefCore discussions, the “layer” idea has been introduced. My concern is simple: how many layers make sense? End users don’t want to have to figure out what lots of layers mean. If we had “OpenStack HPC” and “OpenStack Scientific” and “OpenStack Genomics” layers, that would just be confusing. Let’s keep it simple – use “common” as a layer, but be explicit that it will change to reflect common practice (of course, anything in common is self-reinforcing in that new players will defer to norms and implement common services, thereby entrenching common unless new ideas make services obsolete).

Quick question – we have Cloud Foundry in private beta now, is there anyone in the Ubuntu community who would like to use a Cloud Foundry instance if we were to operate that for Ubuntu members?

This is a series of posts on reasons to choose Ubuntu for your public or private cloud work & play. When you see Ubuntu on a cloud it means that Canonical has a working relationship with that cloud vendor, and the Ubuntu images there come with a set of guarantees:

1. Those images are up to date and secure.
2. They have also been optimised on that cloud, both for performance and cost.
3. The images provide a standard experience for app compatibility.

That turns out to be a lot of work for us to achieve, but it makes your life really easy.

Fresh, secure and tasty images

We update the cloud images across all clouds on a regular basis. Updating the image means that you have more of the latest updates pre-installed so launching a new machine is much faster – fewer updates to install on boot for a fully secured and patched machine.

1. At least every two weeks, typically, if there are just a few small updates across the board to roll into the freshest image.
2. Immediately if there is a significant security issue, so starting a fresh image guarantees you to have no known security gotchas.
3. Sooner than usual if there are a lot of updates which would make launching and updating a machine slow.

Updates might include fixes to the kernel, or any of the packages we install by default in the “core” cloud images. We also make sure that these updated images are used by default in any “quick launch” UI that the cloud provides, so you don’t have to go hunt for the right image identity. And there are automated tools that will tell you the ID for the current image of Ubuntu on your cloud of choice. So you can script “give me a fresh Ubuntu machine” for any cloud, trivially. It’s all very nice.

Optimised for your pocket and your workload

Every cloud behaves differently – both in terms of their architecture, and their economics. When we engage with the cloud operator we figure out how to ensure that Ubuntu is “optimal” on that cloud. Usually that means we figure out things like storage mechanisms (the classic example is S3 but we have to look at each cloud to see what they provide and how to take advantage of it) and ensure that data-heavy operations like system updates draw on those resources in the most cost-efficient manner. This way we try to ensure that using Ubuntu is a guarantee of the most cost-effective base OS experience on any given cloud. In the case of more sophisticated clouds, we are digging in to kernel parameters and drivers to ensure that performance is first class. On Azure there is a LOT of deep engineering between Canonical and Microsoft to ensure that Ubuntu gets the best possible performance out of the Hyper-V substrate, and we are similarly engaged with other cloud operators and solution providers that use highly-specialised hypervisors, such as Joyent and VMware. Even the network can be tweaked for efficiency in a particular cloud environment once we know exactly how that cloud works under the covers. And we do that tweaking in the standard images so EVERYBODY benefits and you can take it for granted – if you’re using Ubuntu, it’s optimal. The results of this work can be pretty astonishing. In the case of one cloud we reduced the Ubuntu startup time by 23x from what their team had done internally; not that they were ineffective, it’s just that we see things through the eyes of a large-scale cloud user and care about things that a single developer might not care about as much. When you’re doing something at scale, even small efficiencies add up to big numbers.

Standard, yummy

Before we had this program in place, every cloud vendor hacked their own Ubuntu images, and they were all slightly different in unpredictable ways. We all have our own favourite way of doing things, so if every cloud has a lead engineer who rigged the default Ubuntu the way they like it, end users have to figure out the differences the hard way, stubbing their toes on them. In some cases they had default user accounts with different behaviour, in others they had different default packages installed. EMACS, Vi, nginx, the usual tweaks. In a couple of cases there were problems with updates or security, and we realised that Ubuntu users would be much better off if we took responsibility for this and ensured that the name is an assurance of standard behaviour and quality across all clouds. So now we have that, and if you see Ubuntu on a public cloud you can be sure it’s done to that standard, and we’re responsible. If it isn’t, please let us know and we’ll fix it for you. That means that you can try out a new cloud really easily – your stuff should work exactly the same way with those images, and differences between the clouds will have been considered and abstracted in the base OS. We’ll have tweaked the network, kernel, storage, update mechanisms and a host of other details so that you don’t have to, we’ll have installed appropriate tools for that specific cloud, and we’ll have lined things up so that to the best of our ability none of those changes will break your apps, or updates. If you haven’t recently tried a new cloud, go ahead and kick the tires on the base Ubuntu images in two or three of them. They should all Just Work TM.   It’s frankly a lot of fun for us to work with the cloud operators – this is the frontline of large-scale systems engineering, and the guys driving architecture at public cloud providers are innovating like crazy but doing so in a highly competitive and operationally demanding environment. Our job in this case is to make sure that end-users don’t have to worry about how the base OS is tuned – it’s already tuned for them. We’re taking that to the next level in many cases by optimising workloads as well, in the form of Juju charms, so you can get whole clusters or scaled-out services that are tuned for each cloud as well. The goal is that you can create a cloud account and have complex scale-out infrastructure up and running in a few minutes. Devops, distilled.

This is a series of posts on reasons to choose Ubuntu for your public or private cloud work & play.

We run an extensive program to identify issues and features that make a difference to cloud users. One result of that program is that we pioneered dynamic image customisation and wrote cloud-init. I’ll tell the story of cloud-init as an illustration of the focus the Ubuntu team has on making your devops experience fantastic on any given cloud.

Ever struggled to find the “right” image to use on your favourite cloud? Ever wondered how you can tell if an image is safe to use, what keyloggers or other nasties might be installed? We set out to solve that problem a few years ago and the resulting code, cloud-init, is one of the more visible pieces Canonical designed and built, and very widely adopted.

Traditionally, people used image snapshots to build a portfolio of useful base images. You’d start with a bare OS, add some software and configuration, then snapshot the filesystem. You could use those snapshots to power up fresh images any time you need more machines “like this one”. And that process works pretty amazingly well. There are hundreds of thousands, perhaps millions, of such image snapshots scattered around the clouds today. It’s fantastic. Images for every possible occasion! It’s a disaster. Images with every possible type of problem.

The core issue is that an image is a giant binary blob that is virtually impossible to audit. Since it’s a snapshot of an image that was running, and to which anything might have been done, you will need to look in every nook and cranny to see if there is a potential problem. Can you afford to verify that every binary is unmodified? That every configuration file and every startup script is safe? No, you can’t. And for that reason, that whole catalogue of potential is a catalogue of potential risk. If you wanted to gather useful data sneakily, all you’d have to do is put up an image that advertises itself as being good for a particular purpose and convince people to run it.

There are other issues, even if you create the images yourself. Each image slowly gets out of date with regard to security updates. When you fire it up, you need to apply all the updates since the image was created, if you want a secure machine. Eventually, you’ll want to re-snapshot for a more up-to-date image. That requires administration overhead and coordination, most people don’t do it.

That’s why we created cloud-init. When your virtual machine boots, cloud-init is run very early. It looks out for some information you send to the cloud along with the instruction to start a new machine, and it customises your machine at boot time. When you combine cloud-init with the regular fresh Ubuntu images we publish (roughly every two weeks for regular updates, and whenever a security update is published), you have a very clean and elegant way to get fresh images that do whatever you want. You design your image as a script which customises the vanilla, base image. And then you use cloud-init to run that script against a pristine, known-good standard image of Ubuntu. Et voila! You now have purpose-designed images of your own on demand, always built on a fresh, secure, trusted base image.

Auditing your cloud infrastructure is now straightforward, because you have the DNA of that image in your script. This is devops thinking, turning repetitive manual processes (hacking and snapshotting) into code that can be shared and audited and improved. Your infrastructure DNA should live in a version control system that requires signed commits, so you know everything that has been done to get you where you are today. And all of that is enabled by cloud-init. And if you want to go one level deeper, check out Juju, which provides you with off-the-shelf scripts to customise and optimise that base image for hundreds of common workloads.

Kudos to all the speakers, panellists, designers and engineers who made ODS Atlanta such a great event last week. And thanks in particular to the team at Canonical that helped pull together our keynote, I had a very large number of compliments that really belong to all of you!

For those that didn’t make it, here are a few highlights.

First, Ubuntu is the leading OpenStack distribution, with 55% of all production are using Ubuntu, nearly 5x the number for RHEL. There is a big squabble at the moment between vendors in the RHEL camp; for the record, Canonical is happy to work with vendors of alternative OpenStack distributions on Ubuntu as long as we have a commercial agreement that enables us to support users. Nonetheless, the standard way to do OpenStack starts with Ubuntu followed by the addition of Canonical’s cloud archive, installing OpenStack using those packages.

Second, vendors are focused on interoperability through Canonical’s OpenStack Interop Lab (OIL). We build OpenStack thousands of ways every month with permutations and combinations of code from many vendors. Bring us a Juju charm of your work, sign up to the OIL program and we’ll tell you which other vendors you need to do more work with if you want to be interoperable with their OpenStack offerings.

Third, Juju and MAAS are growing support for Windows and CentOS, with other operating systems on the horizon too (patches welcome!). Thanks to contributions from CloudBase Solutions, you’ll get amazing orchestration of Windows and Linux apps on any cloud or bare metal. If you have a Windows app that you want charmed up, they are the guys to talk to! We did a live on-stage install of OpenStack with Ubuntu KVM and Windows Hyper-V with the beta code, and expect it to land in production Juju / MAAS in the coming weeks.

I’m particularly excited about a new product we’ve announced, which is a flat-fee fully managed on-premise OpenStack solution. Using our architecture and tools, and your hardware, we can give you a best-of-breed OpenStack deployment with SLA for a fixed fee of $15 per server per day. Pretty amazing, and if you are considering OpenStack, definitely an option to evaluate.  Give us a call!

Every detail matters, and building great software means taking time to remove the papercuts. Ubuntu has over the past 5 years been refined in many ways to feel amazingly comfortable on the cloud. In the very early days of EC2 growth the Ubuntu team recognised how many developers were enjoying fast access to infrastructure on demand, and we set about polishing up Ubuntu to be amazing on the cloud.

This was a big program of work; the Linux experience had many bad assumptions baked in – everything had been designed to be installed once on a server then left largely untouched for as long as possible, but cloud infrastructure was much more dynamic than that.

We encouraged our team to use the cloud as much as possible, which made the work practical and motivated people to get it right themselves. If you want to catch all the little scratchy bits, make it part of your everyday workflow. Today, we have added OpenStack clouds to the mix, as well as the major public clouds. Cloud vendors have taken diverse approaches to IAAS so we find ourselves encouraging developers to use all of them to get a holistic view, and also to address any cloud-specific issues that arise. But the key point is – if it’s great for us, that’s a good start on making it great for everybody.

Then we set about interviewing cloud users and engaging people who were deep into cloud infrastructure to advise on what they needed. We spent a lot of time immersing ourselves in the IAAS experience through the eyes of cloud users – startups and industrial titans, universities and mid-sized, everyday companies. We engaged the largest and fastest-moving cloud users like Netflix, who have said they enjoy Ubuntu as a platform on the cloud. And that in turn drove our prioritisation of paper-cuts and significant new features for cloud users.

We also looked at the places people actually spend time developing. Lots of them are on Ubuntu desktops, but Windows and MacOS are popular too, and it takes some care to make it very easy for folks there to have a great devops experience.

All of this is an industrial version of the user experience design process that also powers our work on desktop, tablet and phone – system interfaces and applications. Devops, sysadmins, developers and their managers are humans too, so human-centric design principles are just as important on the infrastructure as they are on consumer electronics and consumer software. Feeling great at the command line, being productive as an operator and a developer, are vital to our community and our ecosystem. We keep all the potency of Linux with the polish of a refined, designed environment.

Along the way we invented and designed a whole raft of key new pieces of Ubuntu. I’ll write about one of them, cloud-init, next. The net effect of that work makes Ubuntu really useful on every cloud. That’s why the majority of developers using IAAS do so on Ubuntu.