When once you have tasted flight, you will forever walk the earth with your eyes turned skyward, for there you have been and there you will always long to return.
— Leonardo da Vinci
This is a series of posts on reasons to choose Ubuntu for your public or private cloud work & play.
We run an extensive program to identify issues and features that make a difference to cloud users. One result of that program is that we pioneered dynamic image customisation and wrote cloud-init. I’ll tell the story of cloud-init as an illustration of the focus the Ubuntu team has on making your devops experience fantastic on any given cloud.
Ever struggled to find the “right” image to use on your favourite cloud? Ever wondered how you can tell if an image is safe to use, what keyloggers or other nasties might be installed? We set out to solve that problem a few years ago and the resulting code, cloud-init, is one of the more visible pieces Canonical designed and built, and very widely adopted.
Traditionally, people used image snapshots to build a portfolio of useful base images. You’d start with a bare OS, add some software and configuration, then snapshot the filesystem. You could use those snapshots to power up fresh images any time you need more machines “like this one”. And that process works pretty amazingly well. There are hundreds of thousands, perhaps millions, of such image snapshots scattered around the clouds today. It’s fantastic. Images for every possible occasion! It’s a disaster. Images with every possible type of problem.
The core issue is that an image is a giant binary blob that is virtually impossible to audit. Since it’s a snapshot of an image that was running, and to which anything might have been done, you will need to look in every nook and cranny to see if there is a potential problem. Can you afford to verify that every binary is unmodified? That every configuration file and every startup script is safe? No, you can’t. And for that reason, that whole catalogue of potential is a catalogue of potential risk. If you wanted to gather useful data sneakily, all you’d have to do is put up an image that advertises itself as being good for a particular purpose and convince people to run it.
There are other issues, even if you create the images yourself. Each image slowly gets out of date with regard to security updates. When you fire it up, you need to apply all the updates since the image was created, if you want a secure machine. Eventually, you’ll want to re-snapshot for a more up-to-date image. That requires administration overhead and coordination, most people don’t do it.
That’s why we created cloud-init. When your virtual machine boots, cloud-init is run very early. It looks out for some information you send to the cloud along with the instruction to start a new machine, and it customises your machine at boot time. When you combine cloud-init with the regular fresh Ubuntu images we publish (roughly every two weeks for regular updates, and whenever a security update is published), you have a very clean and elegant way to get fresh images that do whatever you want. You design your image as a script which customises the vanilla, base image. And then you use cloud-init to run that script against a pristine, known-good standard image of Ubuntu. Et voila! You now have purpose-designed images of your own on demand, always built on a fresh, secure, trusted base image.
Auditing your cloud infrastructure is now straightforward, because you have the DNA of that image in your script. This is devops thinking, turning repetitive manual processes (hacking and snapshotting) into code that can be shared and audited and improved. Your infrastructure DNA should live in a version control system that requires signed commits, so you know everything that has been done to get you where you are today. And all of that is enabled by cloud-init. And if you want to go one level deeper, check out Juju, which provides you with off-the-shelf scripts to customise and optimise that base image for hundreds of common workloads.
Every detail matters, and building great software means taking time to remove the papercuts. Ubuntu has over the past 5 years been refined in many ways to feel amazingly comfortable on the cloud. In the very early days of EC2 growth the Ubuntu team recognised how many developers were enjoying fast access to infrastructure on demand, and we set about polishing up Ubuntu to be amazing on the cloud.
This was a big program of work; the Linux experience had many bad assumptions baked in – everything had been designed to be installed once on a server then left largely untouched for as long as possible, but cloud infrastructure was much more dynamic than that.
We encouraged our team to use the cloud as much as possible, which made the work practical and motivated people to get it right themselves. If you want to catch all the little scratchy bits, make it part of your everyday workflow. Today, we have added OpenStack clouds to the mix, as well as the major public clouds. Cloud vendors have taken diverse approaches to IAAS so we find ourselves encouraging developers to use all of them to get a holistic view, and also to address any cloud-specific issues that arise. But the key point is – if it’s great for us, that’s a good start on making it great for everybody.
Then we set about interviewing cloud users and engaging people who were deep into cloud infrastructure to advise on what they needed. We spent a lot of time immersing ourselves in the IAAS experience through the eyes of cloud users – startups and industrial titans, universities and mid-sized, everyday companies. We engaged the largest and fastest-moving cloud users like Netflix, who have said they enjoy Ubuntu as a platform on the cloud. And that in turn drove our prioritisation of paper-cuts and significant new features for cloud users.
We also looked at the places people actually spend time developing. Lots of them are on Ubuntu desktops, but Windows and MacOS are popular too, and it takes some care to make it very easy for folks there to have a great devops experience.
All of this is an industrial version of the user experience design process that also powers our work on desktop, tablet and phone – system interfaces and applications. Devops, sysadmins, developers and their managers are humans too, so human-centric design principles are just as important on the infrastructure as they are on consumer electronics and consumer software. Feeling great at the command line, being productive as an operator and a developer, are vital to our community and our ecosystem. We keep all the potency of Linux with the polish of a refined, designed environment.
Along the way we invented and designed a whole raft of key new pieces of Ubuntu. I’ll write about one of them, cloud-init, next. The net effect of that work makes Ubuntu really useful on every cloud. That’s why the majority of developers using IAAS do so on Ubuntu.
Check out “loving the bottom edge” for the most important bit of design guidance for your Ubuntu mobile app.
This work has been a LOT of fun. It started when we were trying to find the zen of each edge of the screen, a long time back. We quickly figured out that the bottom edge is by far the most fun, by far the most accessible. You can always get to it easily, it feels great. I suspect that’s why Apple has used the bottom edge for their quick control access on IOS.
We started in the same place as Apple, thinking that the bottom edge was so nice we wanted it for ourselves, in the system. But as we discussed it, we started to think that the app developer was the one who deserved to do something really distinctive in their app with it instead. It’s always tempting to grab the tastiest bit for oneself, but the mark of civility is restraint in the use of power and this felt like an appropriate time to exercise that restraint.
Importantly you can use it equally well if we split the screen into left and right stages. That made it a really important edge for us because it meant it could be used equally well on the Ubuntu phone, with a single app visible on the screen, and on the Ubuntu tablet, where we have the side stage as a uniquely cool way to put phone apps on tablet screens alongside a bigger, tablet app.
The net result is that you, the developer, and you, the user, have complete creative freedom with that bottom edge. There are of course ways to judge how well you’ve exercised that freedom, and the design guidance tries to leave you all the freedom in the world while still providing a framework for evaluating how good the result will feel to your users. If you want, there are some archetypes and patterns to choose from, but what I’d really like to see is NEW patterns and archetypes coming from diverse designs in the app developer community.
Here’s the key thing – that bottom edge is the one thing you are guaranteed to want to do more innovatively on Ubuntu than on any other mobile platform. So if you are creating a portable app, targeting a few different environments, that’s the thing to take extra time over for your Ubuntu version. That’s the place to brainstorm, try out ideas on your friends, make a few mockups. It’s the place you really express the single most important aspects of your application, because it’s the fastest, grooviest gesture in the book, and it’s all yours on Ubuntu.
In the open source community, we celebrate having pieces that “do one thing well”, with lots of orthogonal tools compounding to give great flexibility. But that same philosophy leads to shortcomings on the GUI / UX front, where we want all the pieces to be aware of each other in a deeper way.
For example, we consciously place the notifications in the top right of the screen, avoiding space that is particularly precious (like new tab titles, and search boxes). But the indicators are also in the top right, and they make menus, which drop down into the same space a notification might occupy.
Since we know that notifications are queued, no notification is guaranteed to be displayed instantly, so a smarter notification experience would stay out of the way while you were using indicator menus, or get out of the way when you invoke them. The design story of focusayatana, where we balance the need for focus with the need for awareness, would suggest that we should suppress awareness-oriented things in favour of focus things. So when you’re interacting with an indicator menu, we shouldn’t pop up the notification. Since the notification system, and the indicator menu system, are separate parts, the UNIX philosophy sells us short in designing a smart, smooth experience because it says they should each do their thing individually.
Going further, it’s silly that the sound menu next/previous track buttons pop up a notification, because the same menu shows the new track immediately anyway. So the notification, which is purely for background awareness, is distracting from your focus, which is conveying exactly the same information!
But it’s not just the system menus. Apps can play in that space too, and we could be better about shaping the relationship between them. For example, if I’m moving the mouse around in the area of a notification, we should be willing to defer it a few seconds to stay out of the focus. When I stop moving the mouse, or typing in a window in that region, then it’s OK to pop up the notification.
It’s only by looking at the whole, that we can design great experiences. And only by building a community of both system and application developers that care about the whole, that we can make those designs real. So, thank you to all of you who approach things this way, we’ve made huge progress, and hopefully there are some ideas here for low-hanging improvements too 🙂
The desktop remains central to our everyday work and play, despite all the excitement around tablets, TV’s and phones. So it’s exciting for us to innovate in the desktop too, especially when we find ways to enhance the experience of both heavy “power” users and casual users at the same time. The desktop will be with us for a long time, and for those of us who spend hours every day using a wide diversity of applications, here is some very good news: 12.04 LTS will include the first step in a major new approach to application interfaces.
This work grows out of observations of new and established / sophisticated users making extensive use of the broader set of capabilities in their applications. We noticed that both groups of users spent a lot of time, relatively speaking, navigating the menus of their applications, either to learn about the capabilities of the app, or to take a specific action. We were also conscious of the broader theme in Unity design of leading from user intent. And that set us on a course which led to today’s first public milestone on what we expect will be a long, fruitful and exciting journey.
The menu has been a central part of the GUI since Xerox PARC invented ’em in the 70’s. It’s the M in WIMP and has been there, essentially unchanged, for 30 years.
The original Macintosh desktop, circa 1984, courtesy of Wikipedia
We can do much better!
Say hello to the Head-Up Display, or HUD, which will ultimately replace menus in Unity applications. Here’s what we hope you’ll see in 12.04 when you invoke the HUD from any standard Ubuntu app that supports the global menu:
Snapshot of the HUD in Ubuntu 12.04
The intenterface – it maps your intent to the interface
This is the HUD. It’s a way for you to express your intent and have the application respond appropriately. We think of it as “beyond interface”, it’s the “intenterface”. This concept of “intent-driven interface” has been a primary theme of our work in the Unity shell, with dash search as a first class experience pioneered in Unity. Now we are bringing the same vision to the application, in a way which is completely compatible with existing applications and menus.
The HUD concept has been the driver for all the work we’ve done in unifying menu systems across Gtk, Qt and other toolkit apps in the past two years. So far, that’s shown up as the global menu. In 12.04, it also gives us the first cut of the HUD.
Menus serve two purposes. They act as a standard way to invoke commands which are too infrequently used to warrant a dedicated piece of UI real-estate, like a toolbar button, and they serve as a map of the app’s functionality, almost like a table of contents that one can scan to get a feel for ‘what the app does’. It’s command invocation that we think can be improved upon, and that’s where we are focusing our design exploration.
As a means of invoking commands, menus have some advantages. They are always in the same place (top of the window or screen). They are organised in a way that’s quite easy to describe over the phone, or in a text book (“click the Edit->Preferences menu”), they are pretty fast to read since they are generally arranged in tight vertical columns. They also have some disadvantages: when they get nested, navigating the tree can become fragile. They require you to read a lot when you probably already know what you want. They are more difficult to use from the keyboard than they should be, since they generally require you to remember something special (hotkeys) or use a very limited subset of the keyboard (arrow navigation). They force developers to make often arbitrary choices about the menu tree (“should Preferences be in Edit or in Tools or in Options?”), and then they force users to make equally arbitrary effort to memorise and navigate that tree.
The HUD solves many of these issues, by connecting users directly to what they want. Check out the video, based on a current prototype. It’s a “vocabulary UI”, or VUI, and closer to the way users think. “I told the application to…” is common user paraphrasing for “I clicked the menu to…”. The tree is no longer important, what’s important is the efficiency of the match between what the user says, and the commands we offer up for invocation.
In 12.04 LTS, the HUD is a smart look-ahead search through the app and system (indicator) menus. The image is showing Inkscape, but of course it works everywhere the global menu works. No app modifications are needed to get this level of experience. And you don’t have to adopt the HUD immediately, it’s there if you want it, supplementing the existing menu mechanism.
It’s smart, because it can do things like fuzzy matching, and it can learn what you usually do so it can prioritise the things you use often. It covers the focused app (because that’s where you probably want to act) as well as system functionality; you can change IM state, or go offline in Skype, all through the HUD, without changing focus, because those apps all talk to the indicator system. When you’ve been using it for a little while it seems like it’s reading your mind, in a good way.
We’ll resurrect the (boring) old ways of displaying the menu in 12.04, in the app and in the panel. In the past few releases of Ubuntu, we’ve actively diminished the visual presence of menus in anticipation of this landing. That proved controversial. In our defence, in user testing, every user finds the menu in the panel, every time, and it’s obviously a cleaner presentation of the interface. But hiding the menu before we had the replacement was overly aggressive. If the HUD lands in 12.04 LTS, we hope you’ll find yourself using the menu less and less, and be glad to have it hidden when you are not using it. You’ll definitely have that option, alongside more traditional menu styles.
Voice is the natural next step
Searching is fast and familiar, especially once we integrate voice recognition, gesture and touch. We want to make it easy to talk to any application, and for any application to respond to your voice. The full integration of voice into applications will take some time. We can start by mapping voice onto the existing menu structures of your apps. And it will only get better from there.
But even without voice input, the HUD is faster than mousing through a menu, and easier to use than hotkeys since you just have to know what you want, not remember a specific key combination. We can search through everything we know about the menu, including descriptive help text, so pretty soon you will be able to find a menu entry using only vaguely related text (imagine finding an entry called Preferences when you search for “settings”).
There is lots to discover, refine and implement. I have a feeling this will be a lot of fun in the next two years 🙂
Even better for the power user
The results so far are rather interesting: power users say things like “every GUI app now feels as powerful as VIM”. EMACS users just grunt and… nevermind ;-). Another comment was “it works so well that the rare occasions when it can’t read my mind are annoying!”. We’re doing a lot of user testing on heavy multitaskers, developers and all-day-at-the-workstation personas for Unity in 12.04, polishing off loose ends in the experience that frustrated some in this audience in 11.04-10. If that describes you, the results should be delightful. And the HUD should be particularly empowering.
Even casual users find typing faster than mousing. So while there are modes of interaction where it’s nice to sit back and drive around with the mouse, we observe people staying more engaged and more focused on their task when they can keep their hands on the keyboard all the time. Hotkeys are a sort of mental gymnastics, the HUD is a continuation of mental flow.
Ahead of the competition
There are other teams interested in a similar problem space. Perhaps the best-known new alternative to the traditional menu is Microsoft’s Ribbon. Introduced first as part of a series of changes called Fluent UX in Office, the ribbon is now making its way to a wider set of Windows components and applications. It looks like this:
You can read about the ribbon from a supporter (like any UX change, it has its supporters and detractors ;-)) and if you’ve used it yourself, you will have your own opinion about it. The ribbon is highly visual, making options and commands very visible. It is however also a hog of space (I’m told it can be minimised). Our goal in much of the Unity design has been to return screen real estate to the content with which the user is working; the HUD meets that goal by appearing only when invoked.
Instead of cluttering up the interface ALL the time, let’s clear out the chrome, and show users just what they want, when they want it.
Time will tell whether users prefer the ribbon, or the HUD, but we think it’s exciting enough to pursue and invest in, both in R&D and in supporting developers who want to take advantage of it.
Other relevant efforts include Enso and Ubiquity from the original Humanized team (hi Aza &co), then at Mozilla.
Our thinking is inspired by many works of science, art and entertainment; from Minority Report to Modern Warfare and Jef Raskin’s Humane Interface. We hope others will join us and accelerate the shift from pointy-clicky interfaces to natural and efficient ones.
Roadmap for the HUD
There’s still a lot of design and code still to do. For a start, we haven’t addressed the secondary aspect of the menu, as a visible map of the functionality in an app. That discoverability is of course entirely absent from the HUD; the old menu is still there for now, but we’d like to replace it altogether not just supplement it. And all the other patterns of interaction we expect in the HUD remain to be explored. Regardless, there is a great team working on this, including folk who understand Gtk and Qt such as Ted Gould, Ryan Lortie, Gord Allott and Aurelien Gateau, as well as designers Xi Zhu, Otto Greenslade, Oren Horev and John Lea. Thanks to all of them for getting this initial work to the point where we are confident it’s worthwhile for others to invest time in.
We’ll make sure it’s easy for developers working in any toolkit to take advantage of this and give their users a better experience. And we’ll promote the apps which do it best – it makes apps easier to use, it saves time and screen real-estate for users, and it creates a better impression of the free software platform when it’s done well.
From a code quality and testing perspective, even though we consider this first cut a prototype-grown-up, folk will be glad to see this:
Overall coverage rate:
lines......: 87.1% (948 of 1089 lines)
functions..: 97.7% (84 of 86 functions)
branches...: 63.0% (407 of 646 branches)
Landing in 12.04 LTS is gated on more widespread testing. You can of course try this out from a PPA or branch the code in Launchpad (you will need thesetwo branches). Or dig deeper with blogs on the topic from Ted Gould, Olli Ries and Gord Allott. Welcome to 2012 everybody!
Our goal with Unity is unprecedented ease of use, visual style and performance on the Linux desktop. With feature freeze behind us, we have a refined target render of the Dash for Oneiric, and here it is:
We’ve moved from the idea of “Places” to a richer set of “Scopes and Lenses”. Scopes are data sources, and can tap into any online or offline data set as long as they can generate categorised results for a search, describe a set of filters and support some standard interfaces. Lenses are various ways to present the data that come from Scopes.
The Scopes have a range of filtering options they can use, such as ratings (“show me all the 5 star apps in the Software Center please”) and categories (“… that are games or media related”). Over time, the sophistication of this search system will grow but the goal is to keep it visual and immediate – something anyone can drive at first attempt.
This delivers on the original goal of creating a device-like experience that was search driven. Collaboration with the always-excellent Zeitgeist crew (quite a few of whom are now full time on the Unity team!) has improved the search experience substantially, kudos to them for the awesome work they’ve put in over the past six months. Since we introduced the Dash as a full screen device-like search experience, the same idea has made its way into several other shells, most notably Mac OS X Lion. While we’re definitely the outsider in this contest, I think we can stay one step ahead in the game given the support of our community.
The existing Places are all in the process of being updated to the Scopes and Lenses model, it’s a bit of a construction site at the moment so hard-hats are advised but dive in if you have good ideas for some more interesting scopes. I’ve heard all sorts of rumours about cool scopes in the pipeline 😉 and I bet this will be fertile ground for innovation. It’s pretty straightforward to make a scope, I’m sure others will blog and document the precise mechanisms but for those who want a head start, just use the source, Luke.
Panel evolution
In the panel, you’ll see that the top left corner is now consistently used to close whatever has the focus. Maximising a window keeps the window controls in the same position relative to the window – the top left corner. We have time to refine the behaviour of this based on user testing of the betas; for example, in one view, one should be able to close successive windows from that top left corner even if they are not maximised.
It’s taken several releases of careful iteration to get to this point. Even though we had a good idea where we were headed, each step needed to be taken one release at a time. Perhaps this might make a little clearer the reasons for the move of window controls to the left – it was the only place where we could ultimately keep them consistent all the way up to a maximised window with the title bar integrated into the panel. I’m confident this part will all be settled by 12.04.
As part of this two-step shuffle, the Dash invocation is now integrated in the Launcher. While this is slightly less of a Fitts-fantastic location, we consider it appropriate for a number of reasons. First, it preserves the top left corner for closing windows. Second, the Dash is best invoked with the Super key (sometimes erroneously and anachronistically referred to as the “Windows” key, for some reason ;-)). And finally, observations during user testing showed people as more inclined to try clicking on items in the Launcher than on the top left icon in the panel, unless that icon was something explicit like a close button for the window. Evidence based design rules.
Visual refinements
Rather than a flat darkening, we’re introducing a wash based on the desktop colour. The dash thus adjusts to your preferred palette based on your wallpaper. The same principle will drive some of the login experience – choosing a user will shift the login screen towards that users wallpaper and palette.
We’ve also integrated the panel and the dash, so indicators are rendered in a more holographic fashion inside the dash. Together with efforts to mute the contrast of Launcher icons the result is a more striking dash altogether: content is presented more dramatically.
Since we have raw access to the GL pipeline, we’re taking advantage of that with some real-time blur effects to help the readability and presentation of overlay content in the Dash, too. Both Nux in the case of Unity-3D and Qt in the case of Unity-2D have rich GL capabilities, and we’d like to make the most of whatever graphics stack you have on your hardware, while still running smoothly on the low end.
Growing community and ecosystem
A project like this needs diverse perspectives, talents and interests to make it feel rounded and complete. While Canonical is carrying the core load, and we’re happy to do so in order to bring this level of quality to the Ubuntu desktop user experience, what makes me particularly optimistic is the energy of the contributors both to Unity directly and to the integration of many other components and applications with the platform.
On the contribution front, a key goal for the Unity community is to maintain velocity in contributor patch flows. You should expect a rapid review and, all being well, landing, for contributions to Unity that are in line with the design goals. In a few cases we’ve also accepted patches that make it possible to use Unity in ways that are different to the design goals, especially where the evidence doesn’t lean very heavily one way or the other. Such contributions add some complexity but also give us the opportunity to test alternatives in a very rich way; the winning alternative would certainly survive, the other might not.
Contrary to common prognostication, this community is shaping up to be happy and productive. For those who do so for love and personal interest, participating in such a community should be fun and stimulating, an opportunity to exercise skills or pursue interests, give something back that will be appreciated and enjoyed by many, and help raise the bar for Linux experiences. I’d credit Jorge and others for their stewardship of this so far, and my heartfelt thanks to all of those who have helped make Unity better just for the fun of it.
Outside of the core, the growing number of apps that integrate sweetly with the launcher (quicklists), dash (scopes), indicators (both app-specific and category indicators) is helping to ensure that API’s are useful, refined and well implemented, as well as improving the experience of Ubuntu users everywhere. Now that we’re moving to Unity by default for both 2D and 3D, that’s even more valuable.
Under the hood
In this round, Unity-3D and Unity-2D have grown together and become twin faces on the same underlying model. They now share a good deal of common code and common services and – sigh – common bugs :-). But we’re now at the point where we can be confident that the Unity experience is available on the full range of hardware, from lightweight thin client systems made of ARM or Atom CPU’s to CADstations with oodles of GPU horsepower.
There’s something of a competition under way between proponents of the QML based Unity-2D, who believe that the GL support here is good enough to compete both at the high end and on the low end, and the GL-heads in Unity-3D, who think that the enhanced experiences possible with raw GL access justify the additional complexity of working in C++ and GL on the metal. Time will tell! Since a lot of the design inspiration for Unity came from game interfaces, I lean to the “let’s harness all the GL we can for the full 3D experience” side of the spectrum, but I’m intrigued with what the QML team are doing.
A wit said of Google Wave “if your project depends on reinventing scrollbars, you are doing something wrong.” But occasionally, just occasionally, one gets to do exactly that.
Under the Ayatana banner, we’ve been on a mission to make the desktop have less chrome and more content. The goal is to help people immerse themselves in their stuff, without being burdened with large amounts of widgetry which isn’t relevant to the thing they are trying to concentrate on. And when it comes to widgetry, there are few things left which take up more space than scrollbars.
For example, I spend plenty of time in a full screen terminal, and it’s lovely to see how clean that experience is on Natty today:
…but that scrollbar on the right seems heavy and outdated. We took inspiration from mobile devices, and started exploring the idea of making scrollbars be more symbolic, and less physical. Something like this:
Of course, since the desktop isn’t often a touch device, we need to think through pointer interactions. We wanted to preserve the idea of keeping content exposed as much as possible, while still providing for pointer interaction when needed. We also decided to drop the “one line scroll” capability, while preserving the ability to page up and down. Take a look at the result:
The design work behind this has been done by Christian Giordano, who worked through the corner cases (literally) and provided a mockup for testing purposes. And the heavy lifting for Natty is being done by the indefatigable Andrea Cimitan, who is currently polishing up a gtk implementation of the concept for the release. Christian put together a blog post on the subject, and a great video which talks through the design process and a few of the challenges and solutions found:
Multitouch is just as useful on a desktop as it is on a phone or tablet, so I’m delighted that the first cut of Canonical’s UTouch framework has landed in Maverick and will be there for its release on 10.10.10.
You’ll need 4-finger touch or better to get the most out of it, and we’re currently targeting the Dell XT2 as a development environment so the lucky folks with that machine will get the best results today. By release, we expect you’ll be able to use it with a range of devices from major manufacturers, and with addons like Apple’s Magic Trackpad.
The design team has lead the way, developing a “touch language” which goes beyond the work that we’ve seen elsewhere. Rather than single, magic gestures, we’re making it possible for basic gestures to be chained, or composed, into more sophisticated “sentences”. The basic gestures, or primitives, are like individual verbs, and stringing them together allows for richer interactions. It’s not quite the difference between banging rocks together and conducting a symphony orchestra, but it feels like a good step in the right direction 😉
The new underlying code is published on Launchpad under the GPLv3 and LGPLv3, and of course there are quite a lot of modules for things like X and Gtk which may be under licenses preferred by those projects. There’s a PPA if you’re interested in tracking the cutting edge, or just branch / push/ merge on LP if you want to make it better. Details in the official developer announcement. The bits depend on Peter Hutterer’s recently published update to the X input protocols related to multi-touch, and add gesture processing and gesture event delivery. I’d like to thank Duncan McGreggor for his leadership of the team which implemented this design, and of course all the folks who have worked on it so far: Henrik Rydberg, Rafi Rubin, Chase Douglas, Stephen Webb at the heart of it, and many others who have expanded on their efforts.
In Maverick, quite a few Gtk applications will support gesture-based scrolling. We’ll enhance Evince to show some of the richer interactions that developers might want to add to their apps. Window management will be gesture-enabled in Unity, so 10.10 Netbook Edition users with touch screens or multi-touch pads will have sophisticated window management at their fingertips. Install Unity on your desktop for a taste of it, just apt-get install ubuntu-netbook and choose the appropriate session at login.
The roadmap beyond 10.10 will flesh out the app developer API and provide system services related to gesture processing and touch. It would be awesome to have touch-aware versions of all the major apps – browser, email, file management, chat, photo management and media playback – for 11.04, but that depends on you! So if you are interested in this, let’s work up some branches 🙂 Here’s the official Canonical blog post, too.
The Ayatana Indicators work has given us a crisp, clean basis for indicators in the panel. We’ve said they will all look a particular way, and behave a particular way. And we’ve said they will be placed on the right of the panel.
But why limit indicators to the panel? Let’s make it possible for applications to use indicators themselves, for all the things that indicators are good at:
Conveying a particular state, such as whether or not the application is connected,
Providing a handle for the indicator menu, to modify that state
We’ll start with “window indicators”, or “windicators” for fun. Windicators are indicators displayed in the window title bar that behave just like the indicators in the panel: they have an icon which shows state, and clicking on the icon brings up a menu. Applications can create, update and remove window indicators using an API more or less like the AppIndicator framework first put to use in 10.04 LTS.
Window indicators, or "windicators", shown in a sample application window.
We’ve carefully placed all the panel indicators on the right, and we’ve carefully put the window controls and window title on the left. So now we have all this space on the right. As a pattern, it would fit to put the window indicators there.
Cody Russell is leading some work in Canonical around the technology which actually draws the window title bar and borders. It’s called “client side window decorations”. We are moving the rendering of the window decorations into the app itself, so that you don’t have the window manager and application drawing those pieces separately. That simplifies certain things (of course it also makes some things harder).
One of the most interesting consequences of the client-side decorations work is that it means that the application could more easily draw into the titlebar (because the application is drawing the title bar). And that makes it even more natural for the application to control the right side of the window title bar as well.
Update: Several commenters correctly pointed out that window indicators could just as easily be rendered by window managers in cases where the theme is not CSD-based. CSD provided the inspiration for giving that space to the application, it’s not essential to the implementation. It would be fantastic for window indicators to be available on 😉
Less chrome, more content: banish the status bar
I’m on a “less is more” kick with our design efforts, and one of the things I want to banish is wasted vertical space. For netbooks, that’s particularly important. And a lot of applications have status bars at the bottom, for no good reason other than it was that way in Windows 3.1.
Typically the application status bar has:
Some status icons (“online”)
Some tools (“Yslow”)
A transient status message (“Saving draft…”)
We can replace these with a combination of windicators and temporary, overlay status bars. I really liked the Chrome browser’s use of overlay status messages, so kudos and thanks to them for the inspiration. The net result of those two steps, in apps where we can, is to save about 5% of the vertical space for your stuff – real content.
Prioritising examples for implementation
If you’re interested in this idea, please join the Ayatana mailing list and participate in the design discussions there. We’d like to develop some patterns that are generic, so that we can use a common icon and possibly also common indicator menu entries for addressing the same issue in diverse applications. Of course, applications will be free to use the mechanism for things that are unique to them.
Candidates for 10.10
It would be fantastic to implement a few of these window indicators for 10.10. Please help us choose the most useful cases! Currently on the list are:
Online / offline status indicator and toggle options for the mail client, chat program or Gwibber, the broadcast messages application.
An “unsaved” indicator, that tells people that the contents of the file they are working on have changed and potentially lets them save it or set autosave properties.
Progress indicators, which show that an action is in progress, and possibly also indicate the extent of the progress. The associated menu would enable one to pause or cancel the operation, and perhaps define the behaviour on completion of the action.
A “basket” indicator, which shows if any items have been selected for purchase,
Sharing indicators, which would show if a document is shared with multiple people, and enable one to setup such a share.
Volume indicators, which would show the loudness of application audio streams, and enable one to set the volume for that specific application.
The key thing is that these indicators are entirely application-specific, and ideally only relevant to the window that you are actually looking at.
Just like Panel Indicators…
From a visual design point of view, again the goal would be to ensure that indicators are symbolic. They would follow the same styling as Ayatana indicators:
Monochrome by default, with shape indicating the function of the indicator
Semantically colored: with red for critical problems, orange for alerts, green for positive status changes and blue for informative states that are not the default or usual state.
Integrated with the Netbook Edition Smart Panel
Last week I blogged about our decision to adopt a single, global menu for all applications, in the panel. And I also said we would explore putting the window title *and* menu into the panel, when the window is maximised. Of course, that means that we need to accommodate the window indicators in the panel as well.
So: when the window is maximised, and we are using a smart which can include both indicators and window titles, the window indicators will be inserted into the panel as well. They will appear on the right of the panel, and be the leftmost indicators. For example, here is the application, maximised (note the dodgy Ubuntu logo in the top left – that’s the panel, not the window title bar you’re looking at):
In this configuration, the system achieves “singular purpose”: the entire screen is devoted to a single application, yet the Ayatana elements continue to serve their purpose, either systemic (the battery indicator) or application specific.
When we set up Project Ayatana to improve the usability of the whole desktop, we called it Ayatana because we were focused on the “sphere of consciousness”, one’s awareness of what’s going on outside of the current application. There are two key aspects to the work:
Notifications are “awareness distilled” in the sense that you cannot interact with them at all. We designed them as ephemeral “click-transparent” messages, implemented in Notify-OSD. Their sole purpose is to notify you of transient events.
Indicator Menus combine persistent awareness of a state with a set of options for modifying that state.
In this blog I’ll outline the arc of our work on indicator menus to date, and the trajectory we expect it to follow. We’re about a year into the effort, all told, and I think it will take another 18 months before we can consider it baked. It should be done by 12.04 LTS. This is an iterative process, and things are in flux right now. I hope, when we are happy that we can commit to ABI stability, that Gnome and KDE will adopt the work too. For the moment, the rapid pace of evolution has meant that we’re depending on fantastic upstreams to keep up with us as things move.
Goals of the Ayatana Indicators
The indicators are designed to create a persistent awareness of state, or an awareness of a persistent state. They complement notifications: they are persistent, when notifications are ephemeral. You might miss a notification, but you should always be able to check your indicators. You can interact with indicators, using their menus, in contrast with the un-clickable notifications.
We value:
Support for both GNOME and KDE. Both desktop environments are important in Ubuntu. We encourage the teams to reflect a pure vision of each, but it’s also the case that users will want to run a GNOME application on Kubuntu occasionally, or vice versa. So we have to make sure the work is considered from the perspectives of developers on either side, and we have to provide APIs and libraries that work in both environments.
Accessibility. Indicators are critical elements of awareness. Whether you are connected, what the time is, whether you are online, whether your battery will last long enough for you to finish your work, whether you have messages… these are all vital to a complete computing experience. We have to make sure that visual and other disabilities can be addressed.
Familiarity and Innovation. As always, these are in tension with one another. Innovation helps us put free software at the front of the curve, but it creates the risk of breaking people’s habits and expectations.
Consistency and Usability. We want the end result to be more usable in the whole, and we are willing to lose individual nuggets if that helps make the whole more valuable.
Streamlining. There are too many indicators, that aren’t clear enough about their intent. There are also many indicators from different applications which do roughly the same thing, but in slightly different ways. The value of all the indicators is enhanced if there are fewer of them, and they are more obvious to discover and use.
Some firm decisions
Those values lead us to some anchor decisions:
D-Bus for communications. A messaging approach makes it straightforward to adopt consistent patterns across different desktop environments. We will provide wrapper libraries for both Gtk and Qt applications to access the indicator capabilities. A Qt application running on Ubuntu should “feel native” when it’s using indicators correctly. And vice-versa. The messaging approach also lets us handle accessibility in a better way: we don’t have to accommodate every possible disability visually, because we can have agents that interpret the indicator messages and handle it in ways that are appropriate for a particular disability.
Opinionated placement. We will place all indicators at the top right of the screen on GNOME. We’ll place them in a particular order, too, with the “most fundamental” indicator, which controls the overall session, in the top right. The order will not be random, but predictable between sessions and screen sizes. There will be no GUI support for users to reorder the indicators.
Constrained behaviour. All the indicators will take the form of an indicator (icon or text), and a menu. Clicking on an indicator will open its menu. Keyboard navigation will always work, and left and right arrows will translate either into submenu navigation or flipping from indicator to indicator. The whole set of indicators on the panel will be navigable as a single menu, in essence. We won’t support “right click” on indicators differently from “left click”, and there’ll be no ability for arbitrary applications to define arbitrary behaviours to arbitrary events on indicators.
Symbolic visuals. We want to pare back the visual representation of status presented by the icons. We don’t believe that visual accessibility for the disabled need drive the design of the standard icon set, as there will be both alternative icons, renderings, and entirely different options such as speech or custom devices to handle those. Colors on the indicators should have semantic purpose and be used mainly for alerts and awareness, while the shape of the icon should define its purpose.
The first part of our work was pure housekeeping. The panel in Ubuntu is very generic, it lets you put all sorts of different gadgets in all sorts of different places, and those gadgets can behave in all sorts of different ways. The result has been to stimulate innovation, but it has also made the panel very inconsistent and ultimately less useful.
We reviewed the way Ubuntu-specific applications were using the panel, and set out to clean them up. Update-manager lost its persistent notification in favour of the more direct popup window. Others will follow.
We decided to introduce a new gadget on the panel which would be a container for all the indicators which follow our new Ubuntu Ayatana pattern. And we started work on a set of indicators that would fit inside that container. Thus far, we’ve done the session, “me”, and sound indicators.
We also created a framework for applications which want to display their own indicator. That’s the AppIndicators work, which has been fantastically lead in 10.04 LTS by Jorge Castro, coordinating with many upstreams to ensure that their applications feel tightly integrated into the panel.
The icon visual design turned into a conversation about “-symbolic” icons at UDS in Dallas, and is now being realised in the ubuntu-mono icon theme in 10.04 LTS. There is work under way to make symbolic icons a more formal and rigorous construct that can be themed, and we’ll participate in that effort or offer an alternative implementation.
9 parts perspiration, 1 part innovation
The detailed design of a large set of systemic indicators, together with the work to make them all look, feel and behave in a consistent fashion, has been substantial effort involving MPT, Ted Gould, Cody Russell and many others. There’s still a lot of work to do. Conor Curran and Kalle Valo joined the team in this latest cycle. There is a great deal that remains to be done.
We also aspire to introduce some new and innovative concepts.
Category Indicators
In order to reduce the number of indicators and improve the persistence and usefulness of the indicators that remain, we’ve introduced the idea of “category indicators”. These are indicators into which multiple, similar applications can embed themselves. Instead of having a different indicator each application, we have one indicator for the whole category.
The messaging indicator, which aggregates awareness about many different types of messages from real people, is an example. Instead of having three different icons for email, IM and Identi.ca or Twitter, Ubuntu has just one messaging indicator, which can make you aware of important messages in any of those applications.
The three default applications for those lines of communication all share the same indicator. They are part of the same category. There are custom API’s for messaging applications which let them:
Insert entries in the messaging menu which are displayed even when the application is not running. Useful for helping people go straight to the activity. Instead of having to check if the email client is running, then switching to it or launching it, then going to the message composition window, I should *always* be able to compose a new message with just two clicks, regardless of whether or not the mail client is running initially.
Add custom menu entries to the messaging menu that are relevant to them. Each applications gets a “section” in the category indicator menu, and they can add custom menu entries to their section.
Register themselves as applications that should be shown in the messaging menu, or remove themselves from that menu. The default applications will show up there unless they are uninstalled or expressly configured not to use the messaging menu. Other applications will put themselves there by default when they are run by that user, who can also configure them not to display there.
Show whether they are running, a state which is indicated with a small “play” style triangle next to the application icon in the menu.
There are also some behaviours which are collective across all the applications in the category. For example, any of the applications can set the messaging indicator to an alert state, signalling that it’s worth clicking on.
Each category indicator supports a unique API that’s relevant for that category. There are some common features, for example the ability of applications to register and de-register for the indicators and the ability to add menu entries, but the details might vary substantially from one category to another.
The underlying goal is to make it clearer to users “what all of those icons are about”. There are fewer of them, and the ones that are there are more persistent – they are always there, and they always do the same sort of thing. “You’ve got a message” is useful no matter which channel the message came through. The net result is that the whole set of indicators feels tighter and better defined.
The session indicator, which displays the shutdown / restart menu, has a similar capability that replaced the “restart required” panel icon in 10.04 LTS. Since the session menu already contains the “restart” menu option, the session menu will now be set into an alert state when you need to restart. The “Restart…” menu option is changed to “Restart Required…” (though I would now prefer something like “Restart, completing updates…”).
The battery indicator shows the status of all of your batteries, from laptop to UPS to mouse and wireless keyboard. Other applications and devices which have battery information should be able to insert themselves there appropriately.
Similarly, all the calendar and alarm applications might fit into the Clock Indicator. And perhaps all the applications which have downloads might use a single category for that – there’s some discussion along those lines on the Ayatana list at the moment.
Timelines and iterations
The basic “add an indicator with a menu” capability is there now, and was used for Application Indicators in 10.04 LTS.
What complicates the picture from a delivery perspective is our evolving understanding of how best to organise the category indicators. For example, at the moment we are aggregating received messages in the messaging indicator, but the send or broadcast elements of those same communications channels are accessed through the Me menu, where we track presence. That has been controversial – sensible folks think we should perhaps restructure that to bring the elements together.
Each arrangement of category indicators involves shaping the API’s in new ways, because the categories are fundamentally different from one another, and we want to design custom indicators for each category. Not only are the individual category indicator designs changing, but the arrangement of categories themselves is subject to active debate and experimentation, which is important to getting a crisp final result.
We can’t be certain that the current configuration is the best one, and want the flexibility to continue to evolve and reshape the APIs accordingly. We expect it will take at least three iterations of Ubuntu to be certain, and that we can commit to ABI stability for 12.04 LTS onwards.
