Archive for the 'launchpad' Category

Soyuz: distro package management in Launchpad

Tuesday, November 1st, 2005

The last few evenings have been devoted to the part of Launchpad called Soyuz, which tracks all the distributions and packages that are registered in Launchpad. It’s the heart of the “distro management” capability of Launchpad, that allows customers to manage their own Ubuntu derivative through the web. And of course we will be dogfooding it constantly, but managing Ubuntu itself in it.

Wednesday, all going well, we will open up the Dapper archive based on Launchpad, which will be the first time that we will have this stuff in production. Expect delays for the first few days as we iron out the kinks.

My own work has been the first cut of the pages that display which package is published where in the archive at any given time. That sounds very straightforward, until you realise how fiendishly complicated a distro package archive can be. They are very much structures that have evolved, and so a lot of the complexity comes from having to deal with the badness of reality.

For example, there is no guarantee that a newly upload package will build on all the supported architectures. So say I upload Apache 2.1.3-1 source to Ubuntu, and it builds on all architectures. You would expect to have those binaries published on all architectures in short order. Now, we upload 2.1.3-2, which builds on all architectures except for PPC. This is where the fun starts. The “current” version of Apache is now different, depending on which architecture you are looking at.

One resolution might be to say “build everywhere and reject if it fails anywhere”. But we are constantly adding new architectures, and as those become more esoteric the chances of a build failure rises dramatically. We can’t hold back the distro when a package fails to build on ARM. So the system has to be able to handle this divergence between architectures.

But it gets even more interesting. There is no rule, for example, that a package called foo on i386 should have the same description, or dependencies, as a package with the same name (and even the same version) on amd64. They are just entirely different beasts. So to do this justice, someone has to drill all the way down to the lowest level before they see stuff that’s human readable. Even though 99.9% of the time, pacakges with the same name and version on different architectures WILL have the same details.

So I’ve put quite a lot of work into trying to balance ease of user interface and technical correctness. At the lowest level, you should see exactly what’s published, which versions, when they were built (and more detail on those than you could have imagined you really, really wanted). At a higher level, however, you should be able to search and find the “obvious answers”.

I’m sure there are bugs, and improvements to come, but it’s nice finally to see this code going into production. Try following the link above to the Ubuntu pages in Launchpad, then searching for packages, and send me your suggestions on how we can improve things.

Postgres FTI with SQLObject issue resolved

Friday, October 14th, 2005

Ah joy. Figured out that SQLObject can be extended fairly easily to handle the ranking problem I referred to previously.

The goal is to end up with a query like this:

SELECT Foo.id, Foo.name, Foo.title, rank(Foo.fti, ftq('browser')) AS rank
FROM Foo   WHERE Foo.fti @@ ftq('browser')
ORDER BY rank

So, hacking SQLObject to add a selectAlso= option to SQLObject.select gives us the ability to write code like this:

results = Foo.select("fti @@ ftq('browser')",
selectAlso="rank(fti, ftq('browser')) AS rank",
orderBy='-rank')

Voila. I’ll pass the SQLObject patch on to Stub for a review and submission upstream.

Postgres FTI and SQLObject

Thursday, October 13th, 2005

Postgres is a truly awesome database. When we started working on Launchpad I wasn’t sure if it would be up to the job. I was so wrong. It’s been robust, fast, and *professional* in every regard.

Stub has got full text search pretty smoothly integrated into Launchpad, using tsearch2. Today I setup the source package searching engine in the Soyuz part of Launchpad (see the page at http://launchpad.net/distros/ubuntu/+search) and used tsearch2. What I could not figure out, however, was how to rank the results by the quality of match.

We are using SQLObject. So the code ends up looking like this:

result = DistributionSourcePackageCache.select("fti @@ ftq(%s)" % sqlvalues(searchtext))

Now I have the resultset, but it does not seem to be in the optimally ranked order, and I can’t figure out how to get it that way. Does anyone know? Ah well, Stub will wake up shortly, will bother him then.

Launchpad hacking in Sau Carlos, Brazil

Friday, July 22nd, 2005

I’m in Sao Carlos, Brazil for a few weeks hacking on The Launchpad, and enjoying some of the local scenery. It struck me on yesterday’s early morning run (Kiko’s suggestion – I don’t generally like running) how lucky I am to be able to take a few weeks and work with this team, despite the intensity of everything else that’s going on at Canonical.

Normally I guess the Founder or CEO of a project would not have the luxury of diving into the heart of the technical challenges we face, and you could argue I should be focused on some of the more corporate activities a startup needs to deal with. But with the mix of non-profit and for-profit goals in Canonical and Ubuntu, I sort of feel it’s my great privilege to be able to participate in the hacking, too. There’s plenty of time to build value that will grow the project beyond what I can provide philanthropically – for the moment it’s all about creating interesting platforms for collaboration in the open source world. And that’s the part that I’m particularly interested in myself.

This week the focus has been Baz (the revision control system) as well as a web view of the distributed revision control world (all the branches people have publicly released) which we call “The Bazaar“. At the moment it’s pretty vestigial – it just shows some stats about the number of baz branches we know about that are related to upstream projects we care about. But in future it will let you see more details of each of those branches. The idea being that you can get a high-level view of all the hacking that is going on AROUND a project upstream, not just on the mainline branch. In a distributed revision control world, like the one the Linux kernel guys adopted initially with BK and now with Git, you might have lots of really interesting work going on outside of the mainline tree, so this web service will give you a view of all of that work. That should help create better collaboration between people interested in a particular feature.