Good architectural layering, and Bzr 1.1

Wednesday, January 9th, 2008

I completely failed to blog the release of Bzr 1.0 last year, but it was an excellent milestone and by all accounts, very well received. Congratulations to the Bazaar community on their momentum! I believe that the freeze for 1.1 is in place now so it’s great to see that they are going to continue to deliver regular releases.

I’ve observed a surge in the number of contributors to Bazaar recently, which has resulted in a lot of small but useful branches with bugfixes for various corner cases, operating systems and integrations with other tools. One of the most interesting projects that’s getting more attention is BzrEclipse, integrating Bzr into the Eclipse IDE in a natural fashion.

I think open source projects go through an initial phase where they work best with a tight group of core contributors who get the basics laid out to the point where the tool or application is usable by a wider audience. Then, they need to make the transition from being “closely held” to being open to drive-by contributions from folks who just want to fix a small bug or add a small feature. That’s quite a difficult transition, because the social skills required to run the project are quite different in those two modes. It’s not only about having good social skills, but also about having good processes that support the flow of new, small contributions from new, unproven contributors into the code-base.

It seems that one of the key “best practices” that has emerged is the idea of plug-in architectures, that allow new developers to contribute an extension, plug-in or add-on to the codebase without having to learn too much about the guts of the project, or participate in too many heavyweight processes. I would generalize that and say that good design, with clearly though-through and pragmatic layers, allow new contributors to make useful contributions to the code-base quickly because they present useful abstractions early on.

Firefox really benefited from their decision to support cross-platform add-ons. I’m delighted to hear that OpenOffice is headed in the same direction.

Bazaar is very nicely architected. Not only is there a well-defined plug-in system, but there’s also a very useful and pragmatic layered architecture which keeps the various bits of complexity contained for those who really need to know. I’ve observed how different teams of contributors, or individuals, have introduced whole new on-disk formats with new performance characteristics, completely orthogonally to the rest of the code. So if you are interested in the performance of status and diff, you can delve into working tree state code without having to worry about long-term revision storage or branch history mappings.

Layering can also cause problems, when the layers are designed too early and don’t reflect the pragmatic reality of the code. For example, witness the “exchange of views” between the ZFS folks and the Linux filesystem community, who have very different opinions on the importance and benefits of layering.

Anyhow, kudos to the Bazaar guys for the imminent 1.1, and for adopting an architecture that makes it easier for contributors to get going.

12 comments:

  1. gord allott says: (permalink)
    January 9th, 2008 at 2:22 pm

    plugins can be fun but in the end you just end up with software thats not peer reviewed being part of the ‘core’ system (even if its not distrabuted with the system, such as everyone and their cat saying you *have* to have extension x if you use firefox). i think a better system is to allow for plugins/extensions but try and get them into the mainline development process once they become serious.

  2. Andy Price says: (permalink)
    January 9th, 2008 at 3:05 pm

    (Allow me to play devil’s advocate for a moment) One argument is that more functionality should be encapsulated in libraries that any application can make use of and anything that isn’t user-interface specific should be relocatable and reusable. By breaking out functionality into libraries packages are kept small and manageable and follow the UNIX philosophy closer. Conversely the plug-in model could be a way of saying “I built too much functionality into my core application and now I need to add complexity (the plug-in system) to my application in order to make it act like a library.” Many smaller libraries which do one thing well could arguably increase contributions by reducing complexity and therefore lowering the barrier to entry. It’s just a thought and I don’t feel strongly either way but it’s an area I find very interesting.

    Mark Shuttleworth says: Good point! I think this is why there’s a strong emphasis on libbzr, which makes it straightforward to embed Bazaar into any Python-based application. Some folks have talked about a C implementation too but I don’t think anyone has felt it necessary yet.

  3. Kevin Menard says: (permalink)
    January 10th, 2008 at 12:41 am

    I constantly have a love-hate relationship with plugins. They are great because they do lower the barrier to entry for the average developer. As a guy that needs to install them, though, it’s a pain in the neck. On top of that, you do run into the support issue tha Gord mentions. I’ve run into plenty of situations where I rely on plugins X & Y for some product P. X doesn’t get updated for the new version of P’s API, while Y does, and Y won’t backport bug fixes to the old version. So, now I’m stuck with two plugins I rely on that are largely incompatible with one another.

    I do think the installation issue is one that requires greater attention, however. It’s the reason that I’ll grab Opera before I grab Firefox or IntelliJ IDEA before Eclipse. When the core is stripped down and you need to grab 10 different plugins to get a reasonable system and there’s no way to easily automate it for all of your work environments or all the work environments of your group, plugins lose their novelty really quickly.

    I’d advocate an easy way to install plugins (i.e., have a plugin repository rather than listing them on each individual developer’s site), easy way to duplicate an installation and/or automated deployment, and maybe even wrap up a distribution of the app’s core plus the most popular set of plugins.

    For what it’s worth, everytime I go to check out bzr I see that I need plugins to integrate with other version control systems and I’m immediately put off. It’s far simpler to just stay with what I have than it is to play platform compatibility roulette.

    Mark Shuttleworth says: I agree wholeheartedly with your observations about plug-in installation, updating, maintenance and version control. Firefox does this best, IMO. It automatically looks for updates, and it assesses compatibility between plug-ins and new versions of the browser.

  4. Robins says: (permalink)
    January 10th, 2008 at 1:41 am

    I have been recently been working heavily on a project that i developed both at home and office. Funnily, this meant that two operating systems were used (XP / Gutsy), two databases were used (PostGres, MSSQL) and the icing, two servers were used (FreeBSD / Windows Server 2003).

    But only one feedback after it all. Bzr worked flawlessly !

    Mark, the way I look at it, different development architectures have their own advantages / disadvantages, but what I believe is far more important are the policy decisions that guide the developers. For e.g. personally I believe bzr’s idea of making renaming a first class citizen is *the most important reason* why I chose bzr for this project even though at that time a similar DVCS (HG) was considered better than bzr 0.92 by a few online reviews.

    Put it another way, I believe that in the long run, the thinking behind a project provides a big hint as to whether the project would suit your needs. Then I find in bzr in harmony with my thinking.

  5. Kevin Menard says: (permalink)
    January 10th, 2008 at 2:33 am

    Mark,

    As a side note, you probably want to update the copyright year in your page footer.

  6. tecosystems » links for 2008-01-10 says: (permalink)
    January 10th, 2008 at 5:20 am

    [...] Mark Shuttleworth » Blog Archive » Good architectural layering, and Bzr 1.1 “It seems that one of the key “best practices” that has emerged is the idea of plug-in architectures, that allow new developers to contribute an extension, plug-in or add-on to the codebase without having to learn too much about the guts of the projec (tags: markshuttleworth architecture plugins firefox bazaar layers add-on extension) [...]

  7. Bonne conception architecturale en couches et Bzr 1.1 says: (permalink)
    January 10th, 2008 at 1:28 pm

    [...] Traduction française de l’article “Good architectural layering, and Bzr 1.1“. [...]

  8. Krenar Qehaja says: (permalink)
    January 12th, 2008 at 5:05 pm

    @Kevin Menard

    this could help Mark if it is on php,

    [php]
    echo “Copyright © 2006 – ” .date(“Y”). ” Mark Shuttleworth”;
    [/php]

  9. tecosystems » Do or Do Not Do, There is No OS X says: (permalink)
    January 16th, 2008 at 7:47 am

    [...] “It seems that one of the key ‘best practices’ that has emerged is the idea of plug-in architectures, that allow new developers to contribute an extension, plug-in or add-on to the codebase without having to learn too much about the guts of the project.” Mark Shuttleworth [...]

  10. Ian Skerrett says: (permalink)
    January 18th, 2008 at 6:42 pm

    Mark,

    I definitely agree that a well defined plugin architecture helps accelerate the participation in an open source community. We have definitely seen this within the Eclipse community, for instance BzrEclipse. :-) Tim O’Reilly coined the term ‘Architecture of Participation’ and well defined architecture is at the heart of this concept.

  11. Bazaar: un CVS distribuito | PettiNix says: (permalink)
    February 1st, 2008 at 11:18 am

    [...] stesso patron di Ubuntu, Mark Shuttleworth, spinge nell’aumento del suo utilizzo, indicandolo come soluzione del futuro nella gestione dei [...]

  12. tecosystems » GNOME Do and The Network Backed Plugin Architecture says: (permalink)
    June 23rd, 2008 at 7:02 pm

    [...] Shuttleworth, I think, was right when he said: It seems that one of the key “best practices” that has emerged is the idea of plug-in [...]