Distributed Development with Mercurial

Every clone of a Mercurial repository can act as a consistent, fully functional repository itself. This is a very useful property of a DVCS; one that you can take advantage of to make development a lot easier, especially for teams that are dispersed in multiple places of the globe.

Imagine for a moment that your team is not sharing the same physical space every day. In fact, half of your team works in an office in the west coast of the United States and the other half is located somewhere in Europe. With a centralized VCS you would have to pick one of the offices and install a central repository there, e.g. the west coast offices:

Two teams sharing a central repository

The result of a setup like this is that your team is now split in two parts, with different access speeds and round-trip times to the repository.

The west coast team is closer to the repository and can access everything in the repository with satisfactory speed. Pulling changes from the repository is fast. Pushing changes back to the repository is equally fast. Browsing the repository history over a web interface is reasonably fast too.

The European team is the child of a lesser god. To access any part of the repository they have to go through a transatlantic link. The speed of creating a new clone of the full Mercurial repository depends on all sorts of factors, including (but alas not limited to):

  • The speed of their connection to the US repository. The availability of “fast enough” speeds to the outside world may be small, entirely non-existent, or it may cost a prohibitively large amount of money. Since this is not a technical problem of the central repository, the European developers may be penalized with excruciatingly slow access speeds for reasons they do not control and cannot do anything to change.
  • Bandwidth limits or traffic volume limits from Europe to the US repository may not be enough. For a sufficiently large repository, like the OpenSolaris onnv-gate branch or the FreeBSD head branch, it may be required to transfer a couple hundred megabytes of data, or even more, for a full clone of the branch. If the local European connection has traffic volume limits per-day or per-month it’s easy to hit a volume cap like this in a mere matter of days (even in less than a day, if you have a dozen developers).
  • The availability of someone who can help with repository issues at the US side may be limited or not available at convenient times. Timezone differences play an important role here. With a timezone drift of even 3 or 4 hours, it is often impractical or difficult to find someone at the other end of the time drift in the wee hours of the morning or very late in the afternoon.

With a DVCS, like Mercurial, you can take advantage of the “mirror only” clones described in an earlier post, to set up a local mirror of the US repository in Europe:

A local Europe-based mirror of the US repository

With a DVCS, like Mercurial, the European repository mirror is a fully functional “repository” of its own, allowing Bill, Chloé and Marie to access any part of the repository history, to create as many local clones as they see fit, and to keep working with the repository mirror even if the remote repo-1 repository is down. A cron job may be set up in the European side, to pull changes from the US-based repo-1 repository, periodically synchronizing the reference repositories of the two sub-teams. The changesets pushed to the US repository are transferred only once, by the cron job, instead of multiple times per developer, every time someone from Europe wants to see what the US sub-team has been working on.

It is still possible for someone at the European side to pull changes directly from the US-based repo-1 repository. It just doesn’t make sense to do this all the time, because it is inherently slower.

There is, however, a catch. At the mirroring diagram I have only shown how the European repository can be synchronized with the US-based repository. How do the developers working from the US side see what their European colleagues have been doing? Having two repositories solves the immediate problem of access speed for every sub-team, but it does not seem amenable or particularly enabling to multiple concurrent lines of development and mutual synchronization of these two repositories.

One way to synchronize the two repositories is to assign the role of “integrator” to one of the developers of each side. An integrator is responsible for pulling from both sides of the world, merging the changes of each repository in a consistent and functional whole, testing that everything still works and then pushing the merged state of the repository to both sides. Bill could take the role of the integrator for the European sub-team, for example, and push changesets back into the US clone:

Integration of the European and US clones and push-back to the US repository by Bill

Bill can now schedule an “integration window” in the European side, and coordinate the push-back to the US repository with both sub-teams. He can pick a known, stable version of the European repository, run all the tests he finds useful in his personal integration clone, and keep testing the European sources until he feels confident that they pass all his tests or that they satisfy the requirements of the US team.

In the meantime, the European sub-team can keep working in their “repo-2” repository, pretty much oblivious to everything that happens to the integration tree of Bill. To them, the particular state of Bill’s integration clone at any random moment in time is pretty much irrelevant. If there are important changes happening in Bill’s clone, they will get to see them after the push-back to repo-1 and repo-2 happens. This means there is a far smaller need for hard locks in either of the two master repositories. Nobody has to sit back and wait in thumb-dwiddling mode until Bill is done.

Anyone who has had to wait for a branch to “unlock” in order to keep working on a bug-fix or even just to commit that small, innocuous documentation fix they have in the queue, will probably feel a profound sense of relief when they realize that this is possible. Hard branch locks are always annoying. Nobody really likes them, but it is only through the flexibility and power of a DVCS like Mercurial that the need for this sort of inconvenience is easily and conveniently eliminated.

A DVCS can go even further than that though. It can help your development processes become safer too. One of the problems with the integration and push-back shown in the previous diagram is that Bill is now a single point of failure. What happens when Bill is on sick leave for a month, or is fired, or leaves the team for his own reasons? What happens if the US team is working hard and fast on that great feature for tomorrow but Bill is not available because it’s the middle of the night in Europe?

Well, the answer to that is simple too. Someone from the US team can pick up the ball and perform an integration step on either side of the world. James, for example, could act as a fallback integrator when Bill is absent or unavailable:

Integration at both sides of the fence

The symmetry of the picture is not a requirement of synchronization between two teams. I deliberately chose to present the two integrators in a very similar manner, to show that there is nothing in the way the two Mercurial repositories work that prevents one of the teams to act in either of the two roles: the master/reference repository and the target repository of a synchronization. This is a direct consequence of the fact that all Mercurial clones are fully functional, stand-alone repositories. The roles of master/target are not assigned by the DVCS itself, depending on the location or owner of a clone. These roles can switch back and forth, to fit your own needs at any particular moment in time.

Mercurial clones, when seen under this light, are an instance of the old BSD adage:

Provide tools, not policies. Provide tools that are flexible, and the users will implement the policies that fit their own needs and satisfy their own requirements.

What is more important is that with a local, fast, complete mirror of the repository in each site, the developers of each location can really do their work faster and easier, working most of the time with a vastly smaller level of annoyance or dependency on the unpredictable round-trip of a remote central repository. That’s distributed development at work, and that’s why you should really have a look at Mercurial if you are not using it already.

You can start today, by pointing your browser to: http://mercurial.selenic.com/

1 thought on “Distributed Development with Mercurial

  1. mperedim

    FWIW, a number of problems that you raise here may be supposedly alleviated if the centralized VCS supports a proxy-like architecture and a proxy is installed in the European office.

    Supposedly :-P

Comments are closed.