Currently I work at the NATO Programming Centre (www.npc.nato.int) as Software Architect. I am in charge of the development of a new console application for MASE (Multi-AEGIS Site Emulator), one of the Air Command and Control applications maintained at the centre. Being one of the most experienced engineers in the software engineering team, I am also the technical mentor for the team in the area of the real-time systems and tactical data links. Angelo has posted 2 posts at DZone. View Full User Profile

Module Re-Use Under Version Control Using Mercurial Guest Repositories

01.31.2013
| 3149 views |
  • submit to reddit

The advantages of modularity are clear to every developer: separation of concerns, divide and conquer, sharing of the development tasks, and increased maintainability.

Another advantage of modularity is code re-use: the functionality provided by a module can be used by multiple applications. For this reason, an API should always strive to be modular, allowing its users take on board only the modules they need.

All these advantages from the development side have a price to pay when it comes to version control: the API modules have their own development path and their users must keep track of which particular version they are compatible with, each release build has to record the versions of the various modules used by the application.

To tackle this problem many tools and frameworks have been and are being created: OSGi, Apache Maven, and Oracle's Jigsaw.

Not as "famous" as the previous ones, is the NetBeans Platform Module System API, which provides an excellent framework for developing modular applications. The Module System API allows to clearly define a dependency from one module onto a second module in terms of its specification and implementation version.

Something the mentioned tools do not provide is version control at source code level. To achieve it, they have to be complemented by a "modules-aware" version control tool.

In this article, I will explain how I achieved this for the projects I am responsible for.

Where I work, our corporate tool for version control is Mercurial and, given this constraint, I had to devise a way that would allow efficient collaboration among projects sharing a code base while keeping, at the same time, strict version control of the source code and of the releases.

While Mercurial is very efficient as a distributed version control tool, by itself it is not module-aware and does not provide the functionality to handle them. Luckily, Mercurial has a number of plug-ins that expand its basic functionality.

The first idea I had was to use the subrepos extension, but reading the page that describes the extension, even Selenic defines it as a "feature of last resort" not leaving big hope about my chances to have it working correctly.

Luckily, looking further, a pretty new extension came to my help: guestrepos.

In order to demonstrate the solution I adopted, I have created a simple scenario showing two separate applications ("modules suites" in NetBeans teminology) that share a common module suite.

Scenario

For the purposes of this demo, the Storage and Display classes in the Enhanced_RCP_Core module suite represent two services used by Application1:

Scenario

In order to be able to access the Enhanced_RCP_Core classes, Application1 has to include the Enhanced_RCP_Core project as shown in the following screenshots:

Scenario

Scenario

Scenario

Scenario

Once the Application1 suite has got visibility of the Enhanced_RCP_Core suite, it is now possible for its modules to add dependencies to public modules contributed by the added project (only the packages explicitly made public are visible from outside the enclosing module).

Scenario

Scenario

Scenario

Once the dependencies are set, the appropriate imports are available in the module classes:

Scenario

And the application compiles correctly.

Scenario

Following the same procedure, the other application Application2 uses the classes of the Enhanced_RCP_Core.

Let's now imagine that the Enhanced_RCP_Core is modified to provide a new storage system:

Scenario

In order to make public that the new service is not available in version 1.0 of the module, the specification version number is increased to 2.0.

Scenario

Application1 developers immediately make use of the new feature:

Scenario

In order to declare the dependency on the modified version of the Enhanced_RCP_Core also the module dependency is set to require version 2.0 of the library.

Scenario

From the version control point of view, we now have Application1 that depends on version 2.0 of the Enhanced_RCP_Core, while Application2 still depends on version 1.0.

NetBeans module system, is well aware of the dependencies, on the other end, the mercurial repositories of the three projects are completely independent and do not know anything about which change-set provides which version of the API.

Only using mercurial is not possible to reproduce a particular build after the applications have been released.
One possible solution to this problem would be to use mercurial tags, the following figure depicts a possible scenario:

Application1

Release Application Change-set Platform Change-set
1.0 2 1
2.0 4 1
3.0 6 3

Application2

Release Application Change-set Platform Change-set
1.0 2 1
2.0 4 2

Following the tags approach, platform change-sets are tagged with the identifiers of the releases of the applications that make use of it.
Scenario

Using this system, the platform repository is polluted with information related to its users,  such a procedure is cumbersome to maintain and difficult to automate in the build process.

Mercurial guest repositories allow to implement a more elegant solution.

From the guestrepos page:

This is an extension for enterprises needing to handle modules and components
Use case:
  • Many products, multiple projects within those products, multiple developers within those projects, multiple branches per developer

  • Source code shared in a multiplicity of configurations

  • Third party code (possibly patched on some branches)

  • A given module may have multiple versions going into the same product

Hg subrepos do not handle the sharing of components well, due to the recursive merge from the top (super) repo and requirement to lock at a specific version.
Guestrepo's goal is to overcome these limitations.
The guestrepo extension does not change any existing Mercurial behavior. It only adds new commands.

The second part of the demo shows hot to make use guest repositories in our Application1 and Application2.
The steps are:

  • downloading and enabling the extension,
  • creating two files: .hggrmapping and .hgguestrepo.

To enable the extension, its path must be added to the hgrc file which is contained in the .hg folder of the project.
The .hggrmapping and .hgguestrepofiles contain the external and internal references of the guest repositories.

Scenario

Once the extension has been enabled and the files created, the two files have to be committed to the repository.
The guest repositories are now setup and they can be pulled with the hg grpull command.

Scenario

The grpull command creates a folder with the given name (Platform in this case) and pull the remote guest repository into it.

Scenario

Opening the suite in NetBeans shows the original name and the tooltip shows the location in the file system.

Scenario

The application must be now modified to point to its own private version of the Enhanced_RCP_Core suite. The procedure is the same explained above:

Scenario

After these steps the guestrepos set-up is complete and the developers can continue to work on the project.

The following steps describe a typical workflow that exploit the guestrepos functionality.

Let's imagine that the developers of Application1 implement an improvement of the Enhanced_RCP_Core, they change the advertised specification version of the platform and make the change.

Scenario

Once the changes in Application1 and in the Enhanced_RCP_Core have been completed, the code can be then committed to the two repositories (main and guest):

Scenario

Scenario

After the commit, the command hg grout shows that there are local changes to be pushed to the main location of the guest repository.

The changes can be pushed with the command hg grpush.

Scenario

One aspect to remind is that the guestrepos commands do not affect the main repository and that if the changes in Application1 would need to be pushed to another repository a normal hg push would have to be made as well.

From Application2 (that has been configured to use guestrepos just like Application1), the command hg grin shows that the platform has some new change sets.

The hg grpull command pulls the changes in the local copy of the repository.

Scenario

Until now, we have seen how the exchange of the change sets is performed, but what about the original problem of keeping the releases under version control?

The command hg grfreeze changes the content of the .hgguestrepo file to reflect the current change sets of the guest repositories.

Following the procedure suggested by the creators of the extension, the release is performed branching to the Release branch, and the performing a freeze and commit in the new branch.

Scenario

The history in the Application1 repository shows how the .hgguestrepo file is changed in the Release branch:

Scenario

In my experience, though, this is not practical because it introduced a new head in the repository, working with multiple heads is difficult, sooner or later one of the developers commits changes to the wrong branch.

The way I prefer to work using inactive bookmarks (as the output of the branch command suggests....)

The procedure I use is:

  • perform a hg grfreeze to write the changeset references in the .hgguestrepo file,
  • commit the change
  • create a bookmark (inactive)

Bookmarks Scenario

The release change-set is bookmarked for easy access in case the build has to be reproduced:

Scenario Bookmark

Either branching or using the bookmarks, only the application repositories are aware of the various releases.
Each release change-set contains the .hgguestrepo file with the hash reference of the platform change-set used for the release, using the bookmark it is straightforward to update the repository and its guests to a previous state .

Scenario

To conclude, the NetBeans Module System API and Mercurial guestrepos extension provide, together, everything that is needed to keep a modular application under configuration control at source code level.

NetBeans allows to define for each module what are the modules, and in which version, it depends on.

Mercurial, together with the guestrepos extension, allows to keep all the source code under version control allowing to easily to update the state of the application (main and guest repositories) to a specific version.

Published at DZone with permission of its author, Angelo D' Agnano.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)