Remote Code
Management
Since late 2004, the source
code management of the
HIRLAM system is done
remotely.
How was this done? What are
the possibilities and
pitfalls?
In the beginning, there was COS
And all was well, because Cray had “nupdate”, a local source code
management application.
HIRLAM, like most software maintained at ECMWF, was contained in
“nupdate” program libraries.
However, ECMWF decided to switch from Cray to Fujitsu in 1995.
What to do ?
On the Fujitsu, we were free ...
... to choose our own source code management system.
Like any good Unix-like system at that time it had SCCS (Source
Code Control System).
However, because I suspected that we would have to change
platforms again in the future, I argued for a free alternative for
SCCS, namely RCS (Revision Control System).
During the All Staff Meeting in Madrid in 1995, the HIRLAM
community accepted this choice.
Nevertheless, we kept the model of “one system manager, who has
full control”. RCS didn't allow for any distributed management.
Remote Documentation Site
Between 1995 and 2004 the System Manager for the HIRLAM project
(Gerard Cats) developed a documentation web site at KNMI.
So after 1995 we were familiar with the fact that documentation of
the HIRLAM project, the releases and the release notes were not
some files that were sent along with the source code, but on
actively maintained web pages.
One of his students developed a Fortran code → .html tool that
allowed the source to be viewable on this web site.
Remote Source Code Repository
From having the documentation of the source code online, it wasn't
a large step to put the source code repository online itself.
As I had been working with the GNU Compiler Collection (GCC) in
this way since the revolution of 1997 (“egcs”) I knew it could be
done with CVS (Concurrent Versions System).
I asked my contacts at Cygnus/Red Hat to spell out to the group
(Gerard Cats, Tomas Wilhelmsson, Jacob Weismann Poulsen) how
to set up this remote repository and the access to it.
This went “on the air” in late 2004.
Switch to Subversion (SVN)
In October 2005, after extensive preparations led by Daniel Berlin,
the GCC repository switched from using CVS to Subversion (SVN).
Subversion was sold as (and indeed was) a CVS, but then done
right.
It didn't have any large new features over CVS, but it was far easier
to work with (especially on the server side).
This prompted a discussion in the HIRLAM system group whether we
shouldn't switch too ...
... which we did (work mostly done by Tomas Wilhelmsson);
completed in early 2006.
How does it work – user side
To get the source code, you use the following command:
toon@super:~/tmp$ svn co https://[email protected]/trunk/harmonie
Authentication realm: <https://svn.hirlam.org:443> Hirlam Subversion
repository
Password for 'toon':
A harmonie/msms
A harmonie/msms/Html.pm
...
A harmonie/util/oulan/oulan_yomfthermo.h
A harmonie/util/oulan/ext_lam_tovshirs.F
U harmonie
Checked out revision 8470.
toon@super:~/tmp$ cd harmonie/
toon@super:~/tmp/harmonie$ svn
up
At revision 8470.
How does it work – server side
We chose to use svn-over-http(s) approach.
It has the following advantages:
1. Uses HTTP protocol, standard ports (80 and 443); therefore
it works through most Institute's firewalls.
2. Runs the server under standard HTTP server account.
3. When using Apache as your HTTP server, you need version 2.0.
(can be run alongside a 1.3 setup – however, on a non-standard
port).
How does it work – access policy
All machines in the domains of the HIRLAM consortium's member
states have default access to the repository at hirlam.org – for
reading.
Write access to the repository is only granted to a few persons
specifically configured on the server to have write access. The
access is granted by username/password combination (as shown
earlier). This is not restricted to the user's machine being in the
aforementioned domains.
Example check-out command of the repository by someone who isn't
interested in updating the code (note: no user name):
svn co https://svn.hirlam.org/branches/harmonie-36h1.1
How does it work – update policy
The HIRLAM group decided to go for a “System” group – a small
number of people who would be responsible for maintaining the
repository.
This group is headed by the Project Leader for System Management.
All scientists who work on improving the HIRLAM system deliver
their updates (which can be tested independently of the contents
of the repository by using the script system around the Fortran
code) to their “nearest” System group person, who will commit
them to the repository.
Of course, it must be shown that the change was adequately tested.
How does it work – update policy
Outside the System Group, write access will be granted to persons
who prepare large changes on development branches.
One famous example of the past few years in HIRLAM was the
'newsnow' branch – for implementing a new snow
parametrization – it existed for several years.
The 'maintainer' of a branch is responsible for keeping it up to date
with the other developments on the 'trunk'.
'Phasing in' of this development then consists of one final merge of
trunk into branch and then to merge the result back into trunk.
Phasing phased out
In this way, a massive phasing effort is not needed: the differences
between a new development (on a branch) and the main
development (on the trunk) are kept to a minimum by regular
merging of the trunk into the branch.
Because the machine the source code repository resides on is
reachable from all HIRLAM member states, there is no need for
extended phasing visits.