Software Infrastructure
Supplement for the OSG
Proposal
OSG Document #1029
Table of Contents
Software Infrastructure Supplement for the OSG Proposal ................................................... 1
1.
2.
Scope and Contents of the VDT (OSG Software Distribution) ....................................... 1
1.1.
1.2.
Functional Software Subsets .............................................................................................................1
Full set of software components ......................................................................................................3
Process for updating the VDT .................................................................................................... 4
2.1.
2.2.
Inputs to the Software Release Process ........................................................................................4
Software Release Process ...................................................................................................................4
1. Scope and Contents of the VDT (OSG Software
Distribution)
The OSG distributes software via the VDT (Virtual Data Toolkit). As described in the
main text of the proposal, the scope of the VDT is “everything in the OSG fabric of
services”. To better help readers understand exactly what that entails, we provide two
lists. First is a list of functional subsets of software that users may install. Second is a
roughly categorized list of everything distributed as part of the VDT as of March 2011.
1.1. Functional Software Subsets
Compute Element: Set of software to provide job submission gateway to a site. There are
two types of compute elements: those based on Globus GRAM, and those based on
CREAM. The latter is work in progress.
Storage Element: Set of software to provide access to a site’s storage. There are several
types of storage elements depending on the exact software used, which includes Bestman,
dCache, XRootd, and HadoopFS.
Site Authorization Service: Using the GUMS software, provides centralized authorization
service that supports other services such as the compute and storage elements.
Client: Software used to access the other services, such as the compute and storage
elements.
Worker Node: Software provided on each computer (worker node) available for running
jobs behind the compute element.
Virtual Organization Membership Management: Software (VOMS, VOMS-Admin…)
used to manage the membership and roles within a virtual organization.
1
Accounting Service: Based on Gratia, allows a site or virtual organization to collect
accounting information about jobs and data.
Monitoring (RSV): Allows a site or virtual organization to monitor a site or set of sets to
validate that the fabric of services is running correctly.
Each of these subsets brings along the complete set of software in order to run the
service. For example, the virtual organization membership management software subset
brings along:









VOMS
VOMS Admin (web application)
Tomcat (web application hosting environment)
Apache (web server)
MySQL (database)
MySQL JDBC (library to connect VOMS Admin to MySQL)
Globus GSI (security infrastructure)
Utilities to manage Certificate Authority certificates and revocation lists
Utilities to manage configuration and update softwware
2
1.2. Full set of software components
Please note that many of these tools in turn rely on other software components.
Job & Workflow Management
Condor/Condor-G/Condor-Cron
CREAM (Pending)
GlideinWMS (Pending)
Globus GRAM
Globus GRAM 5 (Pending)
OSG Matchmaker
Pegasus
Data Transfer/Storage Management
Bestman
Curl
Dccp
dCache
EDG GridFTP Client
FTS Client
Globus GridFTP 4
Globus GridFTP 5 (Pending)
Globus RLS
Hadoop File System
LCG Info
LCG Infosites
LCG Utils
LFC
SRM Client utilities
Squid
UberFTP
Wget
Xrootd
Security Tools
Certificate Authority management tools
Cert. Revocation List management tool
Certificate management tools
Glexec
GSIOpenSSH
GUMS
LCAS
LCMAPS
MyProxy
PRIMA
VOMS
VOMS Admin
VOMRS
Support Services
Apache
EDG Make Gridmap
MySQL
Tomcat
Information Management & Accounting
Generic Information Provider
Gratia Accounting Service
Gratia Probes
Monalisa
OSG Discovery Tools
OSG RSV
Networking
Perfsonar Clients:
BWCTL
NDT
NPAD
Nmap
OWamp
VDT Configuration
Configuration for individual services
OSG-specific configuration management
VDT installation & update tools
Utilities (Support above, not user-visible)
Bouncy Castle
CEMon-Server
CGSI-gSOAP
Chrpath
Expat
GPT
Misc. Perl Modules
MyODBC
OpenLDAP client
PyGlobus
TclGlobus
unixODBC
3
2. Process for updating the VDT
The following diagram illustrates the process for updating the VDT. Updates can be of
any sort, including adding a new software tool to the VDT and updating an existing
software tool.
2.1. Inputs to the Software Release Process
The process is controlled by inputs from four sources:
1) The software obtained from the software providers. Usually the providers are outside
of OSG, but occasionally they may be from within OSG.
2) The requirements from the OSG stakeholders and/or other OSG areas. OSG areas
provide requirements because some software is needed as part of the OSG-provided
services or because they have additional operational or security constraints. Requirements
include the precise list of what is needed, the time frame in which it is needed, and any
constraints on the configuration that needs to be supplied.
3) The priorities from the OSG management. When the Software Infrastructure team is
unable to meet all stakeholder needs, how are the various software tasks prioritized? (See
the OSG Project Management Process supplement document.)
4) The evaluations done by the Technology Architecture and Investigation Area provide
background knowledge needed to add the software. It may also affect the requirements
and priorities based on our best understanding of what can be done with the software.
(See the main text of the proposal for more detail about the Technology Architecture and
Investigation Area.)
2.2. Software Release Process
Broadly speaking, the process is divided into four steps: (See Figure 1, below)
1) Integrate the software. This includes any evaluation of the software that is still needed
(the Technology Investigation Area primarily evaluates new software and not ongoing
updates), building the software, testing it, providing configuration for the software,
documenting it, and packaging it for distribution.
2) We have a small wide-area testbed called the Validation Testbed where a few
dedicated people can quickly do “smoke” tests to ensure that the software is ready to be
tested by a larger audience. This prevents some large problems from wasting the time of a
large number of testers.
3) We have a medium-size wide-area testbed called the Integration Testbed where a
larger set of people do deeper testing of the software, including in some cases the running
of real stakeholder workflows to ensure the software works correctly. In some cases
where the update is primarily targeted at a single user community, that community may
do deeper testing instead of the Integration Testbed.
4) We then release the software into production and provide ongoing support for it.
4
This process is largely the same whether we are adding new software to the VDT,
providing a regular update of the software, or providing an urgent update to the software.
The complexity and urgency of the update affect the time that this process takes. For
urgent security updates it may take as little as two days (with some shortcuts made in
order to deliver the security update quickly), while major new software additions may
take weeks or even months and several iterations through the process.
Figure 1
Authors:
Alain Roy University of Wisconsin-Madison
Revisions:
4 3/3/11 AJR Formatted as OSG document
3 3/2/11 AJR Clarified text, added functional subsets
5