Doc. Identifier:
Note
DataGrid-xx-NOT-nnnn
Date: 28/07/2017
Subject: WP1-WP4 INTERACTIONS
Author: Massimo Sgaravatto ([email protected])
Partner: INFN Padova
Diffusion: Internal
Information:
1. OVERVIEW
WP 1 is implementing a resource broker, responsible to choose the resources where to submit the jobs
submitted by users.
We propose to consider as resource for the PM 9 release (since it has been agreed that for the the PM
9 release we will rely upon the Globus toolkit):
 a queue of an underlying local resource management system (e.g. a LSF queue, a PBS queue,
…). We assume that a queue represents a set of “homogeneous” resources, that is when a job
is submitted to a specific queue, it doesn’t matter in which node of this queue the job is
dispatched (since all the nodes of this queue has direct access to the same storage elements,
the required run time environments are installed on all nodes of the queue, etc…)
 a “single” node, that is a host that doesn’t rely upon a local resource management systems, and
therefore uses the fork system call as job manager
We assume that one or more storage elements are “attached” to the resource, that is these storage
elements can be directly accessed using a local protocol (e.g. NFS, AFS, etc…). We also assume that
these storage elements have the same authorization policies (i.e. the same grid-mapfile in the Globus
based model) of their corresponding resources.
For the PM 9 release the resource broker will dispatch a job on a resource “attached” to a storage
element where the required input data are stored. The output data produced by the job will be
“written” to a storage element “attached” to the executing resource (in case the output data set will
then be moved to an other storage element, using the services provided by WP 2).
We also assume that a local disk area is associated to each resource (possibly to each node served to
the same queue): this local disk is used by the job during its execution, and the disk space is released
once the job has been completed. The size of the minimum local disk space available within a resource
will be advertised along with the other resource characteristics (see below).
For this broker functionality, information on these resources is needed. We assume that all this
information will be published in one Grid Information Service, and we assume that the new release of
the Globus Grid Information Service, also known as MDS-2, will be used.
In section 3 a proposal for a schema for the Information Service (that is a set of attributes on
characteristics and status on resources, needed by the resource broker) is proposed. Specific
information providers (entities able to provide the Information Service with this information) must
therefore be implemented.
IST-2000-25182
INTERNAL
1/5
Doc. Identifier:
Note
DataGrid-xx-NOT-nnnn
Date: 28/07/2017
2. MDS-2
WP 1 is planning to use the Globus MDS-2 as backbone of the PM 9 information infrastructure, since
we think that we can’t rely on the current implementation of the Globus GIS (MDS), not even for a
prototype system.
Besides improvements in the performance, MDS-2 should include the possibility to extend the schema
in a user-defined way, and include associated plugin information producers.
WP 1 is going to evaluate Globus MDS-2, and in particular is going to:

Propose a schema, for what concerning the information about characteristics and status of
resources), since WP 1 is the major consumer of this information (see a first proposal in
section 3)

Implement the Index plugin module for MDS-2, which will be used as first filter in the
resource discovery process (to find a first set of suitable resources where to dispatch the
submitted job)
WP 1 is expecting that WP 4 will implement the required information producers, according to an
agreed schema (see section 3 for a proposal), considering the set of supported local resource
management systems (see section 4).
3. PROPOSAL FOR A SCHEMA
Having defined in section 1 what do we mean as resource, here is the information on resources that we
think should be published in the Grid Information Service:


ResourceId
A univocal identifier for the resource
GlobusResourceContactString
The Globus Resource Manager contact string (e.g. lxde01.pd.infn.it:2119/jobmanager-lsf),
which identifies in the Globus environment the resource

QueueName
The name of the queue (applicable for those LRMS’s that use queues)

ResourceManagementType
Defines the type of resource management system (e.g. LSF, PBS, Condor, …)

ResourceManagementVersion
The version of local resource management system

GRAMVersion
IST-2000-25182
INTERNAL
2/5
Doc. Identifier:
Note
DataGrid-xx-NOT-nnnn
Date: 28/07/2017
The GRAM version

Architecture
The architecture of this resource (it the resource is a queue, the architecture of the machines
associated to this queue)

OpSys
The operating system of this resource (if the resource is a queue, the operating system of the
machines associated to this queue)

PhysicalMemory
Minimum available physical memory for this resource (e.g. if the resource is a queue, the
minimum physical memory for the machines associated to this queue)

LocalDisk
Local disk footprint associated to the resource, available for a job submitted to this resource

TotalCPUs
Number of total CPUs associated to this resource

FreeCPUs
Number of free processors associated to this resource (that is processors able to run, in that
moment, jobs submitted to this resource)

NumSMPs
Number of SMP processors associated to this resource

TotalJobs
Total number of jobs submitted to this resource (running and idle jobs)

RunningJobs
Number of running jobs submitted to this resource

IdleJobs
Number of idle jobs submitted to this resource

MaxTotalJobs
The maximum number of jobs (running and idle) allowed for this resource

MaxRunningJobs
The maximum number of running jobs allowed for this resource

ExpectedStartTime
An estimation of the start time for a job submitted to this resource in that moment

Status
The status of the resource (for example, for a queue, if it is active, that is if it ready to dispatch
jobs to the executing machines)

RunWindows
The time windows that define when the queue is active

Priority
The priority associated to this resource

MaximumCPUtime
IST-2000-25182
INTERNAL
3/5
Doc. Identifier:
Note
DataGrid-xx-NOT-nnnn
Date: 28/07/2017
The maximum CPU time allowed for jobs submitted to this resource

MaximumWallClockTime
The maximum wall clock time allowed for jobs submitted to this resource

ComputingPower
The computing power of this resource.
Which format ??

AuthorizationPolicies
For the PM 9 release we propose to consider the Globus grid-mapfile as authorization policy

RunTimeEnvironments
The application/run time environments installed on this resource (e.g. Objectivity version X,
LHC++ version Y, CMS environment version Z, …)

StorageElements
The identifiers of the storage elements “attached” to this resource
For what concerning the storage elements, we think that the following information should be
published:

StorageElementId

An identifier for this storage element

URIPrefix
The URI prefix that identifies all the physical files stored on this storage element.
In fact the resource broker will find in which storage element(s) the required input data for a
job are stored, querying the ReplicaCatalog (provided by WP 2): given a logical file name, the
ReplicaCatalog provides the corresponding physical file name(s), expressed using URIs, with
the prefix of these URIs identifying the storage elements where these files are stored

AvailableSpace
The available space available on this storage element to store the output data produced by jobs
4. LRMS
Who recommends what Local Resource Management System (LRMS) to use ?
Who supports their integration with Globus ?
Saying that a LRMS is integrated with Globus, we mean that:
1. This LRMS can be configured as Globus resource, that is it is possible to submit Globus jobs
to this LRMS
2. The LRMS (via the so-called GRAM reporter) provides the MDS (aka GIS, Grid Information
Service), with information on characteristics and status of the resources managed by the
LRMS
IST-2000-25182
INTERNAL
4/5
Doc. Identifier:
Note
DataGrid-xx-NOT-nnnn
Date: 28/07/2017
Concerning the first bullet, Globus already supports different LRMS, which include for example LSF,
Condor and PBS, and which not include for example BQS (used in Lyon).
We (WP 1) have already tested the submission to Globus resources to:

Hosts that don’t rely upon a LRMS (that is, using just the fork system call as job manager)

LSF farms

Condor pools

PBS farms
We also tested the job submission via Condor-G (a job submission service built on top of the Globus
GRAM service, that WP 1 is planning to use in its first prototype) to these LRMS’s (actually for PBS
only preliminary tests have been performed, but we don’t expect major problems).
WP 1 is not planning at the moment to “implement” the integration of other LRMS’s with Globus.
For the PM 9 release, WP 1 plan to rely on the current implementation of the Globus GRAM service.
The Condor team is augmenting the GRAM service, in order to support a two phase commit
submission protocol and a persistent job manager. WP 1 will test this new release (that should be back
compatible): if it suitable for our needs and it works without problems, WP 1 will suggest to WP 6 to
deploy this new GRAM release.
Concerning the second bullet, the current implementation of the GRAM reporter is made up of shell
scripts that are used to publish some information about characteristics and status of local resources,
according to a well-defined schema.
We have already evaluated the current (Globus 1.1.3) implementation of the GRAM reporter
(considering as Globus resources machines managed by the fork system call, LSF farms and Condor
pools), and we think that the default schema is not fine for our needs, but, as reported in section 2, we
plan to rely upon MDS-2, which should feature plug-in additional schemas and associated information
producers.
Since we assume that it is up to WP 4 to implement these information providers, we think that WP 4
will have to propose (to WP 6 ?) the set of LRMS’s that should be supported.
Note

What happens if a job “disappears” ” without any exit job status (for example because of a crash
of the executing machine) ? Can we assume that we are relying upon a LRMS able to manage
these events? For example Condor provides this functionality: if a job “disappears” without any
exit status, it is resubmitted. LSF provides this functionality as well (called “Automatic Job
Rerun”). What about PBS and the other local resource management systems that we have to
support? Otherwise, if the underlying LRMS is not able to resubmit the job, Globus assumes that
the job has been completed (it is not able to understand about the problem). Neither Condor-G is
able to manage these kinds of problems.
IST-2000-25182
INTERNAL
5/5