Slides

Co-ordination & Harmonisation of
Advanced e-Infrastructures
for Research and Education Data Sharing
Research Infrastructures
Grant Agreement n. 306819
Heterogeneous CE Configuration
Christos Kanellopoulos, GRNET
AA ROC All-Hands Meeting, 25 March 2013
Heterogeneous CE Configuration
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
2
Heterogeneous CE Configuration
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
3
Heterogeneous CE Configuration
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
4
Heterogeneous CE Configuration
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
5
Heterogeneous CE Configuration
6
Heterogeneous CE Configuration
7
Heterogeneous CE Configuration
After some time:

3 Generations of Hardware

Resources allocated to specific VOs or Users

Resources with extra features (GPGPUs, High Mem etc)
8
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Heterogeneous CE Configuration
CREAM-CE & the gLite Cluster


glite-Cluster publishes information about cluster and subclusters in a site that can
be referenced by any number of compute elements

Allows to deal with sites with multiple CREAM CE nodes and/or multiple subclusters
(i.e. disjoint sets of worker nodes, each set having sufficiently homogeneous
properties)

However the gLite WMS requires the publication of exactly one SubCluster per
Cluster (and hence per batch queue).

Sites with heterogenous hardware can:

9

Publish a SubCluster with a representative/minimum hardware description (e.g.
the minimum memory on any node)

define separate batch queues for each hardware configuration, e.g. low/high
memory queues, and attach the corresponding GlueCE objects to separate
Cluster/SubCluster pairs
Without the use of the gLite-cluster, YAIM allows configuring a single Cluster per
CREAM head node
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Heterogeneous CE Configuration


Installation of a CREAM CE node in cluster mode

yum install yum-protectbase.noarch

yum install ca-policy-egi-core

yum install xml-commons-apis

yum install emi-cream-ce

yum install emi-torque-server

yum install emi-torque-utils
Installation of a glite-cluster nore

yum install yum-protectbase.noarch

yum install ca-policy-egi-core

yum install emi-cluster
10
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Heterogeneous CE Configuration

Configuration of a CREAM CE node in cluster mode

Install host certificate

Configure the siteinfo.def file
11

Variables which are required in cluster mode are described at
https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#cream_CE

When the CREAM CE is configured in cluster mode it will stop publishing information about
clusters and subclusters

That information should be published by the glite-CLUSTER node type instead

A specific set of yaim variables has been defined for configuring the information which is still
required by the CREAM CE in cluster mode

Be sure that CREAM_CLUSTER_MODE is set to yes
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Heterogeneous CE Configuration

Configuration of a CREAM CE in cluster mode (with glite-CLUSTER
deployed on a different node) using Torque as batch system, with
the CREAM CE being also Torque server


Configuration of a CREAM CE in cluster mode (with glite-CLUSTER
deployed on the same node of the CREAM-CE) using Torque as
batch system, with the CREAM CE being also Torque server


/opt/glite/yaim/bin/yaim -c -s <site-info.def> -n creamCE -n TORQUE_server -n
TORQUE_utils
/opt/glite/yaim/bin/yaim -c -s <site-info.def> -n creamCE -n TORQUE_server -n
TORQUE_utils -n glite-CLUSTER
Configuration of a CREAM CE in cluster mode (with glite-CLUSTER
deployed on the same node of the CREAM-CE)) using Torque as
batch system, with the CREAM CE NOT being also Torque server

12
/opt/glite/yaim/bin/yaim -c -s <site-info.def> -n creamCE -n TORQUE_utils -n
glite-CLUSTER
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Heterogeneous CE Configuration

If a CREAM CE is configured in cluster mode:

The resource BDII running on the CREAM CE publishes just the following
objectclasses:

ComputingEndpoint



EndPoint for CEMon

"Child" of ComputingService

Published only if CEMon is deployed
All the other objectclasses are published by the resource BDII running on the
gLite-CLUSTER node

ComputingService

ComputingManager

ComputingSharw

MappingPolicy for ComputingShare

ExecutionEnvironment

Benchmark

ToStorageService

ApplicationEnvironment

EndPoint for RTEPublihser

13
AccessPolicy for ComputingEndpoint
"Child" of ComputingService
Heterogeneous CE Configuration

Another approach could be the usage of Tags and reservations

At the Torque level apply tags to each node that is part of the
cluster e.g. (gen1, gen2, gen3, app1, app2, gpgu, highmem)

14
In /var/torque/server_priv/nodes:
wn001.localdomain np=4
wn002.localdomain np=4
wn003.localdomain np=4
wn004.localdomain np=4
wn005.localdomain np=4
wn006.localdomain np=4
wn007.localdomain np=8
wn008.localdomain np=8
wn009.localdomain np=48
wn010.localdomain np=48
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
lcgpro
gen1
gen1
gen1
gen2
gen2
gen2
gen2
gen3
gen3
gen3
highmem
highmem
gpgpu
app1
app1 app2
Heterogeneous CE Configuration

At the maui level use the tags to make node reservations:

15
In /var/spool/maui/maui.cfg:
SRCFG[app1]
SRCFG[app1]
SRCFG[app1]
PERIOD=INFINITY
USERLIST=user01,user02,user03
HOSTLIST=wn1[4-5],wn19,wn20,wn2[4-5]
SRCFG[gpgpu]
SRCFG[gpgpu]
SRCFG[gpgpu]
PERIOD=INFINITY
USERLIST=user01,user02,user03
HOSTLIST=wn10,, wn49, wn50
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria
Further Information

https://twiki.cern.ch/twiki/bin/view/LCG/Siteinfo_configuration_variables#CLUSTER

https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdminist
ratorGuideForEMI2#1_3_3_2_Installation_of_the_CREA

https://wiki.italiangrid.it/twiki/bin/view/CREAM/CreamGlue2#2_
2_CREAM_CE_in_cluster_mode
16
Christos Kanellopoulos | [email protected] | AA Roc All Hands Meeting 03/2013 | Pretoria