The World’s Southernmost Grid Site for the Compact Muon Solenoid Experiment
Dr Philip Allfrey, Department of Physics, University of Auckland
Yuriy Halytskyy, Centre for eResearch, University of Auckland
The CMS Experiment
The Compact Muon Solenoid Experiment (CMS) [2] is one of
four high-energy physics experiments at the Large Hadron
Collider, a particle accelerator located at CERN, Geneva.
When operating at design values the experiment expects to
produce petabytes (PB) of data per year. The computing and
data storage requirements of the LHC experiments, which are
carried out by large international collaborations, were early
drivers for Grid computing.
We have established a Tier-3 Grid site supporting the CMS experiment at the University of Auckland. This site uses the gLite
middleware[1] developed by the European Grid Initiative (EGI, formerly EGEE). The Centre for eResearch provided Xen virtual
machines to run the middleware, and integrated the worker node into their computing cluster. All components worked “out of
the box”, after site-specific configuration. The site has been certified by EGI’s Asia-Pacific Regional Operations Centre.
Workflow
The CREAM Computing Element accepts jobs submitted by
members of the CMS Virtual Organisation (as determined by
VOMS extensions to X509 Grid certificates) and adds them to
the glite queue on the Centre for eResearch Cluster’s Torque
batch system, where they are executed on a dedicated eightcore Worker Node.
CMS uses a four-tier Grid infrastructure:
•1 Tier-0 site (CERN) which stores one copy of the entire data
set and does the first pass of processing
•7 Tier-1 sites with ~5000 CPU cores and ~10 PB storage,
which store 1/7 of the data set and perform second pass
reconstruction
•~50 Tier-2 sites with ~500-1000 CPU cores and at least 200
TB storage, where most of the user analysis jobs are run and
their output stored
•~50 Tier-3 sites, with no minimum requirements. Usually
hosted at smaller universities, mainly used for user analysis
Jobs may stage in/out files to the Storage Element, which
currently has 1 TB of SAN storage managed by the Disk Pool
Manager application. All files are owned by a system user; a
virtual directory structure and file permissions are managed
via the Disk Pool Namespace Server, which has a MySQL
backend.
Figure 2: The architecture of the gLite-based Tier-3 Grid
site established at the University of Auckland
CMS components
The CMS software is installed on an NFS-mounted partition on
the worker node. Versions are centrally installed/ removed by
jobs submitted by a user with a privileged VOMS Role.
The CMS data stored at Tier-1 or Tier-2 sites is transferred to
the storage element by the PhEDEx application, which uses
the srmv2.2 protocol.
To reduce the load on the central database at CERN which
contains the calibration constants and detector geometry for
the CMS experiment, a Squid proxy server is used by all CMS
sites.
Figure 1: Map showing locations of Grid sites supporting the
CMS Virtual Organisation (VO) [4]
In addition to the EGI monitoring, CMS-specific functionality is
monitored by regularly submitted jobs and remote queries to
the applications, and visualised on the CMS dashboard.
The Storage and Computing Elements report their status to
the site BDII (Berkeley Database Information Index), an LDAP
application, which in turn reports the site’s status to a toplevel BDII hosted by the nearest large Grid site; this is used
for service discovery.
Once a day the APEL processor parses the Torque server logs
to determine per-user and per-VO usage of the site’s
resources which it then publishes via ActiveMQ to a global
registry.
All EGI sites are centrally monitored by Nagios probes:[3]
some tests are run in a job submitted to the site, others
query the BDII or storage element directly. The current status
of the site is visualised on the GStat website.[4]
References
[1] See http://glite.web.cern.ch/glite/ for details of all gLite components
[2] http://cms.cern.ch
[3] https://sam-ap-roc.cern.ch/nagios
[4] http://gstat-prod.cern.ch