Grid Computing in Compact Muon Solenoid
Experiment at CERN
Dr. Nitish Dhingra
Postgraduate Department of Physics
G.H.G. Khalsa College
Ludhiana
Workshop on High Performance Computing – 2015
16-17 March, 2015
1
WOHPC 2015, Panjab University, Chandigarh
Outline
 Introduction – Large Hadron Collider (LHC)
 CMS experiment
 Need of Distributed Computing
 Worldwide LHC Computing Grid (WLCG)
 CMS Remote Analysis Builder (CRAB)
 Summary/Overview
2
WOHPC 2015, Panjab University, Chandigarh
Introduction – Large Hadron Collider
 The LHC machine located at CERN, Geneva is the most powerful accelerator-cumcollider in the world
 Designed to collide two counter-rotating proton beams at center-of-mass energy of
14 TeV (13 TeV for upcoming run) in an underground (~100 m) circular ring having
27 Km circumference
 Each beam consists of bunches of protons (2808 bunches at full intensity)
 ~1011 protons per bunch
 Four main experiments at the LHC
 Meant to record interesting physics events occuring during p-p collisions
3
WOHPC 2015, Panjab University, Chandigarh
Compact Muon Solenoid (CMS)
 One of the major experiments at the LHC, represents a global collaboration of
about 4000 people from 182 institutes across 42 countries
 Aims mainly to study the physics of p-p collisions
 Capable of detecting & measuring energies of particles produced in collisions
 Confirmed the existence of a Standard Model Higgs-like boson in 2012 along with
ATLAS experiment
 Physics Nobel Prize (2013) for theoreticians - François Englert & Peter Higgs
4
WOHPC 2015, Panjab University, Chandigarh
Need of Distributed Computing
 LHC produces around 600 million collisions per second!
 Particles produced in each collision event often decay in complex ways into
even more particles
 Electronic circuits record the passage of each particle through the detector as
a series of electronic signals
 Electronic data is sent to CERN Data Centre (Tier 0) for digital “reconstruction”
 Digitized summary is recorded as a "collision event”
 In order to find interesting physics in recorded events, physicists must sift
through the 30 petabytes (1 PB = 1015 B) or so of data produced annually!
 Impossible for CERN to crunch all of
the data on-site
 Moreover, community of physicists
is spread all around the globe
 Use Worldwide LHC Computing
Grid (WLCG) to share the burden with
computer centers all around the world
5
CERN T0
WOHPC 2015, Panjab University, Chandigarh
Worldwide LHC Computing Grid (WLCG)
 WLCG is based on the technology of World Wide Web (invented at CERN – 1989)
 Composed of four levels called Tiers – 0, 1, 2, 3
 Each Tier is made up of several computer centers & provides a specific set of
services
 Tier 0 → CERN Data Center
 Raw data from detectors is passed to Tier 0
 First pass reconstruction of raw data into meaningful information
 Raw data & reconstructed data is passed to next level i.e. Tier 1s
 Tier 1 → 13 computer centers around the globe (10 Gbps optical link from CERN)
 Canada, Germany, Spain, France, Italy, Netherlands, Republic of Korea, Taipei,
UK, USA, Nordic countries
 Provide 24x7 support for the grid
 Responsible for storing a proportional share of raw & reconstructed data
 Perform large scale reprocessing & store corresponding output
 Store a share of Monte Carlo simulated data
 Reconstructed data is passed to next level i.e. Tier 2s
6
WOHPC 2015, Panjab University, Chandigarh
WLCG (cont.)
Tier 2 → Universities & Scientific institutes (155 Tier 2s including TIFR-Mumbai)
 Ability to store sufficient data & provide adequate computing power for
analysis tasks
 Handle a proportional share of production & reconstruction of simulated
events
 Tier 3 → Local cluster (Panjab University’s Tier 3)
 Grid can be accessed through Tier 3
7
WOHPC 2015, Panjab University, Chandigarh
Some Other Grid Networks
A number of grid networks have been developed across the globe
 EGI→ European Grid Infrastructure
 OSG→ Open Science Grid (US initiative)
 EU-IndiaGrid → Collaborative project between EGI & Indian-Garuda Grid
Links several thousand researchers to the distributed computing resources around the world
Scientific applications of grid computing are boundless
 High Energy Physics
 Nuclear Physics
EGI: http://www.egi.eu/
 Nanoscience
OSG: http://www.opensciencegrid.org/
 Structural Biology
EU-IndiaGrid: http://www.euindiagrid.eu/
 Cancer treatment
Garuda: http://www.garudaindia.in/index.aspx
 Solar energy research
 Community VO (multiple sciences)
World Community Grid (http://www.worldcommunitygrid.org/)
Enables anyone with a computer, smartphone or tablet to donate their unused computing
power to advance cutting-edge scientific research on topics related to health, poverty &
sustainability
With the contributions of over 650,000 individuals & 460 organizations, supported 24
research projects till date including searches for more effective treatments for cancer,
HIV/AIDS & neglected tropical diseases
8
WOHPC 2015, Panjab University, Chandigarh
CRAB – CMS Remote Analysis Builder
 CRAB serves as a tool to communicate to WLCG
 CRAB is a Python program intended to simplify the process of creation and
submission of analysis jobs at CMS experiment in a grid manner
 Submission of jobs to grid with CRAB requires
 An account on User interface (any machine with WLCG commands installed)
 A personal grid certificate issued by an appropriate Certification Authority (CA)
 Personal grid certificate registered in CMS Virtual Organization (CMSVO)
 Grid certificate is like an electronic ATM/Debit card
 Can be issued by any of the trusted CA recognized
by WLCG
 Is an identity for authorization to grid access
 Like ATM pin, don’t disclose your grid password!
 Once these steps are completed, one can setup the CMS
software environment and test grid proxy generation
9
WOHPC 2015, Panjab University, Chandigarh
CRAB Workflow
 RB = Resource Broker
 DBS (DAS)= Data Bookkeeping Service (Data
Aggregation System)
 SE = Storage Element
 DLS = Data Location Service
 CRAB is configured with a config. file
 Specify:
Dataset you want to analyze
Python file to run usercode over dataset
Number of events to analyze
Output file name
Location to return output
10
WOHPC 2015, Panjab University, Chandigarh
CRAB Workflow (cont.)
 Process of analyzing MC/data via CRAB involves a number of steps
 Grid Workload Management System (WMS or Resource Broker) optimizes the
process of creation & submission of jobs
 Step 1: Creation of jobs
 Contact DBS/DAS Services
 Check the available sites hosting
requested dataset
 Find the possible sites to run jobs
 Step 2: Submission of jobs
 WMS/RB decides where to submit the jobs
according to the resource availability metrices
 Step 3: Check the status of jobs
 Step 3a: Wait, if running
 Step 3b: If completed, retrieve the output
 Output is generally a ROOT file with tree(s)/histograms
containing event information
11
WOHPC 2015, Panjab University, Chandigarh
Step 1: Jobs Creation
Available Sites
hosting the
requested dataset:
Fermilab (USA)
12
Requested Dataset
to be analyzed
WOHPC 2015, Panjab University, Chandigarh
Contacting DAS
(DBS) for the
requested dataset
Step 2: Jobs Submission & Online Tracking
Connection with
Remote Host:
Nebraska-Lincoln
(USA)
13
Submission of
jobs
WOHPC 2015, Panjab University, Chandigarh
Step 3: Command Line-Status Check & Retrieval
Jobs Running at
Fermilab
computing
resources
Zero exit code →
jobs completed
successfully
Retrieving the
results
14
WOHPC 2015, Panjab University, Chandigarh
Fruit of Hardwork
15
WOHPC 2015, Panjab University, Chandigarh
Overview
 Distributed computing is the need of the hour!
 WLCG is the most sophisticated data-taking & analysis system ever built for
science, providing nearly real-time access to LHC data
 Represents a global collaboration involving 40 countries & 170 computing
centres around the globe with 2 million jobs running every day
 The world wide web & WLCG are the key components to make LHC
project a grand success!
 WLCG computing enabled physicists to discover Standard Model Higgs-like
boson on 4 July, 2012 and hence the Physics Nobel Prize for 2013
 Grid computing applications are not restricted only to HEP!
If needed, grab the opportunity of distributed computing for your project!
16
WOHPC 2015, Panjab University, Chandigarh
17
WOHPC 2015, Panjab University, Chandigarh