International Workshop on Large Scale of Computing
February 8-10, 2006
Variable Energy Cyclotron Centre
1/AF, Bidhan Nagar
Kolkata – 700 064, India
Tier-2 in India for Alice
Susanta Kumar Pal, VECC, Kolkata
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
1
What is GRID computing?
Computational Grid is a collection of distributed, heterogeneous resources
which can be used as an ensemble to execute large-scale applications. The
emphasis is on :
 Distributed supercomputing
 High throughput & data intensive applications.
 Large scale storage
 High speed network connectivity
Why ALICE is interested in GRID?
1 year of Pb-Pb running: 1 PBytes of data
1 year of p-p running : 1 PBytes of data
Simulations :
2 PBytes
Total Data storage:
4 Pbtyes/year
ALICE computing requirements:
Simulations, Data reconstruction & analysis will use about 10,000 PCyears.
GRID is the solution for ALICE
Connect high performance computers from all collaborating countries with a high speed
secured network. implementing one virtual environment that is easy for the “end user’’.
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
2
The MONARC Multi-Tier Model
(1999)
CERN – Tier 0
IN2P3
Tier 1
FNAL
RAL
Uni n
Lab a
Tier2
Uni b
Department

Lab c


Desktop
MONARC report: http://home.cern.ch/~barone/monarc/RCArchitecture.html
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
3
The opportunity of
Grid technology
LHC Computing
Model
Lab m
2001 - evolving
CMS
Uni x
regional group
ATLAS
Uni a
CERN Tier 1
Lab a
UK
USA
Tier3
physics
department
The
opportunity of
Tier 1
Uni n
Grid technology
LHCb
The LHC
Computing
Centre
Tier2
France
CERN
CERN Tier 0
Italy
……….
Desktop

Lab b
Germany
Lab c
physics group


Susanta K Pal
……….
Uni y
Uni b
Tier 0 Centre at CERN
IWLSC, Feb 8-10, 2006, Kolkata, India
4
What is a Tier-2?
The following is taken from the LCG RTAG 6 on Regional Centre
Category and Service Definition.
A possible categorization scheme for regional centres could be based
on service qualities as follows:
Tier1
* CPU cycles (Grid enabled computing elements), advance reservation
* Disk storage, resident and temporary (Grid enabled storage elements),
advance reservation
* Mass storage (Grid enabled storage elements), advance reservation
* State-of-the-art network bandwidth, quality of service
* Commitment to provide access to primary/master copy of data over
lifetime of LHC
* Commitment to provide long-term access to specific analysis data
* Commitment to resource upgrades as required
* 24/7 services and resource support
* National support role
* Training and user support
* Interactive support for particular applications
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
5
Tier2
* CPU cycles (Grid enabled computing elements)
* Disk storage, maybe temporary only (Grid enabled storage elements)
* May have mass storage
* Sufficient network bandwidth for inter-operability
* A weaker commitment to provide access to data over LHC lifetime
* A weaker commitment to provide long-term access to specific analysis data
* A weaker commitment to resource upgrades
* Focused user support
* 24/7 service but with no guaranteed short-time "crash" response
Tier3
* CPU cycles (Grid enabled computing elements)
* Local storage (not necessarily Grid enabled)
* Focused commitment to data access or resource upgrade
* Only local user support
* Focused services for agreed and day-by-day analysis activities
* Local interactive support
Tier4
* Enable Grid access
* Provide experiment specific tools
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
6
Calib
&
Reco
1
Raw
Calib
ESD
AOD
Tag
RAW data delivered by DAQ undergo
Calibration and Reconstruction which
produce for each event 3 kinds of objects:
1.
2.
3.
ESD object
AOD object
Tag object
Done in Tier-0 site.
Further reconstruction and calibration of RAW
data will be done at Tier 1 and Tier 2.
ESD
Reco
n
AOD
The generation, reconstruction, storage and
distribution of Monte-Carlo simulated data
will be the main task of Tier 1 and Tier 2.
Tag
DPD (Derived Physics Data) objects will be
Processed in Tier 3 and Tier 4.
Ana
DPD
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
7
Why Tier 2 ?
1. Tier-2 is the lowest level to be accessible by the
whole collaboration
2. Each sub-detector of ALICE has to be
associated with minimum Tier-2 because of
large volume of calibration data
3. PMD and Muon Arm are the important subdetectors of ALICE
4. We are solely responsible for PMD – from
conception to commissioning
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
8
ALICE
Setup
LHC Utilization -- ALICE
HMPID
Size: 16 x 26 meters
Weight: 10,000 tons
TOF
TRD
Cost 120 MCHF
TPC
PMD
ITS
Muon Arm
PHOS
Indian contribution
to ALICE : PMD, Muon Arm
IWLSC, Feb 8-10, 2006, Kolkata, India
9
Susanta K Pal
ALICE Layout : another view
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
10
ALICE PMD (=2.3 – 3.5)
Honeycomb counters
for ALICE PMD are
Modified edition of
STAR PMD
Two
Halves
In
Vertical
When not
in use
Two planes of honeycomb proportional
counters
3 X0 thick lead converter
Arranged in two halves in vertical plane
Installed at z=360 cm from I.P.
PMD in ALICE : Fully Indian Contribution
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
11
ALICE PMD Layout (Data taking position)
Converter
+ Support
Plate
Super Module
8 Supermodules
in 2 planes
Total channels
Preshower+Veto
= 221,184
Unit Module
48 Nos. Total
Cooling by air
circulation
Unit module components
Honeycomb (4608 cells)
Top, bottom PCBs
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
12
Components of a unit module
TOP PCB
(details)
4 X 16 cells for one MCM Board
32-pin connector
Copper honeycomb
Edge frame
Bottom PCB
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
13
PMD unit module assembled
Unit cell parameters
421 mm
Cell cross-section
23 mm2
Cell depth
5 mm
Centre-to-centre dist.
5 mm
Cell wall thickness
0.4 mm
Anode wire
(Au-plated tungsten)
20 µm dia
Anode-cathode distance
(on PCB)
750 µm
Unit Module, 4608 cells
421 mm X 260 mm
Fabrication of unit modules at Kolkata, Jammu and Bhubaneswar
Final assembly at Kolkata
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
14
PMD Parameters
Pseudorapidity coverage
2.3 – 3.5
Azimuthal coverage
2p
Distance from vertex
350 cm
Detector active area
2 m2
Detector weight
1000 kg
No. of planes
2 (veto + preshower)
Lead plate + SS plate thickness
3 radiation length
Detector
Gas detector with hexagonal cells
Hexagonal cell dimensions
Depth: 0.5 mm, cross section: 0.22 cm2
Total number of cells
221184 (=>in each plane: 110592 cells)
Detector gas
Ar+CO2 (70%+30%)
Total gas volume
0.02 m3 (20 Liters)
No. of Supermodules per plane
4
No. of unit modules per plane
6 (HV isolation at the unit module level)
No. of HV channels
48
Average Occupancy (at full multiplicity)
13% for CPV & 28% for PMD
Photon counting efficiency
64%
Purity of photon sample
60%
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
15
muon tracking quadrants assembled in Kolkata
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
16
Cell-to-cell gain calibration for PMD
Need : To have uniform response throughout the detector
In data look for single
How to do :
isolated cell ADC spectrum
0
0
Test beam studies tell us – hadron
response from our detector
0
ADC>0
0
0
0
STAR PMD Isolated cell
ADC spectra
Single cell
is affected
Susanta K Pal
Pulse height
spectra landau in a
cell : Mean can be
used for calibration
IWLSC, Feb 8-10, 2006, Kolkata, India
17
Cell-to-cell gain calibration for PMD
ALICE PMD has 221184 cells to be calibrated
 For this we need isolated cell ADC spectra for each cell
 Minimum number of entries need for a good Landau
distribution ~ 1000
 We need at least 1 Million events for calibration
So data volume ~200Kx0.25x4BytesX1Million ~ 200GB
Although calibration is done once for a period of
Running, it may be advisable to check the calibration
constants time-to-time.
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
18
MANAS Calibration for Tracking Detectors of MuonSpectrometer
With and Without MANAS Calibration – preproduction
batch : gain spread 5%
Expected Gain Spread in MANAS
production : ~ 2.3 %
Channel gain calibration may not
be essential any more
Production Batch
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
19
Physics data
Expected Data rate for MuonArm
Total number of readout pads in FMS: 1.1 X 106
Trigger rate – 1 kHz
Average occupency ~ 2.5%
1 month of Pb-Pb data
== 300 TB
Pedestal data:
Every run ~ 5 MB (σ values of each channel
needs to be known for analysis.
One pedestal run/hour
40 GB/month
Electronic Calibration data
The frequency will depend on the observed gain
spread (if it is <2.5% then uncalibrated resolution
will be satisfactory)
Every run ~10 MB (for 2 point calibration)
GMS data
Comparable or less than pedestal data.
Susanta K Pal
Estimation from SINP group
IWLSC, Feb 8-10, 2006, Kolkata, India
20
Required Computing Resources for Tier 2
By 2006:
# CPU(Intel P3) Disk Space (TB)
Tier 0:
28800
727
Tier 1 + 2
7200
238
Total 6 Tier 1 centers and for each Tier 1 there will be several (~ 5-6)
Tier 2 Centers. Tier 2 centres should have the capacity roughly 30% of
Tier1 + 2 capacity.
At VECC
2004
2005
2006
2007
2008
Total
# CPU
(Intel Xeon)
8
8
48
64
64
= 192
Disk space
(TB)
0.5
0.5
12
24
12
= 49
Bandwidth: Tier 2 centers rely entirely on associated Tier 1 centers for
Reconstruction data storage. For efficient and fast analysis a bandwidth of
1.5 Gb/s is a reasonable value – although smaller values are viable.
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
21
Our Experience with Alice-Grid & PDC’04
The following H/W and S/W infrastructures were used :
•8 Node Cluster consisting of dual Xeon CPU & 400GB Disk Space.
•PBS Batch System with one Management server and eight Clients
under OSCAR cluster management environment
•ALICE Environment ( AliEn ) was installed
•Data Grid has been registered at cern.ch
•AliROOT, GEANT and other production related packages are tested
successfully in both ways
•Linked with CERN via 2Mbps available Internet link and Participated in
PDC’04
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
22
Our Experience with Alice-Grid & PDC’04
AliEn Architecture in General
Core Middleware
Low level
Susanta K Pal
User
Application
Logger
Package
Mgr
(…)
V.O.
Packages
&
Commands
Monitor/Gatekeeper
Config
Mgr
Storage Element
SOAP/XML
Core
Modules
External
Libraries
Computing Element
User Interface
Resource Broker
Interfaces
API
(C/C++/perl/java)
Authenticator
LDAP
File & Metadata
Catalogue
ADBI
Database
Proxy
DBI
DBD
RDBMS
Core Components & Services
FS
CLI
GUI
Web
Portal
High level
IWLSC, Feb 8-10, 2006, Kolkata, India
23
Our Experience with Alice-Grid & PDC’04
Grid Services
Workload Management
– Reliable Job Submission
– Matchmaking
– Logging & Bookkeeping
Data Management
– Replica Management
– Metadata Management
Resource Management
– Gatekeeper (batch
Interface)
– Storage (disk, tape, etc.)
– Database (SQL, …)
– Network
Susanta K Pal
Information System
– Service Discovery
– Service State
Information
Security
– Authentication
– Authorization
IWLSC, Feb 8-10, 2006, Kolkata, India
24
Our Experience with Alice-Grid & PDC’04
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
25
Preparing for next PDC
 Changed the cluster software from OSCAR to QUATTOR

OSCAR
 Performance degrades with more Nos. of Nodes ( > 32 Nodes )

QUATTOR
 Perfomence does not degrade with increase of nodes
 Better performance with more Nos. of Nodes
 Dedicated Band-Width for Tier2@Kolkata -> 4 Mbps
 Separate Domain Name for Tier2; ‘tier2-kol.res.in’
 Addition of More Nos. of CPUs (48-Xeon) and Storage(7TB)
 Installation of gLite is in progress
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
26
Preparing for next PDC
GAS
WM
DM
RB
TQ
FPS
FTQ
PM
ACE
FC
JW
JA
LRC
LJC
SE
CE
SRM
CR
Grid Access Service
Workload Mgmt
Data Mgmt
Resource Broker
Task Queue
File Placement Service
File Transfer Queue
Package Manager
AliEn CE (pull)
File Catalogue
Job Wrapper
Job Agent
Local Replica Catalogue
Local Job Catalogue
Storage Element
Computing Element
Storage Resource Mgr
Computing Resource
(LSF, PBS,…)
Susanta K Pal
gLite Architecture
API
GAS
WM
DM
TQ
FTQ
gLite
PM
FC
Middleware
JW
CE
SE
(JA) L
Services
L
ACE
CR
(LSF,..)
JC
IWLSC, Feb 8-10, 2006, Kolkata, India
RC
SRM
27
Present Status
VECC- Cluster :
High Availability Quattor
CERN
Management Node
ROUTER
Switch
Gigabit Network
4Mbps
Management Node
(Stand-by)
Internet Cloud
FireWall
SINP- Cluster :
High Availability Quattor
Susanta K Pal
Tier-2@Kolkata
IWLSC, Feb 8-10, 2006, Kolkata, India
28
Present Status
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
29
Present Status
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
30
Present Status
grid.veccal.ernet.in graphs last hour sorted descending
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
31
Tier-II Centre for ALICE
(Update on VECC and SINP Activities)
 New Domain name “tier2-kol.res.in’ has been registered and
work is going on
 CONDOR Batch System is running with one server and eight
Clients under QUATTOR cluster management environment
 AliROOT, GEANT and other production related packages are
tested successfully in both ways
 ALICE Environment ( AliEn ) at present NOT running
 Data Grid has been registered at cern.ch
 Linked with CERN via 2Mbps available Internet link.
 4MBPS band-width is already installed and commissioned
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
32
Tier-II Centre for ALICE
(Update on VECC and SINP Activities)
Infrastructure Status:
•Fully Equipped GRID Room is Ready
(Sufficient
to accommodate projected Tier-II H/W)
Main Room
Main
Main Room
Room
Six Auto Switchable AC units
1.
20ft x 19ft
2.
Auto
Switchable
units
SixSix
Auto
Switchable
ACAC
units
NW to VECC main
30ft x 20ft
VECC
SINP
UPS room
10ft x 19ft
UPS room
20ft x 15ft
UPS room
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
33
VECC Grid Computing Facility Room
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
34
Proposed Indian ALICE-GRID Project
Indian ALICE Collaborators
•VECC, SINP - Kolkata •Panjab Univ., Chandigarh
•IOP, Bhubaneswar
•Jammu Univ., Jammu
•Rajasthan Univ., Jaipur •AMU, Aligarh
CERN
Chandigarh
Kolkata
Alice-Grid
Router
Modem
Router
Modem
Jammu
Jaipur
Router
Internet Back-Bone
Modem
Router
Modem
Gateway Router +Modem
Bhubaneswar
Aligarh
Modem
Router
Susanta K Pal
Router
Modem
IWLSC, Feb 8-10, 2006, Kolkata, India
35
Contribution towards LCG-GRID PROJECTS
LCG-AliEn-SE Interface using GridFTP
1st-July’03 to 18th Dec’03
Status:
The project was successfully completed
before schedule
Milestones of the Project were as follows :

Test Bed Installation for Grid Environment The
configuration consists of one central server and two
Sites.

Certification Authority Server has been installed
A simple certification authority has been installed to
generate certificates for authentication purpose.

Installation of GridFTP Library under AliEn
The GridFTP daemon in.ftpd has been used as server
and globus-url-copy has been used as client

Development of AliEn-SE Interface via GridFTP
These newly developed modules along with
necessary GridFTP libraries and changes made in
existing AliEn Code have been committed to CVS
Server at CERN.
Susanta K Pal
Quality Assurance and Test Environment
For AliEn-ARDA* Prototype
23rd Dec’03 to 31st March’04
The Project was successfully completed.
Milestones of the Project were:

Exploration and Design of Test Scripts using perl.

Implementation of Test Scripts for each
Individual perl sub-module of AliEn.
Individual perl sub-modules of AliEn code were
tested for proper functionalities. It Generates a
detailed report of the individual tests and maintains
a log.

Validation of Test-Scripts and Procedures.

Testing
Modules
with
perl
Harnessing
Environment.
The Complete Suit was tested at CERN under perl
Harnessing Environment for testing AliEn online
and generating online consolidated report of the test.

Inline Documentation to the extent possible.
ARDA Architectural Road-Map towards Distributed
Analysis
IWLSC, Feb 8-10, 2006, Kolkata, India
36
To Summarize
•Infrastructure is ready to accommodate Tier-2 H/W
•Upgradation of present AliEn cluster with gLite is in progress
•Middleware are installed in the Current Hardware with latest
Scientific Linux OS platform
•With limited resources, VECC took part in ALICE ProductionData-Challenge(PDC-2004)
• Getting ready for Next PDC with upgraded infrastructure
•Upgradation of CE and SE is in the process as per requirement
•One FTE and 2 Engineers with 30% of time are engaged
•Two more FTEs are approved
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
Thank you
37
PMD
V0
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
38
PMD Split Position
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
39
Super Module and Unit module
arrangement of ALICE PMD
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
40
Super Module Type B
UM :
- 12 - FEE Boards in a row
- 6 Nos. of rows
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
41
Super Module Type A
UM:
24-FEE Boards
in a Row
3- Rows
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
42
Contribution towards LCG Project
•LCG-AliEn Storage Element Interface (addendum-5)
1st-July’03 to 18th Dec’03
Value: 150 KCHF
•Test Suite for perl harnessing with AliEn (addendum-6)
23rd Dec’03 to 31st March’04
Value: 150 KCHF
•VECC as a part of the Alice Data-Challenge Team run offline
production on the existing infrastructure.
•Currently main emphasis is on participation in ALICE Physics
Data Challenge 2005-06.
Future Projects:
• Development of Test Environment for ADRA (Architectural
Roadmap for Distributed Analysis) code
• Testing ARDA code under Perl Test harness guidelines
• Part in EGEE ( Enabling Grid for E-Science Euro) prototype development
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
43
Our Experience with Alice-Grid & PDC’04
Security: Overview
User side
Getting a Certificate
Becoming a member of the VO
Server side
Authentication /CA
Authorization / VO
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
44
Our Experience with Alice-Grid & PDC’04
Security: Authentication
Overview
Once in every year
Once in every year
cert-signing
grid-cert-request
grid-cert-request
cert-signing
CA
ca/url update
cert-request
certificate
cert.pkcs12
convert
grid-proxy-init
USER
host-request
host-cert
CA-certificate
Once in life-time of
the VO (only DN)
Every night/week
crl
gridmap
Registration
proxy-cert
SERVICE
mkgridmap
VO-LDAP
Once in every 12/24 Hours
host/proxy certificate exchanged
Every night/week
Susanta K Pal
IWLSC, Feb 8-10, 2006, Kolkata, India
45