Maximize your calibration event rate with partial
events
Ignacio Aracena (SLAC)
SLAC ATLAS forum
March 18th 2009
Outline
• Introduction
–
–
–
–
Motivation
The ATLAS detector
Calibration requirements
The ATLAS TDAQ system
• Partial events for calibration
• Calibration with partial events
–
–
–
–
Inner detector alignment
LAr calibration
Tile calibration
Test results
• Rate studies with partial events
• Open issues
• Summary
2
Motivation
Physics at the LHC:
New physics with (relatively) high pt at few Hz
vs SM physics at ~kHz
ATLAS trigger strategy for new physics:
• Improve physics object selection at three trigger levels
• Region of Interest (RoI) minimizes data
requests/shipping through network
L1 out 100kHz
L2 out 3kHz
5GB/s
EF out 200Hz
300MB/s
Level 1 : Identifies RoI to seed L2 @ 100kHz
Level 2 : Improves RoI eta, phi, pT by using
high detector granularity, accept event rate ~3kHz
Event Filter : full event data available for further
improvement, output rate ~200Hz, i.e. 300MB/s to
permanent storage for offline analyses.
Trigger architecture driven by physics programme
How much trigger rate can we afford for calibration events during physics run?
3
The ATLAS detector
ATLAS subdetectors
EM calorimeter:
LAr calorimeter
Hadronic calorimeter:
Tile calorimeter (barrel)
LAr calorimeter (endcaps)
Tracking :
Pixel, SCT, TRT
Muon spectrometer
TGC/RPC (trigger chambers)
CSC/MDT (precision chambers)
Precise and fast calibration of different subdetectors is crucial for achieving performance goals
4
Event data volume composition
• Event data in terms of Readout buffers (ROBs) requested by
TDAQ:
– LAr 762 ROBs
– Tile 64 ROBs
– Pixel 132 ROBs
– SCT 86 ROBs
– TRT 192 ROBs
– Muons ~400 ROBs
– Total ~1600ROBs
• 1ROB ~ 1kB
• Full event with data of all the ROBs is ~1.5MB (about 50%
from LAr...)
How much data is needed for calibration of subdetectors?
5
Calibration requirements
• Inner detector alignment
– Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours for
TRT
– Requires ~50Hz of isolated tracks
• LAr calibration
– Study pulse shape of individual cells
– Requires ~5 Hz to achieve precision of <1%
• Muon spectrometer
– Need to collect ~106 muons/day for MDT t0 calibration
– ~1kHz of muon tracks
• Tile calorimeter
– Illuminate full calorimeter with dedicated tile calibration system, run
during empty bunch crossings
• New calibration types?
6
Calibration vs Physics
Can we take calibration data during physics runs?
Calibration rate appear to be not matching the
rate limits of TDAQ design scheme at first glance!
L1 out 100kHz
L2 out 3kHz
5GB/s
EF out 200Hz
300MB/s
Muon spectromter calibration ~1kHz
≈ L2 accept rate
Inner detector alignment ~50Hz
25% of EF output rate
LAr calibration ~5Hz
3% of EF output rate
7
hardware
The ATLAS TDAQ system
100 kHz
L1
ReadOut Systems (ROS)
Readout buffers (ROBs)
RoI
3 kHz
L2
Event builder (SFI)
3 kHz
EF
software
200 Hz
Data logger (SFO)
200 Hz
Tier0
ATLAS TDAQ system is based on:
L1 100kHz → L2 3kHz → EF 200Hz
full event size ≈ 1.5MB, RoI size ≈ 3% of full event
Event dataflow
Level1 : If RoI found, all detector data is kept in the
Readout buffers (ROBs) of the Readout system (ROS).
Level2 : Requests data from RoI, with full detector
granularity. If event is accepted, the event is fully built
Event builder : For L2 accepted events the full event
data from the ROS is pulled to build the full event.
Event Filter : Receives full event, which can be used for
further improvements. Sends accepted events to the data
logger (SFO)
SFO : Writes events into different data streams, sends
data to offline mass storage
8
hardware
The TDAQ system
100 kHz
L1
ReadOut Systems (ROS)
Readout buffers (ROBs)
RoI
3 kHz
L2
Event builder (SFI)
(5600MB/s)
3 kHz
EF
software
200 Hz
(300MB/s)
Data logger (SFO)
200 Hz
Tier0
ATLAS TDAQ system is based on:
L1 100kHz → L2 3kHz → EF 200Hz
full event size ≈ 1.5MB, RoI size ≈ 3% of full event
Network requirements:
bandwidth = event rate * event size
Max. bandwidth is fixed by the system installed at P1
Calibration events
should use small fraction of total bandwidth
need high trigger rate
don’t need the full event!
Can increase calibration event rates for same
bandwidth budget by using partial events!
9
hardware
Partial events for calibration
Select calibration event at Level 2:
• Fill list of detector (ROB) identifiers needed for a
specific calibration purpose.
• In the event builder (SFI) use this list to pull out data
only from identifiers in this list and build partial event
• Send partial event to SFO, no need to go through EF.
100 kHz
L1
ReadOut Systems (ROS)
Readout buffers (ROBs)
RoI
3 kHz
L2
Event builder (SFI)
(5GB/s)
3 kHz
partial event
building
EF
software
200 Hz
(300MB/s)
Data logger (SFO)
200 Hz
event stripping
Tier0
Select calibration event at the Event Filer:
• All events that are processed in the EF are fully built
• Fill list of detector (ROB) identifiers needed for a
specific calibration purpose.
• In the SFO use this list to extract only data fragments
from listed ROB identfiers
• Create a partial event from the full event : event
stripping, ship and store only partial event
For an allocated bandwidth partial events allow higher
trigger rates of calibration events, i.e.
partial events save bandwidth at the SFI and SFO
10
Calibration + physics events
Overlapping events:
events that are selected by calibration triggers and physics triggers
Logic to recover the partial event for calibration implemented
• If only overlap at L2, apply event stripping in the EFD
• Allows further save of bandwidth in the EFD
• If overlap at EF, apply event stripping at the SFO
EFD = Event Filter data flow program
PT = EF processing task
11
Calibration requirements and partial events
• Inner detector alignment
– Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours for TRT
– Requires ~50Hz of isolated tracks
– partial event build
• LAr calibration
– Study pulse shape of individual cells
– Requires ~5 Hz to achieve precision of <1%
– event stripping
• Muon spectrometer
– Need to collect ~106 muons/day for MDT t0 calibration
– ~1kHz of muon tracks
– partial event building? Muon group has developed a standalone “TDAQ”
system to extract muon tracks. Less flexible than using partial event from
TDAQ
• Tile calorimeter
– Illuminate full calorimeter with dedicated tile calibration system, run during
empty bunch crossings
– partial event building
12
Inner Detector alignment
Requirements
• Get high rate O(50Hz) over wide pT range
Chains in current Lumi1E31 menu (athena 15.0.0):
• trk9i_id, prescale 40
• trk16i_id, no prescale
• Route to “calibration_IDTracks” stream
Fill list of ROBs at L2
use case for partial event building
(Anna Sfyrla, Carlos Escobar)
13
LAr calibration
Requirements
• Sufficient rate (5Hz) given by photon triggers
into EF, no need for additional rate using partial
EB
• Minimize bandwidth at the EFD/SFO
LVL1
EM RoI
Chains in current Lumi1E31 menu (athena 15.0.0):
LVL2
photon selection
•
•
•
•
•
•
EF
photon selction
g3_larcalib (PS 10^6)
g10_larcalib (PS 100)
g20_larcalib (PS 1)
g20i_larcalib
j10_larcalib (PS 10^6)
j80_larcalib (PS 10^6)
Fill list of ROBs only at EF selection
use case for event stripping
event stripping
use only ROBs
inside RoI
(Nicolas Berger, Guillame Unal, Isabelle Wingerter-Seez)
14
Tile calibration
Tile calorimeter has dedicated calibration system:
• Laser injection (timing studies)
• Charge injection (pulse shape)
which illuminates the full tile calorimeter
Used during emtpy bunch crossing
Chains used in cosmic trigger menu
• TileCalib_laser
• TileCalib_cis
LVL1
CALREQ
LVL2
fill list with all
tile ROBs
partial event build
Trivial use case of partial event with a
complete subdetector
(Andrea Dotti, Oleg Solovyanov)
15
Tests with AtlasHLT-14.5.0
• Setup multihost partition on lxplus with AtlasHTL-14.5.0
(tdaq-02-00-00)
– 2 L2 segments (L2SV, L2PUs),
– 1 EBEF segments (2 SFIs, 1 SFO, 1 EF segment, 4PTs)
– 1 ROS segment (8 ROSs)
• Data sample : /pcatrsrv1/data/files/aagaard/14.5.0/enhancedBias10TeV/BS/ (1200
events)
– “enhanced bias” = L1 preselection EM3, MU4, TAU5, J18, FJ18,
XE25, TE250
– preload the data on ROS (ROS application running on lxplus node)
• Lumi1E31 menu
– testing the ID and LAr calibration streams
• Recorded monitoring histograms after ~1hour run
• Monitoring plots from this test on the following slides
16
Partial event building performance – event size
Size of built events in the SFI
Event size vs streams in the SFI
overlapping events (calib + phys)
need to be stripped
Full event size ~2MB (MC data)
Partial event ~50kB (~3% of full event)
partial events for ID alignment
About 30% of ID alignment events overlap
with physics events
Overlap fraction can be tuned with the menu
Impact on resources? See next slide
17
Partial event building performance – rates
Contribution to event rate at the SFI
stream X event rate / total event rate
Bandwidth contribution at the SFI
stream X event build rate / total event building rate
overlapping events (calib + phys)
need to be stripped
ID alignment event rate comparable to
physics event rates
Pure ID alignment events use very small
fraction of total bandwitdh
Partial event building minimizes the bandwidth fraction used for calibration events
18
Event stripping performance – event size
Event size vs streams in the SFO
(full events from calibration_LArCells,
bug in menu)
Full event size ~2MB (MC data)
Partial event ~50kB (~3% of full event)
stripped events for LAr calibration
(ignore calibration_LArCells,
bug in menu)
19
Event stripping performance – event size
Event size vs streams in the SFO
(full events from calibration_LArCells,
bug in menu)
ID alignment events not stripped
bug, fixed in 15.0.0
Full event size ~2MB (MC data)
Partial event ~50kB (~3% of full event)
stripped events for LAr calibration
(ignore calibration_LArCells,
bug in menu)
20
Event stripping performance – rates
Contribution to event rate at the SFO
stream X event rate / total event rate
LAr calib event rate comparable to
physics event rates
Bandwidth contribution at the SFO
stream X event build rate / total event building rate
LAr calib events use very small fraction of
total bandwitdh
Event stripping minimizes the bandwidth fraction used for calibration events
21
Examples from cosmic data taking runs
• Previous slides showed only tests with MC data
• Tile calibration and ID alignment stream exercised during
cosmic data taking runs in following slides
22
Tile calibration – example from cosmic
Exercised during cosmic runs, example here from run 92082
• Full event size ~5MB (LAr requesting 10 samples)
• Tile calibration event size ~200kB, i.e. ~1% of full event
• No rate monitoring plots for this run (Atlas release 14.2.23), but with given event
size uses only ~1% of bandwidth
• Event successfully used for calibration
LVL2
fill list with
all tile ROBs
partial event
build
Number of events
LVL1
CALREQ
partial event ~200kB
full event ~5MB
Event Size (kB)
23
Inner detector alignment with cosmics

Use chains developed for cosmic data (e.g. CosmicsAllTeTRTxK*, CosmicsAllTeSiTrack*,
CosmicsAllTeIDScan*) and apply PEB on those
✘ … CosmicsAllTe chains do not have ‘well defined’ RoIs
✘ … chains that do tracking at the TRT are not good for testing since TRT does not provide η measurement.

Modify the TrigROBListWriter.cxx to loop over the tracks and get η-φ range from the tracks

Apply modified TrigROBListWriter.cxx to two chains:
o
o
CosmicsAllTeIDSCAN_AllPhysics_TrkHypo and
CosmicsAllTeSiTrack_AllPhysics_TrkHypo

Generate new cosmic menu that contains these new chains, and run with this menu on cosmic data.

The generated raw data will contain information from the ID only and for the η-φ range that is defined by
the modified TrigROBListWriter.cxx

Compared raw data generated with and without
the PEB applied in the chains
Run 90943
Partial events gives same results as full event data
(Anna Sfyrla, Carlos Escobar)
24
HLT rate studies with partial events
• Use partial event to store only trigger decision bits from
L1/L2/EF
– Allows high statistics rate studies
– https://twiki.cern.ch/twiki/bin/view/Atlas/BeatenbergActions#General_
Action_Items , see “T4”
• Could also be combined with monitoring information for
TDAQ performance tests
• Basic TDAQ infrastructure ready. Need to configure menu
“chain”, study the impact on the resources
25
Open issues
• Only one list of ROBs (subdetectors) per event. List is union
of all partial events. Results in overlapping ROBs (e.g. LAr
ROBs showing up in ID alignment events)
• Tests on MC show no overlap between LAr calibration and ID
alignment. More studies needed
26
Open issues
• Only one list of ROBs (subdetectors) per event. List is union
of all partial events. Results in overlapping ROBs (e.g. LAr
ROBs showing up in ID alignment events)
• Tests on MC show no overlap between LAr calibration and ID
alignment. More studies needed
• Bandwidth budget per sub-detector calibration?
– Need to establish rules/policy
– Needs to be discussed across sub-detector groups and TDAQ group
27
Summary
• Calibration with partial events
– Using partial events for calibration of sub-detectors allows to increase
the event rate for a given bandwidth budget
– Flexible system that allows calibration data taking with all subdetectors
• Being used by LAr, ID, Tile
– Partial event already tested and used at P1 during technical runs, cosmic
data taking
– Muon MDT t0 calibration? Standalone system does not offer flexibility
of partial events. Next customer on the list
• Potential new use case discussed in Beatenberg
– Partial event build with only trigger decision bit information for trigger
rate studies
• New calibration types?
– Advantage of partial event is flexibility, can accommodate all subdetectors
28
Outlook
• Run test at P1
– Uses real system
– Larger statistics available
– Can re-play cosmics and/or pre-load MC data
• Implement monitoring
– Rate plots shown here not implemented yet in official SFI/SFO tags
– New monitoring plots?
29