CBM DAQ and Event Selection
Walter F.J. Müller, GSI, Darmstadt
for the CBM Collaboration
International Workshop on
Transition Radiation Detectors – Present and Future
Cheile Gradistei, Romania, September 24-28, 2005
Outline

CBM (very briefly)



FEE/DAQ/Event Selection



25 September 2005
observables
setup
requirements
challenges
strategies
TRD Workshop, Cheile Gradistei, September 24-28, 2005
2
CBM at FAIR ... evolving
SIS 100 Tm
SIS 300 Tm
U: 35 AGeV
p: 90 GeV
Compressed
Baryonic Matter
Experiment
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
3
CBM Physics Topics and Observables

In-medium modifications of hadrons
 onset of chiral symmetry restoration at high ρB
 measure: , ,   e+e- (μ+ μ-)
Good e/π separation
open charm: D0, D±

Strangeness in matter
Vertex detector
 enhanced strangeness production
 measure: K, , , , 

Indications for deconfinement at high ρB
 anomalous charmonium suppression ?
 measure: D0, D±
J/  e+e- (μ+ μ-)

Low cross sections
→ High interaction rates
→ Selective Triggers
Critical point
 event-by-event fluctuations
 measure: π, K
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
Hadron identification
4
CBM Setup
 Radiation hard Silicon pixel/strip detectors in a magnetic dipole field
 Electron detectors: RICH & TRD & ECAL: pion suppression up to 105
 Hadron identification: RPC, RICH
 Measurement of photons, π0, η, and muons: ECAL
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
5
CBM Setup... evolving
HBD
μID
ZCAL
 Add ZCAL to determine centrality
 Consider HPD for early lepton tagging (optional)
 Consider μID detector (optional)
 Consider various Silicon tracker implementations
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
6
CBM Trigger Requirements

In-medium modifications of hadrons
 onset of chiral symmetry restoration at high ρB
offline
 measure: , ,   e+e±
open charm (D0, D ) trigger

Strangeness in matter
 enhanced strangeness production
 measure: K, , , , 
offline

Indications for deconfinement at high ρB
 anomalous charmonium suppression ?
trigger
 measure: D0, D± trigger
J/  e+e

Critical point
 event-by-event fluctuations
 measure: π, K
25 September 2005
assume archive rate:
few GB/sec
20 kevents/sec
trigger on
displaced vertex
drives FEE/DAQ
architecture
trigger on high pt
e+ - e- pair
offline
TRD Workshop, Cheile Gradistei, September 24-28, 2005
7
Open Charm Detection





Example: D0  K-+ (3.9%; c = 124.4 m)
reconstruct tracks
find primary vertex
find displaced tracks
find secondary vertex
target
few 100 μm
5 cm


standard procedure....
for CBM, this is in 1st level 'trigger'
like for LHcB or BTeV (rip)
25 September 2005
first two planes
of vertex detector
TRD Workshop, Cheile Gradistei, September 24-28, 2005
8
CBM DAQ Requirements Profile


D and J/Ψ signal drives the rate capability requirements
D signal drives FEE and DAQ/Trigger requirements




Problem similar to B detection, like in LHCb or BTeV (rip)
Adopted approach:
displaced vertex 'trigger' in first level, like in BTeV (rip)
Additional Problem:
DC beam → interactions at random times
→ time stamps with ns precision needed
→ explicit event association needed
Current design for FEE and DAQ/Trigger:


Self-triggered FEE
Data-push architecture
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
9
Conventional FEE-DAQ-Trigger Layout
Especially
instrumented
detectors
Detector
L0 Trigger
fbunch
Trigger
Primitives
FEE
Dedicated
connections
Buffer
Limited
capacity
Modest
bandwidth
L2 Trigger
Archive
25 September 2005
Standard
hardware
L1 Accept
DAQ
Cave
Limited
L1 trigger
latency
TRD Workshop, Cheile Gradistei, September 24-28, 2005
Shack
L1 Trigger
Specialized
trigger
hardware
10
Limits of Conventional Architecture
Decision time for first
level trigger limited.
typ. max. latency 4 μs for LHC
Not suitable for complex
global triggers like secondary
vertex search
Only especially instrumented
detectors can contribute to
first level trigger
Limits future trigger
development
Large variety of very
specific trigger hardware
High development cost
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
11
The way out .. use Data Push Architecture
Especially
instrumented
detectors
Detector
L0 Trigger
ffbunch
clock
Trigger
Primitives
FEE
Time
distribution
Dedicated
connections
Buffer
Limited
capacity
Modest
High
bandwidth
L2
L1 Trigger
L2Archive
Trigger
25 September 2005
Standard
Special
hardware
L1 Accept
DAQ
Cave
Limited
L1 trigger
latency
TRD Workshop, Cheile Gradistei, September 24-28, 2005
Shack
L1 Trigger
Specialized
trigger
hardware
12
The way out ... use Data Push Architecture
Detector
fclock
FEE
Cave
Shack
DAQ
High
bandwidth
L1 Trigger
L2Archive
Trigger
25 September 2005
Special
hardware
TRD Workshop, Cheile Gradistei, September 24-28, 2005
13
The way out ... use Data Push Architecture
Detector
Self-triggered front-end
Autonomous hit detection
fclock
FEE
No dedicated trigger connectivity
All detectors can contribute to L1
Cave
Shack
Large buffer depth available
System is throughput-limited
and not latency-limited
Use term: Event Selection
25 September 2005
DAQ
High
bandwidth
L1 Select
L2
Select
Archive
Special
hardware
TRD Workshop, Cheile Gradistei, September 24-28, 2005
Modular design:
Few multi-purpose rather
many special-purpose
modules
14
Front-End for Data Push Architecture





Each channel detects autonomously all hits
An absolute time stamp, precise to a fraction
of the sampling period, is associated with each hit
All hits are shipped to the next layer
(usually concentrators)
→ power efficient
front-end
→ low jitter
clock/time
distribution
→ high bandwidth
Association of hits with events done later
using time correlation
Typical Parameters:

with few 1% occupancy and 107 interaction rate:

some 100 kHz channel hit rate

few MByte/sec per channel

whole CBM detector: 1 Tbyte/sec
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
15
Typical Front-End Electronics Chain
PreAmp





Si Strip
Pad
GEM's
PMT
APD's
Shaper
Shaping
time:
5-200 ns
ADC
Sample rate:
10-100 MHz
Dyn. range:
8...>12 bit
digital
Filter
1/t Tail
cancellation
Baseline
restorer
Hit
Finder
Hit
parameter
estimators:
Amplitude
Time
Backend
& Driver
FIFO-Buffering
Line protocol
All in one mixed-signal chip
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
16
Towards a Multi-Purpose FEE Chain
PreAmp
Several selectable
input stages;
→ pos/neg signals
→ capacitance match
25 September 2005
Shaper
switched capacitor filter
→ programmable
shaping time
ADC
programmable
sampling rate
TRD Workshop, Cheile Gradistei, September 24-28, 2005
digital
Filter
Hit
Finder
programmable
hit finder and
estimator
17
Towards an Over All Architecture
to/from next
neighbor hit
detection
N PreAmplifier
Blocks
Hit Detector
• Leading Edge
• peak
•…
DAC
Programmable
Finite State
Machine
send
Fast
Readout
DATA-out
start
Ain
Digital
Pipeline
Hit TDC
TIMER
stop
CLOCK
we
1..N
WE
.
.
.
RE
Analog
Filter
to/from next
neighbor hit
detection
25 September 2005
sample
ADC
ANALOG
FIFO
Event
Buffers
Digital
Filters
different combinations
• sampling rate
• resoluion
• power
leveraging TRAP knowledge
and expertise
TRD Workshop, Cheile Gradistei, September 24-28, 2005
18
A CBM M-MPW Run

CBM organized a Multi-Multi Project Wafer run in UMC 0.18 μm CMOS



6 different parts combined on a 5 x 5 mm2 wafer
submitted June to Europractice (IMEC)
dies (already cut) just delivered, now come the moments of truth
Test
structures
PreAmp for
Si Strip
Atkin
Fischer
Brüning
Coordination:
Marcus Dorn @ KIP
Content addressable
memory
DLL based TDC
Clock-Data recovery
Deppe
Muthers
12bit 50MSPS ADC
Tontisirin
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
19
FEE Summary

Concrete R&D on FEE ASIC buildings blocks



MPWs done in 2005 covering essential blocks
analyze results, plan now next steps
Other ongoing developments:

Fast PreAmp-Shapers by H.K. Soltveit




in 0.35 μm AMS CMOS as spin-off of DETNI
in 0.13 μm IBM CMOS
next generation TAC by H. Flemming
FOPI and HADES work on RPC electronic


both learn to build mid-sized systems
self-triggered multi-hit TDC based on
CERN HPTDC as HADES spin-off planned
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
20
CBM DAQ and Online Event Selection



needed
for D
needed
for J/μ
More than 50% of total data
volume relevant for first level
event selection
Aim for simplicity
Simple two layer approach:
1. event building
2. event processing
usefull
for J/μ
STS, TRD, and ECAL data used
in first level event selection
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
21
Logical Data Flow
Concentrators:
multiplex channels
to high-speed links
Time distribution
Buffers
Build Network
Processing resources for
first level event selection
structured in small farms
Connection to
'high level'
selection processing
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
22
Bandwidth Requirements
Data flow:
~ 1 TB/sec
Gilder helps
Moore helps
1st level selection:
10~14 operation/sec
Data flow:
few 10 GB/sec
archive target: 1 GB/sec
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
23
CBM DAQ / L1 Event Selection
Requirements Profile Revisited

So far assumed:



D and J/Ψ measured simultaneously @ 107 int/sec
requirements driven by D (displaced vertex @ L1)
Current scenarios:





open charm runs with ~106 int/sec
look into D0, D±, Ds, and Λc
look into J/Ψ → e+elook into J/Ψ → μ+μlook into p-A and p-p
less computational B/W
open charm L1 select
Λc has cτ = 60μm only
lower σ → 107 int/sec
all new game, still open
even lower σ → 108+ int/sec
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
24
CBM DAQ / L1 Event Selection
Requirements Profile Revisited
System
Observable
Interaction
Rate
Au+Au
open charm
106
C+C
open charm
107
p+p,p+A
open charm
Au+Au
J/Ψ
107
C+C
J/Ψ
108
p+p,p+A
J/Ψ
few 108
p+A
Y
few 108
25 September 2005
few
107
MAPS/DEPFET pile-up again
primary vertex determination ?
TRD Workshop, Cheile Gradistei, September 24-28, 2005
events no longer well
separated in time....
25
Feasibility & Options: BNet
Data flow in
Event Builder
~ 1 TB/sec
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
26
Example: InfiniBand Switches

TODAY:





likely in a few years:





288 4x port QDR
likely same or lower cost
1152 GByte/sec switching speeds
adequate for CBM...
Conclusion:


25 September 2005
Voltaire ISR 9288 InfiniBand switch
288 4x ports; non-blocking
cost today ~120 kEUR (or ~400 EUR/port)
288 GByte/sec switching bandwidth
BNet switch is not a major issue
There will be a COTS solution
TRD Workshop, Cheile Gradistei, September 24-28, 2005
27
Feasibility & Options: PNet
Initial proposal:
Network of FPGA and
CPU resources
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
28
Shown in CBM Collaboration
Meeting October 2004
Slide from CHEP'04
Dave McQueeney
IBM CTO US Federal
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
29
The Cell Processor


one 'normal' PowerPC CPU (PPE)
8 Synergistic Processing Element
(SPE) each with





256 kB memory; 128 x 128 bit register
4 SP floating point units (SIMD)
designed for best GFlop/W
runs at ~ 3 GHz → 200 GFlop peak
Software support state:




Linux integration planned for 2.6.13
SPE interface via spufs
gcc/gdb support for C/C++
SIMD instructions via instrinsics...
Developed by
Sony, Toshiba and IBM
Market: VIDEOGAMES
Budget: 500 M$
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
221 mm2
die size
in 90 nm
30
The Cell Processor cont.

Processors like the Cell offer




Even Intel has realized
now that this is the
essential figure of merit
This is a very promising platform for L1 tracking



massive single precision floating point capabilities
large memory bandwidth
very good power efficiency (GFlop / W)
great for algorithms with a lot of arithmetic
probably much easier to handle than FPGAs
Next steps



analyze CA/KF tracker algorithm
determine the SIMD potential
re-implement it for speed


Note: might even improve time on a normal PC when SSE is used....
and of course, get a Cell system ...

25 September 2005
Note: 1000 Cell processors have 200 SP TFlop peak ....
TRD Workshop, Cheile Gradistei, September 24-28, 2005
31
CBM FEE/DAQ Demonstrator

Mission: Provide a platform to

demonstrate essential architecture elements of the CBM FEE-DAQ concept











self-triggered, data push, conditional RoI based readout
combined data, time, control, and RoI traffic
low jitter clock and synchronization over serial links
high bandwidth, RDMA based architecture
integrated approach for DCS/ECS
provide test bed for all future FEE/DAQ prototyping in


FEE:
CNet:
TNet:
BNet:
E/DCS:
hardware
firmware
controlware
software
perform beam tests with detector prototypes
form basis for medium-scale applications in intermediate-term experiments
Be operational by end 2006


avoid cathedrals, go for the bazaar, try and learn
build a first generation (G1) demonstrator quickly
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
32
System Layout
FEE boards
FEE
CNet links
CNet
DCB: data combiner boards
TNet
Clock/Trigger distribution
CDL: CBM detector links
ABB: active buffer boards
BNet
EB Switch
Backend processors
General Network
HNet
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
33
G1 Demonstrator: PNet
FEE
CNet
TNet
BNet


PNet

HNet

No explicit PNet in 1st
generation demonstrator
Will be included with later
upgrades
For quick start normal PCs
for all processing
HNet will be a GE
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
34
G1 Demonstrator: BNet
FEE

CNet
Key architectural element in
modern high speed data
transport is RDMA


TNet

Separate


BNet

data transport
via zero-copy RDMA
status management
via reliable messages
Proposal:


HNet
e.g. Infiniband
e.g. iWARP, iSCSI
Architecture choice:
CBM BNet based on RDMA
(e.g. uDAPL or other API)
Implementation choice:
use Infiniband
(readily available now)
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
35
ABB and BNet – A closer look
BNet technology
choice and ABB
now decoupled
EB Switch
Traffic on
BNet links
is now
bidirectional
CDL: detector links
ABBs in a subset
of the backend
processors
ABB
BNet
NIC
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
36
G1 Demonstrator: CNet / DCB
FEE

CNet link proposal:

use either

CNet


TNet
possibly augment with


BNet
external clock recovery to
provide low-jitter clock
small CPLD to handle low
level protocol and provide
separate ports for control
and data

DCB (data combiner boards)
are the CNet switch nodes
G1 prototype FPGA based

CDL (CBM detector link):

HNet
OASE
Virtex-4 MGT

2.5 Gbps
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
37
DCB: Data Combiner Board
............
OASE
OASE
OASE
MEM
OASE
bi-directional optical link;
data, trigger, RoI, control, clock
OASE or MGT based
Processing
Block
for processing:
small Virtex-4 FX
- use logic for data handling
- use processor for control
use OASE or MGT and possibly
external clock recovery
Ether
MGT
and
SFP
Comm
Block
uni-directional optical link;
trigger, aux. control, clock
uni- or bi-directional optical link;
2.5 Gbps; data
CNTL
25 September 2005
DATA
CLK/TRG
Ethernet as main control interface
TRD Workshop, Cheile Gradistei, September 24-28, 2005
38
G1 Demonstrator: TNet
FEE
CNet

TNet
TNet proposal:


TNet link proposal:


BNet


reuse CNet components
just a CNet link
run unidirectional
possibly use optical splitters
TNet controler proposal:

just a DCB with other
firmware
HNet
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
39
TNet Controller and Setup
DCB
......
DCB
DCB
......
DCB
passive optical splitter
or as alternative
another layer of DCBs
Use DCB as TNet controller
- different firmware
- little extra hardware
- trigger inputs
- clock generator
DCB-TC
optionally
connect to ABB
25 September 2005
DATA
Trigger
hardware trigger inputs
and likely also 1-2 outputs
TRD Workshop, Cheile Gradistei, September 24-28, 2005
40
G1 Demonstrator: FEE
FEE

Get started with a


CNet

TNet

general purpose
chamber-mountable
modest density
ADC + FPGA based
board

Plausible parameters:

BNet



HNet
16 channels
100 MHz sampling
10 bit
If needed, a pipeline TDC
based FEE module to support
Time-over-Threshold based
systems might be added too
to archive
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
41
............
Processing
Block
Communication
Block
ADC
ADC
ADC
ADC
Front-End Board
for example 8 dual ADC's
for processing:
medium Spartan-3
small Virtex-4 FX
use OASE or MGT and possibly
external clock recovery
bi-directional optical link;
data, trigger, RoI, control, clock
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
42
OASE
MGT
MGT
GE
The minimal test bench setup
maximal 8 FEE
local clock from DCB
modest data date via GE
GE
GE
DCB
OASE
OASE
OASE
OASE
OASE
OASE
FEE
OASE
OASE
FEE
Micro Setup: 1 DCB + 1 PC
PC
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
43
OASE
OASE
OASE
OASE
OASE
OASE
OASE
OASE
MGT
MGT
GE
GE
OASE
GE Switch
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
Next to minimal test bench setup
few DCB
clock/trigger from DCB-TC
modest data date via GE
DCB
MGT
MGT
GE
GE
DCB
OASE
FEE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
OASE
FEE
OASE
FEE
OASE
OASE
FEE
Mini Setup: Ethernet based
PC
25 September 2005
OASE
MGT
MGT
GE
GE
GE
GE
DCB-TC
PC
TRD Workshop, Cheile Gradistei, September 24-28, 2005
44
PC
Controls
25 September 2005
PC
PC
GE
GE
PC
TRD Workshop, Cheile Gradistei, September 24-28, 2005
OASE
MGT
MGT
OASE
OASE
OASE
OASE
OASE
OASE
OASE
OASE
GE Switch
OASE
MGT
MGT
GE
GE
OASE
DCB
GE
GE
GE Switch
MGT
MGT
GE
GE
OASE
MGT
MGT
GE
GE
DCB
ABB
GE
GE
ABB
GE
GE
ABB
GE
FEE
OASE
FEE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
Full Setup: Low Performance
DCB
DCB-TC
Use GE for event building
DAQ – Event Building & Selection
45
PC
Controls
25 September 2005
PC
PC
HCA
GE
PC
TRD Workshop, Cheile Gradistei, September 24-28, 2005
OASE
MGT
MGT
OASE
OASE
OASE
OASE
OASE
OASE
OASE
OASE
IB Switch
OASE
MGT
MGT
GE
GE
OASE
DCB
GE
GE
GE Switch
MGT
MGT
GE
GE
OASE
MGT
MGT
GE
GE
DCB
ABB
HCA
GE
ABB
HCA
GE
ABB
GE
FEE
OASE
FEE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
OASE
OASE
OASE
OASE
FEE
FEE
OASE
OASE
FEE
OASE
Full Setup: High Performance
DCB
DCB-TC
Use IB for event building
DAQ – Event Building & Selection
46
DCB/ABB Prototype
leading but also
bleeding edge ...
Virtex-4 FPGA FX20
three MGT
on DB connector
designed for
12 V supply
one GE
on RJ45
one MGT
on SFP
connector
PCI express Interface
4 lanes → 1 GB/sec
25 September 2005
8 high speed connectors for
* OASE daughter board
* SRAM daughter board
* probe connector board
TRD Workshop, Cheile Gradistei, September 24-28, 2005
Gläß & Brüning
Mannheim
47
The End
Thanks for
your attention
We acknowledge the support of the European CommunityResearch Infrastructure Activity under the FP6
"Structuring the European Research Area" programme
(HadronPhysics, contract number RII3-CT-2004-506078).
25 September 2005
TRD Workshop, Cheile Gradistei, September 24-28, 2005
48