Commissioning of the new electronics and
online system for the Super-Kamiokande Experiment
S. Yamada1, K. Awai1, Y. Hayato1, K. Kaneyuki1, Y. Kouzuma1,
S. Nakayama1, H. Nishino1, K. Okumura1, Y. Obayashi1, Y. Shimizu1,
M. Shiozawa1, A. Takeda1, Y. Heng2, B. Yang3, S. Chen2, T. Tanaka4, T.
Yokozawa1, Y. Koshio1, S. Moriyama1 for the Super-Kamiokande
collaboration, Y. Arai5, K. Ishikawa6, T. Uchida7 , A. Minegishi6
1, Institute for Cosmic Ray Research, University of Tokyo
2, Department of Engineering Physics, Tsinghua University
3, Department of Physics, Seoul National University
4, Solar Terrestrial Environment Laboratory, Nagoya University
5, High Energy Accelerator Research Organization (KEK )
6, Iwatsu Test Instruments Corporation
7, Department of Physics, University of Tokyo
1
1, Super-Kamiokande detector
Water Cherenkov detector
13,000 PMTs are equipped in
50,000 tons water tank
ν
Cherenkov
light
Charge and timing information of
PMT hits are recorded
©Scientific American
Obtain Cherenkov
Ring images
List of physics topics of the SK detector
- Atomospheric neutrino oscillation : Δm23, θ23
- Solar neutrino oscillation : Δm12, θ12
- Neutrino from accerelator(T2K) : search for θ13
- Search for neutrino from supernova (burst or diffused)
- Proton decay search
2
2, Motivation of upgrading DAQ
■ new Front-end Electronics to achieve Good Data Quality
- Low Noise, good timing and charge response
- Wide Charge Dynamic Range
■ Process all the hits from PMTs in the online system
• Previous system
Hardware trigger module
counts hit # of PMTs
Exceeds
threshold
Only Triggered hit data was
sent to online system
Issue Event Trigger
Higher rate
Complex trigger
• New system
All the hit data is sent to the online system → event selection is done by software.
・ Lower energy threshold for solar neutrino measurement
・ Use complex trigger to reduce background for Relic Supernova neutrino search
→
New system needs high transfer rate and data processing speed.
3
3, New Electronics
QTC-Based Electronics with Ethernet
(QBEE)
Network
Interface Card
PMT
signal
Ethernet
Readout
60MHz Clock
TDC Trigger
Calibration Pulser
QTC
TDC
FPGA
Features
• 24channel input
• QTC (custom ASIC)
• Charge measurement
• wide dynamic range
(>2000pC)
• multi-hit TDC (AMT3)
• Data is sent to Online system
via Ethernet
• External 60MHz clock is used
for synchronization with other
Qbees
• On-board pulsar for charge
calibration
• Low power consumption
( < 1W/ch )4
Analog performance of QBEE
Dynamic Range
Charge Resolution
ADC count (– Pedestal)
QBEE
previous Electronics
saturated @ ~600pC
0
600
1200
1800
2400
input charge (pC) (1p.e.=~ 2 pC)
No saturation over 2000 pC !!
(RMS resolution)/(input charge) [%]
10
QBEE
Prev. Electronics
Large
Medium
Small
1
0.1
10-1
1
10
102
103
input charge (p.e.)
Good charge resolution
~5% @ 1 p.e.
< 2% @ > 3 p.e.
5
Data-readout via Ethernet
QBEE throughput from analog pulsar
Input to a readout PC
throughput rate (MB/s)
Custom Network Interface Card
12
10
8
6
4
2
0
MAX :
11.8MB/s
(~95Mbps)
Requirement
0
4
8
12
16
20
input data rate (MB/s)



TCP/IP firmware (SiTCP)
and interface logic are
implemented on FPGA
IP address is set by dip switch
32MB SDRAM


Required data transfer speed :
(PMT dark noise) 10kHz x 6byte x 24ch
= 1.5MB/sec/board
Fast enough. Reaches the
theoretical limit of 100BASE-TX !!
6
4, New Online System
24PMTs
30QBees
QBee
QBee
.
.
.
.
.
.
13,000
PMTs
QBee
QBee
1hit cell
= 6bytel
(ch, T, Q)
Ethernet
Sorting
Data from 30Qbee
Merger
Software
trigger
Software
Merger
trigger
Merger
Software
trigger
550
20
10
QBees
Front-end PCs
Merger PCs
Organizer
Front
End PC
Front
End PC
Front
End PC
QBee
Event builder
Recorded
Data:
9MB/s
typical
Disk
Offline
analysis
- LINUX Multi-threaded softwares are running on Online PCs equipped with
4 CPU cores.
- From electronics to offline disk, data is transferred using TCP/IP protocol,
with commercial Ethernet network equipments.
7
Function
To collect data From 30 Qbees
and sort the hit cells in time order
To make use of multi core CPUs,
data in different time blocks are
sorted in parallel by multithreaded functions
↓
effective for the improvement of
throughput
～550
QBs
20
10
1
Front Front Orga
End End nizer
PC
PC
PC
offline
Performance of Front-end PC
Performance with dummy data
Front-end PC can handle up
to 15kHz dark rate
( PMT dark rate = 4 ~ 5kHz )
8
Data flow manager
To Distribute load of event building, data flow manager controls data flow
between 20 front-end PCs and 40 Merger processes.
Data flow manager
Request rate from data flow
manager to front-end PCs vs
Number of Requests issued at
the same time
FEPC
FEPC
Block
3
Switch
Block
3
Block
1
Block
1 Merger
Block
2
Block
2
PC
Required rate
Merger PC
To avoid network congestion, data flow
manager issues plural number of requests at
one time. →
Effective for distributing the destination of
the data and improvement of throughput
Number of Requests at one time
9
Software trigger for event building
From front-end PCs
Software trigger
Merged and
Sorted data
200ns
200ns
Number of hits exceeds the threshold,
“Software”
event
“Software”
event
Send
downstream
Several triggers can be applied for each interest in physics.
(different threshold and gate width)
Basic hitsum
triggers
Trigger type
* Super Lowe
* LowE, HIghE
• T2K trigger
beam line)
* External
( lower threshold, for solar neutrino analysis )
( higher threshold, for Atmospheric neutrino)
( with beam spill information sent from T2K
( to synchronize a calibration light source) 10
5, Installation and Basic performance
Replacement work was done for 2weeks in the end of Aug. 2008.
After the installation of new DAQ system, it started working since Sep. 6.
DAQ system is stable now and 24hrs operation is ongoing.
Installed new DAQ system in an elec. Hut
Trigger rate for 12hrs
Front-end PCs
and network switches
Qbees
11
Charge and timing response of Qbee with the SK detector
Charge and timing responses were measured by putting LED or Laser light
source in the SK detector.
Single p.e. Distribution
Timing Resolution
QBEE
Previous Electronics
QBEE
Prev.
Electronics
- High S/N for single p.e. detection
- Good agreement with single p.e.
distribution by previous electronics
Comparable timing resolution with
previous one
~2 nsec @ 1 p.e.
12
~0.5 nsec @ 100 p.e.
～550
QBs
Change the threshold of the software trigger
and measure the efficiency of Online DAQ’s data processing.
process data w/o loss
20
Front
End
PC
10
Mer
ger
PC
1
Orga
nizer
PC
offline
Performance of the Online system
- 12kHz of
Event Trigger rate
can be processed
without data loss,
which is much
larger than the max.
Trigger rate in the
previous System
(~4kHz )
- Bottleneck is the
disk write on the
organizer PC
( max. ~50MB/s)
13
7, Other performances with new DAQ system
Capability of measuring nearby Supernova Burst
Flush rate [MHz]
Light pulser
Nearby-supernova burst →
Need to handle very high rate events
2
1.2
Pulse light is injected to SK
tank to mimic nearby
supernova burst (Light pulse
rate follows step-like function ).
- 100 times improvement from
the prev. system
1
5
10(s)
# of events/10s
- 6Mevents/10s can be processed
w/o losing SN burst data
(Bottle neck is online system)
0.2
New
Prev
Distance between earth and SN (kpc)
14
Measurement of Neutron captured events by software “neutron trigger”
Neutron source : ( Am/Be source + BGO ) in SK water tank
Prompt signal : Scintillation light of
4.4MeV prompt gamma from BGO
Delayed signal :
2.2MeV-γ from neutron capture in water
( It cannot be triggered as a prompt signal
because its energy is too low.)
prompt
Delayed (neutron trigger)
?
time
Neutron capture time in water
To search delayed signal, issue
“neutron trigger “ after the normal trigger
and save the data.
Condition for neutron trigger:
- neutron trigger gate width = 800 us.
- Need prompt signal
- No signal in outer cosmic Muon
VETO detector.
This trigger will be used to reduce b.g. for relic-supernova neutrino search
15
T2K (neutrino beam from Tokai to Kamioka) trigger

By using GPS data of SK and Tokai sites, PMT hits within ±500μs are
recorded as T2K triggered event ( 1st priority in software trigger )
T2K triggered
data
check
From Tokai
Tokai-GPS
Data
Offline
triggered
Disk T2K data
3rd Reduction
SK-GPS data
HITSUM
Triggered data
2nd Reduction
Merger
+
Software
trigger
1st Reduction
Every hit
data
- In Apr., 1st neutrino beam was produced at J-PARC in Tokai.
- Still beam is in commissioning status, but in SK DAQ, T2K
trigger and reduction are now applied to SK data and being
checked.
Online trigger monitor at SK
Spill information
coming from J-PARC
16
8, Summary
* Newly developed Electronics and online system for
Super-Kamiokande was installed and started running.
** With new electronics, larger charge dynamic range,
lower power consumption, larger data transfer speed and
good charge and timing resolution were achieved.
** Succeed to handle every hit data from the SK detector
using Ethernet based online system by distributed data
processing.
** DAQ is stable and continues taking data. Physics
analysis is now ongoing.
17
18
Develop a new Calibration source for SK using external trigger
Qbee
(Ni captured)
Fission
detector
g
n
Special
Cable hit as an
external trigger
n
g
events
- We have used gammas from neutron capture by Ni as a calibration
source in SK
- In a currently developed calibration source, signal from a fission
detector is used as a external trigger for non-biased data taking.
- Hit data in the following 500us window is selected by the software
trigger to obtain neutron-captured gamma data from the source.
Cf
Ni+PE+Epoxy ball
0
time
SK
100
200 0 300 400
500
Time from the nickel trigger
(micro sec)
19
T2K (neutrino beam from Tokai to Kamioka) trigger
How to process the T2K gps data
PMT hits within ±500μs are
recorded as T2K triggered event ( 1st
priority in software trigger )
Tspill+TOF
SK PMT hits

SK
-500ms
T
+500ms
Time stamp
GPS
receiver
Store as “T2K DST”
Transfer route from Tokai to SK
Kamioka Kenkyuto
Off-line
sw
Neutrino ( J-Parc )
Strage
Front-end
GPS recv.
Merger PC
SK
Reflective
memory
VPN in Sinet
Underground
DMZ
Edge sw.
GPS
recv.
FW
Magnet control
etc
Near Detector Switch
(routing)
Beam DAQ
20
Data flow manager (cont’d)
To avoid network congestion, data flow manager issues plural number of requests
at one time. -> effective for distributing the destination of the data and
improvement of throughput
Example)
Number of entries
Number of entries
in request queue = 1 in request queue = 3
Request :destination
20 FEPC send
data to the same
PC simultaneously.
Req10 : MGR0
Req11 : MGR1
Req12 : MGR2
Req13 : MGR3
Req14 : MGR4
Req15 : MGR5
Depending on
the progress
in each FEPC,
destinations are
distributed over
3 PCs.
Required rate
time
time
Req10 : MGR0
Req11 : MGR1
Req12 : MGR2
Req13 : MGR3
Req14 : MGR4
Req15 : MGR5
Request :destination
Request rate from data flow manager
to front-end PCs vs Number of
Requests issued at the same time
21
Specification of QTC
Discriminator
Self
Number of channel
3
Procesing speed
About 900nsec / cycle
Chage integration gate
400nsec
Gain stage and gain
3 （1:7:49）
threshold
-0.3 ~ -14mV（S range）
Dynamic range
0.2 ~ 51pC（S range）
~ 357pC（M range）
~ 2500pC（L range）
Charge resolution
~ 0.2pC（S range）
linearity
Less than ±1%
Timing resolution
0.3nsec（2pC input）
0.2nsec以下（>10pC input）
Power consumption
< 100mW / channel
CMOS process
0.35 um
package
100pin CQFP
22
Custom ASIC QTC
CAL
QTC Block Diagram
Charge Integration
Delay
(Low Pass Filter)
DAC
+
OUT
Input
+
OUT
Disharging
Start
-
Preamp.
Charging
Start
To All Channels
Vth
Discriminator
DAC
Output
-
Gate & Output Generator
OUT
+
Discriminator
QTC
DAC
timing
charge
3ch input
DAC
3ch x 3gain
output
HIT
Gate Generation
 built-in discriminator
 400nsec charge gate
~ 1msec / cycle
 3 gain stages (ratio 1:7:49)
 Only the data from the proper
gain stage are left by FPGA
23
RMS resolution (nsec)
Timing Resolution
3
QBEE (QTC)
ATM
20-inch PMT
resolution
2
1
0
10-1
1
10
102
input charge (p.e.)
Good timing resolution
for 20-inch PMT signal
103
residual from linear fit (nsec)
QTC performance (Timing
measurement) T : 0.52 nsec / count
Timing Linearity
QBEE (QTC)
ATM
20
10
0
10
1000
3000 5000 7000
hit timing (nsec)
Perfect timing linearity !!
24
Breakdown of throughput speed
QTC (x8)
WR speed
180Mb/s
(1.3MHz/ch)
TDC (x4)
PMT
signal
DSM (x4)
(FPGA)
SIC
(FPGA)
FIFO
(1.5Mb)
L1 buffer 256W
1.1 MHz/ch (900nsec
gate win.)
~300kHz/ch
1.6MHz/ch
(simulation)
Ethernet (SiTCP)
11.8Mbyte/sec
(82kHz/ch)
TKO transfer
~15Mb/s
(~100kHz/ch)
SDRAM 32Mbyte
Daughter Board
25
Data transfer latency
26
27
MCLK output specification


Output [ 2 pairs in 1 UTP cable ]

(1,2) pair
60 MHz clock

(5,6) pair
Trigger + 32 bit event # + TDC reset

(3,4) and (7,8) pairs
(not used, for future unification of CLK/TRG and
100BASE-TX)
Spec. of serial signal [ 1 bit = 1 clock, total 38 clocks = 633 nsec ]
Start at a negative edge of the clock
60 MHz clock
Serial signal
Header (always 1)
Trigger (Narrow/Wide + Pedestal + Split)
Trigger on/off + TDC reset on/off
• Trigger w/o TDC reset
(10)
• Trigger w/ TDC reset
(11)
• TDC reset only (No Trigger)
(01)
32 bit event # (MSB  LSB )
28
Clock jitter and clock/trigger phase check
Clock jitter measurement
yellow : clock
blue
: trigger
Clock/Trigger phase check
16.67nsec
~4.3nsec
trigger here
measure the variation
of zero-crossing point
jitter = 26 psec RMS
requirement (by AMT)
< 30 psec
no problem
duty cycle :
Tlow/Thigh ~ 1.24
29