KM3NeT CLBv2
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
1
CLB Reset after configuration:
Solution in Firmware:
 Use End Of Startup (EOS signal in primitive
STARTUPE2) to trigger a timer that keeps system reset
asserted for 8.2 us. This ensures proper reset after
FPGA configuration

8,2 us
counter expires
System Rst
System Clk
(62.5 MHz)
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
EOS
~9 us PLL startup -> clock
~300 ns low after
configuration done
September 18, 2014
CLBv2, Vidyo
2
XST versus Precision
Revised StMachine and TDC files
 Cause: usage of VHDL libraries std_logic_arith and
std_logic_unsigned in combination with numeric_std.
 Std_logic_arith is an old Synopsys implementation and
should not be used since it is a non standard IEEE library
(as such XST and Precision both have their own
implementations!)
 Don’t use “initial values” when defining Signals and
Variables

Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology

Now synthesis via XST and Precision give equal results!
September 18, 2014
CLBv2, Vidyo
3
Monitoring channel Implemented!
stmachine (2)
IPMUX
Ch-0
Ch-1
Ch-2
Ch-3
TTDC
Hits-0
TMCH
TAES
Test
Hits-30
Clr
LM32
LM32
IRQ
timeslice_start
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
TDC
Busy
Set
DPRAM
Latch on
“timeslice_start”
Clear on
“timeslice_start”
Start register transfer on
“timeslice_start”
Added two registers in StMachine:
Monitor Memory Base
Monitor Memory Words (length: 1-256 words of 32 bits)
September 18, 2014
CLBv2, Vidyo
4
Flow control

Implemented state machine “Control Status” register
containing:
1.
2.
3.



TDC, AES and MCH enables
TDC, AES and MCH flushes
Monitor busy (for LM32_2nd polling)
Enable stops data forwarding at UDP packet boundaries
Flush forces a clear of all pending data in the pipeline
Both needed to facilitate a clean start and stop/pause.
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
5
Flow control on system level




Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
How to implement DAQ back pressure that ultimately
will overflow the front-end FIFOs?
Overflow will cause loss of bookkeeping!
We can signal this but action should be taken at system
level.
Currently:
◦ we are studying issues with the LM32_2nd Slow control that is
being blocked sometimes
◦ Solved? Yesterday => Rev 140917-00 (tagged in SVN since it is
flashed for the CLB for DOM-1)
◦ …backup slides “IPMUX”
September 18, 2014
CLBv2, Vidyo
6
Channel synchronicity


Studying channel synchronicity (TDC, AES and MCH)
showed failures.
Revise State Machine, TDC, AES and MCH front-ends.
◦ Ongoing

…backup slides
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
7
UDP Test packets generator
Implemented on the 4th IPMUX channel.
 Controllable via “UDP test packet control register”:

◦ Trigger (bit 0 OR DIP-Switch 0)
◦ Continuous mode (bit 1 OR DIP-Switch 1)
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
8
G-Board underway:
8
165
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
9
EMC and Temperature Test Reports


Test reports have been made for the assembly of a
CLBv2.2.1 and PBv2.3 in a mushroom
Temperature:
◦ https://docs.google.com/a/km3net.de/file/d/0B6xQ9LNuU6S8TUQ4T3dVXzZ6SWM
/edit
◦ https://isvn.ific.uv.es/repos/KM3NeT/CLBv2/trunk/fw/CLBv2_Design/clb/doc/Therm
alPictures/CLBv2_2_1_Plus_PBv2_3_Plus_Mushroom/KM3NeT_ELEC_WD_2014_
004_ThermalMeasurements_PBv2_3_CLBv2_2_1_Mushroom_PJ_Draft.docx

EMC:
◦ https://docs.google.com/a/km3net.de/file/d/0B6xQ9LNuU6S8RlUxQmpYUWdQUD
A/edit
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
◦ https://isvn.ific.uv.es/repos/KM3NeT/CLBv2/trunk/fw/CLBv2_Design/clb/doc/EMC_
Measurments/KM3NeT_ELEC_WD_2014_005_EMC_Measurements_PBv2_3_CLB
v2_2_1_Mushroom_PJ_Draft.docx
September 18, 2014
CLBv2, Vidyo
10
Switch tests


(Standard) WR switch needs Auto Negotiation on in
order to get the link up.
Switch specialist Tristan Seurink:
◦ Test one switch with one port used for TX and another port used
for RX
◦ Auto Negotiation off at CLB LM32_WR and at Switch ports
◦ Packets successfully received at both ends. (ping!)
2 separate switch ports
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Data to CLB
Data from CLB
September 18, 2014
CLBv2, Vidyo
11
Switch tests details

Juniper (type)
◦ Did the job!

Arista (type?)
◦ Not able to disable Auto Negotiation

Arista (7150-64)
◦ Not able to unlock SFP identification.
Need dummy 8/10B coded stream to
get switch (Tx) port up
(the Rx CDR has no lock and the
port is signalled down)
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Data from CLB
Data to CLB
September 18, 2014
CLBv2, Vidyo
12
Switch tests next steps
Does this scheme work over multiple levels of switches?
 Try other switch vendors

Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
13
Todo list



Fix State Machine, TDC, AES and MCH (David, Antonio, me)
Re-arrange MCH channel (proper time-tag; not lagging one time-slice
=> me)
Flow-Control study:
◦ Fix Slow Control => thoroughly test yesterdays solution
◦ define system level actions when frontends overflow


ICAPE2 / multiboot / watchdog / golden image tests
Software: Communication interface between LM32_2nd and
LM32_WR
◦ SFP readout and setting (loopback, PRBS, wavelength tuning)
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
◦ Auto negotiation control

Ethernet flow control (received pause frame implementation in the
CLB)
September 18, 2014
CLBv2, Vidyo
14
Backup slides
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
15
Data Channel Synchronicity
TimeSlice_Start
TimeSlices
State Machine Enables
TDC FIFO
AES FIFO
MCH
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Example:
TDC has no data and thus no header is composed (state “waiting” since fifo is
empty). Fifo gets non-empty at the first TimeSlice_Start; now a (yellow) header
is composed.
AES has data as soon as it is enabled. Therefore is starts to compose a (red)
header (state “waiting” -> “header” when fifo non-empty)
MCH only starts to compose a (yellow) packet when TimeSlice_Start is asserted
September 18, 2014
CLBv2, Vidyo
16
Data Channel Synchronicity
TimeSlice_Start
TimeSlices
State Machine Enables
TDC FIFO
AES FIFO
MCH
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Example:
TDC and AES have data as soon as they are enabled. Therefore they start to
compose a (red) header (state “waiting” -> “header” when fifo non-empty)
MCH only starts to compose a (yellow) packet when TimeSlice_Start is asserted
September 18, 2014
CLBv2, Vidyo
17
IPMUX
Stream
Selector
TDC
AES
UDP Packet
Buffer
Anyone
has
data?
MCH
FIFO 512
FIFO 512
FIFO 512
FIFO 512
Priority!
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
EOD
1) Header will be transferred
first, tail of packet is underway
2) IPMUX Stream selector
locks on TDC stream,
transfers header and waits…
3) Until End Of payload Data
(EOD) marker is passed (at
the end of a time-slice)
Lm32_2nd

If time slices are chosen to be “long” (1 second) then IPMUX channels keeps
lock on the selected stream for that amount of time.

Even the high priority of the LM32 doesn’t help now!

Funny, but with high rates each stream is served as expected!
September 18, 2014
CLBv2, Vidyo
18
IPMUX-2
Stream
Selector
TDC
AES
UDP Packet
Buffer
Anyone
has
data?
MCH
FIFO 8192
FIFO 8192
EOD
No matter the size of the FIFO
the problem persists
FIFO 8192
FIFO 8192
Priority!
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Lm32_2nd

Increase size of input FIFOs…

This does solve the key problem but it reliefs other input channels from “full”
situations.

More is needed…
September 18, 2014
CLBv2, Vidyo
19
IPMUX-3
Stream
Selector
TDC
E
O
D
FIFO 8192
E
O
D
UDP Packet
Buffer
Anyone
has a
complete
packet to
transfer?
AES
FIFO 8192
FIFO 8192
FIFO 8192
MCH
FIFO 8192
FIFO 8192
FIFO 8192
Priority!
EOD
FIFO 8192
Lm32_2nd
1) Since “data” FIFO can now
hold at least one packet…
2) …We can keep track of any
packet tails (EOD) present in
a parallel “EOD” FIFO (you
actually only need the FIFO
flag structure, no need to
store any data!)
3) Now the Stream selector
only initiates an action when a
complete packet can be
transferred
Peter Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
September 18, 2014
CLBv2, Vidyo
20