Wiener Setup
CAEN Setup
(see pag. 8)
Photos: beginning of August 2015
Wiener
PS - 6U
Active Heat
Exchanger
Empty unit
Pavia’s
fan tray
Space for
custom fan
Tray & Heat
exchanger
Wiener Bin
Wiener Bin
Wiener’s
HB fan tray
3.3 V PS
Inactive Heat
Exchanger
PS for Pavia’s
fan
Detailed view of the Pavia’s fan tray
Space for
custom fan
Tray & Heat
exchanger
CAEN PS 9U
1. Report on cooling tests for the Wiener Setup: single
HB fan - June 15-19 2015
- S. Citraro, P. Giannetti, A. Lanza, H. Nasimi
We measure the temperatures on old boards used in the past in the Vertical Slice tests, the AMBslim shown
in the figure below. We used a single fan, the Wiener HB placed below the bin, running at 5000 rpm.
Sensor’s
temperature
Temperature
on pins with
termocamera
Temperature on the centre of the package w termocamera
AMB1
UP
ML
ML
MR
MR
AMB1
DN
FL on chip
We use AMchip03 built for CDF. The consumption is more than 2.5 W per chip when all the inputs swap at
each clock cycle. Most of the power is provided from 48 V (the core) part of it from 3.3 V (the I/O).
We have 2 types of boards: (a) type 1 with 96 AMchips for a total consumption of 280-240 W, type 2 with
64 AMchips for a total consumption of 185-190 W. For type 1 AMBslim the two LAMB mezzanines near the
front panel have chips on both sides (64 chips near the front panel, 32/LAMB) while the two LAMBs on the
rear side, near the VME connectors, have chips only on the top side (32 chips in total, 16 chips/LAMB). We
perform 2 types of measurements: (a) we use temperature sensors glued on the LAMB PCBs, as you can
see in the figure above. The most important are the ones in the RED circles on the top of the board, where
less cold air arrives. In particular the one in the left top corner is predicted by the T-simulation performed
by IMEC to be the hottest corner of the board, so we focus our attention in particular on it. (b) We also
measure the temperature on the top left corner AMchip using a termo-camera from the front when the
front panel is missing (T on chip pins) and with a Z-module with a hole corresponding to the center of that
chip (T on package). The T on package is ~20o higher than T on pins, the T on pins is ~6 o higher than PCB T.
Slots
1-5
Slot Slot Slot Slot
6
7
8
9
Slots
10-13
Slots
14-21
Load
Boards
286 235 183 190
W W W wo
3v3 1V2 1V3 name
We build a test setup using 4 AMBslims, one near the other. On the left are the two type 1 boards on the
right the two type 2, as shown above. We fill the rest of the crate with old CDF boards (black in the figures)
and Load Boards full of resistors (yellow in the figure), however no one of them is connected at power, they
are there just to produce the correct resistance to the air. The only boards connected to power are the 4
AMBslims. The crate measures a total current of 15 A on 48 V and 48 A on 3.3 V.
We want to test all the slots of the crate in the central region since we know from the Temperature
Simulation performed by IMEC that they are not equally cooled by the fans
(https://indico.cern.ch/event/387419/material/4/0.pdf ). We test all the slots between 7 and 14 placing
the red board (1v2) on each of these slots and repeating each time the measurements. The 4 boards are
moved of one slot at time on the right, maintaining the relative position: the violet board, the one in slot 6
in the figure, is 3v3 and has the maximum consumption of 286 W. In the middle (slot 7 and 8) are the two
boards with sensors, a type 1 (1v2, red, on the left) that has a consumption of 235 W, and a type 2 (1v3,
blue) that is on the right and has a consumption of 183 W. Finally the board in slot 9 has no sensor and a
consumption of 190 W.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
better
1v2 240W
56o
1v3 185W
16 18 20
15 17 19
21
Worse
for front
73,6o
68,5o II 63o 51o
60o
64o 66o
67o 44o
56o
o
o
o
o
o
44 48 49 50
39 40o 42o 47o
Slot 6
Slot 8
Slot 7
Temperatures on sensors FL
The figure above shows clearly that different slots are not equivalent. In the figure are reported all the FL
sensor measurements from slot 7 to slot 14 for the red board (1v2 type 1) and from slot 8 to slot 15 for the
blue board (1v3 type 2). Simulation predicted that the highest temperatures would have been in the left
top corner of the slots that overlap the metal between 2 wheels, like for example slot 8 (see the photo on
the right). The measurements are reported below the figure for each slot and reveal a different behaviour.
The worse temperatures correspond to the left part of the wheel and the maximum is near the centre of
the wheel (slot 10 in the figure).
The red board has systematically higher temperatures since in the left top corner are 2 chips for a total
local delivery of power of 5 W roughly. The blue board has half chips and half consumption in that area.
However the two types of boards have the same trend showing the maximum temperature at the center of
the wheel. In the slots of the half wheel where the FL sensor has lower temperatures, the FR sensor has
higher values, while in the half wheel where the FC sensor becomes hot the FR sensor has lower values.
Looking at the photo on the right of the figure above, you can see that the wing of the wheel as an angle,
whose effect during the rotation could generate what we observe. Rotation in the right part of the wheel
seems to push air towards the front of the board while rotation in the left part of the wheel seems to push
the air towards the rear. We know that the IMEC T-simulation did not take into account this effect because
only the vertical component of the airflow was used as input to board level temperature simulations. They
did not observe any asymmetry in the flux of air with respect the centre of the wheel, along the slot.
Pavia personnel is starting now the simulation of the air flux in the crate, they will try to use all components
of the air velocity.
80
70
60
50
one fan FL
40
one fan FC
30
20
10
0
Slot7
Slot8
Slot9
Slot10
Slot11
Slot12
Slot13
Slot14
Slot15
The plot above summarizes the behaviour of the temperature as a function of the slots for both FL and FC
sensors.
The measurement in slot 10 has been repeated swapping the two external boards in the quartet of
AMBslims. This swap puts the tester board in the middle of the two type 2 boards, so the tester board has
less heat on its left. In this condition the measured temperature is 5 degrees below.
Considering that the centre of the package is 26 o above the sensor, the red board has the top-left AMchip
package over 85o in all the slots from 8 to 12 (only 6, 7, 13 and 14 are below) while for the blue board
(more similar to our AM06 case) the temperature is always below 80o. The blue board is more similar since
it has AMchips on a single face of the LAMB, 16 chips per LAMB. The power dissipated by AMchip03 with
all his input swapping at each clock cycle is very similar to the heat produced by AM06. However the
AMchip package in the old board is very different from the AM06 one, since it is not a BGA (it is a PQFLP)
and it dissipates much less, this is why the temperature difference between the package and the PCB is so
large (~26 degrees). AM06 simulation shows that the new package (flip chip with heat slug) is extremely
efficient dissipating heat and the difference of temperature between the silicon, the package and the PCB is
expected to be few degrees. So we expect on the new board a similar temperature on the PCB and the
package, being lower of the package temperature we measure now.
Finally we measured again the temperatures putting the 4 boards in slots separated by empty slots, as
shown in the figure below (white means “empty”):
Slots
1-6
7
Slot Slot Slot Slot Slot Slot Slot Slot
8
9 10 11 12 13 14 15
286
W
3v3
235
W
1V2
183
W
1V3
Slot
16-21
190
wo
name
The temperature in slot 10 (1v2 board, type 1) was 63o more than 10 degrees below what measured when
the boards were into contiguous slots.
2. Report on cooling tests for the Wiener Setup: double
fan tray- July 20-24 2015
P. Giannetti, A. Lanza, P. Kalaitzides, I. Maznas
We have inserted a second fan tray on the top of the bin and we performed equivalent measurements in
this new condition. It is a custom fan produced by Agostino Lanza at Pavia. Since the temperature
simulation showed a weak action of the HB fans on the front of the crate, very near the front panels, we
thought that moving the fan centre to the front of the crate would have been beneficial. So the first raw of
3 fans of the custom tray are prominent outside of the rack, on the front, to better cover the points found
hot in the T-simulation. The rows of custom fans are not aligned with the HB rows.
Each row in the custom tray can work at independent speed. We set all of them at the maximum speed,
above 6000 rpm. The HB fan instead was running at 5000 rpm.
The plot below shows for the FL sensor (the one on the top left corner of the board )a comparison of the
measurements performed with one and two fans working in the same conditions for all the other system
parameters.
80
70
60
50
one fan FL
40
two fans FL
30
20
10
0
Slot7
Slot8
Slot9
Slot10
Slot11
The results are good because the temperature seems to be much more flat as a function of the slot # and
the value in slot 10 (the maximum for measurements with a single fan) is decreased of 26o.
The effect of the second fan on the FC sensor (placed on the top centre of the board) is instead negligible
(see the plot below), but in this case the temperature was acceptable already with a single fan.
80
70
60
50
one fan FC
40
two fans FC
30
20
10
0
Slot7
Slot8
Slot9
Slot10
Slot11
In conclusion with two fans the temperature is slightly higher at the centre of the board (~60o in the centre
and ~50o in the top left corner), is much more uniform as a function of the slot #, and is acceptable even if:
1. The HB fan was running at 5000 rpm
2. The Heat Exchanger below the bin was not active
3. Two extra PS where placed below the bin, one to provide the 3,3 V to the old boards and one to
give power to the custom fan.
We could not repeat the measurements into all the slots measured with a single fan since one of the 4
boards, 1v3, stopped to work suddenly. We tested more slots using only 3 boards instead of 4. We expect
the results are similar, since the test board (1v2) was still in the middle of the 3, the only change was 190
Watts on the right instead of 183 Watts. The plots below show the behaviour of the temperature in other
slots, for both the FC and FL sensors.
60
50
40
Two fans FL- 4 boards
30
Two fans FL - 3 boards
20
10
0
Slot10
Slot11
Slot12
Slot13
Slot14
Slot15
Slot16
70
60
50
40
Two fans FC - 4 boards
30
Two fans FC - 3 boards
20
10
0
Slot10
Slot11
Slot12 Slot13
Slot14
Slot15
Slot16
The temperatures on the board sensors are never above 60o. This corresponds to temperatures always
below 85 degrees on old PQFP packages, when the local heat dissipation is 5 Watts (two AM chips one
on TOP and one on bottom of the PCB) in a PCB area of 30×30 mm2.
3. Report on cooling tests in CAEN rack: two custom
fans - November 2015
- A. Lanza, S. Citraro, P. Giannetti, I. Maznas
The
52
51
50
49
48
47
46
45
44
43
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
CAEN
PS
has
been
installed
and
power
has
been
provided
to
the
bin.
TURBINE
HEAT EXCHANGER
Fan tray
for cables
9U VME bin
(AM PU)
Fan tray
HEAT EXCHANGER
External PS unit
HEAT EXCHANGER
Fan tray
for cables
9U VME bin
(AM PU)
Fan tray
Air deflector
The setup has been adjusted:
1.
The 3.3 V PS has been installed to power the old AMBslim
boards;
2.
The two custom fan trays (one on top of the bin and one below
the bin) have been connected to CAEN PS).
other services
We repeated the temperature measurements in the same conditions described for the Wiener
rack:
1. Tree AMBslim boards were used putting the one with sensors (1V2 235 W) in the slot 10
(the worse slot in previous measurements), the board 1v3 (280 W) in the slot 9 and the
third one (190 W) in slot 11.
2. The two custom fans were running at 6800 rpm.
3. The smaller fan to be inserted below the bin, inside it, was used in two positions: inserted
as much as possible (22 mm outside the rack) or aligned with the fan on the top fan (42
mm outside the rack). This was done to examine two different positions of the fans with
respect the AM chips aligned on the LAMBs.
These are the results of the T measurements on the board 1V2:
Two fan units - the bottom one displaced by 22 mm
70,0
65,0
60,0
55,0
50,0
°C 45,0
40,0
35,0
30,0
25,0
20,0
Slot 10
FL
TOP
FC
TOP
RC
TOP
RR
TOP
ML
TOP
MR
TOP
FL
FC
BOT BOT
RC
BOT
RR
BOT
ML
BOT
MR
BOT
Position
Two aligned fan units (both 42 mm outside the rack)
65,0
60,0
55,0
50,0
45,0
°C
40,0
35,0
30,0
25,0
20,0
Slot 10
FL
TOP
FC
TOP
RC
TOP
RR
TOP
ML
TOP
MR
TOP
FL
FC
BOT BOT
RC
BOT
RR
BOT
ML
BOT
MR
BOT
Position
The results observed in the Wiener rack are reproduced: the maximum T is in the FC sensor and it
can be kept below 60 degrees.
AM06 Slow corner chips: tests at USA15 – February 2016
-
A. Lanza, G. Volpi, S. Citraro, N. Biesuz
The AMBSLP_V3 has been used with 4 LAMB_v3s assembled with AM06 chips to measure the
temperature on the final LAMBs. The first figure below shows the old motherboard (3 GE DC-DC
converter of 40 A each for 2 LAMBs) with 4 LAMBs, while the second figure shows the position of
the sensors on the top of the LAMB where the temperature is expected to be higher.
Temperature sensors
The power for the core of AM06 chips was set 1,2 V, a conservative value since the slow corner
chips run correctly at 1,15 V and the fast chips works fine at 1 V. It is probable that the typical
chips will be able to run at substantially lower voltage.
The bin was loaded with the same load boards full of resistors described in section 1 (see figures
below).
The AMBSLP_v3 under test was in slot 15 and had on the right in slot 16 a load board, while on the
left the load board was in slot 13, not the nearest one, for mechanical constraints (the photo on
the right shows the details in the relevant slots, with the AMBSLP in the center. This time the load
boards where powered and were able to produce a power consumption of 2,7 kW. An AUX card
was also in the rear cage of the bin to send data to the AMBSLP. The load boards, however, are
not able to exchange heat with the air as AMBOARDs, so their use in the test is not optimal, we
should repeat the test with 3 AMBSLPs full of AM chips (not available now) in contiguous slots
(choosing the less favourite slots 10-11) to make final conclusions.
We were running a test sending hits in input continuously in the single AMBSLP (loop mode for
events of ~450 hits average size), only a 2,5% of idle words was inserted at each event end. Bits
were flipping at 50% level, as expected in normal condition. Hits received by the AM bank are the
main cause of consumption.
The temperature in the fan above the CAEN PS and below the bin was not uniform: the right half
was much warmer than the left one, due to the fact that only one bin is in the rack and half PS is
not providing any current. However the AMBSLP_V3 was above the warm half fan. The fan was
running at 80% of its maximum capability.
The water in the heat exchangers was turned off, as in other tests described the section above,
however for the cooling test this is a conservative condition. The air temperature at USA15 in that
area is ~16 degree.
The temperature in the upper LAMBs was roughly 6-8 degrees higher than temperatures in the
lower LAMBs. The plot below shows the two LAMB2 sensor temperatures (the left and the right)
measured at 4 different times, roughly each 5 minutes. We see that very quickly the system reach
the plateau and the sensor on the left has higher temperature than the sensor on the right.
LAMB2_Tsens_right
LAMB2_Tsens_left
80
80
70
70
60
60
50
50
40
LAMB2_Tsens_left
40
30
30
20
20
10
10
0
LAMB2_Tsens_right
0
1
2
3
4
1
2
3
4
80
70
60
50
40
30
20
10
After ~15 minutes
At the beginning
0
The plot above shows the temperatures of the four sensors on the top of the board (two inside
LAMB2 on the left and two inside LAMB0 on the right. The LAMB on the left, near the front panel,
ha slight higher temperatures, however even the hottest point is below 70 degree and the
temperature inside the package is few degree above the PCB temperature.
Observed features about the new PS that have been communicated to
CAEN:
(1) The fan turn-off should have an independent switch or a delay with respect the PS turn-off to allow
the turn-off of the fan after the power of the bin is removed, when the temperature is not any
more too high. Now a single power off is used for both the bin and the fans. May be also at turn on
the PS could be delayed few seconds with respect the PS.
(2) The measurement of the current on the display is not stable, it fluctuates few A, even for the 48 V.
The voltage on the backplane instead is stable and the tests on the AMBSLPV3 were successful, so
the problem is certainly on the current monitoring.
CAEN is going to examine the problem and solve them for the next production of 2 pieces.
ADDITIONAL CHANGES FOR THE NEXT CAEN PSs:
1. We don’t need anymore 120 A on 5 V, since SSB will powered on 48 V taken by J0 30 A is enough
2. We require to change the fan power from 30-35 A @24 V to 20 A @48 V to be able to use more
powerful cooling if necessary.