Operation of the CAEN SY127 High Voltage System
P. Hanlet, Illinois Institute of Technology
August 11, 2010
Abstract
This document describes user controls and monitoring systems for the CAEN SY127 high
voltage crates. This is tied to the Controls and Monitoring systems using EPICS. No prior
knowledge of EPICS is required, and the first section can therefore be skipped; however
Be sure to read Section 1.2.1!!!
1
1.1
Introduction
Introduction to EPICS
EPICS stands for Experimental Physics and Industrial Controls Systems. It is an open source
software package which is developed and supported by an international community.
EPICS provides a uniform means of connecting to a wide variety of hardware devices, via device
drivers which are connected to EPICS support drivers, as shown in Fig. 1.1. This connection allows
the developer to write IOCs (Input Output Controllers) in which Process Variables, “PVs”, or
“records”, are defined which represent attributes of the device to which the IOC is connected.
Examples of PVs might be current, voltage, temperature, pressure, level, etc. Each PV has a list
of fields which further describe the record; examples include units, alarm limits, security levels for
access, data type, frequency of reading, etc. The IOCs themselves are processes which run on VME
crates, linux pcs, windows pcs, etc.
User
Access
CA
Ctrl
CA
ALH
Mon
CA
Arch
Ctrl
CA
Mon
LAN
IOC
(VME)
Process
Variables
flow
heater
temp
valve
vacuum
level
IOC
(VME)
IOC
(PC)
Drivers
IOC
(VME)
...
Magnet PSU:
Hardware Cryo:
pump, chiller, ... I, V, ramp, ...
...
IOC
(PC)
HV:
I, V, ramp, ...
Figure 1: A typical EPICS system showing physical devices connected via hardware drivers to
EPICS IOCs, which in turn are connected to the LAN. Users access PVs via CA clients.
1
The backbone of the EPICS system is a local area network, or LAN. All of the IOCs are either
directly connected on the same subnet, or connected via a gateway to a different subnet.
The other components of EPICS are Channel Access clients, or CAs. These are user interfaces to
the PVs, and might be of graphical user interfaces, plots, gateways, Alarm Handlers, and Channel
Archivers, to name a few. When a CA is in need of a PV, it broadcasts that need over the LAN;
the IOC which has the PV, then answers back and establishes a connection with the CA. In this
way, users can view and/or control the process variables in a device independent way.
1.2
CAEN SY127 Description
This document describes a specific IOC and interfaces which allow users to control and monitor
the CAEN SY127 HV system. The interfaces include a channel access GUI, “HVSY127”, alarm
handler, “ALH”, and channel archiver, “Arch”, as seen in Fig. 1.2. Descriptions of the later two
CA’s can be found in the references.
Channel
Access
HVSY127
ALH
Control/Monitor:
I,V,ramp,trip,...
Arch
LAN
IOC
(miceiocpc1)
Hardware
Crates
RS232
CAEN SY127: (1)
I, V, ramp, trip, ...
CAEN SY127: (2)
Caenet
I, V, ramp, trip, ...
Figure 2: A portion of the EPICS system showing the components associated with the CAEN
SY127 HV system. The LAN extends to other IOCs and CA clients.
1.2.1
Communication with CAEN HV Controller – *** PLEASE READ ***
As of the writing of this document, MICE has two separate CAEN SY127 HV crates which are daisy
chained together via a Caenet link. The connection between the controlling computer, miceiocpc1,
and the first crate in the chain is an RS232 cable. The previous control of the two crates was
performed with a minicom program via an xterm window. ASCII strings were sent back and forth
across the RS232 line for communication and monitoring. The problem with this was that there
was no means of monitoring, archiving, or raising alarms without human intervention.
Due to the nature of the physical connection, the IOC which controls the HV crates literally
communicates with the same ascii strings seen when using minicom. The EPICS protocols are
written to parse the strings to both read and write asynchronously from/to the CAEN controller. As
a result, it is imperative that communication not be interrupted with strings of other commands and
that the strings which the controller is sending be fully read. This has the unfortunate consequence
of requiring delays in the code which renders the IOC to appear slow. Additionally, the baud rate
of the IOC was reduced to 4800 bps to accommodate the asynchronous communication.
2
2
HVSY127 Monitoring
Control and Monitoring of the CAEN SY127 crates is performed with an EPICS gui (EPICS
speak: “EDM” panel). This gui, can be found in application launcher under PID (particle ID) on
any MLCR (MICE local control room) CAM (Controls and Monitoring) terminal. Click on the
“HVSY127” link and the following EDM panel will appear, see Fig. 2.
Figure 3: The HVSY127 EDM panel for controls and monitoring of the CAEN SY127 crates.
Fig. 2 shows the EDM panel for controlling and monitoring the SY127 crates. The left side of the
panel is dedicated to controls and the right to monitoring. Beginning from the upper left-hand
corner, the “HV ON” or “HV OFF” light indicates whether the key is turned to the On or Off
position on the crate. The next three boxes give the present values of crate number, channel number, and channel name; labeled “CRATE”, “CHANNEL”, and “NAME”, respectively (clever, don’t
you think?). The next two boxes, labeled “FUNCTION REQUEST” and “VALUE REQUEST”
show the labels (same as used on the crate front panel) indicating the function to be performed
and initially, the present value, then the requested value. The “CHANNEL STATUS” box gives
the same indicator lights as on the crate front panel. These lights only update when a new read is
performed, therefore, they will often not follow transitions.
Monitoring of the CAEN SY127 crates is performed using the visual displays on the EDM panel,
the alarm handler, and the channel archiver. The visual displays on the EDM panel are on the right
hand side of the panel and allow the user to view the present crate’s: channel parameters, map of
the slots, and list of the channel errors. This section describes the available monitoring.
Initialization of the IOC begins with making a map of the crate
electronic modules (HV pods) and storing the results. A successful read of the map (see figure to the right) is followed by a
reading of all of the channels in the crate. These two functions
initialize the arrays of existing channels, set values, and names
for existing channels (required for changing channels). A check is
then made for a second crate, the crate count is computed, and
if existing, the next crate is selected and the procedure repeated.
3
If a second crate does not exist, the crate number and data for the lone crate is stored with a crate
count of 1. No further initialization will be required for the life of the IOC. A stop and restart of
the IOC will always perform a new initialization cycle.
The HVSY127 IOC monitors the two SY127 HV crates by performing periodic reads of the crates.
After initialization, periodic reads are performed as long as the IOC is running. The procedure is
to read one crate, wait 60 seconds, switch crates, and read the second crate. This cycle continues
throughout the life of the IOC. However, when a control operation is performed, see Sec. 3, then
a flag is set to prevent a read from occurring. Once the control function is complete, the flag is
released, a read is performed, the clock is reset to zero, and the read cycle continues.
2.1
Viewing Crate’s Channel Parameters
The HVSY127 visual channel monitor can be accessed in several ways. It is automatically updated
with each read cycle for each crate. It can also be forced, along with a new read, via the “Update
Display” button on the “CONTROL” menu. It also occurs automatically as part of a change crate
control function.
Figure 4: Example of visual monitoring of channel parameters: Crate 1 (left) and Crate 2 (right).
Only existing channels are displayed. The columns printed are:
• channel number, 0-39
• channel name as known by the crate
• a binary value indicating whether the channel is On or Off
• read voltage in “V” – this is what is monitored
• read current in “µA” – this is what is monitored
• set voltage in “V” – requested value
• set current limit in “µA” – channel trips when current draw exceeds this limit
• ramp up rate and ramp down rates in “V/s”
• trip time in “0.1s” – time before channel trips if current exceeds set current limit
4
2.2
Viewing Crate’s Slot Map
During initialization of the IOC, the list of HV pods (up to 10) in each crate is read and stored
internally. A view of this list can be found from the “CONTROL” menu button labeled “HV
Module Map”. Depending on the present crate the user is view one of the following figures, see
Fig. 2.2.
Figure 5: HV Pod locations in crate’s slots: Crate 1 (left) and Crate 2 (right).
2.3
Viewing Channel Errors
During initialization of the IOC, the list of HV pods (up to 10) in each crate is read and stored
internally. A view of this list can be found from the “CONTROL” menu button labeled “Error Table”. Depending on which crate the user is view, will show one of the following figures, see Fig. 2.3.
Figure 6: Example of HV error table
Note that channel number, channel name, Set and Monitored voltages and currents, as well as
relevant difference values are recorded.
2.4
SY127 Alarm Handler
One of the principle reasons to migrate to the EPICS based control of the CAEN SY127 HV crates
is its native ability to raise alarms on parameters being monitored. A screenshot of the Alarm
Handler and its associated screens is shown in Fig. 2.4.
With the periodic reading of the voltages and current draws in the crates, these values are compared
to the set values with each read. At the time of writing this document, a minor voltage alarm is
raised if the magnitude of the different between set and read voltages is ≥ 5 V ; a major alarm is
5
Figure 7: Alarm handler window for the CAEN SY127 system. Figure (right) is a list of channels
with error conditions; this is the same table as shown on the main EDM panel. The lower window
shows the “guidance” window.
raised if the difference is ≥ 10 V .
There are no lower limits on the current draw. Rather, a minor alarm is raised when the current
draw Ilim − Imon ≥ 20 µA and a major alarm is raised when Ilim − Imon ≥ 10 µA; where Ilim is the
trip limit and Imon is the monitored current draw.
These checks are only performed for channels which are ON.
Having an alarm on each individual channel of the HV system would result in a busy alarm handler.
Since any alarm conditions raise an alarm, only one alarm is required per crate. The IOC along
with the Alarm Handler raises alarms based on an OR of all channels in each crate. A record of
each channel can be found from the EDM panel, as described in Sec. 2.3.
From the Alarm Handler window, which is started from the “Tiered Application Launcher” under
HVSY127, an alarm would change the color of the box preceding each crate: yellow for minor alarm
and red for major alarm. The “G” button (circled in green in Fig. 2.4) gives “guidance” by opening a window with instructions (end of green arrow). The “P” button (circled in red in Fig. 2.4)
opens an additional EDM panel which displays the same error tables as shown in Fig. 2.3. This
EDM panel indicates to which crate it is displaying information in the upper right-hand corner.
At the bottom left-hand corner of the panel, one can press the button to change the crate error table.
Just for the record, I cannot change the label on the buttons.
2.5
SY127 Channel Archiver
As of the writing of this document, this is not yet implemented. However, it will run continuously
and record any changes to voltage or current limits.
6
3
HVSY127 Control
Control of the CAEN SY127 crates is performed from the EPICS EDM panel HVSY127, which
is found from the application launcher under PID on any MLCR CAM terminal. Click on the
“HVSY127” link and the EDM as seen in Fig. 2 will appear.
Due to the restrictions on communicating with the crates, see Sec. 1.2.1, only one such panel can be
opened and no simultaneous minicom sessions can be connected. Furthermore, an effort has been
made to remove user control buttons when the crate is busy. Future versions will complete this
task. For now, one must refrain from giving more then one input signal via button or
text entry box.
Control of the HV crates is accomplished using the following buttons (adjacent figure) which appear
on the EDM panel when the IOC is not busy. The buttons are:
• “Select Crate” for choosing between crates 1 and 2
• “Select Channel” for choosing between channels 0 and 39
• “Set Parameters” for changing a channel:
– Name, Voltage, Current Limit, Ramp Up/Down rates, Trip Times
• “Select Group” for change channel groups to view
3.1
Changing Crates
The two crates in the present system are located in the MICE experimental hall. Crate 1 controls
the Ckov and GVA1 counters and is located at the west end of the entrance aisle once inside the
hall. Crate 2 controls the 4 Fermilab BPMs, and is located in a rack on the outside, north-west
corner of the DSA.
In order to change crates, one presses the “Select Crate” button
from the “CONTROL” menu. A second window will appear as
shown in the adjacent figure. From window, one selects either
crate. Once the crate is selected, the IOC will change crates and
perform a new read of the channels. Selection of the same crate
will not result in an error, but will initiate a new read of the
crate.
The control buttons will not be available during the read, however, while the IOC is changing crates,
the buttons remain available.
Do not press any buttons after the first selection is made!
I will fix this in future version.
3.2
Changing Channels
A completely populated SY127 crate can have up to 40 channels. EPICS does not allow for changing
the EDM panels dynamically, so all of the possible channels are displayed when changing channels.
7
In order to change a channel, one presses the “Select Channel”
button from the “CONTROL” menu. A second window will
appear as shown in the adjacent figure. From window, one selects
the channel of choice from the list of buttons, 0-39. The IOC
will then find the correct name of the channel and change the
channel. After changing the channel, the result is compared to
the request, and the HVSY127 EDM panel channel number and
name are updated.
The control buttons will not be available during the read, however, while the IOC is changing
channels, the buttons remain available.
Do not press any buttons after the first selection is made!
I will fix this in future version.
3.3
Changing Channel Parameters
Once a channel is selected, one can change the channel’s parameters, namely:
channel name, set voltage (in V ), maximum current limit (in µA), ramp up and down rates (in
V /s), trip time (in 0.1 s, and On/Off status.
In order to change a channel’s parameters, one presses the “Set
Parameters” button from the “CONTROL” menu. A second
window will appear as shown in the adjacent figure. From this
window, select the desired function. At this point, the “FUNCTION REQUEST” window will display the desired function and
the “VALUE REQUEST” window will display the present value.
Also appearing will be the prompt for
the text entry box labeled “INPUT
VALUE” at the bottom of the EDM
panel. Adjacent to the left are screenshots of the control side (left side) of the
EDM panel:
• (a) – “Set Channel Name”
• (b) – “Set Voltage (V)”
• (c) – “Set Current Limit (uA)”
• (d) – “Set Ramp Up (V/s)”
• (e) – “Set Ramp Down (V/s)”
• (f) – “Set Trip Time (0.1s)”
The control buttons will not be available during the read, however, while the IOC is changing
parameters, the buttons and text box remain available.
Do not press any buttons or enter values after the first selection is made!
8
3.4
Changing Viewing Group
This is not yet implemented.
4
4.1
4.1.1
Known Problems
Multiple Entries
Problem
The biggest known problem with the HVSY127 IOC is the possibility that under certain circumstances, a user may enter one or more values more than once. Due to the nature of the communication with the crate controller, the commands will become confused since any serial stream must
complete before another stream travels on the RS232 line.
4.1.2
Solution
A gray “Reset” button might fix the problem, but this is not guaranteed if the IOC is already
in a loop. The only certain, albeit slower, fix is to kill the IOC from the “Remote Applications
Launcher” and then restart it. This does not require a restart of the HVSY127 EDM panel.
It is important to note that a restart of the IOC will not affect the channel parameters stored in
the crates. Barring an even more serious problem, the channel parameters are will not change with
starting/stopping the IOC.
4.2
4.2.1
Slow Response of Functions
Problem
The IOC is slow in changing parameters, in particular changing crates.
4.2.2
Solution
This is not a solution, but reread (or read) Sec. 1.2.1, to understand why it is slow. I may try to
tune it in the future, but have already made attempts to do so.
4.3
4.3.1
Unimplemented Functions
Problem
Some functions are not yet implemented.
4.3.2
Solution
They are on my To Do list. I’m publishing this because having a robust working version to monitor
is more important than having all the bells and whistles.
9
5
Conclusion
This document describes both the Control and Monitoring of the CAEN SY127 High Voltage
crates. The EPICS based control and monitoring is requisite to allow for constant monitoring, the
raising of alarms when parameters go out of limits, and archiving of any changes to the HV systems.
At present there are known problems if users do not follow instructions. Otherwise the system is
robust. Future versions will attempt to remove possible human errors from affecting the operation
of the IOC.
10