WinCC-OA Upgrades
in LHCb
Clara Gaspar, May 2016
Control System Scope
❚ Different Scope
Experiment Control System
Detector & General Infrastructure (Power, Gas, Cooling, etc.)
Detector Channels
Trigger
TFC
Front End Electronics
Readout Boards
High Level Trigger (Farm)
Storage
Monitoring
DAQ
External Systems (LHC, Technical Services, Safety, etc.)
❚ Different Requirements?
Clara Gaspar, May 2016
2
Control System Architecture
ECS
Status & Alarms
Commands
INFR.
SubDet1
LV
DCS
SubDet1
DCS
SubDet2
DCS
SubDet1
TEMP
SubDet1
GAS
…
TFC
SubDetN
DCS
DAQ
SubDet1
DAQ
SubDet1
FEE
SubDet2
DAQ
SubDet1
RO
HLT
…
LHC
SubDetN
DAQ
Legend:
Control
Unit
LV
Dev1
LV
Dev2
…
LV
DevN
FEE
Dev1
FEE
Dev2
…
FEE
DevN
Device
Unit
❚ Some Stats:
❙
❙
❙
❙
~420 000 datapoints
~14 000 000 datapoint elements
~8 000 Control Units
~40 000 Device Units
Clara Gaspar, May 2016
3
Control System Deployment
❚ WinCC-OA:
❙ ~170 projects
❙ Distributed over ~150 PCs
All virtual machines
All Linux
❚ + Driver Machines
❙ Only Drivers and/or
Scattered Managers
❙ ~30 Windows
(ccpc/USB boxes of 4)
❙ ~15 Linux
(ccpc/Specs boxes of 2)
❙ 1 DIP Machine
Clara Gaspar, May 2016
4
Control System Deployment
❚ Some deployment details
❙ Projects in “/localdisk”
❘ “/group” (G: in windows) visible everywhere
❘ “GlobalProject” in Config of All Projects, ex:
pvss_path = "/opt/WinCC_OA/3.11"
proj_path = "/group/velo/pvss/fwComponents_VEECS1"
proj_path = "/group/online/ecs/PVSSShortcutProjects311/GlobalProject"
proj_path = "/localdisk/wincc/VEECS1"
❙ Operation via “Shortcuts”
❘ Automatically generated, for Windows and Linux, ex:
pvss_path = "C:/Siemens/Automation/WinCC_OA/3.11"
proj_path = "G:/online/ecs/PVSSShortcutProjects311/CentralRepository"
proj_path = "G:/online/ecs/PVSSShortcutProjects311/VELO/VELORepository"
proj_path = "G:/velo/pvss/fwComponents_VEECS1"
proj_path = "G:/online/ecs/PVSSShortcutProjects311/GlobalProject"
proj_path = "G:/online/ecs/PVSSShortcutProjects311/VELO/VEECS1_W“
❙ Repositories (containing panels and libs) generated every night
❘ Created from FwComponents => Per sub-detector => LHCbRepository
Clara Gaspar, May 2016
5
Control System Tools
❚ “Shortcut” Project
❙ “Database” of ALL Projects, allows:
❘ Generate and manage Shortcuts
❘ Generate repositories
❘ Execute scripts or import DPs
on any group of projects
Clara Gaspar, May 2016
6
Upgrading…
❚ Requirements/Wishes:
❙ Do everything centrally
❙ As transparent as possible for all types of users
❘ Developers, Operators, etc.
❙ As fast as possible
❚ HW and/or Operating System
❙ Normally comes with new WinCC-OA version
❘ -> See later
Clara Gaspar, May 2016
7
Upgrading…
❚ WinCC-OA patches
❙ Try to agree with users to do all in one go (but can be negotiated):
❘ Done by Sys-admins:
〡 Stop projects (Linux service) -> Install Patch RPM -> ReStart projects
❚ WinCC-OA Major Versions
❙ Depends on “Upgrade” procedure…
❙ Worst case example:
(move to virtual machines, slc6, windows to linux and 3.8 to 3.11 at once)
❘ Extracted ALL DPs to project's dplist dir
❘ Copied all Projects to a central area
❘ Prepared a script: create new project, import DPs, copy back folders
(in the process fix Windows paths and WinCC-OA novelties)
❘ Sys-admins created the machines and installed WinCC-OA
❘ Run the script for all projects
❘ Ask sub-detectors to check if everything still worked
Clara Gaspar, May 2016
8
Upgrading…
❚ Framework Components
❙ LHCb Components
❘ Dump new files (panels, scripts, libs) in “GlobalProject”
❘ Import DPs to relevant projects, if needed
❘ If necessary restart managers or project
(either transparently or in agreement with users)
❙ JCOP Framework Components
❘ Same as above, but:
〡Need to prepare LHCb FW based on JCOP FW
〡Merge LHCb changes: Trending, Alarm Handling
〡Verify carefully all changes…
❚ In general we don’t use the Installation tool except to
create New projects and for tests
Clara Gaspar, May 2016
9
JCOP FW Upgrades
❚ Requests:
❙ Please do not load new libraries in the config file
Always use #uses
❙ Please do not overwrite settings in the dpl file, ex:
fwTT_TrendTree.root
_FwTreeType
""
0x101
_AESProperties_fwAes.Alerts.FilterTypes.AlertSummary
28.07.2005…
_AESProperties
2
❙ If a post-install script is needed for an upgrade would be
nice to write it in the release notes
❙ If the dpl files have changed would be nice to write it in
the release notes
❙ Would be nice to have an fwInstallation dpl file or to
understand exactly when the DPs are upgraded
Clara Gaspar, May 2016
10
JCOP FW Upgrades
❚ Other “features”:
❙ Normally the time of a new FW release and the time
window to install it before start up is very short.
❘ So by the time I understand all new “features” and how to
merge them with LHCb, there is no time to wait for a new
version.
❘ So I end up “fixing” my local copy and after is done, I forget to
report it back… Until the next release…
❙ This applies mainly for fwTrending and Alarm Screen
❙ But also for example I changed the “Corporate Color”
(I think in 3 different places), plus it’s against the guidelines
❘ Ok(ish) for DeviceEditorNavigator (development tool)
❘ But not for AlarmScreen (an operation tool), completely different
look and feel from all other LHCb Operation panels
Clara Gaspar, May 2016
11
JCOP FW Upgrades
❚ Other “features”: DB Schemas
❙ FwConfigurationDB schema
❘ We only find out that the schema needs upgrading long after
installation…And very close to LHC startup
❘ Then we need to open the ConfDB panel and upgrade each
schema (once per sub-detector)
❘ The upgrade always fails for some schemas-> need Piotr’s help
❙ Oracle Archive schema
❘ Every new version brings new problems of compatibility
between standard API/CERN API/patch level, etc.
Clara Gaspar, May 2016
12
JCOP FW Upgrades
❚ Other “features”:
❙ Problem with WCCOAascii
Since WinCC-OA 3.11 and all Linux
❘ Ascii Manager never starts while the project is running
FATAL, 135, ASCII export cannot be started, WinCC OA project
start possibly in progress. Please try again later.
❘ Need to stop project and start only main managers
❙ Why? Need help…
❚ And Notes:
❙ We do use occasionally fwInstallation (without DB)
❘ So we need to keep it functional
❙ We also use the fwSystemOverview tool for upgrades
❘ To stop/restart projects, To restart selected managers, etc.
Clara Gaspar, May 2016
13
Upgrading…
❚ Applications
❙ Continuous Process
❘ Modify files in /localdisk (or better in a test project)
❘ Test
❘ When happy move them to FwComponents (central area)
❙ Backups are there for “mistakes”…
❚ Why do we do it like this?
❙ While DCS part is quite stable
❙ DAQ projects change every day…
❘ No way (for me) to keep up with proper components
❘ So couldn’t impose it on the others…
❙ I think fwInstallation needs proper/stable components
Clara Gaspar, May 2016
14