Machine Checkout and
Setup Periods
Jan Uythoven
Thanks to: O.Aberle, R.Bailey, F.Bordry, O.Brunner, L.Bottura,
E.Carlier, P.Charrue, E.Ciapala, S.Claudet, R.Denz, D.ForkelWirth, B.Goddard, M.Jimenez, R.Jones, M. Lamont,
V. Mertens, P.Ninin, B.Puccio, S.Ramberger, F.Rodriguez
Mateos, R.Schmidt, L.Serio, B.Todd, M.Zerlauth
1
Machine Checkout:
WHEN ?
 Year 1: Following Hardware Commissioning
 Assume all equipment tests are done in the past 18 months
 Most of the system tests are done in the past 18 months
 Subsequent years: Following Shutdown
 Assume most equipment tests are done
Work
 Machine Checkout
 First time machine complete cold and closed
 All systems running together from the control room
 Before
First Beam
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 2
Tasks - Summary

Redo a subset of tests performed during the Hardware
Commissioning period





New Tests





Because some tests more than a year ago
Because the final configuration of the system might be different,
especially the interface to other equipment
If hardware commissioning period was squeezed in time, this will have
been at the cost of the system tests
As a result: will need more time for the Machine Checkout
Additional system tests
All systems together – machine wide, interdepedencies
Machine completely closed, last access / final checks
Under the responsibility of the Operations Group
Green light from DSO, info for INB
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 3
Time
Phases of Machine
Checkout
Hardware Commissioning
Sectors 1-2 and 2-3
Cooldown of all
sectors
Close machine
ELQA tests
completely;
Some phases
in parallel
Equipment tests
Contr. Access per 2 sectors:
Interlocks,
Power Converters
Machine Checkout phase I
RF, kickers, BDI,..
Machine Checkout
phase II
Machine Checkout
phase III
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Complete Machine
Cycle from CCC
DSO tests
First beam
Page 4
Phase I

Infrastructure for the complete machine


Water, Ventilation, Power, Controls Network
Radiation Protection (RAMSES)
 Check response ionisation chambers
 Interface with other systems
UPS, AUG
When full
inventory
 Cooldown of the complete machine
of He
available?
 Sectors T up to 200 K: need 10 days for cooldown
 If T > 80 K: need to redo Electrical Quality Assurance tests
(ELQA), first time  3 weeks
 If T < 100 K: cooldown needs about 5 days

Up to about 5 weeks for sectors warmed up > 80 K.
Need to start cooldown in parallel with Hardware
Commissioning. Manpower.
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 5
Phase I, continued
Access System

Close the complete machine



New tasks – before only done per sector (during HC)






Test of LHC Access Control System (LACS)
Test of LHC Access Safety System (LASS)
New tasks: interface adjacent sectors, machine wide
 Important Safety Element in point X must interface over the global machine and
beam dump in point 6
Interface with Radiation Protection equipment: access veto
Interface with HR data base, biometrical data
Some equipment was not in final state during Hardware Commissioning tests
Exercise procedures for ‘patrouille’, giving access, back from access, …
Start first with sectors needed for Machine Checkout equipment
tests phase II

If necessary, time to sort out problems parallel to phase II programme
Ideally only controlled access from phase I onwards
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 6
Phase II

Per two sectors



Check interlocks: PIC, WIC, QPS
 See
Test all power converters, two sectors at
a time
 See
Determines the
pace of tests in
Phase II
Simultaneously in other
sectors

Other equipment checks, see
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 7
Phase II
Interlock Systems


Before power converters can be tested, test interlock system
Quench Protection System (QPS)




Powering Interlock Controller (PIC)




Visual check of equipment, half day per sector
Automatic test procedure, 8 hours (same as monthly procedure): fires the
quench heaters without current in magnets
Test interface to BIS -> check in phase III
Takes information from the QPS
Automated test procedures and visual control
Needs about 8 hours for testing
Warm Interlock Controller (WIC)


No automated procedures foreseen
Some manual testing and visual control: 8 hours
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 8
Phase II
Power Converters

Followed by testing of Power Converters
two sectors at a time





Run all converters to full power, perform tests at full power
Calibration already done during Hardware Commissioning
12 h continuous running at full power
Need about half a week per two sectors
Proceed around the ring
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 9
Phase II
Control System
 Control System Communication Mechanisms
 Field buses operational machine wide. All elements to be
checked: need about 8 days
 Already start during phase I
 FrontEnds, BackEnds and general services
 Should have been made available during the Hardware
Commissioning period sector by sector and left running
 Timing generation and
 6- 8 days for the first year
distribution
 Databases access
 1 day, at least
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 10
Phase II
Control System, ctd.

Application software



Tested continuously during the last months / years
Specific projects need testing
 Interlocks, realtime feedback, logging, postmortem, safe
beam flag etc.
 Estimate 10 days for test …..???
Final tests when complete machine available:
See phase III
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 11
Other Equipment tests:
RF


Basic work of testing cavities, cryo, LLRF and control
during hardware commissioning period
Number of tests which would clearly accelerate RF
operations and improve performance:
Ed’s Wish List
- Reliability of RF system: power & cryogenics
- Software and diagnostics developments / improvements
- New conditioning hardware for subsequent years
- RF noise and ripple measurements
- Customisation of klystrons (positioning of resonance)
- Linearisation of klystron response
- Effect of detuning and coupler position on loop stability
- Optimisation of loops for higher intensity
- Fine adjustment of trip level for fast transients
- Setting-up of the longitudinal damper
- Preparation
of
feed-forward
and
1-T Checkout
feedback
Jan Uythoven,
AB/BT
Chamonix@CERN
2005,
Machine
Page 12
Other Equipment tests:
Warm Magnets

Visual inspection


No spanners lying around…
All protection covers in place
 Cooling system
 All valves open
 No visible leaks

o.k.
Expect 1 day work per sector, 8 days total
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 13
Other Equipment Tests:
Vacuum

Assume machine already cold & closed

Check interlocks


Sector valves
Connection to BIC
3 days per octant
2 teams in parallel
12 days
 Not to forget
 Vacuum interlocks on transfer lines: 2 days
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 14
Other Equipment tests:
Collimators and Beam Absorbers

Redo some of the hardware
commissioning tests in local & remote



Interfaces with more intelligent software
from the CCC to manage settings


Motor control, position switches
Temperature sensors, cooling system
Not defined yet, should include BIC
Couple of days

In parallel with other tests
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 15
Other Equipment tests:
Pulsed Magnets




Beam dump kickers (MKD and MKB point 6)
Injection kickers (MKI points 2 and 8)
Tune and Aperture kickers (MKQA point 4)
For all of them



Verification of local interlocks
Pulse at full power – MKI some re-conditioning time
Interface with many systems across the machine
 RF point 4 (rev. freq.)
 Energy from dipoles either side point 6
 Timing, controls, machine protection, post mortem
 OASIS, logging
 Cross connections between MKI and MKD
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 16
Other Equipment tests:
BDI

Large distributed systems: BPMs & BLMs



Interlocked Systems – check interface BIC (phase III)


Hardware fully checked during HC, but re-check
Main tasks: software and logging, links to other users
BLM, BCT point 4, BPMs point 6, LDM (Longitudinal Density Monitor)
Diagnostics systems – no special requirements

Q and Q’ measurements, Schottky systems, Sychrotron radiation
monitors, OTR screens, Wire scanners
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 17
Phase III

Status after Phase I & II




All equipment and systems (re-)tested, sector by sector
Individual interlocks tested
Controlled access
Phase III






Machine wide tests from the CCC
Go through complete machine cycles without beam
Accent will be on control system and application software
Interleaved with final tests of access system and green light by DSO
In principle no access should be necessary any more
Control System Tests: Build on experience !
 To be tested throughout the yearS with the maximum number of ‘dry runs’
 TI 8 tests with beam
 Hardware Commissioning of all sectors including system tests
 Sector test with beam
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 18
Phase III
Complete Machine Cycle
Without Beam

Initial machine cycle (45 minutes) to get to initial field conditions



Generate functions for the power converters
Download all functions to power converters – synchronous ramping of
all power converters including energy ramping of beam dumping system






All fixed displays, video, trajectory and orbit correction
Feedback applications: trajectory, tune, chromaticity, transverse, longitudinal
Logging of all parameters during the cycle
Vacuum displays, alarm screens, radiation monitoring
Interface with the Beam Interlock System



Timing (slow and fast), OASIS to set timing of kickers
Interface with RF, injection kickers, beam dump kickers, machine protection system
Applications – communication with all equipment


Settings management, trimming of functions
Machine wide timing of the systems, pre-pulses etc.


No further tests possible – real magnetic field to be found out with beam
Safe beam flag depending on energy
Post mortem after beam dump
Some kind of sequencer to automate tasks (LHCexec)
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 19
Phase III, continued

Now obligatory to have machine completely closed



Followed by Beam Interlock System (BIS) tests (or BIC)




Need “Green Light” from all systems – towards the end of test period
16 different users, 150 ‘user boxes’
 Every box needs to receive an interlock, provoked by the equipment connected
to it. Check 4 x dump request per equipment.
Estimate 3 x 8 hours to test – 2 days
Rubber stamping of systems by DSO – data for INB


If search broken during phase II: Complete patrouille with several teams:
2 days
Re-test complete access system before DSO test: 2 x 12 hours
 Machine protection system has probably been touched upon during phase II
Need about 1 day, if everything goes fine
Rubber stamping by ‘Mr. Interlock’

Need formal test procedures and acceptance criteria
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 20
Time Required

Phase I: 1 week minimum


Cooldown and closing machine
Phase II: 3 weeks minimum


Partially parallel
with hardware
commissioning
Interlocks, equipment &
system tests
Phase III: 2 weeks minimum



Complete cycles without beam
Final test access system and BIS
DSO tests
Jan Uythoven, AB/BT
Minimum of 6
weeks on planning!
Equipment setup
with beam, see
Chamonix@CERN 2005, Machine Checkout
Page 21
Equipment Tests With
Beam

BDI



RF






Check interlocks of the system
Safe beam flag
Beam Dumping System



Logging beam data, post mortem
Steering – feedback
Machine protection tests


Low level control (typically 3 weeks)
Phasing
Feed back
Control system


Calibration BPMS, BCTs
Set-up of tune and chromaticity measurements
Beam dump to be set up at the different energies
Verification of post mortem, logging and analysis
Cryogenics with significant beam load
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 22
Not to underestimate:
Transfer Lines

TI 8 will need recommissioning



Tested before
 With beam up to TI 8 TED end 2004,
 Not tested yet: last 100 m + collimators
 Sector test end 2006
But will need to be recommissioned before first beam
TI 2: never fully commissioned before



Will probably only be ready for commissioning at about the
same time as the last LHC sector (transport of all LHC
magnets through the TI 2 tunnel)
Could be done in parallel with hardware commissioning of the
last sector: preferred solution but impact on HC schedule
Other option is to do this in parallel with ring 2 beam
commissioning: higher risk in case of problems
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 23
Machine Checkout
For Subsequent Years




No Hardware Commissioning period but shutdown work to
be finished
Outline stays the same with similar tests
Some tests should be faster, because systems are better
known
Foresee about 5 weeks minimum: longer than SPS / LEP
because systems are more complex, machine protection to
be taken more seriously
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 24
Conclusions

After the hardware commissioning period the machine is not
ready for beam



At least 6 weeks of tests need to figure on the LHC planning
for Machine Checkout





Start earlier for cooling down the sectors warmed up > 80 K for ELQA
tests, in parallel with hardware commissioning
Start earlier for software and machine protection: DRY RUNS
to be planned and scheduled
Many tasks in parallel: need careful planning and good coordination
Some running in of equipment with beam is also required


Confirmed by experience from LEP & SPS, need a period to get ready for
beam under the responsibility of OP
Even with all equipment groups 100 % ready on time (!): complete
systems tests remain and their interdependencies need to be verified
BDI being the obvious candidate, but also RF, beam dumping system
Don’t forget the transfer lines
Same applies every year after a shutdown
Jan Uythoven, AB/BT
Chamonix@CERN 2005, Machine Checkout
Page 25