Liverpool HEP – Site Report
May 2007
John Bland, Robert Fay
Staff Status
Two members of staff left in the past year:
• Michael George August 2006, Andy Washbrook December 2006
Replaced by two full time HEP system administrators
• John Bland, Robert Fay December 2006
One full time Grid administrator
• Paul Trepka
One part time hardware technician
• Dave Muskett
Current Hardware
Desktops
• ~100 Desktops: Upgraded to Scientific Linux 4.3, Windows XP
• Minimum spec of 2GHz x86, 1GB RAM + TFT Monitor
• 48 new machines, rest upgraded to equivalent spec
Laptops
• ~30 Laptops: Mixed architectures, specs and OSes.
Batch Farm
• Scientific Linux 3.0.4, + software repository (0.7TB), storage (1.3TB)
• 40 dual 800MHz P3s with 1GB RAM
• Split 30 batch / 10 interactive
• Using Torque/PBS
• Used for general analysis jobs
Current hardware – continued
Matrix
• 10 node dual 2.40GHz Xeon, 1GB RAM
• 6TB RAID array
• Used for CDF batch analysis and data storage
HEP Servers
•
•
•
•
•
User file store + bulk storage via NFS (Samba front end for Windows)
Web (Apache), email (Sendmail) and database (MySQL)
User authentication via NIS (+Samba for Windows)
Quad Xeon 2.40GHz shell server and ssh server
Core servers have a failover spare
Current Hardware – continued
MAP2 Cluster
• 960 node (Dell PowerEdge 650) cluster
• 280 nodes shared with other departments
• Each node has 3GHz P4, 1GB RAM, 120GB local storage
• 12 racks (480 nodes) dedicated to LCG jobs
• 5 racks (200 nodes) used for local batch processing
• Front end machines for ATLAS, T2K, Cockcroft
• 13 dedicated GRID servers for CE, SE, UI etc
• Each rack has two 24 port gigabit switches
• All racks connected into VLANs via Force10 managed switch
Storage
RAID
• All file stores are using at least RAID5. New servers starting to use
RAID6.
• All RAID arrays using 3ware 7xxx/9xxx controllers on Scientific Linux
4.3.
• Arrays monitored with 3ware 3DM2 software.
File stores
• New User and critical software store, RAID6+HS, 2.25TB
• ~3.5TB general purpose ‘hepstores’ for bulk storage
• 1.4TB + 0.7TB batchstore+batchsoft for the Batch farm cluster
• 1.4TB hepdata for backups and scratch space
• 2.8TB RAID5 for LCG storage element
• New 10TB RAID5 for LCG SE (2.6 kernel) with 16x750GB SATAII.
Network
Topology
MAP2
WAN
Switch
NAT
Switch
Offices
Servers
Force10
Gigabit
Switch
LCG servers
1GB link
Proposed Network Upgrades
Topology - Future
MAP2
3GB
3GB
WAN
firewall
Force10
Gigabit
Switch
LCG servers
Offices
Servers
1GB link
VLAN
Network Upgrades
Recently upgraded core Force10 E600 managed switch to increase
throughput and capacity.
• Now have 450 gigabit ports (240 at line rate)
• Use as central departmental switch, using VLANs
• Increasing bandwidth to WAN using link aggregation to 2-3GBit/s
• Possible increase to departmental backbone to 2GBit/s
• Adding departmental firewall/gateway
• Network intrusion monitoring with snort
• Most office PCs and laptops are on internal private network
Wireless
• Wireless is currently provided by Computer Services Department
• HEP wireless in planning
Security & Monitoring
Security
• Logwatch (looking to develop filters to reduce ‘noise’)
• University firewall + local firewall + network monitoring (snort)
• Secure server room with swipe card access
Monitoring
• Core network traffic usage monitored with ntop/MRTG (all traffic to be
monitored after network upgrade)
• Use sysstat on core servers for recording system statistics
• Rolling out system monitoring on all servers and worker nodes, using
SNMP, MRTG (simple graphing) and Nagios
• Hardware temperature monitors on water cooled racks, to be
supplemented by software monitoring on nodes via SNMP.
Printing
We have three group printers
•
Monochrome laser in shared area
•
Colour led
•
Colour ink/phaser
•
Accessible from Linux using CUPS and automatic queue browsing
•
Accessible from Windows using Samba/CUPS, and auto driver installs
•
Large format posters printed through university print queues.