AFCRI/CBIO Mission Continuity Planning – Key Accomplishments
Abramson Family Cancer Research Institute
Department of Cancer Biology
Dorothy Hunter – Manager, Research Facilities
898‐0266; [email protected]
James J. Riley – Director, Fiscal Operations
746‐5520; [email protected]
Original Process and Goals
• Began during FY2009
• Originally surrounded IT operations
• Quickly evolved to encompass all operations
– Information Technology, Research Facilities, Human Resources, Grants Management, Financial Reporting, Agency Management, etc.
• Goals included:
– Identify and address areas of opportunity
– Document policies, procedures and job responsibilities
– Establish key relationships internally and externally
Key Discoveries
• Identified targets for improvement
–
–
–
–
–
–
IT backup strategy
Server location
Ultra‐low freezer management
Administrative inconsistency
Standardization of financial reporting to faculty
Enhance internal collaborations
• Validated existing procedures
–
–
–
–
–
–
Availability of terminal server Server‐based file storage
Focus on web database applications
RF emergency spill kits
Strong vendor relationships
RF freezer failure response
Major Accomplishments
• Information Technology Improvements
– Enhanced backup strategy
– Server relocation
• Research Facilities
– Improved emergency power planning
– Web‐based freezer monitoring
– Information dissemination among lab staff
• Business Office
– Enhanced PI Reporting
– Increased efficiency
– Cost reductions
Information Technology Improvements
• Servers
– Relocated to properly equipped facility
• Backup strategy
– Dramatic improvement in managing potential for data loss
•
•
•
•
Worst case down to ~4 hours
Implemented new tape backup and archival systems
Standardized “Undelete” policy
Implemented routine interval backups
Research Facilities (RF) Improvements
• Increased emergency generator access to all ultra‐low freezers
• Standardized backup freezer availability
– 1 x ‐80 per floor; ‐20 and cryogenic capacity
• Developed plan for catastrophic power failure in advance of major storms
• Implemented remote temperature monitoring system for freezers and cold rooms
RF – Freezer Monitoring: The Problem
• Ultra‐low freezers contain most valuable, potentially irreplaceable research material
– Loss impacts careers, funding, prior findings, etc.
• Previous failure detection system
– Relies on audible unit alarms and individual unit temperature readings, battery operated
• Are easily disabled
– During overnight hours and weekends, rely on touring security guard visits @ 1‐2 per shift
RF – Freezer Monitoring: Former Strategy
RF – Freezer Monitoring Timeline
• Under discussion for several years
• Summer 2012‐ Secured funding commitment based on $75k estimate
• Fall 2012 – Review existing campus options and begin vendor contact
• Spring 2013 – Evaluate 5 vendors
• Summer 2013 – Work with PMACS, PSOM SPO, ISC and Purchasing to finalize arrangements and contract
• Fall 2013 – Installed > 150 devices across 6 research floors in two buildings
• Present – fully installed, implementing minor adjustments, documenting SOP’s, etc. Identified first failure 12/23/13.
RF – Monitoring System Evaluation
•
•
•
•
•
•
•
•
Performance
Price
Support
Scalability
Fewest “Moving parts”
Scope of environmental monitoring
Compliance
Control of communication and software
– Installed at Penn, using AirPennNet, behind firewalls
RF – Freezer Monitoring: How it Works
• Temperature and Power monitors to support redundancy
– Currently: 87 freezer temps, 30 cold rooms, 30 power devices
• Communicate readings via AirPennNet‐Guest every 20 minutes (adjustable)
– Server on‐campus
• Alarm parameters set to meet necessary standards
• Cascading alarm system with 3 levels of notification that requires and tracks acknowledgement
• Reporting capabilities to support compliance
• Data readily available
RF – Freezer Monitoring: Dashboard
RF – Freezer Monitoring: Door Opening Incident
RF – Freezer Monitoring: Full Freezer Failure
Freezer Monitoring: Larger Mission Continuity Potential
• PSOM Space Planning and Operations constituents
• ULAR animal facility environmental monitoring regulatory compliance
• Clinical sample regulatory compliance
• Power consumption comparison and evaluation
• Laboratory behavior modification
• Asset tracking via GPS location
Freezer Monitoring: Power Comparison – Residential Unit
Freezer Monitoring: Power Comparison – Commercial Unit
Freezer Monitoring ‐ Cost
Freezer Monitoring – Supporting the Cost?
• Developed Funding Partnerships
–
–
–
–
Department of Cancer Biology
Abramson Family Cancer Research Institute
Institute for Immunology
PSOM Space Planning and Operations
• Opportunity cost: freezer monitor vs. other asset investments
• Strategic repurposing of existing assets
• Multi‐year savings plan related to operating costs
• Adjusted cost per unit < $500
• Potential for expansion built into initial contract
• Final determination: We couldn’t afford *not* to do this!
Freezer Monitoring – Vendor Contact
• Aegis Scientific
–
–
–
–
http://www.aegisfridge.com/
http://www.aegismonitoring.com/
Sales Rep: Dee Jones, 800‐796‐2344
Technical Info: Ryan Kostishion, 800‐796‐2344
• Penn Purchasing vendor site
– http://www.purchasing.upenn.edu/buyinfo/suppliers/aegis_scientific.php
• Penn Purchasing news release
– http://www.purchasing.upenn.edu/news/131011_01.php