Allstate Migration and
Infrastructure for XI R2
Tim Rodine, TCR Solutions, Inc.
Alex Timberg, Allstate Insurance Company
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 2
Business Intelligence at Allstate
History of Business Intelligence at Allstate
EDS
Tool 1
Prior State: up to 2004
Interim State
AllCorp
AllCorp
Enterprise
Reporting
Tool 2
Historically, multiple Allstate areas held
responsibility for business intelligence,
which created:
Overlapping tool functionality
Overlapping organizational
responsibilities
A lack of clear tool direction and
appropriate usage
Slide 3
Enterprise
Business
Intelligence Tools
Ivantage
Tool 3
Centralized BI
Center of
Excellence
BusinessObjects 6.5 for
ad hoc reporting
Crystal Enterprise 10 for
operational reporting.
Drive an enterprise BI strategy to
deliver business value, lower costs,
and reduce inefficiencies
Consolidate tools and support under
one organization (EBIT)
Business Intelligence at Allstate
Current state
 BusinessObjects XI R2 – common Business Intelligence platform
Slide 4
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 5
Starting with Separate Environments
BusinessObjects 6.5
 Primarily used for analysis with the Allstate data warehouse
 Supported ~20 independent ad hoc and reporting projects in addition to
the Allstate data warehouse
 Rolled out in 2004, supported ~8,000 users
 Production in Ohio data center, Dev and QA in Illinois
 Total number of servers: twelve, eight in production
 IBM WebSphere as an application server
Percentage of Users Per Project
All Other
Groups
21%
Project 5
17%
Project 6
4%
Project 4
Project 1
18%
7%
Project 3
3%
Slide 6
Project 4
12%
Project3
7%
Project 2
11%
Starting with Separate Environments
BusinessObjects 6.5 environment
DEVELOPMENT
TEST/QA
DATA STORES
PRODUCTION
Users
Developers
VIP/Load Balancer
http://Infoview
PRIMARY
NODES
VIP/Load Balancer
http:/ADWTEST
http://PREPRO.INFOVIEW
PRIMARY
NODE
SECONDARY
NODE
SECONDARY
NODES
DATA CONNECTION THROUGH DBMS CLIENTS OR NATIVE DRIVERS
Testers
ALLSTATE DATA WAREHOUSE
UNIX ORACLE
MVS DB2
WINDOWS SQL SERVER
Slide 7
Starting with Separate Environments
Crystal Enterprise 10
 Rolled out in 2004, supported ~22,000 users
 Variety of data sources
• Oracle
• SQL Server
• DB2
• SAP
 Distributed between two data centers: Illinois and Ohio
 Access through ePortfolio/Infoview or custom applications
• Most custom apps hosted on their own web servers
 Production cluster of six servers
Slide 8
Starting with Separate Environments
Crystal Enterprise 10 environment
SANDBOX
DEVELOPMENT
TEST/QA
DATA STORES
PRODUCTION
Users
Developers
Testers
VIP/Load Balancer
http://CE/logon
WEB/APP TIER (.NET)
http://CESBX/logon
ALL TIERS
http://CEDEV/logon
ALL TIERS
http://CESTG/logon
ALL TIERS
INTELLIGENCE/PROCESSING
TIER
ILLINOIS DATA CENTER
PROCESSING TIER
DATA CONNECTION THROUGH DBMS CLIENTS OR NATIVE DRIVERS
Developers
Admins
DATA SERVERS
MVS DB2
DATA SERVERS
UNIX ORACLE
DATA SERVERS
UNIX SAP
OHIO DATA CENTER
DATA SERVERS
WINDOWS SQL SERVER
Slide 9
Starting with Separate Environments
Crystal Enterprise 10 – continued
 Multi-tier design for load balancing and failover
• Two Web Servers (IIS/.NET) with load balancer
• Two Intelligence/Processing Servers in Illinois data center
• Two Processing Servers in Ohio data center
 Predecessor for XI R2 design
 Tested for disaster recovery
 Total number of servers: nine
Slide 10
Starting with Separate Environments
Crystal Enterprise 10 production - detail
VIP/Load Balancer
http://CE/logon
VIP/Load Balancer
USERS
Web/App Tier
ACTIVE SERVICES:
- WEB APPLICATION SERVER
Other Web Servers
CRYSTAL
ENTERPRISE SDK
Intelligence/Processing Tier
Processing Tier
ACTIVE SERVICES:
Web/App/Intelligence Tier
- CRYSTAL MS
- FILE REPOSITORY SERVERS
- EVENT SERVER
- PROGRAM JOB SERVER
- REPORT JOB SERVER
- PAGE SERVER
- CACHE SERVER
- REPORT APPLICATION SERVER
REPLICATION
ILLINOIS DATA SERVERS
ILLINOIS DATA CENTER
Slide 11
ACTIVE SERVICES:
CMS DB
ORACLE
CMS DB
ORACLE
- REPORT JOB SERVER
- PAGE SERVER
- CACHE SERVER
- REPORT APPLICATION SERVER
BACKUP
REPORTS
I/O FRS
REPORTS
I/O FRS
NAS
BACKUP
OHIO DATA SERVERS
OHIO DATA CENTER
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 12
Planning for the Migration
In 2005 developed the strategy for migrating to BusinessObjects
XI R2 and continued with participation in the Beta program
Detailed planning started in April 2006
Q2 2006
Strategy
Detailed
Planning
New
Servers
Built
Q3 2006
Q4 2006
Q1 2007
XI R2 Environment Available
Crystal 10 Migration to XI R2
BusinessObject 6.5 Migration to XI R2
Retire
Platform
Retire
Platform
BusinessObject 5 Migration to XI R2
Slide 13
Planning for the Migration
Migrate existing content only
 Audit reports used to develop list of content and validate usage
• Some projects/reports were no longer needed
• If the project needed enhancements then this started in the
BusinessObjects XI R2 development environment and went through
the entire project life cycle on the new platform
No re-write to another tool
 No conversion from BusinessObjects Reporter to WebIntelligence
No enhancements to existing content
 No Enhancements as part of the migration!!!
Infrastructure build - incremental approach
 Build parallel XI R2 environment
 Migrate applications one at a time
 Move servers as load changes
Slide 14
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 15
Design of XI R2 Cluster
How many servers?
Requirements:
 30,000 users
• 8,000 WEBI, 10% concurrent
• 22,000 Crystal Reports, 10% concurrent
• Up to 400 simultaneous requests
 Scalable to 40,000 users
 Data distributed between multiple data centers
• Allstate Data Warehouse in Ohio
• SAP in Illinois
• Other sources in Ohio, Illinois, or Texas
Slide 16
Design of XI R2 Cluster
Sizing exercise




Call your friendly Business Objects consultants
Three days of shuffling numbers
Sizing estimator spreadsheet as a basis
The most important info:
• Ability to handle 470 simultaneous requests
Verdict:
 60 CPU’s
 Processing servers in two data centers
 Separate tiers for web/application, intelligence, and processing
Slide 17
Design of XI R2 Cluster
Distributed design
 Web/application tier – two servers
• IIS/.NET/WebSphere
 Intelligence tier – two servers
• Central management servers
– High availability Oracle data base on Unix
• I/O file repository servers
– FRS on network attached storage
• Event server
• Cache servers
 Processing tier – eight servers
• Four- eight CPU machines in Ohio Data Center
• Four- four CPU machines in Illinois Data Center
 Expandable to 40,000 by adding one server to web/app and one to
intelligence tiers
Slide 18
Design of XI R2 Cluster
BusinessObjects Enterprise XI R2 production environment
VIP/Load Balancer
http://BOE/logon
VIP/Load Balancer
USERS
Web/App Tier
Other Web Servers
BOE SDK
Processing Tier
Intelligence Tier
Web/App/Intelligence Tier
Processing Tier
CMS DB
ORACLE
CMS DB
ORACLE
REPLICATION
REPORTS
I/O FRS
REPORTS
I/O FRS
NAS
BACKUP
OHIO DATA SERVERS
OHIO DATA CENTER
Slide 19
BACKUP
ILLINOIS DATA SERVERS
ILLINOIS DATA CENTER
Design of XI R2 Cluster
The Bottom line:





Slide 20
Number of servers: twelve (savings of nine)
Support staff: four (savings of two)
Flexible design
Easy to scale up – just add servers as needed to the appropriate tier
Load balancing and failover built in
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 21
Performance, Resiliency, and Failover
Objective: blend of performance and fault tolerance
Separation of tiers improves performance
 Processing tier servers installed close to data
Servers in pairs for load balancing and failover
 Single server failure should be barely noticeable
All services installed on all servers
 Simplified standard installation
 Only needed services switched on
 Easy to readjust servers if needed
Remote processing servers used as a standby failover
 No backup servers sitting idly in a “hot site”
Slide 22
Performance, Resiliency, and Failover
Servers in Illinois data center preconfigured in case of a
disaster in Ohio
 Start Central Management Server (CMS) on backup servers to
register with the cluster, then stop and set to manual
 Set location of File Repository Server (FRS) root to backup NAS
share
 All services configured to point to @BOEPRD cluster, not individual
server name
CMS and FRS replicated from Ohio to Illinois
Disaster recovery procedures identified, documented
and tested
Slide 23
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 24
Migration – the Scope
Migrate existing content only
 Audit reports used to develop list of content and validate usage
• Some projects/reports were no longer needed
• If the project needed enhancements then this started in the
BusinessObjects XI R2 development environment and went through
the entire project life cycle on the new platform
No re-write to another tool
 No conversion from BusinessObjects Reporter to WebIntelligence
No enhancements to existing content
 No Enhancements as part of the migration!!!
Infrastructure build - incremental approach
 Build parallel XI R2 environment
 Migrate applications one at a time
 Move servers as load changes
Slide 25
Migration – the Scope
Content Migrated
•
•
•
•
•
•
Users/Groups
Universes and connections
Standard reports (Crystal and BusinessObjects Classic)
Personal documents
Historical instances of Crystal reports
Custom code (.NET viewers) for application integration
Content Not Migrated
• Inbox documents > Save as personal document first
• Scheduled documents > Need to reschedule on new platform
• Preferences
Slide 26
Migration - the Team
The Team




Project Owner/Sponsor
Project Manager
Technical and Process Lead
Infrastructure Team ~ 4 part time team members
• The Infrastructure Team were Technical Analysts/Engineers
 Migration Coordinators ~ 20 part time team members
• The Migration Coordinators were Business Analysts or
Developers who were familiar with the applications and users
Slide 27
Migration – the Process
Business
Objects 6.5
Crystal
Enterprise 10
Migration Coordinators
Crystal
Application
Infrastructure
Team
Move to
Staging
Migration Coordinators
Application
Test & Report
Data
Validation
Migration Coordinators
Successful
?
No
Business
Objects
Application
Shut Down Old
Application
Slide 28
Production
Environment
BusinessObjects XI R2 Staging Environment
Log Issue &
Track
Resolution
Clients
User
Acceptance
Testing
Infrastructure
Team
Successful
?
No
Log Issue &
Track
Resolution
Move to
Production
Migration Coordinators
Redirect
Users to
Production
Verify Client
Accessibility
Topics
Business Intelligence at Allstate
Starting with Separate Environments
Planning for the Migration
Design of XI R2 Cluster
Performance, Resiliency, and Failover
Migration
Lessons Learned and Tips
Q&A
Slide 29
Lessons Learned and Tips
The basics





Invest time in strategy and planning
Clearly identify roles, responsibilities and processes
Meet regularly with migration team members
Log all issues in a central location
Ensure regular / thorough communication occurs
Don’t take the upgrade/migration lightly
 Start with smaller projects to migrate
 Plan for issues and delays
 Keep the end user in mind to minimally impact their needs
Slide 30
Lessons Learned and Tips
Tip: Spend time to thoroughly know the technical environment




Slide 31
Numerous new services to set up, configure and monitor
Spend some time each day in the environment
Review the Readme from the latest release
Set up naming standards for folders, groups, connections…
Lessons Learned and Tips
Tip: Spend time to thoroughly know the user environment
 Shake down key functionality to make sure it works as expected
 When starting, spend some time each day in the environment
 Review the ‘XIr2 What’s New’ document
Slide 32
Lessons Learned and Tips
Tip: Spend time on the security setup




Slide 33
Security drastically changed from BusinessObjects Classic
Have a checklist for the setup of any project
Standardize functionality roles to ease maintenance
For Supervisor like interface use the Security Viewer and CMC
together for administering the system.
Lessons Learned and Tips
Tip: Spend time to understand the Import Wizard




Slide 34
Decide if it’s a one time or gradual migration
Where did the reports or that universe go?
Why do I have multiple connections with similar names?
Find out where the Import Wizard log is and how to read it
Lessons Learned and Tips
Tip: Work with Business Objects support closely
 There may be a resolution already
 There may be issues that needs resolution from Business Objects
 Most of the content migrated without an issue however a quick
modification fixed others
Slide 35
Lessons Learned and Tips
Tips
 Set up common reference documents for easy access
 Set up training / support sites internally early
Slide 36
Lessons Learned and Tips
The results
 Majority of the content was migrated without any issue or intervention
 Most users needed minimal to no training on the new environment
 The Knowledge Accelerator content improved dramatically which
helped with any gaps in training
 Reduced the amount of infrastructure needed with BusinessObjects
XI R2 compared to the separate Crystal Enterprise 10 and
BusinessObjects 6.5 deployments
 Infrastructure support staff was freed up to work on other efforts
 Many positive comments by users on the new interface and
functionality with BusinessObjects XI R2
 Few remaining apps delayed due to external reasons
…all of this was accomplished while new development efforts were
underway as well as another major upgrade project for Oracle 10g
was rolled out at Allstate
Slide 37
Q&A
Questions
 Tim Rodine, President, TCR Solutions Inc.
 Alex Timberg, Systems Consultant, Allstate Insurance Company
 We will repeat questions to ensure everyone can hear
Contact information
 Email: [email protected]
 Email: [email protected]
Slide 38