• Ravi Ammamuthu
• Sarayu Damodaran
• Virendra Galotra
© 2011 Cisco and/or its affiliates. All rights reserved.
Sep 25, 2013
Cisco Confidential
1
Integrated Global Monitoring
Database Provisioning
Database Patching
Q&A
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
2
IT manages hundreds of databases if not thousands
Global Data Centers
Heterogeneous Systems
Complexity
Lack of Agility
Flat/Shrinking IT budget
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
3
Integrated
Global
Monitoring
© 2011 Cisco and/or its affiliates. All rights reserved.
Database
Provisioning
Database
Patching
Database
Upgrade
Cisco Confidential
4
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
5
Prod
293 *
Non-Prod
1011 *
Standalone
86
Standalone
486
RAC
207
RAC
525
Standby
96
Standby
40
© 2011 Cisco and/or its affiliates. All rights reserved.
* Total excluding Standby DBs
Cisco Confidential
6
Global OEM users
PROD1
PROD2
Prod
Targets
NonProd
Targets
STG &
DEV
 Total targets - ~22000
 Targets by category are:
• Prod - 488 hosts, 293 DBs
• NonProd – 1216 hosts,
1011 DBs
• DG – 69 hosts, 96 DBs
 Targets on UCS, Sun-OS
 DB Repository Size
• 600 GB for PROD1
• 1500 GB for PROD2
 112 Metrics
 32 Reports & 20 Jobs
© 2011 Cisco and/or its affiliates. All rights reserved.
Legend
Cloud Control & Targets
Targets Only
DR
Cisco Confidential
7
Agents
Richardson
Primary
Amsterdam
RTP
Global Site Selector (GSS)
DR
ACE
ACE
Active
OMS
S/W Library Replication
Snapmirror-Async
Passive
OMS
Redo Shipping
DB Repos
DB Repos
RCDN
RTP
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
8
Agents
RTP
Richardson
Amsterdam
Global Site Selector (GSS)
DR
MVDC
ACE
ACE
ACE
S/W Library
Metrocluster
Active
OMS
Active
OMS
Redo Shipping
Passive
OMS
Redo Shipping
DB Repos
DB Repos
RCDN
Allen
© 2011 Cisco and/or its affiliates. All rights reserved.
S/W Library
Snapmirror- Async
DB Repos
RTP
Cisco Confidential
9
• Self Healing Through Corrective Actions
Feature
Restart Database, Listener
• Tablespace Space
Addition
• File System Cleanup
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
10
• Cluster Database Performance
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
11
• Historical Information
• Ability to drill down and narrow down
• Graphical Representation
• Comparison of configurations across
environments
• Agile report generation
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
12
• Segregation of Prod and Non-Prod Targets
• Apply Standard Templates Periodically
• Adopt Metric Extensions
• Avoid Storing Job Outputs in Repository
• Always be on Latest Patches
• Eliminate Customization as much as Possible
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
13
2009-2011
• OEM 10.2.0.5
• Centralized
Monitoring of Prod
and Non-Prod DBs
• Data Masking
• User Defined
Metrics (UDM)
• OEM Infrastructure
Migration to new
Data Center
2012
• Enterprise wide
Patching of DBs
using OEM
• OEM 12.1.0.1
Upgrade
• Monitoring ASM
DBs
2013
• Provision Grid
Infrastructure
• Provision RAC &
Standalone DBs
• OEM 12.1.0.2
Upgrade
• Metric Extensions
• Self Healing
• Automatic feed of
OEM data to Portal
• 4 OEM Cloud
Control LC Envs
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
14
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
15
Situation
• IT Vision :
Provision
Infrastructure Stack
in 5 Days
Position
• Identify Tool to
scale horizontally &
vertically to meet
demands - OEM
© 2011 Cisco and/or its affiliates. All rights reserved.
Complication
• Complexity :
Virtualization,
Cloud, Interconnected
Services
Implication
• Scalability :
Enhancement to
Custom Tool for
each new release
Action
• Use Out-of-Box
Solution with little
or no
Customizations
Benefits
•
•
•
•
•
Faster Provisioning
Optimization
Quality
Adhere Standards
Less OPEX
Cisco Confidential
16
Software Deployment
Standalone Database
RAC Database
Validate Server, Network,
Storage, Network, Storage
Validate Server, Network,
Storage
Validate Server, Network,
Storage
Set up Oracle Home from
existing Environment
Install Oracle Software
Install Grid Infrastructure
Create Standalone
Database
Configure Grid
Infrastructure
Setup/Validate Cisco
Standards
Install Oracle Software
Create RAC Database
Setup/Validate Cisco
Standards
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
17
• Manual Installation Process


scripts, silent mode
Solution:
pre-requisite checks
Enterprise
Manager (OEM) to
• Lack of Standardization
Provision
• Long and Error Prone
Manual Process
Databases
• Increased Resource Requirement
• Maintain Up-to-date Releases
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
18
Clone
Oracle
Home
Non-RAC
A.1
Create
Database
A.2
Deployment Procedures
A. Cisco Provision Oracle
Database
1. Deploy Database Software
2. Create a New Database
B. Cisco Provision Oracle
RAC Database
1. Deploy Grid Infrastructure
2. Deploy Database Software
3. Configure Grid
Infrastructure
4. Create a New Database
DB
Type
?
RAC
Clone,
Configure
GI
B.1, B.3
© 2011 Cisco and/or its affiliates. All rights reserved.
Clone
Oracle
Home
B.2
Create
RAC
Database
B.4
Cisco Confidential
19
• Launch Cisco Procedure and provide inputs
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
20
• Real Time monitoring of step status
• Drill down to log levels directly
Deployment procedures
extended to include CISCO
Customizations
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
21
Prod
293 *
Non-Prod
1011 *
Standalone
86
Standalone
486
RAC
207
RAC
525
Standby
96
Standby
40
© 2011 Cisco and/or its affiliates. All rights reserved.
From
Feb’13
OEM Provision
222
Standalone DP
• DB
43
• w/o DB 65 *
RAC DP
• DB
• w/o DB
48
69 *
* Total excluding Standby DBs
* 60% software only deployments
Cisco Confidential
22
Standalone DB
RAC DB
7
18
16
6
65%
Savings
H
5
4
H 14
o
o
4
u
3
r
0.5
2
Create nonRAC DB
3
s
2
1
Clone Oracle
Home
74%
Savings
12
10
u
6
3
Create RAC DB
8
r
6
s
4
8
2
0.5
2
2
Clone Oracle
Home
Install &
Configure GI
0
0
Manual
OEM
Manual
OEM
Multiple Environments are also provisioned at the same time
Environments Provisioned using EM12c = 222 (from Feb 2013)
Avg. No. of Concurrent Envs. Provisioned = 6
* Grid and Oracle Home includes patch storage
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
23
Value
Metrics
Faster Provisioning of
DBs
• 4x reduction for RAC cluster
Increased Agility
• 75% SLA Improvement
Reduced OPEX
• Scaled down resources from 4
to 1 for 222 Envs
• Multiple clusters in a single go
• No Extra Efforts for Scale-Ins
Increased Scalability
Increased Quality
© 2011 Cisco and/or its affiliates. All rights reserved.
• Standards Maintained
• Incident Prevention
• Minimal Errors
Cisco Confidential
24
DB Templates not supporting multiple file locations
• Apply patch for DBCA in Cisco’s Gold Code Tree
Sudo privileges prevent running Auto-Fix automatically
• Fix manually; Proceed with ignore option
Longer time to copy software across WAN
• Use reference target from same DC as source
Database listeners are not registered on RAC DB
• Plug-in scripts to create database listeners
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
25
Standardization using Gold Image
Leverage hooks for any customizations
Use Referenced Components for Remote DBs;
saves bandwidth and time
Define Templates; RAC, Non-RAC, ASM, dNFS
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
26
Enhance Standard Deployments
• Provisioning Profile with Locked inputs to
Deployment Procedures
• Segregation of Designer and Operator roles
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
27
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
28
Audit
Requirement
Challenges
Solutions
© 2011 Cisco and/or its affiliates. All rights reserved.
• Quarterly Patching
• PSU, CPUs, One-off Patches
• 3200 DB Instances
• Extended Outages
• Resource Intensive
• Not Scalable
• Leveraged OEM to Patch DBs
Cisco Confidential
29
STEPS
Manual Mode
Login
Login to individual Centralized login
Boxes
to EM
Schedule Patch
EM Mode
N/A
Download Patch
Run Pre-Patch
Shutdown DB
Automated using
Enterprise
Manager
Apply Patch
Startup DB
Post Patch
Health Checks
Manual process by every single
DBA on every Database
© 2011 Cisco and/or its affiliates. All rights reserved.
Multiple patches can be
scheduled by one DBA or by
multiple DBAs
Cisco Confidential
30
• Patch RAC & Standalone DBs
• Customized DP for Cisco’s processes using hook-ups
 Pre and Post requisite checks
 Application start/stop scripts
• Merge & Apply many patches in a plan
• Rolling Patching
• Rules to notify failures/success
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
31
Downtime (Hrs)
Direct
4000
Before OEM
2000
Cost
$385K
Before OEM
$65K
After OEM
Effort (Hrs)
4000
Before OEM
660
After OEM
After OEM
Indirect
1000 DBs
Savings 3 DBAs.
Reduced
Human Error
Reduced
Incidents
Highly
Scalable
Disclaimer: Cisco specific numbers
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
32
Jobs Failed to
Submit
• Cause : Duplicate Cluster Target;
unable to override named credential
• Disabled Auto-Discovery of Targets
Multiple
Patch Plans
• Cause : Opatch Bug 17417108
• STF
: Group into multiple plans
Performance
Issue in UI
• Working with Oracle
Opatch
download
process fails
• Cause : Corrupted software library
• Fix
: Rebuilt software library
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
33
• Run Patch Analysis before maintenance window
• Use MOS Online option for Patching, helps in direct
patch download and conflict resolution
• Use Rolling Patch option to minimize downtime for
RAC DBs
• Include multiple patches in a single plan to go in
one downtime
• Manage Oracle resources using Central Inventory
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
34
Ease of patching for 12
Node Clusters
2 DBAs to patch 42 DBs
in 6 Hrs.
* Increased Productivity
* Met Auditing timelines
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
35
Reduce Risk and Downtime further
• Out of Place Patching of Standalone and RAC
Reduce Documentation:
• Better use of Patch Template to reduce
documentation and communication to DBAs
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
36
Q1FY14
Q2FY14
Q3FY14
OEM MAA (Level-4)
using MetroCluster
Cron Less Monitoring
EM Target Self
Management
Auto Agent Install
Agent Restart
Monitor TimesTen
FS Clean Up
DB Upgrade
DB Mender
Monitor FMW
DG Build
Auto Space Addition
© 2011 Cisco and/or its affiliates. All rights reserved.
Monitor Golden Gate
Upgrade OMS
Cisco Confidential
37
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
38
Manager
• Ravi Ammamuthu
Architect
• Sarayu Damodaran
• Kunal Chakrabarty
Technical Team
• S Venky Sadagopan
• Pascal Laporte
• Manickaraja Kumarappan
• Surekha Jampala
• Venkat Kiran Gunturi
• Manikandan Ganesan
• Sairam Kantheti
PM
• Meena Chanda
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
39
Thank You
© 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
40