Using Oracle Enterprise Manager to Deliver Multitenant DBaaS on Oracle Exadata: Lessons Learned Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Presenters • Manish Shah - Manager, Exadata and OEM Engineering, The Hartford • Brian Bennett - Database Architect, The Hartford • Courtney Llamas - Consulting Member of Technical Staff Oracle Special Thanks To: • Barb Yankauskas – who could not be here but have been instrumental in our OEM re-engineering efforts as well as preparation of this presentation. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 2 The Hartford – Who We Are Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 3 OEM 12c Environments Summary OEM 12c Development Environment • OEM 12.1.0.3 • Release and Upgrade testing • 261 Targets • 6 different Oracle Database versions • 4 OS Platforms: • Linux, SunOS, HP-UX, Windows • Targets restricted to development Life Cycle Status • 162 OEM user accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. • OEM 12c Production Environment • OEM 12.1.0.3 • 5,557 Targets • 11 different Oracle Database versions • 3 OS Platforms: • Linux, SunOS, HP-UX • Targets span all Life Cycle Status: • Production, Dev, DR, QA, Staging • 391 OEM user accounts 4 OEM 12c Architecture Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 5 Why? Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 6 Framework Approach Our Requirements + OEM Capabilities = OEM Alerting and Monitoring Framework that satisfies the requirements Trick: You need to have an intimate understanding of the OEM capabilities first to be able to map your requirements. Challenge: Mapping your requirements to OEM capabilities is not intuitive Solution: We have done the hard work for you! If you follow gathering your requirements in a manner presented here, it will be easy for you to arrive at the framework that is customized for your environment and that can be used to map your specific requirements back to OEM 12c Capabilities. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 7 Requirement Drivers Requirements Driver 1: Your Organizational Structure Target Mix in your environment Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. How These Two Maps to Create a Division Of Responsibility 8 Requirement Drivers Requirements Requirements: Security (1) LOB Teams should have privileges on only those DB Targets that belong in their LOB (e.g blackouts, view and manage targets etc) (2) Platform Teams should have privileges to manage not only DB Targets but also all non-DB targets (e.g. host, ASM, Cluster etc) they manage for a given platform Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 9 Requirement Drivers Requirement Driver 2: DBaaS Catalog SLAs: Drives alerting requirements (e.g. # of alerts and thresholds differ by Metal tiers) Metal Tier Phy. Stby Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 10 Requirement Drivers Requirements Requirements: Alerting (1) Different Metric Thresholds for Gold, Silver and Bronze to satisfy response and resolution SLAs. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 11 Requirement Drivers Requirements Requirements: Alerting (2) Gold, Silver and Bronze targets will be alerted on different # of metrics to satisfy response and resolution sLAs. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 12 Requirement Drivers Requirements Requirements: Accountability in Alerting (3) LOB Teams are responsible to resolve business specific alerts (e.g. # of sessions) and platform DBAs are responsible to resolve platform specific alerts. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 13 Other Requirements: Following other requirements came from requirement drivers shown earlier 1. Need to group related targets to LOB teams 2. Assign gold-metric-template to Gold Targets, Silver to silver targets and bronze to bronze targets 3. Set accountability based alerting between LOB & Platform teams 4. Ability to differentiate between prod and non-prod alerts to determine whether to send alert to e-mail, pager or both 5. Allow customization of alerts as needed by specific teams 6. Maintain metric consistency by re-applying metrics periodically to maintain master alerts. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 14 Requirement Driver Requirements Requirement Driver 3: Exadata is a relatively new Platform for The Hartford. Databases are still migrated from Non Exadata to Exadata Platform Requirements: 1. Auto assign targets to LOB teams where they belong as new targets are added 2. Auto assign to platform teams 3. Auto assign metrics (# of) and alerting thresholds based on Metal tier status of targets that are added. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 15 How? Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 16 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 17 Targets & Target Properties • Targets – all targets were in scope • Auto-assignment of targets to security, administrative and alerting groups based on Oracle global target properties. • A flexible, scalable, self-supportive OEM environment with: • Metrics settings, templates, and template collections • Dynamic and administrative groups • Enterprise rule sets • An OEM framework that aligned with our organizational model Lessons Learned • Dynamic and Admin groups cannot use User Defined target properties • Do not use spaces in the description for Life Cycle Status Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 18 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 19 Security Groups Security Groups: • • • • Use target properties to drive security and alerting Simplify security administration Are distinct to each DBA team Drive membership of alerting groups Security Group Integrity: • Every target must have target properties value set • An OEM report will be used to confirm that all targets have values assigned to their target properties Security Groups in Our Implementation: • One for each of the three LOB (AppDBA) Teams • One for each of the two platform (SYSDBA) Teams Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 20 Security Groups Application DBA Security groups are distinct by: • Department • Contact • Target Type: Cluster Database, Database Instance, PDB System Database Administrator security groups are distinct by: • Department • Comment : “Non-Exadata” • Target Type: All Exadata Administrator security groups are distinct by: • Department • Comment : “Exadata” • Target Type: All Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 21 Security Group Lessons Learned: Oracle Database 12c Container and Pluggable Databases Pluggable and Container databases presented unique issues: • Each pluggable database will only be in one security group • The container database can have multiple PDBs, each can be supported by different teams Consider the items below where pluggable databases can be supported by multiple teams: • Which team actually supports the container database? • Do the teams that support the PDBs in the container database also support the container database? • Or does the system DBA support the container database? Lesson Learned: Don’t share container databases with multiple support teams Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 22 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 23 Roles Roles: • Implement a consistent set of privileges across OEM • Simplify OEM account administration by associating each DBA team’s security group to a role. Benefits of Roles: • • • • • Ease of user privilege implementation More efficient OEM user creation Consistent user privilege assignment Enabled reporting on team membership and privileges Reduces the use of “shared” user accounts Roles In Our Implementation: • One for each of the three LOB (AppDBA) Teams • One for each of the two platform (SYSDBA) Teams • One for each Of the LOB Leads Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 24 Roles • LOB DBA Team 1 Role: This is a distinct role for LOB DBA teams to perform these functions on targets in their security group: • to view and connect to targets • to place/free blackouts on targets, • to follow-up on target events and incidents • LOB DBA Team 1 Lead Role : A distinct role for LOB DBA team leads to perform these functions on targets in their security group: • • • • All of the privileges for LOB DBAs, plus the following Set, modify, and remove target metrics Create and apply templates Create and manage enterprise rule sets Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 25 Roles Platform DBA Roles: A role for Platform DBAs who are responsible for a given platform, privileges include: • Install and patch agents • Discover and view targets • Set target properties • Create Templates • Create Groups OEM Admin Role: • System DBAs who are responsible for OEM maintenance, patching, upgrading, discovery • Includes privileges to support and maintain the OEM Infrastructure – Note: These privileges are very powerful. Use with caution! Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 26 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 27 Alerting Groups Group type: simple OEM PPG group Purpose: allow LOB DBAs through the use of rules to monitor and alert on targets they support Alerting Groups In Our Implementation: • One for each of the three LOB (AppDBA) Teams • One for each of the two platform (SYSDBA) Teams The Alerting Groups are maintained by an automated process (sql) • Derives all the targets related to DB targets in the security group • Dynamically creates emcli commands to add/remove targets from alerting group • Uses OEM jobs Note: The alerting group is also assigned to the LOB DBA team role with view target privilege refer to appendix for view example code Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 28 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 29 Incident Rules Each DBA team has one ruleset Our implementation of Incident Rules & Rulesets Each ruleset has 2 rules: metric event & availability event Each Rule has 2 actions: pager & email, email only Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 30 Incident Rules - Lessons Learned • Rules are very flexible. • Take the time to plan out the design and standards you want to enforce • Identify the events that require alerting: availability and metric • Identify the alerting priority: by life cycle status, metal tier, alerting account? • Develop standard naming conventions for all rules & rulesets • Limit the number of administrators that have the ability to create enterprise rulesets • When creating metric alert rule, specify the events you want to alert on • Ensure the alerting account has view access to the target being alerted Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 31 Incident Rules - Lessons Learned • • • • Rules are very flexible. Create default rules on OEM alerting groups only Limit ruleset creation to the Lead DBA roles Rulesets and Rules in a ruleset are executed in order • Be careful when automatically clearing or executing a job from a rule can prevent following rules from firing • Beware of the “Rubik’s Cube” effect: • As target type, targets, number of metric & target availability events increase → administration becomes more difficult • Be consistent! Be consistent! Be consistent! Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 32 Alerting Accounts Are: • • • • • • • • An administrative account that supports notifications Associated to DBA teams Secure: the password is not provided Limited to View on Targets privilege on the alerting group Also useful for non-OEM users who need to receive notifications Flexible: teams can have multiple alerting accounts Flexible: notification schedules can switch between email and pager Low maintenance: A new email or DL can added to the account and all an incident rules without changing each rule Lesson Learned: • Keep it simple • Employ a standard naming convention Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 33 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 34 Metrics, Settings & Templates Metric settings can be set: • Locally at the target level for individual targets • Globally through the use of templates. As a best practice: • Establish a standard set of monitoring metrics across all targets • Identify which metrics need to be enabled and have thresholds set • Create a default template for each target type. • Set the default template for each target type Lessons Learned: • Start small → focus on metrics that are critical • Some metrics such as “Disk Device Busy” promoted hundreds of alerts. Choose metrics with caution! • Focus only on metrics & settings that are common for all targets by target type • Use metric extensions sparingly Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 35 Monitoring Framework Setup Workflow Targets Metric Settings Target Properties Templates Template Collections Admin. Groups Dynamic PPG Groups (Security) Simple PPG Groups (Alerting) Roles Alert Notifications & Alerting Accounts Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. Incident Rules & Rulesets 36 Administrative Groups Primary function: maintain metric settings, and support monitoring. • At first glance ► intimidating! • Our implementation used: • First Level ► Department • Second level ► Life Cycle Status • Third Level ► Location (Metal Tier) • Each target can be in only one administrative group • Only global target properties are used to create the group. Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 37 Administrative Groups By Department, Life Cycle, Metal Tier The revised admin groups, showing the Dev, Invest, PTE nodes merged ► They share common metrics and metric settings Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 38 Template Collections Primary function: A collection of templates to be applied to the admin group to maintain metric settings, and support monitoring at each leaf level. • A template collection is simply a set of templates that will be applied to an admin group • Each template collection contains templates that are unique by target type. • Each branch of the admin group hierarchy can/will have different template collections • The branch level will override settings in a parent group The auto apply of template collections to admin groups ensure consistent metrics Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 39 Administrative Groups: Lessons Learned • Administrative groups only use global target properties • The more values a target property has, the deeper, broader, and more complex the administrative hierarchy became. • Keep it simple! • To get the right design, it may take several iterations • Merge administrative groups that have templates with same metrics and settings. • Administrative groups will not use user defined target properties • Any manipulation of the admin group basically used to be a rebuild of the hierarchy which has been fixed in 12.1.0.3 • The primary purpose of the admin groups is to apply template collections Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 40 Summary By using We achieved an OEM 12c environment that: • Target properties, Dynamic, PPG, and Administrative groups, • Roles , Alerting Accounts, Incident rules & rulesets • Templates and Template collections • OEM Auditing • • • • • • Is flexible and self-supporting Allows customization on metal tier Remains “current” Is organized Provides an enterprise view Ensures accountability Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 41 Overall Lessons Learned • How do you want OEM to be used? • How will privileges be handled? • What do your OEM targets support: You need to identify: You need to ensure: • Gold tier business applications? • Silver tier business applications? • Bronze tier business applications? • Which metrics support your metal tiers? • What are the thresholds by platform or architecture? • 3 to 6 months for planning: how do you want to design and implement a multitenant OEM environment? • OEM best practices and recommendations are used! Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 42 Using Oracle Enterprise Manager to Deliver Multitenant DBaaS on Oracle Exadata: Lessons Learned • Questions? [email protected] [email protected] [email protected] Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 43 Using Oracle Enterprise Manager to Deliver Multitenant DBaaS on Oracle Exadata: Lessons Learned References • Out-Of-Box Roles – http://docs.oracle.com/cd/E24628_01/doc.121/e36415/app_a.htm#r5c1-t2 • OEM Resource and Target Privileges (12c Release 4 Enterprise Manager Cloud Control Security Guide) – http://docs.oracle.com/cd/E24628_01/doc.121/e36415/sec_features.htm#CJADDBGA • OEM Target Privileges: – http://docs.oracle.com/cd/E24628_01/doc.121/e36415/sec_features.htm#sthref90 • Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c (Oracle White Paper) Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 44 Appendix- Populating Alerting Groups: • To maintain Alerting Groups we created an automated process: • Start by creating and “seeding” the group with one member (seeding not required if using emcli) • Use PL/SQL, shell scripts, sysman and customized views, and security groups to generate emcli commands to update alerting groups • The views: • • • • • Identify the current alerting group members ►“Current members” Generate the alerting group members ► “Derived members” Add an alerting target ► “Derived members” – “Current members” Remove an alerting target ► “Current members” – “Derived members” Refer to the Appendix for the SQL details Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 45 Appendix- Populating Alerting Groups: Process Design sysman views: mgmt$manageable_entities, mgmt$target, and mgmt$target_associations identify the “Derived alerting members”, “Current alerting members” ./emcli modify_group -name="EDM_AppDBA1_AllTargets" add_targets="LISTENER_SCAN1_xdhfd2-eop-clus:oracle_listener" Create an OEM job to call the process and execute the emcli commands Check for sql and emcli warnings and errors Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 46 Appendix- Populating Alerting Groups: The SQL Behind the Scene VIEW HTDF.VIEW_MNTRNG_GRP_TRGTS_CRRNT SELECT DISTINCT assoc_target_name AS target_name, assoc_target_type AS target_type, security_group_name, monitoring_group_name FROM sysman.mgmt$target_associations, HTDF.view_appdba_admin_groups WHERE source_target_name = monitoring_group_name AND source_target_type = 'composite'; VIEW HTDF.VIEW_MNTRNG_GRP_TRGTS_DRVD SELECT -- the database, rac database, and container databases in the security group assoc_target_name AS target_name, assoc_target_type AS target_type, security_group_name, monitoring_group_name FROM sysman.mgmt$target_associations, HTDF.view_appdba_admin_groups WHERE source_target_name = security_group_name AND assoc_target_type IN ('oracle_database', 'rac_database', 'oracle_pdb') AND assoc_def_name = 'contains' UNION SELECT -- the targets that are associated to the databases targets in the security group ta2.assoc_target_name AS target_name, ta2.assoc_target_type AS target_type, va.security_group_name, va.monitoring_group_name Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 47 Appendix - Populating Alerting Groups: The SQL Behind the Scene VIEW HTDF.VIEW_MNTRNG_GRP_TRGTS_DRVD (continued) FROM HTDF.view_appdba_admin_groups va, sysman.mgmt$target_associations ta1, --ta1(source=group, assoc=db) (SELECT source_target_name, source_target_type, assoc_target_name, assoc_target_type FROM sysman.mgmt$target_associations UNION SELECT -- the listener targets that are associated to the databases (connecting through host) ta4.source_target_name AS source_target_name, ta4.source_target_type AS source_target_type, ta3.target_name AS assoc_target_name, ta3.target_type AS assoc_target_type FROM sysman.mgmt$target ta3, sysman.mgmt$target_associations ta4 WHERE ta3.target_type = 'oracle_listener' AND ta4.assoc_target_type = 'host' AND ta4.source_target_type IN ('oracle_database', 'rac_database', 'oracle_pdb') AND ta4.assoc_def_name = 'hosted_by' AND ta3.host_name = ta4.assoc_target_name) ta2 --ta2 (source=db, assoc=assoc targets) WHERE ta1.source_target_name = va.security_group_name AND ta2.source_target_name = ta1.assoc_target_name AND ta1.assoc_target_type IN ('oracle_database', 'rac_database', 'oracle_pdb') AND ta1.assoc_def_name = 'contains' AND ta2.assoc_target_type IN ('rac_database', 'oracle_database', 'oracle_pdb', 'oracle_listener', 'host', 'oracle_emd', 'osm_instance', 'osm_cluster', 'cluster'); Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 48 Day In A Life Of Targets Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 49 Security and Roles - Metadata Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 50 Alerting Groups – Metadata Copyright © 2014 by The Hartford. Confidential. For internal distribution only. All rights reserved. No part of this document may be reproduced, published or posted without the permission of The Hartford. 51
© Copyright 2026 Paperzz