University of Edinburgh
_____________________________________________________________________________________________
Migrate IDM to Oracle SOA Suite 11g and
Upgrade Grouper
High Level Design
COM007
Document Version: 1.1
Date: 29/07/13
__________________________________________________________________________
Information Services
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
Contents
1
1.1
1.2
2
2.1
3
3.1
3.2
4
DOCUMENT MANAGEMENT .................................................................. 3
Contributors ........................................................................................ 3
Version Control ................................................................................... 3
INTRODUCTION ...................................................................................... 4
Grouper upgrade R&D ........................................................................ 4
IDM PATCHING STRATEGY ................................................................... 5
SOA patching ...................................................................................... 5
Database patching .............................................................................. 5
ITERATIONS ............................................................................................ 6
4.1
Iteration One – Outbound processing migration .............................. 6
4.1.1
10g Outbound processing issues ................................................... 6
4.1.2
Queue management ...................................................................... 6
4.1.3
Queue archive and Error handling ................................................. 7
4.1.4
Composite design .......................................................................... 8
4.1.5
Push versioning ............................................................................. 8
4.1.6
Service definitions and notification control ..................................... 8
4.1.7
IDM Administrative UI .................................................................... 9
4.2
Iteration Two – Inbound processing migration ............................. 10
4.2.1
Current inbound processing issues .............................................. 10
4.2.2
11g queue management .............................................................. 10
4.2.3
Queue archive and Error handling ............................................... 10
4.2.4
Composite design ........................................................................ 11
4.2.5
API clean-up ................................................................................ 11
4.3
Iteration Three – Pull web services migration ................................ 12
4.3.1
10g Pull web services issues ....................................................... 12
4.3.2
“Get” web services ....................................................................... 12
4.3.3
Change push subscription............................................................ 13
4.3.4
Get identity service....................................................................... 13
4.4
Iteration Four – Grouper upgrade .................................................... 14
4.4.1
LDAP Provisioning ....................................................................... 14
4.4.2
Default group type ........................................................................ 14
4.4.3
Grouper Hook .............................................................................. 14
4.5
5
Iteration Five – VRS Fixes ................................................................ 14
REFERENCES ....................................................................................... 15
__________________________________________________________________________
Page 2 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
1 Document Management
1.1 Contributors
Please provide details of all contributors to this document.
Role
Author
Reviewer
Reviewer
Reviewer
Reviewer
Reviewer
Unit
Applications Division
Applications Division
Applications Division
Applications Division
Applications Division
Applications Division
Name
Richard Good
Chrisann Macsween
Michael Sun
Alex Carter
Chris McKay
Stevie Conlon
1.2 Version Control
Please document all changes made to this document since initial distribution.
Date
10/06/13
29/07/12
Version
1.0
1.1
Author
Richard Good
Richard Good
Section
All
4.1.3
4.1.5
4.2.3
4.3.4
Amendment
Initial Draft
Add indexing note
End to end processing
clarification
Note on UUN addition to
archive, indexing.
Clarify calling client can
change settings
__________________________________________________________________________
Page 3 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
2 Introduction
This document details the high level design for both the migration of the IDM
from SOA 10g over to 11g, and the upgrade of Grouper.
SOA 11g is different in a number of ways:




It is now deployed on Weblogic
It requires the use of Service Composite Architecture applications
(henceforth referred to as SCA)
It can use BPEL 2.0 processes
It has an easy to use Event Delivery Network (henceforth referred to as
EDN)
NOTE: For more details on new features, refer to the SOA 11g feature
evaluation.
In order to safeguard the transition from 10g to 11g, the project will not go for
a “big bang” approach. It will instead transition various functional components
in a series of iterations.
The main sections which follow cover the iterations we will follow to migrate
the IDM.
NOTE: This high level design will not exhaustively go through the
resolution of all issues identified in the IDM Known Errors list, they will
be addressed via the individual design specifications for an iteration.
2.1 Grouper upgrade R&D
An evaluation of Grouper has been carried out, the results of which can be
found on Insite at:
https://www.wiki.ed.ac.uk/display/insite/Grouper+2.1.x+evaluation
The main summary points are:


We will be moving to Grouper 2.1.4
We will move to using incremental provisioning for Central Auth
__________________________________________________________________________
Page 4 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
3 IDM Patching strategy
As part of the migration, all code will be moved over to SVN, and will conform
to the Software version and release standards.
3.1 SOA patching
SOA patching will be done via Apache Ant automated deployment, as per the
SOA development standards.
3.2 Database patching
The standards currently do not include database patching specifically. In order
to address changes to the IDM table and PL/SQL API, a database patching
strategy will be adopted such that:


A tagged release contains roll-forward and roll-back patching scripts for
any database changes.
We introduce the ability to patch the IDM database schemas in an
automated fashion (either forwards or backwards).
To do so, we will make use of Apache ANT, and introduce a table in the IDM
schema which indicates which release version of the IDM the database is
using.
NOTE: We will re-use scripts which have been created for other IS
Applications software to do this.
__________________________________________________________________________
Page 5 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4 Iterations
Each of the following iterations will be deployed to live in a discrete package,
and will have a separate System Design Specification.
The introduction of SCA applications in SOA 11g gives us a convenient way
of grouping together functional service components. This also allows us to
effectively “hide” services from the outside world, which gives added security.
4.1 Iteration One – Outbound processing migration
This iteration concentrates on migration of the outbound “push” SOA
processes for the IDM.
4.1.1 10g Outbound processing issues
On the current 10g version of the IDM, the following are seen as the main
issues with outbound processing:





The outbound processing queue gets too large, and causes the queue
processing to stall
Services receive many notifications which are of no interest to them
The service definitions which control notifications do not work as
specified
Notifications for the same identity occasionally go out together, but in
the wrong order (i.e. update before insert), and cause certain services
to fail notifications
It is often difficult to troubleshoot failures at a java connector level (the
log files have to be used)
4.1.2 Queue management
In order to better manage the queue, we will leverage a few new features of
SOA 11g.
Firstly, notifications will be given a priority when they are inserted into the
queue. This will allow us to fast-track certain changes through the system.
Queue entries will now be de-queued (deleted) from the notification queue.
This will help keep the table size down, and should therefore speed up
processing. De-queued changes will be published as an event to the EDN.
In addition, we will make use of mediator message re-sequencing in order to
prioritise the messages.
__________________________________________________________________________
Page 6 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
Message
(Priority 1)
Message
(Priority 2)
Queue
SOA
Handler
Event Publisher
Message
(Priority 3)
Figure 1. Event queue publisher
4.1.3 Queue archive and Error handling
Currently, a nightly process runs which archives off notification messages.
Since the queue will change to de-queuing messages, both archival and error
handling will have to change.
Instead, successful process of a change message will insert the message
directly into the archive, and failure to process a message will insert a
message back into the queue with a failure status code.
Receive
Message
Process
Notification
Submit
change error
Archive
change
Figure 2. Archive and error handling
NOTE: The move to immediate archival will include the creation of
indexes on the archive table(s) for support and reporting purposes.
__________________________________________________________________________
Page 7 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4.1.4 Composite design
A composite will be built which conforms to the following:




The composite will have unit tests which cover all aspects of process
flow, including error testing
There will be no externally callable components, except where they are
for the support of unit testing
It will make use of the EDN publish/subscribe to generate notification
events
In order to ensure that timeout of calls to external services do not
cause the notification to , the notification composite should be moved
to asynchronous, and should have an extended timeout to manage the
correct error handling
4.1.5 Push versioning
There are currently 2 versions of push which run on the IDM:


Version 1, or legacy connector is to support ITI Unix services
Version 2 is for the other services
It is proposed that we create a Version 3 service push model, whereby a
service connector is written purely in a composite, as a subscriber to
notification events.
The benefit of doing so will be to see full end-to-end notification processing
from within the Weblogic UI, which would be of great benefit for support
purposes.
As part of this, we will produce a Version 3 service as a proof of concept. It is
proposed to use the Discussion Forums service, as that is both low risk, and
also experiences a significant amount of errors on the current live IDM which
slows down processing.
NOTE: Full end-to-end notification processing view via the Weblogic UI
would only be possible for Version 3 services, other push services would
only show viewing of service response (and not the audit of what the
service did with the notification).
4.1.6 Service definitions and notification control
In order to better control both the number of notifications generated, and also
reduce the required complexity of connectors, the service definition should be
updated. Broadly, on a service by service basis:
__________________________________________________________________________
Page 8 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________




Notification generation and notification push settings are separate
things and should have separate on/off switches
Entitlement notifications should be able to be switched on/off
Insert/Update/Delete notifications should be able to be switched on/off
Group and Identity notifications should be able to be switched on/off
In addition, the way in which a contact is added to the IDM should be updated,
such that:



A contact’s primary key is email address, and there should only be one
per unique contact
It should be possible to create/update/delete contacts
Contacts should be easily assignable to a service
4.1.7 IDM Administrative UI
In order to alleviate some of the problems with the current Cold Fusion
VRS/IDM User Interface, it is proposed to implement a new UI in order to
implement the service administration screens.
VRS UI
VRS/IDM
UI
IDM UI
Doing so will allow us to make the necessary changes to the administration
UI, and also allow us to transition elements of the existing Cold Fusion UI at a
future date.
Features included in this iteration would be:

Service administration (for existing services)
__________________________________________________________________________
Page 9 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4.2 Iteration Two – Inbound processing migration
This iteration focuses on migration of all the SOA processes responsible for
receiving with inbound requests, making changes to the IDM, and generating
notifications.
4.2.1 Current inbound processing issues
On the current version of the IDM, the following are seen as the main issues
with inbound processing:



The inbound processing queue can become large, which can cause the
queue processing to stall
Timeouts can occur for large/complicated processing changes
Changes for the same identity occasionally come in together, and get
processed in the wrong order (i.e. update before insert), causing
processing issues
4.2.2 11g queue management
In order to better manage the queue, we will leverage a few new features of
SOA 11g.
Firstly, changes will be given a priority when they are inserted into the queue.
This will allow us to fast-track certain changes through the system.
Queue entries will now be de-queued from the process queue. This will help
keep the table size down, and therefore make it far less likely that the query
on the table will end up timing out. De-queued changes will be published as
an event to the Event Delivery Network (see Figure 1).
In addition we will make use of mediator message resequencing in order to
better manage the ordering of notifications. Specifically, we will use Best
Effort resequencing, ordered by the change date and grouped by priority. That
way changes should be sequenced in the correct order, and grouped by
priority so that high priority messages are processed first.
4.2.3 Queue archive and Error handling
Currently, a nightly process runs which archives off process queue messages.
Since the queue will change to de-queuing messages, both archival and error
handling will have to change.
Successful process of a change message will now insert the message directly
into the archive, and failure to process a message will insert a message back
into the queue with a failure status code (see Figure 2).
__________________________________________________________________________
Page 10 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
In addition, a new table should store a list of identities which have failed
changes, and therefore require synchronisation.
Finally, failed changes should be archived after a suitable period of time (to be
agreed).
NOTE: Archival of an identity change without a uun (such as from HR)
will add the UUN at the archival stage. The archive will also be indexed
appropriately for support and reporting purposes.
4.2.4 Composite design
A composite will be built which conforms to the following:





The composite will have unit tests which cover all aspects of process
flow, including error testing
There will be no externally callable components, except where they are
for the support of unit testing
It will make use of EDN publish/subscribe to generate change events
In order to avoid transaction timeouts, the composite should be moved
to asynchronous, and should have an appropriate timeout (to be
agreed) set to manage the correct error handling
Grouper interactions will be moved into a set of externally callable SOA
processes
o getGroup_ep
o createGroup_ep
o setGroup_ep
o deleteGroup_ep
o addGroupMember_ep
o deleteGroupMember_ep
o getGroupMembers_ep
4.2.5 API clean-up
The current API contains many redundant procedures and functions. As part
of the migration process, all redundant code and packages should be
removed, leaving only procedures which are actually in use.
__________________________________________________________________________
Page 11 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4.3 Iteration Three – Pull web services migration
This iteration focuses on migration of the SOA processes supporting pull
requests.
4.3.1 10g Pull web services issues


Only a limited subset of web services are currently used
The identity web service returns all data for an identity, which is often
more than a service needs
4.3.2 “Get” web services
For reference, the current pull web services are:







bpelIDMAcknowledgeChange – Allows a service to acknowledge a
processed change
bpelIDMChangePushSubscription – Allows a service to change their
push subscription
bpelIDMGetChanges – Allows a service to request a number of
notification changes
bpelIDMGetGroup – Allows retrieval of a group (from Grouper)
bpelIDMGetGroupMembers – Allows retrieval of group memberships
(from Grouper)
bpelIDMGetIdentity – Allows retrieval of a specific identity
bpelIDMIsGroupMember – Returns whether an identity is a member
of a group
All of the web services in the above should be reviewed and enhanced as
necessary, including the creation of unit tests to support them.
The services should also be renamed as per the new naming standards,
which will mean as follows:

bpelIDMAcknowledgeChange → setChangeAcknowledged_ep

bpelIDMChangePushSubscription → setPushSubscription_ep

bpelIDMGetChanges → getChanges_ep

bpelIDMGetGroup → getGroup_ep

bpelIDMGetGroupMembers → getGroupMembers_ep

bpelIDMGetIdentity → getIdentity_ep

bpelIDMIsGroupMember → isGroupMember_ep
In addition, the externally callable Grouper processes in the inbound iteration
should be made available for authorised services to use.
__________________________________________________________________________
Page 12 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4.3.3 Change push subscription
This service should be updated as per the changes recommended in Section
3.1.6.
4.3.4 Get identity service
It should be possible to do a simple lookup of an identity without returning all
the details of get identity. The current service should be modified to support a
calling client being able to:






Switch affiliation retrieval on/off
Switch address retrieval on/off
Switch email retrieval on/off
Switch org retrieval on/off
Switch programme retrieval on/off
Switch course retrieval on/off
__________________________________________________________________________
Page 13 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
4.4 Iteration Four – Grouper upgrade
This iteration will focus on upgrading Grouper to 2.1.4.
4.4.1 LDAP Provisioning
The new version of Grouper allows for incremental change based provisioning
of LDAP. This iteration will move the new Grouper installation to use change
based provisioning.
4.4.2 Default group type
A new default group type should be introduced, which performs the following:


Creates a gid attribute
Creates a source attribute (which Golden Copy source system the
group comes from)
4.4.3 Grouper Hook
The grouper hook should be modified as follows:


The creation of the gid attribute should be removed, as it will be
created by the default type
Group notification creation should respect the service definition settings
in Section 3.1.6
4.5 Iteration Five – VRS Fixes
The final iteration will be a series of fixes to the VRS interface, as per the IDM
Known Errors list pulled together for the start of this project.
Other than the move to create a new administration UI (see Section 3.1.7),
there is nothing specific to add to this iteration.
__________________________________________________________________________
Page 14 of 15
IDM Migration – High Level Design
[Version: 1.0]
_____________________________________________________________________________________________
5 References





Grouper evaluation
Mediator resequencing
SOA 11g feature evaluation
SOA Development standards
Software Version and Release Standards
__________________________________________________________________________
Page 15 of 15