What’s New in
OpenEdge Replication?
June 5, 2017
• Jeff Owen, Dapeng Wu, Safina Khanem, Jim Clark
Agenda
 PICA Queue Monitoring
 Agent Restart
 Replication Sets
 Replication Monitor, VSTs, DSRUTIL, builddb Utility
2
10.2B08 and 11.3.0: Improved PICA Queue Monitoring
3
10.2B08: PICA Queue Monitoring Improvements
Display Db Service Manager Screen
04/27/17
Status: Database Service Manager
Communication Area Size :
8193.09 KB
Total Message Entries
:
74908
Free Message Entries
:
74908
Used Message Entries
:
0
Used HighWater Mark
:
1
Area Filled Count
:
0
Service Latch Holder
:
-1
Access Count
:
5304
Access Collisions
:
0
4
Useful for sizing –pica
Is PICA Queue filling
often?
Is there contention for
the queue?
11.6.0: Restart Agent
5
What is Restart Agent?
 The ability to restart the Replication Agent without restarting the target database
 Command - dsrutil <targetdbname> –C restart agent
 Helpful in a scenario where replication is stopped or agent has been terminated and needs
to be restarted without shutting down the target database and having to restart the database
to allow the agent to connect with the Replication server.
6
Restart Agent - Use Cases
1
Improvements to disaster recovery
environment
2
Make changes to properties files
3
Agent terminates due to timeout while applying
AI Files
7
11.7.0: Synchronous Commit Mode For Two Targets
8
Two target support for Synchronous Commit Mode
Source Replica
AI Block
AI Block
Commit!
Commit!
Target Replica One
Target Replica Two
9
11.7.0: Replication Sets
10
Replication Set:
A replication environment that
contains a source replica and
two target replicas that can
transition together
Replication Set Inter-agent Communication
12
Inter-agent Communication
 Allows the targets to transition together if the source database is gone.
 One becomes new source, the other remains a target and replication can continue.
 Targets communicate their location relative to source.
 Used to determine if AI extents can be unlocked.
 If one target is behind, the healthier target may need to keep a number of extents locked to
synchronize with the less healthy target.
 Allows a health check at the beginning of a recovery transition to determine
coordinator.
13
Inter-agent Communication
 Specify replication-set=1 in the [transition] section of the properties file
[transition]
replication-set=1
database-role=reverse
transition-to-agents=agent1,agent2
restart-after-transition=1
source-startup-arguments=-S 51902 -pi replserv
 When the agents receive the inter-agent communication message from the server,
they will initiate their connections and send a response to the server.
14
What if…
 Server goes offline?
 Targets remain connected if they remain running
 When server is restarted, it will re-initiate the network.
 Agent goes offline?
 Replication between source and other agent continues.
 When the agent is restarted and server told to connect to it, the network will be re-initiated.
 Agents are started without Server?
 Agents will depend on their properties files to connect to each other.
15
Inter-agent Communication Visibility
 Messages in database log file:
 Source: (18458) Attempting to establish inter-agent network connection on agent <agent name>.
 Target: (18476) The inter-agent network on agent <agent name> is complete.
 Monitor and VST enhancements to track the status on target database.
 New _Repl-InterAgentActivity VST
 Replication inter-agent status screen in dsrutil –C monitor
 New status on Replication Server for initiating and recovering inter-agent connections.
 Short-lived status, unlikely to be seen unless there is a problem with an agent responding.
16
AI Extents Remaining Locked for a Long Time on a Target
 Check that the agents are connected to each other:
 Through monitor or VST.
 Verify the local agent’s information for remote agent’s last acknowledged TX Sequence.
Inter-agent status screen in monitor.
_ReplIntAgtAct-RemoteLastBlockSeq and _ReplIntAgtAct-RemoteSourceTRID
 Check if an agent is falling significantly behind the source.
17
Inter-agent Status in Monitor
OpenEdge Replication Monitor
Database:
Page 1
/testdb/target2
Database is enabled as OpenEdge Replication:
Target
Local Agent:
ID:
Host name:
Port:
Server latency at last message:
Synchronization points:
agent2
2
127.0.0.1
4388
0
second(s)
1
Remote Agent:
ID:
Host name:
Port:
Database name:
Connection state:
Last message sent at:
Last message received at:
Last ping sent at:
Last ping received at:
Messages sent:
Messages received:
Synchronization points:
Last acknowledged Source AI sequence:
Last acknowledged Transaction:
agent1
1
localhost
4387
/testdb/target1
Connected
Wed May 3 15:42:23
Wed May 3 15:42:23
Wed May 3 15:42:23
Wed May 3 15:42:23
266
266
1
2
22
18
2017
2017
2017
2017
Fail Over Transition
19
Fail Over Transition:
A transition where a source
replica and a target replica
switch roles
Initiating Failing Over
dsrutil <source_db_name> -C transition failover
So how is the target database specified?
transition-to-agents=agent1,agent2
21
Fail Over Transition Configuration
Target One
Source
Target Two
Target One
Source
22
When is Transition Failover Useful?
 Maintenance is required for the server hosting the source replica
 The server hosting the source replica is being replaced
 The server hosting the source replica is being moved
23
Best Practices for Fail Over Transition
 All replica properties files must have these properties set:
transition=manual
role=reverse
agent-shutdown-action=recovery
replication-set=1
transition-to-agents=agent1,agent2
24
Best Practices for Fail Over Transition
 All replica properties files should define a [control-agents.xxx] section(s)
 Define the same transition-to-agents property in each replica:
 transition-to-agents=agent1,agent2
 All replica properties files should define an [agent] section
25
Best Practices to Revert Configuration
 Edit the current source replica’s properties file, identifying new target replica:
 transition-to-agents=agent0
 Terminate and restart the replication server running on replica 1
 Trigger role switch for the replicas:
 dsrutil targetdb1 –C transition failover
 After transition completes the original configuration is restored:
 The newest target replica will have reverted back to is original role as a source
 The new source replica will have reverted back to is original role as a target
 Edit previous source replica’s properties file, restoring the original values:
 transition-to-agents=agent1,agent2
 Terminate and restart agent1 on replica 1
26
Replication Set: Recovery Transition
27
Recovery Transition:
The transition of a database
from the role of a target replica
to a source replica after the
failure of the original source
replica
Recovery Transition
 Database role changes after a failure in the source database
 Target can transition to a new source, or a normal database
Source
 Possible failures:
 Source database crashes
 Site failures
 Storage failures
 Network failures
Target Two
Target One
Inter-Agent
Connection
29
11.7.0: Two Ways to Configure Recovery Transition
 Manual Transition

Transition manually with “dsrutil –C transition” command

Source is lost, but AI data still available

Apply AI extents before transition

Recover as much data as possible

Targets can be restarted after failures, then begin transition

[Server] properties:
transition=manual
 Auto Transition

Transition happens automatically after “transition-timeout”

Transition as soon as possible (HA)

[Server] properties:
transition=auto
30
11.7.0: Which Agent will transition?
 Priority of the agents to transition is identified in the properties file.
 The transition-to-agents property specifies the priority:
[transition]
replication-set=1
database-role=reverse
transition-to-agents=agent1,agent2
restart-after-transition=1
source-startup-arguments=-S 51876 -pi replserv
31
Introducing the Transition Coordinator
 A healthier Target that serves as temporary Source to bring remaining targets
up to date

Doesn’t have to be the same as the priority target
 Coordination Phase:

A phase in recovery transition process

Perform health check on the targets at the beginning of the transition

Decide which target will become the coordinator

Target databases will temporarily switch roles

Synchronize with the less healthy target
32
Transition Coordinator: Synchronization with less healthy target
Source
Target 2
Target 1
Transaction #1
Transaction #2
Source crash!
Health Check and Synchronization
Transaction #2
33
How Does the Coordinator Work?
 Which target will be the coordinator if the source is lost?
 Which AI extents will the coordinator use to bring the other target up to date?
 Which AI extents on a target can be unlocked and recycled?
34
11.7.0: From where to synchronize with the less healthy target?
TxnSeq
AI Seq
100
1
200
2
300
3
TxnSeq
AI Seq
AI extent (.a1)
50
1
AI extent (.a2)
100
2
200
3
AI extent (.a1)
AI extent (.a2)
Target 1
Source
AI extent (.a3)
TxnSeq Exchange
AI extent (.a1)
AI extent (.a2)
AI extent (.a3)
Target 2
AI extent (.a3)
Source
Txn Seq/AI Seq
Target 1
Local/Remote TxnSeq
Target 2
Local/Remote TxnSeq
100/1
100/50
50/100
200/2
200/100
100/200
300/3
300/200
200/300

Which target will be the coordinator if the source is lost?

Which AI extents will the coordinator use to bring the other target up to date?

Which AI extents on Target 1 can be unlocked and recycled?
35
11.7.0: Recovery Transition step by step
Source
Target 1
Target 2
Source crash!
Enter Pre-Transition
Enter Pre-Transition
Transition Starts
Determine the Coordinator
Target1 is the Coordinator
OR Target2 is the Coordinator
Coordinator starts up and synchronizes with the other Target
Database role changes
The Source starts up and synchronizes with the new Target
Time
New Target
New Source
36
11.7.0: Best Practices For Configuring Recovery Transition
 All databases should have identical transition-to-agents properties
 Consider a larger connection-timeout value for the targets
 Allow for time to apply AI extents to the priority target, if desired
 Consider the time required to reseed a new target when:
 Setting the control-agents properties
 control-agents=agent2 for priority target (agent1)
 Setting the connection-timeout properties
 Manage AI with AI Archiver on Source and Targets
 Recovery Backup
37
11.6.0: Replication Virtual System Tables: Enhancements
38
What’s New With VSTs
1
Three VST tables expanded
2
Four new VST tables
3
Static vs. Dynamic distinction in tables
39
Expanded VSTs
1
_Repl-Server
2
_Repl-Agent
3
_Repl-AgentControl
40
New VSTs
1
_DbServiceManager
2
_DbServiceManagerObjects
3
_Repl-AgentActivity
4
_Repl-AgentControlActivity
41
11.7.0: Replication Virtual System Tables: Enhancements
42
What’s New With VSTs
1
Replication Properties
2
New VST: Inter-agent Communication Data
43
What’s New With VSTs
1
_Repl-Server and _Repl-Agent
2
_Repl-InterAgentActivity
44
11.7.0: Replication Monitor Enhancements
45
Replication Monitor Enhancements
 Prior to 11.7, the monitor was missing some information found in only the VST.
 All additions to existing screens are at the bottom.
 Should not break your screen scraper
 Notable enhancements include addition of transition properties to the agent and server
status screen, as well as other properties.
 Allows sanity checks on transition behavior before invoking transition!
 A new Inter-agent status screen on target databases.
46
11.7.0 builddb Enhancement
47
11.7.0: Prostruct builddb support
 Re-creates a control area (.db) from the structure description (.st) of an existing
database.
Used to recover when an existing database control area is lost or damaged.
 Prior to OpenEdge 11.7, the command will disable replication and cause a rebase.
 In OpenEdge 11.7, replication will remain enabled and no rebase is needed.
48
11.7.0: DSRUTIL Status Verbose
49
dsrutil status verbose
Example:
Command Line:
dsrutil db1 -C status agent1 -verbose
3048: Startup Synchronization
50
For more information…
 Demonstrations of the new features will be available at the Reception and Expo
 Other sessions for your consideration:
 541: What’s new with the VSTs
 419: OpenEdge Roadmap and Vision
 437: OpenEdge Deployment Info Exchange
51
We Have Questions for you!
52
Questions for us?
53
Appendix 1: Replication server state values and their description
Status
Description
0
6001
Unknown
Server Initialization
6002
Connecting to Agents
6003
Configuring Agent(s)
6004
Recovery Processing
6005
Startup Synchronization
6021
Normal Processing
6022
Performing Transition
6023
Replication is Suspended
6060
Server is ended
54
Appendix 2: Replication Agent state values and their description
Status
Description
Status
Description
Status
Description
0
1001
Unknown
Initial Connection
1038
1063
Agent is Terminated
Agent is Ended
3051
Online backup of Target
Database
1002
Initializing
2080
Pre Transition
3052
Target Database in a Quiet Point
1003
2081
Applying After-image Extent
3053
Target Database in BI Stall
1032
Target Database in Quiet
Point
Initial Connection Failed
2082
Transitioning
3054
Target Database in AI Stall
1033
Recovery Failed
2083
Listening
3055
Being Transitioned
1034
Invalid Target Database
Configuration
Agent Failed
2084
3048
Waiting while JTA transactions
are resolved
Startup Synchronization
3049
Normal Processing
3050
Recovery Synchronization
1035
1036
1037
Agent is Ignored
Agent is Stopped
55