Reconciliation Common Errors and Resolution Tips
Reconciliation process is the key to ensure data provider(s) generate the complete and
correct production copy of CI and its relationship. When a CI is not identified or merged
as expected from the corresponding RE job definitions, questions are raised around how
we can find out what goes wrong.
In order to trouble shoot the RE problems, you will need to have RE log for analysis. So
the first step is to turn on Reconciliation Engine log:
1) Go to Reconciliation Engine tab in CMDB console.
2) From the left navigation pane, click “Settings”.
3) From the Server Settings dialogue window. Set “Debug Mode” to True and save
the change.
RE job logs are located at what’s defined in the “Log File Path” field on the Server
Settings form.
There’re many RE common problems which we list here along with error messages in the
RE logs and their resolution tips.

Found multiple matches in find-in-dataset during identification process.
Symptom
RE log message
CI remains unidentified in the source dataset after identification
process is finished.
Sample error messages in the RE log:
[Fri Feb 23 2007 13:40:05.7650] [ ERROR ] [TID:
002188] : Found multiple matches in dataset <BMC
Asset>
[Fri Feb 23 2007 13:40:05.7850] [ ERROR ] [TID:
002188] : Cannot identify the instance of class
<BMC_ComputerSystem> with instance id
<AG005056C00008wQfeRQ66iBAQ0owA>
Root Cause
Resolution

Find-in-dataset (BMC.Asset in this example) has two or more CIs
which share the same value on identification attribute.
1) Search for CI based on instance id (In our sample log,
instance id is AG005056C00008wQfeRQ66iBAQ0owA.).
2) Search CIs from find-in-dataset (BMC.Asset) based on
identification attribute values of the CI from step 1). For
example, if TokenId attribute is used in the identification
rule, search for CI based on dataset id equals find-in-dataset
id and TokenId equals to what you see from the CI instance
from step 1).
3) If multiple CI instances are found from step 2), delete the
duplicates entry so only one CI exists in find-in-dataset; if
multiple CI instances refer to different CIs, re-examine the
identification rule or the data quality and make appropriate
adjustment.
4) Run identification process to get the CI identified.
Unique index violation during identification process.
Symptom
RE log message
CI remains unidentified in the source dataset after identification
process is finished.
Sample error messages in the RE log:
[Fri Feb 23 2007 14:17:37.1120] [ ERROR ] [TID:
002188] : ARERR[120092] The dataset ID and
Reconciliation Identity combination is not unique.
Root Cause
Resolution
There’s a duplicate identified CI in the source dataset.
1) Search for CI based on instance id.
2) Search CIs from find-in-dataset (BMC.Asset) based on
identification attribute values of the CI from step 1). For
example, if TokenId attribute is used in the identification
rule, search for CI based on dataset id equals find-in-dataset
id and TokenId equals to what you see from the CI instance
from step 1). Make a copy of ReconciliatonIdentity attribute
value if a CI is found.
3) Search for CI based on source dataset id and
ReconciliationIdentity value from step 2).
4) Check to see whether the found CI is marked as softdeleted. If it’s marked as soft-deleted, delete the record; if
it’s not marked as soft-deleted, re-examine the identification
rule and make appropriate modifications.
5) Run identification process to get the CI identified.

Missing end points during merge process.
Symptom
Relationship CI can’t be found from BMC.Asset after merge
process is finished while such identified CI exists in source dataset.
RE log message
[Thu Jan 25 2007 11:58:27.9156] [ ERROR ] [TID:
000008] : Cannot find the endpoint of
relationship::BMC_Application
[Thu Jan 25 2007 11:58:27.9161] [ ERROR ] [TID:
000008] : ARERR[0] Cannot find the endpoint of
relationship::BMC_Application
[Thu Jan 25 2007 11:58:27.9162] [ ERROR ] [TID:
000008] : Merging of record failed:
Root Cause
Modification to the CI from source dataset without modifying
corresponding relationship CIs. For example, resetting
reconciliation identity of BMC_ComputerSystem CI without
resetting the associated relationship CIs’ reconciliation identity.
1) Reset reconciliation identity to be “0” for all instances in the
specific dataset.
2) Re-run identification and merge process.
Resolution

Incremental merge becomes full merge.
Symptom
Only a few CIs were modified since last RE merge. But the ongoing merge takes much
longer than usual.
RE log
message
[Fri Feb 23 2007 13:40:08.6690] [ INFO ] [TID: 003172] : Merging class: BMC_ComputerSystem
[Fri Feb 23 2007 13:40:08.7190] [ INFO ] [TID: 003172] : Total number of records found which are
to be merged are <527867> This number is much bigger than anticipated.
Root
Cause
Merge activity’s “Last Activity Time” is reset to NULL for the following reasons:
1) The previous RE job run is aborted.
2) RE job definition such as schedule or merge algorithm is changed.
Resolution:
1) Open “RE:Activity” form with Remedy Administrator Tool. Scroll down to the
bottom part of the form to view page holder with pages System_tab & Debug_tab.
Make sure the page holder is not set to “Hidden”.
2) With the Remedy User Tool, open “Reconciliation Activities” search form and
search for the merge activity and get the activity id from the “instanceId” field.
3) With Remedy User Tool, open search from “RE:Activity_Runs” and fill in
“Activity Instance ID” field with activity id retrieved from previous step and
search.
4) Fill in the appropriate value in both “Last Activity Start Time” and “Last Activity
End Time” fields.
5) Now RE merge process should perform on delta basis.

Merge algorithm selection.
Symptom
The same CI seems to be merged more than once during merge
process.
RE log message
[Fri Feb 23 2007 13:40:08.6090] [ INFO ] [TID:
003172] : Algorithm: Including child CIs and
committing together
Root Cause
Resolution
Unnecessary complicated merge algorithm selection.
If no selectivity is needed for merge activity, choose “Stand Alone”
algorithm.

Run RE with a single thread.
Symptom
RE has poor performance in all activities.
RE log message
Root Cause
Resolution

RE log only shows two unique TIDs (Thread ID).
You forget that RE can be set with multiple threads.
Use Remedy Admin Tool to set RE to be multi-threaded.
RE has poor performance due to incorrect database settings.
Symptom
RE log message
Root Cause
Resolution
RE has poor performance in all activities.
Oracle 10g has default cursor_sharing set to be ‘exact’.
Oracle database instance has small memory allocation.
SQL server has small tempdb space allocation.
Set cursor_sharing to be ‘force’ or ‘similar’ to boost
performance by up to 30%.
2) Get DBA involved ( run statspack to identify the database
performance bottleneck ).
3) Allocate more dedicated fast disk space for SQL Server
tempdb.
1)
2)
3)
1)
Database indexing can be critical to RE performance. These areas
require good attention.
 Identification attributes are indexed properly.
 The out of box BMC Atrium indexes are not
modified or deleted.
 Index statistics are updated regularly.

RE aborted due to temporary disk space limit.
Symptom
RE log message
RE aborted.
Root Cause
Reconciliation process leads AR API requesting temporary disk
space which is not available in the file system.
Add more disk space to the server TEMP disk or redirect TEMP to
another disk which has more space.
Resolution

[Wed Nov 01 02:47:50 2006] [ ERROR ] [TID: 002836] :
ARERR[316] Failure with a temporary file
[Wed Nov 01 02:47:50 2006] [ ERROR ] [TID: 002836] :
No space left on device
No memory available during reconciliation process.
Symptom
RE log message
Root Cause
Resolution
RE aborted.
ARERR 300 Malloc failed on server
Memory usage is too high to allow new memory allocation.
1) Make sure the server has enough memory resource. We
recommend CMDB/RE is configured to be on a separate
server so there will be no share resource for CMDB/RE.
2) Check virtual memory setting to make sure there’s enough
virtual memory space. Due to intensive data manipulation,
there may be needs to use more than what’s available in
system physical memory.
3) Fine tune RE multi-thread number. When using “include
child CIs” algorithms, reducing the number of RE threads
will decrease the required memory by Reconciliation
process.

AR server timeout issue.
Symptom
RE log message
RE aborted or RE seems to be hanging.
[Mon Oct 16 20:10:43 2006] [ ERROR ] [TID: 005256] : Failed in
getting instances from class <BMC_PRODUCT>::
[Mon Oct 16 20:10:43 2006] [ ERROR ] [TID: 005256] :
ARERR[94] Timeout during database query -- consider using more
specific search criteria to narrow the results, and retry the operation
[Thu Nov 9 10:51:23 2006] [ ERROR ] [TID: 000008] : Error
while loading instances with api AROSGetMultipleInstances()::
[Thu Nov 9 10:51:23 2006] [ ERROR ] [TID: 000008] :
ARERR[93] Timeout during data retrieval due to busy
server -- retry the operation
Root Cause
Resolution
Bad SQL statement by AR or RE.
1) There’s a known issue with AR generated SQL statement
for AR joint form if there’re attachment/attachment pool
fields involved. Workaround can be either to delete the
unused attachment/attachment pool fields or to modify the
joint form SQL statement manually.
2) If timeout happens when RE is merging millions of CIs
from a specific class, we can divide the merge process into
smaller RE jobs to limit the data retrieved from AR server.
For example, we can define multiple jobs of merging
BMC_Product class grouping by alphabetical name ranges
and having one job trigger the other one to start with
“execute job” activity sequentially.
3) Consult with DBA for troubleshooting.