Infosys
Documentum XPLORE Indexing And Configuration
Infosys
Documentum XPLORE Indexing And Configuration
April, 2014
INFOSYS LIMITED
Bangalore
Document No.
Ver. Rev. :
Authorized by:
Signature/:
Date:
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
COPYRIGHT NOTICE
©2014 Infosys Limited, Bangalore, India. All rights reserved. Infosys believes the
information in this document is accurate as of its publication date; such information is
subject to change without notice. Infosys acknowledges the proprietary rights of other
companies to the trademarks, product names and such other intellectual property rights
mentioned in this document. Except as expressly permitted, neither this document nor any
part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by
any means, electronic, mechanical, printing, photocopying, recording or otherwise, without
the prior permission of Infosys Limited and/or any named intellectual property rights
holders under this document.
Infosys Limited
Hosur Road
Electronic City, 3rd Cross
Bangalore 560 100
India.
Telephone: (91) (80)28520 261-270
Fax: (91) (80) 8520 362
Website: http://www.infosys.com
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
Author(s): Mayuri Kothari
Date written (MM/DD/YY): 08/05/2014
Target readers: Documentum users
Project Involved: AZAF2IMS
Keywords: Documentum, Index Server, Index Agent, XPLORE
Purpose of the document: to share project experience and learning, xPlore is
recently launched search engine. It will enhance user’s knowledge in search used
in Documentum
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
Introduction
This guide describes xPlore Indexing management and internal relationship between
fulltext objects for Documentum xPlore server. Microsoft has ended supporting fast search
engine for search in Documentum. Documentum 6.5SP3 and 6.6 are last versions
compatible with fast search engine. Documentum xPlore is now installed with Documentum
versions 6.7SP2 and 7.0 for search in Documentum. Documentum xPlore is bundled with
content server 6.7 and 7.0 and covered by same license.
Infosys
FAST
xPlore
[Template for Internal Tuto
Fig1. Switching from FAST to xPlore
Infosys
Powerful new search features with xPlore improve capability and stability of search in
Documentum. Quality of search is improved, it can support over 80+ languages. xPlore
uses lemmatization to help improve search. Searches for ‘test’ will find documents ‘testing’
and ‘tests’. xPlore stores original form and lemma for each token to ensure exact match for
all languages. xPlore supports 800+ versions of formats. Certain formats can be turned off
for indexing if desired by modifying dm_format object and setting can_index to false.
Documentum xPlore is supported on Linux and windows (physical or virtual) Operating
system. Documentum content server can be run in usually supported operating systems
<Title of Internal
(windows, Linux, Solaris, AIX and HPUX)
Tutorial
<Month, year>
INFOSYS LIMITED
Bangalore
©2014 Infosys Limited, India
Document No.
Ver. Rev. :
Authorized by:
Signature/:
Date:
Infosys
Documentum XPLORE Indexing And Configuration
Internal Relationship between Fulltext Objects:
It is necessary to ensure that all fulltext objects are configured properly on Index server.
PFB diagrams which explains relationship between xPlore objects.
dmi_queue_item
dm_ftindex_agent_config
Name
Name
…..
Queue user
Index_name
…..
Fig2. Relation between objects dmi_queue_item and dm_ftindex_agent_config
dm_ftindex_agent_config
dm_fulltext_index
Index_name
Ftengine_id
Is_standby
Name
Indexagent.xml
Queue user
<instance_name>
Index_name
…..
…..
Install_loc
Fig3. Relation between objects dm_fulltext_index, dm_ftindex_agent_config and
indexagent.xml
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
dm_location
dm_fulltext_index
dm_ftengine_config
Object_name
Index_name
R_object_id
…..
Ftengine_id
Filter_config_id
Is_standby
…..
Install_loc
Fig4. Relation between dm_location, dm_fulltext_index and dm_ftengine_config
dm_ftfilter_config
R_object_id
Types_to_exclude
Cabinets_to_exclude
dm_ftengine_config
R_object_id
Filter_config_id
…..
Folders_to_exclude
….
Fig5. Relation between dm_ftfilter_config and dm_ftengine_config
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
You can have quick check on all configurations for fulltext objects by simple DQL and API
queries. Below are some of the DQL and API queries.
1. DQL >Select * from dm_fulltext_index
Go
2. DQL>Select * from dm_ftengine_id
Go
3. API>dump,c,r_object_id
To update objects from standby to active, you need to run update query to make changes in
database. To run SQL query from DQL use below query
1. EXECUTE exec_SQL with query='update dm_fulltext_index_s objects set
is_standby=0 where r_object_id=''r_object_id'''
To update objects from active to standby, you need to run update query to make changes in
database. To run SQL query from DQL use below query
1. EXECUTE exec_SQL with query='update dm_fulltext_index_s objects set
is_standby=1 where r_object_id=''r_object_id'''
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
xPlore Indexing:
PART A: INTRODUCTION
After the index agent is installed successfully, you need to start Index agent from URL. In
browser enter the URL that contains the Index Agent host DNS name and port number.
http://host:port/IndexAgent/login_dss.jsp
Index agent can be started in normal mode or migration mode. If you start index agent in
normal mode, it indexes content that is added to or modified in repository from that point of
time. If you start index agent with option ‘start new re-indexing operation’, it indexes all
content in repository. object_types that are registered for full-text indexing in dmi_registry
are indexed, then dm_acl and dm_group objects are indexed. If you start indexing with
option ‘continue re-indexing’, it will start indexing from time when you have stopped
indexing.
Get the id of DSearch Fulltext Engine Configuration by using simple DQL query
DQL> select r_object_id, object_name from dm_ftengine_config
You need to add the following line to server.ini file and restart content server and xPlore
services
ftengine_to_use = <r_object_id>
To find object_types that are registered for full-text indexing can be found using below
DQL query
DQL> select name from dm_type where r_object_id in (select distinct registered_id from
dmi_registry where user_name='dm_fulltext_index_user')
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
xPlore Indexing:
PART B: TO SPEED UP INDEXING
Sometimes it is observed that indexing is slow. If count of total documents to be processed
is large users can not afford to wait for long to complete the indexing. In that case you need
to speed up the ongoing indexing. Indexing is in total related to process and memory hand
in hand. You can follow below recommended changes to speed up the indexing.
1. Increase XMX memory in startup script of Dsearch

Stop all xPlore services

Edit the script that launches an xPlore server. Script is located at,
Dsearch_home/jboss5.1.0/server

Increase the value of XMX, default is 2GB increase it to 8GB
USER_MEM_ARGS="-Xms256m -Xmx2048m -XX:PermSize=64m -XX:MaxPermSize=256m Xss1024k
USER_MEM_ARGS="-Xms256m –Xmx8192m -XX:PermSize=64m -XX:MaxPermSize=256m
-Xss1024k

Start all xPlore services
2. Tune xDB.properties increase memory parameter

Stop all xPlore services

Edit xDB.properties located at
Dsearch_home/jboss5.1.0/server/DctmServer_PrimaryDsearch/deploy/dsearch.war/
WEB-INF/classes


XHIVE_MAX_MEMORY=1024M

XHIVE_MIN_MEMORY=128M
Default values for XHIVE_MAX_MEMORY and XHIVE_MIN_MEMORY are as below

XHIVE_MAX_MEMORY=128M

XHIVE_MIN_MEMORY=32M
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
3. Increase CPS daemon count – Multiple CPS daemons for indexing

Login to the dsearchadmin console

Expand Instances | PrimaryDsearch and click Content Processing Service

Click Configuration tab

If Daemon count is set to 1 or nothing, change it to 2

Retsart CPS Instance
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
xPlore Indexing:
PART C: ERRORS OBSERVERVED DURING INDEXING AND RESOLUTIONS
1. ERROR WorkerThread
[Worker:ExportAction:#2][DM_INDEX_AGENT_EXCEPTION_IN_WORKER_THREAD]
"Exception in Worker Thread. Message: {1}"
com.documentum.fc.common.DfRuntimeException:
[DFC_PRIVILEGE_ERROR_ABUSE_ATTEMPT] failed attempt to abuse escalation:
'incompatible roles found for dmc_policy_coordinator'

The above error can be removed by clearing BOF cache from xPlore as well as
content server

xPlore Path Dsearch_home/jboss5.1.0/server/DctmServer_Indexagent/data/IndexAgent/c
ache/6.7.xx

CS path ->$DOCUMENTUM_SHARED/cache

Stop all Documentum related services on content server and xPlore server and Index
agent on xPlore server

Clear the BOF cache from xPlore server and content server as per above location

Start Documentum services and xPlore services

Start Re-Indexing
2. ERROR WorkerThread
[Worker:ExportAction:#4][DM_INDEX_AGENT_EXCEPTION_IN_WORKER_THREAD]
"Exception in Worker Thread. Message: {1}"
com.documentum.fc.common.DfRuntimeException: [DFC_PRIVILEGE_EMPTY_ROLE_CLAIM]
"privileged entity name 'dmc_policy_coordinator' claims no escalation roles"

The above error can be removed by DAR files re-installation and clearing
BOF cache from content and xPlore server

Stop all Documentum related services on content server and xPlore server
and Index agent on xPlore server

Clear the BOF cache from xPlore server and content server as per above
location

Start Documentum services and xPlore services
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration

Start Re-Indexing
3. DOCUMENT_WARNING CPS Warning [Data too large to perform linguistic processing
Text size 19644164, mime (Microsoft Excel 2010 Workbook),threshold 10485760

Change the max_text_threshold value to 20MB in
PrimaryDsearch_local_configuration.xml file. Default threshold value is set to
10MB

This file is located at dsearch_home/dsearch/cps/cps_daemon location on
xPlore server

Stop xPlore server and Index agent

Edit PrimaryDsearch_local_configuration.xml file
<max_text_threshold>20971520</max_text_threshold>

Start xPlore server and Index agent

Start Re-Indexing
4. WARN PrepWorkItem [Worker:ExportAction:#4]Content size for 0902152d80164361
id: 0602152d800bff42 58293844 exceeds limit of 41943040 skipping content

Edit contentSizeLimit value in indexagent.xml file on xPlore Server

Default value for contentSizeLimit is 20MB , in above error contentSizeLimit
was set to 40MB

Indexagent.xml file is located at
Dsearch_home/jboss5.1.0/server/DctmServer_Indexagent/deploy/IndexAgen
t.war/WEB-INF/classes

Stop Index agent on xPlore server

Edit indexagent.xml file on xPlore Server
<parameter_name>contentSizeLimit</parameter_name>
<parameter_value>3000000000</parameter_value>

Change contentSizeLimit as per the requirement(in bytes), it is size of documents
submitted for indexing

Start Index agent on xPlore server

Start Re-Indexing
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
5. 2014-04-22 13:56:43,987 ERROR IndexingStatus
[PollStatusThread][DM_INDEX_AGENT_PLUGIN] Document represented by key
0902152d8040862d failed to index into earchive, error:Content Processing Service
Exception: [Batch request error. CPS error code 5, error message Time out].

Navigate to dsearchadmin console -> instances -> PrimaryDsearch -> CPS >configuration -> edit request timeout time ( set it to 1200 and check , if
error occurs again increase it to 2500) default request time out is set to 600
6. [dmadmin tools]$ ./ftintegrity_for_earchive.sh
Executing stateofindex
Connected to the docbase earchive
DfIdNotFoundException:: THREAD: main; MSG:
[DM_API_E_EXIST]error: "Document/object specified by 0802152d80000d5f does not
exist."; ERRORCODE: 100; NEXT: null
at
om.documentum.fc.client.impl.docbase.DocbaseExceptionMapper.newException(DocbaseExc
eptionMapper.java:49)
The above error have occured while running ftintegrity tool. After analysis it was observered
that there was configuration error, server was on standby, below DQL queries can be used
to remove particluar error by making server to active state.

EXECUTE exec_SQL with query='update dm_fulltext_index_s objects set
is_standby=1 where r_object_id=''<r_object_id>'''
EXECUTE exec_SQL with query='update dm_fulltext_index_s objects set
is_standby=0 where r_object_id=''<r_object_id>'''
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
xPlore Indexing:
PART D: RUNNING FTINTEGRITY TOOL
ftintegrity and the state of index job (in Content Server 6.7) are used to verify indexing
after migration or normal indexing. ftintegrity tool can be run in normal mode only.
To run ftintegrity tool Navigate to dsearch_home/setup/indexagent/tools.

Open the script ftintegrity_for_repositoryname.sh (Linux) Substitute the repository
instance owner password in the script (replace <password> with your password).

Run the script ftintegrity_for_repositoryname.sh
ftintegrity result files and their interpretation :
After you run ftintegrity script, 4 files are created at location
dsearch_home/setup/indexagent/tools. PFB file names which are created:

ObjectId-common-version-match.txt

ObjectId-common-version-mismatch.txt

ObjectId-dctmOnly.txt: This file will contain object ids of document which have
failed indexing. You can re-submit this file for indexing manually.

ObjectId-indexOnly.txt
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
xPlore Indexing:
PART E: MANUALLY INDEXING FAILED OBJECTS
Copy the text file ObjectId-dctmOnly.txt after running ftintegrity tool. Steps to run
ftingretity are mentioned in document above.
Send this file for indexing, from Index Agent URL, PFB snapshot.
Click on object file, click on browse and select particular file containing r_object_ids of
objects to be indexed.
©2014 Infosys Limited, India
Infosys
Documentum XPLORE Indexing And Configuration
Acknowledgements: Mandeep Behal, Visharad Sharma
References:
1. Documentum- xPlore 1.2 Administration-and-development-Guide
©2014 Infosys Limited, India