IBM Initiate Workbench
CloverETL User's Guide
Version 9 Release 7
SC19-3165-02
IBM Initiate Workbench
CloverETL User's Guide
Version 9 Release 7
SC19-3165-02
Note
Before using this information and the product that it supports, read the information in “Notices and trademarks” on page
87.
© Copyright IBM Corporation 1995, 2011.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
Contents
Introduction to CloverETL in IBM Initiate
Workbench . . . . . . . . . . . . . v
Where to go for more information on CloverETL
System requirements for CloverETL . . . .
How to get help . . . . . . . . . . .
Technical Support Contacts . . . . . .
On the web . . . . . . . . . . .
By telephone . . . . . . . . . . .
.
.
.
.
.
.
. v
. v
. vi
. vi
. vi
. vi
Chapter 1. CloverETL features and
benefits . . . . . . . . . . . . . . . 1
Sample graphs in IBM Initiate Workbench . . .
About Master Data Extract . . . . . . .
Using the CloverETL interface . . . . . . .
Switching to the Clover perspective . . . .
The CloverETL layout . . . . . . . . .
Callout handler registration for CloverETL graphs .
.
.
.
.
.
.
1
2
2
2
2
3
Chapter 2. CloverETL graph components 5
Overview of CloverETL components . . . . . . 5
CloverETL Readers . . . . . . . . . . . 5
CloverETL Transformers . . . . . . . . . 5
CloverETL Joiners . . . . . . . . . . . 5
CloverETL Writers . . . . . . . . . . . 5
CloverETL Edges . . . . . . . . . . . . 6
Other CloverETL components . . . . . . . 6
Deprecated CloverETL components . . . . . . 6
Working with CloverETL graphs. . . . . . . . 6
Preparing the CloverETL extract file – best
practices . . . . . . . . . . . . . . . 6
Using a CloverETL sample graph . . . . . . 7
Building a CloverETL graph . . . . . . . . 7
Adding and configuring a CloverETL Reader . . 8
Adding and configuring a CloverETL Transformer 9
Using CloverETL Edges to connect components
10
Adding and configuring a CloverETL Writer . . 12
Connections in CloverETL . . . . . . . . . 13
Creating a database connection for CloverETL
graph execution . . . . . . . . . . . . 13
Creating an Initiate connection for CloverETL
graph execution . . . . . . . . . . . . 14
Creating a JMS Connection for CloverETL graph
execution . . . . . . . . . . . . . . 14
Executing a CloverETL graph . . . . . . . . 15
Executing CloverETL graphs with the madconfig
launch_etl utility . . . . . . . . . . . 15
Using madconfig launch_etl to execute a
CloverETL graph . . . . . . . . . . . 16
Recording responses to the madconfig utility . . 16
Viewing logs and error messages about CloverETL
graph execution . . . . . . . . . . . . . 17
The CloverETL JMS Reader component . . . . . 17
How the CloverETL JMS Reader component and
Edges process messages . . . . . . . . . 18
© Copyright IBM Corp. 1995, 2011
Using the CloverETL JMS Reader components's
timeouts to manage message flow . . . . . .
Limiting the CloverETL JMS Reader component
max message count for debugging. . . . . .
Using the CloverETL JMS Reader max message
counts and timeouts together . . . . . . .
Managing the CloverETL Edge's message buffer
for immediate message processing . . . . . .
The CloverETL Clean and Deduplicate Sample
Graph . . . . . . . . . . . . . . . .
Linking The CloverETL Clean and Deduplicate
sample Reader to a source file . . . . . . .
Adding metadata for the CloverETL Clean and
Deduplicate sample graph . . . . . . . .
Editing the Ext Filter component in the
CloverETL Clean and Deduplicate sample graph .
Specifying a sort key in the Ext Sort component
of the CloverETL Clean and Deduplicate sample
graph . . . . . . . . . . . . . . .
Specifying a dedup key in the CloverETL Clean
and Deduplicate the sample graph . . . . .
Writers in the CloverETL Clean and Deduplicate
sample graph . . . . . . . . . . . . .
The IBM Initiate Member Model Transform Graph
Input metadata in the Member Model Transform
graph . . . . . . . . . . . . . . .
Linking to source data files and output
directories in the Member Model Transform
graph . . . . . . . . . . . . . . .
Configuring the IBM Initiate Member Model
Transform Graph . . . . . . . . . . .
The CloverETL Extract Household Graph . . . .
Creating a project for the CloverETL Extract
Household graph . . . . . . . . . . .
Customizing the extract_household graph . . .
19
19
19
20
20
21
21
21
23
23
24
24
25
26
27
28
28
28
Chapter 3. IBM Initiate custom
components . . . . . . . . . . . . 31
Using Key fields in CloverETL MemPut,
MemSearch, TskSearch, TskPut, and Handler Writer
to join records . . . . . . . . . . . .
IBM Initiate custom Reader components . . . .
Comparing the IBM Initiate custom Reader
components . . . . . . . . . . . .
The IBM Initiate MemGet component. . . .
The IBM Initiate MemSearch component. . .
The IBM Initiate Handler Reader component .
The IBM Initiate TskGet component . . . .
The IBM Initiate TskSearch component . . .
IBM Initiate custom Writer components . . . .
The IBM Initiate MemPut component. . . .
The IBM Initiate MemDelete component . . .
The IBM Initiate Handler Writer component .
The IBM Initiate TskPut Component . . . .
. 31
. 32
.
.
.
.
.
.
.
.
.
.
.
32
32
36
40
42
46
50
50
54
56
57
iii
IBM Initiate custom Transformer components for
CloverETL. . . . . . . . . . . . . .
The IBM Initiate MemScore component . . .
The IBM Initiate Attribute Reformat component
The IBM Initiate MemSeqNoIncrementer
component . . . . . . . . . . . .
IBM Initiate custom address-standardization
components for CloverETL . . . . . . . .
Attribute requirements for address
standardization in CloverETL . . . . . .
Deciding between batch and transactional
standardization in CloverETL . . . . . .
The CloverETL IBM Initiate Address Verification
Interface component . . . . . . . . .
iv
CloverETL User's Guide
. 63
. 64
67
Configuring the CloverETL standardization and
validation graph . . . . . . . . . . . . 79
IBM Initiate Address Verification Interface
transliteration in CloverETL . . . . . . . . 83
. 69
Legal Statement. . . . . . . . . . . 85
. 71
Notices and trademarks . . . . . . . 87
. 71
Index . . . . . . . . . . . . . . . 91
. 71
Contacting IBM . . . . . . . . . . . 93
. 72
Introduction to CloverETL in IBM Initiate Workbench
The CloverETL application in IBM® Initiate® Workbench enables you to ensure data
quality easily by executing extract, transform and load (ETL) operations in a
graphical format that is similar to a flow chart.
ETL operations are a part of Master Data Management that enable you to
v extract data from a source or sources, such as the IBM Initiate Master Data
Service® or flat files
v transform the data format, table structure, or other characteristics into a data
model that fits your data target
v load the transformed data into the target, for example a data warehouse, or even
back into the IBM Initiate Master Data Service
Within the CloverETL graph editor, you can create the flow chart structure, which
is called a graph or template, by dragging and dropping graphical components. The
configuration behind the graph is stored as an editable XML document; however,
many users find that the graphical format is much easier to work with.
CloverETL takes advantage of the IBM Initiate Master Data Service APIs by
providing many components that perform interactions such as MemPut,
MemSearch, and MemGet.
Where to go for more information on CloverETL
The IBM Initiate Workbench Clover User's Guide is not intended to provide detailed
information about all CloverETL functionality; rather, it discusses specific
components and procedures that are specific to the IBM Initiate suite of products.
For more information on the CloverETL application, refer to the following
documents and resources:
v The CloverETL user documentation, available via the IBM Initiate Workbench
Help menu, or at http://www.cloveretl.com/documentation
v The CloverETL wiki at http://wiki.clovergui.net/doku.php
For related information on IBM Initiate products, refer to the following documents:
v IBM Initiate Workbench User's Guide
v IBM Initiate Workbench Installation Guide
For additional information on address standardization solutions, refer to
supplemental documentation as follows:
v IBM Initiate Address Verification Interface: See the IBM Initiate Address
Verification Interface proprietary documentation that was provided in PDF
format with your IBM Initiate Address Verification Interface software
System requirements for CloverETL
This document assumes that IBM Initiate Workbench has been installed according
to the instructions published in the IBM Initiate Workbench Installation Guide. Refer
to IBM Initiate Master Data Service System Requirements for detailed information.
© Copyright IBM Corp. 1995, 2011
v
How to get help
You can get additional help for all products by contacting your organization's
Technical Support Contact. You can also find additional information on the web, or
get help by telephone.
Technical Support Contacts
Each organization should designate two (2) or more individuals to serve as
Technical Support Contacts to interface with IBM Software Support. These
individuals should form an internal core software support team who become
experts in the usage and maintenance of the IBM Initiate software
If you have questions or concerns about the software, and if the information in this
guide does not answer your questions, contact your internal Technical Support
Contacts. The Technical Support Contacts can engage the IBM Initiate Master Data
Service Support team to resolve your issue.
On the web
To find help on the web, go to https://www.ibm.com/support/. This site contains
the IBM Initiate Master Data Service library, news, links to web resources, and the
ability to open a Service Request.
By telephone
If you are in North America, call 1-800-IBM-SERV (1-800-426-7378).
If you are outside of North America, check the web page http://www.ibm.com/
planetwide/ for contact information for your area.
vi
CloverETL User's Guide
Chapter 1. CloverETL features and benefits
CloverETL can benefit your business operation in several ways, including the
following scenarios:
Batch inbound processing for a bulk load of data
You want to load a large set of data into the Hub (IBM Initiate Master Data
Service); however, certain data-cleansing processes must be executed before you
can load the data. Running CloverETL can ensure that consistently formatted, clean
data is inserted into the Hub, particularly at the initial load time in an
implementation or when bringing on a new source system.
Batch outbound processing for a bulk extract of data
You want to extract a large set of data from the Hub into some other system, for
example, a reporting database, or you want to synchronize systems. CloverETL
extracts the data from the Hub, runs it through a set of data-cleansing processes,
and ensures that it is properly formatted for use in the target system.
Real-time data load or update
In this ongoing, long running service, one record at a time is run through a
data-cleansing process and then inserted into the Hub. The single transaction
typically uses a graph similar to one that is used for cleansing the data at initial
load.
Real-time data extraction
CloverETL accepts a record from the Hub during this ongoing, long running
service, and processes it through a set of data-cleansing processes to prepare it for
consumption by another system.
In each scenario, CloverETL processing can include an Initiate custom component
that calls the IBM Initiate Address Verification Interface address standardization
feature. As a result, each record's address information is properly formatted for
your business needs, either for temporary use and storage or database update.
Sample graphs in IBM Initiate Workbench
Sample graphs are included with the IBM Initiate Workbench component. You can
customize these graphs to use in your environment.
The following sample graphs are supported in the base IBM Initiate Master Data
Service product:
v Clean and Deduplicate
v Initiate Member Model Transform (as a wizard)
v Extract Full All
v Extract Household
© Copyright IBM Corp. 1995, 2011
1
About Master Data Extract
Master Data Extract is an ETL product which includes wizards that create the
customized extraction graphs:
v Initiate Member Model MMCA (Member Most Current Attributes) graph
v Initiate Member Model EMCA (Entity Most Current Attributes) graph
These graphs extract both incremental and initial MMCA and EMCA data from the
Hub.
Using the CloverETL interface
The following sections describe how to use the CloverETL interface.
Switching to the Clover perspective
To switch to the Clover perspective in IBM Initiate Workbench, follow these steps:
Procedure
1. Click the Open Perspective icon.
2. Choose Other to open the Other Perspective window group.
3. In the Open Perspective window, choose CloverETL.
4. Click OK.
The CloverETL layout
The CloverETL layout utilizes the most common features of the Workbench
development tool, including perspectives, views and editors. Detailed information
about these features can be found in the IBM Initiate Workbench User's Guide.
When using the CloverETL function, you will work in the following areas.
The CloverETL Navigator view
The Navigator view provides a tree structure for browsing Workbench artifacts,
including Workbench projects and graphs.
The CloverETL Outline view
The Outline view gives an overview of the components used in a graph, and
shows graph-related information such as database connections, parameters,
metadata, and sequences.
Within the Outline view, you can toggle between a hierarchical view and a graphical
view of the graph by clicking the view buttons:
The CloverETL Graph editor
The Graph editor is the main workspace for creating, editing, and viewing graphs.
Graphs are created by dragging graph components from the Palette and dropping
them into the Graph editor.
The CloverETL Palette
The Palette is where you select graph components – Readers, Writers,
Transformers, Joiners, and other graph components – as well as Edges for joining
components, and editing tools.
Click the headings within the palette to expand or collapse the list of components.
2
CloverETL User's Guide
Note: To deactivate an active component or tool selected in the palette, press Esc.
The CloverETL Tabs view
The lower-right portion of the screen shows a tabbed list of views that provide
additional information and help with graph execution and troubleshooting.
v The Properties tab enables you to view and edit properties of the graph
components, such as XML attributes, filtering, expressions, and metadata
information.
v The Console tab shows progress messages and errors.
v The Problems tab provides a list of configuration and validation problems
v The Clover-Regex Tester tab is an editor that assists with testing of regular
expressions.
v The Clover-Log tab displays logging details for the graph execution.
Callout handler registration for CloverETL graphs
Some CloverETL graphs require certain handler class information and handler
arguments.
For information about packaging a CloverETL graph as a callout, refer to the
“Callout handlers” information in the IBM Initiate Workbench User's Guide.
Chapter 1. CloverETL features and benefits
3
4
CloverETL User's Guide
Chapter 2. CloverETL graph components
CloverETL graph components provide a visual representation of the flow of the
extract, transform, and load process.
Components are joined by Edges; data flows through Edges, which are structured
by metadata. Graph components are selected from the Palette list.
Overview of CloverETL components
IBMInitiateIBM Initiate Workbench uses several CloverETL components in the
extract, transform and load processes. Most components require some
configuration to work correctly within a graph. Configuration is done through the
component's editor, which displays all the component's properties.
v Missing property values that are required are flagged with a yellow warning
icon.
v Default property values are displayed in gray italics.
CloverETL Readers
Readers can be thought of as the starting point for the ETL process, in that they
“read” the input data that is to be processed. Readers access data from specified
data sources, such as external files, LDAP repositories, or databases.
Each graph must include at least one Reader.
CloverETL Transformers
Transformers perform operations on the data, such as filtering, sorting,
de-duplicating, and merging. Transformers receive data via input ports and export
it to output ports.
You can concatenate a series of Transformers to perform multiple operations
against your data. When using multiple Transformers, care should be taken with
the sequence in which they are executed. For example, it is necessary to sort your
data before doing a deduplication operation.
CloverETL Joiners
Joiners can put together records with different metadata, including those with
different numbers of fields, according to a specified key and transformation.
These components perform some of the same transformations that the Transformer
components perform.
Joiners receive data via multiple input ports and export it to a single output port.
CloverETL Writers
Writers output data to specified output sources, such as external files, database
tables, or LDAP repositories. They are often the terminal components in graphs.
.
© Copyright IBM Corp. 1995, 2011
5
Note: The Trash component is a quick way to terminate a graph for testing.
CloverETL Edges
Edges appear as the connecting lines between Readers, Transformers, and Writers.
Every Edge carries metadata information for structuring the data. This includes
information such as field names, field types, and delimiters.
Metadata can be derived from header rows in the data, entered manually, or
applied via a format file (*.fmt). In addition, metadata can be propagated from one
Edge to another.
Edges are always bound to component ports. Components read data from input
ports and write data to export ports. Some components provide specific numbers of
ports, and others provide an infinite number of ports. Filter components, for
example, have two output ports: one (port 0) for writing valid records, and another
(port 1) for writing invalid records.
Other CloverETL components
CloverETL also supports components that execute graphs and some system, Java,
and database commands.
In addition, there are components that perform lookup and foreign key definition
functions.
Deprecated CloverETL components
Some CloverETL components have been deprecated but are still available in the
product. These components have been replaced by components that function more
efficiently; however, their behavior might not be exactly the same as the
components that they replace.
A deprecated component will continue to function, but in a future release might be
removed.
Note: The Console log will include the following warning message when you use
a deprecated component:
WARN [main] - [COMPONENT_NAME] - Component is of type
COMPONENT_TYPE, which is deprecated
For a more thorough discussion of CloverETL components, refer to the Clover
documentation
Working with CloverETL graphs
Workbench includes several sample graphs that you can customize for your own
use; these graphs can also provide a resource for you to learn how graphs are
built.
Preparing the CloverETL extract file – best practices
You should observe the following best practices when you prepare your data for
transformation in CloverETL:
v Sort your data on an index or key. This speeds up the processing time and can
eliminate the need for a Sort step in your graph.
6
CloverETL User's Guide
v Include header rows where possible. This facilitates the creation of metadata
required by the graph. However, when configuring some Reader components,
remember to set the properties related to skipping lines in source data
appropriately.
Using a CloverETL sample graph
To view and use a sample graph you must import it into a project.
For information about working with projects, refer to the IBM Initiate Workbench
User's Guide.
Importing a CloverETL graph into a project
You can import a graph anywhere in the project folder. Creating a folder just for
storage of graphs helps with efficient file management.
About this task
Procedure
1. In the Navigator pane, right-click within the project folder (or on the folder in
which you want to store graphs), and choose Import.
2. In the Import - Select dialog box, select Import graphs – version conversion.
3. Click Next.
4. In the Import – Clover.ETL Graphs dialog, click the Browse buton beside the
From directory field.
5. Navigate to and select the <ROOTDIR>\<Workbench (version)>\samples\
graphs directory (where ROOTDIR is your the program files installation
directory, and <Workbench (version)> is the directory where Workbench is
installed).
6. Click OK.
7. The Import – Clover.ETL Graphs window is now populated with all available
sample graphs in that directory.
In the right pane, select each item that you want to import, or select graphs in
the left pane to select all the items.
8. Click Finish to complete the import procedure.
Building a CloverETL graph
The section discusses the basic steps for creating and executing a CloverETL graph.
About this task
Depending on your business needs, the graphs that you build will vary in design
and complexity; however, the basic steps include:
v selecting graph components
v connecting them with Edges
v defining their properties
v selecting sources for data that will be transformed
v selecting targets for the transformed data
v identifying metadata and assigning it to the Edges
v running the graph
Chapter 2. CloverETL graph components
7
For more detailed information, refer to the CloverETL Designer User's Guide, and to
“The CloverETL Clean and Deduplicate Sample Graph” on page 20.
Before you can design and execute a graph, you must create a graph placeholder
in Navigator within your project.
Procedure
1. Right-click the folder in which you want to create the graph, and select New >
ETL Graph to open the New graph wizard.
2. In the CloverETL node, click ETL Graph.
3. In the Create new graph dialog box:
a. Type a name for the graph.
b. (Optional) Type a description of what the graph accomplishes.
c. Clear the Allow inclusion of parameters from external file? check box.
4. Click Next.
5. In the Output dialog box, verify the parent folder (the parent project and
folder) for the graph.
(Optional) You can change the name of the graph by updating the File name
field.
Note: As a best practice, create the graph under the Initiate project folder that
is associated with your Hub and with the data you will use this graph to
transform.
6. Click Finish. The new graph name is displayed in the Navigator tree pane
within the folder in which you created it.
Adding and configuring a CloverETL Reader
Adding and configuring a reader enables your graph to read in data from an
external source such as a text file, database, or LDAP repository.
About this task
Readers access data from external sources.
Procedure
1. In the Palette, click the Readers header to expand the list of Readers.
2. Drag the type of Reader you want to use into the Graph editor.
3. Double-click the Reader to open the Edit component dialog.
Note: You can also access the properties of the component via the Properties
tab at bottom of the Workbench. When using the Properties tab, ensure that
you have selected the correct component in the Graph editor. In the Properties
tab, port information is listed under Input Port and Output Port in on the
Properties tab.
4. Enter a File URL linking to your source data:
a. Click in the Value field for File URL (on the Properties tab). An ellipsis icon
is displayed.
b. Click the ellipsis icon to open a file browser.
c. Navigate to and select your source data file, and click Open.
5. Edit the other Reader properties as needed. Available properties vary
depending on the type of Reader you have chosen; detailed information about
8
CloverETL User's Guide
Reader-specific properties is available in the CloverETL Designer User's Guide. In
most cases you can accept the default properties.
6. Click OK.
Results
Note: A yellow warning icon is displayed at the upper right of the Reader when
no data source is selected.
Adding and configuring a CloverETL Transformer
Transformers perform operations on the data, such as filtering, sorting,
de-duplicating, and merging. You can concatenate multiple Transformers to
perform complex operations on your data.
About this task
To add a Transformer component to your graph and configure it, follow these
steps.
Procedure
1. In the Palette, click the Transformers header to expand the list of Transformers.
2. Drag the type of Transformer you want to use into the Graph editor.
3. Double-click the Transformer to open the Edit component window.
Note: You can also access the Properties of the component via the Properties
tab. Note, however, that when using the Properties tab, you must be sure the
correct component is selected in the Graph editor. In the Properties tab, port
information is listed under Input Port and Output Port.
4. Edit the Transformer properties as needed. Available properties will vary
depending on which type of Transformer you have selected.
Required properties with missing values are marked with a yellow warning
icon. For example, a sort key is required for the Ext Sort Transformer, and a
filter expression is required for the Ext Filter Transformer.
Detailed information about Transformer-specific properties is available in the
CloverETL Designer User's Guide.
Note: Some properties, such as filter expressions, cannot be validated until
metadata has been defined for the data being transformed. See “Adding
CloverETL metadata” on page 10 for more information.
Before you can use the Dedup component for deduplication operations, data
must be sorted. If your source data is already sorted, you can direct data
directly from the Reader to the Dedup component. If your source data has not
been sorted, you must direct your data through an Ext Sort component before
directing it to the Dedup component.
5. When you have you entered the Transformer's properties, click OK.
Using regular expressions with the CloverETL filterExpression
editor
The Ext Filter Transformer can use regular expressions and built-in functions for
filtering data.
Chapter 2. CloverETL graph components
9
Each input record is processed by the filter expression. If the result of the
expression is true, the record is passed to port 0 (accepted); otherwise it is passed
to port 1 (rejected).
You can find additional information about constructing regular expressions in the
CloverETL Designer User's Guide, and on the Web at sites such as
http://jakarta.apache.org/regexp and http://java.sun.com/docs/books/tutorial/
essential/regex/.
CloverETL Expression tester
CloverETL includes an editor that can assist you with testing of regular
expressions. The expression tester is accessible from the Tabs pane.
More information about the expression tester is available from the CloverGUI
Help, available via IBM Initiate Workbench's help menu.
Using CloverETL Edges to connect components
Graph components are linked with Edges. Edges represent the data flow between
components.
Edges require metadata to define the data that flows between components. Adding
Edges to the graph, then, involves several steps:
v Connecting components with an Edge
v Adding metadata information
v Applying metadata to an Edge
v Propagating metadata from one Edge to adjacent Edges
These steps are described in detail below.
Connecting CloverETL components with an Edge
To pass data from one component to the next in your graph, follow these steps to
connect the components with Edges.
Procedure
1. In the Palette, click to select the Edge tool. The mouse icon changes to show a
plug, indicating that the Edge tool is active.
2. Click the output port of the component you want to connect from. Output
ports appear as notches along the right edge of a component.
Note: For components with multiple ports, be sure to connect to the correct
port. You can mouse over a component's port to see a brief description of the
port's output.
3. Click the input port of the component you want to connect to. Input ports
appear as notches along the left edge of a component.
Note: When the Edge is displayed as a dotted line, this indicates that metadata
is missing. A solid-line Edge indicates that all required information is present.
Adding CloverETL metadata
Metadata defines the data that flows between components via Edges.
10
CloverETL User's Guide
About this task
While CloverETL offers multiple ways to add metadata, this section describes the
recommended way for use with the IBM Initiate sample graphs. Refer to the
CloverETL Designer User's Guide for information on alternative ways of adding
metadata to an Edge.
Procedure
1. In the Outline pane, right-click Metadata and choose New Metadata.
Note: You can also access the New Metadata menu by right-clicking on an
Edge in the graph.
2. From the New Metadata menu choose Extract from flat file.
Note: For detailed information on other methods of adding metadata, refer to
the CloverETL Designer User's Guide.
3. In the Flat file dialog, click Browse.
4. Browse to and select your source data file. Click Open.
5. Click Next.
6. If your file is delimited, choose a delimiter from the Delimiter drop-down list.
If your delimiter is not listed in the Delimiter drop-down list, you can add it
by following these steps:
a. From the Delimiter drop-down list, choose Other “<specify>”.
b. In the field to the right of the Delimiter list, enter the delimiter character
(such as a pipe).
c. Click the << icon to add the delimiter to the drop-down list.
7. If your source data file includes header records, check the Extract names box
to pull field names from the source file.
Note: If your source data file does not include header records, you will be
able to add field names on the next tab.
8. Click Next. The metadata is displayed.
9. If you want to give this metadata a specific name rather than accepting the
default name, click the Record:default name field (in the Name column) and
enter the new name.
10. If your source file does not include header records, you can edit the default
field names (listed in the Name column) as needed.
11. If you want to preview the data before continuing, click the Preview tab.
Note: The Preview tab is available only when adding new metadata. If you
open an existing metadata record from the Outline pane, you will not see the
Preview tab.
12. Click Finish.
Applying CloverETL metadata to an Edge
About this task
Once you have created metadata for a graph, you can associate the same metadata
to other Edges.
Chapter 2. CloverETL graph components
11
Procedure
1. Using the Palette's Select tool, right-click the Edge to which you want to apply
metadata.
Note: You can apply metadata to multiple Edges at once by using Ctrl-click to
select multiple Edges.
2. From the drop-down list, select Select metadata. A list of saved metadata for
this graph appears. Choose the metadata you want to apply to this Edge.
Note: You can also apply metadata to an Edge by editing the Metadata
property on the Edge's Properties tab.
Propagating CloverETL metadata from one Edge to another
Metadata can be propagated from one Edge to adjacent Edges recursively,
providing a quick and convenient alternative to assigning metadata manually to
adjacent Edges.
About this task
To propagate metadata to an adjacent Edge, right-click on the Edge with metadata
already assigned and choose Propagate metadata.
Note: Edges with metadata assigned appear as solid lines. Edges missing metadata
appear as dotted lines. When you have propagated the metadata to adjacent Edges,
those Edges will appear as solid lines.
Debugging a CloverETL graph Edge
CloverETL offers a debug feature, to help you troubleshoot problems in your
graphs.
About this task
To debug a graph Edge, right-click on the Edge and choose Debug > Enable
Debug. A green “bug” icon is displayed on edges with debugging enabled. Debug
information is captured when the graph is run.
You can view debug data after the graph is run by right-clicking the edge and
choosing Debug > View Data.
Adding and configuring a CloverETL Writer
Writers write transformed data to specified output files.
About this task
To use a Writer in your graph, you must add to the graph, configure it, and
connect it to the other component(s) that will provide the data to write.
Procedure
1. In the Palette, click the Writers header to expand the list of Writers.
2. Drag the type of Writer you want to use into the Graph editor.
3. Double-click the Writer to open the Edit component dialog.
12
CloverETL User's Guide
Note: You can also access the Properties of the component via the Properties
tab. Note however that when using the Properties tab, you must be sure the
correct component is selected in the Graph editor.
4. Edit the Writer properties as needed. Available properties will vary depending
on which type of Writer you have selected. For example, the DB2 Data Writer
requires database connection information, and the LDAP Writer requires an
LDAP URL.
Required properties are marked with a yellow warning icon. For example,
database connection information is required for the DB2 Data Writer. Detailed
information about Writer-specific properties is available in the CloverETL
Designer User's Guide.
Note: You can easily discard records by using the Trash Writer.
5. When you have you entered the Writer's properties, click OK.
6. Connect the Writer to the appropriate Transformer via an Edge. Refer to “Using
CloverETL Edges to connect components” on page 10 for more information.
Connections in CloverETL
Connections in CloverETL let your graph communicate with external data
resources.
To execute some graphs, you must establish connections to resources such as the
IBM Initiate Master Data Service or to databases from which data will be extracted.
Creating a database connection for CloverETL graph
execution
A database connection lets your CloverETL graph components connect to a
database.
About this task
After creating a database connection, you can use any of the database readers in
graphs that you build.
Procedure
1. Open a graph.
2. In the Outline pane, right-click Connections and choose Connections > Create
internal. This opens the database connections window.
Note: You must open a graph in the Graph editor to see the nodes, including
the Connection node, in the Outline pane.
3. Click to select a database driver from the available drivers window.
Note: It is recommended that you use one of the supplied drivers.
4. Enter a Name for the connection.
5. Enter the User and Password for connecting to the database.
6. Click the Validate button to validate the database connection.
7. Click Finish.
Chapter 2. CloverETL graph components
13
Creating an Initiate connection for CloverETL graph execution
To use IBM Initiate custom components you must create an IBM Initiate
connection, which is a connection to the IBM Initiate Master Data Service.
About this task
To create your IBM Initiate connection, you will need to specify the hostname,
port, user name and password for connecting to your Master Data Engine (Hub)
instance.
Procedure
1. Open a graph.
2. In the Outline pane, right-click Connections and choose Connections > IBM
Initiate connection. This opens the Initiate connection window.
3. Select internal.
4. Enter a Connection name for the connection.
5. Enter the Host and Port for your Hub.
6. Enter the User and Password for connecting to your Hub.
7. To configure reconnection attempts after a network, engine, or database outage:
a. Set a number of attempts in the Retry count field.
b. Set the interval between attempts in the Retry delay field.
8. Click Finish.
Creating a JMS Connection for CloverETL graph execution
Use a JMS connection to connect your graph to a message queue source.
About this task
After creating a JMS connection you can use the JMS Reader and JMS Writer
components in graphs that you build.
Procedure
1. Open a graph.
2. In the Outline pane, right-click Connections and choose Connections > JMS
internal connection. This opens the JMS connection window.
Note: You must open a graph in the Graph editor to see the nodes, including
the Connection node, in the Outline pane.
3. Enter a Name for the connection.
4. Click the green plus symbol to browse to and select the Libraries for the
connection. These reference any custom jar files that may need to be
appended to the classpath. The prefix for your libraries should be file:
For example, a library entry for Apache ActiveMQ would take a format such
as
file:C:/apache-activemq-5.2.0/activemq-all-5.2.0.jar
5. Enter an Initial ctx factory class. This is the fully-qualified class name
implementing the JNDI API and allowing lookup of the connection factory.
This item will detect class name automatically when a jar file with the class is
specified. Otherwise, the class name may be specified manually.
6. Choose a URL from the drop-down list. The URL specifies the connection's
protocol.
14
CloverETL User's Guide
7. Choose a Connection factory JNDI name from the drop-down list. This is the
JNDI name of the factory creating JMS connections. Values in the drop-down
list will vary depending on the JMS provider.
8. Enter a Destination JNDI for the connection. This is a JMS provider-specific
address that identifies queue/topic of messages.
9. Enter the User and Password for connecting to the JMS.
10. Check the Encrypt password box if you want to encrypt the password.
Note: Click Validate connection to validate the connection information. If the
connection does not validate, make sure that the Master Data Engine service is
running and that you have entered valid connection parameters.
11. Click Finish.
Executing a CloverETL graph
To execute a graph, ensure that necessary connections are established, and launch
the graph using the Run icon.
About this task
To execute a graph, click the Run icon
in the toolbar, or right-click in the
graph editor and choose Run As > Clover.ETL graph. When a graph has finished,
the number of records processed along each Edge is displayed.
Note: The CloverETL GUI is intended for graph development and debugging.
Production level graphs with large data sets should be run by using the
command-line option (refer to “Executing CloverETL graphs with the madconfig
launch_etl utility” ) or on the same server where the data is located (refer to the
IBM Initiate Workbench User's Guide). Running graphs by using the command line
or remote server enables improved performance due to the availability of increased
memory and 64-bit processing.
For detailed information about graph runtime options, refer to the CloverETL
Designer User's Guide.
Executing CloverETL graphs with the madconfig launch_etl
utility
The madconfig launch_etl utility provides a means of running a CloverETL graph
automatically, for example as part of a scheduled job.
About this task
After you have configured the sample templates and created CloverETL graphs,
you can execute them automatically by using the madconfig utility as part of a
scheduled job. To update parameters, such as audit record number range, when
you run the scheduled job you can create an external properties file that the
scheduled madconfig utility can reference.
You can also use the madconfig launch_etl utility to execute a graph
independently of a job.
Chapter 2. CloverETL graph components
15
Using the madconfig utility to create a properties file for a
scheduled job
Parameters for running a scheduled job are stored in a properties file. You can
create a properties file for a scheduled graph execution using the madconfig utility.
About this task
Incremental extracts typically select data that is based on a range of audit record
numbers that change each time the graph is run. Although you may manually set
the range of record numbers to extract manually in the graph, it may be more
practical to generate a properties file automatically via a scheduled job. The
properties file then supplies the graph with the appropriate values for the record
number range.
This section describes how to use the madconfig utility to launch a graph using a
designated, external properties file. You can set up a scheduled job to launch the
madconfig utility on a regular basis.
Note: It is outside the scope of this document to describe how to set up a
scheduled job that generates the properties file. You can use a standard utility such
as the Windows Task Scheduler or a Unix chron utility (or other methods) to set up
a scheduled job.
Using madconfig launch_etl to execute a CloverETL graph
The madconfig launch_etl operation can automatically launch a CloverETL graph.
It can optionally use a properties file to provide input to the madconfig utility.
About this task
The madconfig launch_etl operation can make use of a properties file that contains
auditor record number files.
Procedure
1. From a command prompt, run madconfig launch_etl
Note: This utility is run from the ROOTDIR\Engine x.x.x\scripts directory
2. At the prompt, enter the path to the graph (*.grf file) you want to run.
3. (Optional) At the prompt, enter the path to your configuration file (that is, the
file containing the properties for your graph's audit record number parameters).
4. At the prompt, enter a heap memory size setting or accept the default (256
MB).
5. (Optional) At the prompt, enter a statement that includes any additional
parameters needed to execute the graph.
Note: For complete documentation of the madconfig utilities, refer to the IBM
Initiate Master Data Service Engine Installation Guide.
Recording responses to the madconfig utility
If you want to launch a graph by using madconfig on a scheduled basis, you can
record a set of responses to the madconfig utility's prompts.
16
CloverETL User's Guide
About this task
v To record a set of responses to the madconfig launch_etl function, run
madconfig -recordfile myfile.properties launch_etl where myfile.properties is the
name of the file that will store your responses.
Note: In addition to recording your responses, this command also executes the
graph.
v To run madconfig using the recorded responses, run madconfig –propertyfile
myfile.properties launch_etl, where myfile.properties is the name of the file where
your responses are stored.
Viewing logs and error messages about CloverETL graph execution
Warning and error messages, processing information, and graph status are
captured on the Console, Problems, Clover–Graph tracking, and Clover Log tabs.
About this task
Refer to the CloverETL Designer User's Guide for detailed information about the
contents of these tabs.
The CloverETL JMS Reader component
The JMS Reader component processes messages in accordance with Java's standard
JMS API. Messages can be routed directly to a Writer component, or can be
transformed before writing, using any of CloverETL's transformer components.
The table below lists JMS Reader properties which are typically configured. For a
full description of all JMS Reader component properties, refer to the CloverETL
User's Guide or the CloverETL online help in Workbench.
Table 1. JMS Reader component properties
Property
Required?
Default
JMS connection
yes
The JMS connection ID. JMS connections
are created in the Connections node of the
Outline. For more information on creating a
JMS connection, see “Creating a JMS
Connection for CloverETL graph execution”
on page 14. You can also refer to the
CloverETL User's Guide or the CloverETL
online help in Workbench.
Processor code
no
Inline Java code defining the processor
class. Processor code is applied only if the
Processor class is not specified
Processor URL
no
URL to file which contains Java source of
the processor class. This is applied only if
the Processor class and Processor code are
not specified
Processor class
no
org.jetel.
component.
jms.JmsMsg2
DataRecord
Properties
Description
Name of the Processor class. The default
value (
org.jetel.component.jms.JmsMsg2
DataRecordProperties
) is applied only if the Processor code and
Processor URL attributes are not specified.
Chapter 2. CloverETL graph components
17
Table 1. JMS Reader component properties (continued)
Property
Required?
JMS message
selector
no
Default
Description
JMS selector specifying messages to be
processed.
Processor source no
charset
ISO-8859-1
(check
DataParser.
DEFAULT_
CHARST_
DECODER in
default
Properties)
The charset of the Processor code, if it is
specified by the processorURL attribute.
Message charset
ISO-8859-1
(check
DataParser.
DEFAULT_
CHARST_
DECODER in
default
Properties)
The charset of the Message
Max msg count
no
The maximum number of messages to
process before exiting the graph. Setting
this to 0 means to read forever (that is, to
enforce no maximum). When debugging a
graph, it can be useful to set the max
message count to a manageable number of
messages for review.
Timeout
no
The maximum amount of time (in
milliseconds) to wait for the next message
before exiting the graph. Setting to 0 means
to wait forever (that is, to enforce no
timeout).
Message body
field
no
Maps the message body to a field on the
output port's Edge metadata. Once your
metadata is defined, the message body field
can be selected from the drop-down.
How the CloverETL JMS Reader component and Edges
process messages
When configuring the JMS Reader component, it is important to understand how
messages are processed, in order to avoid suboptimal performance.
Once connected to the message queue source, the JMS Reader component reads
messages as they arrive, and passes them via the component's output port to the
Edge, which in turn routes the messages on to the next component in the graph.
The JMS Reader's default behavior is to read and process each message as it
arrives, regardless of the number of messages, or the rate at which they arrive. The
JMS Reader, by default, does not time out or impose any kind of maximum count
on the messages it processes.
The Edge that carries messages from the JMS Reader component to the next
component by default buffers the messages before routing them to the next
component. This buffering behavior is designed to improve performance; however,
it is important to realize that in a low-traffic scenario, it may take time for the
18
CloverETL User's Guide
Edge's buffer to fill up and trigger the sending of messages to the next component.
This may result in delays which are unacceptible from a performance standpoint.
The sections below describe how to use timeout values, max message counts, and
Edge properties to determine the manner and rate of message processing.
Using the CloverETL JMS Reader components's timeouts to
manage message flow
The CloverETL JMS Reader component's timeout properties help you manage
message flow for optimum performance.
With message queues, unlike (for example) text or database files, there is often no
clear end point for the message stream. A message queue can contain anything
from no messages at all to a virtually endless stream of messages.
The JMS Reader component's default behavior is to process messages as they
arrive, and to continue to process messages indefinitely. If you have a scenario in
which you want the JMS Reader to process a set of messages and then, when that
set is completely processed, to stop and close the graph, you can use the timeout
property. A typical use case for using a timeout is when you have stopped the
Master Data Engine for a time, and know that during this downtime you have
built up a queue of messages to process; in this scenario you want to “drain” the
existing queue, and when all the messages are processed, to have your graph stop
automatically. To do so, set the timeout value in the JMS Reader properties to a
number other than zero.
The timeout property is set in milliseconds. If you set the timeout value to, for
example, 10,000 milliseconds, the JMS Reader component will wait no more than
ten seconds for the next message to come. If ten seconds elapse with no further
messages, the graph will cease processing and close.
The timeout property's default setting of 0 means to wait forever (that is, to
enforce no timeout).
Limiting the CloverETL JMS Reader component max message
count for debugging
The CloverETL JMS Reader component's maxMsgCount property determines the
maximum number of messages to process before ceasing processing and closing
the graph.
Typically this is set to 0, to allow the graph to process all messages, without
maximum. In a debugging situation, however, you may want to set the
maxMsgCount to a manageable number, for reviewing data and troubleshooting
any graph problems.
Using the CloverETL JMS Reader max message counts and
timeouts together
This table shows how the maxMsgCount and timeout values can be used together
for different kinds of processing behavior:
Chapter 2. CloverETL graph components
19
Table 2. JMS Reader max message count and timeout behavior
Max msg count
timeout
Behavior
0
0
The JMS Reader keeps waiting for new messages.
The phase in which this node is embedded never
stops.
greater than 0
0
The JMS Reader reads new messages until its
count reaches maxMsgCount, regardless of how
long it takes.
0
greater than 0
The JMS Reader waits for new messages, as long
as the last one arrives within the specified
number of milliseconds, regardless of how many
messages it reads.
greater than 0
greater than 0
The JMS Reader stops when the count of read
messages reaches the maxMsgCount value, or
when the timeout occurs, whichever comes first.
Managing the CloverETL Edge's message buffer for immediate
message processing
Use the CloverETL Edge's message buffer to ensure quick processing of messages.
About this task
The default behavior for Edges connected to a JMS Reader component is to buffer
messages, waiting to pass the messages downstream to the next graph component
until the buffer is full. In situations where the message traffic is low, this can mean
long waits for messages to be processed.
To ensure quick processing, you can set a "direct fast propagate" property on the
graph's Edges, to direct the graph to output messages as they are processed,
regardless of buffer status. This setting must be applied to the Edge of the JMS
Reader component's output port and all downstream Edges.
Any time you are propagating messages in real time, it is recommended that you
use “direct fast propagate.”
These instructions assume the graph has been created, and Edges have been
populated with metadata.
Procedure
1. Right-click the Edge connected to the JMS Reader component's output port.
2. From the context menu, choose Edge Type.
3. Set the Edge Type to “Direct fast propagate”.
Note: Repeat these steps for all downstream Edges in the graph.
The CloverETL Clean and Deduplicate Sample Graph
A sample data cleaning and deduplication graph (clean_and_dedup.grf) is
provided with CloverETL to demonstrate verification of a load file.
The sections below describe the sample graph components and how to use them.
20
CloverETL User's Guide
You must import the sample graph into IBM Initiate Workbench in order to access
it in CloverETL. Refer to “Importing a CloverETL graph into a project” on page 7.
Linking The CloverETL Clean and Deduplicate sample Reader
to a source file
About this task
The sample graph includes a Universal Data Reader to read sample data from an
extract file. In order to use this component, you must link the Reader to your
source file.
Procedure
1. Double-click the Reader to open the Edit component dialog.
2. Enter a File URL linking to your source data:
a. On the Properties tab, click in the Value field for File URL. An ellipsis icon
is displayed.
b. Click the ellipsis icon to open a file browser.
c. Navigate to and select your source data file, and click Open.
3. Click OK to close the Edit component dialog.
4. Check the data for a byte order mark (BOM) at the beginning of the file:
a. Right-click on the Reader and choose View Data.
b. Choose the number of records to view (you can accept the default value of
records 1 through 10) and whether to display them in a plain text or grid
format.
c. Click OK.
d. Review the data in the View Data dialog.
If you see problematic characters at the beginning of the file, you can use a
text editor such as Notepad to open and correct the file.
Adding metadata for the CloverETL Clean and Deduplicate
sample graph
The sample graph requires metadata defining your source data.
About this task
Detailed instructions for adding and applying metadata can be found in “Adding
CloverETL metadata” on page 10 and “Applying CloverETL metadata to an Edge”
on page 11.
Note: After you have added metadata, be sure to propagate it to the rest of the
Edges in the graph. Right-click on the Edge containing metadata and choose
Propagate metadata.
Editing the Ext Filter component in the CloverETL Clean and
Deduplicate sample graph
The first Transformer in the sample graph is an Ext Filter component that filters
records containing problematic characters from the data. A regular expression has
been defined for the sample graph's Ext Filter component to perform this filtering.
Chapter 2. CloverETL graph components
21
About this task
Records that meet the criteria (that is, records that contain only alphanumeric
characters) are written to the Ext Filter's output port 0, and records that do not
meet the criteria are written to output port 1.
Before using the graph, you must edit the filter expression in this component to
(put the fields in place of placeholder fields, also to add fields you need.)
Procedure
1. Double-click on the Ext Filter component to open the Edit component dialog.
2. On the Properties tab, click in the Value field for Filter expression. An ellipsis
is displayed in the Value field.
3. Click the ellipsis to open the Filter expression dialog. The sample filter
expression is shown in the lower pane on the dialog.
Note: Fields from the metadata are displayed in the upper left area of this
dialog. If you have not created and applied metadata, you will not see a list of
fields in this area.
4. The first line of the filter expression provides an example of how to check for
specific values in a field:
($Field0 ~= "[AB]")
and
// Can be A or B
This example verifies that the value in Field0 is “A” or “B”. You can edit this
line as needed to check for specific values, or comment it out by adding two
forward slash (/) characters to the beginning of the line.
Note: To replace Field0 with a field from the metadata, double-click Field0 to
highlight it, then double-click the value in the list of metadata fields to insert
it in the filter expression in place of Field0.
5. The second line in the filter expression checks that a Field1 contains only
alphanumeric characters:
($Field1 ~= "[0-9A-Za-z]+")
and
// Can be alphanumeric characters
You can copy and edit this field as needed to filter for alphanumeric
characters. Substitute the field name in your data for Field1 as needed.
6. The third line of the filter expression checks that Field2 contains only
alphanumeric or empty characters:
(isnull($Field2) or ($Field2 ~= "[0-9A-Za-z]+"))
(empty) or alphanumeric characters
and
// Can be Null
You can copy and edit this field as needed to filter for alphanumeric and
empty characters. Substitute the field name in your data for Field2 as needed.
7. The fourth line of the filter expression checks that Field3 contains only
alphanumeric and designated other characters (double quotes, forward slash,
single quote, dash, period, comma, and space):
(isnull($Field3) or ($Field3 ~= "[\\"/’-., 0-9A-Za-z]+"))
(empty) or characters listed
// Can be Null
You can copy and edit this field as needed to filter for alphanumeric and
special characters. Substitute the field name in your data for Field3 as needed.
8. If you want to validate the filter expression, click the Validate button. Any
errors encountered during validation are displayed at the bottom of the
dialog.
9. When you have finished editing the filter expression, click OK to save the
changes and close the Filter expression dialog.
22
CloverETL User's Guide
10. Click OK to close the Edit component dialog.
Specifying a sort key in the Ext Sort component of the
CloverETL Clean and Deduplicate sample graph
The Ext Sort component in the sample graph sorts the records from the Ext Filter's
output port 0 (that is, records that met the Ext Filter component's filter criteria) and
sorts them by a specified key; in the sample graph, the key is RecordID.
About this task
You must edit this component to specify one or more sort keys for your data.
Procedure
1. Double-click the Ext Sort component to open the Edit component dialog.
2. On the Properties tab, click in the Value field for Sort key (in the Basic node).
An ellipsis is displayed in the Value field.
3. Click the ellipsis to open the Edit key dialog.
4. Select the field(s) you want to sort on from the Fields list on the left. Use the
arrows to move them to the Key parts list on the right. You can sort on more
than one key. The fields' positions in the Key parts list determines the order in
which sorting is performed.
5. Click OK to save the changes and close the Edit key dialog.
6. To choose whether to sort in ascending or descending order, click in the Value
field for Sort order (on the Properties tab in the Basic node) and choose
Ascending or Descending.
Note: If a sort order is not specified, the component will default to sorting in
ascending order.
7. Click OK to close the Edit component dialog.
Specifying a dedup key in the CloverETL Clean and
Deduplicate the sample graph
The Dedup component removes duplicate records from the sample data, based on
specified keys; in the sample graph, the key is RecordID..
About this task
Unique records are written to output port 0, and duplicates are written to output
port 1.
Note: In order for the Dedup component to work, you must sort the data based on
a dedup key first so that records with duplicate key fields are in sequence.
You must edit this component to specify one or more dedup keys for your data.
Procedure
1. Double-click the Dedup component to open the Edit component dialog.
2. On the Properties tab, click in the Value field for Dedup key (in the Basic
node). An ellipsis is displayed in the Value field.
3. Click the ellipsis to open the Edit key dialog.
4. Select the field(s) on which you want to deduplicate from the Fields list on the
left. Use the arrows to move them to the Key parts list on the right. You can
Chapter 2. CloverETL graph components
23
deduplicate on more than one key. The fields' positions in the Key parts list
determines the order in which deduplication is performed.
5. Click OK to save the changes and close the Edit key dialog.
6. To specify whether to keep the first or last copy of a duplicate record, click in
the Value field for Keep (on the Properties tab in the Basic node) and choose
First or Last.
7. Click OK to close the Edit component dialog.
Writers in the CloverETL Clean and Deduplicate sample graph
The Writers in the sample graph write output from the filer and deduplication
transformations to a series of output files.
v The Ext Filter component writes “dirty” records (that is, records that do not
meet the filter criteria) to a Universal Data Writer labeled Write dirtyExtract.txt
v The Dedup component writes unique records to a Universal Data Writer labeled
Write cleanExtract.txt, and duplicate records to a Universal Data Writer labeled
Write dupExtract.txt
To use these Writers, you must specify an external file to which the component
writes the data.
Note: For data tracking and organization, it is useful to assign each Writer a name
that matches or references the name of the external file data is written to.
Information on how to rename a component is given in the steps below.
Specifying an external file for data writing in the CloverETL
Clean and Deduplicate sample graph
Procedure
1. Double-click the Writer to open the Edit component dialog.
2. Enter a File URL linking to the external file. You can either type in a file name,
or browse to link to an existing file. To browse to an existing file:
a. Click in the Value field for File URL (on the Properties tab). An ellipsis icon
is displayed.
b. Click the ellipsis icon to open a file browser.
c. Navigate to and select the external file, and click Open.
Note: If you place the extract file in the folder that contains the project, you
can open and view the output files from the Navigator pane. For example, if
the project folder is C:\Program Files\IBM\Initiate\Workbench970\workspace\
SampleProject, put the output files under the ..\SampleProject folder to make
them appear in the Navigator under SampleProject. You may have to refresh
the Navigator in order for changes to appear.
3. If you want to rename the component to reference the name of the external file,
update the Component name value (in the Visual properties node on the
Properties tab).
4. Click OK to close the Edit component dialog.
The IBM Initiate Member Model Transform Graph
You create this graph by executing a wizard. The graph prepares source data into
the database unload format (*.unl files) required for data derivation and for bulk
load into the Hub. As such, it provides a graph-based alternative to the member
model transformation part of the mpxdata function.
24
CloverETL User's Guide
Note: After Member Model Transform Graph transforms the data to *.unl files,
data derivation must be run. You can run mpxfsdvd as a command-line function
or as a job in IBM Initiate Workbench to perform data derivation. For information
about the mpxfsdvd job, refer to the IBM Initiate Workbench User's Guide.
The IBM Initiate Member Model Transform Graph requires that certain source data
is located within your project:
v Input metadata, in the form of a *.fmt file
v Prepared source data, such as the output files of the clean_and_dedup.grf sample
graph
v A target directory for the *.unl files produced by the IBM Initiate Member
Model Transform graph
Input metadata in the Member Model Transform graph
There are multiple ways to create an input metadata (*.fmt) for the graph:
v Export metadata from an existing graph outline
v Import IBM Initiate Hub metadata
(Import > Clover ETL> IBM Initiate metadata)
v Import metadata from DDL
(Import > Clover ETL > Import metadata – transform from DDL)
v New Metadata from Database
(New > Clover ETL > Metadata (Database))
v New Metadata defined by hand
(New > Clover ETL > Metadata (Define by Hand))
v New Metadata from flat file
(New > Clover ETL > Metadata (Flat File))
This document discusses the first two in detail below. For more information on the
other ways to create input metadata, refer to the CloverETL Designer User's Guide.
Creating a metadata file by exporting metadata
About this task
You can create an input metadata file by exporting metadata from an existing
graph outline. If you have not defined any metadata at all, see “Adding CloverETL
metadata” on page 10 for information on how to create it.
Procedure
1. In the Outline pane, right-click on the metadata you want to create the file
from, and choose Export metadata.
2. Select the project folder for the metadata file, and enter a Name for it.
3. Click OK.
A .fmt file with the Name you entered is displayed in the Project folder you
specified.
Chapter 2. CloverETL graph components
25
Importing IBM Initiate Hub metadata to the Member Model
Transform graph
About this task
You can create metadata format files directly from data in your IBM Initiate Hub
with the Import IBM Initiate metadata feature. This feature enables you to select
attributes from which to create metadata.
Procedure
1.
2.
3.
4.
From the File menu, choose Import.
In the Select wizard, expand the Clover ETL node.
Select IBM Initiate metadata , and click Next.
In the Import dialog, enter the connection properties for connecting to your
Initiate Hub.
a. Enter Host, Port, User, and Password values for connecting to your IBM
Initiate Hub.
b. If you are using SSL security, check the Use SSL box and select the SSL
Version from the list. If you want IBM Initiate Workbench to verify the SSL
certificate returned from the Hub, click Verify SSL Certificate.
Note: You can click Validate to validate the connection information. If the
connection does not validate, make sure that the Master Data Engine service is
running, and that you entered correct connection parameters. Note that the
Validate option validates only the host and port, not the user name and
password.
5. Click Next. IBM Initiate Workbench connects to the IBM Initiate Hub and
retrieves attributes.
6. In the Import dialog, use Ctrl-click to select the attributes for which you want
to import metadata. You can use the Select All button to select all the listed
attributes.
7. Specify a Destination folder for the metadata files. It is good practice to specify
a project folder where the Member Model Transform Graph will also reside.
8. Click Finish. The metadata files appear in the Navigation pane.
Linking to source data files and output directories in the
Member Model Transform graph
About this task
Source data for the IBM Initiate Member Model Transform Graph must be in a
directory accessible through a Project folder in the Navigator pane. You can link to
a file or folder anywhere in your file system in order to make it accessible from the
Navigator.
While it is not required that the output directory be linked in a Navigator Project
folder, you may find it useful to link to the output file in the same manner you
link to your source data, so that related files are accessible in the same area.
Procedure
1. In the Navigator pane, right-click on Project folder you want to add files or
directories to and choose New > Folder (or New > File).
2. In the New Folder or New File dialog, click Advanced.
3. Check the Link to folder (or file) in file system box.
26
CloverETL User's Guide
4. Click Browse to browse to the file or folder you want to add.
5. Click Finish.
Configuring the IBM Initiate Member Model Transform Graph
About this task
After the metadata, source files, and output directory are ready, you can launch the
IBM Initiate Member Model Transform Graph wizard to configure the graph for
the data.
Procedure
1. In the Navigator pane, right-click on a project and choose New > Other.
2. In the New wizard, choose IBM Initiate Member Model Transform Graph (in
the CloverETL node) and click Next.
3. Enter a Name and optional Description for the graph.
4. Click Next.
5. Select the parent folder (that is, the parent project) for the graph, and click
Next.
6. Enter configuration information for the graph:
v Input metadata: Click Browse to browse to and select the metadata format
(*.fmt) file for this graph.
Note: Refer to “Input metadata in the Member Model Transform graph” on
page 25 for information on how to create input metadata files.
v Input file: Click Browse to browse to and select the source data to input
into this graph.
v Output directory: Click Browse to browse to and select the directory where
the output (*.unl) files will go.
v Initiate connection properties:
– Host: Enter the host name for your IBM Initiate Hub.
– Port: Enter the port for your IBM Initiate Hub.
– User: Enter the user name for connecting to your Initiate Hub.
– Password: Enter the password for connecting to your Initiate Hub.
– Load from file: if you have created and saved an external Initiate
connection, you can browse to it here.
– Use SSL: If you are using SSL security, check the Use SSL box and select
the SSL Version from the list. If you want IBM Initiate Workbench to
verify the SSL certificate returned from the Hub, click Verify SSL
Certificate.
Note: You can click Validate to validate the connection information. If the
connection does not validate, make sure that the Master Data Engine service is
running and that you entered correct connection parameters.
7. Click Next.
8. The Attributes dialog lists the attributes in your Hub. Select the attributes you
want to use (using shift-click or control-click).
9. Click Next.
10. The Model Mapping dialog displays tabs for each of the attributes you
selected. Drag input fields from the Input fields column to the Attribute fields
(in the Model mapping area) you want to map them to.
Chapter 2. CloverETL graph components
27
Note: Text values entered in any of the attribute fields must be enclosed in
double quotes.
11. Click Next. A summary screen is displayed.
12. If the summary information on the summary screen is OK, click Finish.
Note: When you create an IBM Initiate Member Model Graph, CloverETL
generates a lookup table, as a flat file, named SRCHEAD that is used to
perform lookups for MemIdent and Memhead sources. You must edit the
SRCHEAD lookup table with the sources that are applicable to your project.
13. The graph is displayed in the Graph editor. Click the Run icon
the graph.
to execute
The CloverETL Extract Household Graph
The extract_household graph extracts data that represents the association of
physical locations (households) and individuals living in them as they have been
grouped by the Hub. The output of the graph is four delimited text files that you
can use as source data for other tasks.
Creating a project for the CloverETL Extract Household graph
About this task
To use the extract_household graph, you must create an Initiate project that uses
the Householding template.
Procedure
1. From the Initiate menu, select New Initiate Project. More detailed information
on creating projects is located in the IBM Initiate Workbench User's Guide.
2. In the New Initiate Project - New Initiate Project dialog box, type a descriptive
name for the project in the Project Name field, and click Next.
3. In the New Initiate Project - Registered Hubs dialog box, select a hub to
associate with the project, and click Next..
4. In the New Initiate Project - Templates dialog, select the
Initiate_Demo_Householding template from the Available Templates list, and
click Finish.
The graph is now available in the Navigator pane when you create the project.
5. In the Navigator pane, double-click the extract_household graph to load the
graph in the Editor.
Customizing the extract_household graph
Most specifications that you need to run the graph are pre-defined; however, there
are configuration settings that you may want to change. The sections below
describe how to customize the graph.
Editing the Database Reader components for the
extract_household graph
About this task
The extract_household graph uses DBInputTable Reader components that include
SQL statements for querying the database. For more information about
DBInputTable Readers, refer to the CloverGUI Help.
28
CloverETL User's Guide
A database connection is defined in the DBInputTable Readers. To execute the
graph you will need to edit the database connection.
Procedure
1. In the Outline pane, expand Connections.
2. Right-click the database connection placeholder, and click Edit.
3. In the Database connection dialog box, edit the connection properties and
credentials.
a. Enter a Name for the database connection.
b. Enter the User and Password for connecting to the database.
c. Click to select a database driver from the available drivers window.
Note: It is recommended that you use one of the supplied IBM Initiate
drivers.
d. Edit the URL as needed to substitute your connection information.
4. Click the Validate button to validate the database connection.
5. Click OK.
Editing the query for the extract_household graph
About this task
Follow this procedure if you need to edit the SQL in the Reader components:
Procedure
1. Double-click the Reader to open the Edit component dialog box.
2. Click in the Value column of the SQL query property, and click the ellipsis that
is displayed.
3. In the SQL query editor, edit the Query text (in the lower pane) as needed.
4. Click OK.
Editing the parameters for the extract_household graph
About this task
The SQL queries written into the database Reader components use parameters, or
named constants, to represent attributes in the database and the project location.
However, these parameters might not coincide with those in your environment.
Procedure
1. In the Outline pane, expand Parameters.
2. Right-click any one of the parameters, and select Edit from the context menu.
3. In the Graph parameters dialog box, click in the Name and Value columns to
make changes as needed.
4. Click Finish to save the changes.
Specifying a location for the extract_household graph's output
files
About this task
The Writer components (relationship_links, delimited_name, delimited_phone,
delimited_addr) write output from the deduplication transformations to a series of
text files in the output folder; however, you can specify an external location for
each output file.
Chapter 2. CloverETL graph components
29
Note: The default File URL uses the {WORKSPACE} parameter to define a relative
path to your workspace.
Procedure
1. Double-click a Writer component to open the Edit component dialog box.
2. To edit the output file, click in the Value column of the File URL property, and
click the ellipsis that is displayed.
3. In the Path field, you can type a file location, or click the control and browse to
a location.
4. Click OK to close the URL File Dialog box.
5. In the Edit component dialog box, complete other customization as needed. For
more information about the Universal Data Reader component, refer to the
CloverGUI Help
30
CloverETL User's Guide
Chapter 3. IBM Initiate custom components
The CloverETL application in IBM Initiate Workbench includes several components
that are available only to IBM Initiate users. These components behave like IBM
Initiate interactions, taking full advantage of the IBM Initiate APIs.
The Initiate custom components are categorized as follows:
v Readers
– IBM Initiate MemGet
– IBM Initiate MemSearch
– IBM Initiate Handler Reader
– IBM Initiate
– IBM Initiate
v Writers
– IBM Initiate
– IBM Initiate
– IBM Initiate
– IBM Initiate
TskSearch
TskGet
MemPut
MemDelete
Handler Writer
TskPut
v Transformers
– IBM Initiate MemScore
– IBM Initiate Attribute Reformat
– IBM Initiate MemSeqNo Incrementer
– IBM Initiate Address Verification Interface
Using Key fields in CloverETL MemPut, MemSearch, TskSearch,
TskPut, and Handler Writer to join records
About this task
The IBM Initiate MemPut, IBM Initiate MemSearch, IBM Initiate TskPut, and IBM
Initiate Handler Writer components enable you to use Key fields to join records
from different input ports.
Note: The Master key always comes from data on input port 0, and slave keys
come from data on input ports 1-n.
Procedure
1. Double-click the component to open the Edit component dialog.
2. Click in the Value field next to the Key fields parameter. An ellipsis is
displayed.
3. Click the ellipsis to open the Join key dialog. Note that there is one tab for the
Master key, and Slave key tabs for as many other input ports are being used.
4. On the Master key tab, click to highlight a master key field from the Fields list,
and click the arrow icon to move it to the Master key column.
5. On each Slave key tab, chose the field you want to map to the Master key field
from the Fields list and drag it to the Slave key field.
6. Click OK on the Join key dialog to save the updates and close the dialog box.
© Copyright IBM Corp. 1995, 2011
31
7. Click OK on the Edit component dialog to save the updates and close the
dialog.
IBM Initiate custom Reader components
This section describes the IBM Initiate Reader components: IBM Initiate MemGet,
IBM Initiate MemSearch, IBM Initiate Handler Reader, IBM Initiate IBM Initiate
TskSearch, and IBM Initiate TskGet.
Comparing the IBM Initiate custom Reader components
When deciding which of these custom Initiate Readers to use in your graph,
consider the following:
v The IBM Initiate MemGet component retrieves data by using a member key
value, such as an EntRecno, SrcCode and MemIdNum, or MemRecno.
v The IBM Initiate MemSearch component returns members that, in comparison
with the input data, have a score higher than the minimum score defined.
v The IBM Initiate Handler Reader component retrieves data from the dictionary
according to the key inputMember.
v The IBM Initiate TskGet component retrieves task-related data using a known
member value such as an Member ID number (memidnum). or Member Record
number (memrecno) in the same manner as the TskGet API interaction
v The IBM Initiate TskSearch component takes search criteria input (for example,
task type, task owner, or task status) and returns a list of members with task
issues
The IBM Initiate MemGet component
The MemGet interaction is a Reader that retrieves individual members and their
attributes, using a known member value such as an Enterprise ID (EID), or Source
Code (srccode) and Member ID number (memidnum). The IBM Initiate MemGet
component retrieves data in the same manner as the MemGet API interaction.
Arguments are supplied to the IBM Initiate MemGet component in the form of
component parameters.
Note: For additional information on MemGet, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
The IBMInitiate MemGet component requires input from a CloverETL Reader. The
Reader supplies a connection to the external source of the data, and can also apply
filtering or other criteria to determine which data is passed on to the IBM Initiate
MemGet component.
CloverETL provides a number of Readers that can read data from sources such as
databases, text files, and LDAP repositories.
The IBM Initiate MemGet component has one input port and an unlimited number
of output ports. Output port 0 is used for successful results of the MemGet
interaction. This output port can be linked to a Writer in a CloverETL graph, to
write the results of the MemGet interaction to the target destination (such as
database table, text file) of your choice.
Additional ports 1-n can be used for mapping MemGet output. Different
combinations of attributes can be mapped to each output port. For example, first
name and last name can be mapped to port 0, while street, city, state are mapped
32
CloverETL User's Guide
to port 1. Segment and attribute values are cached when the Mapping dialog box
is opened, eliminating multiple trips to the server when configuring the
component.
About IBM Initiate MemGet metadata
CloverETL graphs using an IBM Initiate MemGet component use two distinct sets
of metadata: the input port metadata defines which data will be retrieved from the
IBM Initiate Hub, and the output port metadata defines the data being written to
the target destination. In many cases these two sets of metadata will not be
identical, and must be defined and applied independently.
Creating a graph with an IBM Initiate MemGet component
About this task
The basic steps required for creating a graph with an IBM Initiate MemGet
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to the external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Readers section of the Palette, select and drag an IBM Initiate
MemGet component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate MemGet component.
5.
6.
7.
8.
9.
Note: You can also add Transformers between the Reader and the IBM Initiate
MemGet component to filter, sort, or otherwise transform the data before
passing it to the IBM Initiate MemGet component.
Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being retrieved from the IBM Initiate Hub.
From the Writers section of the Palette, select and drag a Writer onto the
graph. This Writer component will process the information retrieved by the
IBM Initiate MemGet component.
Add an Edge to connect the IBM Initiate MemGet component to the Writer.
Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being written to the target file.
Double-click the IBM Initiate MemGet component to open the Edit
Component dialog.
10. Specify a connection to the IBM Initiate Hub by clicking in the Value field for
the IBM Initiate connection property and choosing a connection from the
drop-down list.
Note: If there are no connections in the drop-down, you can create one using
the steps outlined in “Connections in CloverETL” on page 13.
11. Map the IBM Initiate Hub attributes to the fields defined in the output
metadata:
a. Click in the Value field for the Mapping property, and click the ellipsis
control to open the Mapping dialog box.
Chapter 3. IBM Initiate custom components
33
Note: You must have an Initiate connection and metadata for the Edges
leading into and out of the IBM Initiate MemGet component defined
before you can map attributes.
b. On the Attributes tab, select the attributes to map using Ctrl-click. Click
Next.
c. On the Output port tab, drag the metadata Input fields (attributes) to the
output fields you want to map them to.
you can enable CloverETL to
d. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
You can also type in names of attribute values if the attribute values you
want to use are not in the Input list.
Note: The Output fields listed on the Output port tab correspond to the
fields in the Output Edge's metadata. If you have not added a Writer,
joined it to the IBM Initiate MemGet component with an Edge, and added
metadata to the Edge, as described in the steps above, you will not see
Output fields on the Output port tab.
e. Click Finish.
12. Enter additional properties in the Edit Component dialog, as needed to
configure the IBM Initiate MemGet component. Required properties are
marked with a yellow warning icon. Property definitions are given in the
MemGet Properties table below.
13. Save the graph.
IBM Initiate MemGet properties
The table below lists the properties you can define for the IBM Initiate MemGet
component.
Note: For additional information on MemGet, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 3. MemGet properties
Property
Value
Initiate connection
The MemGet component requires a connection to an IBM Initiate
Hub in order to retrieve attributes. Detailed instructions on how
to create a connection and specify it for this parameter are given
in “Connections in CloverETL” on page 13.
Get type
Get type is used to determine the types of data that is retrieved:
v As member: Returns only members that match the search
criteria
v As entity: Returns members that match the search criteria and
any members that are linked to those matching members
Key type
Key type defines the key that is used for retrieving data:
v MemIdnum: SrcCode and MemIdnum is the retrieval key
v MemRecno: MemRecno is the retrieval key
v EntRecno: EntRecno is the retrieval key
34
Member type
Specifies the type of Members to get
Entity type
Specifies the type of Entities to get
CloverETL User's Guide
Table 3. MemGet properties (continued)
Property
Value
Mapping
Use the Mapping parameter to map metadata fields to your IBM
Initiate Hub attributes. See “Creating a graph with an IBM Initiate
MemGet component” on page 33 for detailed instructions on
mapping.
Segment code filter
Filters by segment code. You can enter multiple values in this
field, separated by commas. The list of segment codes can be
found in the segCode field of the Identity Hub mpi_seghead table.
In order to retrieve any member segments, at least the
MEMHEAD segCode must always be specified.
In addition to specific segment codes, you can use the following
values as noted:
v ALL: all segments
v MEMALL: all member segments
v DICALL: all dictionary segments
v AUDALL: all audit segments
v MEMATTRALL: all member attribute segments
Segment attribute filter Filters results by segment attribute. You can enter multiple values
in this field, separated by commas.
Record status filter
Filters results by record status. Although optional, no attributes
(only memHead records) will be returned if this is not set.
Typically you would want to get at least 'A'ctive attributes. You
can select multiple status codes in the dialog box.
v A: Active
v I: Inactive
v D: Deleted
v S: Shadow
Source code filter
Filters results by source code. You can enter multiple values in
this field, separated by commas.
Member status filter
Filters results by member status. You can enter multiple values in
this field, separated by commas. Valid member status values are:
v A: Active
v O: Overlay
v M: Merged
v D: Deleted
Composite view name
Sets the Composite View Name. Composite views enable Initiate
Hub administrators to create and name a view that controls the
data that is returned for the interaction. Use "DEFAULT" to get the
default composite view.
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Use error port
Specifies how the last output port is used:
v True: port is used an error port. The tab for this port will
display ERR_CODE and ERR_MESSAGE attribute fields that
you can map to the applicable output fields.
v False: port is used as a standard port
Skip rows
Specifies how many rows should be skipped from the source port.
This is useful for handling files where first rows are a header
rather than real data.
Chapter 3. IBM Initiate custom components
35
Table 3. MemGet properties (continued)
Property
Value
Number of records
Specifies how many records should be read from the Initiate Hub
source.
Data policy
Specifies how to handle misformatted or incorrect data:
v Strict: aborts processing
v Controlled: logs the entire record while processing continues
v Lenient: attempts to set incorrect data to default values while
processing continues
The IBM Initiate MemSearch component
The MemSearch interaction uses selected criteria (such as minimum score, name,
address, date of birth, or ID number) to search and return a list of records
matching the criteria. The interaction creates a “virtual” member based on the
attribute criteria submitted and compares those attributes with other records in the
database.
Note: For additional information on MemSearch, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
The IBM Initiate MemSearch component requires input from a CloverETL Reader
component. The Reader supplies a connection to the external source of the data,
and can also apply filtering or other criteria to determine which data is passed on
to the Initiate MemSearch component.
CloverETL provides a number of Readers that can read data from sources such as
databases, text files, and LDAP repositories. Data can be routed to the IBM Initiate
MemSearch component via a Reader in the same way that it can be routed to an
Initiate MemPut component, as shown in the illustration in the section “The IBM
Initiate MemPut component” on page 50.
Creating a graph with an IBM Initiate MemSearch component
About this task
The basic steps required for creating a graph with an IBM Initiate MemSearch
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to your external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Readers section of the Palette, select and drag an IBM Initiate
MemSearch component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate MemSearch
component.
Note: You can also add Transformers between the Reader and the Initiate
MemSearch component to filter, sort, or otherwise transform the data before
passing it to the Initiate MemSearch component.
36
CloverETL User's Guide
5. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being retrieved from the Initiate Hub.
6. From the Writers section of the Palette, select and drag a Writer onto the
graph.
7. Add an Edge to connect the IBM Initiate MemSearch component to the
Writer.
8. Create and apply metadata to the Edges, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being written to the target file.
9. For multiple outputs, add other components as needed.
10. Double-click the IBM Initiate MemSearch component to open the Edit
Component dialog box.
11. Specify a connection to the IBM Initiate Hub by clicking in the Value field for
the IBM Initiate connection property and choosing a connection from the
drop-down list.
Note: If there are no connections in the drop-down, create one by following
the steps outlined in “Connections in CloverETL” on page 13.
12. Map your Initiate Hub attributes to the fields defined in the output metadata:
Note: You must have an IBM Initiate connection and metadata for the Edges
leading into and out of the IBM Initiate MemSearch component defined
before you can map attributes.
a. Click in the Value field for the Input Mapping property. An ellipsis is
displayed.
b. Click the ellipsis to open the MemSearch input mapping dialog.
c. Select the attributes to map by using Ctrl-click. Click Next.
d. The MemSearch input mapping dialog displays tabs for each of the
attributes you selected. Drag the metadata (input) fields to the attribute
fields you want to map them to.
Click each tab to expose the input and attribute fields for the selected
attributes.
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
g. In the Edit Component dialog box, click in the Value field for the Output
Mapping property. An ellipsis is displayed.
h. Click the ellipsis to open the MemSearch output mapping dialog.
i. Select the attributes to map using Ctrl-click. Click Next.
Note: You can select attributes for output mapping that were not selected
for input mapping, and then map the additional attributes to additional
output ports. This capability enables you to pass additional data through
from the Hub to the output when the search for the selected attributes is
successful.
j. On the Output port tab, drag the metadata input (attribute) fields to the
output fields you want to map them to.
Chapter 3. IBM Initiate custom components
37
You can also type in names of attribute values if the attribute values you
wish to use are not present in the Input list.
Note: You can define an output port for error codes and error messages.
You must define the Value for the property Use error port as True in the
Edit component dialog box to enable this functionality.
k. Click Finish.
13. Enter additional properties in the Edit Component dialog, as needed to
configure the IBM Initiate MemSearch component. Required properties are
marked with a yellow warning icon. Property definitions are given in the
MemSearch Properties table below.
14. Save the graph.
IBM Initiate MemSearch properties
The table below lists the properties you can define for the IBM Initiate
MemSearch component.
Note: For additional information on MemSearch, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 4. MemSearch properties
Property
Description
Initiate connection
The MemSearch component requires a connection to an IBM
Initiate Hub in order to retrieve attributes. Detailed instructions
on how to create a connection and specify it for this parameter are
given in “Connections in CloverETL” on page 13.
Get type
Get type specifies the type of member-level aggregation to
perform during retrieval operations:
v AsMember: Returns only members that match the search
criteria
v AsEntity: Returns members that match the search criteria and
any members that are linked to those matching members
Search type
SearchType is used by the interaction classes to specify the type of
member-level aggregation to perform during a search operations.
v AsMember: Tells the search to compare the input on a member
by member basis
v Unknown: Used as a placeholder when a Search type is not set
38
Member type
Member type specifies the Member type to search for.
Entity type
Entity type specifies the Entity type to search for.
Input Mapping
Use the Input Mapping parameter to map metadata fields to your
IBM Initiate Hub attributes. See “Creating a graph with an IBM
Initiate MemSearch component” on page 36 for detailed
instructions on mapping.
Output Mapping
Use the Output Mapping parameter to map metadata fields to
attributes for the output data. See “Creating a graph with an IBM
Initiate MemSearch component” on page 36 for detailed
instructions on mapping.
Key fields
The Key fields parameter specifies how records from different
input ports are joined. See “Using Key fields in CloverETL
MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to
join records” on page 31 for detailed instructions on using Key
fields.
CloverETL User's Guide
Table 4. MemSearch properties (continued)
Property
Description
Segment code filter
Filters by segment code. You can enter multiple values in this
field, separated by commas. The list of segment codes can be
found in the segCode field of the Identity Hub mpi_seghead table.
In order to retrieve any member segments, at least the
MEMHEAD segCode must always be specified.
In addition to specific segment codes, you can use the following
values as noted:
v ALL: all segments
v MEMALL: all member segments
v DICALL: all dictionary segments
v AUDALL: all audit segments
v MEMATTRALL: all member attribute segments
Segment attribute filter Filters results by segment attribute. You can enter multiple values
in this field, separated by commas.
Record status filter
Filters results by record status. Although optional, no attributes
(only memHead records) will be returned if this is not set.
Typically you would want to get at least 'A'ctive attributes. You
can select multiple status codes in the dialog box.
v A: Active
v I: Inactive
v D: Deleted
v S: Shadow
Source code filter
Filters results by source code. You can enter multiple values in
this field, separated by commas.
Member status filter
Filters results by member status. You can enter multiple values in
this field, separated by commas. Valid member status values are:
v A: Active
v O: Overlay
v M: Merged
v D: Deleted
Composite view name
Sets the Composite View Name. Composite views enable Hub
administrators to create and name a view that controls the data
that is returned for the interaction. Use "DEFAULT" to get the
default composite view.
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Min score
Sets the minimum score that a search result must have in order to
be returned in the output. Use zero to indicate that any score is
acceptable.
Max rows
Sets the maximum number of members to return in the output.
Entities returned are created from the member record results.
Use zero to indicate that any number of members or entities may
be returned.
Max candidate
Sets the maximum number of candidates that will be examined
during a search. A setting of zero indicates that any number of
candidates will be examined.
Chapter 3. IBM Initiate custom components
39
Table 4. MemSearch properties (continued)
Property
Description
Use error port
Specifies how the last output port is used:
v True: port is used an error port. The tab for this port will
display ERR_CODE and ERR_MESSAGE attribute fields that
you can map to the applicable output fields.
v False: port is used as a standard port
Skip rows
Specifies how many rows should be skipped from the source port.
This is useful for handling files where first rows are header rows
rather than real data.
Number of records
Specifies how many records should be read from the source.
Data policy
Specifies how to handle misformatted or incorrect data:
v Strict: aborts processing.
v Controlled: logs the entire record while processing continues
v Lenient: attempts to set incorrect data to default values while
processing continues
The IBM Initiate Handler Reader component
The IBM Initiate Handler Reader component retrieves data from the dictionary. It
passes attributes from a member row list into a graph where they can be
reformatted for temporary display and use, or passed back to the dictionary or to a
permanent file by a Writer component.
The IBM Initiate Handler Reader component has no input ports; the data is passed
in from the dictionary through an Initiate connection. There are multiple output
ports available for connecting to components such as Transformers and Joiners.
Creating a graph with an IBM Initiate Handler Reader component
About this task
The basic steps required for creating a graph with an IBM Initiate Handler Reader
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. From the Readers section of the Palette, select and drag an IBM Initiate
Handler Reader component onto the graph.
3. Add the appropriate components, such as Readers and Transformers, needed
to complete the graph.
4. Create and apply metadata to the Edges, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
Note that the metadata you define here describes the data being retrieved
from the IBM Initiate Hub.
5. Double-click the IBM Initiate Handler Reader component to open the Edit
Component dialog.
6. Specify a connection to the IBM Initiate Hub by clicking in the Value field for
the IBM Initiate connection property and choosing a connection from the
drop-down list.
40
CloverETL User's Guide
Note: If there are no connections in the drop-down, you can create one using
the steps outlined in “Connections in CloverETL” on page 13.
7. Click Apply.
8. Map the IBM Initiate Hub attributes to the fields defined in the output
metadata:
a. Click in the Value field for the Mapping property, and click the ellipsis
control to open the Mapping dialog box.
b. Select the attributes to map using Ctrl-click.
c. Click Next.
d. On the Output port tab, drag the metadata input (attribute) fields to the
output fields you want to map them to.
Note: The Output fields listed on the Output port tab correspond to the
fields in the Output Edge's metadata. If you have not added a Writer,
joined it to the IBM Initiate Handler Reader component with an Edge, and
added metadata to the Edge, as described in the steps above, you will not
see Output fields on the Output port tab.
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
9. Click Finish.
10. Enter additional properties in the Edit Component dialog as needed to
configure the IBM Initiate Handler Reader component. Required properties are
marked with a yellow warning icon. Property definitions are given in the
Handler Reader Properties table below.
11. Save the graph.
IBM Initiate Handler Reader properties
The table below lists the properties you can define for the IBM Initiate Handler
Reader component.
Table 5. Handler Reader properties
Property
Value
Initiate connection
The IBM Initiate Handler Reader component requires a connection
to an IBM Initiate Hub in order to retrieve attributes. Detailed
instructions on how to create a connection and specify it for this
property are given in “Connections in CloverETL” on page 13.
Dictionary key
Specifies the key that the component uses to find member rows in
the dictionary. The default value is inputMember.
Mapping
Use the Mapping property to map metadata fields to your IBM
Initiate Hub attributes.
Component name
Default label for the component. You can change this label to
make it more specific to your business uses.
Description
Optional user-defined explanation of the component's use.
ID
Internal identification.
Component type
Internal identification.
Specification
Description of the component's use.
Chapter 3. IBM Initiate custom components
41
Table 5. Handler Reader properties (continued)
Property
Value
Phase
For graph executions that include multiple graphs running
sequentially, indicates the location in the sequence for the graph
containing this component. Phase identifiers begin with 0 (zero).
Enabled
Status of the component within the graph. The options are:
v enabled — the component is active in the graph. It will parse
data and pass it to the next component.
v disabled — the component is inactive in the graph. A disabled
component will neither parse data nor pass data to the next
component.
v passThrough — the component is inactive in the graph. It will
not parse data, but will pass data to the next component
Input port
Output port
If the component's status is passThrough, enables you to specify
which input port should receive the data records and which
output port should pass them to the next component.
The IBM Initiate TskGet component
The IBM Initiate TskGet component retrieves task-related data using a known
member value such as an Member ID number (memidnum). or Member Record
number (memrecno) in the same manner as the TskGet API interaction. The TskGet
performs a cross-match using a member or entity task record as input to the cross
match, and returns all members in the same entity as the input member, as well as
any members that match above the Clerical Review threshold.
Arguments are supplied to the IBM Initiate TskGet component in the form of
component parameters.
Note: For additional information on TskGet, refer to the SDK interaction examples
and the IBM Initiate Java SDK Javadoc Information.
The Initiate TskGet component has one input port and an unlimited number of
output ports. Output port 0 is used for successful results of the TskGet interaction.
This output port can be linked to a Writer in a CloverETL graph to write the
results of the TskGet interaction to the target destination (such as database table,
text file) of your choice.
Additional ports 1-n can be used for mapping TskGet output. Different
combinations of attributes can be mapped to each output port.
About IBM Initiate TskGet metadata
CloverETL graphs using an IBM Initiate TskGet component use two distinct sets of
metadata: the input port metadata defines which data will be retrieved from the
IBM Initiate Hub, and the output port metadata defines the data being written to
the target destination. In many cases these two sets of metadata will not be
identical, and must be defined and applied independently.
Creating a graph with an IBM Initiate TskGet component
About this task
The basic steps required for creating a graph with an IBM Initiate TskGet
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
42
CloverETL User's Guide
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to the external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Readers section of the Palette, select and drag an IBM Initiate
TSKGET component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate TSKGET component.
Note: You can also add Transformers between the Reader and the IBM Initiate
TskGet component to filter, sort, or otherwise transform the data before
passing it to the IBM Initiate TskGet component.
5. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being retrieved from the IBM Initiate Hub.
6. From the Writers section of the Palette, select and drag one or more Writers
onto the graph. The Writer components will process the information retrieved
by the IBM Initiate TSKGET component.
7. Add one or more Edges to connect the IBM Initiate TSKGET component to
the Writer(s).
8. Create and apply metadata to the Edge(s), using the steps out lined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being written to the target file.
9. Double-click the IBM Initiate TSKGET component to open the Edit
Component dialog.
10. Specify a connection to the Initiate Hub by clicking in the Value field for the
IBM Initiate connection property and choosing a connection from the
drop-down list.
Note: If there are no connections in the drop-down, you can create one using
the steps outlined in “Connections in CloverETL” on page 13.
11. Map the IBM Initiate Hub attributes to the fields defined in the output
metadata:
a. Click in the Value field for the Mapping property, and click the ellipsis
control to open the Mapping dialog box.
Note: You must have an Initiate connection and metadata for the Edges
leading into and out of the IBM Initiate TSKGET component defined
before you can map attributes.
b. Select the attributes to map using Ctrl-click. Click Next..
Note: If you map non-segment attributes for output, such as entXtsk,
memXtsk, or idtXtsk, you must also select MEMHEAD or no records will
be returned.
c. On the Output port tab(s), drag the metadata input (attribute) fields to the
output fields you want to map them to.
you can enable CloverETL to
d. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
Chapter 3. IBM Initiate custom components
43
You can also type in names of attribute values if the attribute values you
want to use are not in the Input list.
Note: The Output fields listed on the Output port tab correspond to the
fields in the Output Edge's metadata. If you have not added a Writer,
joined it to the IBM Initiate TskGet component with an Edge, and added
metadata to the Edge, as described in the steps above, you will not see
Output fields on the Output port tab.
e. Click Finish.
12. Enter additional properties in the Edit Component dialog, as needed to
configure the IBM Initiate TSKGET component. Required properties are
marked with a yellow warning icon. Property definitions are given in the
TSKGET Properties table below.
13. Save the graph.
IBM Initiate TSKGET properties
The table below lists the properties you can define for the IBM Initiate TSKGET
component.
Note: For additional information on TSKGET, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 6. TSKGET properties
Property
Value
IBM Initiate connection The TSKGET component requires a connection to an Initiate Hub
in order to retrieve attributes. Detailed instructions on how to
create a connection and specify it for this parameter are given in
“Connections in CloverETL” on page 13.
Get type
Get type is used to determine the types of data that is retrieved:
v As member: Returns only members that match the search
criteria
v As entity: Returns members that match the search criteria and
any members that are linked to those matching members
Key type
Key type defines the key that is used for retrieving data:
v TskRecno: TskRecno is the retrieval key
v MemIdnum: SrcCode and MemIdnum is the retrieval key
v MemRecno: MemRecno is the retrieval key
v EntRecno: EntRecno is the retrieval key
44
Member type
Specifies the type of Members to get
Entity type
Specifies the type of Entities to get
Mapping
Use the Mapping parameter to map metadata fields to your IBM
Initiate Hub attributes. See “Creating a graph with an IBM Initiate
TskGet component” on page 42 for detailed instructions on
mapping.
CloverETL User's Guide
Table 6. TSKGET properties (continued)
Property
Value
Segment code filter
Filters by segment code. You can enter multiple values in this
field, separated by commas. The list of segment codes can be
found in the segCode field of the Identity Hub mpi_seghead table.
In order to retrieve any member segments, at least the
MEMHEAD segCode must always be specified.
In addition to specific segment codes, you can use the following
values as noted:
v ALL: all segments
v MEMALL: all member segments
v DICALL: all dictionary segments
v AUDALL: all audit segments
v MEMATTRALL: all member attribute segments
Segment attribute filter Filters results by segment attribute. You can enter multiple values
in this field, separated by commas.
Record status filter
Filters results by record status. Although optional, no attributes
(only memHead records) will be returned if this is not set.
Typically you would want to get at least 'A'ctive attributes. You
can select multiple status codes in the dialog box.
v A: Active
v I: Inactive
v D: Deleted
v S: Shadow
Source code filter
Filters results by source code. You can enter multiple values in
this field, separated by commas.
Member status filter
Filters results by member status. You can enter multiple values in
this field, separated by commas. Valid member status values are:
v A: Active
v O: Overlay
v M: Merged
v D: Deleted
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Use error port
Specifies how the last output port is used:
v True: port is used an error port. The tab for this port will
display ERR_CODE and ERR_MESSAGE attribute fields that
you can map to the applicable output fields.
v False: port is used as a standard port
Skip rows
Specifies how many rows should be skipped from the source port.
This is useful for handling files where first rows are a header
rather than real data.
Number of records
Specifies how many records should be read from the IBM Initiate
Hub source.
Data policy
Specifies how to handle misformatted or incorrect data:
v Strict: aborts processing.
v Controlled: logs the entire record while processing continues
v Lenient: attempts to set incorrect data to default values while
processing continues
Chapter 3. IBM Initiate custom components
45
Table 6. TSKGET properties (continued)
Property
Value
Advanced > IxnTskGet When using the Advanced Issue Management feature's
version
implementation-defined tasks, it is recommended that you use the
IxnTskGet2 version.
IxnTskGet2 is similar to the traditional IxnTskGet get interaction,
but IxnTskGet2 populates the new MEMRECNOs array in all task
types. The traditional IxnTskGet interaction leaves this field null.
In addition, IxnTskGet2 "rolls up" idtxtsk records with the same
TSKRECNO into a set so that only one implementation-defined
task object (idtxtsk) is returned for a taskset. This differs from the
traditional IxnTskGet, which will preset all mpi_idtxtsk records.
The IBM Initiate TskSearch component
The TskSearch interaction takes search criteria input (for example, task type, task
owner, or task status) and returns a list of members with task issues.
Note: For additional information on TskSearch, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
The IBM Initiate TskSearch component requires input from a CloverETL Reader
component. The Reader supplies a connection to the external source of the data,
and can also apply filtering or other criteria to determine which data is passed on
to the Initiate TskSearch component.
CloverETL provides a number of Readers that can read data from sources such as
databases, text files, and LDAP repositories. Data can be routed to the IBM Initiate
TskSearch component via a Reader in the same way that it can be routed to an
IBM Initiate MemPut component, as shown in the illustration in the section “The
IBM Initiate MemPut component” on page 50.
Creating a graph with an IBM Initiate TskSearch component
About this task
The basic steps required for creating a graph with an IBM Initiate TskSearch
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to your external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Readers section of the Palette, select and drag an IBM Initiate
TskSearch component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate TskSearch
component.
Note: You can also add Transformers between the Reader and the IBM Initiate
TskSearch component to filter, sort, or otherwise transform the data before
passing it to the IBM Initiate TskSearch component.
46
CloverETL User's Guide
5. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being retrieved from the IBM Initiate Hub.
6. From the Writers section of the Palette, select and drag one or more Writers
onto the graph.
7. Add one or more Edge(s) to connect the IBM Initiate TskSearch component
to the Writer(s).
8. Create and apply metadata to the Edges, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10. Note that the metadata
you define here describes the data being written to the target file.
9. Double-click the IBM Initiate TskSearch component to open the Edit
Component dialog box.
10. Specify a connection to the Initiate Hub by clicking in the Value field for the
IBM Initiate connection property and choosing a connection from the
drop-down list.
Note: If there are no connections in the drop-down, create one by following
the steps outlined in “Connections in CloverETL” on page 13.
11. Map your IBM Initiate Hub attributes to the fields defined in the output
metadata:
Note: You must have an IBM Initiate connection and metadata for the Edges
leading into and out of the IBM Initiate TskSearch component defined before
you can map attributes.
a. Click in the Value field for the Input Mapping property. An ellipsis is
displayed.
b. Click the ellipsis to open the TskSearch input mapping dialog.
c. Select the attributes to map by using Ctrl-click. Click Next.
d. The TskSearch input mapping dialog displays tabs for each of the
attributes you selected. Drag the metadata (input) fields to the attribute
fields you want to map them to.
Click each tab to expose the input and attribute fields for the selected
attributes.
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
g. In the Edit Component dialog box, click in the Value field for the Output
Mapping property. An ellipsis is displayed.
h. Click the ellipsis to open the TskSearch output mapping dialog.
i. Select the attributes to map using Ctrl-click. Click Next.
Note: You can select attributes for output mapping that were not selected
for input mapping, and then map the additional attributes to additional
output ports. This capability enables you to pass additional data through
from the Hub to the output when the search for the selected attributes is
successful.
j. On the Output port tabs, drag the metadata input (attribute) fields to the
output fields you want to map them to.
Chapter 3. IBM Initiate custom components
47
You can also type in names of attribute values if the attribute values you
wish to use are not present in the Input list.
Note: In the example above, the last output port is being used for error
codes and error messages. You must define True as the Value for the
property Use error port in the Edit component dialog box to enable this
functionality.
k. Click Finish.
12. Enter additional properties in the Edit Component dialog, as needed to
configure the IBM Initiate TskSearch component. Required properties are
marked with a yellow warning icon. Property definitions are given in the
TskSearch Properties table below.
13. Save the graph.
IBM Initiate TskSearch properties
The table below lists the properties you can define for the IBM Initiate TskSearch
component.
Note: For additional information on TskSearch, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 7. TskSearch properties
Property
Description
IBM Initiate connection The TskSearch component requires a connection to an IBM Initiate
Hub in order to retrieve attributes. Detailed instructions on how
to create a connection and specify it for this parameter are given
in “Connections in CloverETL” on page 13.
48
Member type
Member type specifies the Member type to search for.
Entity type
Entity type specifies the Entity type to search for.
Input Mapping
Use the Input Mapping parameter to map metadata fields to your
IBM Initiate Hub attributes. See “Creating a graph with an IBM
Initiate TskSearch component” on page 46 for detailed instructions
on mapping.
Output Mapping
Use the Output Mapping parameter to map metadata fields to
attributes for the output data. See “Creating a graph with an IBM
Initiate TskSearch component” on page 46 for detailed instructions
on mapping.
Key fields
The Key fields parameter specifies how records from different
input ports are joined. See “Using Key fields in CloverETL
MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to
join records” on page 31 for detailed instructions on using Key
fields.
MemRecnos key fields
Key fields of input metadata which group multiple memRecnos to
IdtXtsk rows. Use this option if you want to find multiple
members associated with a single custom task instance; the
custom task instance is associated with a “primary” member
record in one input file, and the additional member records are
mapped to the “primary” member record.
CloverETL User's Guide
Table 7. TskSearch properties (continued)
Property
Description
Segment code filter
Filters by segment code. You can enter multiple values in this
field, separated by commas. The list of segment codes can be
found in the segCode field of the Identity Hub mpi_seghead table.
In order to retrieve any member segments, at least the
MEMHEAD segCode must always be specified.
In addition to specific segment codes, you can use the following
values as noted:
v ALL: all segments
v MEMALL: all member segments
v DICALL: all dictionary segments
v AUDALL: all audit segments
v MEMATTRALL: all member attribute segments
Segment attribute filter Filters results by segment attribute. You can enter multiple values
in this field, separated by commas.
Record status filter
Filters results by record status. Although optional, no attributes
(only memHead records) will be returned if this is not set.
Typically you would want to get at least 'A'ctive attributes. You
can select multiple status codes in the dialog box.
v A: Active
v I: Inactive
v D: Deleted
v S: Shadow
Source code filter
Filters results by source code. You can enter multiple values in
this field, separated by commas.
Member status filter
Filters results by member status. You can enter multiple values in
this field, separated by commas. Valid member status values are:
v A: Active
v O: Overlay
v M: Merged
v D: Deleted
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Max rows
Sets the maximum number of members to return in the output.
Entities returned are created from the member record results.
Use zero to indicate that any number of members or entities may
be returned.
Max candidate
Sets the maximum number of candidates that will be examined
during a search. A setting of zero indicates that any number of
candidates will be examined.
Use error port
Specifies how the last output port is used:
v True: port is used an error port. The tab for this port will
display ERR_CODE and ERR_MESSAGE attribute fields that
you can map to the applicable output fields.
v False: port is used as a standard port
Skip rows
Specifies how many rows should be skipped from the source port.
This is useful for handling files where first rows are header rows
rather than real data.
Chapter 3. IBM Initiate custom components
49
Table 7. TskSearch properties (continued)
Property
Description
Number of records
Specifies how many records should be read from the source.
Data policy
Specifies how to handle misformatted or incorrect data:
v Strict: aborts processing.
v Controlled: logs the entire record while processing continues
v Lenient: attempts to set incorrect data to default values while
processing continues
IBM Initiate custom Writer components
This section describes the IBM Initiate Writer components: IBM Initiate MemPut,
IBM Initiate MemDelete, and IBM Initiate Handler Writer.
The IBM Initiate MemPut component
A MemPut interaction inserts or updates member data in the Hub database. The
IBM Initiate MemPut component processes and inserts or updates data in the same
manner as the MemPut API interaction. Arguments are supplied to the IBM Initiate
MemPut component in the form of component parameters.
Note: For additional information on MemPut, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
The IBM Initiate MemPut component requires input from a CloverETL Reader. The
Reader supplies a connection to the external source of the data, and can also apply
filtering or other criteria to determine which data is passed on to the MemPut
component.
CloverETL provides a number of Readers that can read data from sources such as
databases, text files, and LDAP repositories. The diagram below illustrates some of
the ways you can route data to the MemPut component via a Reader.
50
CloverETL User's Guide
Creating a graph with an IBM Initiate MemPut component
About this task
The basic steps required for creating a graph with an IBM Initiate MemPut
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to the external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Writers section of the Palette, select and drag an IBM Initiate
MemPut component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate MemPut component
Note: You can also add Transformers between the Reader and the IBM Initiate
MemPut component to filter, sort, or otherwise transform the data before
passing it to the IBM Initiate MemPut component.
5. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
6. Double-click the IBM Initiate MemPut component to open the Edit
Component dialog box.
7. Map IBM Initiate Hub attributes to the fields defined in the metadata:
a. Ensure that you have defined an IBM Initiate connection.
b. Click in the Value field for the Mapping property, and then click the
ellipsis control to open the MemPut Mapping dialog.
c. Select the attributes to map using Ctrl-click.
d. Click Next.
e. The MemPut Mapping dialog displays tabs for each of the attributes you
selected. Drag the metadata (input) fields to the attribute fields you want
to map them to.
you can enable CloverETL to
f. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
g. You can specify a row indicator setting for an individual attribute that
overrides a global row indicator setting in the Edit component dialog box.
1) Select the attribute.
2) Beside the RowInd/Filter table, click the green plus (+) button.
CloverETL inserts a row indicator value of Insert into the table.
3) To change the value, click the row in the RowInd column to display
the drop-down list, and then select the applicable value.
4) To add a filter expression to the row indicator value, click the row in
the Filter column to display the Filter dialog box. Build the expression
and click OK.
h. Click Finish.
8. If you want to map failure messages to an output file:
a. Add a Writer to capture the failure message.
Chapter 3. IBM Initiate custom components
51
b. Connect the Writer to the MemPut component output port 1, and add
Metadata to the Edge.
c. Open the MemPut component and click Output mapping, to map your
output as needed. When mapping failure messages, keep in mind the
following:
v Failure output should include the memrecno.
v Is it recommended that failure output include ERR_CODE and
ERR_MESSAGE.
9. Enter additional properties as needed to configure the IBM Initiate MemPut
component. Required properties are marked with a yellow warning icon.
Property definitions are given in “Initiate MemPut properties.”
10. Save the graph.
Initiate MemPut properties
The table below lists the properties you can define for the Initiate MemPut
component.
Note: For additional information on MemPut, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 8. MemPut properties
Property
Description
Initiate connection
Specify a connection to the Initiate Hub by clicking the Value field
and selecting from the drop-down list. The MemPut component
requires a connection to an Initiate Hub in order to retrieve
attributes.
If no connections are listed, you can create one using the steps
outlined in “Connections in CloverETL” on page 13.
Put type
PutType limits the types of changes that can occur for a given
interaction.
For example, you might want to update a Task. By using the
UpdateOnly setting, the Task will only be updated if it already
exists. By using the InsertOnly setting, the Task will only be
inserted if it does not already exist.
v InsertUpdate: If the member data does not exist, create it; or if
the member data already exists, update it
v InsertOnly: If the member data does not exist, create it; or if
the member data already exists, abort the interaction
v UpdateOnly: If the member data already exists, update it; or if
the member data does not exist, abort the interaction
52
CloverETL User's Guide
Table 8. MemPut properties (continued)
Property
Description
Member mode
Member mode specifies the type of input data present in the
member data presented to the interaction:
v Partial: The input member attributes do not contain a complete
picture of all the members attributes. The attributes will be
inserted and no overlay tasks will be created.
v AttrComp: For each given attribute, all values for that attribute
are present in the input member, so the Engine can make
decisions regarding the Active/Inactive status of historical
attributes. During put processing, this flag indicates that
attributes in the database that share the same attrRecno, but are
not included in the input should have their recStat set to
Inactive.
v Complete: Complete member is present in the input record.
This is the only setting that supports the creation of Overlay
tasks. The Engine can compare this complete input member to
the image in the database to make decisions regarding the
creation of an overlay task.
v Explicit: The input member is used as a map for what to
insert/update/delete. No overlay tasks will be created
Match mode
Match mode specifies the type of matching to perform after the
put interaction:
v Immediate: Cross match the member immediately following the
update of the member data. This option has no additional effect
in asynchronous mode
v Deferred: Member matches when Entity Manager gets to this
spot in the queue
v DoNothing: Update the member data, but do not cross match
the updated data against other members in the Hub database.
The interaction returns control to the caller as soon as the
update is complete. This blocks this member from being
matched when it is put, but another put that has buckets in
common could cause this member to be matched.
Mapping
Use the Mapping parameter to map metadata fields to your Hub
database attributes. Refer to “Creating a graph with an IBM
Initiate MemPut component” on page 51 for more information on
mapping.
Output Mapping
If you want to map failure messages to a file, use the output
mapping option to map failure messages to port 1.
Key fields
The Key fields parameter specifies how records from different
input ports are joined. See “Using Key fields in CloverETL
MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to
join records” on page 31 for detailed instructions on using Key
fields.
Entity type
Specifies the type of Entity to process.
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Chapter 3. IBM Initiate custom components
53
Table 8. MemPut properties (continued)
Property
Description
RowInd
Specifies a global action that the Master Data Engine should
perform on all rows that are passed in. The values are:
v (I)gnore
v (I)nsert
v (U)pdate
v (D)elete
The RowInd property is also available for individual attributes.
The individual attribute setting overrides the global setting.
Number of skipped
records
Specifies how many records/rows should be skipped from the
source port. This is useful for handling files where first rows are a
header not a real data.
Number of written
records
Specifies how many members should be written to the Initiate
Hub. If no number is specified, the number of records written is
not limited.
The IBM Initiate MemDelete component
A MemDelete component inactivates member data in the Hub database in the
same manner as the MemDelete API interaction. Arguments are supplied to the
IBM Initiate MemDelete component in the form of component parameters.
Note: For additional information on MemDelete, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
The IBM Initiate MemDelete component requires input from a CloverETL Reader
component. The Reader supplies a connection to the external source of the data,
and can also apply filtering or other criteria to determine which data is passed on
to the IBM Initiate MemDelete component.
Creating a graph with the IBM Initiate MemDelete component
About this task
The basic steps required for creating a graph with an IBM Initiate MemDelete
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to your external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Writers section of the Palette, select and drag an IBM Initiate
MemDelete component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate MemDelete
component.
Note: To filter, sort, or otherwise transform the data before passing it to the
IBM Initiate MemDelete component, add Transformers between the Reader
and the IBM Initiate MemDelete component.
5. Press [ESC] or click Select in the Palette to turn off the Edge tool.
54
CloverETL User's Guide
6. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
7. Double-click the IBM Initiate MemDelete component to open the Edit
Component dialog.
8. Specify a connection to the IBM Initiate Hub by clicking in the Value field for
the IBM Initiate connection property and choosing a connection from the
drop-down list.
If no connections exist in the drop-down list, refer to “Connections in
CloverETL” on page 13 to create a connection.
9. Click the Value field for the Key type property to select the key to be used for
retrieving data.
10. Specify the mapping of metadata fields to the IBM Initiate Hub attributes by
clicking the Value field of the Mapping property to open the Mapping dialog
box. Refer to “Creating a graph with an IBM Initiate MemSearch component”
on page 36 for more information about mapping.
11. If you want to map failure messages to an output file:
a. Add a Writer to capture the failure message.
b. Connect the Writer to the MemDelete component output port 1, and add
Metadata to the Edge.
c. Open the MemDelete component and click Output mapping, to map your
output as needed. When mapping failure messages, keep in mind the
following:
v Failure output should include the memrecno.
v Is it recommended that failure output include ERR_CODE and
ERR_MESSAGE.
12. Specify other properties as needed, and then click OK to save the graph.
IBM Initiate MemDelete properties
This table lists the properties you can define for the IBM Initiate MemDelete
component.
Note: For additional information on MemDelete, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 9. MemDelete properties
Property
Values
IBM Initiate
connection
The MemDelete component requires a connection to an IBM Initiate
Hub in order to retrieve attributes. Detailed instructions on how to
create a connection and specify it for this parameter are given in
“Connections in CloverETL” on page 13.
Key type
Key type defines the key that is used for retrieving data:
v MemIdnum: SrcCode and MemIdnum combination is the
retrieval key.
v MemRecno: MemRecno is the retrieval key.
Key types are mutually exclusive. Every input record is mapped to
a single MemDelete execution.
Mapping
Use the Mapping parameter to map metadata fields to your Hub
database attributes.
Output mapping
If you want to map failure messages to a file, use the output
mapping option to map failure messages to port 1.
Chapter 3. IBM Initiate custom components
55
The IBM Initiate Handler Writer component
The IBM Initiate Handler Writer component writes data to a shared resource or
temporary memory. It is often used to display reformatted data for temporary use
rather than to produce a permanent file or update a storage system.
The IBM Initiate Handler Writer component has no input ports; the data is passed
in from the dictionary through an IBM Initiate connection. There are multiple
output ports available for connecting to components such as Transformers and
Joiners.
Creating a graph with an IBM Initiate Handler Writer component
About this task
The basic steps required for creating a graph with an IBM Initiate Handler Reader
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. From the Readers section of the Palette, select and drag a Reader component
onto the graph.
3. From the Writers section of the Palette, select and drag an IBM Initiate
Handler Writer component onto the graph.
4. Add intermediate components needed to complete the graph.
5. Create and apply metadata to the Edges, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
6. Double-click each component to open its Edit Component dialog and edit the
component's properties.
7. In the IBM Initiate Handler Writer Edit component dialog, specify a
connection to the IBM Initiate Hub by clicking in the Value field for the IBM
Initiate connection property and choosing a connection from the drop-down
list. Click Apply.
Note: If there are no connections in the drop-down, you can create one using
the steps outlined in “Connections in CloverETL” on page 13.
8. Map the IBM Initiate Hub attributes to the fields defined in the output
metadata:
a. Click in the Value field for the Mapping property, and click the ellipsis
control to open the Mapping dialog box.
b. Select the attributes to map by using Ctrl-click.
c. Click Next.
d. Drag the metadata input (attribute) fields to the output fields you want to
map them to.
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
9. Enter additional properties in the Edit Component dialog, as needed to
configure the IBM Initiate Handler Writer component. Required properties are
56
CloverETL User's Guide
marked with a yellow warning icon. Property definitions are given in “IBM
Initiate Handler Writer properties.”
10. Save the graph.
IBM Initiate Handler Writer properties
The table below lists the properties you can define for the IBM Initiate Handler
Writer component.
Table 10. Handler Writer properties
Property
Value
IBM Initiate connection The Handler Writer component requires a connection to an IBM
Initiate Hub in order to display attributes. Detailed instructions on
how to create a connection and specify it for this property are
given in “Connections in CloverETL” on page 13.
Dictionary key
Specifies the key that the component uses to store member rows
in the dictionary. The default value is outputMember.
Mapping
Use the Mapping property to map metadata fields to your IBM
Initiate Hub attributes.
Key fields
The Key fields property specifies how records from different input
ports are joined. See “Using Key fields in CloverETL MemPut,
MemSearch, TskSearch, TskPut, and Handler Writer to join
records” on page 31 for detailed instructions on using Key fields.
Component name
Default label for the component. You can change this label to
make it more specific to your business uses.
Description
Optional user-defined explanation of the component's use.
ID
Internal identification.
Component type
Internal identification.
Specification
Description of the component's use.
Phase
For graph executions that include multiple graphs running
sequentially, indicates the location in the sequence for the graph
containing this component. Phase identifiers begin with 0 (zero).
Enabled
Status of the component within the graph. The options are:
v enabled — the component is active in the graph. It will parse
data and pass it to the next component.
v disabled — the component is inactive in the graph. A disabled
component will neither parse data nor pass data to the next
component.
v passThrough — the component is inactive in the graph. It will
not parse data, but will pass data to the next component
Input port
Output port
If the component's status is passThrough, enables you to specify
which input port should receive the data records and which
output port should pass them to the next component.
The IBM Initiate TskPut Component
The IBM Initiate TskPut component is designed to let you insert custom task data
into the Hub and associate it with existing member records. This functionality is
part of the Advanced Issue Management feature, which allows users to create
implementation-defined or custom task types, to supplement Initiate's predefined
task types.
Chapter 3. IBM Initiate custom components
57
For an overview of the Advanced Issue Management feature, including
information about creating custom task types in IBM Initiate Workbench,
determining which data to include as part of your custom tasks, and resolving
custom tasks in IBM Initiate Inspector, refer to the IBM Initiate Master Data Service
Hub Overview. You should be familiar with the information in the Advanced
Issue Management section of the IBM Initiate Master Data Service Hub
Overview before creating an IBM Initiate TskPut graph.
The IBM Initiate TskPut component takes custom task data from an external
source, inserts it into the Hub, and associates it with existing member records
which are already present in the Hub.
In a CloverETL TskPut graph, custom task data is read into the graph via a
CloverETL Reader component. The Reader supplies a connection to the external
source of the data, and can also apply filtering or other criteria to determine which
data is passed on to the TskPut component.
When planning your IBM Initiate TskPut graph, it is helpful to keep the following
in mind:
v The IBM Initiate TskPut component can be configured to use either IxnTskPut or
IxnTskPut2 as the ixnTskPut version. When you are putting custom task data
which references more than one member record, always use IxnTskPut2.
v If you will assign a single custom task instance to multiple member records
using IxnTskPut2, you should plan to provide separate input for the custom task
data and for the member record data. You can map custom task data to member
record data in the TSKPUT component via key fields. The sample graph in the
following section illustrates how this is done.
v When mapping your ouput, you should include both the taskrecno and the
memrecno; otherwise you risk outputting incomplete information.
v Hints let you tell the IBM Initiate Inspector application to highlight certain fields
in the task resolution screen, to help IBM Initiate Inspector users find and
resolve data issues, and descriptions let you add descriptive text to the custom
task data, to further refine the level of detail for your custom tasks. If you plan
to use hints or descriptions, be sure to map these as part of your TSKPUT
output. Hints are mapped from the idtXtsk table (idtXtsk.hint); descriptive text is
also mapped from the idtXtsk table, but the field name in the TSKPUT
component is “text” (idtXtsk.text).
Creating a graph with an IBM Initiate TskPut component
About this task
Note: Before creating an IBM Initiate TskPut graph, you should review the
overview of the Advanced Issue Management feature in the IBM Initiate Master
Data Service Hub Overview.
The basic steps required for creating a graph with an IBM Initiate TskPut
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
The example here shows how to take custom task data from multiple external
sources (in this case, a set of custom task data, and a list of member records the
custom data will be applied to) and map them together. This use case is a typical
one if you want to apply a given custom task instance to more than one member
record. If you have a one-to-one relationship between your custom tasks instances
58
CloverETL User's Guide
and member records (that is, a given custom task is applied only to a single
member record), you do not need to add a second external data source.
When mapping custom task instances to multiple member records, you must
include a single member record number (memrecno) in the custom task data
(which will serve as your ‘primary” member record), then use keys to map
additional member records from the other data source(s) to the “primary” member
record in the custom task data.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader for your custom task data and connect it to the external data
source, using the steps outlined in “Adding and configuring a CloverETL
Reader” on page 8.
3. Optional. Add a second Reader for your additional member record data, and
connect it to the external data source.
4. From the Writers section of the Palette, select and drag an IBM Initiate
TskPut component onto the graph.
5. Add Edges to connect the Readers to the IBM Initiate TskPut component.
Note: You can also add Transformers between the Reader and the IBM Initiate
TskPut component to filter, sort, or otherwise transform the data before
passing it to the IBM Initiate TskPut component.
6. Create and apply metadata to the Edges, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
When creating metadata for the custom task and member record Edges, keep
in mind the following:
v If you are using hints or descriptions, be sure that your source data and
Edge metadata include the hint and text fields.
v In this example, which maps additional member records to the custom task
data, the member record source data and Edge metadata should include
keys to map the member records in the member record source data to the
“primary” member record listed in the custom task data.
The following figures show sample metadata for a custom task data set
(idxtsk) and a second set of data for additional member records
(memRecnos).
Chapter 3. IBM Initiate custom components
59
Figure 1. Sample metadata for a custom task set (idxtk)
Figure 2. Sample metadata for additional task records in the custom task set (idxtsk)
7. Double-click the IBM Initiate TskPut component to open the Edit Component
dialog box.
8. Map IBM Initiate Hub attributes to the fields defined in the metadata:
a. Ensure that you have defined an IBM Initiate connection.
b. Click in the Value field for the Mapping property, and then click the
ellipsis control to open the TskPut Mapping dialog.
c. Select the any attributes or non-attribute segments to map, using Ctrl-click.
60
CloverETL User's Guide
Note: You must include at minimum the IdtXtsk non-attribute segment.
d. Click Next.
e. The TskPut Mapping dialog displays tabs for each of the attributes you
selected. Drag the metadata (input) fields to the attribute fields you want
to map them to.
you can enable CloverETL to
By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. You can specify a row indicator setting for an individual attribute that
overrides a global row indicator setting in the Edit component dialog box.
1) Select the attribute.
2) Beside the RowInd/Filter table, click the green plus (+) button.
CloverETL inserts a row indicator value of Insert into the table.
3) To change the value, click the row in the RowInd column to display the
drop-down list, and then select the applicable value.
4) To add a filter expression to the row indicator value, click the row in
the Filter column to display the Filter dialog box. Build the expression
and click OK.
g. Click Finish.
9. If you want to map success and/or failure messages to output files:
a. Add a Writer for each kind of output you want to capture (success and/or
failure).
b. Connect the Writers to the TskPut component's output ports, and add
Metadata to the Edges. Port 0 is for success messages and port 1 is for
failures.
c. Open the TskPut component and click Output mapping, to map your
output as needed. When mapping success and failure messages, keep in
mind the following:
v Both success and failure output should include the memrecno.
v Is it recommended that failure output include ERR_CODE and
ERR_MESSAGE.
v If you used keys to join data from multiple input files, include these
keys in your output as well.
10. Enter additional properties as needed to configure the IBM Initiate TskPut
component. Required properties are marked with a yellow warning icon.
Property definitions are given in “Initiate TskPut properties.”
11. Save the graph.
Initiate TskPut properties
The table below lists the properties you can define for the Initiate TskPut
component.
Note: For additional information on TskPut, refer to the SDK interaction examples
and the IBM Initiate Java SDK Javadoc Information.
Chapter 3. IBM Initiate custom components
61
Table 11. TskPut properties
Property
Description
Initiate connection
Specify a connection to the Initiate Hub by clicking the Value field
and selecting from the drop-down list. The TskPut component
requires a connection to an Initiate Hub in order to retrieve
attributes.
If no connections are listed, you can create one using the steps
outlined in “Connections in CloverETL” on page 13.
Put type
PutType limits the types of changes that can occur for a given
interaction.
For example, you might want to update a Task. By using the
UpdateOnly setting, the Task will only be updated if it already
exists. By using the InsertOnly setting, the Task will only be
inserted if it does not already exist.
v InsertUpdate: If the member data does not exist, create it; or if
the member data already exists, update it
v InsertOnly: If the member data does not exist, create it; or if
the member data already exists, abort the interaction
v UpdateOnly: If the member data already exists, update it; or if
the member data does not exist, abort the interaction
Member mode
Member mode specifies the type of input data present in the
member data presented to the interaction:
v Partial: The input member attributes do not contain a complete
picture of all the members attributes. The attributes will be
inserted and no overlay tasks will be created.
v AttrComp: For each given attribute, all values for that attribute
are present in the input member, so the Engine can make
decisions regarding the Active/Inactive status of historical
attributes. During put processing, this flag indicates that
attributes in the database that share the same attrRecno, but are
not included in the input should have their recStat set to
Inactive.
v Complete: Complete member is present in the input record.
This is the only setting that supports the creation of Overlay
tasks. The Engine can compare this complete input member to
the image in the database to make decisions regarding the
creation of an overlay task.
v Explicit: The input member is used as a map for what to
insert/update/delete. No overlay tasks will be created
Match mode
Match mode specifies the type of matching to perform after the
put interaction:
v Immediate: Cross match the member immediately following the
update of the member data. This option has no additional effect
in asynchronous mode
v Deferred: Member matches when Entity Manager gets to this
spot in the queue
v DoNothing: Update the member data, but do not cross match
the updated data against other members in the Hub database.
The interaction returns control to the caller as soon as the
update is complete. This blocks this member from being
matched when it is put, but another put that has buckets in
common could cause this member to be matched.
62
CloverETL User's Guide
Table 11. TskPut properties (continued)
Property
Description
Mapping
Use the Mapping parameter to map metadata fields to your Hub
database attributes. Refer to “Creating a graph with an IBM
Initiate TskPut component” on page 58 for more information on
mapping.
Output Mapping
If you want to map success and/or failure messages to a file, use
the output mapping option to map success messages to port 0 and
failure messages to port 1.
Key fields
The Key fields parameter specifies how records from different
input ports are joined. See “Using Key fields in CloverETL
MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to
join records” on page 31 for detailed instructions on using Key
fields.
MemRecnos key fields
Key fields of input metadata which group multiple memRecnos to
IdtXtsk rows. Use this option if you want to associate a single
custom task instance with more than one member; the custom
task instance is associated with a “primary” member record in one
input file, and the additional member records, in a different input
file, are mapped to the “primary” member record.
Entity type
Specifies the type of Entity to process.
Client args
Client args are used for external pre- and post-invocation
handlers. For more information, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
RowInd
Specifies a global action that the Master Data Engine should
perform on all rows that are passed in. The values are:
v (I)gnore
v (I)nsert
v (U)pdate
v (D)elete
The RowInd property is also available for individual attributes.
The individual attribute setting overrides the global setting.
Number of skipped
records
Specifies how many records/rows should be skipped from the
source port. This is useful for handling files where first rows are a
header not a real data.
Number of written
records
Specifies how many members should be written to the Initiate
Hub. If no number is specified, the number of records written is
not limited.
Advanced > IxnTskPut Specifies which version of of the IxnTskPut interaction to use:
version
IxnTskPut, or IxnTskPut2. When working with custom task data
which references more than one member record, always use
IxnTskPut2.
IBM Initiate custom Transformer components for CloverETL
This section describes the IBM Initiate Transformer components: IBM Initiate
MemScore, IBM Initiate Atrribute Reformat, and IBM Initiate MemSeqNo
Incrementer.
IBM Initiate custom components for address standardization are described in “IBM
Initiate custom address-standardization components for CloverETL” on page 71.
Chapter 3. IBM Initiate custom components
63
The IBM Initiate MemScore component
The IBM Initiate MemScore component runs the scoring algorithm of the IBM
Initiate Hub on 2 or more records that are not stored in the IBM Initiate Hub; the
first record is the target against which the others are scored.
The IBM Initiate MemScore component requires input from a CloverETL Reader.
The Reader supplies a connection to the external source of the data, for both the
first record (that is, the target) and other records that will be scored against it.
CloverETL provides a number of Readers which can read data from sources such
as databases, text files, and LDAP repositories.
The IBM Initiate MemScore component allows multiple input ports for both the
target record and the records scored against it. The target record data is supplied
on the Master port(s), and the comparison records are supplied on the Slave
port(s). If you will use multiple Master ports (for example, to supply a name and
address from one data source, and a phone number from a second data source),
the data must be joined via a Master key field. Likewise, comparison data supplied
via multiple Slave ports must be joined by a Slave key field.
Note: For additional information on IBM Initiate MemScore, refer to the SDK
interaction examples and the IBM Initiate Java SDK Javadoc Information.
Creating a graph with an IBM Initiate MemScore component
About this task
The basic steps required for creating a graph with an IBM Initiate MemScore
component are described below. For detailed information about working with
Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to your external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8. If you
will use multiple data sources for either the Master or Slave inputs, add and
connect a Reader for each.
3. From the Transformers section of the Palette, select and drag an IBM Initiate
MemScore component onto the graph.
4. Add an Edge to connect each Reader to the IBM Initiate MemScore
component. Note that you can connect many Readers to a single IBM Initiate
MemScore component.
5. Create and apply metadata to the Edge(s), using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
6. Add a Writer component to write the output of the IBM Initiate MemScore
component:
a. From the Writers section of the Palette, select and drag Writer component
onto the graph.
b. Add an Edge to connect the IBM Initiate MemScore component to the
Writer.
c. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
7. Double-click the IBM Initiate MemScore component to open the Edit
Component dialog.
64
CloverETL User's Guide
8. Specify a connection to the IBM Initiate Hub by clicking in the Value field for
the IBM Initiate connection property and choosing a connection from the
drop-down list.
Note: If there are no connections in the drop-down, you can create one using
the steps outlined in “Connections in CloverETL” on page 13.
9. Specify the Count of Master ports.
10. Specify the Count of Slave Members. This is the number of slave records that
will be scored against the master record.
11. Map the Input Master attributes to the fields defined in the metadata:
a. Click in the Value field for the Input Master mapping property. An ellipsis
is displayed.
b. Click the ellipsis to open the MemScore master mapping dialog.
Note: You must have an IBM Initiate connection defined before you can
map attributes.
c. Select the attributes to map using Ctrl-click. Click Next.
d. The MemScore master mapping dialog displays tabs for each of the
attributes you selected. Drag the metadata (input) fields to the attribute
fields you want to map them to.
Note: The Count specified for the Master ports in the step above
determines which ports are read for the Master mapping. For example, if
the Master port count is set to one, the Master mapping dialog will display
input fields for one port (Port 0); if the Master port count is set to two, the
Master mapping dialog will display input fields for two ports (Ports 0 and
1).
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
12. Map the Input Slave attributes to the fields defined in the metadata:
a. Click in the Value field for the Input Slave mapping property. An ellipsis is
displayed.
b. Click the ellipsis to open the MemScore slave mapping dialog.
Note: You must have an IBM Initiate connection defined before you can
map attributes.
c. Select the attributes to map using Ctrl-click. Click Next.
d. The MemScore slave mapping dialog displays tabs for each of the
attributes you selected. Drag the metadata (input) fields to the attribute
fields you want to map them to.
Note: The Count specified for the Slave ports in the step above determines
how many ports are displayed in the input fields area of this dialog.
e. By clicking the Auto mapping button, you can enable CloverETL to
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
13. Map the Output attributes to the fields in the metadata.
Chapter 3. IBM Initiate custom components
65
a. Click in the Value field for the Output mapping property. An ellipsis is
displayed.
Note: If you have not added a Writer to the graph and connected it to the
MemScore component via an Edge, you will not be able to open the
MemScore output mapping dialog.
b. Click the ellipsis to open the MemScore output mapping dialog.
Note: You must have an IBM Initiate connection defined before you can
map attributes.
c. Select the attributes to map using Ctrl-click.
d. Drag the metadata input (attribute) fields to the output fields you want to
map them to.
e. Click OK.
14. If you want to map failure messages to an output file:
a. Add a Writer to capture the failure message.
b. Connect the Writer to the MemScore component output port 1, and add
Metadata to the Edge.
c. Open the MemScore component and click Output mapping, to map your
output as needed. When mapping failure messages, keep in mind the
following:
v Failure output should include the memrecno.
v Is it recommended that failure output include ERR_CODE and
ERR_MESSAGE.
15. Save the graph.
IBM Initiate MemScore properties
The table below lists the properties you can define for your IBM Initiate MemScore
component.
Note: For additional information on MemScore, refer to the SDK interaction
examples and the IBM Initiate Java SDK Javadoc Information.
Table 12. MemScore Properties
Property
Values
IBM Initiate connection The MemScore component requires a connection to an IBM
Initiate Hub in order to retrieve attributes. Detailed instructions
on how to create a connection and specify it for this parameter are
given in “Connections in CloverETL” on page 13.
66
Member Type
Member type specifies the Member type to score.
Entity type
Specifies the type of Entity to process.
Input Master mapping
Use the Input Master mapping parameter to map master metadata
fields to your IBM Initiate Hub attributes. “Creating a graph with
an IBM Initiate MemScore component” on page 64 for detailed
instructions on mapping.
Input Slave mapping
Use the Input Slave mapping parameter to map slave metadata
fields to your IBM Initiate Hub attributes. “Creating a graph with
an IBM Initiate MemScore component” on page 64 for detailed
instructions on mapping.
CloverETL User's Guide
Table 12. MemScore Properties (continued)
Property
Values
Output mapping
Use the Output mapping parameter to map metadata fields to
attributes for the output data. See “Creating a graph with an IBM
Initiate MemScore component” on page 64 for detailed
instructions on mapping. If you want to map failure messages to a
file, use the output mapping option to map failure messages to
port 1.
Count of Master Ports
The number of input ports that are in the Master set (for example,
if Count of Master Ports is 3, then the first 3 ports are in the
Master set, and the remaining ports are in the Slave set). These
ports are used to read the Master member, against which the
other members (read from the Slave port set) are scored.
If you will use more than one Master port to combine data from
multiple sources, you must use the Master key field to join
records from these multiple sources.
Count of Slave
Members
The number of Slave members read from the Slave set of ports,
which are used in one MemScore interaction.
Master Key fields
This parameter specifies how records from different input ports
are joined. The key is used to construct the Master member from
the data coming from the master ports. This attribute is
mandatory if there is more than 1 Master port.
See “Using Key fields in CloverETL MemPut, MemSearch,
TskSearch, TskPut, and Handler Writer to join records” on page 31
section for detailed instructions on using Key fields.
Slave Key fields
This parameter specifies how records from different input ports
are joined. The key is used to construct the Slave members from
the data coming from the Slave ports. This attribute is mandatory
if there is more than 1 Slave port.
See “Using Key fields in CloverETL MemPut, MemSearch,
TskSearch, TskPut, and Handler Writer to join records” on page 31
for detailed instructions on using Key fields.
The IBM Initiate Attribute Reformat component
The custom IBM Initiate Attribute Reformat component, similar to the standard
CloverETL Reformat component, receives data from a Reader through a single
connected input port, transforms it in a user-specified way, and sends the
reformatted outgoing data records to the connected output port(s). However, the
IBM Initiate Attribute Reformat component includes the additional Attribute code
property, in which you can identify the attribute record number of an attribute
from the IBM Initiate Master Data Service.
Transformations that this component can execute include:
v
v
v
v
v
changing the number of fields in a record
changing case for alpha characters
converting data types (from one type to another)
concatenating or reordering data fields
changing metadata
Chapter 3. IBM Initiate custom components
67
Creating a graph with the IBM Initiate Attribute Reformat
component
About this task
The basic steps required for creating a graph with an IBM Initiate Attribute
Reformat component are described below. For detailed information about working
with Readers and metadata, see “Working with CloverETL graphs” on page 6.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. Add a Reader and connect it to your external data source, using the steps
outlined in “Adding and configuring a CloverETL Reader” on page 8.
3. From the Writers section of the Palette, select and drag an IBM Initiate
Attribute Reformat component onto the graph.
4. Add an Edge to connect the Reader to the IBM Initiate Attribute Reformat
component.
Note: To filter, sort, or otherwise transform the data before passing it to the
IBM Initiate Attribute Reformat component, add Transformers between the
Reader and the Initiate Attribute Reformat component.
5. Press [ESC] or click Select in the Palette to turn off the Edge tool.
6. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
7. Double-click the IBM Initiate Attribute Reformat component to open the Edit
Component dialog.
In the edit dialog box, you must specify how the data is to be transformed by
editing one of the Transform properties.
v In the Transform class property, you can specify the path and file name of a
class, jar, or zip file.
v In the Transform property, you define the transform within the graph by
using Java or Clover transformation language (CTL).
v In the Transform URL property, you can specify the path and file name of a
Java or CTL file.
8. Click the Transform property that you want to use, and then click the ellipsis
that is displayed.
For detailed information about the Transform properties, refer to the
CloverGUI Help.
v For the Transform class property, the Open Type wizard is displayed. Type
a few characters in the text field that identify the file to populate the wizard
with file names that match.
Select the appropriate file, and then click OK.
v For the Transform property, the Transform editor is displayed.
You can define the transformation in several ways:
– dragging and dropping input fields in the Transformations tab
– using CTL in the Source tab
– clicking Open Tab to open a window in the graph editor and using CTL
v For the Transform URL property, the URL File dialog is displayed.
Browse to the local or remote file, select it, and then click OK.
68
CloverETL User's Guide
9. To specify that the transformation is applicable to an individual attribute, click
the Attribute code property and the resulting ellipsis.
Note: This feature requires an IBM Initiate connection. If you have not
already done so, you can define an IBM Initiate connection by following the
steps outlined in “Connections in CloverETL” on page 13.
10. In the Select attribute dialog box, select an IBM Initiate connection and click
Connect.
The dialog box is populated with the attribute codes from the source data.
11. Select the applicale attribute code, and then click OK.
The Attribute code field is populated with the AttrRecNo for the selected
attribute.
12. Click OK to save the transformation properties and close the dialog box.
The IBM Initiate MemSeqNoIncrementer component
The IBM Initiate Member Sequence Number Incrementer
(MemSeqNoIncrementer) component is used to assign memseqno values to data, in
a batch mode. The memseqno is a system-generated value that is used to provide a
unique primary key per attribute within a member. For example, a record may
include two phone attributes—one for home phone number (homePhone) and
another for mobile phone number (mobilePhone); each are of the type memphone, but
each would be assigned a unique memseqno.
The MemSeqNoIncrementer component is used primarily with the IBM Initiate
Member Model Transform Graph. It is created automatically as part of the graph
produced by the Member Model Transform Graph wizard.
For more information on creating a Member Model Transform Graph, see “The
IBM Initiate Member Model Transform Graph” on page 24.
However, you can also use the IBM Initiate MemSeqNoIncrementer component
in other graphs, to assign memseqno values to specified fields, one by one. Records
with changed memseqno values can be reformatted according to a formula specified
in the IBM Initiate MemSeqNoIncrementer component's Field Mapping attribute.
IBM Initiate MemSeqNoIncrementer attributes
This section describes the attributes of the IBM Initiate MemSeqNoIncrementer
component. You can use this information to undertand how IBM Initiate
MemSeqNoIncrementer components are generated as part of an IBM Initiate
Member Model Transform Graph, or to build your own graphs utilizing the IBM
Initiate MemSeqNoIncrementer component.
IBM Initiate MemSeqNoIncrementer components should be concatenated to
assign memseqno values to more than one kind of attribute.
To edit an IBM Initiate MemSeqNoIncrementer component's attributes,
double-click the component to open the Edit component window.
Note: You do not need to manually edit the attributes of a IBM Initiate
MemSeqNoIncrementer component that has been created by the IBM Initiate
Member Model Transform Graph; the attributes are populated automatically based
on values you entered in the IBM Initiate Member Model Transform Graph wizard.
Chapter 3. IBM Initiate custom components
69
There are three required fields in the IBM Initiate MemSeqNoIncrementer
component:
v Not null fields
The Not null fields are fields that make up the attributes. These are essentially
the fields to which you want to assign memseqno values, in the event that one of
the fields is not null.
To select Not null fields, click on the ellipsis to open the Edit key dialog. The
dialog lists available attributes, based on the metadata from the Edge connecting
the Reader to the IBM Initiate MemSeqNoIncrementer component. Select fields
from the Fields column and use the arrow buttons to move them to the Key
parts column.
v MemSeqNo field
The MemSeqNo field is the field to increment as memseqno values are assigned.
To select a MemSeqNo field, click on the ellipsis to open the Edit key dialog.
The dialog lists available attributes, based on the metadata from the Edge
connecting the Reader to the IBM Initiate MemSeqNoIncrementer component.
Select fields from the Fields column and use the arrow buttons to move them to
the Key parts column.
v Field mapping
The Field mapping dialog maps fields from the component's input file to its
output file(s).
To map input fields to output fields, click on the ellipsis to open the Mapping
dialog..
Note: You can not open the Field mapping dialog unless you have defined
metadata for the Edge connecting the IBM Initiate MemSeqNoIncrementer to the
Writer. If no metadata has been applied to this Edge, the ellipsis for opening the
Field mapping dialog does not appear.
The Input Fields column lists attributes based on the metadata from the Edge
connecting the Reader to the IBM Initiate MemSeqNoIncrementer component.
The Field Mapping table's Output fields are fields defined in the metadata for the
Edge connecting the IBM Initiate MemSeqNoIncrementer component to the
Writer on Port 0. Drag fields from the Input Fields column to the Output fields
you want to map them to.
IBM Initiate MemSeqNoIncrementer component Input and Output
ports
The IBM Initiate MemSeqNoIncrementer component has two output ports.
v Port 0 is required. It is used for reformatted records for which the memseqno
was changed. When at least one of the key fields is not null, the value of the
field set as memseqnoField is unique for this attribute, and the record is
reformated according to the formula given in the Mapping attribute and sent to
port 0.
v Port 1 is optional. It is used for records with the same structure as on input, but
with modified memseqno values. It outputs a copy of the input record, but the
field set as memseqnoField can be incremented by one (if at least one of the key
fields is not null). If you wish to concatenate multiple IBM Initiate
MemSeqNoIncrementer components, use Port 1 to route data from one IBM
Initiate MemSeqNoIncrementer component to the next.
70
CloverETL User's Guide
IBM Initiate custom address-standardization components for
CloverETL
Address standardization is the process of formatting and validating address data
according to standards from postal agencies world-wide. Address standardization
also involves delivering correct address data to applications and business processes
that need it, and maintaining standardized address data as it changes over time.
CloverETL provides address standardization functionality that integrates with
several address-verification vendors through the use of Transformer components. If
you have a license for one of these vendors, you can use the applicable CloverETL
component to connect with the vendor while you are working in IBM Initiate
Workbench.
IBM Initiate Workbench includes an IBM Initiate Address Verification Interface
custom component for building graphs for address standardization.
Attribute requirements for address standardization in
CloverETL
Before you can perform address standardization and store the results in the Hub,
you must structure the data model to accommodate the standardized address data.
At a minimum you would define an attribute for the standardized street address,
but the type of standardization and verification that you perform dictates your
modification of the data model. Requirements for the attributes will depend on the
type of address data you will be handling. When you set up a CloverETL graph to
perform standardization, you will map the standardized data returned by the
callout to the attribute(s) you have defined for the standardized address data.
Deciding between batch and transactional standardization in
CloverETL
Standardization supports both batch and transactional scenarios. As you determine
the best standardization approach for your organization, you will need to balance
performance requirements and the currency of the verification platform’s data (that
is, the data used to standardize addresses). Retrieval speed can be faster if
standardized data is stored in the Hub database, but because the postal reference
databases are updated regularly, the retrieval results are likely to be more up to
date if standardization is done “on the fly” as data is requested.
File-to-file batch standardization in CloverETL
File-to-file batch standardization can be accomplished by a simple graph that
includes three components: a Reader to retrieve the source data records, a
Transformer component to standardize and verify the data, and a Writer to create
useful output. For example, you might retrieve records from a list of international
customers in order to create mailing labels or to display in IBM Initiate Inspector.
Transactional processing by using a callout in CloverETL
For transactional address standardization, you might consider registering a callout
that includes the address-standardization graph. With this model, to
standardization process flows as follows:
v a request for address data is made
v the database extracts the requested data
v the Master Data Engine invokes the callout, which contains an
address-standardization graph
Chapter 3. IBM Initiate custom components
71
v the Transformer component sends the data to the vendor
v the vendor returns the standardized, verified address data
v the standardized, verified data is displayed to the requester
Complete the following steps in order to set up this type of transactional
processing:
1. Create and register a callout handler in IBM Initiate Workbench. Refer to the
“Callout handlers” chapter in the IBM Initiate Workbench User's Guide.
2. Create a standardized address Attribute in the data model. Refer to the
“Configuration editor” chapter in the IBM Initiate Workbench User's Guide.
3. Create the address-standardization graph in CloverETL.
The CloverETL IBM Initiate Address Verification Interface
component
IBM Initiate Address Verification Interface is a subscription-based product that
gives you access to postal reference data that is standardized by national postal
services from around the world. The postal reference data is updated on a
quarterly basis, enabling the IBM Initiate Address Verification Interface component
to access the most up-to-date address data for any given country.
Updates are made available the first week of each calendar quarter (January 1st,
April 1st, July 1st, and October 1st) and are only enabled for a period of 4 months.
Important: Because each update to the postal reference data is only enabled for 4
months, you should schedule regular updates to be applied, in order maintain
functionality.
The IBM Initiate Address Verification Interface component is a Transformer
component that you can use together with a Reader component (for example,
Initiate Handler Reader) and a Writer component to perform the
address-standardization callout.
For details about changes between the IBM Initiate Address Verification Interface 4
(offered in versions 8.7 and earlier of IBM Initiate Workbench) and version 5
(which replaces version 4 with the December 2009 patch), see “Changes from IBM
Initiate Address Verification Interface version 4 to IBM Initiate Address Verification
Interface version 5 in CloverETL” on page 73.
About postal reference data
The postal reference data to which IBM Initiate Address Verification Interface gives
you access is provided to you at the time of purchase and is updated regularly.
This data must be stored in a directory that is accessible to the IBM Initiate Master
Data Service server.
Refer to the IBM Initiate Address Verification Interface documentation (provided to
you at the time of purchase) for more details about working with postal reference
data, including information on how to update the data.
The IBM Initiate Address Verification Interface is supported by technology and
postal reference data provided by AddressDoctor. AddressDoctor has compiled the
postal reference database used by the IBM Initiate Address Verification Interface
using data obtained from data suppliers. In collating this data, AddressDoctor has
proceeded diligently and made checks for correctness, completeness and validity.
72
CloverETL User's Guide
Nevertheless, it may be possible that individual data may be incorrect, incomplete
or invalid. Neither AddressDoctor nor IBM provides any warranty in this regard.
The software libraries used by AddressDoctor for the verification of postal
addresses use fault-tolerant methods and fuzzy matching algorithms. They can
lead to wrong corrections or suggestions even if the input data is correct. It may be
possible that an address is corrected incorrectly (false positive) or that an address
that would be correctable is not corrected (false negative).
Changes from IBM Initiate Address Verification Interface version
4 to IBM Initiate Address Verification Interface version 5 in
CloverETL
In release 9.2 of IBM Initiate Workbench, the version of the IBM Initiate Address
Verification Interface component has been upgraded from version 4 to version 5. If
you are upgrading from a previous version of IBM Initiate Workbench, you must
recreate any existing IBM Initiate Address Verification Interface graphs using the
new IBM Initiate Address Verification Interface 5 component in the current version
of IBM Initiate Workbench.
For complete details on new features and changes to existing features, refer to the
IBM Initiate Address Verification Interface proprietary documentation that was
provided to you at time of purchase.
Some key changes to note include the following:
v The addressformat.cfg file, which in version 4 had to be present in the directory
which held your postal database(s), is no longer used.
v You can now specifiy multiple unlock codes for your IBM Initiate Address
Verification Interface component. Refer to the IBM Initiate Address Verification
Interface proprietary documentation for information on how multiple codes
work together.
v IBM Initiate Address Verification Interface component can be configured via a
Clover dialog, or by referencing a configuration XML file that conforms to the
SetConfig.DTD provided in the IBM Initiate Address Verification Interface
proprietary documentation.
v IBM Initiate Address Verification Interface component parameters can be set via
a Clover dialog, or by referencing a parameters xml file that conforms to the
parameters.DTD provided in the IBM Initiate Address Verification Interface
proprietary documentation.
v Input and output mapping has changed; additional options are now available.
Refer to the documentation below, and to the IBM Initiate Address Verification
Interface proprietary documentation, for details.
v The “Preloading” option (for preloading postal databases into memory for
improved performance) is now set with the database (in the Parameters dialog
on the Database tab)
v The Validation parameter in version 4 has been replaced with a Mode option in
version 5. To set the validation mode, use the Mode setting on the Process tab of
the Parameters dialog.
v Refer to the IBM Initiate Address Verification Interface proprietary
documentation for additional details on IBM Initiate Address Verification
Interface 5 features and functions.
Note: Any address standardization graphs you have created using an earlier
version of IBM Initiate Workbench and version 4 of IBM Initiate Address
Chapter 3. IBM Initiate custom components
73
Verification Interface must be recreated using the new IBM Initiate Address
Verification Interface5 component in the updated version.
Country and CASS certification in CloverETL
Some countries offer a certification process for address validation software. In the
United States, the postal service offers the CASS system (Coding Accuracy Support
System), which improves the accuracy of carrier routes, ZIP codes, and delivery
point codes that appear on mail, and is required for sending bulk mail at
discounted rates.
To use CASS certification with IBM Initiate Address Verification Interface,
additional databases that contain CASS related information (such as Carrier Route
codes, EWS, ZIPMOVE, LACSLink, DPV, DFS2, SuiteLink etc.) are required. This
information is contained in files named USA5C1.MD to USA5C21.MD. Please refer
the IBM Initiate Address Verification Interface proprietary documenation for more
detail. (At the time of this writing, CASS information is included in section 3.2.2
Special Remarks for USA CASS Certified Mode in the IBM Initiate Address
Verification Interface documentation.)
For more information about country certification, refer to the postal authority for
the applicable country.
Enabling the IBM Initiate Address Verification Interface
component
The IBM Initiate Address Verification Interface must be enabled before you can use
an address verification component in your graphs. This section explains how.
About this task
When you purchase address database(s), you are provided with an enablement file,
IAVI.txt, which contains an encrypted key that gives you full access to the
address standardization features within IBM Initiate Workbench for those
database(s). This file must be placed in the same directory as your address
database(s), and a path to the database(s) and enablement file must be set in each
IBM Initiate Address Verification Interface component you use in a graph.
Procedure
1. Place a copy of the IAVI.txt file in the same directory as your address
database(s). Note that if you use multiple directories to store various address
database(s), a copy of IAVI.txt must be placed in each one.
2. In each graph that uses an IBM Initiate Address Verification Interface
component, follow these steps to set the path to the directory containing the
database and the IAVI.txt file.
a. Double-click the IBM Initiate Address Verification Interface component to
open the Edit Component dialog
b. Click in the value field for Configuration to open the Configuration dialog.
c. On the DataBase tab, click the green plus sign to add a row for your
database.
d. Modify the CountryISO3, Type, and Preloading type as needed.
e. In the Path field, enter the path to the directory where the database and
IAVI.txt files are located. When entering paths, note the following:
v Paths can be direct (for example, C:\Mydata\) or parameterized (for
example, ${my_db_dir})
74
CloverETL User's Guide
v UNIX paths must use forward slashes (/); Windows paths can use
forward slashes (/) or use backslashes (\)
v Network paths (UNIX or Windows) must use forward slashes (/)
f. Optional. If you want to verify that your component is enabled, click the
UnlockCode tab. This tab lists the validation status for any databases
referenced on the DataBase tab ("valid" for correctly enabled, or "invalid" if
the IAVI.txt file is missing or not functional).
g. Click OK to save your changes and return to the Configuration dialog.
h. Click Apply to apply your change.
Using a manual unlock code to enable the IBM Initiate Address Verification
Interface:
In some cases, such as demonstrations or debugging scenarios, you may need to
manually enter an unlock code, as an alternative to using the IAVI.txt enablement
file. This section explains how.
About this task
When you manually enter an unlock code for the IBM Initiate Address Verification
Interface, you must enter the code for every IBM Initiate Address Verification
Interface component in your graphs.
Procedure
1. In the graph that contains the IBM Initiate Address Verification Interface
component, double-click the IBM Initiate Address Verification Interface
component to open the Edit Component dialog
2. Click in the value field for Configuration to open the Configuration dialog.
3. On the UnlockCode tab of the configuration dialog, click the green plus sign to
add a row for your unlock code.
4. Type or copy your unlock code over the “Enter an unlock code” text in the new
row. You can enter multiple unlock codes in this manner. Refer to the IBM
Initiate Address Verification Interface proprietary documentation for
information about how multiple unlock codes work together.
5. Click OK to return to the Configuration dialog.
6. Click Apply to apply your change.
Expiration of the unlock code:
The IBM Initiate Address Verification Interface unlock code expires on a periodic
basis. Although the unlock code does not expire as frequently as the postal
reference data, it is good practice to update your unlock code each time you
update your postal reference data, to ensure continued operation.
About this task
To update your unlock code, you must obtain an updated IAVI.txt file from IBM.
Updates to this file are provided by IBM along with postal reference data. It is
good practice to update your IAVI.txt file any time you update your postal
reference data.
Failure to install this new key will result in your address standardization process
discontinuing. The follow error will be reported in your log files, but will not be
displayed on the user interface:
Chapter 3. IBM Initiate custom components
75
ERROR com.addressdoctor.AD_Exception: [-100] (CRITICAL) Expired
Creating a graph with an IBM Initiate Address Verification
Interface component
About this task
The basic steps required for creating an address standardization graph with an
IBM Initiate Address Verification Interface component are described below. For
detailed information about working with Transformers, see “Working with
CloverETL graphs” on page 6.
In an address standardization graph, data to be standardized can be read from any
data Reader, or from Initiate dictionary data if used in a callout. For callouts, the
graph will use a special Reader component, the Initiate Handler Reader, which is
specially designed to read data from the dictionary area. Likewise, a custom Writer
component, the Initiate Handler Writer is used to return or write standardized
data.
Note: An address standardization graph requires that both input and output data
are mapped appropriately. This mapping cannot be completed until the Reader,
Transformer (IBM Initiate Address Verification Interface), and Writer components
are placed in the graph and connected with Edges containing metadata. For this
reason, Initiate recommends that you add and connect all the elements of the
graph before beginning to configure each element.
Procedure
1. Create and open a new graph using the steps outlined in “Building a
CloverETL graph” on page 7.
2. From the Readers section of the Palette, select and drag an Initiate Handler
Reader component onto the graph, and connect it to your external data
source, using the steps outlined in “Adding and configuring a CloverETL
Reader” on page 8.
3. Add the IBM Initiate Address Verification Interface (AddressDoctor)
component, and connect it to the Initiate Handler Reader with an Edge:
a. From the Transformers section of the Palette, select and drag an IBM
Initiate Address Verification Interface (AddressDoctor) component onto the
graph.
b. Add an Edge to connect the Initiate Handler Reader to the IBM Initiate
Address Verification Interface (AddressDoctor) component.
c. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
The metadata should include both the attributes you want to standardize,
and any attributes you will “pass through” to the requesting application.
4. Add the Initiate Handler Writer component, and connect it to the IBM Initiate
Address Verification Interface (AddressDoctor) component with an Edge:
a. From the Writers section of the Palette, select and drag an Initiate Handler
Writer component onto the graph.
b. Add an Edge to connect the Initiate Handler Writer to the IBM Initiate
Address Verification Interface(AddressDoctor) component.
c. Create and apply metadata to the Edge, using the steps outlined in “Using
CloverETL Edges to connect components” on page 10.
76
CloverETL User's Guide
The metadata should include attributes for both your source data and the
standardized version of it, as well as any other attributes you will “pass
through” to the requesting application.
5. Map the Initiate Handler Reader attributes that will be passed to the IBM
Initiate Address Verification Interface component.
Note: Remember that the metadata for the Edge connecting the Initiate
Handler Reader to the IBM Initiate Address Verification Interface component
must be defined and applied before mapping can be done.
a. Double-click the Initiate Handler Reader to open the Edit Component
dialog.
b. Click the ellipsis in the Value field for the Mapping property to open the
Attributes and Non-attribute segments dialog.
c. Select the attributes and non-attribute segments to map.
On the Attributes tab, HOMEADDR and LEGALNAME are typically
mapped. Remember that you need to map every attribute you want to
pass, even if not all these attributes represent data to be standardized. For
example, if customer's name is included in the data that will be passed on
to the requesting application, you should include it here even though it
will not be standardized
On the Non-attribute segments tab, MEMHEAD is typically mapped,
because the memrecno is needed to identify the record.
d. Map records from the selected attributes to the relevant output fields.
you can enable CloverETL to
e. By clicking the Auto mapping button
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
f. Click Finish.
6. Double-click the IBM Initiate Address Verification Interface(AddressDoctor)
component to open the Edit Component dialog.
7. Click Configuration to enter the following required configuration properties in
the IBM Initiate Address Verification Interface component:
a. On the Unlock tab, enter your unlock code(s). Unlock codes are provided
at the time of purchase.
b. On the Database tab, enter information about your postal reference
databases. The postal reference data was provided to you at the time of
purchase.
1) Choose a Country ISO3 for the database.
2) Choose the Type for the database. The type must match the filetype of
the database; for example, choose FASTCOMPLETION as the type for a
database with a filetype of FastCompletion.
3) Enter the Path to the postal reference database.
4) Choose a Preloading mode.
v PARTIAL loads the metadata and indexing structures of the selected
databases.
v FULL loads the complete selected databases into memory.
v NONE disables loading of the CASS certification database, and
should not be selected for use with the database of any other
country.
5) Click OK to exit from the Configuration dialog.
Chapter 3. IBM Initiate custom components
77
8. Click Parameters and enter any additional parameters for your graph. Refer to
the IBM Initiate Address Verification Interface proprietary documentation for
detailed information about parameter options. Press OK when finished.
Note: The input and result encoding values are set during run-time based on
the operating system.
9. Click Input Mapping to map the IBM Initiate Address Verification Interface
input address properties to your input metadata fields. The AddressDoctor
address properties are the fields in the IBM Initiate Address Verification
Interface postal reference data that map to the Initiate attributes you want to
standardize.
The following table summarizes some IBM Initiate Address Verification
Interface address properties and the common address elements to which you
would map them.
Table 13. Mapping IBM Initiate Address Verification Interface (AddressDoctor) properties to
address elements
AddressDoctor
address property
Example user-friendly address input field
Country
Country name
Province
County (UK), State (United States), Province (Canada), Kanton
(Switzerland), Prefecture (Japan)
Postal code
Division within a city (post code, ZIP code)
Locality
City name, municipality name
Street
Street name (1000 Harvard DR)
House number
Building address number (1000 Harvard DR)
Building
Recognizable building name (Emirates Palace Hotel, Sears Tower)
Subbuilding
Division within a building (Apartment 3B, Seventh floor, Suite 120)
The Subbuilding property is contained within the Building
property.
v From the Properties list, select a Building property.
v In the Mapping dialog box, select the Building property. The
Property Configuration: Building type drop-down list box is
displayed.
v From the drop-down list, select the Subbuilding property.
v Drag the applicable address input field to the Building property.
a. On the Address properties dialog, use Ctrl-click to choose the properties
you want to be available for mapping. Click Next.
b. Map the input fields to the selected IBM Initiate Address Verification
Interface address properties.
c. Click Finish, then click OK to close the IBM Initiate Address Verification
Interface component.
10. Click Output Mapping to map the IBM Initiate Address Verification Interface
output address properties to your output metadata fields. In this step, you are
mapping standardized data from the IBM Initiate Address Verification
Interface postal reference data to fields in your output metadata, that will be
passed to the requesting application.
a. On the Address properties dialog, use Ctrl-click to select address
properties for mapping, and click Next.
78
CloverETL User's Guide
11.
12.
13.
14.
b. On the Address output mapping dialog, map the IBM Initiate Address
Verification Interface address properties to the output metadata.
c. Click Finish.
Double-click the Initiate Handler Writer component to open the Edit
component dialog.
Specify an Initiate connection for the Initiate Handler Writer.
Map the output attributes:
a. Click the ellipsis in the Value field of the Mapping property to open the
Handler Writer mapping dialog.
b. Select the attributes to map, and click Next.
c. Map your selected attributes to the appropriate Input fields.
d. By clicking the Auto mapping button, you can enable CloverETL to
perform the mapping by making a “best guess” based on the attribute
codes and attribute names.
e. Click Finish.
If you want to select key fields to designate master keys in your data, use the
Key fields property to select your master key(s).
15. Save the graph.
Configuring the CloverETL standardization and validation
graph
About this task
The address standardization process takes an existing address record from an
external file or from the IBM Initiate Hub, verifies it against a standardization or
verification platform, and creates output with a standardized version of the
address. The output can reside in temporary memory or as a record to be written
to a file or database, depending on the design of the graph. The standardized
output does not replace the original record, but supplements it with the
standardized data.
To illustrate, here is some sample, nonstandard input:
Example Company Inc | 200 W. Madison | Suite 2300 | Chicago | Ill | US
Example Company Inc | 5001 Plaza on the Lake | Suite 111 | Austin | Texas | USA
Example Company Inc | 2550 Union Hills Drive | Suite 201 | Phoenix | Ariz | USA
In the example, postal codes are missing and the names of streets, state, and
country are inconsistent.
To standardize and validate the input data, set up a graph with the following
settings:
Input to IBM Initiate Address Verification Interface component:
v “delivery address line 1” = street number/name
v “delivery address line 2 = suite number
Output from IBM Initiate Address Verification Interface component:
v “delivery address line 2” = street number/name
The following data results from executing the graph:
Example Company Inc|200 W Madison St Ste 2300|Chicago|IL|60606|United
States|36536022|1|04434022
Chapter 3. IBM Initiate custom components
79
Example Company Inc|5001 Plaza On the Lk Ste 111|Austin|TX|78746|United
States|36536022|1|04434022
Example Company Inc|2550 W Union Hills Dr Ste 201|Phoenix|AZ|85027|United
States|36536022|1|04434022
In this example output, the addresses are in a standard and consistent format, the
correct postal codes have been added, and there are three additional fields of data.
Let's look at how the data and graph are configured to produce this result.
Fielding source data for best results in the CloverETL
standardization and validation graph
How source data is structured and associated with IBM Initiate Address
Verification Interface fields can affect the reliability of validation.
v Fully fielded data is, in general, broken down into individual columns, each of
which contains only one address element that is assigned to one IBM Initiate
Address Verification Interface field. This structure yields the most accurate and
reliable results. In our example, the data is considered fully fielded.
Note: In fully fielded data, HouseNumber and Street values can reside in the same
column and be assigned to the Street field.
v With partially fielded data, some address elements are grouped into the same
column and assigned to the more generic ‘Address Line' fields. The grouping of
similar but different data makes validation less reliable.
v Unfielded data has no explicit structure and is usually not validated. The
Formatted Address fields are used with unfielded data, which usually is
formatted according to line feeds or other delimiters, to produce usable output.
More detailed information about data structure and IBM Initiate Address
Verification Interface fields is available in the IBM Initiate Address Verification
Interface documentation.
Edge metadata for the CloverETL standardization and validation
graph
On the IBM Initiate Address Verification Interface input Edge, we are including all
the columns of the source file.
However, on the output Edge, we are passing additional metadata to complete the
addresses and display information that is helpful for understanding how the data
was standardized and validated.
80
CloverETL User's Guide
We have added the following items to the output metadata.
v ZIP: to include the missing postal codes.
v ElementResultStatus: an indicator of whether and how each address element was
modified. Refer to the IBM Initiate Address Verification Interface proprietary
documentation for the significance of ElementResultStatus codes.
Mapping the CloverETL IBM Initiate Address Verification Interface
component
This section discusses mapping for the IBM Initiate Address Verification Interface
component.
About IBM Initiate Address Verification Interface address properties
Mapping involves an understanding of the IBM Initiate Address Verification
Interface properties. There are three logical groups:
v Address element: Examples are Organization, Street, Province (State),
PostalCode, and Locality (City). These properties set and retrieve values for
individual address elements.
v FormattedAddress: This set of properties presents alternatives to the address
element properties. You can use them to map unstructured (unfielded) data that
cannot be used with the more structured address element properties.
v Control and status: These properties, for example the DefaultCountry property,
can determine how IBM Initiate Address Verification Interface works. They also
can be used to set and retrieve information about operations; the
ElementResultStatus is an example.
For more detailed information about IBM Initiate Address Verification Interface
properties, refer to the IBM Initiate Address Verification Interface documentation.
Input mapping in the example
Our sample exercise included the following input mapping.
Chapter 3. IBM Initiate custom components
81
These input fields are the same as the input Edge metadata. They are mapped to
the corresponding IBM Initiate Address Verification Interface address properties.
For input mapping, the Delivery address line 1 property accesses the Street1 data
field; Delivery address line 2 accesses the Street2 data field, which, in our example,
contains the Suite number.
Output mapping in the example
The following configuration is mapped for the output data.
In the output mapping, the fields for output data are the same as the output Edge
metadata. They are mapped to the corresponding IBM Initiate Address Verification
Interface properties.
82
CloverETL User's Guide
For output mapping, the Delivery address line 1 property is mapped to only the
Street1 input field. However, because DeliveryAddress properties can use partially
fielded data, the Suite number (which is mapped to Street2 in the input mapping)
will be included in the output on the same line as the Street1 data, as shown in the
output data:
200 W Madison St Ste 2300
Before you can store the standardized address data in the Hub, you must create
attributes for the standardized address elements, for example, StandardName,
StandardStreet, and StandardCity.
Graph design tips for batch and transactional address
processing in CloverETL
The Transformer components for each vendor are different; however, these
suggestions might improve your results and performance.
v If your source file includes a header row, be sure to set the Skip first line value
to True if that option is available in the Reader component.
v Remember to map every attribute that you want to pass through to the Writer,
including those that will not be standardized and verified.
v If the Transformer component enables parsing of the input data, avoid accepting
this option; it can cause a conflict with the postal reference databases.
v Preloading at least part of the postal reference databases results in faster
performance. To avoid errors, ensure that you have sufficient memory to enable
loading of large databases.
v Ensure that the output metadata includes attributes for all the source data and
standardized data, as well as attributes that are being passed without
standardization.
v If you edit a callout graph after running it, you must save, close, and reopen the
graph in order for IBM Initiate Workbench to effect the change. Closing and
reopening is not required for edited graphs that are not included in callouts.
IBM Initiate Address Verification Interface transliteration in
CloverETL
Transliteration is the process of transcribing words or text from one writing system
or character set to another (for example, transcribing Japanese characters to a Latin
alphabet), usually based on phonetic equivalences. Transliteration refers only to
changes in the writing system, and does not include translation.
In the Initiate Hub, transliteration is performed via a CloverETL graph, using the
IBM Initiate Address Verification Interface component.
Creating a transliteration graph
About this task
Any address can be transliterated to the Latin script. The basic steps required for
creating a transliteration graph are similar to the steps for creating an address
standardization graph. The key difference is that for transliteration, you must set
certain properties correctly to format your output:
v Set the following parameters on the Parameters dialog's Process tab:
– Mode: Set the mode to PARSE to transliterate without processing the address.
– Optimization level: In general for transliteration, an optimization level
STANDARD will produce the best results.
v Set the following parameters on the Parameters dialog's Result tab:
Chapter 3. IBM Initiate custom components
83
– Preferred language: Set to ENGLISH or your preferred language.
– Preferred script: Set to LATIN or your preferred script
For detailed information about creating a basic IBM Initiate Address Verification
Interface graph, see “Creating a graph with an IBM Initiate Address Verification
Interface component” on page 76.
84
CloverETL User's Guide
Legal Statement
Licensed Materials – Property of IBM
© Copyright IBM Corporation, 1995, 2011. US Government Users Restricted Rights
- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp. IBM, the IBM logo, Initiate, and Initiate Master Data Service are
trademarks of IBM Corp., registered in many jurisdictions worldwide. Java and all
Java-based trademarks and logos are trademarks or registered trademarks of
Oracle and/or its affiliates. Other product and service names might be trademarks
of IBM, or other companies. This Program is licensed under the terms of the
license agreement accompanying the Program. This license agreement may be
either located in a Program directory folder or library identified as "License" or
"Non-IBM License", if applicable, or provided as a printed license agreement.
Please read this agreement carefully before using the Program. By using the
Program, you agree to these terms.
© Copyright IBM Corp. 1995, 2011
85
86
CloverETL User's Guide
Notices and trademarks
This information was developed for products and services offered in the U.S.A.
Notices
IBM may not offer the products, services, or features discussed in this document in
other countries. Consult your local IBM representative for information on the
products and services currently available in your area. Any reference to an IBM
product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product,
program, or service that does not infringe any IBM intellectual property right may
be used instead. However, it is the user's responsibility to evaluate and verify the
operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not grant you
any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785 U.S.A.
For license inquiries regarding double-byte character set (DBCS) information,
contact the IBM Intellectual Property Department in your country or send
inquiries, in writing, to:
Intellectual Property Licensing
Legal and Intellectual Property Law
IBM Japan Ltd.
1623-14, Shimotsuruma, Yamato-shi
Kanagawa 242-8502 Japan
The following paragraph does not apply to the United Kingdom or any other
country where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or
implied warranties in certain transactions, therefore, this statement may not apply
to you.
This information could include technical inaccuracies or typographical errors.
Changes are periodically made to the information herein; these changes will be
incorporated in new editions of the publication. IBM may make improvements
and/or changes in the product(s) and/or the program(s) described in this
publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for
convenience only and do not in any manner serve as an endorsement of those Web
© Copyright IBM Corp. 1995, 2011
87
sites. The materials at those Web sites are not part of the materials for this IBM
product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it
believes appropriate without incurring any obligation to you.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact:
IBM Corporation
J46A/G4
555 Bailey Avenue
San Jose, CA 95141-1003 U.S.A.
Such information may be available, subject to appropriate terms and conditions,
including in some cases, payment of a fee.
The licensed program described in this document and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement or any equivalent agreement
between us.
Any performance data contained herein was determined in a controlled
environment. Therefore, the results obtained in other operating environments may
vary significantly. Some measurements may have been made on development-level
systems and there is no guarantee that these measurements will be the same on
generally available systems. Furthermore, some measurements may have been
estimated through extrapolation. Actual results may vary. Users of this document
should verify the applicable data for their specific environment.
Information concerning non-IBM products was obtained from the suppliers of
those products, their published announcements or other publicly available sources.
IBM has not tested those products and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products.
All statements regarding IBM's future direction or intent are subject to change or
withdrawal without notice, and represent goals and objectives only.
This information is for planning purposes only. The information herein is subject to
change before the products described become available.
This information contains examples of data and reports used in daily business
operations. To illustrate them as completely as possible, the examples include the
names of individuals, companies, brands, and products. All of these names are
fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which
illustrate programming techniques on various operating platforms. You may copy,
modify, and distribute these sample programs in any form without payment to
88
CloverETL User's Guide
IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating
platform for which the sample programs are written. These examples have not
been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or
imply reliability, serviceability, or function of these programs. The sample
programs are provided "AS IS", without warranty of any kind. IBM shall not be
liable for any damages arising out of your use of the sample programs.
Each copy or any portion of these sample programs or any derivative work, must
include a copyright notice as follows:
© (your company name) (year). Portions of this code are derived from IBM Corp.
Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rights
reserved.
If you are viewing this information softcopy, the photographs and color
illustrations may not appear.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of
International Business Machines Corp., registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the web at "Copyright and
trademark information" at www.ibm.com/legal/copytrade.shtml.
The following terms are trademarks or registered trademarks of other companies:
Adobe is a registered trademark of Adobe Systems Incorporated in the United
States, and/or other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other
countries, or both.
Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in
the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Oracle and/or its affiliates.
Notices and trademarks
89
90
CloverETL User's Guide
Index
A
E
adding metadata for sample graphs 21
address certification 74
address standardization 71
about 71
attributes 71
creating a graph 76
enabling components 74
postal reference data 72
Address Verification Interface
certification 74
creating an address standardization
graph 76
ElementResultStatus values 81
enabling components 74
postal reference data 72
see also address standardization 71
types of address properties 81
applying metadata 11
applying metadata to Edges 11
Attribute Reformat
CloverETL component 67
attributes
address standardization 71
automatic graph execution 15
Edges
about 6
applying metadata 11
debugging 12
propagating metadata 12
editing the database Reader
components 28
editing the parameters 29
editing the query 29
executing a graph 15
expression tester 10
extract files
best practices 6
extract household graph 28
B
best practices for extract files
6
C
callout handler registration 3
CASS certification 74
CloverETL, what is v
Coding Accuracy Support System
(CASS) 74
command line processing 15
configuring the household_extract
graph 28
configuring the IBM Initiate Member
Model Transform graph 27
connections
database 13
JMS 14
creating input metadata 25
customer support
contacting 93
D
database connection 13
database Reader components
deprecated components
about 6
28
© Copyright IBM Corp. 1995, 2011
F
features and benefits
J
JMS connection
Joiners
about 5
14
K
1
key fields
G
graphs
address standardization 71
extract household graph 28
household_extract graph 28
IBM Initiate Member Model
Transform graph 27
IBM Initiate Member ModelTransform
graph 24
running 15
samples installed with Workbench 1
transliteration 83
troubleshooting 17
H
household extraction graph
customization 28
householding project 28
IBM Initiate MEMSEARCH component
for CloverETL 36
IBM Initiate TSKGET component 42
IBM Initiate TSKSEARCH component 48
IBM Initiate TSKSEARCH component for
CloverETL 46
importing IBM Initiate Hub metadata 26
Initiate Demo Householding project 28
Initiate MEMPUT component 52
Initiate TSKPUT component 61
28
I
IBM Initiate Address Verification Interface
creating a transliteration graph 83
IBM Initiate Attribute Reformat 67
IBM Initiate Handler Writer
component 56
IBM Initiate Hub metadata 26
IBM Initiate Member Model Transform
graph 24, 27
IBM Initiate Member Sequence Number
Incrementer component 69
IBM Initiate MEMDELETE
component 54
IBM Initiate MEMSCORE 64, 66
IBM Initiate MEMSEARCH
component 38
31
L
launching graphs
legal notices 87
16
M
madconfig 16
launching graphs 16
scheduling jobs 16
madconfig launch_etl utility 15
memDelete
CloverETL component 54
memGet
CloverETL component 34
CloverETL component metadata 33
memPut
CloverETL component 52
memScore
CloverETL component 64, 66
memSearch
CloverETL component 36, 38
metadata 11
adding for the sample graph 21
applying 11
creating input metadata 25
exporting 25
file 25
IBM Initiate MEMGET component 33
IBM Initiate TSKGET component 42
importing IBM Initiate Hub 26
propagating 12
O
other components
about 6
output directories 26
91
P
W
postal reference data 72
propagating metadata 12
Writers
about 5
IBM Initiate Handler Writer 56, 57
IBM Initiate MEMDELETE 54
Initiate Handler Writer 76, 83
sample graph 24
R
Readers
about 5
IBM Initiate Handler Reader 41
IBM Initiate MEMGET 34
IBM Initiate MEMSEARCH
component 36, 38
IBM Initiate TSKGET 42, 44
IBM Initiate TSKSEARCH
component 46, 48
Initiate Handler Reader 76, 83
running a graph 15
S
sample graphs 24, 28
scheduled jobs 16
software services
contacting 93
source data files 26
support
customer 93
switching to the Clover perspective
2
T
Technical Support Contact vi
trademarks
list of 87
Transformers
about 5
Address Verification Interface 76
Dedup 23
Ext Filter 10, 22
Ext Sort 23
IBM Initiate Address Verification
Interface Transliteration 83
IBM Initiate Attribute Reformat 67
IBM Initiate Member Sequence
Number Incrementer 69
IBM Initiate MEMSCORE 66
Initiate IBM MEMSCORE 64
Initiate MEMPUT 52
Initiate TSKPUT 61
MemSeqNoIncrementer 69
using regular expressions 10
transliteration
creating a graph 83
troubleshooting graphs 17
TSKGET
CloverETL component 42, 44
CloverETL component metadata 42
TskPut
CloverETL component 61
tskSearch
CloverETL component 46, 48
U
using madconfig utility
92
15
CloverETL User's Guide
Contacting IBM
You can contact IBM for customer support, software services, product information,
and general information. You also can provide feedback to IBM about products
and documentation.
The following table lists resources for customer support, software services, training,
and product and solutions information.
Table 14. IBM resources
Resource
Description and location
IBM Support Portal
You can customize support information by
choosing the products and the topics that
interest you at www.ibm.com/support/
entry/portal/Overview/Software/
Information_Management/IBM
Initiate_Master_Data_Service
Software services
You can find information about software, IT,
and business consulting services, on the
solutions site at www.ibm.com/
businesssolutions/
My IBM
You can manage links to IBM web sites and
information that meet your specific technical
support needs by creating an account on the
My IBM site at www.ibm.com/account/
Training and certification
You can learn about technical training and
education services designed for individuals,
companies, and public organizations to
acquire, maintain, and optimize their IT
skills at http://www.ibm.com/software/swtraining/
IBM representatives
You can contact an IBM representative to
learn about solutions at
www.ibm.com/connect/ibm/us/en/
Providing feedback
The following table describes how to provide feedback to IBM about products and
product documentation.
Table 15. Providing feedback to IBM
Type of feedback
Action
Product feedback
You can provide general product feedback
through the Consumability Survey at
www.ibm.com/software/data/info/
consumability-survey
© Copyright IBM Corp. 1995, 2011
93
Table 15. Providing feedback to IBM (continued)
Type of feedback
Action
Documentation feedback
To comment on the information center, click
the Feedback link on the top right side of
any topic in the information center. You can
also send comments about PDF file books,
the information center, or any other
documentation in the following ways:
v Online reader comment form:
www.ibm.com/software/data/rcf/
v E-mail: [email protected]
94
CloverETL User's Guide
Printed in USA
SC19-3165-02