Basic
CA-ESP Workload Automation
Course – Agent Overview
What You Will Learn
- What is an ESP System Agent
- Basic Agent Architecture
- How ESP System Agents Interact With CA ESP
Workload Automation Engine
- Basic Configuration Options
- Defining non-mainframe workload
- Value that ESP Agents can provide
- ESP Agent for Databases
2
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Product Components
3
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP Agents
- Lean, non-invasive conduits (Approx 50MB disk space)
- Extends CA ESP Workload Automation’s solutions across a
variety of operating systems and ERP environments
- Unlimited scalability, throughput
- Each platform has unique ESP Agent
- Manage through a single point of control, will be integrated
with CA Job Management products (CA-7, CA-Autosys)
4
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP System Agents - Scalability
- Volumes and Concurrency
- Large number of concurrent processes
- Lab tested at 1,000 concurrent jobs, largest known field
implementation 500 concurrent jobs
- Highly scalable
- Can keep up with volumes manager can handle
5
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
What is an ESP Agent?
- Processes work
- Notifies Manager of job status
- Started
- Running
- Complete/Failed
- Stores output from jobs in a spool file
- Allows users to control workload
- Waits for work
- Persistence
6
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
7
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Workload Objects
Distributed OS Integration
File
Management
ERP Integration
Agent Monitoring
Web App Integration
OS Resource Integration
Database Integration
Mainframe Integration
8
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Who Controls the ESP Agent?
- Schedulers
- Schedule jobs regardless of platform & Have
control over entire enterprise
- Operations
- Monitor entire enterprise & Control every
production job
- System Administrators
- Install ESP Agent & Maintain ESP Agent
9
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP System Agent Architecture
- ESP System Agents for UNIX Based Platforms are started
processes.
- ESP System Agents for Microsoft Windows are installed as
Windows Services
- All ESP System Agents Utilize SUN Microsystems JAVA
Runtime Environment as the Base Architectural Component
- All ESP System Agent Functions are Created as JAVA PlugIn’s
10
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP System Agent Architecture
Spawns &
Tracks
Submitted
Commands
& Scripts
Processes &
Monitors File
Trigger &
FTP
Requests
Processes
Machine
Resource
Monitoring
Requests
Processes
Processes
TCP/IP
EJB & JMS
Communications
Publish
& SNMP
Subscribe
Requests
Requests
Job Execution
JAVA Plug-In
File
Monitoring
JAVA Plug-In
Object
Monitoring
JAVA Plug-In
J2EE
Execution
JAVA Plug-IN
Agent
Management
JAVA Plug-In
SUN Java Runtime Environment Base Code
11
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Communication
- ESP Managers and ESP System Agents
communicate asynchronously using message
queues.
- ESP Managers and Agents communicate by
sending Automated Framework Messages or AFMs.
- Most Agent commands deal with the control of
these AFMs.
12
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Communication
- ESP Managers and ESP System Agents have
Sender and Receiver Ports.
- The receiver listens on a predefined TCP/IP port.
- When the sender has messages to transmit, it connects
to the receiver’s port, sends the messages, and then
closes the connection.
13
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Communication
- ESP Managers and Agents have sender and
receiver ports.
- Each Agent has one dynamic sender port and
one receiver port.
- The ESP Host can have multiple receiver ports (for
example, to separate encrypted and non-encrypted
message traffic) and has one dynamic sender port for
each connected Agent.
14
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP Agent Basic Communications
15
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Communication
- Automated Framework Messages include:
- ESP System Agent Name
- Command or Script to Execute
- UserId to Execute the Workload Under
- Arguments for the Script or Command
- Environment Variables
- Job Execution Status
16
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Communications
- Sample AFM Messages:
20050908 12060405+0500 CM_DE54 DLEIGH_WIN_AGENT
WINNT1/WLMDEMO5.20/MAIN State EXEC SetStart Status(Executing at
DLEIGH_WIN_AGENT) Jobno(1860) User(DLEIGH_WIN_AGENT) Host(DLeigh)
20050908 12063416+0500 CM_DE54 DLEIGH_WIN_AGENT
WINNT1/WLMDEMO5.20/MAIN State COMPLETE Cmpc(0) SetEnd
User(DLEIGH_WIN_AGENT) Host(DLeigh)
20050908 12290376+0500 CM_DE54 DLEIGH_WIN_AGENT
CYBDL01K/WLMDEMO3.3/MAIN State SUBERROR Failed SetEnd
Status(Command file not found) Cmpc(20007) User(DLEIGH_WIN_AGENT)
Host(DLeigh)
17
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- All ESP Agent Configuration Settings are stored in
a single file called agentparm.txt
- This file is created by the installation process and
can be manipulated using any text editor
- This Configuration File Defines the Name of the
Agent, Ports Used, and Other Configurations that
Pertain to Other Functions (FTP, J2EE, SNMP, etc.)
18
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- Sample agentparm.txt File:
# ESP System Agent for Microsoft Windows parameters
#
# Log
#
log.level=5
log.maxsize=1024000
#
# Agent name
#
agentname=DLEIGH_WIN_AGENT
#
# Communications
#
communication.managerid_1=CM_DE54
communication.manageraddress_1=lparc
communication.managerport_1=6666
communication.monitorobject_1=DLEIGH_WIN_AGENT/AGENTMON1.0/MAIN
communication.managerid_2=ESPRESSO_CALYPSO_47500
communication.manageraddress_2=calypso
communication.managerport_2=47507
communication.monitorobject_2=DLEIGH_WIN_AGENT/AGENTMON2.0/MAIN
19
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- Sample agentparm.txt File:
communication.managerid_3=ESPRESSO_ELVIS_7500
communication.manageraddress_3=elvis
communication.managerport_3=7507
communication.monitorobject_3=DLEIGH_WIN_AGENT/AGENTMON3.0/MAIN
communication.inputport=9900
communication.prefixlevel=2
#
# Security
#
security.filename=D:/Cybermation/ESP System Agent R6.1/security.txt
security.keystorage=D:/Cybermation/ESP System Agent R6.1/keys.txt
security.cryptkey=0x3132333435363738
security.level=off
#
# Initiators
# There will be separate line for each pair of <class, number of initiators>
# initiators.class_N, where N is number running from 1 to ...
# Class is user definable, (should be the same here and in AFM)
# Soft shutdown mode waits for all jobs to be completed
#
initiators.class_1=Default,1000
20
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- Sample agentparm.txt File:
21
# Persistence
#
persistence.gcinterval=10000
persistence.level=2
#
# Core parameters (for plugins)
#
core.address=localhost
core.port=35800
#
# General characteristics
#
spooldir=./spool
COLD_START=false
#
# Runner plugin parameters
#
runnerplugin.managerport=35801
runnerplugin.requestport=35802
#
# Start JVMs
#
oscomponent.attachjvm=true
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- Sample agentparm.txt File:
#Path to the JRE
oscomponent.javapath=D:/Cybermation/ESP System Agent R6.1/jre/bin
#type of jvm (used only if attachjvm=true)
oscomponent.jvm=client
#What plugins we want to start by the core Java agent
plugins.start_internal_1=runner
plugins.start_internal_2=fileMon
plugins.start_internal_3=objmon
plugins.start_internal_4=management
plugins.start_internal_5=ftp
plugins.start_internal_6=j2ee
plugins.start_internal_7=router
#
# SNMP specific
#
management.snmp.mibfile=D:/Cybermation/ESP System Agent R6.1/cybermib.txt
management.snmp.host=DLeigh
management.snmp.port=162
management.snmp.community=public
22
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Basic Configuration
- Sample agentparm.txt File:
#
# FTP specific
#
# ftp.noserver=false
ftp.serverport=21
#ftp.client.ssl=true
#ftp.client.ssl.truststore=
#ftp.client.ssl.truststore.password=055A55EB863D2A5D
#ftp.server.ssl=true
#ftp.server.ssl.keystore=
#ftp.server.ssl.keystore.password=C8B98BFA6652520BC0
#
# Windows Service name
#
oscomponent.servicename=ESP System Agent for Microsoft Windows R6.1
oscomponent.servicedisplayname=ESP System Agent for Microsoft Windows R6.1
23
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Successful Communication depends
on…
Agent Information that must match…
Agentparm.txt
24
Topology in ESP dSeries
agentname
Name
communication.inputport
Agent Port number
communication.managerid
Manager Instance Host name
communication.manageraddress
DNS name or IP Address of dSeries Server
coomunication.managerport
ESP Server Manager Port
security.cryptkey
Encryption key used from server to agent
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Test Your Memory
- What Type of Workload Can an ESP System Agent
Execute?
Windows Scripts, Windows Commands, FTP Processes,
Machine Resource Monitors, J2EE Based Processes,
etc…
- How Does the ESP System Agent Communicate
with an ESP Server?
Utilizing Automated Framework Messages over TCP/IP
25
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Test Your Memory
- What is the Receiver Port Used for?
The receiver listens on a predefined TCP/IP port. When the
sender has messages to transmit, it connects to the
receiver’s port, sends the messages, and then closes the
connection.
- What Job States Can the ESP System Agent for
Windows Send to an ESP Server?
READY, EXEC, COMPLETE, FAIL, SUBERROR, &
SUBDELAY
26
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Additional Automation
capabilities
Approaches for use in your
environment
- Event-Driven Workload Automation
- “Batch” Environment Monitoring
- Run Book Automation
28
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Event-Driven Automation
- Trigger off new applications based on monitoring capabilities
of the R7 Agent, using ESP Alerts:
- Error message being written to a log file
- A service coming online
- Server’s CPU stuck at 100%
- ESP:dSeries currently implemented as Jobs, also several
monitors implemented as Event Triggers
29
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
“Batch” Environment Monitoring
- The ability to monitor critical environmental dependencies of
an ESP Application to ensure successful completion
- Example: Find out you are going to run out of disk space
hours before the server dies
- Example: If you know you’re going to start running a job
on your SAP server at 2am, and the database server for
that job crashes at 10pm, why wait 4 hours for everyone
to be surprised?
- (and getting called in the middle of the night)
- Simply another step towards better managed SLA’s
30
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Run Book Automation
- Latest fad in IT Operations that many
of you, have been doing for years
- Focus on managing (executing,
controlling, monitoring) IT Operations
tasks in a controlled, workflow-style
manner much like workload
automation
- Example: Shutting down certain
processes on a server, running
maintenance, starting back up the
processes
- You can, naturally, do much of this
with your existing ESP Scheduler and
ESP Agents
31
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
FTP Workload Objects
Automating File Transfers
- To schedule FTP workload, use the FTP_JOB job type.
- You can automate file transfer with an FTP job. The job can
use an existing FTP server or the Agent’s in-built FTP server.
- Note: To use the Agent as an FTP server, you need to
configure the Agent during installation or set the Agent
parameter ftp.noserver to false. The Agent configured as an
FTP server does not support anonymous file transfers
33
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Automating File Transfers
- In this sample, a file called /temp/cyberftp181006.txt is
downloaded from a UNIX machine (rem_unix) then copied to a
local machine, a Windows PC. Note that the two locations include
a complete path statement. After the download is complete, the
job completes in ESP dSeries:
34
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Properties for ‘Monitor’
Workload Objects
Adding Monitoring Capabilities to ESP
- Starting with Release 6 of the ESP System Agent,
new monitoring capabilities were introduced:
- Windows
- Windows Event Log Monitoring
- Windows Services Monitoring
- Windows/UNIX/OS400
-
Disk Space Monitoring
Process Monitoring
TCP/IP Address/Port Monitoring
CPU Monitoring
Text File Monitoring
- J2EE
- JMS Publish/Subscribe to Queues and Topics
36
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Services
- What is it?
- This job type allows you to monitor Windows
Services on a local machine
- Sample Definition:
37
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Services
- Service Name corresponds to
the name of the Service as
identified in the Services
Application
38
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Services
- Status refers to the state you with the Service to be in for the
monitor to complete
- Valid Statuses include: RUNNING, STOPPED,
CONTINUE_PENDING, PAUSE_PENDING, START_PENDING,
STOP_PENDING, EXISTS, NOTEXISTS
- Can be coded with either WAIT or NOW. Now will return a
COMPLETE/FAIL immediately, while WAIT will remain until the
condition is met or the JOB is forced complete
39
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Event Log
- What is it?
- A job to monitor the Event Log of a Local
Windows Server. It can monitor any one of three
types of logs
- Application Log (Programs)
- System Log (System Components, e.g. a Driver)
- Security Log (Security Events like an invalid
login, file access)
40
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Event Log
41
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Event Log
- Sample Job Definition
42
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Windows Event Log
- EVENTLOG values are generally Application, Security, or System
- EVENTTYPE values may be ERROR, WARN, INFO, AUDITS, AUDITF
(AUDITS and AUDITF related to EVENTLOG Security only)
- EVENTSOURCE value is typically generated by software vendor.
Values with spaces requires quotes
- EVENTCATEGORY represents a classification by the
EVENTSOURCE, in this case, ‘Norton AntiVirus’ is catagorizing this
as a Disk event.
43
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Disk Space
- What is it?
- Allows you to find out how much disk space has been used
or is free
- Can be expressed in MB, GB, or as a Percentage
- Can be CONTINUOUS monitored using an ESP Alert
44
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
IP Monitoring
- What is it?
- It allows you to monitor specific IP addresses or IP Address/Port
combinations to validate that network resources are accessible
and listening ports are available
Valid Status include RUNNING and STOPPED and are monitored for
immediate state NOW or monitored until condition is met WAIT
45
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
CPU Monitoring
- What is it?
- Allows you to monitor CPU utilization to determine
success/failure criteria of your job monitor
- Why would I want to do this?
- Provides you with a means of ensuring that the machine has
sufficient CPU available before submitting workload to it
- If a machine is in a continuously busy state, allows you to fire off
notifications
- CPU Monitoring is also tied into Physical Resources, which can be
used for physical load balancing (required HPO for ESP:mSeries)
- How is CPU utilization measured?
- Windows uses CPU as recorded by the system
- UNIX uses Load Average
46
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
CPU Monitoring
- This example completes if the CPU Usage is greater than
80%.
- Can be used to send warnings to System Administrators.
47
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Text File Monitoring
- Allows you to monitor INSIDE a text file for a
specific string of text for a matching value
- Monitoring has a great deal of flexibility with
Regular Expressions
48
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Text File Monitoring
- This example monitors the file by lines for a text
string, but only the first 20 lines
49
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Text File Monitoring
Example using Regular Expressions:
50
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Properties for JMS
Workload Objects
- JMS Monitoring
JMS Subscribe
- What is it?
- Java Messaging Service
- A Java-based standard for connecting to, and
communicating with, message brokers such as
MQ Series, Sonic, WebLogic, or Tibco
- Allows you to monitor a JMS Queue or Topic for a
specific message that matches your criteria
52
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
JMS Subscribe
Receive
ESP
Topic or
Queue
Agent
Acknowledge
- JMSS_JOB job type
- Subscribe to Topics
or Queues
- Filter results
53
JMS Provider
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
JMS Subscribe
54
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
JMS Subscribe
- A few uncommon terms
- INITIAL_CONTEXT
- CONNECTION_FACTORY
- JNDIUSER
55
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP Agent for
Databases
ESP Database Agent
- Provides integration to a variety
of database platforms
- Utilizes the agent plug-in
architecture to adapt to the
System Agent core
- Written entirely in JAVA and the
JDBC API
57
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
ESP Database Agent Features
ESP Database
Agent
TRG
SP
SP
SQL Updates
and Queries
58
Stored
Procedures
ESP Database
Agent
Events
Parameters
ESP Database
Agent
Results
Results
SQL
ESP Database
Agent
-
+
Table Monitor
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
TRG
Data Triggers
Event Driven SQL Statements
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
SQL
59
ESP Agent for
Databases
Results
- Event-driven execution of SQL
commands i.e. Insert, update,
delete, etc.
- Success criteria specified through
regular expressions
- Output stored to file
Invoking Stored Procedures
- Event-driven execution of stored
procedures
- Retrieve output values generated by
the stored procedure
ESP Agent for
Databases
SP
SP
60
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Monitor Database to Trigger an
ESP Event
- Monitor a table for a net change in
size
ESP Agent for
Databases
- i.e. number records added or deleted
which satisfy a user defined condition
- Complete or selected (columns) records
can be defined
- One-time or continuous alerts
- +
61
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Database Triggers
- Database generates the events as they
occur
- Excellent performance, with very low
overhead
- Generate an event for every record added,
deleted, or modified which satisfies a user
defined criteria
- One-time or continuous alerts
ESP Agent for
Databases
Events
- Very granular notification of changes to a
database table
TRG
TRG
62
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Database Agent Technical
Details
- Supported databases
- Oracle 9i & up
- Microsoft SQL Server 2000 & up
- IBM DB2
63
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Database Agent Examples
- This example uses SQL to Insert a line into a Table
called ‘stores’ in the ‘pubs’ database
64
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
Questions?
65
Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.