Zen and the Art of
REQADM Maintenance
Steve Willoughby
PMD Engineering Computing
12 April 2000
Copyright © 2000 Intel Corporation.
What This Course Is
• Introduction to the REQADM system
components for the REQADM
Administrator.
• Tips for configuring and maintaining the
system.
• Provides familiarity with system to help you
troubleshoot your system.
• A peek ahead at REQADM 2.0, and how
it’ll change what you do.
What This Course Is Not
• User-level introduction to REQADM clients
for either systems administrators or
customers.
• In-depth look at report generation.
• How to organize a help center.
• There are separate presentations available
for each of those.
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Agenda
• What is REQADM, inside and out?
– System Architecture
• Directory Layout
– Ticket States and Statuses
– Overview of Clients
– Server Design and Components
System Architecture
• Server stores and manages tickets.
– All access to ticket data goes through server.
• Ensures reliable data, lock management, (minimal)
security, sanity-checking, common interface.
• Clients provide various interfaces to server.
– New interfaces added independently, don’t
require server or other client changes.
– API library
Directory Layout
/opt/reqadm*
archive
bin
db
lib
Backup
sbin
spool
queue1
queue2
Database
files
for
ticket
data
Configuration
files,
ASCII
“Core”
files
REQADM
for
closed
Supplimental
scripts
Nightly
cron
script
Purged
tickets
REQADM
server’s
files
stored
in in
*.pidcompiled
files,
etc.
tickets
system
stored
programs
and
makes
database
archived
here
for later
DBMS
directories
named
(server,
clients,
of
1,000
programs,
not
“core”
backup
copies
andfor
retrievalmodule.
(compressed
the
REQADM
queues
tickets
maintenance
each.
scripts).
REQADM
services.
recovery
files
here.
archives)
*default name; YMMV
Fixed
D0
D1000
Agenda
• What is REQADM, inside and out?
– System Architecture
• Directory Layout
– Ticket States and Statuses
– Overview of Clients
– Server Design and Components
Ticket States
New Ticket
Assigned
Prioritized
Work in Progress
On Hold
Categorized
Resolved
Closed
Ticket Status
• REQADM rates each ticket’s status:
– Invalid:
• Malformed ASCII file or other anomaly. Normally,
you wouldn’t even see these, as the clients generally
filter them from view.
– Proto:
• A new ticket still being built by REQADM. Not
really a request yet. Again, clients tend not to show
these to anyone until they’re full-blown tickets.
Ticket Status
• REQADM rates each ticket’s status:
– Re-queued:
• Ticket has been moved to a different queue; this
number is the old ticket number it used to have.
• Kept for historical references.
• Read-only copy; can’t be changed further.
• REQADM 2.0’s flat numbering space will render
this obsolete.
Ticket Status
• REQADM rates each ticket’s status:
– Resolved:
• Work completed. SLA “clock” stopped. Referred to
customer for approval or denial.
– Closed:
• Work completed and approved by customer.
– Pending:
• “On hold” pending action by 3rd-party or customer.
SLA “clock” stopped while on hold, unless
explicitly committed to a specific date and time.
Ticket Status
• REQADM rates each ticket’s status:
– New:
• Submitted but not yet assigned to an owner.
– Due:
• Due (by SLA or commitment) within one hour.
– Stale
• Not modified within grace period (site
configurable).
– Active:
• Any ticket not matching any of the above.
Agenda
• What is REQADM, inside and out?
– System Architecture
• Directory Layout
– Ticket States and Statuses
– Overview of Clients
– Server Design and Components
Overview of Clients
• REQADM includes several standard clients
which provide user interfaces to the system.
• Also maintenance clients (discussed later).
• Custom clients may be written locally in
C++ or Perl.
ASCII Tools
• reqadm
– If ticket number is already known, can
interactively operate on ticket, doing just about
anything the X interface can do.
– Batch-capable: can do all operations using
command line arguments in non-interactive
mode.
– Can perform multiple operations on many
tickets in one step.
ASCII Tools
• reqadm-find
– Displays queue list (one ticket per line).
– Use to search for tickets or monitor queues.
– Options to look for specific ranges of tickets.
• Phonetic subject search.
• Search by date ranges, people involved, etc.
ASCII Tools
• reqadm-create
–
–
–
–
Creates new ticket for customer.
Support may invoke on behalf of customer.
ASCII input field editing.
Spawns external text editor for multi-line fields
(user option) or uses line-at-a-time entry
system.
GUI Tools
• xreqadm
– Integrated tool with actively-updated queue
display and “editor” windows to work with
individual tickets.
– Can perform multiple operations on a single
ticket, or a single operation on multiple tickets.
– Arguably the most useful way for support
personnel to use REQADM.
GUI Tools
• reqadm-create
– If a $DISPLAY variable is set in the
environment, reqadm-create will attempt to
shift to X mode rather than ASCII mode.
– Displays an interactive GUI form to fill out.
GUI Tools
• reqadm-prefs
– Allows users (or support on behalf of a user) to
customize their personal settings.
– “Look and feel” of all REQADM clients
affected.
– E-mail sent to user (from whatever tickets)
customized to their own preferences.
Web Tools
• Most of the GUI functionality also available
on web server (clients respond to being run
as CGI rather than stand-alone).
• CGI environment seriously hinders what
tools can do, so this is not the most
desirable way to access REQADM,
especially for support personnel.
Mail Tools
• reqadm-mail-receiver
– Invoked to handle all incoming mail to
REQADM, for new tickets and notes added to
old ones.
– Written in Perl, so lots of customization
possible.
– Invokes the batch “reqadm” commands to do
the actual work.
Report Tools
• reqadm-report
– General-purpose reporting package for
REQADM.
– Pulls collection of tickets from server, analyzes
them for various metrics.
– Produces Structured ASCII, PostScript or
Comma-Separated ASCII outputs, suitable for
printing or importing into spreadsheets and
other programs.
Agenda
• What is REQADM, inside and out?
– System Architecture
• Directory Layout
– Ticket States and Statuses
– Overview of Clients
– Server Design and Components
Server Design
• Server is the central manager of all activity
within REQADM:
– Storage and retrieval of tickets.
– Management of simultaneous access between
users (lock management, update management).
– Tracks user customization and site
configuration information.
– Interacts with external services such as CDIS
and LDAP.
Server Design
Individual Client
Connections
...
Client Session Management layer
Server Command Interpreter
Primitive Operations layer
Lock
Configuration Personnel
DBMS
Manager
Manager
Manager
Connections to External Services
Server Subsystems
• DBMS
– Performs database management services.
– REQADM 1.x uses this to store “overview”
data (essential attributes of tickets) for fast
queries and displays. Details still stored in
ASCII files.
– REQADM 2.0 will move to all ticket data
being stored in the database, for increased
speed, and enhanced search capabilities.
Server Subsystems
• Lock Manager
– Arbitrates who gets to access tickets.
– Currently supports both read (simultaneous)
and write (exclusive) locks, but most common
clients expect to update ticket, so always get
write locks.
• This means anyone viewing a ticket will lock out all
other users from touching it.
– Locks expire after a site-configurable length of
time, or the client disconnects from the server.
Server Subsystems
• Lock Manager, continued
– Locks will be broken only if already expired
and another client wants the same lock.
– Even if lock broken, original writer allowed to
update if the intervening locker(s) didn’t
modify the ticket.
– REQADM 2.0 will introduce more advanced
locking, allowing simultaneous reads/writes.
Server Subsystems
• Configuration Manager
– Serves site configuration to clients. Discussed
in detail later.
• Personnel Manager
– Handles user information in the database.
– Serves user info to clients, allows clients and
automated processes to update database from
external sources (e.g., Human Resources).
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
External Services vs. REQADM
• REQADM does not operate in a vacuum.
• Gets information about customers and
support personnel from external personnel
information servers and/or files.
• Interacts with mail system to send and
receive tickets and updates.
• Web interface interacts with web server and
authentication systems.
User Identification Infrastructure
• User ID needed
for tracking and
security.
• Initially loaded
from static files
– /etc/passwd
– site “user table”
CDIS
Request
Server
LDAP
Usertable*
passwd
*and CDIS and stdusers files.
• Dynamic lookup possible if you use LDAP and/or
CDIS servers, which REQADM will talk to.
User Identification Infrastructure
CDISID”
• All users are identified by “world-wide
number.
Request
Server
– Any number between 1 and
134,217,727.
• (That’s 0x00000001 and 0x07ffffff.)
LDAP
– Allows usernames, full names, e-mail
addresses, user ID numbers,
etc. passwd
to change
usertable
without losing ticket tracking history.
• But how do we map a person to a WWID in
the first place, and maintain that mapping?
Static User ID Files
CDIS
• The baseline REQADM system reads
four
static files to get this user
information:
Request
Server
– /etc/passwd (or passwd NIS
map)
LDAP
• Defines full name, user ID number and username
(therefore E-mail address) for each local user.
– Site “usertable” file
usertable passwd
• Associates username with WWID number.
Static User ID Files
• Usertable is a simple ASCII file, oneCDIS
line
per user, to map WWIDRequest
to UNIX username:
1042:steve
1100:scaggs
Server
LDAP
usertable
WWID
username
passwd
Static User ID Files
CDIS
• The baseline REQADM system reads
four
static files to get this user
information:
Request
– /opt/reqadm/lib/stdusers Server
LDAP
• Defines WWIDs (with names and mail addresses)
for users not in the password file (including
administrative accounts which wouldn’t be in the
usertable passwd
usertable file).
– Site “CDIS” data file
• Defines personal information (phone numbers,
office location, etc.) for each WWID. Optional.
Static User ID Files
CDIS
• /opt/reqadm/stdusers is formatted just
like
usertable, but is for defining
non-user IDs
Request
within REQADM. May Server
include additional
LDAP
fields:
100000000:reqadmad:reqadmad@mydo
main.com:REQADMusertable
Administrator
passwd
WWID
username
E-mail
Full name
Static User ID Files
CDIS
• /opt/reqadm/stdusers could be used for:
– The REQADM administrator
account (which
Request
Server
probably won’t be in CDIS)
– Other “non-human” accounts which may LDAP
send
or manipulate tickets
– Accounts for customers
or vendors
who aren’t
usertable
passwd
in your password file
Static User ID Files
• CDIS file is supposed to be importedCDIS
on a
regular basis from someRequest
corporate HR
Server or other
database, customer database,
LDAP
appropriate source
• Tab-delimited ASCII file with one line per
usertable
passwd
user, identified by WWID
number.
• REQADM only uses certain fields from this
file:
Static User ID Files
•
•
•
•
•
•
•
00
06
16
19
21
24
25
CDIS
Full name of user
Request
E-mail address
Server
Mailstop
LDAP
Phone number for messages (secretary?)
Office location (room/cubicle
number)
usertable passwd
Pager number
Office phone number
Static User ID Files
CDIS
• 32 WWID number
Request
• 33 Cellular phone number
Server
• 39 Date and time entry was last updated
LDAP
– In format such as mm/dd/yyyy hh:mm. See
Intro.txt for description of other legal formats.
passwd
– This date field is used usertable
by REQADM
to
determine whether CDIS data is more recent
than REQADM’s own database (triggering an
update from this file into the REQADM
database).
Static User ID Files
• Nightly process updates REQADM CDIS
database from these files.
Request
Server
• This may not be timely enough,
or contain
LDAP
all users who want to use REQADM.
• Dynamic updates of user information
possible using externalusertable
services:passwd
– LDAP
– CDIS
E-Mail Address Resolution
• When REQADM needs to identify aCDIS
user
based only on E-mail address
Request (such as when
Server
processing incoming mail
messages):
– If address looks like “domain\username”LDAP
(NT
style), we assume the local NT domains are in
sync with the UNIX password
so we drop
usertable file,
passwd
the domain\ part and search for username:
• In REQADM’s database of known usernames
• Using a CDIS server, if one is available.
E-Mail Address Resolution
CDIS of
• Otherwise, if the address is only a string
digits with no domain orRequest
punctuation, we
Server and look it
assume it is a WWID number,
LDAP
up as explained below.
– We have off-site users who authenticate to the
web using their WWIDusertable
number and
a
passwd
password, so this is how they appear to the
web-based REQADM clients.
E-Mail Address Resolution
CDIS
• Otherwise, we check REQADM’s database
of known E-mail addresses,
with and
Request
Serveraddress (we’ll
without canonizing the mail
try adding the local mail domain to LDAP
unqualified addresses).
usertable
passwd
E-Mail Address Resolution
CDIS
• If all the above fails, we contact an LDAP
directory server (if one is
available) which
Request
is configured to translateServer
mail addresses to
WWID numbers for our organization. LDAP
• If no such user could be found, the client
passwd
may elect to generate ausertable
temporary
WWID to
use for this user.
WWID Lookups
• If we are given a WWID number forCDIS
a user,
we look them up in the system
Request by:
Server we retrieve the
– If the WWID is a group number,
LDAP
group information from the REQADM database
(which comes from workgroups.conf).
– Otherwise, if the WWID
isn’t found
in
usertable
passwd
REQADM’s database already, a CDIS server is
contacted (if available) to provide supplimental
info for the user, and that is stored in
REQADM’s database for future use.
User ID Resolution
• Most clients look at the user ID theyCDIS
are
running under, and resolve
that to a
Request
Server
REQADM WWID number:
LDAP
– REQADM’s database of user ID’s is searched.
– getpwuid() called to translate user ID to
username.
usertable passwd
– Site “usertable” file opened and searched to get
a WWID number for that username.
– CDIS is contacted to get supplimental info and
a new record is created in the database.
LDAP Server
CDIS
• The basic idea here is that you can set
up an
LDAP sever which contains
all the valid ERequest
Server your
mail addresses known within
organization, and associates them withLDAP
WWID numbers.
usertable like
passwd
• Useful for large organizations
Intel
which have lots of users who may have
accounts at multiple company sites.
LDAP Server
CDIS
• The LDAP server used by REQADM
should accept queries ofRequest
the general form:
– o=Organization-name Server
– (mail=address-being-searched-for)
LDAP
• The response should include an attribute
usertable
passwd
called “employeenumber”
which
contains
the WWID for that address.
• All of the above query components are site
configurable to match your server settings.
CDIS Server
• This is a simple server which servesCDIS
corporate personnel information
in response
Request
Server
to a query in one of the following
formats:
– iname
– wnumber
LDAP
usertable passwd
• If the specified local username
or WWID
number is not known, CDIS responds:
– Key ’name’ not found.
CDIS Server
• If a record was found, data lines are CDIS
returned:
WWID= 1234
Request
DomainAddress= [email protected]
Server
Fname= JOHN
LDAP
Lname= RANDOM
Mailstop= X1-111
usertable passwd
MI=
X
OfficeLoc= X1-1-11
MsgNum=
5551111
CDIS Server
CDIS
• Easy to implement if you have the corporate
data somewhere you canRequest
get to it (on-line
database, static file, etc.).Server
LDAP
• Can be written in Perl without much effort.
• We intend to include a small sample CDIS
usertable passwd
server in REQADM 2.0’s
distribution.
Mail Server Interaction
Incoming Ticket
Update Notices
Response to Notice
Mail
Server
Approval/Denial
Request
Server
DB
New Ticket Receipts
Mail Server Interaction
Incoming integrated
Ticket
• Tightly
with mail services
Update Notices
• “Use the tool you’re Mail
currently using”
Response
to Notice
– Customer
gets E-mail
notice of ticket activity
Server
New Ticket Receipts
– Can simply “reply” to mail—reply goes back to
Approval/Denial
REQADM, is noted in ticket history.
• New commentary at top of each mail
message—gives feel of casual E-mail
Request
conversation between
parties.
Server
DB
Mail Server Interaction
Incoming
Ticketsends its own outgoing mail by
• Each
client
contacting the SMTP server. Update Notices
Mail
Notice
• Response
Returntomail
is spooled
and fed into the
Server
New Ticket Receipts
REQADM server by reqadm-mail-receiver
Approval/Denial
script
– Generally run every 5 minutes via cron.
Request
Server
DB
Web Server Interaction
Browser
Request
Server
Web
Server
Browser
Browser
Browser
NT Auth
Server
htpasswd
Web Server Interaction
• Most REQADM functions available via CGI.
Browser
• Problem
Request of authenticating user to REQADM.
– Server
What functions to allow (security)? Browser
Webaction on tickets?
– Whom to log as taking
Server
Browser
• Easiest is to make CGI directory passwordprotected,
Browser
entries
NT Auth and make sure all users have
in Server
the web server password file.
htpasswd
Web Server Interaction
• We assume the web username is the same as:
–Request
UNIX username,
Browser
– Server
Numeric-only WWID number (no “P” prefix),
Browser
– E-mail address, or Web
Server
– NT-style username (domain\username)Browser
• It’s
the
web
server’s
responsibility
to
Browser
NT Auth
authenticate
the user by whatever means
Server
necessary (NT auth server, .htpasswd file, etc.)
htpasswd
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Installation
• We won’t go through a tutorial of the
installation process here.
• But please refer to all these documents
before installing:
– README file
– RELEASE-NOTES file
• What’s in this release, known issues, etc.
– Intro.txt file
• Lots and lots of infrastructure details, etc.
Installation
• However, there are some setup parameters
you need to figure out before you install.
– These will be “burned in” to the REQADM
binaries when installed. Changing them will
require a re-install.
– Why? Mostly because it’s difficult to get so
many customers to set environment variables or
dotfiles to point REQADM clients to the local
server and configuration directories. So we
burn that info into REQADM directly.
Installation Parameters
• REQADM server hostname and TCP port
– Clients will connect sockets there to talk to server.
• Pathname to Perl 5 interpreter
– Written into #! lines of Perl scripts.
• Pathname to GNU Tar utility
– Written into some scripts which use tar, also used
by the install program to unpack files.
Installation Parameters
• REQADM home directory
– Called “/opt/reqadm” here; server files will be
installed there.
• Pathname REQADM web images, HTML dir
– Setup will install files there.
• Pathname to client directory
– Clients will look here to find help files.
Installation Parameters
• Assigned site codes “A” and “B”
– Together, these form a 64-bit number which clients
send to assure the server they are the correct
clients for that server. (Not really air-tight
security, but helps keep accidents from happening.)
– Each can be any number from 99,999,999 to
2,147,483,647. Just be consistent throughout all
your installations on all platforms.
Installation Parameters
• REQADM account user ID
– Numeric UNIX user ID number for the account set
aside to run the REQADM server. The setup
program will chown all files to this account.
• Web server account user ID
– Numeric UNIX user ID number for the account
which runs your web server. If a REQADM client
is run by this account, it assumes it’s a CGI client
and reacts accordingly.
Installation Parameters
• Other parameters are configured at runtime
and may be changed at will.
• Caveat: web pages contain many of these
runtime configuration values (e.g., your
organization name and phone number).
– These are static HTML files, so you must
configure these items before installing the web
support. Changing these fields requires reinstalling the web support.
Installation Parameters
• The fields affected are listed in Intro.txt.
• In REQADM 2.(mumble), the static pages will
all be replaced by a dynamic document
server, which can get these values in real
time from the REQADM server, so this
won’t be a problem anymore.
Runtime Configuration
• The parameters used to configure
REQADM are stored in several files:
–
–
–
–
–
–
reqadm.conf
catcodes.conf
hours.conf
queues.conf
systems.conf
workgroups.conf
Configuration: reqadm.conf
• This is the master configuration file for the
REQADM system.
• Values here are available to any client.
• Sets most of the policies and behavior of
REQADM.
Configuration: reqadm.conf
• REQADM is very touchy about mistakes:
– A missing field will crash the server and/or
client which was trying to access that field.
– Be very careful to follow the file’s syntax rules.
– Since this controls so much of REQADM’s
internal code, be very sure you know what you
are doing before changing these values.
Configuration: reqadm.conf
• ASCII file, defining one “field” (i.e.,
parameter) per line.
• Comments begin with a “#” as the first
character on the line (no whitespace allowed
to the left).
– “#” elsewhere on the line has no special meaning.
• Lots of comments in the file, explaining what
the fields do. Pay close attention to them.
Configuration: reqadm.conf
• All fields have a four-character name.
– Can actually be any four bytes, including nonprintable characters (but we recommend sticking
to printable ASCII).
– Case-sensitive.
– REQADM will never define fields with names
beginning with “$” (e.g., “$ABC”), so you are
free to use those names for custom clients which
need central configuration fields.
Configuration: reqadm.conf
• Text fields defined as:
SUPN=Acme Widgets, Inc. Support
Field
name
Field
value
Configuration: reqadm.conf
• Multi-line text fields defined as:
CAT@=E-mail problem
CAT@=Workstation crashed
CAT@=Need software installed
Field
name
It’s a multi-line (array) field if
the field’s name ends with “@”
Field
value
Configuration: reqadm.conf
• Numeric fields defined as:
LTO#=3600
Field
name
It’s a numeric field if the
field name ends with a “#”
Field
value
Configuration: reqadm.conf
• Numeric fields may be in different bases:
LTO#=3600
LTO#=$3FF
LTO#=@277
LTO#=%010111111
Decimal
Hex
Octal
Binary
Configuration: reqadm.conf
• This file sets policies enforced by
REQADM:
– Grace period before resolved tickets
automatically close w/o customer approval,
– Lock timeout interval,
– Definition of “stale” ticket (how old),
– Account to notify for all ticket resolutions and
commitments
Configuration: reqadm.conf
• This file sets identity information about your
group:
–
–
–
–
–
Support center name and phone number,
Escalation mail address if customer not satisfied,
E-mail domains local to your group,
Return address and ID name for REQADM mail,
Your local area code
Configuration: reqadm.conf
• This file sets resource location information:
–
–
–
–
–
–
Pathnames of other configuration files,
URLs of web components,
Host info for SMTP, CDIS, LDAP, etc.,
Commands for printing, paging, etc.,
Pathname to client help files,
Pathnames to static user ID files
Configuration: reqadm.conf
• This file defines custom content:
– HTML code to add to stock REQADM home
page,
– Text of error messages to be mailed out,
– Which web graphics are used
• Very skilled (and brave) admins can use these to
change the appearance of the web forms used by
REQADM.
• REQADM 2.(mumble) will extend this to include GUI
tools’ graphic images and colors as well.
Configuration: reqadm.conf
• This file controls REQADM behavior:
–
–
–
–
–
–
Mail receiver parameters, alias names, etc.,
Category to use when merging tickets,
What queues customers can submit tickets into,
Input problem type list,
Required and/or edit-locked input fields,
Standard pre-define queue views for xreqadm
Configuration: reqadm.conf
• Changes made to this file will be seen as
soon as the reqadm server daemon is sent a
HUP (1) signal.
• Clients may have these values cached; they
will need to be restarted to load fresh values
again.
– REQADM 2.0 will include code to notify the
clients when these values change, so that will
no longer be necessary.
Configuration: catcodes.conf
• Defines all root-cause categories available
to categorize tickets.
– Different than the problem type (CAT@) in
reqadm.conf, which was the simple “what kind
of complaint I have” data point supplied by the
customer.
– This is the more detailed “what actually caused
the problem in the first place” data point
supplied by the support tech resolving the
ticket.
Configuration: catcodes.conf
• These may be used to analyze the major
issues and problem trends.
• Hierarchial list of categories; for example:
– Software Problems
• Need upgrade
• Bug in application
– Workstation Problems
• Dead hardware
• Misconfigured system
Configuration: catcodes.conf
• File defines one category per line.
– Comments as with reqadm.conf
– Use “/” to show hierarchy (like a pathname):
sw
sw/bug
sw/upgrade
ws
ws/dead
ws/config
=
=
=
=
=
=
Software Products
Bug in application
Upgrade needed
Workstation Problems
Dead hardware
Mis-configured system
Configuration: catcodes.conf
• File defines one category per line.
– Comments as with reqadm.conf
– Use “/” to show hierarchy (like a pathname):
sw
= Software Products
sw/bug
= Bug in application
sw/upgrade = Upgrade needed
ws
= Workstation Problems
Category code (used
Category
ws/dead
= Dead
hardware description
Obsolete
2nd
internally and
with
field (can be
ws/config
= Mis-configured(multiple
system
command-line
any nonoptions
words okay)
space text)
Configuration: catcodes.conf
• Don’t forget to include the category defined
in reqadm.conf’s CDUP field.
• Changes to this file may not be noticed until
the server is restarted. This will eventually
be fixed so that sending a HUP signal will
suffice.
Configuration: hours.conf
• Defines the hours your center is “open”.
• Used for calculating SLA-based due dates
for tickets.
• A single line containing hour numbers for
“open” and “close” time, Sunday through
Saturday.
• Will probably be moved into reqadm.conf at
some point.
Configuration: hours.conf
0 0 8 17 8 17 8 17 … 8 17 0 0
Closed
Sunday
Open Monday
8:00-17:00
Open Tuesday
8:00-17:00
Open Friday
8:00-17:00
Closed
Saturday
Configuration: queues.conf
• Defines the ticket queues used by REQADM.
• File defines one queue per line.
– Comments as with reqadm.conf.
– First queue is the default queue for all clients.
general
restore
project
005 010 040 400 0
010 010 010 000 0
000 000 000 000 0
Configuration: queues.conf
• Defines the ticket queues used by REQADM.
• File defines one queue per line.
– Comments as with reqadm.conf.
– First queue is the default queue for all clients.
general
restore
project
Queue
name
SLA
hours
(urgent)
005 010 040 400 0
010 010 010 000 0
000 000 000 000 0
Not used
SLA
hours
(low)
Configuration: queues.conf
• Removing a queue from this file will not
delete it from REQADM. Once a queue is
set up (especially if it contains tickets),
REQADM is too paranoid to delete it.
– You could, however, scratch and reload your
database and just happen to fail to load that
queue back in.
– But beware if that (deleted) queue had tickets
requeued to other active queues, or vice versa.
REQADM will still need to have it around.
Configuration: queues.conf
• Changes to this file will not be noticed by
REQADM until the server is restarted.
• This will eventually be fixed so sending a
HUP signal will suffice.
Configuration: systems.conf
• Defines an initial set of host names, locations
and platform types to associate with tickets.
• File defines one system per line.
Intel::Linux|lenny|John’s Office
Intel::Linux|laverne|Mary’s Office
Sun::3/60|balthmus|Joey’s Office
Intel::Windows::NT 4|pc123|Accounting
Intel::Windows::98|demo27|Sales Laptop
Configuration: systems.conf
• Defines an initial set of host names, locations
and platform types to associate with tickets.
• File defines one system per line.
Intel::Linux|lenny|John’s Office
Intel::Linux|laverne|Mary’s Office
Location
Sun::3/60|balthmus|Joey’s Office
Platform name.
Intel::Windows::NT 4|pc123|Accounting
Note hierarchy
Hostname
Laptop
with “::” Intel::Windows::98|demo27|Sales
(unique
in
separating each
database)
level.
Configuration: systems.conf
• Removing a system from this file will not
remove it from the database. This file is
just used to load into (merge with) the
current database.
• The contents of this file are re-read when
the reqadm server is started or receives a
HUP signal.
Configuration: workgroups.conf
• Defines privileged users: those who can
own tickets and have “administrator” access
to the REQADM tools.
– I.e., your systems administration staff.
• Also defines what group(s) each user
belongs to.
– Can then see a queue view for all tickets owned
by the group (those owned by any members of
the group), to simplify managing the group.
Configuration: workgroups.conf
• Also defines “rotations”
– “Fake” groups corresponding to areas of
ownership (accounts, postmaster, etc.).
– Tickets may be assigned to the rotation group,
then given to the actual owner who is in the
rotation at that point in time.
• Manually by a dispatcher or the recipient themselves
• Or automatically via tools like REQMGR.
Configuration: workgroups.conf
• One group/rotation/user per line.
• Users must be defined after the group they
belong to.
– Multiple entries for the same user indicate they
belong to multiple groups.
• Groups may be nested.
• Each group is assigned a number in the
same range as WWID’s (but separate
numbering space; will not conflict)
Configuration: workgroups.conf
• Sample file:
group 1 N support
member 1042 steve
group 2 N mailteam 1
member 1157 joey
member 1291 beth
group 100 Y postmaster
group 101 Y accounts
Configuration: workgroups.conf
• Sample file:
group 1 N support
member 1042 steve
group 2 N mailteamGroup
1 name (shouldn’t be
Start member 1157 joey the same as an existing
defining
user)
member
1291
beth
a group
group 100 Y postmaster
101 Y accounts
Thisgroup
is
The group may not own
group #1
tickets (not a rotation)
Configuration: workgroups.conf
• Sample file:
group 1 N support
member 1042 steve
group 2 N mailteam 1
Start member 1157 joey
defining a
UNIX username (must
1291 beth
membermember
of
match with WWID in
the support
group 100 Y postmaster
database)
group
group 101 Y accounts
WWID
Configuration: workgroups.conf
• Sample file:
group 1 N support
member 1042 steve
group 2 N mailteam 1
Start member 1157 joey
definingmember 1291 beth
This group is directly
a group
group 100 Y postmaster under group #1 in the
hierarchy (so “support”
101 Y accounts includes everyone in
Thisgroup
is
group #2
“mailteam” too).
Configuration: workgroups.conf
is
• This
Sample
file:
group #100
group 1 N support This group may own
Start member 1042 steve tickets, so this is
actually defining a
defining
1
a groupgroup 2 N mailteam“rotation”.
member 1157 joey
member 1291 beth
group 100 Y postmaster
group 101 Y accounts
Configuration: workgroups.conf
• Changes to this file will be reflected when
the REQADM server receives a HUP
signal.
• You’d usually do this when hiring a new
support person, adding them to the list of
privileged users.
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Maintenance
• Normally, REQADM should run fairly
smoothly on its own, with little or no
supervision.
• Maintenance tasks are automated and run
from cron.
• The only usual exceptions are reconfiguring
or reacting to a problem.
Nightly Cron Script
• /opt/reqadm/sbin/reqadm-nightly runs out of
cron when users are not expected to need
the system.
–
–
–
–
Closes requests which have not been approved
Suspends request server
Backs up database (only n days worth kept)
Updates personnel info from all static user ID
files.
– Allows server to continue running again.
Nightly Cron Script
• Watch the mailed output from this script to
make sure it’s not indicating that disk space
is running low or any other problems are
occurring.
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Suspending the Server
• Some of the following actions require
suspending the REQADM server while they
are performed (e.g., any external access to
the database files).
• This is done with the reqadm-control
command.
• Like everything else we’ll discuss, this is in
/opt/reqadm/sbin. Examples assume you’re
in that directory.
reqadm-control
• ./reqadm-control -locks
– Prints a list of all tickets currently locked:
G ID- TIME- QUEUE----- NUMBER---W 037 00:05 request
130000
HOSTNAME/IP-NUM USERNAME COMMENT--192.168.1.1
steve
REQADM
reqadm-control
• ./reqadm-control -locks
– Prints a list of all tickets currently locked:
G ID- TIME- QUEUE----- NUMBER---W 037 00:05 request
130000
Lock
grade
(write)
HOSTNAME/IP-NUM USERNAME COMMENT--192.168.1.1
steve
REQADM
Session
ID
Lock
Age
Ticket
reqadm-control
• ./reqadm-control -locks
– Prints a list of all tickets currently locked:
G ID- TIME- QUEUE----- NUMBER---W 037 00:05 request
130000
HOSTNAME/IP-NUM USERNAME COMMENT--192.168.1.1
steve
REQADM
System
connecte
d from
Username
Program
running
reqadm-control
• Use to see who’s getting in the way of your
shutting down the server.
• Also useful if people complain that they
have requests locked: find the
system/process where their client is still
running.
reqadm-control
• ./reqadm-control -suspend
– Suspend the server when remaining locks
released, and do something according to other
options specified:
-verbose
– Print informational messages showing what’s happening.
-time mm:ss
– Abort attempt to suspend server if the specified time
expires before locks are released.
reqadm-control
• ./reqadm-control -suspend
– Following the options is one of the following:
mm:ss
– Suspend for the specified length of time, then resume
server operations.
commands
– Execute the shell commands specified. When they
complete, resume server operations.
Checking Database Integrity
• To check to see if the database is intact:
./dbchk ../db/reqadm-db.sc
– Verifies that database is internally consistent
and valid for the DBMS to use.
– Does not check that the data inside the database
actually make sense.
– Database password is “lao:ban0”.
• SUSPEND SERVER BEFORE RUNNING!
Clearing Database Lockfiles
• If something died and left a lockfile behind,
clear it with:
./dbchk -u ../db/reqadm-db.sc
– Be very certain the lockfile is not valid (i.e.,
that the database server or other program isn’t
really running, accessing the database now).
– Database password is “lao:ban0”.
• SUSPEND SERVER BEFORE RUNNING!
Recovering Trashed Databases
• If something catastrophic happened, you
can recover the database from backups, to
quickly get the system back up and running.
• This is tricky (we need a script to automate
this), and if you don’t have a huge number
of tickets, it’s probably better to clear and
reload the database instead.
• In fact, I’m not going to teach it here until
we do get a script for it...
Reloading Database From
Scratch
• Delete the entire contents of the database:
./dbinit ../db/reqadm-db.sc
– Database password is “lao:ban0”.
– If this won’t work because the database is
locked, you can use a -u option just like with
dbchk (and with the same cautions).
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• If the dbinit failed because the database is
too corrupt even for that, you’ll have to
recover an old reqadm-db.sc file. These are
saved nightly as ../db/Backup/REQADM_
DBS.*.*.A. Copy one of these to the ../db
directory under the name reqadm-db.sc and
try the dbinit again.
Reloading Database From
Scratch
• Now that the database is empty, load all your
personnel information from the static files:
./reqadm-update-db -people -verbose
• For speed, this blasts the data directly into the
database itself, bypassing the REQADM
server (which must not be running).
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• Now we need to load all the tickets from the
ASCII files into the REQADM database.
• If you have time to spare, leave the server
down and use this program to bulk-load all the
tickets directly into the database (bypassing the
server, which must not be running):
./reqadm-update-request-db
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• If time is not on your side, and the server must
be started ASAP, you can load only the most
critical tickets by naming them as explicit
arguments to the update program (this loads
only those tickets, and prevents the normal
queue scan):
./reqadm-update-request-db queue/n ...
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• What’s most critical? Maybe you want to load
all the open tickets but not old, closed ones:
cd ../spool
foreach queue (*)
../sbin/reqadm-update-request-db \
$queue/[0-9]*
end
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• If you only load partial tickets, there is one
important set of tickets you must load before
bringing up the server: always load the highestnumbered ticket in each queue!
– These might be closed already, so hunt in the
Fixed/D* directories for them.
– Load them explicitly as explained already, but note
they are specified as queue/number even if stored in
the Fixed directories.
• SUSPEND SERVER BEFORE RUNNING!
Reloading Database From
Scratch
• The reason for this is that if a new ticket is
received, REQADM will assign a new ticket
number higher than what it currently sees in the
queue. If the highest existing ticket was not
loaded, a duplicate ticket number will be issued
by REQADM.
• Once (at least) the highest ticket in each queue
is loaded, you may restart the server.
Bulk-Loading Tickets
• If you wish to bulk-load tickets while the
server is up and running (perhaps to update
status of all tickets, or maybe to finish
loading a set of tickets partially loaded as
noted a moment ago), run:
./bulk-loader
– This scans all your queues, invoking reqadmbulk-load-request-db to load the
tickets into the server.
Bulk-Loading Tickets
• This takes longer, but can be done while the
server’s up, and people are accessing the
request system.
Adding/Correcting Individual
Tickets
• If you want to update the database to reflect
what’s in a ticket (perhaps after editing the
ASCII ticket file directly):
./reqadm-bulk-load-request-db q/no
– This is done with the server up and running
normally.
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Reqmgr
• Request queue manager
– Monitors open requests
– Automatically takes action based on state
• Highly customizeable
– Modular design
– Uses Perl API
REQMGR material by John Mechalas, PMD EC
Reqmgr
• Useful tasks
– notify people when a request is submitted
– auto-assign request from “rotations” to people
– escalate “ignored” requests (extended time w/o
a human owner)
– auto-assign requests with special “Subject:”
tags
Getting Started with reqmgr
• reqmgr.conf
– defines “trigger” conditions
– module called when request state matches a
rule
– modules actually do the work
reqmgr.conf
• Directives
– “AddModule” specifies a module to be loaded
– “Match” determines whether or not multiple
rules can match a single request
– see the man page
• Directive syntax
AddModule <modname>
Match <single|multiple>
reqmgr.conf
• Macros
–
–
–
–
–
substitue one string for another
expanded in rulesets only!
no quoting: value string taken literally
can be nested…
…but there’s no recursion protection
• Macro syntax
Define macroname value string …
reqmgr.conf
• Variables
– Sets a variable name to a text string
– No quoting: value string taken literally
– Visible to all reqmgr modules
• Variable syntax
Set varname value string …
reqmgr.conf
• Rulesets
– define “trigger” conditions that activate a
module
– processed in the order they are defined
• Ruleset syntax
Rule ruleid:
field ::= target value
Module modname
Rulesets
• Rule expression
– possible request fields defined in Reqmgr(3p)
– ::= is an equality, expression match, or list
lookup operator (can be negated)
– value is the data that field is compared to
– see reqmgr.conf(5) for details
• modname must be declared via
AddModule
Sample config
AddModule rotation
Set rot_config rotation.conf
Set rot_count_queues request incident restore
Set rot_assign_msg (reqmgr assignment)
Define ROTATION unixreq postmaster ntreq
Rule 1:
owner =@ ROTATION
Module rotation
Sample Config
• Activates module “rotation” if owner one
of:
unixreq
postmaster
ntreq
• Sets variables to be used by “rotation”
module
Sample Config
• Module “rotation”
– distributed with REQADM
– uses config file to define rotation membership
• day of week or date
• time of day
• rotation name
– assigns requests to rotation members based on
request load
reqmgr Modules
• Module API
– written in Perl
– documented in ReqmgrModuleAPI(3p)
– see sample modules for development hints
Agenda
•
•
•
•
•
•
•
What is REQADM, inside and out?
How does it interact with external services?
How do I set up and configure REQADM?
What standard maintenance is required?
What can I do for special cases or problems?
How can I automate tasks with REQMGR?
How can I stay informed about changes?
Keep In Touch
• Subscribe to the reqadm-users-l mailing list:
– Send mail to [email protected]
– In the message body place the text:
subscribe reqadm-users-l
end
– You’ll receive announcements of new releases,
problems discovered, enhancements planned.
– Also a discussion forum for asking questions of
interest to all REQADM users around the world.
Read the Files
• When you download a new release of
REQADM, be sure to read the RELEASENOTES file to see what has changed since
the last release.
Further Reading
• The REQADM distribution includes an
unadvertised technical documentation section.
• Assuming you installed REQADM’s web
files to /reqadm on your server, go to:
http://.../reqadm/reqadm-tech.html
• Includes design information, database
schemas, client/server protocol
documentation, etc.
Copyright © 2000 Intel Corporation. All Rights Reserved. May be distributed under same terms and conditions as the REQADM distributions.