OPS-12: Caring for an Ailing AppServer™
Roy Ellis
Principal QA Engineer
Agenda
 Anatomy of the AppServer
 Diagnosing Problems
 Help Treating your AppServer
Listening Port
clien
t
2
C
l AppServer
i Broker
e
n
t
s
OPS-12: Caring for an Ailing AppServer
Agent
A
g
e
n
t
s
Agent
Agent
Agent
© 2008 Progress Software Corporation
Anatomy of the AppServer
What is the OpenEdge AppServer?
®
3
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
Or What is the Progress® AppServer?
 Broker
•
•
•
•
4
JavaTM process
Handles communication from clients to agents
Manages client and agent pools
Writes to
<brokername>.broker.log
<brokername>.broker.log
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
Or What is the Progress AppServer?
 Agent
•
•
•
•
5
“C” process
Handles requests and replies to the clients
Communicates “STATUS” to broker
Writes to
<brokername>.server.log
<brokername>.server.log
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
4 Operating Modes




6
State-aware
One to one
State-reset - Default
Stateless
Many to few
State-free
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
2 Connection Models
 Client to Broker gets redirected to Agent
• State-aware
• State-reset
 Client to Agent through the Broker
• Stateless
• State-free
7
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
22 Connection
Connection Models
Models
State-aware
Stateless
Agent
State-reset
State-free
Agent
Agent
Agent
clien
t
AppServer
Broker
8
OPS-12: Caring for an Ailing AppServer
Agent
© 2008 Progress Software Corporation
Anatomy of the AppServer
AppServer Broker Thread Pools
Listening Thread L-3090
Client Threads
AppServer
Server Threads
C-0001
Broker
S-0001
C-0002
S-0002
C-0003
S-0003
C-0004
S-0004
C-0005
S-0005
C-0006
…
9
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
Broker communication with Agents
Broker starts agent and waits for
agent to respond, status set to:
STARTING
Once started the agent reports to
broker and broker sets status to:
Client request agent, is redirected
to agent port, broker sets status to:
AVAILABLE
Client connects to agent, agent
reports to broker, broker sets status
to:
Agent completes request, notifies
broker, broker sets status to:
10
OPS-12: Caring for an Ailing AppServer
CONNECTING
CONNECTED
AVAILABLE
© 2008 Progress Software Corporation
Anatomy of the AppServer
Reading Log Files
[08/04/04@11:41:31.334-0400] P-001268
T-009968
2 AS AS Application
P-001268
T-009968
2 AS AS
[08/04/04@11:41:31.334-0400]
Server connected with connection id:
172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358)
Application
Server connected
withT-009968
connection
id: 172.30.1.47:
[08/04/04@11:41:31.334-0400]
P-001268
2 AS 4GLTRACE
Run
performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789"
:asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc.
(8358)
[Main Block - authcard.p]
[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run
AS 4GLTRACE
Run
“-H localhost
-S 6789”
connectAuthSvc
"-HconnectAuthSvc
localhost -S 6789 " [performCardAuth
- authcard.p]
[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run
[performCardAuth
- authcard.p]
createAuthMsg
"John Doe 1234567890
Visa " [performCardAuth authcard.p]
[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run
sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa "
[performCardAuth - authcard.p]
[08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run
disconnectAuthSvc "1000" [performCardAuth - authcard.p]
[08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application
Server disconnected. (8360)
11
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Anatomy of the AppServer
Supporting Processes
 AdminServer - Required
• JAVA process
• Manages all OpenEdge Brokers and Servers
• Writes to admserv.log
 NameServer - Optional
• JAVA process
• Traffic manager from clients to AppServers
• Writes to <nameservername>.ns.log
12
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Agenda
 Anatomy of the AppServer
 Diagnosing Problems
 Help Treating your AppServer
13
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems
$DLC/bin/asbman -name asbroker1 -start
OpenEdge Release 10.1C01 as of Fri May 9 21:58:30 EDT 2008
Connecting to Progress AdminServer using rmi://localhost:20931/Chimera
(8280)
Searching for asbroker1 (8288)
Connecting to asbroker1 (8276)
Unable to start asbroker1 (8297)
14
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems - <brokername>.broker.log
=====================================================================
[08/05/18@20:05:25.933-0400] P-000283 T-Main 1 --- --/usr1/stat/progress/101c/wrk/asbroker1.broker.log opened.
[08/05/18@20:05:25.939-0400] P-000283 T-Main 1 --- --- Logging level set to =2
[08/05/18@20:05:25.940-0400] P-000283 T-Main 1 --- --- Log entry types activated:
UBroker.Basic,[08/05/18@20:05:26.020-0400] P-000283 T-Main 2 UB Basic
ubroker
version: v101c (07-May-08) (8038)
[08/05/18@20:05:26.314-0400] P-000283 T-L-9090 2 UB Basic
Started listenerthread: L9090 (8043)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- Could not listen on port:
9090 (8044)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- java.net.BindException:
Address already in use (errno:226)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at
java.net.PlainSocketImpl.socketBind(Native Method)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at
java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at
java.net.ServerSocket.bind(ServerSocket.java:325)
[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at
java.net.ServerSocket.<init>(ServerSocket.java:186)
15
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems
$DLC/bin/asbman -name asbroker1 -start
Connecting to Progress AdminServer using rmi://localhost:20931/Chimera
(8280)
Searching for asbroker1 (8288)
Connecting to asbroker1 (8276)
Starting asbroker1. Check status. (8296)
$DLC/bin/asbman -name asbroker1 -query
Connecting to Progress AdminServer using rmi://localhost:20931/Chimera
(8280)
Searching for asbroker1 (8288)
Connecting to asbroker1 (8276)
Broker: asbroker1 not running (8313)
16
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems - <brokername>.broker.log
[08/05/18@20:15:21.807-0400] P-000435 T-S-0004 2 UB Basic
Started server: /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel
2 -logfile /usr1/stat/progress/101c/wrk/asbroker1.server.log -ubpid 435
-Ms 1 -logname asbroker1 -logentrytypes ASPlumbing,DB.Connects logthreshold 0 -numlogfiles 3 -ASID 4 –ubpropfile
/usr1/stat/progress/101c/dlc/properties/ubroker.properties -svrefresh ipver IPv4 -db mydatabase (8108)
[08/05/18@20:15:22.210-0400] P-000435 T-S-0004 1 UB ---------Message from server : Startup Procedure Failed (8113)
[08/05/18@20:15:22.241-0400] P-000435 T-S-0004 1 UB ---------IOException reading message from server : java.io.EOFException:
pipe to server broken (8117)
[08/05/18@20:15:22.241-0400] P-000435 T-S-0004 2 UB Basic
ERROR: cannot start server. (8100)
17
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems - <brokername>.server.log
[08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- ** There is no
server active for database mydatabase on -H cruz -S 6543. (704)
[08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- Error
initializing the application server. (5479)
[08/05/18@20:29:13.992-0400] P-000667 T-000000 2 AS AS Application
Server Shutdown. (5476)
[08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- ** There is no
server active for database mydatabase on -H cruz -S 6543. (704)
[08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- Error
initializing the application server. (5479)
[08/05/18@20:29:15.002-0400] P-000668 T-000000 2 AS AS Application
Server Shutdown. (5476)
18
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems - <brokername>.server.log
[08/04/03@13:03:33.470-0400] P-008824 T-004332 2 AS AS Starting
application server for asbroker1. (5560)
[08/04/03@13:03:33.501-0400] P-008824 T-004332 2 AS AS Application
Server Startup. (5473)
[08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -C:\tmp\ex08\roy\eg1\asstart.r Database sports2000 not connected.
(1006)
[08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -C:\tmp\ex08\roy\eg1\asstart.r startup procedure ended with an
ERROR condition. (8025)
[08/04/03@13:03:33.814-0400] P-008824 T-004332 2 AS AS Application
Server Shutdown. (5476)
19
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Startup Problems
 All “INITIAL” agents must start or the AppServer will
fail to start
• Set initial agents to start to zero
–“initialSrvrInstance=0” in ubroker.properties
• (IPv6 enabled on a machine can cause agents not to
communicate with broker)
 Add “–Djvmstart.debug=1” to the “jvmargs=“ in
AdminServerPlugins.properties
• View $WRKDIR/jvmStart.log file
20
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Phantom Agents
Broker Name
: asbroker1
Operating Mode
: State-reset
Broker Status
: ACTIVE
Broker Port
: 3090
Broker PID
: 25319
Active Servers
:5
Busy Servers
:0
Locked Servers
:0
Available Servers
:5
Active Clients (now, peak) : (0, 6)
Client Queue Depth (cur, max) : (0, 11)
Total Requests
: 124
Rq Wait (max, avg)
: (259 ms, 11 ms)
Rq Duration (max, avg) : (5543 ms, 24 ms)
PID
State
Port nRq
nRcvd nSent Started
Last Change
25320 CONNECTED 02003 000020 000020 000020 May 19, 2008 06:28 May 19, 2008 07:28
25321 AVAILABLE 02018 000031 000031 000031 May 19, 2008 06:28 May 19, 2008 07:28
25323 AVAILABLE 02019 000014 000014 000014 May 19, 2008 06:28 May 19, 2008 07:28
25322 CONNECTED 02006 000026 000026 000026 May 19, 2008 06:28 May 19, 2008 07:28
25324 AVAILABLE 02021 000033 000033 000033 May 19, 2008 06:28 May 19, 2008 07:28
21
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Phantom Agents
ps -ef | grep _proapsv
statqa 25324 25319 0 06:28:47 ?
0:24
/usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf
statqa 25323 25319 0 06:28:46 ?
0:56
/usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf
statqa 25321 25319 0 06:28:45 ?
0:55
/usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 –log
protrace.25320
protrace.25322
22
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Diagnosing Application problems

Look at log files
•
•
•

<brokername>.broker.log
<brokername>.server.log
Database log files where agent is connected
Increase logging levels
•
•
srvrLoggingLevel=4 (extended)
srvrLogEntryTypes=4GLTrace
– UBNet - New for 10.1C
23
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Hanging Agents
Broker Name
: asbroker1
Operating Mode
: Stateless
Broker Status
: ACTIVE
Broker Port
: 3090
Broker PID
: 25319
Active Servers
:1
Busy Servers
:0
Locked Servers
:0
Available Servers
:1
Active Clients (now, peak) : (0, 1)
Client Queue Depth (cur, max) : (0, 1)
Total Requests
: 11
Rq Wait (max, avg)
: (11 ms, 11 ms)
Rq Duration (max, avg) : (24 ms, 11 ms)
PID
State
Port nRq
nRcvd nSent Started
Last Change
25320 SENDING 02003 000001 000001 00001 Apr 4, 2008 11:40 Apr 4, 2008 11:41
24
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Diagnosing Problems
Diagnosing Agent problems
[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application
Server connected with connection id:
172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358)
[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run
performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789"
[Main Block - authcard.p]
[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run
connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p]
[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run
createAuthMsg "John Doe 1234567890 Visa " [performCardAuth authcard.p]
[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run
sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa "
[performCardAuth - authcard.p]
[08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run
disconnectAuthSvc "1000" [performCardAuth - authcard.p]
[08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application
Server disconnected. (8360)
25
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Agenda
 Anatomy of the AppServer
 Diagnosing Problems
 Help Treating your AppServer
26
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
asbman
 -query
• Get status and history of AppServer broker
and agents
 -kill
• Shutdown an AppServer with busy or hanging
agents
• May need to run twice in a row
27
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
LogRead




28
Available on PSDN
ABL source code available
Read and Sort multiple log files
Unsupported
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
Database Connections
 Don’t connect using the startup parameters
• Any database down to a single database will
cause your AppServer agents to be
unresponsive
 Connect using the “CONNECT” call in the
• srvrStartupProc=
• srvrConnectProc=
• srvrActivateProc=
 Use “IF CONNECTED” ABL
29
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
Lifespan
 New in 10.1B
 Described in ubroker.properties.README
 Read the CAUTION!
• Use only when directed by Tech Support
 Periodically force the shutdown of all agents
and allow new agents to start when
requested
30
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
OpenEdge Management
31
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Under Development
D I S C L A I M E R
 The next slide includes information about potential
future products and/or product enhancements.
 What I am going to say reflects our current
thinking, but the information contained herein is
preliminary and subject to change. Any future
products we ultimately deliver may be materially
different from what is described here.
D
32
I
S
OPS-12: Caring for an Ailing AppServer
C
L
A
I
M
E
R
© 2008 Progress Software Corporation
Help Treating your AppServer
List AppServer Connections
 New in 10.2A
 Described in ubroker.properties.README
 List of all clients known to the AppServer,
includes:
•
•
•
•
33
Host/IP
Port
Connected Agent
More!
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Help Treating your AppServer
Help from Progress
 Technical Support
 Knowledge Base (kbase)
 PSDN
34
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
In Summary
 Understand your AppServer
 Diagnose your Problem
 Care for your Ailing AppServer
Listening Port
clien
t
35
C
l AppServer
i Broker
e
n
t
s
OPS-12: Caring for an Ailing AppServer
Agent
A
g
e
n
t
s
Agent
Agent
Agent
© 2008 Progress Software Corporation
Relevant Exchange Sessions
 OPS-15: What was Happening with my
Database, AppServer, Operating System…
Yesterday, Last Month, Last Year?
 SOA-38: OpenEdge Middleware Road Map
and Info Exchange
 OPS-19: What is IPv6 and Why Should I
care?
 OPS-13: Building and Deploying a Highly
Available Application
36
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
?
Questions
37
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
Thank You
38
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation
39
OPS-12: Caring for an Ailing AppServer
© 2008 Progress Software Corporation