Generic logging layer for the
distributed computing
by
Gene Van Buren
Valeri Fine
Jerome Lauret
1
Logging API for HENP application
• Most HENP experiment software includes a
logging or tracing API allowing for displaying in a
particular format important feedback coming
from the core application.
• However, inserting log statements into the code
is a low-tech method for tracing the program
execution flow and often leads to a flood of
messages in which the relevant ones are
occluded.
• In a distributed computing environment,
accessing the information via a log-file is no
longer applicable and the approach fails to
provide runtime tracing
2
Practical Grid case example
• A “job” is often a complex workflow or Meta-job
– A Meta-Job can be composed of step1 = several sub-jobs
running in parallel, the result collected and N jobs in step2
process the result etc ….
– We want to have a consistent picture of the Meta-Job (not only
individual ones)
• A given job may be re-directed elsewhere (pre-emption,
re-submit)
• An individual job log may warrant the triggering of some
action (abort the entire task or Meta-job, recovery
service take action, workflow is altered etc …)
• To accommodate for the needed components, we first
need to provide the necessary tools to collect the
information => logger
3
Consistency
• Running a job involves a chain of events where
many components are involved often written in
diverse languages and not offering a consistent
and easily adaptable interface for logging
important events.
• Meta-Job requires a Meta-log
– Workflow should be represented
– All steps may be inspected in details (individual steps)
– An overview of the a global log (Meta-log) MUST be
possible
4
GRID jobs
Job log file
Meta-log of meta-job
STAR GRID job scheduler
GRID realm
STAR meta-job
workflow
GRID jobs
Job log file
5
The job steps …
•
Job initiation
–
–
•
•
Job initialization at the remote node – Shell script
Application (user) job
–
–
–
•
–
•
Starting, application preparation – C++, Fortran, …
Running the application itself – C++, Fortran, …
Job finishing (from the application user point of view) –
C++/Fortran/etc
Job normal completion
–
•
•
– local OS shell script
Job generation/ submission – Java code (for example, STAR
Unified Meta-Scheduler, SUMS)
transferring the log files and clean up – Shell script, Services
(SRM)
file registration, …
job abnormal completion at remote site – Perl script, …
job abnormal completion at submitting side – Java
code,…
6
… etc …
Jakarta log4j project
• An approach based on a new generic layer built
on top of a logger family derived from the Jakarta
log4j project that includes log4cxx, log4c, log4perl
packages may provide a consistency across
packages and framework
• A community supported project
7
Main components
• Log4cxx has 3 main components:
– loggers,
– appenders and
– layouts.
• These three types of components work together
to enable developers to log messages according
to message type and level, and to control at
runtime how these messages are formatted and
where they are reported.
8
Current job
logging
configuration
Production
Manager
STAR Scheduler
Job
Log4cxx.xml
Hierarchy
of the
loggers
List of layouts
List of appenders
Abstract STAR logger
Abstract STAR logger
Abstract STAR logger
Abstract STAR logger
“MySQL Appender
“Console” Appender
“File” Appender
“Custom” Appender
“HTML” Appender
9
Event severity & verbosity
• The first and foremost advantage of any logging
API over plain stdout resides
– in its ability to disable certain log statements while
allowing others to print unhindered.
– This capability assumes that the logging space, that
is, the space of all possible logging statements, is
categorized according to some developer-chosen
criteria
– Repetitive events can be filtered
• Selection of log event severity or verbosity level
10
Run-time log configuration
• Additionally, the power of using log4j design, is the
possibility to enable logging (or features) at runtime
without modifying the application binary or the
wrapper layers.
• The picture shows a dedicated layer implemented as
C++ abstract class library that serves as a proxy
between the application framework and the distributed
environment.
• The approach is designed so that the debugging
statements can remain in shipped code without incurring
a heavy performance cost.
• Logging equips the developer with as detailed context as
necessary for application failures, from testing, quality
assurance to a production mode limited amount of
information.
11
STAR logger configuration file
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true" xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="D1" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-3c{2}:%-5p - %m%n"/>
</layout>
</appender>
<appender name="D2" class="org.apache.log4j.FileAppender">
<param name="File"
value="temp.html" />
<layout class="org.apache.log4j.HTMLLayout">
<param name="ConversionPattern" value="%x: %-5p %c{2} - %m%n"/>
</layout>
</appender>
<appender name="MYSQL" class="org.apache.log4j.MySQLAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="INSERT INTO StarLogger VALUES (&quot;%-3c{2}:%-5p %d
%m&quot;);"/>
</layout>
</appender>
-
</appender>
<root>
<priority value ="DEBUG" />
<appender-ref ref="D1" />
<appender-ref ref="D2" />
<appender-ref ref="D3" />
<appender-ref ref="MYSQL" />
</root>
</log4j:configuration>
12
• The implementation of the STAR logger layer via log4cxx
API allows to independently set the level of the
verbosity for each STAR production “module”. It is
convenient for the debug purposes of some particular
piece of the code. Each developer knows his/ her
module and can set the message level” for that module.
• Capability to include the special appenders like
“MySQL”and “HTML” for certain critical points does allow
to get the job status on-line (via remote Db or Web page)
before the GRID deliveries the entire usually huge log
file to dig through.
• If the needs of the special treatment arisen the custom
dynamically loaded appender filter can be deployed also.
13