FJPPL meeting
Lyon, 17th of February 2010
Sylvain Reynaud
SAGA specification (overview)
JSAGA
2
SAGA specification / JSAGA impl.
Execution
JSAGA
Data
Security
3
Goal: seamless job submission
job
desc.
JSAGA
infrastructures heterogeneity
JSAGA
(e.g. EGEE, OSG, DEISA)
• which security context to use?
• which transfer path (according to
firewalls, protocols, security…) ?
• which environment variables ?
• how to use commands available
on workers (wget, srmcp, Sget…) ?
based
on
gLite
plug-ins
Globus
plug-ins
JDL
WMS
RSL
staging
graph
SRM
input
data
SAGA
GridFTP
middleware heterogeneity
• how to use the legacy APIs ?
LCG-CE
LCG-CE
job
JSAGA
firewall
(e.g. gLite, Globus, Unicore)
WS-GRAM
WS-GRAM
job
4
interface
Both implementer & user of SAGA
user applications
JSAGA
jobs collection
SAGA
JSAGA
JSAGA
JSAGA

Ready to use software, adapted
to targeted scientific field

A single interface for using all
grid infrastructures

A single interface for using all
middlewares

As many interfaces as design
approaches

As many interfaces as used
technologies
core engine
plug-ins interfaces
plug-ins
legacy APIs
JSAGA
implementation
5
Plug-ins interfaces in JSAGA
SAGA

Close to application developer
needs
JSAGA

– object-oriented
– high-level
– uniform interface for all the
supported technologies

design objectives
– easy to use
… but << certainly not simple to
implement >> (T. Kielmann)
plug-ins
interfaces
Close to existing middleware
APIs
– service-oriented
– low-level
– as many interfaces as design
approaches
+ optional interfaces

design objectives
– easy to implement
– enable efficient usage of
middleware APIs
• engine code = 2 x plug-ins code
JSAGA
6
Plug-ins interfaces
JSAGA
7
Plug-ins interfaces - job (streams)
Streaming
Plug-in interfaces:
direct/buffered/redirected streams
usedset
before/during/after
execution
stream set stream
get stream
for
for non- for
interactive interactive
interactive
getInput
getOutput
getError
SAGA user
interface:
getInput / getOutput
naregi
cream
local
ssh
unicore6
wsgram
gLite-WMS
gatekeeper
done
construction
JSAGA
planned
Job control
8
Plug-ins interfaces - job (monitoring)

Several ways to monitor jobs
Monitoring
– API mode
Plug-in interfaces:
querying / listening
individualquery
job / listlisten
of jobsquery
/ filtered jobs
• poll job status
• listen to notifications about job status
changes
status for status forstatus for
individual individual filtered
job
job
jobs
– API granularity
• individual jobs
• list of jobs
• jobs filter (e.g. by user, by date, by
tag…)


getState
waitFor
SAGA user interface:
getState / waitFor
Plug-in implements several interfaces
User see a single interface: SAGA
naregi
cream
local
ssh
unicore6
wsgram
gLite-WMS
gatekeeper
done
construction
JSAGA
planned
Job monitoring
9
Plug-ins interfaces - data
Optimizations:
 Optional interfaces for plug-ins
– e.g. third-party transfer,
recursive find with meta-data

Cache management of
– opened connections
– meta-data
– content
• explicit via plug-in
cache
zip
file
ftp
sftp
https
Physical
http
irods
srb
srm
gsiftp
rbyteio
catalog
irods
srb
lfn
rns
done
construction
JSAGA
planned
Logical
cache
10
Plug-ins interfaces - security
plug-in supported attributes
/etc/grid-security/certificates
CA
jsaga-context-init
command line
/tmp/x509up_u_$UID
pass
phrase
proxy
path
proxy
proxy
~/.globus/user*.pem
p12
key
Login / pwd
VOMS
MyProxy
G. RFC820
G. Legacy
Globus
InMemCred
SSH
X509
done
construction
JSAGA
planned
cert
11
JSAGA plug-ins
local
ssh
naregi
cream
unicore6
wsgram
gLite-LB
gatekeeper
local
ssh
naregi
cream
unicore6
wsgram
gLite-WMS
cache
zip
file
ftp
sftp
https
http
irods
srb
srm
gsiftp
rbyteio
catalog
irods
srb
lfn
rns
X509
Login / pwd
SSH
JKS
VOMS
MyProxy
G. RFC820
G. Legacy
Globus
InMemCred
12
done
construction
JSAGA
planned
gatekeeper
Security
Physical
Logical
Data
Job monitoring
Job control
Execution
JSAGA plug-ins

SAGA hides most of the
differences between the
technologies

Information on remaining
differences is available for
each plug-in module…
– on JSAGA web site
• Frequently Asked Questions
• Configuration example
– with command line jsaga-help
• supported protocols
• associated security context
• usage, default attributes
JSAGA
usage: jsaga-help -a <ctxId>.<attr> | --config | -d <mode> | -h | -j
<mode> | -s <mode> | -v
[-D <ctxId>.<attr>=<value>
where:
-a,--attribute <ctxId>.<attr>
--config
-d,--data <mode>
-D <ctxId>.<attr>=<value>
-h,--help
-j,--job <mode>
-s,--security <mode>
-v,--version
Output the value of security
context attribute
Output the effective configuration
Information about data protocols.
<mode> = service | context
Set context instance attribute
(e.g. -DVOMS[0].UserVO=dteam)
Display this help and exit
Information about job services.
<mode> = service | context
Information about security context
instances.
<mode> = usage | default | missing
Output version information and exit
13
JSAGA command line interfaces

JSAGA provides command line interfaces for…
– security
• jsaga-context-init
• jsaga-context-info
• jsaga-context-destroy
– execution management
• jsaga-job-run
• jsaga-job-status
• jsaga-job-cancel

– data management
•
•
•
•
•
•
•
•
•
•
jsaga-cat
jsaga-cp
jsaga-ls
jsaga-mkdir
jsaga-mv
jsaga-rm
jsaga-rmdir
jsaga-stat
jsaga-test
jsaga-logical
Also useful as code examples to start with SAGA
JSAGA
14
Software quality

Build process fully automated
– download and install
• build tools
• external libraries
– generate source code
– execute test-suites
• unitary tests
• integration tests
– generate project web site
• documentation
• reports
– generate installer GUI

Plug-ins
– minimal external dependencies
• e.g. gLite-UI no needed
• OS-independent
– maven 'archetype' to generate
skeleton of new plug-in project
– validated by a middlewareindependent SAGA test suite
# SAGA protocols test-suite configuration
gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/
gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/
gsiftp.context=OpenPlast_proxy
https.base=http://grid.in2p3.fr/html/Private/
https.context=Web_X509
file.base=file:///c:/tmp/
file.base2=file:///c:/
JSAGA
15
Installer GUI
JSAGA
16
Licenses

LGPL license
– for the core engine and for
most plug-ins

Optional licenses
– for plug-ins having external
dependencies, which license is
not compatible with LGPL
– then, end-user must…
• either accept the terms of the
license agreement,
• or go back to previous screen
and uncheck the plug-in(s)
JSAGA
17
Contributions

elis@

• a web portal for submitting jobs
to industrial and research
grid infrastructures
• uses JSAGA to hide grid
infrastructure heterogeneity

contribution from
/
• a multi-protocols file browser

– automated SAGA execution
management test-suite
JSAGA
contribution from P.Calvat
– plug-ins for SRB
– plug-ins for iRODS

logical/physical
plug-ins
JJS (Java Job Submission)
• a tool for running efficiently
short-life jobs on EGEE
– many plug-ins
• Job: gLite-WMS, Unicore 6,
Globus WS-GRAM, SSH, local
• Data: RByteIO, SFTP
JUX (Java Universal eXplorer)

contribution from P.Calvat
– compare overhead with JSAGA
/ with direct calls to Globus API
– scalability tests by JJS users
18
Contributions

SimExplorer

• a set of tools, including a
workflow engine providing
distributed computing facilities,
for managing simulation
experiments

contribution from
– scalability tests
• thousands jobs on many WMS
– tests with long-run workflows
• request + test: auto-reset job
service when context renewed
jCAE (Java Computer Aided Engineering)
• Computer Aided Engineering on
distributed computing environment

contribution from
– currently working on SSH plug-in
• enable controlling and monitoring
a SSH job from another JVM
– maybe other plug-ins later…
• GRIA, LSF, torque ?
– wiki pages
JSAGA
19
Contributions

(not a user of JSAGA)
• Open source technologies to
manage, preserve, and link
digital content

contribution from
– access to their grid services
deployed for D-Grid
• Unicore 6
• Globus
– v2 components (Gatekeeper)
– v4 components (WS-GRAM)

(upcoming) contribution
from P.Y.Jallud
– data plug-in to browse content
from Fedora Commons
repository
– will be included in JUX
• gLite
JSAGA
20
Other ideas for contributions

New users => new use-cases
– new tests
• bug-fix requests
• new wiki pages
– new ideas
• feature requests
• help for testing these features
– need support for additional
technologies
• contribute with new plug-ins

New plug-ins
– for job
•
•
•
•
•
batch systems (with DRMAA)
grids
clouds
peer-to-peer desktop grids
pilot job systems
– for data
• physical/logical file protocols
• application level protocols
– for security
• SSO (Shibboleth, OpenSSO…)
JSAGA
21
Conclusion

JSAGA plug-in interfaces designed to
– minimize amount of code needed to support new technologies
– minimize impact of API uniformity on efficiency and scalability

JSAGA plug-in implementations minimize dependencies
– on external tools and libraries (e.g. gLite-UI)
– on operating system (tested on
)

JSAGA is opened to external contributions
JSAGA
22
JSAGA
23
Backup slides
JSAGA
24
C C'
Transfer path depends on…

grid or site

– network filtering policy
– protocols supported for staging
– commands available on workers

– services available from workers
(close Storage Element, shared FS)
– supported context instances
SMTP
SRB
R1
execution service
transfer protocol
– access mode (RO, WO, RW)
– third-party transfer
– supported data protection level
GSIFTP
GSIFTP

data to stage
–
–
–
–
common
R1
result
E1
std-error
shared by several jobs
installed on some worker nodes
file size
required data protection level
HTTP
C'
job
C
GSIFTP
job
Grid
E1
JSAGA
World
job
25
C C'C'' common
EE
Transfer path depends on…

grid or site

– network filtering policy

– services available from workers
(close Storage Element, shared FS)
– supported context instances
SMTP
SRB
R1
execution service
– protocols supported for staging
– commands available on workers
transfer protocol
– access mode (RO, WO, RW)
– third-party transfer
– supported data protection level
GSIFTP
C'
src
E
src
GSIFTP
D1

data to stage
–
–
–
–
executable
D1
input data
R1
result
E1
std-error
shared by several jobs
installed on some worker nodes
file size
required data protection level
HTTP
C"
job
C
GSIFTP
TAR
E1
JSAGA
job
E
sGet
job
26