Grid Application Support
by the P-GRADE Portal
Gergely Sipos
[email protected]
MTA SZTAKI
Computer and Automation Research Institute
Hungarian Academy of Sciences
www.lpds.sztaki.hu/pgportal
[email protected]
CEOS WGISS Meeting, May 8-12, 2006, Budapest
1
Outline
•
•
•
•
•
•
Motivation of our work
P-GRADE Portal in a nutshell
Grid application development
Grid application execution and monitoring
Support for collaborative work
Conclusions
CEOS WGISS Meeting, May 8-12, 2006, Budapest
2
Layered view of Grid systems
E-Scientists
Applications
Application
toolkits, portals
Higher-level grid services:
brokering, data replica, …
Basic Grid services:
security, job submission, info, …
Graphical grid tools:
P-GRADE Portal
Command line grid tools
Grid middleware:
e.g. Globus, EGEE, …
Resources to be shared
CEOS WGISS Meeting, May 8-12, 2006, Budapest
3
Motivation to the
P-GRADE Portal
• Fast evolution of Grid middleware technologies and
tools:
– GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, …
• Many production Grids are now freely available for eScientists
– EGEE (LCG-2  gLite), UK NGS (GT2),
US Open Science Grid (GT2  GT4), NorduGrid (ARC), …
• The same set of services are available everywhere, but
implemented in different ways
– Computation services, data services, security services, (brokers)
Let’s provide a technology-neutral, intuitive graphical
interface for the most common Grid middleware services!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
4
P-GRADE Portal in a nutshell
• General purpose, workflow-oriented computational
Grid portal. Supports the development and execution of
workflow-based Grid applications.
• Based on standard portlet framework (Gridsphere)
– Easy to expand with new portlets (e.g. application-specific portlets)
– Easy to tailor to community needs
• Grid services supported by the portal:
Service
Job execution
File storage
EGEE grids
Globus grids
Computing Element
GRAM
Storage Element
GridFTP server
Certificate management
Information system
Brokering
MyProxy
BDII
MDS-2
Workload Management System
---
Job monitoring
Mercury
Workflow & job visualization
PROVE
The P-GRADE Portal hides middleware technologies and
solves Grid interoperability problem at the workflow level
CEOS WGISS Meeting, May 8-12, 2006, Budapest
5
What is a P-GRADE Portal workflow?
• a directed acyclic
graph where
– Nodes represent jobs
(batch programs to be
executed on a
computing resource)
– Ports represent
input/output files the
jobs expect/produce
– Arcs represent file
transfer operations
• semantics of the
workflow:
– A job can be executed
if all of its input files
are available
CEOS WGISS Meeting, May 8-12, 2006, Budapest
6
Two levels of parallelism
• The workflow concept of
the P-GRADE Portal
enables the efficient
parallelization of complex
problems
• Semantics of the
workflow enables two
levels of parallelism:
– Parallel execution
inside a workflow
node
– Parallel execution
among workflow
nodes
Multiple jobs
run parallel
CEOS WGISS Meeting, May 8-12, 2006, Budapest
Each job can be
a parallel
program
7
Ultra-short range weather forecast
(Hungarian Meteorology Service)
Forecasting dangerous
weather situations (storms,
fog, etc.), crucial task in the
protection of life and property
25 x
10 x
25 x
5x
Processed information:
surface level
measurements, highaltitude measurements,
radar, satellite, lightning,
results of previous
computed models
Requirements:
•Execution time < 10 min
•High resolution (1km)
CEOS WGISS Meeting, May 8-12, 2006, Budapest
8
Multi-Grid portal:
Demonstration @ EGEE User Forum
Portal Server in
London
P-GRADE
Portal
server
Portal
Client in
Geneva
Manchester
Executable
EGEE (CE)
Leeds
Service
InvocationWestFocus GT4
Grid (UK)
Executable
Poznan
NGS GT2 (UK)
Executable
Job Repository
(GEMLCA)
Budapest
UoW
Brunel
CEOS WGISS Meeting, May 8-12, 2006, Budapest
9
The typical user scenario
Part 1 - development phase
Certificate
servers
SAVE
WORKFLOW
Portal
server
Grid
services
START
EDITOR
OPEN & EDIT
or DEVELOP
WORKFLOW
CEOS WGISS Meeting, May 8-12, 2006, Budapest
10
The typical user scenario
Part 2 - execution phase
Certificate
servers
TRANSFER FILES,
SUBMIT JOBS
DOWNLOAD
PROXY
CERTIFICATES
VISUALIZE
JOBS and
WORKFLOW
PROGRESS
Portal
server
MONITOR
JOBS
Grid
services
SUBMIT
WORKFLOW
DOWNLOAD
RESULTS
DOWNLOAD
RESULTS
CEOS WGISS Meeting, May 8-12, 2006, Budapest
11
Opening the workflow editor
The editor is a Java Webstart application
download and installation is only a click!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
12
Workflow Editor
Define DAG of batch jobs:
1. Drag & drop components:
jobs and ports
2. Connect ports by channels
(no cycles, no loops)
3. Define job and port
properties
CEOS WGISS Meeting, May 8-12, 2006, Budapest
13
Job property window
Properties of a job:
• Binary executable
• (Type of executable)
• (Number of
processors)
• Command line
parameters
• The resource to be
used for the execution:
• Grid
• Broker / resource
CEOS WGISS Meeting, May 8-12, 2006, Budapest
14
Support for manual resource selection:
information system browser
The information system
portlet queries EGEE
and Globus information
systems
CEOS WGISS Meeting, May 8-12, 2006, Budapest
15
Support for broker-based
resource selection
1. Select a broker Grid/VO for the job
2. (Specify extra ranks & requirements for
the job in Job Description Language)
3. The broker will find the best resource for
your job!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
16
Defining input/output data
File type:
input: required by the job
output: produced by the job
File location:
local: my desktop
remote: grid storage site
File name:
unique name of the file
File storage type:
Permanent: final result
of the WF
Volatile: only used for
inter-job data transfer
CEOS WGISS Meeting, May 8-12, 2006, Budapest
17
The typical user scenario
Execution phase – step 1:
Certificate
servers
DOWNLOAD
PROXY
CERTIFICATES
Portal
server
Grid
services
CEOS WGISS Meeting, May 8-12, 2006, Budapest
18
Prerequisite of workflow execution:
Download user certificates
1. Certificate server
details:
• Hostname
• Port number
• User name
• Password
2. Certificate
parameters:
• Lifetime
CEOS WGISS Meeting, May 8-12, 2006, Budapest
19
Result of certificate download
HUNGRID resources
Multiple
certificates
for multiple
Grids/VOs
can be
available
on the
portal
server at
the same
time!
SEE-GRID resources
CEOS WGISS Meeting, May 8-12, 2006, Budapest
20
Workflow Execution
(observation by the workflow portlet)
White/Red/Green color means
the job is initial/running/finished state
CEOS WGISS Meeting, May 8-12, 2006, Budapest
21
Workflow Execution
(observation by the workflow portlet)
White/Red/Green color means
the job is initial/running/finished state
CEOS WGISS Meeting, May 8-12, 2006, Budapest
22
Workflow Execution
(observation by the workflow portlet)
White/Red/Green color means
the job is initial/running/finished state
CEOS WGISS Meeting, May 8-12, 2006, Budapest
23
Workflow Execution
(observation by the workflow portlet)
White/Red/Green color means
the job is initial/running/finished state
CEOS WGISS Meeting, May 8-12, 2006, Budapest
24
Workflow Execution
(observation by the workflow portlet)
White/Red/Green color means
the job is initialised/running/finished
CEOS WGISS Meeting, May 8-12, 2006, Budapest
25
On-Line application monitoring
- The portal
monitors and
visualizes
parallel jobs
- The portal
monitors and
visualizes
workflows:
-Input data
transfer
-Job execution
-Output data
transfer
CEOS WGISS Meeting, May 8-12, 2006, Budapest
26
Downloading the results…
- Download
small files in a
single ZIP file
- Keep large files
in the Grid for
future analysis!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
27
Collaborative work
by the P-GRADE Portal
• Export and import workflows and
execution results from/into the portal
• Share legacy applications as services with
other portal users
• Share successfully executed jobs as
services with other portal users
• Integrate other users’ legacy applications
and jobs into your workflow
CEOS WGISS Meeting, May 8-12, 2006, Budapest
28
References
• P-GRADE Portal service is available for
– SEE-GRID infrastructure
– HUNGRID VO of EGEE
– Central European VO of EGEE
– GILDA: Training VO of EGEE
– UK National Grid Service
• Under preparation for
– US Open Science Grid, Economy-Grid,
Swiss BioGrid, BioInfoGrid, Turkish Grid
• P-GRADE portal can be installed for any
public/private EGEE or Globus Grid/VO
Please contact us!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
29
Main conclusion:
Easy-to-use, technology-neutral
Grid portal for e-Scientists
• The P-GRADE Portal hides differences of Grids
• Globus X – LCG2 – gLite Grid interoperability
• Transparent switching between Grid technologies
• Graphical tools for application development, execution and
monitoring
• Seq & parallel components can be integrated into large Grid
applications
• Your code does not have to contain grid specific calls
• Support for collaborative research
• Share workflows
• Share legacy applications and jobs as application services
• Built by standard portlet API
• customizable to specific application areas and user groups
(such as earth scientists…)
CEOS WGISS Meeting, May 8-12, 2006, Budapest
30
Further information…
– Visit the www.lpds.sztaki.hu/pgportal site
(manuals, slide shows, installation procedure, etc.)
– Visit or request a training event!
(see list of events on P-GRADE Portal homepage)
• Lectures, demos, hands-on tutorials, application
development support
– Get an account for a portal service:
•
•
•
•
VOCE portal - SZTAKI
SEEGRID portal – SZTAKI
HUNGrid portal – SZTAKI
NGS portal – University of Westminster
– Install a portal for your community: If you are the
administrator of a Grid/VO SZTAKI is pleased to help
you install a portal for your community!
CEOS WGISS Meeting, May 8-12, 2006, Budapest
31
• P-GRADE Portal tutorial in Paris (free event!)
22-23, June – lectures, hands-on, application support
See details on P-GRADE Portal homepage!
• Joint Regional CE EGEE and SEE-GRID
Summer School on Grid Application Support
3-8 July, Budapest, SZTAKI
www.egee.hu/grid06
Thank you!
Questions?
CEOS WGISS Meeting, May 8-12, 2006, Budapest
32