Advanced features of the P-GRADE
portal
Peter Kacsuk, Gergely Sipos
MTA SZTAKI
Tamás Kiss
University of Westminster
www.lpds.sztaki.hu/pgportal
[email protected]
1
Contents
• Solving Grid interoperability
• Supporting collaborative Grid application
development
• Classification of Grid portals
2
Grid
interoperability
3
Grid interoperability 1
• P-GRADE portal supports job submission
in 2nd generation Grids:
– LCG – EGEE
– GT2 – UK NGS and US OSG
– NorduGrid
• P-GRADE Portal enables:
– Connecting 2nd generation production Grids
4
Connecting 2nd generation production Grids
UK NGS
GT2
Certificate
server
Proxy
Job
Manchester
SZTAKI Portal Server
User
Workflow
P-GRADE
Portal
v2.3
Leeds
EGEE-VOCE
LCG-2
Budapest
Supports both
direct and
brokered job
submission
Job
Job
LCG-2
broker
Poznan
Job
Brno
5
Grid interoperability 2
• P-GRADE Portal with GEMLCA supports legacy
code service invocation in 2nd and 3rd generation
Grids:
– LCG, gLite – EGEE
– GT2, GT4 – UK NGS
• Combination of jobs and legacy code services
enables P-GRADE Portal to support:
– Extending 2nd generation Grids with legacy code
services
6
Extending 2nd generation Grids with
legacy code services
NGS GT2
UoW Portal Server
User
GEMLCA
P-GRADE
Portal
Manchester
Executable
EGEE/VOCE
Poznan
Leeds
Executable
Job Repository
(GEMLCA)
Budapest
7
Grid interoperability 3
• Combination of jobs and legacy code services
enables P-GRADE Portal to support:
– Transition from 2nd generation Grids to 3rd
generation Grids:
• Extending 2nd generation production Grids with 3rd
generation volunteer Grid sites
• Connecting 2nd generation production Grids with 3rd
generation volunteer Grids
– Connecting 2nd generation and 3rd generation
production Grids
8
Extending 2nd generation Grids with
3rd generation volunteer Grid sites
UK NGS
GT2
UoW Portal Server
User
P-GRADE NGS
GEMLCA
Portal
Manchester
Executable
Executable
Westminster
Oxford
Executable
GEMLCA
Repository
Service
Invocation
UoW site – GT4 with
GEMLCA Service
9
Extending the NGS with legacy code services
and GT4 Resources
GT2 Job submission
to Rutherford
GT4 Service
Invocation at UoW
GEMLCA Legacy
code submitted
to Oxford
GT2 Job submission
to Manchester
10
Extending 2nd generation production Grids
with 3rd generation volunteer Grids
NGS GT2 Grid
UoW Portal Server
User
P-GRADE NGS
GEMLCA
Portal
Executable
Manchester
Leeds
Service
Invocation
WestFocus GT4
Grid
Executable
GEMLCA
Repository
UoW
Brunel
11
Connecting 2nd generation and
3rd generation production Grids
NGS GT2
UoW Portal Server
GEMLCA
P-GRADE
Portal
User
Manchester
Executable
EGEE LCG/gLite
Leeds
Service
InvocationWestFocus GT4
Grid
Executable
Poznan
Executable
Job Repository
(GEMLCA)
Budapest
UoW
Brunel
12
Plan: Connecting major Grids
by GEMLCA and the P-GRADE Portal
In test phase
Available
P-GRADE
GEMLCA
Portal
GEMLCA
Repository
13
Regional coverage of
using P-GRADE Portal
+ GILDA
For education
& training
14
Conclusion:
Easy-to-use, technology-neutral
Grid portal for e-Scientists
• The P-GRADE Portal hides differences of Grids
• Globus 2 and 4 – LCG2 – gLite Grid interoperability
• Transparent switching between Grid technologies
• Graphical tools for application development, execution and
monitoring
• Sequential & parallel components can be integrated into
large Grid applications
• Direct or broker-based resource allocation
• Your code does not have to contain grid specific calls
15
Collaborative work
support
16
Grid usage
Browser
Special client
Define
application
Grid portal
Grid
app.
Execute
application
Get
results
Portal
Grid middleware and
resources
Grid
17
The problem
• Grids have been designed for highly
complex applications (The overhead is too
big to execute small programs…)
• How can a single person define highly
complex applications?
18
The solution today…
E-mail, phone, skype, MSN, …
Browser
Special client
Define
application
Grid
app.
Execute
application
Get Portal
results
Grid
19
The solution today…
E-mail, phone, skype, MSN, …
Browser
Special client
How can I integrate the Define
application
knowledge of many people
Grid
app.
into a single and
consistent
Execute
Get Portal
application
entity in the shortest time?
results
Grid
20
Similar problems
have been already solved…
• Concurrent editing of Web pages:
WIKI, MediaWiki
• Concurrent editing of documents:
Online collaboration tool in MS Word
• Concurrent editing of tables:
Online collaboration tool in MS Excel
• …
21
The solution:
collaborative Grid portals
• Grid portals that support
– concurrent engineering of applications
– concurrent management of application execution
Grid
application
Portal
Grid
22
The Collaborative
P-GRADE Portal
Browser
Special client
edit
results
submit
Browser
Special client
edit
results
P-GRADE Portal
Grid
23
The Collaborative
P-GRADE Portal
No
Special client
time-consuming
edit
communication
…
Browser
results
submit
Browser
Special client
edit
…only
efficient,
workfloworiented
results
collaboration!
P-GRADE Portal
Grid
24
Collaborative Grid
workflows
• Integrate knowledge – development phase
• Generate and distribute results – execution phase
Collaborativ
e workflow
25
An e-Science example
Topology
of the
graph
Input protein
molecules
Collaborativ
e workflow
Protein
folding
algorithm
Experiment
results
26
Input for the
A business example
Graph
topology
market
prediction
application
Collaborativ
e workflow
Market
prediction
model
Results of
the
prediction
workflow
27
Collaborative workflow
development
– The portal has to guarantee that the result is a
consistent workflow
– Protect against data loss  locking mechanisms
– As real-time as possible!  to increase parallelism
during development
28
Workflow development
model
• Locking workflow components (jobs) exclusively
• First come first served
• Only the owner can edit the component
29
Workflow development
model
• Locking workflow components (jobs) exclusively
• First come first served
30
Workflow development
model
• Locking workflow components (jobs) exclusively
• First come first served
31
Screenshot
Locked
for me
Locked for
somebody
else
Unlocked
32
Requesting a lock
Workflow editor #1
Lock
request
Workflow editor #2
Portal server
Lock
accepted
33
Synchronizing the
different views
Workflow editor #1
Push updated
local view
Portal server
Workflow editor #2
Pull global view
34
Result of the collaborative
development
Job
Grid resource
data
data
Job
data
Job
Job
data
data
Job
data
Grid resource
Grid resource
Job
Grid resource
Grid resource
Grid resource
35
Executing collaborative
workflows
1. Concurrent control requests
It can happen in single-user case too
(multiple browsers are opened)
2. Different users’ proxies to be used to execute
different workflow nodes
Grid 1
P
Grid 2
1
P
P
3
2
Grid 3
36
Conclusions
• Extend the single-user version of the
P-GRADE Portal:
– Concurrent engineering of workflow-based
grid applications
– Application execution by multiple users in a
concurrent fashion
– Prototype version already available
37
Classification of
Grid portals
38
Two orthogonal aspects of Grid
portals usage
• Number of accessible Grids
– Single isolated Grid (SI)
– Multiple isolated Grids (MI)
– Multiple collaborative Grids (MC)
• Collaborative vice not collaborative usage
– Multiple isolated users (MI)
– Multiple collaborative users (MC)
39
Portal classification
Multiple Isolated
users
(MIxx)
Multiple Collaborative
users
(MCxx)
Single Isolated Grid
(xxSI)
MISI portals
(current portals)
MCSI portals
Multiple Isolated Grids
(xxMI)
MIMI portals
MCMI portals
Multiple Collaborative
Grids (xxMC)
MIMC portals
MCMC portals
40
MISI Portal: P-GRADE portal 1.0
LCG-2
P-GRADE-Portal
Rome
London
Athens
41
Portal classification
Multiple Isolated
users
(MIxx)
Multiple Collaborative
users
(MCxx)
Single Isolated Grid
(xxSI)
MISI portals
(current portals)
MCSI portals
Multiple Isolated Grids
(xxMI)
MIMI portals
MCMI portals
Multiple Collaborative
Grids (xxMC)
Multi-Grid portals
MIMC portals
MCMC portals
42
MIMI Portal: P-GRADE portal (2.1)
GridLab
P-GRADE-Portal
SEE-Grid
London
Rome
Athens
43
Portal classification
Multiple Isolated
users
(MIxx)
Multiple Collaborative
users
(MCxx)
Single Isolated Grid
(xxSI)
MISI portals
(current portals)
MCSI portals
Multiple Isolated Grids
(xxMI)
MIMI portals
MCMI portals
Multiple Collaborative
Grids (xxMC)
MIMC portals
MCMC portals
44
MIMC Portal: P-GRADE portal (2.1)
GridLab
P-GRADE-Portal
SEE-Grid
London
Rome
Athens
45
Portal classification
Multiple Isolated
users
(MIxx)
Multiple Collaborative
users
(MCxx)
Single Isolated Grid
(xxSI)
MISI portals
(current portals)
MCSI portals
Multiple Isolated Grids
(xxMI)
MIMI portals
MCMI portals
Multiple Collaborative
Grids (xxMC)
MIMC portals
MCMC portals
46
MCMC Portal: P-GRADE portal (3.0)
GridLab
P-GRADE-Portal
SEE-Grid
Rome
London
Athens
47
Summary of classification
P-GRADE 3.0+
P-GRADE 2.3
SI Grid
WF
manager
MI users WF GUI
.
.
.
MC users WF GUI
.
.
.
User
Grid
User
.
..
Grid
User
MI
Grids
WF
manager
MC
Grids
WF
manager
Grid
Grid
.
..
.
.
.
User
.
..
Grid
Grid
.
..
.
..
User
Grid
Grid
.
.
.
User
.
..
.
.
.
User
.
..
User
.
..
.
..
Grid
Grid
.
..
.
.
.
User
.
..
48
Conclusions
•
xxMC portals: execute a workflow simultaneously in many
Grids
P-GRADE portals 2.1 - 2.3 already work as xxMC portal
providing service for several production Grids:
•
–
–
–
–
HunGrid
SEE-Grid
VOCE
UK NGS
www.lpds.sztaki.hu/pgportal
•
MCxx portal: Enables to collaborate through workflow
applications!
•
Collaborative P-GRADE Portal: Prototype was demonstrated
at SC05 exhibition
49
Thank you for the attention!
Questions…
[email protected]
www.lpds.sztaki.hu
50