WNoDeS – a Grid/Cloud
Integration Framework
Elisabetta Ronchieri (INFN-CNAF)
for the WNoDeS Project
http://web.infn.it/wnodes
([email protected])
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
2
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
3
Requirements we started with
• Large computing centers used for a wide range of
applications by multiple user communities
– To determine proper and timely configuration of compute
resources without sacrificing efficiency, flexibility and
security
• User Communities
– To allow accessing resources by using a number of
different interfaces, authenticating via several methods,
exploiting existing local and distributed infrastructures
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
4
In short
• To define a multi-layer framework to optimize the
usage of computing center resources for multiple
communities
… starting from the INFN CNAF center (located in Bologna,
Italy) that
• currently hosts about 10,000 computing cores, and 9 PB of disk
space and 10 PB of tape space
• each day hosts about 50,000 jobs
• supports tens of international experiments of astro-particle
physics (e.g., AMS2, Argo, Auger, Fermi/Glast, Magic, Pamela,
Virgo) with about 15% of its computing resources
• supports the Italian Tier-1 for CERN-based LHC experiments of
high energy physics (i.e, ATLAS, CMS, LHCb, Alice) and Tier-0/1 for
several others
• adopts Platform LSF as batch system and IBM GPFS as shared File
System
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
5
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
6
The Worker Nodes on Demand
Service, a.k.a. WNoDeS, in one slide
•
Is a software framework created by INFN to integrate Grid and Cloud provisioning
through virtualization
– All resources (Grid, Cloud, or else) are taken from a common pool
•
Is scalable and reliable – it is in production at several Italian centers, including the
INFN Tier-1 (CNAF, Bologna) since November 2009
– Currently WNoDeS is managing about 2000 on‐demand Virtual Machines (VMs) there
•
•
Is totally transparent for Grid users and for users of traditional Computing Centers
batch systems
Supports a native Cloud interface
– OCCI (Open Cloud Computing Interface) compliant
– A Cloud Web portal
•
Integrates authentication, policy and accounting
It is not required to convert an entire farm to WNoDeS: it can coexist and share
resources with a traditional computing cluster not using virtual machines
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
7
Key Characteristics
•
Use of Linux KVM to virtualize resources on-demand that are available and
customized for
– direct job submissions by local users [1-2]
– Grid job submissions (with direct support for the EMI CREAM-CE and WMS
components) [1-2]
– instantiation of Cloud resources [6]
– instantiation of Virtual Interactive Pools (VIP) [3]
•
Use of LRMS (a “batch system software”) to handle VM scheduling
– There is no need to develop special (and possibly unscalable, inefficient) resource
brokering systems
– The LRMS is totally invisible to users for e.g. Cloud instantiations
•
Not Use of “Cloud over Grid” (or “Grid over Cloud”)’s concept to handle resources
– WNoDeS simply uses all resources and dynamically presents them to users as users
want to see and access them
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
8
Key Advantages
•
Use of a common pool of resources
– There is no need to dedicate resources to “user interfaces”, “Grid computing”, “Cloud
computing”, “local users”
– All types of resources are taken from a common pool, resulting in overall better
utilization of resources
•
•
Integrated support of old and new use cases (local access, Grid computing, Cloud
computing)
Re-use of ten years of worldwide development, expertise and resources brought
about by Grid Computing applied to the key areas of Authentication,
Authorization, Accounting, Information Systems, Brokering
– This will make it possible for example to inter-connect Clouds without starting from
scratch in these areas again
•
Flexibility and scalability
– At the core of WNoDeS there is a standard batch system used for resource provisioning
and policing a mature, stable piece of software found in any sizeable resource center
– There is no need to rewrite this key and complex part
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
9
Requirements
• Collected on the base of the needs of INFN
users and resource providers to virtualize
systems and services
• Published in the document
‘WNoDES: Requirements for
Virtualized Services’ [7]
– It is a continuous work in progress
due to the dynamic nature of
requirements and the constant
evolution of WNoDeS
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
10
Req. #1: Jobs submitted by users of a
traditional computing center
Users log on to a frontend system of a resource center
and submit jobs to a computing farm
• WNoDeS can associate jobs
belonging to a user (or to a set
of users) to VMs specifically
created for them and
customized to match the job
requirements [1,2]
Virtual
Virtual
Machin
Virtual
Machin
eMachin
e
e
– This is completely transparent
for users
WN
WN
– There is no need to change
WN
anything from the users’ point
WNoDeS enabled
of view
WNs
– The jobs will be executed on
dynamically-created VMs
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
Standard
Standard
job
Standard
job
job
WN
WN
WN
Non WNoDeS
WNs
11
Req. #2: Jobs submitted by users of a
Grid-based distributed infrastructure
• WNoDeS handles these jobs as an extension of the
Req. #1 simply taking into account special Grid
requirements if present [1,2]
• There are two possibilities here:
– All jobs beloning to certain Virtual Organizations (VOs) can
be directed to pre‐packaged VMs
• This is completely transparent for users
– Grid users can insert a special request into their JDL script
for example to specify which VM they want their jobs to
run on
• This is done by using standard EMI (European Middleware
Initiative) gLite job management tools
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
12
Req. #3: Self-allocation of systems by
users of a traditional computing center
Local users need direct and local access (or interactive
access) to the batch system or the distributed
infrastructure without requiring root access.
• WNoDeS provides a mode called Virtual Interactive Pool (VIP)
[3]
– Systems are provisioned so that users can log on with their local
account
– Users can specify characteristics of VM such as machine type, number
of CPUs, amout of RAM, local filesystems to be mounted
– This is a kind of cloud computing applied to a traditional computing
center designed to efficiently offer new services, without incurring the
overhead to dedicate resources for this purpose
– A caching mechanism is available to reduce provisioning time
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
13
Req. #4: Self-allocation of compute resources
by distributed users (Cloud Computing)
Distribute users like to autonomously selfallocate compute resources defined by several
parameters
• WNoDeS can deliver Cloud services via [6]
– A standard API called the Open Cloud Computing Interface
(OCCI), developed by the Open Grid Forum (OGF)
• Rarely employed directly by users
– A Cloud Web portal
• A user-friendly way to self-allocate resources
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
14
Req. #5: Resource access by distributed users
through multiple authentication methods
Users need to be authenticated whatever is the users’
authentication method
• WNoDeS can use any authentication method as long as it can
be mapped into a known and unique resource ownership [2]
– Ownership can be shared across different resource interfaces such as
Grid and Cloud
– Non-Grid users will be authenticated as well using Kerberos or
Shibboleth by the IGI Web Portal [6]
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
15
Status
• WNoDeS 1 is in production at the INFN Tier-1 and at other INFN sites in
Italy
– Focusing on providing virtualization services for local and Grid accesses
• WNoDeS 2 is planned for release in November 2011
– Introducing (among other things) the Cloud portal, the VIP interface and
support for multiple batch systems
• WNoDeS 2 installation and configuration also works with Torque and
Maui as Open Source Batch System (tested at INFN Bari)
– Using properties for queue, jobs and WN
• WNoDeS 2 configuration also works with Lustre as Open Source Shared
File System to serve data and distribute Virtual Node images (tested at
INFN Bari)
– Observed good performance and scalability
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
16
Developments
• Program focuses in particular on:
– Dynamic network virtualization
– VM image abstraction
– Efficient access to large storage systems
• Gateway NFS/GPFS
– Inter-cloud connectivity
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
17
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
18
WNoDeS Cloud Web Interface
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
19
Process Flow for the WNoDeS
Web-Based Cloud Management
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
20
Mixed Mode
• We are investigating the possibility to use the
same physical WN to execute both real jobs and
VM jobs
– This will increase the flexibility of the farm and allow
better optimization of the resource usage
• There is no need to “reserve” WN to WNoDeS
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
21
Mixed Mode
New WNoDeS
Old WNoDeS
Standard
Standard
job
job
Virtual
Virtual
Machine
Virtual
Machine
Machine
WN
WN
WN
WN
WNoDeS enabled WNs
WNoDeS jobs
Batch
Server
Manager
Standard
Standard
job
Standard
job
job
Virtual
Virtual
Machine
Machine
WN
Standard
job
Virtual
Machine
WN
WN
WN
WN
Non WNoDeS WNs
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
WNoDeS jobs
&
Standard jobs
Batch
Server
Manager
22
Software
• WNoDeS has been ported into ETICS during
summer
– To handle daily builds via local
remote build in order to get
repository RPMs
– To automate its deployment via
WNoDeS metapackages
– To adopt ETICS quality checks
– To automate internal test
procedures
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
23
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
24
WNoDeS and IGI
• IGI, the Italian Grid Infrastructure, plans to offer an
integrated computing infrastructure providing
– Grid computing services
– Cloud computing services
– Inter-connection of Cloud computing resource centres for
the Italian and research community
The WNoDeS framework is the production solution for
integrating Grid and Cloud infrastructure offered by
IGI
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
25
Overview
•
•
•
•
•
Problem Statement
WNoDeS as solution
New features and why
WNoDeS and IGI
Details
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
26
Information
• For further information and questions
– WNoDeS mailing list, [email protected]
• The WNoDeS Web site
– http://web.infn.it/wnodes
• Acknoledgements for this talk
– D.Salomoni (INFN CNAF), A. Italiano (INFN CNAF), G. Dalla Torre (INFN
CNAF), D. Andreotti (INFN CNAF), G. Donvito (INFN Bari), V. Spinoso
(INFN Bari)
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
27
References
[1] A. Italiano, et al, WNoDeS, a Tool for Integrated Grid/Cloud Aaccess and Computing
Farm Virtualization, CHEP 2010, Taipei
[2] D. Salomoni, et al, An Authentication Gateway for Integrated Grid and Cloud
Access, CHEP 2010, Taipei
[3] C. Grandi, et al, Virtual Pools for Interactive Analysis and Software Development
through an Integrated Cloud Environment, CHEP 2010, Taipei
[4] D. Salomoni, A. K. Calabrese Melcarne, A. Chierici, G. Dalla Torre, A. Italiano,
“Performance improvements in a large scale virtualization system”, PoS(ISGC 2011
& OGF 31)049
[5] D. Salomoni, D. Andreotti, L. Cestari, G. Potena, P. Solagna, “A Web-based portal to
access and manage WNoDeS Virtualized Cloud resources”, PoS(ISGC 2011 & OGF
31)054
[6] M. Bencivenni, et al, “A portal for an easy access to the IGI grid infrastructure”,
poster 9 at EGI TF, 19-23 September 2011, Lyon, France
[7] D. Salomoni, E. Ronchieri, “WNoDeS: Requirements for Virtualized Services”,
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
28
Thanks!
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
29
Backup Slides
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
30
Measurements
• Using Lustre shared file-system to distribute Virtual
Node images determines:
– Good performance
• Tested already at 3-5GByte/s on a farm with 700TB of storage
– Scalability
• Already tested with thousands of client and hundreds of file
servers
E. Ronchieri, EGI TF, 19-23 September 2011, Lyon, France
31