GENI Instrumentation and Measurement WG:
2nd I&M Workshop and
Priority Architecture Topics
GENI Engineering Conference 8
UC San Diego
GPO System Engineer: Harry Mussman
July 22, 2010
www.geni.net
Sponsored by the National Science Foundation
2nd I&M Workshop
• Objective:
– Gather contributors from key I&M prototyping projects that have
already implemented pieces of I&M functionality in a manner
consistent with GENI goals.
– Define priority pieces of the I&M architecture by consensus
– Assemble teams to complete the discussion and documentation
– Draft a roadmap for implementations during Spirals 2 and 3.
– See http://groups.geni.net/geni/wiki/2ndInstMeasWork#References
• Initial set of priority topics:
– GENI I&M use cases
– GENI I&M services
– GENI measurement plane
– Interfaces, protocols and schema for measurement data in GENI
Sponsored by the National Science Foundation
July 22, 2010
2
(continued)
• Approach:
– Each priority topic was discussed in a structured manner
– The organizers outlined an approach or solution, and representatives from
the key projects discussed it
– The group identified where they had a consensus, and where there were
issues that needed further discussion
– One new topic was identified: how to discover, specify, authorize and assign
all of the various types of resources required for I&M capabilities
– The interfaces topic was split into four topics, divided upon function
– Finally, teams were assembled to discuss each priority topic, come to a
consensus, and write a revised section for the architecture document
• Initial set of priority topics:
–
–
–
–
GENI I&M use cases
GENI I&M services
GENI measurement plane
Interfaces, protocols and schema for measurement data in GENI
Sponsored by the National Science Foundation
July 22, 2010
3
(continued)
• Organizing Committee:
–
–
–
–
–
Paul Barford - University of Wisconsin – Madison
Bruce Maggs – Duke University and Akamai
Harry Mussman – BBN/GPO
Vic Thomas - BBN/GPO
Evan Zhang – BBN/GPO
• Key projects who sent representatives:
–
–
–
–
–
–
–
–
OML (ORBIT Measurement Library) OMF (ORBIT Management Framework)
Instrumentation Tools
perfSONAR
Scalable Sensing Service (S3)
OnTimeMeasure for network measurements
GENI Meta-Operations Center and NetKArma
Virtual Machine Introspection (VMI)
Experiment Management Service – Digital Object Registry
Sponsored by the National Science Foundation
July 22, 2010
4
(continued)
• Final set of priority topics:
–
–
–
–
–
–
Topic 1: GENI I&M Use Cases
Topic 2: GENI I&M Services
Topic 3: GENI I&M Resources
Topic 4: GENI I&M Measurement Plane and Interfaces
Topic 5: GENI I&M Interfaces and Messages/Flows/APIs :
Topic 6: GENI I&M Interfaces and Messages/Flows/APIs :
Data File Transfers
– Topic 7: GENI I&M Interfaces and Messages/Flows/APIs :
Registration and Discovery
– Topic 8: GENI I&M Interfaces and Messages/Flows/APIs :
– Topic 9: GENI Measurement Data Schema
Sponsored by the National Science Foundation
July 22, 2010
Manage Services
Data Flows and
Service
GUIs
5
Topic 1: GENI I&M Use Cases
• Team members:
– Paul Barford - University of Wisconsin – Madison
– Jim Griffioen - Univ Kentucky
– Prasad Calyam* - Ohio Supercomputing Ctr
– Camilo Viecco - Indiana Univ
– Brian Hay – Univ Alaska
*agreed to organize first writing and discussion
• Identify all user groups, and provide basic use cases for each user group:
–
–
–
–
–
–
Experiment researchers
Experiment (opt-in) users
Central (i.e., GMOC) operators
Cluster and aggregate providers and operators
Archive service providers and operators
Researchers that use archived measurement data
Sponsored by the National Science Foundation
July 22, 2010
6
Topic 2: GENI I&M Services
• Team members:
– Harry Mussman* and Evan Zhang – BBN/GPO
– Giridhar Manepalli – CNRI
– Chris Small and Beth Plale - Indiana Univ
*agreed to organize first writing and discussion
• Summarize current view of services
–
–
–
–
–
–
Measurement Orchestration (MO) Service
Measurement Point (MP) Service
Measurement Information (MI) Service
Measurement Collection (MC) Service
Measurement Analysis and Presentation (MAP) Service
Measurement Data Archive (MDA) Service
• Need basic definition of a Measurement Data Archive (MDA) service
Sponsored by the National Science Foundation
July 22, 2010
7
(continued)
• Have identified three different types of services:
– Type 1: Dedicated service platform for customized Information. (Completely
dedicated to an experiment)
– Type 2: Common service platform with dedicated slivers for customized
Information. (Common portion, plus parts associated with different
experiments)
– Type 3: Common service for common or customized information. (Common
service, with data provided to multiple experiments)
Sponsored by the National Science Foundation
July 22, 2010
8
GENI I&M Architecture:
Services and Interfaces
Sponsored by the National Science Foundation
July 22, 2010
9
Topic 3: GENI I&M Resources
• Team members:
– Vic Thomas - BBN/GPO
– Jim Griffioen* - Univ Kentucky
– Martin Swany - Univ Delaware
– Camilo Viecco - Indiana Univ
– Brian Hay – Univ Alaska
– Giridhar Manepalli - CNRI
*agreed to organize first writing and discussion
• How are each of these resources discovered, specified, authorized and
assigned:
–
–
–
–
–
–
Hosts, VMs, etc.
Network connectivity
Software, e.g., I&M software that can be included in an experiment
I&M services
I&M data flows and file transfers
I&M data files stored in archives
Sponsored by the National Science Foundation
July 22, 2010
10
(continued)
• Options:
–
–
–
–
–
–
Always by mechanisms provided by the CF?
With CF plus additional mechanisms?
Consider example of LS in perfSONAR
Consider example of data file stored in archive, owned by an experimenter
Need to understand interop with CF for each option
Does CF setup secondary authorization mechanisms in some cases? If so,
how?
• For each item, consider
–
–
–
–
–
–
–
–
Unique name?
Unique and persistent identifier?
Ownership
What sort of policies the owner may want to apply
How to create
How to name
How to register and discover
How to authorize and assign
Sponsored by the National Science Foundation
July 22, 2010
11
(continued)
• Chart:
–
Sponsored by the National Science Foundation
July 22, 2010
12
Topic 4: GENI I&M Measurement Plane
and Interfaces
• Team members:
– Harry Mussman* – BBN/GPO
– Ezra Kissel – Univ Delaware
– Chris Small - Indiana Univ
*agreed to organize first writing and discussion
• Assume: Use IP backbone network (with Internet access) for most measurement
traffic (like control traffic)
• Consider: Use Layer 2 (VLAN) connections only when necessary
• Issues with measurement traffic:
– Which protocols are active? minimize number to simplify?
– Need access to resources in aggregates, even when resources are in private
address space, via proxies (i.e., gateways)
– Need to provide authentication and authorization, without and with proxies
– Need to be able to reserve bandwidth (provide QoS) to protect measurement
traffic from other traffic, and vice-versa
– Which of these requires the use of Layer 2 (VLAN) connections?
–
Sponsored by the National Science Foundation
July 22, 2010
13
GENI Measurement Traffic Flows
Sponsored by the National Science Foundation
July 22, 2010
14
GENI I&M Interfaces and Messages/Flows/APIs
• 1) Discover Resources and Assign Slivers
– EC srvc uses CF to discover resources, and then assign slivers to slice/researcher for
I&M srvc’s
• 2) Configure and Program Slivers
– EC srvc uses CF and/or ssh to load std or customized software images for I&M srvc’s
– Note: 1) and 2) are not specific to I&M services
• 3) Manage Services
– EC srvc and MO srvc use CF and/or https to check status of I&M srvcs, receive event
notifications, and execute functions such as start, stop, reset, reboot, and checkpoint
• 4) Measurement Data Flows and Data File Transfers
– Measurement data flows and data file transfers between I&M srvcs. Options: Pull,
Push, Pub/Sub.
• 5) Register I&M Service
– Operator configures I&M srvc to register with Lookup Srvc, advertising name, location,
and available metadata
• 6) Discover I&M Service and Establish Meas Data Flow
– ECS or I&M srvc discovers I&M srvc advertisement, and establishes data flow
• 7) Conduct and Observe Experiment
– Researcher uses browser to interact with and observe services via web portals (GUIs)
Sponsored by the National Science Foundation
July 22, 2010
15
GENI I&M Architecture:
Services and Interfaces
Sponsored by the National Science Foundation
July 22, 2010
16
Topic 5: GENI I&M Interfaces and Messages/Flows/APIs:
Manage Services
• Team members:
– Vic Thomas - BBN/GPO
– Ivan Seskar – Rutgers WINLAB
– Max Ott – NICTA
– Sonia Fahmy* – Purdue
– Giridhar Manepalli – CNRI
*agreed to organize first discussion and writing
• Define an approach based on OMF/OML and S3
• Consider:
–
–
–
–
–
HTTP(S)
REST vs SOAP
Authorization by credentials or ? If credentials, how to revoke?
Pass XML fragments
Define basic API
Sponsored by the National Science Foundation
July 22, 2010
17
Topic 6: GENI I&M Interfaces and Messages/Flows/APIs:
Data Flows and Data File Transfers
• Team members:
– Harry Mussman* – BBN/GPO
– Ivan Seskar – Rutgers WINLAB
– Max Ott – NICTA
– Ezra Kissel – Univ Delaware
– Prasad Calyam - Ohio Supercomputing Ctr
– Michael Zink - UMass Amherst
*agreed to organize first writing and discussion
• Consider data flows and data file transfers between all services
– Fundamental to I&M, and not yet defined in GENI
– Linked to measurement data schema, which is primary
Sponsored by the National Science Foundation
July 22, 2010
18
(continued)
• Consensus to allow a wide range of options:
– Both data flows and data files transfers
– Types: Pull, Push, Pub/Sub
– Protocols: SNMP, SCP, FTP, gridFTP, HTTP, XMPP, TCP, SCTP
– Consider: Naming, Discovery, Connectivity , Authentication and
authorization mechanisms
• Goal is to define an initial set:
– Minimum set required for GENI
– Mapped to current projects
– Permit later additions
Sponsored by the National Science Foundation
July 22, 2010
19
Topic 7: GENI I&M Interfaces and Messages/Flows/APIs:
Registration and Discovery of Services
with Available Measurement Data
• Team members:
– Jason Zurawski* – Internet2 (yes)
– Prasad Calyam - Ohio Supercomputing Ctr (yes)
*agreed to organize first writing and discussion
• Consider approach used in perfSONAR
• Summarize for:
– Services with available data flows
– Also sources of file transfers?
– Also GUIs?
Sponsored by the National Science Foundation
July 22, 2010
20
Topic 8: GENI I&M Interfaces and Messages/Flows/APIs:
GUIs
• Team members:
– Jeremy Reed and Zongming Fei - Univ Kentucky (yes)
– Guilherme Fernandes* – Univ Delaware (yes)
– Puneet Sharma - HP Labs (yes)
– *agreed to organize team
• Consider all types of GUIs:
– Control experiments
– Display I&M results
– Report status
– View archive service
• Define overall goals for GENI GUIs
• Consider portal, for access to multiple GUIs
• Consider need for authentication and authorization
Sponsored by the National Science Foundation
July 22, 2010
21
Topic 9 GENI Measurement Data Schema
• Team members:
– Bruce Maggs – Duke University and Akamai
– Max Ott – NICTA
– Ivan Seskar – Rutgers WINLAB
– Martin Swany* - Univ Delaware
– Camilo Viecco - Indiana Univ
– Michael Zink - UMass Amherst
– Jim French – CNRI
*agreed to organize first discussion and writing
• Consider:
– Measurement data schema
– Metadata schema
– Metadata contents
Sponsored by the National Science Foundation
July 22, 2010
22
(continued)
• Goal is to define an initial set:
– Minimum set required for GENI
– Mapped to current projects
– Permit later additions
• Consider measurement data schema and/or metadata schema from:
– perfSONAR
– GMOC-provided
– Current OML approach
– Proposed OML approach using IPFIX
– NetCDF (as used by DI Cloud)
• Provide overall template for GENI metadata:
– Which items in GENI metadata template are required?
– Invariant?
Sponsored by the National Science Foundation
July 22, 2010
23
GENI I&M Architecture Priority Topics:
Current Status
Topic
Group
discussion
First writing
WG review
Topic 1: GENI I&M Use Cases
Topic 2: GENI I&M Services
Topic 3: GENI I&M Resources
Topic 4: GENI I&M Measurement Plane and
Interfaces
Topic 5: GENI I&M Interfaces and
Messages/Flows/APIs : Manage Services
Topic 6: GENI I&M Interfaces and
Messages/Flows/APIs : Data Flows and Data
File Transfers
Topic 7: GENI I&M Interfaces and
Messages/Flows/APIs : Service Registration
and Discovery
Topic 8: GENI I&M Interfaces and
Messages/Flows/APIs : GUIs
Topic 9:
GENI Measurement Data Schema
Sponsored by the National Science Foundation
July 22, 2010
24
GENI I&M Architecture Document:
Current Status
• Status:
– By GEC7 : v0.1 DRAFT completed, by GPO; see
http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurem
entsArchitecture
– By GEC8: v0.5 DRAFT completed, by GPO, with contributions from
WG; see
• Plan:
– By GEC9: v1.0 draft, reviewed by WG
Sponsored by the National Science Foundation
July 22, 2010
25
GENI I&M Architecture Priority Topics and Document:
Next Steps
• Complete group discussion and first writing for each priority topic:
– When?
– Add members to each group?
• Review each priority topic with WG, and rewrite:
– How?
– When?
• Assemble and review revised Architecture Document as a whole
– How?
– Before GEC9
• Draft a roadmap for implementations during Spiral 3.
Sponsored by the National Science Foundation
July 22, 2010
26