XSEDE: Extreme Science and Engineering
Discovery Environment
Program Plan for Program Year 6
13 May 2016
i
XSEDE Senior Management Team
John Towns (NCSA)
PI and Project Director
Kelly Gaither (TACC)
Co-PI and Campus Engagement and Enrichment Director
Ralph Roskies (PSC)
Co-PI and Extended Collaborative Support Service Co-Director
Nancy Wilkins-Diehr (SDSC)
Co-PI and Extended Collaborative Support Service Co-Director
Greg Peterson (UT-NICS)
XSEDE Operations Director
David Hart (NCAR)
Resource Allocations Service Director
David Likfa (Cornell)
XSEDE Community Infrastructure Director
Ron Payne (NCSA)
Program Office Director
Karla Gendler (TACC)
Senior Project Manager
Emre Brookes (UT-San
Antonio)
Chair of the User Advisory Committee (voting member of the
SMT)
Dan Stanzione (TACC)
Chair of the XD Service Providers Forum (voting member of the
SMT)
ii
Table of Contents
Foreword ...................................................................................................................................................vi
1 Executive Summary ......................................................................................................................... 7
1.1
Strategic Goals ........................................................................................................................................... 7
1.2
How to Read this Program Plan ......................................................................................................... 8
2 Discussion of Strategic Goals and Key Performance Indicators ................................... 10
2.1
Deepen and Extend Use ...................................................................................................................... 10
2.1.1
Deepen Use to Existing Communities ................................................................................. 10
2.1.2
Extend Use to New Communities ......................................................................................... 10
2.1.3
Prepare the Current and Next Generation ........................................................................ 11
2.1.4
Raise Awareness of the value of advanced digital services ....................................... 11
2.2
Advance the Ecosystem ...................................................................................................................... 12
2.2.1
Create an Open and Evolving e-Infrastructure ............................................................... 12
2.2.2
Enhance the Array of Technical Expertise and Support Services ........................... 12
2.3
Sustain the Ecosystem ........................................................................................................................ 13
2.3.1
Provide Reliable, Efficient, and Secure Infrastructure................................................. 13
2.3.2
Provide Excellent User Support ............................................................................................ 13
2.3.3
Operate an Effective and Productive Virtual Organization........................................ 14
2.3.4
Operate an Innovative Virtual Organization .................................................................... 14
3 Community Engagement & Enrichment (WBS 2.1) ........................................................... 16
3.1
CEE Director’s Office (WBS 2.1.1) .................................................................................................. 17
3.2
Workforce Development (WBS 2.1.2) .......................................................................................... 17
3.3
User Engagement (WBS 2.1.3)......................................................................................................... 19
3.4
Broadening Participation (WBS 2.1.4) ......................................................................................... 21
3.5
User Interfaces & Online Information (WBS 2.1.5)................................................................. 22
3.6
Campus Engagement (WBS 2.1.6).................................................................................................. 23
4 Extended Collaborative Support Services (WBS 2.2) ....................................................... 27
4.1
ECSS Director’s Office (WBS 2.2.1) ................................................................................................ 28
iii
4.2
Extended Support for Research Teams (WBS 2.2.2) .............................................................. 28
4.3
Novel and Innovative Projects (WBS 2.2.3) ............................................................................... 30
4.4
Extended Support for Community Codes (WBS 2.2.4) .......................................................... 31
4.5
Extended Support for Science Gateways (WBS 2.2.5) ........................................................... 32
4.6
Extended Support for Education, Outreach, & Training (WBS 2.2.6).............................. 33
5 XSEDE Community Infrastructure (WBS 2.3) ...................................................................... 36
5.1
XCI Director’s Office (WBS 2.3.1) ................................................................................................... 37
5.2
Requirements Analysis & Capability Delivery (WBS 2.3.2)................................................. 38
5.3
XSEDE Capability & Resource Integration (WBS 2.3.3) ........................................................ 39
6 XSEDE Operations (WBS 2.4) ..................................................................................................... 42
6.1
Operations Director’s Office (WBS 2.4.1).................................................................................... 43
6.2
Cybersecurity (WBS 2.4.2) ................................................................................................................ 43
6.3
Data Transfer Services (WBS 2.4.3) .............................................................................................. 45
6.4
XSEDE Operations Center (WBS 2.4.4) ........................................................................................ 47
6.5
System Operations Support (WBS 2.4.5) .................................................................................... 48
7 Resource Allocation Service (WBS 2.5) ................................................................................. 50
7.1
RAS Director’s Office (WBS 2.5.1) .................................................................................................. 50
7.2
XSEDE Allocations Process & Policies (WBS 2.5.2)................................................................. 51
7.3
Allocations, Accounting, & Account Management CI (WBS 2.5.3) .................................... 52
8 Program Office (WBS 2.6) ........................................................................................................... 54
8.1
Project Office (WBS 2.6.1) ................................................................................................................. 55
8.2
External Relations (WBS 2.6.2) ....................................................................................................... 56
8.3
Project Management, Reporting, & Risk Management (WBS 2.6.3) ................................ 57
8.4
Business Operations (WBS 2.6.4)................................................................................................... 59
8.5
Strategic Planning, Policy & Evaluation (WBS 2.6.5) ............................................................. 59
A Glossary and List of Acronyms .................................................................................................. 62
iv
Table of Tables
Table 1-1: Summary of key performance indicators (KPIs) for XSEDE. ........................................................... 9
Table 2-1: KPIs for the sub-goal of deepen use (existing communities). ....................................................... 10
Table 2-2: KPIs for the sub-goal of extend use (new communities)................................................................. 11
Table 2-3: KPIs for the sub-goal of prepare the current and next generation. ............................................ 11
Table 2-4: KPIs for the sub-goal of raise awareness of the value of advanced digital services. ........... 12
Table 2-5: KPIs for the sub-goal of create an open and evolving e-infrastructure..................................... 12
Table 2-6: KPIs for the sub-goal of enhance the array of technical expertise and support services. . 13
Table 2-7: KPIs for the sub-goal of provide reliable, efficient, and secure infrastructure. ..................... 13
Table 2-8: KPIs for the sub-goal of provide excellent user support. ................................................................ 14
Table 2-9: KPIs for the sub-goal of operate an effective and productive virtual organization. ............ 14
Table 2-10: KPIs for the sub-goal of operate an innovative organization. .................................................... 15
Table 3-1: Area Metrics for Community Engagement & Enrichment .............................................................. 16
Table 3-2: Area Metrics for Workforce Development. ........................................................................................... 18
Table 3-3: Area Metrics for User Engagement. ......................................................................................................... 20
Table 3-4: Area Metrics for Broadening Participation ........................................................................................... 21
Table 3-5: Area Metrics for User Interfaces & Online Information................................................................... 23
Table 3-6: Area Metrics for Campus Engagement.................................................................................................... 24
Table 4-1: Area Metrics for Extended Collaborative Support Services ........................................................... 27
Table 4-2: Area Metrics for Extended Support for Research Teams. ............................................................... 29
Table 4-3: Area Metrics for Novel and Innovative Projects ................................................................................. 31
Table 4-4: Area Metrics for Extended Support for Community Codes ............................................................ 32
Table 4-5: Area Metrics for Extended Support for Science Gateways ............................................................. 33
Table 4-6: Area Metrics for Extended Support for Education, Outreach, & Training ............................... 34
Table 5-1: Area Metrics for XSEDE Community Infrastructure (XCI).............................................................. 36
Table 5-2: Area Metrics for Requirements Analysis & Capability Delivery................................................... 38
Table 5-3: Area Metrics for XSEDE Capability & Resource Integration .......................................................... 40
Table 6-1: Area Metrics for Operations. ....................................................................................................................... 42
Table 6-2: Area Metrics for Cybersecurity .................................................................................................................. 43
Table 6-3: Area Metrics for Data Transfer Services ................................................................................................ 45
Table 6-4: Area Metrics for XSEDE Operations Center .......................................................................................... 47
Table 6-5: Area Metrics for System Operations Support ...................................................................................... 48
Table 7-1: Area Metrics for Resource Allocation Service...................................................................................... 50
Table 7-2: Area Metrics for XSEDE Allocations Process & Policies................................................................... 51
Table 7-3: Area Metrics for Allocations, Accounting, & Account Management CI ...................................... 52
Table 8-1: Area Metrics for Program Office ................................................................................................................ 55
Table 8-2: Area Metrics for External Relations ......................................................................................................... 56
Table 8-3: Area Metrics for Project Management, Reporting, & Risk Management .................................. 58
Table 8-4: Area Metrics for Business Operations..................................................................................................... 59
Table 8-5: Area Metrics Strategic Planning, Policy & Evaluation ...................................................................... 60
v
Foreword
This program plan for the sixth program year of XSEDE (PY6) is the result of ongoing planning
efforts to assure project activities are in alignment with XSEDE’s three strategic goals and designed
to deliver on XSEDE’s mission and realize XSEDE’s vision. The PY 6 Program Plan follows the goalcentric format to crisply articulate XSEDE’s vision, mission, goals, and associated key performance
indicators (KPIs). The Senior Management Team works with internal and external stakeholders and
with third-party experts to review our vision, mission, and goals and to assess the associated
planned activities and definitions of success. The program plan presented here is a direct product of
this process. As a large, complex, highly-distributed project, this is critical to help XSEDE improve as
an organization and as a provider of services to the compute- and data-enabled science and
engineering research and education community.
This process has also driven a metrics-based approach to reporting on activities of the project..
Based on feedback from XSEDE’s review panels, advisory groups and the community, XSEDE has
continues to refine the KPIs to be metrics that measure progress toward the high-level strategic
goals. The key concept is not that the metrics [KPIs] themselves have a direct causal effect on
eventual outcomes but rather that the metrics are chosen so that actions and decisions which move
the metrics in the desired direction also move the organization in the direction of the desired
outcomes, goals and impact. All other metrics presented are termed Area Metrics. This is intended
to focus the attention of external stakeholders on what is believed to be the best (key) indicators of
progress toward XSEDE’s long-term strategic goals.
The Executive Summary (§1) is intended to effectively and concisely communicate the vision,
mission and three strategic goals of XSEDE and state the project-wide KPIs
A discussion of goals and key performance indicators (§2) provides the next level of detail in
understanding project progress. It decomposes the strategic goals into sub-goals and discusses how
progress is measured toward each of the sub-goals using KPIs that represent measures of impact to
the scientific community where possible.
These first two sections take a project-wide view. A more detailed description of plans from the
view of the areas responsible for supporting each of the sub-goals—and contributing toward the
KPIs associated with those sub-goals—is provided by looking at each of the areas of the project in
the remaining sections (§3, §4, §5, §6, §7, §8). These sections highlight new activities, ongoing
activities and discontinued activities for PY6 and additional area metrics that the area’s staff has
deemed important.
It is anticipated that this document is read in electronic form (PDF) using Adobe Reader®. There is
extensive cross-linking to facilitate referencing content across the document. In general, all text that
has blue underlining (e.g. §1.1) is clickable. Clicking on the underlined text will take you to the
referenced section. These are set up to facilitate moving back and forth between the high level
discussions in §1 and §2, to more detailed discussions regarding specific project areas in §3, §4, §5,
§6, §7, and §8.
vi
1 Executive Summary
Computing in science and engineering is now ubiquitous: digital technologies underpin, accelerate,
and enable new, even transformational, research in all domains. Researchers continue to integrate
an increasingly diverse set of distributed resources and instruments directly into their research and
educational pursuits. Access to an array of integrated and well-supported high-end digital services
is critical for the advancement of knowledge. XSEDE (the Extreme Science and Engineering
Discovery Environment) is a socio-technical platform that integrates and coordinates advanced
digital services within the national ecosystem to support contemporary science. This ecosystem
involves a highly-distributed, yet integrated and coordinated, assemblage of software,
supercomputers, visualization systems, storage systems, networks, portals and gateways,
collections of data, instruments and personnel with specific expertise. XSEDE fulfills the need for an
advanced digital services ecosystem distributed beyond the scope of a single institution and
provides a long-term platform to empower modern science and engineering research and
education. As a significant contributor to this ecosystem, driven by the needs of the open research
community, XSEDE substantially enhances the productivity of a growing community of scholars,
researchers, and engineers. XSEDE federates with other high-end facilities and with campus-based
resources, serving as the foundation for a national e-science infrastructure with tremendous
potential for enabling new advancements in research and education. Our vision is a world of
digitally-enabled scholars, researchers, and engineers participating in multidisciplinary collaborations
while seamlessly accessing computing resources and sharing data to tackle society’s grand challenges.
Researchers use advanced digital resources and services every day to expand their understanding
of our world. More pointedly, research now requires more than just supercomputers, and XSEDE
represents a step toward a more comprehensive and cohesive set of advanced digital services
through our mission: to substantially enhance the productivity of a growing community of scholars,
researchers, and engineers through access to advanced digital services that support open research;
and to coordinate and add significant value to the leading cyberinfrastructure resources funded by the
NSF and other agencies. XSEDE has developed its strategic goals in a manner consistent with NSF’s
strategies stated broadly in the Cyberinfrastructure Framework for 21st Century Science and
Engineering1 vision document, and the more specifically relevant Advanced Computing
Infrastructure: Vision and Strategic Plan2 document.
1.1
Strategic Goals
To support XSEDE’s mission and to guide the project’s activities toward the realization of that
vision, three strategic goals are defined:
Deepen and Extend Use: XSEDE will deepen the use—make more effective use—of the
advanced digital services ecosystem by existing scholars, researchers, and engineers, and
extend the use to new communities. XSEDE will contribute to preparation—workforce
development—of the current and next generation of scholars, researchers, and engineers in
1
2
http://www.nsf.gov/cise/aci/reorgLandingPage.jsp?p=/od/oci/cif21/CIF21Vision2012current.pdf
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12051
7
the use of advanced digital services via training, education, and student preparation; and
XSEDE will raise the general awareness of the value of advanced digital services.
Advance the Ecosystem: Exploiting its internal efforts and drawing on those of others, XSEDE
will advance the broader ecosystem of advanced digital services by creating an open and
evolving e-infrastructure, and by enhancing the array of technical expertise and support
services offered.
Sustain the Ecosystem: XSEDE will sustain the advanced digital services ecosystem by assuring
and maintaining a reliable, efficient, and secure infrastructure, and providing excellent user
support services. XSEDE will further operate an effective, productive, and innovative virtual
organization.
The strategic goals of XSEDE cover a considerable scope. To assure XSEDE is delivering on it’s
mission and to assess progress toward XSEDE’s vision, key metrics have been identified to measure
progress toward meeting each sub-goal. These key performance indicators (KPIs) are a high-level
encapsulation of the project metrics that measure how well XSEDE is meeting each sub-goal.
Planning is driven by XSEDE’s vision, mission, goals and these metrics, which are in turn rooted in
the needs and requirements of the communities XSEDE serve.
The key concept is not that the KPIs themselves must have a direct causal effect on eventual
outcomes, or measure eventual outcomes or long-term impacts, but rather that the KPIs are
chosen so that actions and decisions which move the metrics in the desired direction also move
the organization in the direction of the desired outcomes and goals.
Table 1-1 below summarizes the project’s three strategic goals and associated sub-goals along with
the KPIs used to measure progress toward achieving those goals.
1.2 How to Read this Program Plan
In this program plan, XSEDE continues to refine both the language relating to metrics and the
metrics themselves as the organizational understanding continues to grow. In response to feedback
and as a result of the annual review of processes and standards, careful readers may note two
important changes intended to focus attention on the true best indicators of performance: (a) The
term KPI only refers to indicators of progress for the project’s strategic goals and key
measurements within areas of the project are termed Area Metrics; and (b) Significant effort has
gone into simplifying complex measurements at the project-level. Where once composite
measurements were used that contained results from many areas of the project (often of varying
scale, cost, or composition), XSEDE has carefully chosen what is felt to be the best available
indicator of progress of the project in the related area.
The document contains many links between sections allowing the reader to “drill down” to
increasingly detailed information or to traverse upward for the organization-wide perspective. In
general, §1 and §2 have high-level information about overall strategy of the organization and
important highlights. In §2 each of the three strategic goals are examined and the associated subgoals and KPIs are mapped. In §3, §4, §5, §6, §7, and §8, the planned activities for PY6 that support
the overall project goals are described.
8
Table 1-1: Summary of key performance indicators (KPIs) for XSEDE.
Strategic
Goals
Sub-goals
KPIs
Deepen and Extend Use (§2.1)
Deepen use (existing
communities) (§2.1.1)
 Number of completed ECSS projects
 Average ECSS impact rating
 Average satisfaction with ECSS support
Extend use (new
communities) (§0)
 Number of new users from underrepresented
communities and non-traditional disciplines of XSEDE
resources and services
 Number of sustained users from underrepresented
communities and non-traditional disciplines of XSEDE
resources and services
Prepare the current and next
generation (§2.1.3)
 Number of participants in synchronous and
asynchronous training
 Average impact assessment of training for participants
registered through the XSEDE User Portal
Raise awareness of the value
of advanced digital services
(§2.1.4)




Number of unique visitors to the website
Number of unique visitors to the XSEDE User Portal
Number of social media impressions
Number of media hits
Advance the Ecosystem (§2.2)
Create an open and evolving
e-infrastructure (§2.2.1)
 Number of new capabilities made available for
production deployment
 Average satisfaction rating of XCI services
Enhance the array of technical
expertise and support
services (§2.2.2)
 Average rating of staff regarding how well-prepared
they feel to perform their jobs
Sustain the Ecosystem (§2.3)
Provide reliable, efficient, and
secure infrastructure (§2.3.1)
 Average composite availability of core services
 Hours of downtime with direct user impacts from an
XSEDE security incident
Provide excellent user
support (§2.3.2)
 Mean time to ticket resolution
 Average user satisfaction rating for allocations and
other support services
Operate an effective and
productive virtual
organization (§2.3.3)
 Percentage of recommendations addressed by
relevant project areas
Operate an innovative virtual
organization (§2.3.4)
 Number of strategic or innovative improvements
 Ratio of proactive to reactive improvements
 Number of staff publications
9
2 Discussion of Strategic Goals and Key Performance Indicators
The strategic goals of XSEDE (§1.1) cover a considerable scope. Additionally, the specific activities
within this scope are often very detailed; therefore, to ensure that this significant and detailed
scope will ultimately deliver XSEDE’s mission and realize its vision, the three strategic goals are
decomposed into components or sub-goals to be considered individually.
In determining the best measures of progress toward each of the sub-goals, KPIs that correlate to
impact on the scientific community are used. These often pair measurements of outcome with an
assessment of quality or impact to provide both a sense of scope and significance of the supporting
activities. In some cases, metrics for impact or outcome not currently being collected. In these cases,
the best available metric is used.
2.1 Deepen and Extend Use
XSEDE will 1) deepen the use—make more effective use—of the advanced digital services ecosystem
by existing scholars, researchers, and engineers and 2) extend the use to new communities. XSEDE
will contribute to preparation—workforce development—of scholars, researchers, and engineers in
the use of advanced digital technologies via training, education, and student preparation; and
XSEDE will 4) raise the general awareness of the value of advanced digital research services.
2.1.1
Deepen Use to Existing Communities
Although efforts to identify new technologies and new service providers along with efforts to
evolve the e-infrastructure and enhance the research prowess of current and future researchers all
serve to deepen use, the collaborative, yearlong work done to help research teams more effectively
and broadly use the ecosystem is the best indicator of deeper use. These efforts enable increased
scale and efficiency of usage and allow use of new capabilities for the delivery of science. The
project has chosen three metrics (Table 2-1) that together measure the scope, quality, and impact of
these activities: 1) the total number of projects completed by the Extended Collaborative Support
Services (ECSS) team through work with research teams, community codes, and gateways and the
average ratings for 2) satisfaction with and 3) overall impact from the ECSS work. Satisfaction and
overall impact are scores provided by the PIs of the projects on a 1-5 scale following project
completion.
Table 2-1: KPIs for the sub-goal of deepen use (existing communities).
KPI
PY6
Target
Owner(s)
Number of completed ECSS projects
Average ECSS impact rating
Average satisfaction with ECSS support
50
4 of 5
4.5 of 5
ECSS (§4)
ECSS (§4)
ECSS (§4)
2.1.2
Extend Use to New Communities
New communities are defined as new fields of science, industry, and under-represented
communities. New fields of science are those that represent less than 1% of XSEDE Resource
Allocation Committee (XRAC) allocations. The Novel & Innovative Projects (NIP) team and the
10
Broadening Participation team both work to bring advanced digital services to new communities.
XSEDE measures both the number of new users and the number of sustained users on research
projects from underrepresented communities and non-traditional disciplines of XSEDE resources
and services as the indicators of progress (Table 2-2).
Table 2-2: KPIs for the sub-goal of extend use (new communities).
PY6
Target
KPI
Owner(s)
Number of new users from underrepresented
communities and non-traditional disciplines of
XSEDE resources and services
>200
ECSS – NIP (§4.3)
CEE – Broadening Participation (§3.4)
Number of sustained users from underrepresented
communities and non-traditional disciplines of
XSEDE resources and services
>1100
ECSS – NIP (§4.3)
CEE – Broadening Participation (§3.4)
2.1.3
Prepare the Current and Next Generation
Part of XSEDE’s mission is to provide a broad community of existing and future researchers with
access and training to use advanced digital services via the sub-goal of preparing the current and
next generation of computationally savvy researchers. While many activities support this sub-goal,
such as the various Champion (§3.6), Student Engagement (§3.4), and Education (§3.2) programs,
the training offered through Community Engagement & Enhancement (CEE) impacts the most
people directly. This and a complementary measure of impact as indicated by those same
individuals, are therefore considered the key indicators (Table 2-3) of performance toward this
goal.
Table 2-3: KPIs for the sub-goal of prepare the current and next generation.
PY6
Target
KPI
Number of participants in synchronous and
asynchronous training
Average impact assessment of training for participants
registered through the XSEDE User Portal
2.1.4
Owner(s)
5,600
CEE – Workforce Development (§3.2)
4 of 5
CEE – Workforce Development (§3.2)
Raise Awareness of the value of advanced digital services
While many PY6 activities, such as Workforce Development (§3.2), User Engagement (§3.3) and
Broadening Participation (§3.4) efforts, and the visibility of the Champions and other Campus
Engagement (§3.6) activities, contribute to XSEDE’s ability to raise the general awareness of the
value of advanced digital research services, XSEDE looks at outcomes in three areas (Table 2-4):
website, social media, and public relations. Desirable trends in these key outcomes can be
correlated to success for this sub-goal.
11
Table 2-4: KPIs for the sub-goal of raise awareness of the value of advanced digital services.
KPI
PY6
Target
Owner(s)
Number of unique visitors to the website
Number of unique visitors to the XSEDE User Portal
Number of social media impressions
Number of media hits
80,000
15,000
190,000
140
CEE - UII (§3.5)
CEE - UII (§3.5)
Program Office - ER (§8.2)
Program Office - ER (§8.2)
2.2 Advance the Ecosystem
Exploiting its internal efforts and drawing on those of others, XSEDE will advance the broader
ecosystem of advanced digital services by 1) creating an open and evolving e-infrastructure, and by
2) enhancing the array of technical expertise and support services offered.
2.2.1
Create an Open and Evolving e-Infrastructure
External factors, such as the number of XSEDE Federation members and the variety of services they
provide alongside internal efforts, such as operations (§6) of critical infrastructure and services and
the evaluation and integration of new capabilities, all affect the evolution of the e-infrastructure.
While XSEDE actively seeks new Federation members and Service Providers, and partnerships with
national and international cyberinfrastructure projects, XSEDE views its role as connector of these
elements to be the most impactful. Thus, XSEDE focuses on the number of new capabilities made
available for production deployment along with the satisfaction rating of XSEDE Community
Infrastructure (XCI) services as indicators of performance with respect to this sub-goal (Table 2-5).
Table 2-5: KPIs for the sub-goal of create an open and evolving e-infrastructure.
PY6
Target
KPI
Number of new capabilities made available for production
deployment
Average satisfaction rating of XCI services
2.2.2
Owner(s)
10
XCI (§5)
4 of 5
XCI (§5)
Enhance the Array of Technical Expertise and Support Services
To enhance the technical expertise of XSEDE’s staff to offer an evolving set of support services,
XSEDE will continue many activities including workshops, symposia, and training events hosted by
Extended Collaborative Support Services (ECSS) and Service Providers (§4.6). The average rating
by staff for training will continue to be a KPI although the staff climate survey question that
measures this has been changed to focus on staff having the training they need, instead of staff
turning to XSEDE training to fulfill these needs (Table 2-6). This represents a change in approach to
staff training that reflects budget constraints, where XSEDE will leverage existing training offered
by the Service Providers, universities, professional associations and the broader community
alongside its own training to enhance the expertise of staff.
12
Table 2-6: KPIs for the sub-goal of enhance the array of technical expertise and support services.
KPI
PY6 Target
Average rating of staff regarding how well-prepared they feel
to perform their jobs
4 of 5
Owner(s)
Program Office - Evaluation
(§8.5)
2.3 Sustain the Ecosystem
XSEDE will sustain the advanced digital services ecosystem by 1) assuring and maintaining a
reliable, efficient, and secure infrastructure, and 2) providing excellent user support services.
Furthermore, XSEDE will operate an 3) effective, 4) productive, and 5) innovative virtual
organization.
2.3.1
Provide Reliable, Efficient, and Secure Infrastructure
Many activities support this sub-goal—such as User Information & Interfaces (§3.5), Security
(§6.2), Data Transfer Services (§6.3), Systems Operations and Support (§6.5), support for
Allocations (§7.2), and Accounting & Account Management (§7.3)— but perhaps the truest measure
of an infrastructure’s reliability is its robustness as reflected by sustained availability. Thus the KPIs
for this sub-goal is a composite measure of the availability of XSEDE infrastructure components and
the number of hours of downtime with direct user impacts from a security incident (Table 2-7). The
composite measure is a geometric mean of the availability of critical enterprise services and the
XRAS account management service.
Table 2-7: KPIs for the sub-goal of provide reliable, efficient, and secure infrastructure.
KPI
PY6 Target
Average composite availability of core services
99%
Hours of downtime with direct user impacts from an
XSEDE security incident
< 24
2.3.2
Owner(s)
XSEDE Operations (§6)
RAS (§7)
Cybersecurity (§6.2)
Provide Excellent User Support
Although nearly every group in the organization has some support function, XSDE has chosen to
focus on metrics with respect to two primary support interfaces to the community: the XSEDE
Operations Center (XOC) metrics with respect to ticket resolutions times and the average
satisfaction rating of the allocations process. The XOC is the frontline centralized support group that
either resolves or escalates tickets to the appropriate resolution center depending on the request.
Two KPIs are measured (Table 2-8): the mean time to resolution on support tickets that are
resolved by the XOC or routed to, and resolved by, other XSEDE areas, and the average satisfaction
rating for the allocations process measured via a quarterly survey of users who have interacted
with the allocations request system and the allocations process more generally.
13
Table 2-8: KPIs for the sub-goal of provide excellent user support.
KPI
PY6 Target
Mean time to ticket resolution (hours)
User satisfaction with allocations process
2.3.3
< 24
4 of 5
Owner(s)
XSEDE Operations (§6)
RAS (§7)
Operate an Effective and Productive Virtual Organization
During the first five years of XSEDE, in conjunction with developing a methodology for driving and
assessing performance excellence, XSEDE has adapted the Baldrige Criteria3 to the XSEDE context
and has assessed and applied applicable criteria from all seven criteria by that methodology. These
include annual reviews of the vision, mission, strategic goals, project-wide processes and standards
(KPIs and area metrics); user and staff surveys (§3.3, §8.5); stakeholder communications (§8.2);
advisory boards (§8.1); community engagement (§3); workforce enhancement (§3.2); and the
analysis of organizational data that leads to organizational learning, strategic improvement and
innovation. Thus, XSEDE has chosen to monitor its continuous improvement efforts as an
organizational measurement of recommendations addressed by the relevant project areas (Table
2-9). The idea here is that a stable or expanding number of improvements shows that XSEDE
continues to systematically evaluate the organization and make informed, proactive improvements.
Table 2-9: KPIs for the sub-goal of operate an effective and productive virtual organization.
PY6
Target
KPI
Percentage of recommendations addressed by
relevant project areas
2.3.4
90%
Owner(s)
Project Office - Evaluation (§8.5)
Operate an Innovative Virtual Organization
Measuring innovation for an organization like XSEDE (or for organizations in general) is difficult
and represents an area of open research. A partial measure is the number of staff publications
produced since this shows that XSEDE staff is involved in novel activities that achieve peerreviewed publication. Additionally, after much thought and discussion both internally and with
external stakeholders and advisors, XSEDE has identified two additional indicators that strongly
correlate to innovation: 1) the ratio of proactive organizational improvements to those that were
reactive and 2) the number of improvements that are innovative or lead to innovations. The first
indicator is a measurement of organizational maturity and agility; the second measures innovative
actions directly (Table 2-10). While these provide some indication of innovation, they are still not
satisfactory. These KPIs will continue to be the subject of an open conversation within the
organization and with stakeholders and advisors as XSEDE assesses these measurements and how
3
http://www.nist.gov/baldrige/
14
to best quantify innovation. In particular, this will be discussed in earnest within the Strategic
Planning, Policy and Evaluation team (§8.5).
Table 2-10: KPIs for the sub-goal of operate an innovative organization.
KPI
PY6 Target
Number of strategic or innovative
improvements
Ratio of proactive to reactive improvements
9
Number of staff publications
70
3
Owner(s)
Program Office - Project Management &
Reporting (§8.3)
Program Office - Project Management &
Reporting (§8.3)
Project Office (§8)
15
3 Community Engagement & Enrichment (WBS 2.1)
At the core of Community Engagement & Enrichment (CEE) is the user, broadly defined to include
anyone who uses or may potentially use the array of resources and services offered by XSEDE. The
CEE team is dedicated to actively engaging a broad and diverse cross-section of the open science
community, bringing together those interested in using, integrating with, enabling, and enhancing
the national cyberinfrastructure. Vital to the CEE mission is the persistent relationship with existing
and future users, including allocated users, training participants, XSEDE Conference attendees,
XSEDE collaborators, and campus personnel. CEE will unify public offerings to provide a more
consistent, clear, and concise message about XSEDE resources and services, and bring together
those aspects of XSEDE that have as their mission teaching, informing, and engaging those
interested in advanced cyberinfrastructure.
The five components of CEE are Workforce Development (§3.2), which includes Training, Education
and Student Preparation, User Engagement (§3.3), Broadening Participation (§3.4), User Interfaces
& Online Information (§3.5) and Campus Engagement (§3.6). These five teams will ensure routine
collection and reporting of XSEDE’s actions to address user requirements. They will provide a
consistent suite of web-based information and documentation and engage with a broad range of
campus personnel to ensure that XSEDE’s resources and services complement those offered by
campuses. Additionally, CEE teams will expand workforce development efforts to enable many
more researchers, faculty, staff, and students to make effective use of local, regional, and national
advanced digital resources. CEE will expand efforts to broaden the diversity of the community
utilizing advanced digital resources. The CEE team will tightly coordinate with the rest of XSEDE2,
particularly Extended Collaborative Support Services (§4), Resource Allocation Services (§7),
XSEDE Community Infrastructure (§5), and External Relations (§8.2).
CEE is focused on the personal interactions, ensuring that existing users, potential users and the
general public have sufficient access to materials and have a positive and effective experience with
XSEDE public offerings and frontline user support. As such, the CEE area metrics are designed to
broadly asses this performance. CEE has focused on metrics that will quantify how many users in
aggregate are benefiting from XSEDE resources and services. Additionally, CEE has focused on how
well the user base is sustained over time and how well training offerings evolve with the changing
user community needs.
Table 3-1: Area Metrics for Community Engagement & Enrichment
Area Metric
Number of new users of XSEDE resources
and services
Number of sustained users of XSEDE
resources and services
Number of new users from
underrepresented communities of XSEDE
resources and services
PY6 Target
>1000
>5000
>100
Sub-goal Supported
Deepen/Extend – Deepen use to existing
communities (§2.1.1)
Deepen/Extend – Deepen use to existing
communities (§2.1.1)
Deepen/Extend – Extend use to new
communities (§2.1.2)
16
Area Metric
PY6 Target
Number of sustained users from
underrepresented communities of XSEDE
resources and services
Number of participants in synchronous
and asynchronous training
Average impact assessment of training for
participants registered through XSEDE
User Portal
Number of unique visitors to the website
Number of unique visitors to the XSEDE
User Portal
Sub-goal Supported
>1000
Deepen/Extend – Deepen use to existing
communities (§2.1.1)
>5,000
Deepen/Extend – Prepare the current and
next generation (§2.1.3))
Deepen/Extend – Prepare the current and
next generation (§2.1.3)
4 of 5
80,000
15,000
Deepen/Extend - Raise awareness of the value
of advanced digital services (§2.1.4)
Deepen/Extend - Raise awareness of the value
of advanced digital services (§2.1.4)
3.1 CEE Director’s Office (WBS 2.1.1)
The CEE Director’s Office has been established to provide the necessary oversight to ensure the
greatest efficiency and effectiveness of the CEE area. This oversight includes providing direction to
the L3 management team, coordination of, and participation in, CEE planning activities and reports
through the area’s Project Manager. The Director’s Office also attends and supports the preparation
of project level reviews and activities.
The CEE Director’s Office will continue to manage and set direction for CEE activities and
responsibilities. They will contribute to and attend bi-weekly senior management team calls,
contribute to the project level plan, schedule, and budget, contribute to XSEDE quarterly, annual,
and other reports as required by the NSF and attend XSEDE quarterly and annual meetings.
3.2 Workforce Development (WBS 2.1.2)
The Workforce Development mission is to provide a continuum of learning resources and services
designed to address the needs and requirements of researchers, educators, developers, integrators,
and students utilizing advanced digital resources. This includes providing professional
development for the XSEDE team members.
Workforce Development will provide an integrated suite of training, education, and student
preparation activities to address formal and informal learning about advanced digital resources
addressing the needs of researchers, developers, integrators, IT staff, XSEDE staff, and
undergraduate and graduate faculty and students. CEE – Workforce Development will provide
business and industry with access to XSEDE2’s workforce development efforts including training
services and student internships that have proven beneficial to industry in the first five years of the
project.
Workforce Development is comprised of three areas: Training, Education and Student Preparation.
The Training team will develop and deliver training programs to enhance the skills of the national
open science community and ensure productive use of XSEDE’s cyberinfrastructure. The Education
team will work closely with training and student preparation to support faculty in all fields of study
with their incorporation of advanced digital technology capabilities within the undergraduate and
17
graduate curriculum. The Student Preparation program will actively recruit students to use the
aforementioned training and education offerings to enable the use of XSEDE resources by
undergraduate and graduate students to motivate and prepare them to pursue advanced studies
and careers to advance discovery and scholarly studies.
The Training metrics have been dramatically revised based on lessons learned. Beginning with PY6,
the metrics will include the number of people who attended training rather than the number of
people who registered. This will more accurately represent the number of people who are
benefitting from the training offerings. Further, since the learning environment and usage
modalities for synchronous (which incudes in-person and web-cast events) and asynchronous
(which includes usage of CI-Tutor and Cornell Virtual Workshop tutorials) training are very
different in nature, the number of people benefitting from these will be reported separately. In
addition, the number of unique people benefitting from the full range of training offerings, as well
as the number of people benefitting from each of the synchronous and asynchronous training
offerings, will be reported. These metrics will better represent the impact of training among the
community.
The Education metrics are consistent with those reported in previous years. They will include the
adoption and incorporation of computation and data-enabled techniques and methods within the
curriculum, and the sharing of course materials. This will include offerings of certificate and degree
programs, new and modified course modules, the number of modules shared with the community,
and the downloads of course modules for adoption and/or adaptation in courses.
The Student Preparation metrics will include the number of students engaged in utilizing XSEDE
resources and services through internships, their participation in the annual XSEDE Conference, or
working on XSEDE projects. Also included will be the number of under-represented students who
are involved.
Table 3-2: Area Metrics for Workforce Development.
Area Metric
PY6
Target
Number of unique participants, synchronous training
1600
Number of total participants, synchronous training
(One person can take several classes)
Number of unique participants, asynchronous training
2000
Number of total participants, asynchronous training
(One person can take several classes)
Average impact assessment of training for participants
registered through XSEDE User Portal
4000
Number of formal degree, minor, and certificate
programs added to the curricula
Number of materials contributed to public repository
2000
4 of 5
3
40
Sub-goal Supported
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
18
Area Metric
PY6
Target
Number of materials downloaded from the repository
56,000
Number of computational science modules added to
courses
Number of students benefitting from XSEDE resources
and services
Number of under-represented students benefitting
from XSEDE resources and services
40
50
50%
Sub-goal Supported
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
While the Training area metrics have changed from registrants to actual participants, they are
representative of the attendance figures that have been collected to date for XSEDE training
offerings. The Education area metrics reflect experiences to date and reflect a reduction in the
number of workshops and campus visits that will be conducted annually. The student metrics
represent the funding for internships and student participants at the annual XSEDE Conference. The
diversity figures reflect XSEDE’s success at recruiting and engaging under-represented students.
All of the training activities that were conducted in PY5 will be continued in PY6 at approximately
the same level of effort. They are intended to ensure that the community makes effective use of
advanced digital resources. The team will continue to develop and deliver synchronous (in-person
and web-cast) and asynchronous (self-paced tutorials) training on topics deemed most useful
through surveys, focus groups, and community feedback. The team will continue to offer roadmaps
for guided learning, certifications (including badges and continuing education units), and reviews of
new and revised materials.
The Education activities that will continue to be offered (at a reduced level of effort from PY5)
include promoting the integration of computational science into the curriculum, the collection and
dissemination of shared education and training materials, and promoting opportunities for faculty
professional development in computational science education, and workshops for faculty to focus
on curriculum development.
The Student Preparation activities will continue to recruit students for internships working with
XSEDE staff and researchers, and recruiting students to attend the annual XSEDE Conference.
The small, private, online-style courses offered by the education program, while very popular, will
not be continued due to lack of sufficient funding. There will be fewer Education workshops for
faculty and fewer visits to campuses to work with faculty and administrators on certificate and
degree program development efforts. Also, there will be fewer students supported via internships
and participation in the annual XSEDE Conference. The external evaluation effort will continue at a
significantly reduced level. The instruments developed by the evaluators will be utilized by the
workforce development team to continue to assess program impact and to identify improvements
to the services.
3.3 User Engagement (WBS 2.1.3)
The mission of the User Engagement (UE) team is to capture community needs, requirements and
recommendations for improvements to XSEDE’s resources and services, and report to the national
19
community how their feedback is being addressed. In PY6 and beyond, XSEDE will place greater
emphasis on maintaining consistent user contact, traceability in tracking user issues, and closing
the feedback loop.
The UE team will track three area metrics: the number of active and new PIs that are contacted
quarterly, the number of user issues that are identified via PI responses to the contact emails that
require follow-up, and the number of user issues that are resolved within each reporting period.
The UE team will maintain quarterly contact with all active research teams via email to the project
PI to ensure the team is making progress, any obstacles to the team are being addressed, and user
requirements are being gathered. Startup PIs will be contacted 15-45 days after their allocation
start date and new and renewal project PIs will be contacted 15-30 days following the start of the
new allocation period. Also, active PIs within their second, third and fourth quarters of their
allocation period will be contacted in the first month of each allocation quarter.
Issues that are raised in response to these contact emails will be logged and tickets generated on
the user’s behalf if further action is required. Tickets will be handled by the UE team, assigned to
another XSEDE team, or assigned to the appropriate service provider(s) for resolution. The UE team
will track tickets to ensure resolution. UE will also use the ticket data to identify community needs
and file Use Cases for further follow-up by the XCI area.
Action items resulting from issues and community needs will be recorded and tracked along with
the status and this information will be made available to the user community. The number of user
issues/requirements resolved within each reporting period will be included in each quarterly
report.
Table 3-3: Area Metrics for User Engagement.
Area Metric
PY6 Target
Sub-goal Supported
Number of active and new PIs contacted
quarterly
Number of user requirements
entered/tracked
All active PIs
Sustain - Provide excellent user support
(§2.3.2)
Sustain - Provide excellent user support
(§2.3.2)
Number of user requirements resolved
All issues identified
through regular
contact with PIs
90%
Sustain - Provide excellent user support
(§2.3.2)
As previously mentioned the UE team will maintain quarterly contact with every active PI, log and
track to resolution all user issues resulting from this regular contact, and aims to resolve 90% of the
issues reported by users.
Ongoing activities are in support of maintaining consistent contact with research teams to identify
and track user issues, assigning these issues to the appropriate group for resolution, and tracking
the issues to resolution. Therefore, they are in support of sub-goal “Provide excellent user support.”
In coordination with the Strategic Planning, Policy Evaluation and Organization Improvement
(§8.5) team, the annual user survey will be developed, scheduled, and administered. User surveys
20
will also be conducted by selecting a subset of user survey respondents and conducting interviews
with them. An interview report will be generated and this will be used to identify user issues to
address. These issues will then be passed on to the appropriate groups in XSEDE for resolution and
their progress will be tracked. Every PI will be contacted quarterly via email to identify user issues
to address. These responses will also be mined for user and community needs such as desired
software. Micro-surveys will continue in PY6. Again, these will be coordinated with Strategic
Planning, Policy Evaluation and Organization Improvement (§8.5) to facilitate the survey and User
Interfaces and Online Information (§3.5) to deploy the survey. The UE team will work with Project
Management, Reporting and Risk Management (§8.3) to develop user tracking processes using
project management tools, including JIRA and other issue tracking software.
3.4 Broadening Participation (WBS 2.1.4)
Broadening Participation will engage under-represented minority researchers from domains that
are not traditional users of HPC and from Minority Serving Institutions. This target audience ranges
from potential users with no computational experience to computationally savvy researchers,
educators, Campus Champions, and administrators that will promote change at their institutions for
increased use of advanced digital services for research and teaching.
Broadening Participation will continue the most effective recruitment activities— conference
exhibiting, campus visits, and regional workshops—while increasing national impact through new
partnerships and the utilization of lower cost awareness strategies to continue the growth in new
users from underrepresented communities. The Diversity Forum and the Minority Research
Community listservs and community calls focus on user persistence in their use of XSEDE services
and their deepening engagement through participation in committees such as the User Advisory
Committee (UAC) and XRAC, participation in champions, bridging, and other programs. Persistent
institutional engagement is enabled by curriculum reform and larger numbers of researchers
adopting the use of advanced digital resources as a standard research method.
Table 3-4: Area Metrics for Broadening Participation
Area Metric
Number of new under-represented
individuals using XSEDE resources and
services
Number of sustained under-represented
individuals using XSEDE resources and
services
Longitudinal Assessment of XSEDE
Inclusion & Equity via the Climate
Survey
PY6 Target
Sub-goal Supported
>100
Deepen/Extend – Deepen use to existing
communities (§2.1.1)
>1000
Advance – Enhance the array of technical
expertise and support services (§2.2.1)
25%
improvement
Advance – Enhance the array of technical
expertise and support services (§2.2.1)
The target for new users is a straight line growth based on prior years’ experience. Some in new
communities may make use of science gateways prior to requesting an XSEDE allocation. While
gateways to report user counts for those running jobs on XSEDE resources, they do not report
demographics, nor do they report counts of those using other resources through a gateway. This
21
target for new users may be adjusted down due to these factors. The target for number of sustained
users is approximately 25% of the first time XSEDE users from underrepresented communities
identified since the start of the metrics collection. In past XSEDE climate surveys, some staff
members have identified issues that affect equity and inclusion. The target is to see a 25%
improvement.
Broadening Participation will continue the most effective recruitment activities— conference
exhibiting, campus visits, and regional workshops—while increasing national impact through new
partnerships and the utilization of lower cost awareness strategies. In addition to conference
exhibiting, online XSEDE talks will be offered to organizations and professional societies that host
career development webinars for under-represented minority researchers, early career faculty,
graduate and undergraduate students. Additionally, the team will aggressively promote the
submission of papers at professional societies by XSEDE under-represented minority users and
expand the dissemination to partners to include new initiatives such as the IEEE Special Technical
Committee on Broadening Participation.
The Minority Research Community currently consists of faculty users who share information via a
listserv and participate in two or three community calls throughout the year. This virtual
community will be restructured to encourage more persistent participation, increase the number of
active participants, and facilitate their input into XSEDE strategies and programs.
Using the model of the XSEDE Service Provider’s forum, the Southeastern Universities Research
Association (SURA) will establish the XSEDE Diversity Forum with participation from each of the
outreach and diversity managers at each of the XSEDE partners. The forum participants will share
best practices, cross disseminate information, identify ways to leverage education and outreach
activities and review XSEDE programs to ensure they are encouraging and supporting diversity.
3.5 User Interfaces & Online Information (WBS 2.1.5)
User Interfaces & Online Information (UII) is committed to enabling the discovery, understanding,
and effective utilization of XSEDE’s powerful capabilities and services. UII has immediate impact on
XSEDE users from day one, providing them with an information rich website, the XSEDE User
Portal, and a uniform set of user documentation.
The UII team will track six area metrics: number of new users of XSEDE resources and services,
number of sustained users of XSEDE resources and services, number of unique visitors to the
website and User Portal, and user satisfaction rating of the website, User Portal, and online
documentation.
The number of new users of XSEDE resources and services is comprised of the number of new
XSEDE User Portal accounts that are created in PY6. The number of sustained users of XSEDE
resources and services is the total number of users who have logged into the XSEDE User Portal
during PY6, including new users that created accounts during PY6. The number of unique visitors to
the website and User Portal will be tracked using Google Analytics. This is defined as both new and
returning users that have at least one session within the selected time period. User satisfaction of
the website, User Portal and user documentation will be a rating from the annual XSEDE User
Survey. If additional surveys such as micro-surveys are conducted to measure satisfaction, they will
22
be shared in the appropriate quarterly report.
Table 3-5: Area Metrics for User Interfaces & Online Information.
Area Metric
Number of new users of XSEDE
resources and services
Number of sustained users of XSEDE
resources and services
Number of unique visitors to the
website
Number of unique visitors to the XSEDE
User Portal
User satisfaction with website
User satisfaction with user portal
User satisfaction with user
documentation
PY6 Target
Sub-goal Supported
5,000
Sustain - Provide excellent user support (§2.3.2)
8,000
Sustain - Provide excellent user support (§2.3.2)
80,000
Sustain - Provide excellent user support (§2.3.2)
15,000
Sustain - Provide excellent user support (§2.3.2)
4 of 5
4 of 5
4 of 5
Sustain - Provide excellent user support (§2.3.2)
Sustain - Provide excellent user support (§2.3.2)
Sustain - Provide excellent user support (§2.3.2)
High-end targets for the metrics above were selected based on previous experience The target
number of new users and sustained users is 5,000 and 8,000, respectively. The number of unique
visitors to the website and User Portal reflect the different audiences. The website has a wider
range of audiences and therefore, the PY6 target was selected as 80,000, a number that has not
been reached before. The User Portal has a more targeted audience of XSEDE users and therefore,
the number of unique visitors is targeted at 15,000. The average satisfaction ratings are measured
in the annual user survey as a measurement of quality of the User Portal, website and user
documentation and selected to be greater or equal to 4 out of 5.
The UII team will continue to manage the XSEDE User Portal and technical user documentation
such as getting started guides, user guides, etc.
Extending beyond managing user documentation the UII team will also take responsibility for the
website. The website will undergo a new redesign with the goal of creating a website and User
Portal that have distinctive designs to reflect the different audiences they are serving. Furthermore,
the UII team will incorporate the XSEDE software catalog and administrative interface into their
work (previously funded through the XSEDE Technology Investigation Services team).
The UII team will no longer handle the Knowledge Base (KB) but will work with the KB team to
share information about documentation updates and stale documents but it will no longer be
managed under UII.
3.6 Campus Engagement (WBS 2.1.6)
The Campus Engagement program promotes and facilitates the effective participation of a diverse
national community of campuses in the application of advanced digital resources and services to
accelerate discovery, enhance education, and foster scholarly achievement.
23
Campus Engagement, via the Campus Champions, works directly with institutions across the US
both to facilitate computing and data-intensive research and education, nationally and with
collaborators worldwide, and to expand the scale, scope, ambition and impact of these endeavors.
This is done by increasing scalable, sustainable institutional uptake of advanced digital services
from providers at all levels (workgroup, institutional, regional, national, international), fostering a
broader, deeper, more agile, more sustainable and more diverse nationwide cyberinfrastructure
ecosystem across all levels and cultivating inter-institutional interchange of resources, expertise
and support.
Campus Engagement’s core approach is to assist researchers and educators in identifying and
facilitating the use of the most appropriate advanced digital capabilities for their extant and
emerging needs, including workgroup, institutional, regional, national and/or international level
(which is anticipated in a subset of cases to include XSEDE resources). Campus Engagement’s
activities include, but are not limited to, fostering the expansion of both the scale and the scope of
the national and worldwide community of cyberinfrastructure practitioners, both via internal
initiatives and in collaboration with other related efforts. CE also aims to assist with the
establishment and expansion of consortia (for example, intra-state, regional, domain-specific) that
collaborate to better serve the needs of their advanced computing stakeholders. XSEDE teams (e.g.,
service providers, diversity leadership, Champions) will be leveraged to both reach appropriate
stakeholders and provide them with the information and capabilities they need. CE will also
educate institutional leadership about the value proposition of advanced digital services and the
impact of these capabilities on research and education outcomes, as a driver for stimulating
investment across all scales.
Metrics include the number of institutions with a Champion, the number of unique contributors to
the Champion email list and the number of activities that (i) expand the emerging CI workforce
and/or (ii) improve the extant CI workforce, participated in by members of the Campus
Engagement team. The number of institutions with Champions represents the breadth of the reach
of the Campus Engagement/Champion program. The email list metric demonstrates active
involvement across institutions and shows the extent to which peer mentoring is valued. The
number of activities that expand and/or improve the extant CI workforce demonstrates the efforts
of the team to nurture continued development of the CI workforce across the community.
Table 3-6: Area Metrics for Campus Engagement.
Area Metric
PY6
Target
Sub-goal Supported
Number of Institutions with a Champion
225
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Number of unique contributors to the Champion
email list ([email protected])
Number of activities that (i) expand the emerging CI
workforce and/or (ii) improve the extant CI
workforce, participated in by members of the
Campus Engagement team
50
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
20
24
The number of institutions with Champions is publicized on the XSEDE website and the current rate
of growth indicates that 225 is a reasonable target for PY6. It is anticipated that the number of
Champion institutions will top out somewhat below the number of institutions with Carnegie
Classifications of doctoral university with highest, higher or moderate research activity, which total
329 institutions as of 2015 (http://carnegieclassifications.iu.edu/). The overwhelming majority of
doctoral universities will ultimately have Champions and a modest number of Champions at nondoctoral institutions is expected, based on uptake to date.
The target for the number of unique contributors to the email list is based on previous list analysis
indicating roughly 20% of Champions are regular contributors. The target for the number of
activities that (i) expand the emerging CI workforce and/or (ii) improve the extant CI workforce,
participated in by members of the Campus Engagement team is set at 20. As a new metric, we will
be monitoring this and adjusting the target as appropriate.
All of the Champion subprograms (Campus, Regional, Student, Domain) are continuing. The
Regional Champion subprogram should be very effective in helping to sustain the Champion
program at a much higher level of involvement. The Student subprogram will be very effective in
preparing the next generation of workforce participants. Champions will continue to engage the
Fellows subprogram administered and funded by ECSS. These subprograms all require training and
development support from the XSEDE community to foster good methods and effectiveness
throughout the program and to assist in the creation of an effective virtual program. A marketing
presence will continue to be developed to raise the awareness and impact of the Champion
program. These will bolster the awareness of the value of advanced digital services throughout the
community.
A new concierge service will be developed to integrate new Champions and to provide them
resources and mentorship that will enable new Champions to onboard more rapidly and therefore
to provide better facilitation to the researchers and educators they represent. A new sustainability
task force has been formed to meet regularly and develop strategies to sustain the Champion
community beyond this project. This new activity will begin planning for the post-XSEDE era, in
anticipation of the need for diverse funding sources. A new initiative has recently begun informally,
and will begin formally in PY6, to visit states that don’t currently have ongoing intrastate
cyberinfrastructure collaboration, to help these states identify the benefits of such collaboration
and to begin to adopt models for such collaboration that are appropriate to their statewide research
climates and needs This new activity will extend the impact of research and education facilitation
by providing mechanisms for peer mentoring among cyberinfrastructure practitioners, service
provision to underserved institutions (e.g., non-doctoral), and interfacing with institutional officers
such as Chief Information Officers and Chief Research Officers.
The Campus Engagement co-managers are leading efforts to establish relationships and
collaborations with a variety of related initiatives and organizations. For example, with Advanced
Cyberinfrastructure Research & Education Facilitators (ACI-REF), both Campus Engagement comanagers are part of the pending ACI-REF Research Coordination Network proposal and comanager Henry Neeman is part of the follow-on proposal to the original Clemson-led ACI-REF
grant. Other examples include co-manager Dana Brunson and several Champions are involved in
the Software Carpentry organization and part of the effort to create HPC Carpentry materials,
Brunson is interfacing with Open Science Grid (OSG) personnel to establish mechanisms to
incorporate OSG into the Champion CI portfolio, Neeman is a member of the HPC Systems
25
Professionals SC16 workshop organizing committee and both co-managers are members of the
Coalition for Academic Scientific Computing (CASC).
Quarterly outreach events at XSEDE quarterly meetings will be discontinued.
26
4 Extended Collaborative Support Services (WBS 2.2)
The Extended Collaborative Support Service (ECSS) improves the productivity of the XSEDE user
community both through successful, meaningful collaborations and well-planned training activities.
The goal is to optimize applications, improve work and data flows, increase effective use of the
XSEDE digital infrastructure and broadly expand the XSEDE user base by engaging members of
under-represented communities and domain areas. The ECSS program provides professionals who
can be part of a collaborative team—dedicated staff who develop deep, collaborative relationships
with XSEDE users, helping them make best use of XSEDE resources to advance their work. These
professionals possess combined expertise in many fields of computational science and engineering.
They have a deep knowledge of underlying computer systems and of the design and
implementation principles for optimally mapping scientific problems, codes, and middleware to
these resources. ECSS includes experts in not just the traditional use of advanced computing
systems but also in data-intensive work, workflow engineering, and the enhancement of scientific
gateways.
ECSS projects fall into five categories: Extended Support for Research Teams (ESRT), Novel and
Innovative Projects (NIP), Extended Support for Community Codes (ESCC), Extended Support for
Science Gateways (ESSGW), and Extended Support for Training, Education and Outreach (ESTEO).
Project-based ECSS support is requested by researchers via the XSEDE peer-review allocation
process. If reviewers recommend support and if staff resources are available, the ECSS expert and
the requesting PI develop a work plan outlining the project tasks. The work plan includes concrete
quarterly goals and staffing commitments from both the PI team and ECSS. ECSS managers review
work plans and also track progress via quarterly reports.
The main metrics of ECSS are the number of projects with workplans per year (targets are ESRT
(30), ESCC (10), ESSGW (10)), the average satisfaction with the ECSS support as reported by PIs in
post-project interviews (target is 4.5/5), and the impact that those PIs think the ECSS support has
had on their project (target is 4/5). The other two metrics, number of new users from nontraditional disciplines of XSEDE resources and services and number of sustained users from nontraditional disciplines of XSEDE resources and services are explained in more detail just before
Table 4-3.
Table 4-1: Area Metrics for Extended Collaborative Support Services
Area Metric
Number of completed ECSS projects
Average ECSS impact rating
Average satisfaction with ECSS support
Number of new users from non-traditional
disciplines of XSEDE resources and
services
PY6 Target
50
4 of 5
4.5 of 5
100
Sub-goal Supported
Deepen/Extend – Deepen use to
communities (§2.1.1)
Deepen/Extend – Deepen use to
communities (§2.1.1)
Deepen/Extend – Deepen use to
communities (§2.1.1)
Deepen/Extend – Extend use
communities (§2.1.2)
existing
existing
existing
to
new
27
Area Metric
PY6 Target
Number of sustained users from nontraditional disciplines of XSEDE resources
and services
100
Sub-goal Supported
Deepen/Extend – Extend
communities (§2.1.2)
use
to
new
4.1 ECSS Director’s Office (WBS 2.2.1)
The ECSS Director’s Office has been established to provide the necessary oversight to ensure the
greatest efficiency and effectiveness of the ECSS area. The office consists of two Level 2 CoDirectors (Ralph Roskies, who manages ESRT and NIP activities, and Nancy Wilkins-Diehr who
manages ESCC, ESSGW and ESTEO activities) and two projects managers (Karla Gendler and
Marques Bland).
The Level 2 ECSS Co-Directors, Roskies and Wilkins-Diehr, will continue to advise the XSEDE PI on
many issues, especially those where ECSS activities are relevant. They monitor compliance with
budgets, and retarget effort if necessary. They periodically run staff face-to-face meetings and
respond to concerns, particularly those that emerge from staff climate surveys.
In addition, Roskies and Wilkins-Diehr carry out the post-project interviews with all project PIs
who have received ECSS support, both to get their assessment of how the project went, and to hear
and act on any concerns they may express. Wilkins-Diehr also organizes the monthly symposium
series, one of the contributors to staff training, and runs the Campus Champions Fellows program
(§3.6).
The two project managers aid in the management of the day-to-day activities of ECSS which
includes the management of project requests (XRAC and startups), active projects, project
assignments and staffing. They will refine the ECSS project lifecycle, further defining processes to
aid in the management of over 100 active projects. They will also administer JIRA for the
management and tracking of projects, both for the managers and directors of ECSS and for ECSS
staff. They will provide ECSS information to the XSEDE Project Management (PM) office and relay
information from the PM team to ECSS. They manage and attend biweekly ECSS management and
less frequent full staff meetings as well as PM meetings. They post notes and action items to the
ECSS wiki once meetings have concluded. They also maintain the ECSS wiki and mailing lists.
New activities for PY6 include managing the transition to JIRA project management software.
4.2 Extended Support for Research Teams (WBS 2.2.2)
Extended Support for Research Teams (ESRT) accelerates scientific discovery by collaborating with
researchers, engineers, and scholars in order to optimize their application codes, improve their
work and data flows and increase the effectiveness of their use of XSEDE digital infrastructure.
28
ESRT projects are initiated as a result of support requests or recommendations obtained during the
allocation process. Most projects focus on home-grown codes, as community codes fall under ESCC
(§4.4) but are not exclusively restricted to this classification. The primary mandate of ESRT is the
support of individual research teams within the context of their research goals.
The area metrics for ESRT include the 1) number of completed projects, 2) average impact rating,
and 3) average satisfaction with support. The number of completed projects will be monitored
throughout the project. A completed project is defined as a project that has progressed through the
complete support pipeline, which includes steps such as assignment of a consultant, production of a
work plan, execution of work plan and reporting of progress through quarterly reports, and the
filing of a final project report. The PIs of these completed projects are contacted by ECSS leadership
for an interview where PIs relay their experience with ECSS support and its impact upon their
research. These discussions culminate in a numerical ranking (out of 5 possible) that expresses the
level of satisfaction with ECSS support and its impact upon their research goals.
Table 4-2: Area Metrics for Extended Support for Research Teams.
Area Metric
Number of completed ESRT projects
Average ECSS impact rating
Average satisfaction with ECSS support
PY6 Target
30
4 of 5
4.5 of 5
Sub-goal Supported
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
The number of completed ESRT projects has an annual target of 30 completed projects. This target
is chosen based on experience with the number of projects that are typically requested as well as
evaluation of the available staff effort available within ESRT. The recommendation of 8 or more
projects per quarter is common. This number of recommended projects is sufficient to allow for the
meeting of the metric given that some projects are unable to start for various reasons and are not
included in the number of completed projects. In PY6, the available effort for projects in ESRT is
about 9.3 FTEs. As a result, the target illustrates a reasonable project load of about 3.2 projects per
FTE.
The average impact rating has a target of 4 out of 5 while the average satisfaction rating has a target
of 4.5 out of 5. While initially both targets were 4 of 5, a rating of 4 was found to be easy to achieve.
Impact scores are lower since PI interviews generally occur soon after the ECSS work when impact
may be difficult to ascertain.
The L3 manager will continue to manage ESRT projects via staff assignment, progress tracking, and
reporting. This activity supports deepening use by existing communities via the initiation,
execution, and reporting of projects focused on supporting and improving the use of XSEDE
resources by research teams. ESRT plans to initiate 30 projects recommended through the
allocations process, document 30 project work plans created by interactions between assigned staff
and PIs prior to project execution, execute 30 project work plans via assigned staff effort and
document progress via quarterly reports and document 30 project final reports for completed
29
projects. Most ECSS staff, including those in ESRT, have small fractions of time devoted to training,
education and outreach through ECSS’ ESTEO group. In that capacity, ESRT staff will provide
training to staff on new XSEDE cyberinfrastructure utilized for ESRT projects. In addition, ESRT
management will provide new staff training on work plan creation, project execution, and reporting
as a refresh at the start of the XSEDE2 program. ESRT staff will continue to participate in the review
of XSEDE proposals include XRAC, adaptive and startup requests as well as participate in the
monthly symposium to gain and disseminate knowledge acquired through projects.
New for PY6, ESRT and ECSS management will provide JIRA training to staff in order to facilitate the
utilization of this newly developed project management capability to better track project
assignment, execution, and documentation. This activity supports an effective virtual organization
via the project management capabilities provided that enhance communication and documentation
of projects in addition to increased visibility of available staff effort and skills, which facilitates
more efficient project assignment.
4.3 Novel and Innovative Projects (WBS 2.2.3)
Novel and Innovative Projects (NIP) accelerates research, scholarship, and education provided by
new communities that can strongly benefit from the use of XSEDE’s ecosystem of advanced digital
services. Working closely with the XSEDE outreach team, the NIP team identifies a subset of
scientists, scholars and educators from new communities, i.e. from disciplines or demographics that
have not yet made significant use of advanced computing infrastructure, who are now committed to
projects that appear to require XSEDE services and are in a good position to use them efficiently.
NIP staff then provides personal mentoring to these projects, helping them to obtain XSEDE
allocations and use them successfully.
XSEDE projects generated by, and mentored by, the personal efforts of the NIP experts should
stimulate additional practitioners in their field to become interested in XSEDE. Strategies used will
include building and promotion of science gateways serving communities of end-users and the
enhancement of the Domain Champions program by which successful practitioners spread the
word about the benefits of XSEDE to their colleagues.
A set of 60 fields of science (FOS) have been identified in the XD Central Database, each of whose
usage over the past 10 years is below 0.5% of the total normalized usage. The area metric “number
of new users from non-traditional disciplines of XSEDE resources and services” (an XSEDE KPI) will
report the total number of users on the projects newly activated from these FOS. The area metric
“number of sustained users from non-traditional disciplines of XSEDE resources and services” (an
XSEDE KPI) will report the total number of users on the projects from these FOS that have used at
least 10% of their allocated usage. The area metric “number of new XSEDE projects from target
communities generated by NIP” will report the number of projects from the targeted FOS that will
be generated by the personal efforts of NIP staff members. The area metric “number of successful
XSEDE projects from target communities mentored by NIP” will report the number of projects from
the targeted FOS, personally mentored by NIP staff members that have used at least 10% of their
allocated usage.
30
Table 4-3: Area Metrics for Novel and Innovative Projects
Area Metric
PY6 Target
Sub-goal Supported
Number of new users from non-traditional
disciplines of XSEDE resources and services
100
Deepen/Extend – Extend use to
new communities (§2.1.2)
Number of sustained users from non-traditional
disciplines of XSEDE resources and services
100
Deepen/Extend – Extend use to
new communities (§2.1.2)
Number of new XSEDE projects from target
communities generated by NIP
20
Deepen/Extend – Extend use to
new communities (§2.1.2)
Number of successful XSEDE projects from target
communities mentored by NIP
10
Deepen/Extend – Extend use to
new communities (§2.1.2)
The PY6 targets above were set as an initial expectation based on observations in PY3-PY5. They
will be revised annually.
NIP will continue to recruit and mentor users from target FOS to ensure the success of their
projects. The team will recruit successful users who are influential in their fields to become XSEDE
Domain Champions and provide them with the support they need to help extend use to their
communities.
Working closely with SPs, NIP will recruit and steer appropriate user groups and Domain
Champions to the most suitable resources, especially science gateways, virtual environments, and
data hosting and analysis support.
4.4 Extended Support for Community Codes (WBS 2.2.4)
Extended Collaborative Support for Community Codes (ESCC) extends the use of XSEDE resources
by collaborating with researchers and community code developers to deploy, harden, and optimize
software systems necessary for research communities to create new knowledge.
ESCC supports users via requested projects and initiated projects. ESCC projects may be created in
two different ways. Most ESCC projects are initiated as a result of requests for assistance during the
allocation process. These projects are similar in nature to ESRT projects but involve community
codes rather than codes developed for and by individual research groups. ESCC projects may also
be initiated by staff to support a community’s needs.
ESCC has three key metrics: number of completed projects, impact rating, and satisfaction rating.
These metrics are collected on a quarterly basis although the targets are annual.
The number of completed projects refers to the number of projects that were assigned, progressed
to the workplan phase and completed with a final report. The impact rating is an estimate of the
impact the ESCC project had on a PI’s research based on a number the PI assigns to it. The
satisfaction rating measures the PI’s satisfaction with the support provided by ECSS.
31
Table 4-4: Area Metrics for Extended Support for Community Codes
Area Metric
Number of completed ESCC projects
Average ESCC impact rating
Average satisfaction with ESCC support
PY6 Target
10
4 of 5
4.5 of 5
Sub-goal Supported
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
The goal of 10 completed projects per year was established in the first year of XSEDE and appears
to be a reasonable choice that corresponds well to the average number of requests per year and is
sustainable with current staffing levels.
The impact and satisfaction ratings come directly from the PIs. The L2 lead carries out a PI
interview in which the PI is asked to rate the impact this ECSS project has had on their research and
their satisfaction with the support provided. Satisfaction rating tracks the satisfaction with the
support given by ECSS experts. These ratings and feedback are shared with the ECSS experts who
worked on the project. If there are concerns with the assistance given, the results are discussed
with the ECSS expert, the L2 lead, and the L3 manager.
Ongoing activities for ESCC include the development of workplans for approved projects and the
creation of a final report that accurately summarizes the outcome of the project. ESCC is also
responsible for characterizing the collection of community codes available on SP resources to
determine if there are gaps in XSEDE coverage. This will be much easier with the implementation of
an updated software search tool in the XSEDE User Portal.
ESCC will use the XDMoD (XSEDE Metrics on Demand) portal to characterize community code
usage on XSEDE resources. This will assist ECSS management in identifying gaps in the installed
community software portfolio.
4.5 Extended Support for Science Gateways (WBS 2.2.5)
Extended Support for Science Gateways (ESSGW) broadens science impact and accelerates
scientific discovery by collaborating in the development and enhancement of science-centric
gateway interfaces and by fostering a science gateway community ecosystem.
ESSGW projects primarily begin through user requests from the XSEDE allocation process. Similar
to ESRT and ESCC, ESSGW projects progress through three activities. First, the project is assigned to
an ECSS expert. Second, the project is quantified with the formation of a work plan through
collaboration with the research group. Third, when the project is completed, the ESSGW expert
produces a final report with input from the research group. A successful project is the completion of
all three phases. Each state of the progression is measured to provide an assessment of progress.
Submission of work plans within 45 days of initial contact, 90% of projects with work plans
completed, and 85% of completed projects with final reports within three months are additional
criteria for success.
32
Area metrics are direct measurements of ESSGW’s operations and effectiveness. The target of ten
completed ESSGW projects per year measures ESSGW’s operations and indicates that the area is
both bringing in new projects and concluding previous consultations. The effectiveness of the
consultations is measured by the next two metrics, impact rating and satisfaction of the support
recipients. These two metrics are measured through interviews. The fourth metric measures the
unique number of gateway users across all gateways.
Table 4-5: Area Metrics for Extended Support for Science Gateways
Area Metric
Number of completed ESSGW projects
Average ESSGW impact rating
Average satisfaction with ESSGW support
Number of unique gateway users per quarter
PY6 Target
10
4 of 5
4.5 of 5
3000
Sub-goal Supported
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
The target values were chosen as follows. Targeting ten completed projects per year means that
ESSGW staff are completing approximately 50-75% of their projects if fully assigned. From previous
experience, this is an achievable goal that nevertheless requires diligence from ESSGW
management and staff. Impact and satisfaction ratings are determined through interviews of the PIs
by ECSS management and indicate the PI’s perception of the value obtained from the consultation.
Ongoing activities for ESSGW include the development of workplans for approved projects and the
creation of a final report that accurately summarizes the outcome of the project. ESSGW is also
responsible for supporting the XSEDE science gateway community generally, which will continue to
be done through regular community presentations. ESSGW will continue to support the collection
of gateway usage metrics. ESSGW will continue to work closely with other areas of the XSEDE team
to conduct outreach and training.
ESSGW management will work with Service Providers to determine requirements for gateway
usage metrics that fit well with cloud and data centric XSEDE resources.
4.6 Extended Support for Education, Outreach, & Training (WBS 2.2.6)
Extended Collaborative Support for Training, Education & Outreach (ESTEO) prepares the current
and next generation of researchers, engineers, and scholars in the use of advanced digital
technologies by providing the technical support for Training, Education and Outreach planned
activities.
Typical events include train-the-trainers events, on-site classes requested by Campus Champions,
regional workshops, conferences, and summer schools (including the International HPC Summer
School). Staff also create and review online documentation and training modules. This on-demand
training is increasingly popular with the user community when both time and travel budgets are
limited.
33
A few of the overall activities performed by ESTEO are represented in the Area Metrics table,
namely, metrics surrounding support of the campus champions fellows program, a metric to gauge
the number of live training events staffed, and metrics designed to measure breadth of attendees
participating in both staff training and ECSS symposium activities. The full breadth of ESTEO
activities is captured in the “ongoing activities” section below, which are measured and reported
quarterly in the metrics appendix.
Table 4-6: Area Metrics for Extended Support for Education, Outreach, & Training
Area Metric
Number of Campus Champions fellows
PY6 Target
4
Average Score of fellows assessment
4 of 5
Number of live training events staffed
20
Number of staff training events
2
Attendees at staff training events
40
Attendees at ECSS Symposia
300
Sub-goal Supported
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Deepen/Extend – Prepare the current
and next generation (§2.1.3)
Advance – Enhance the array of
technical expertise and support
services (§2.2.1)
Advance – Enhance the array of
technical expertise and support
services (§2.2.1)
Advance – Enhance the array of
technical expertise and support
services (§2.2.1)
The target for the number of Campus Champion Fellows and the impact rating reflect the desire to
run a contained, but high quality program. The Fellows gain in depth expertise in new areas
represented by the projects that they participate in with their ECSS mentors. The number of live
training events is based on available staff. Funding constraints dictate a shift from live events to
asynchronous events, which can more easily be shared with large numbers of users and potential
users. The biggest driver for live training events are the workshops jointly planned with the
Broadening Participation component of Community Engagement and Enrichment. The target for
the number of staff training events is set to strike a balance between staff-focused and user-focused
activities (with a deliberate bias to user-focused activities). The targets for staff participation in
both staff training and ECSS symposia indicate the desire to maintain a well-prepared workforce.
ESTEO has ongoing activities that support the sub-goal “Preparing the current and next
generation,” namely, delivering live training, reviewing education proposals, mentoring Campus
Champion Fellows, mentoring students, reviewing training modules, producing training modules,
and retiring training modules. These directly deliver on training and education goals encompassed
in this sub-goal. ESTEO also provides services to support the sub-goal “Raising Awareness of the
value of advanced digital services” by responding to requests for service, supporting meetings and
BoFs, supporting Campus Champions and producing and presenting talks and presentations.
Additionally, ESTEO supports the sub-goal “Enhance the array of technical expertise and support
services” with ECSS internal training. A particular focus in 2016 is on new SP resources and the
34
innovative capabilities they will offer. Metrics for ongoing activities are reported quarterly in the
metrics appendix.
Support is being discontinued for large online courses such as Jim Demmel’s “Applications of
Parallel Computers” course, as this is no longer hosted by XSEDE.
35
5 XSEDE Community Infrastructure (WBS 2.3)
The mission of XSEDE Community Infrastructure (XCI) is to facilitate interaction, sharing and
compatibility of all relevant software and related services across the national CI community
building on and improving upon the foundational efforts of XSEDE.
XCI envisions enabling users targeting services allocated by XSEDE (including OSG resources),
campus-based CI facilities, commercial cloud providers, CI software services such as science
gateways and Globus Online, and even the individual researcher who wants to interact effectively
with the national CI via her or his own laptop. Through XCI, XSEDE will serve an aligning function
within the nation, not by rigorously defining a particular architecture, but rather by assembling a
technical architecture that facilitates interaction and interoperability across the national CI
community. The suite of interoperable and compatible software tools that XSEDE will make
available to the community will be based on those already in use but will add additional services
that address emerging needs including data and computational services.
There are four area metrics for XCI: Average satisfaction rating of XCI services; number of new
capabilities made available for production deployment; number of capabilities delivered vs. those
planned; and total number of systems that use one or more CRI provided toolkits.
For the average satisfaction rating of XCI services, the L2 Director and Deputy Director will focus
their attention on current and new community stakeholders who can benefit from XCI services.
This will be a critical method for demonstrating return on investment (ROI) for XCI. They will work
with the XSEDE Evaluation team to survey all community stakeholders on the value of XCI services.
For the number of new capabilities made available for production deployment, XCI will accept Use
Cases from CEE, ECSS and Capability and Resource Integration (CRI) and will quickly provide
capability development plans (CDP) that will be prioritized and executed based on feedback from
the User Requirement Evaluation and Prioritization (UREP) and the SMT.
Measuring the number of capabilities delivered versus those planned is essentially an efficiency
ratio. With proper vetting and prioritization by the UREP and Senior Management Team (SMT), XCI
is committed to the efficient and effective delivery of new capabilities.
The total number of systems that use one or more CRI provided toolkits is a measure of
effectiveness in broadening the community and participation in national cyberinfrastructure. If CRI
is successful, participation in national cyberinfrastructure means sharing resources beyond
institutional boundaries, not just consuming them.
Table 5-1: Area Metrics for XSEDE Community Infrastructure (XCI)
Area Metric
PY6 Target
Average satisfaction rating of XCI services
4 of 5
Number of new capabilities made available
for production deployment
Number of capabilities delivered/ Number
planned
10
1
Sub-goal Supported
Advance - Create an open and evolving einfrastructure (§2.2.1)
Advance - Create an open and evolving einfrastructure (§2.2.1)
Advance - Create an open and evolving einfrastructure (§2.2.1)
36
Area Metric
PY6 Target
Total number of systems that use one or
more CRI provided toolkits
450
Sub-goal Supported
Advance - Create an open and evolving einfrastructure (§2.2.1)
XCI is focused on stakeholder satisfaction and thus the goal for “average satisfaction rating of XCI
services” is a minimum of 4 on a scale of 1-5, where 5 is the most satisfied.
A new capability is defined as the ability for users to do what is detailed in a new Use Case. The goal
for “number of new capabilities made available for deployment” is to deliver what is described in
10 new non-canonical Use Cases. This will be accomplished by a combination of: identifying which
existing production components support a new Use Case, by delivering component upgrades to
support a new Use Case, and by delivering new components that support a new Use Case. XCI will
partner with software/service providers whenever possible to deliver new and upgraded
components.
The “Number of capabilities delivered/Number planned” area metric is an efficiency measure. With
proper vetting and prioritization by the XSEDE User Requirements Evaluation and Prioritization
working group (UREP) and SMT, XCI is committed to the efficient and effective delivery of new
capabilities with an optimal ratio of 1.
The area metric “Total number of systems that use one or more CRI provided toolkits” has a target
of 450 as Globus Transfer systems numbers, in the past, have been in the low hundreds (250-300
but growing steadily each quarter). Ideally, XCI would like to see the other tools deployed on 100 or
150 systems which would allow a goal that is attainable without undue reliance on Globus user
base increases alone.
5.1 XCI Director’s Office (WBS 2.3.1)
The XCI Director’s Office has been established to provide the necessary oversight to ensure the
greatest efficiency and effectiveness of the XCI area. This oversight includes providing direction to
the L3 management team, coordination of and participation in XCI planning activities and reports
through the area’s Project Manager. The Director’s Office also attends and supports the preparation
of project level reviews and activities.
XCI will work with CEE, ECSS, and UREP groups to understand new Use Case requests and their
priorities to ensure timely delivery of necessary capabilities and services. Furthermore, XCI will
work with these same groups to help quantify the importance and usage of new services and
capabilities by the NSF community in determining return on investment.
The XCI Director’s Office will continue to manage and set direction for XCI activities and
responsibilities. They will contribute to and attend bi-weekly SMT calls, contribute to the project
level plan, schedule, and budget, contribute to XSEDE quarterly, annual, and other reports as
required by the NSF and attend XSEDE quarterly and annual meetings. The XCI Director’s office will
also initiate, evaluate, and act on annual stakeholder satisfaction surveys.
37
New activities include a renewed focus on identifying new community stakeholders who can
benefit from XCI services, understanding their requirements, and measuring their satisfaction
qualitatively and, wherever possible, quantitatively.
5.2 Requirements Analysis & Capability Delivery (WBS 2.3.2)
The Requirements Analysis & Capability Delivery (RACD) team facilitates the integration,
maintenance, and support of cyberinfrastructure capabilities addressing user technical
requirements. The process begins by preparing Capability Delivery Plans (CDPs) that describe the
technical gaps in XSEDE’s prioritized Use Cases. To fill the gaps, RACD evaluates and/or tests
existing software solutions, engages with software providers and facilitates software and service
integration. To ensure software and service adoption and ROI, RACD will involve users and service
providers and operators in an integration process that uses engineering best practices and
instrument components to measure usage. Once components are integrated, RACD will facilitate
software maintenance and enhancements in response to evolving user needs and an evolving
infrastructure environment.
RACD helps create an open and evolving e-infrastructure by integrating components that support
XSEDE Use Cases. The first key activity in that process involves preparing Capability Delivery Plans
(CDPs) that detail how RACD plans to address software capability gaps in XSEDE prioritized Use
Cases. Some capability gaps may require engaging with current or new software partners and this
number of engagements is to be tracked. Based on lessons learned in the previous years of the
project, the engineering process further improves both user and service provider engagement
during the capability integration process and thus the satisfaction of this engagement is tracked.
Table 5-2: Area Metrics for Requirements Analysis & Capability Delivery
Area Metric
PY6 Target
Number of capability delivery plans prepared for
prioritized Use Cases
Number of CI integration assistance engagements
7
User rating of components delivered in production
4 of 5
Operator rating of components delivered for
production deployment
Software/service provider rating of our integration
assistance
Responsiveness to defect and support requests
4 of 5
6
4 of 5
45 days or less
Sub-goal Supported
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
Because capabilities can vary in complexity and may contain few or many gaps, the target for
“number of capability delivery plans prepared for prioritized Use Cases“ is a modest target of 7 new
CDPs in PY6. As new XSEDE Use Cases are documented and prioritized, RACD will address as many
as they can, given their current staffing levels and the availability of software partners. RACD has
set a goal to engage with software partners to integrate at least 6 significant new components or
significant expansion of existing components.
38
RACD’s goal is that users and software/service operators will rate components 4 out of 5, and that
external software/service providers will also rate the assistance provided them 4 out of 5. This will
be an interesting target since many components will be integrated by software partners that RACD
works with, so this target will be affected directly by the quality of the products, the engagement of
the software partners, and RACD’s ability to assist those software partners in raising their quality
and engagement.
Once components are integrated and in use, RACD aims to facilitate timely support and
maintenance requests by informing users of the solutions to issues in 45 days or less. Some
problems may be resolved much more quickly, but this goal factors the time needed to investigate
and resolve more complex issues.
RACD will focus on Use Cases given high and medium-high priority by the UREP in early 2016, and
on core capabilities from the proposal. RACD will use CDPs to identify capability gaps and select
activities based on feasibility, user impact and resource availability.
The following activities are a continuation of activities from PY5 and have been prioritized by the
UREP for work in PY6. RACD will implement XSEDE’s group management services using Globus.
Work will continue on InCommon-based authentication that will allow for the ability to
authenticate to XSEDE resources and services using federated identities and two factor, web single
sign-on, and unified and extensible group manage services. Also, this will allow the registration of
an XSEDE identity provider with InCommon, under XSEDE's existing InCommon membership so
that XSEDE identities and attributes can be used to interoperate with other federated services in
the US (via InCommon) and globally (via eduGAIN). RACD will work to provide and access resource
information that will enhance level 1, 2, and 3 resources, software, and services publishing and
discovery and also implement science gateway software and service publishing and discovery.
For PY6, new activities include enabling discovery of XSEDE data analytics resources, software, and
usage examples and assisting science gateways in leveraging large data movement capabilities
delivered in the previous years of the project. Also, capabilities will be provided to select, clean,
supplement, integrate, format, and model data. Users should be able to gather from local or remote
sources, manipulate, translate and organize their data as needed for data mining, modeling or
analysis activities. Other XSEDE services besides the User Portal, such as Science Gateways, will be
able to use XSEDE identity and group management services. RACD will explore opportunities to
enhance traditional access to one or multiple HPC resources. It will build virtual clusters and stock
XSEDE environment virtual machines (VMs) to enable XSEDE component evaluation and testing all
while promoting a growing ecosystem of XSEDE integrated software and services using the XSEDE
Community Software Repository (XCSR).
XSEDE will no longer fund maintenance and enhancement to the XSEDE Wide File System (XWFS),
Genesis II, and UNICORE. However, Genesis II and UNICORE should continue to be made available
in XSEDE by the software providers and by resource operators.
5.3 XSEDE Capability & Resource Integration (WBS 2.3.3)
The mission of the Capability & Resource Integration (CRI) team is to work with SPs, CI providers
and campuses to maximize the aggregate utility of national cyberinfrastructure. CRI will coordinate
interactions between SP’s and XSEDE in the SP Forum, engage with national CI providers, gather
39
requirements for tools and training that assist users of XSEDE and other CI, identify and
disseminate benefits and costs of interoperating with XSEDE and create toolkits that fit and
improve common usage modalities.
CRI’s activities are reflected in the uptake of CRI-integrated toolkits, such as the XSEDE Campus
Bridging Cluster toolkit and XSEDE National Integration Tookit, but also Globus Transfer clients and
other toolkits as developed. The Area Metrics reflect users of CRI toolkits as well as integration of
toolkits on Level 1 SP resources and the aggregate CI (in Teraflops) taking advantage of CRI tools.
The number of systems using one or more CRI provided toolkits includes systems using the Globus
Transfer software, the XSEDE National Integration Toolkit (XNIT), and XSEDE Campus Bridging
Cluster (XCBC), as well as any new toolkits developed by CRI. The number of Level 1 total systems
that incorporate tools reflects the level of adoption of CRI toolkits within the SP community. Thanks
to incorporating SP Forum coordination, CRI hopes to get SP input on requirements for potential
new toolkits, which paves the way towards a broader CI utility set. The number of repository
subscribers indicates the number of campus sites which receive regular updates from the XSEDE
Community Software Repository, indicating regular and prolonged use of the XNIT. Aggregate
Teraflops of cluster systems reflects the uptake of both the XNIT and XCBC software and gives some
measure of the overall capability provided by these toolkits. Metrics for partnership interactions
reflect sustained work on the part of CRI with other CI providers, campuses, and SP representatives
to be responsive to a broad set of needs.
Table 5-3: Area Metrics for XSEDE Capability & Resource Integration
Area Metric
Total number of systems that use one or more CRI
provided toolkits
Number of Level 1 total systems that fully incorporate all
of the recommended tools from the XSEDE Community
Repository
Number of repository subscribers to SCRI cluster and
laptop toolkits
Aggregate number of TeraFLOPS of cluster systems using
SCRI toolkits
Number of partnership interactions between XCRI and
SPs, national CI organizations, and campus CI providers
PY6
Target
450
5
150
1000
12
Sub-goal Supported
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
Advance - Create an open and
evolving e-infrastructure (§2.2.1)
Deepen/Extend – Deepen use to
existing communities (§2.1.1)
Targets for CRI activities reflect a level of impact that allows individual campus CI providers to
contribute to the aggregate infrastructure leveraging CRI toolkits in a meaningful way.
CRI will continue to package scientific software for the existing set of toolkits and provide site-visit
activities in order to assist campuses with integrating with the national cyberinfrastructure. CRI
will review curriculum for and provide two additional webinars for learning the cluster toolkits.
CRI will also continue attending outreach events to describe CRI toolkit capabilities and services.
Some additional outreach is expected in order to explain the transition between XSEDE Campus
Bridging and CRI.
40
CRI will engage in requirements gathering with SPs, campus providers, and end users to select tools
to be incorporated in additional toolkits for enhancing ease of access to CI resources. High on the
list are tools for incorporating InCommon authentication and using XSEDE allocations to
enhance/parallel campus allocation systems.
41
6 XSEDE Operations (WBS 2.4)
The mission of XSEDE Operations is to install, connect, maintain, secure, and evolve an integrated
cyberinfrastructure that incorporates a wide range of digital capabilities to support national
scientific, engineering, and scholarly research efforts.
In addition to the Operations Director’s Office (§6.1), Operations staff is subdivided into four teams
based on the work breakdown structure: Cybersecurity (§6.2), Data Transfer Services (§6.3),
XSEDE Operations Center (XOC) (§6.4), and Systems Operational Support (§6.5). The Operations
management team meets weekly and individual Operations groups meet approximately bi-weekly
with all meeting minutes posted to the XSEDE wiki.
XSEDE Operations has three Area Metrics: (1) average composite availability (as a geometric mean)
of both critical central services and the XSEDE Resource Allocation Service (XRAS) (2) hours of
downtime of critical central services or multiple SPs resulting from a security incident with direct
user impact, and (3) mean time to resolution of user (non-staff) tickets.
Maintaining and evolving an integrated cyberinfrastructure requires the availability, reliability, and
security of digital resources. To that end, XSEDE Operations is charged with monitoring a number of
hardware and software services.
Along with monitoring and maintaining resources, the XOC provides XSEDE with effective and
efficient user support by responding to service requests from end users and staff. All user (nonstaff) service requests handled by the XOC and WBS ticket queues comprise this metric.
Table 6-1: Area Metrics for Operations.
Area Metric
Average composite availability of core
services (geometric mean of critical
services and XRAS)
Hours of downtime with direct user
impacts from an XSEDE security incident.
Mean time to ticket resolution (hrs)
PY6 Target
Sub-goal Supported
99%
Sustain - Provide reliable, efficient, and secure
infrastructure (§2.3.1)
<24
Sustain - Provide reliable, efficient, and secure
infrastructure (§2.3.1)
Sustain - Provide excellent user support
(§2.3.2)
<24
The composite availability of core XSEDE services is a key measure of the health of the entire
cyberinfrastructure. Each of the components of this composite (the geometric mean of critical
enterprise services and the XRAS account management service) has its own availability target.
When the targets for each of these services are met, the average composite availability percentage,
measured as a geometric mean, is at least 99%.
Ensuring user availability of resources also requires diligence with respect to maintaining secure
hardware and software. Mitigating any potential security incidences that might affect availability is
paramount. Therefore, Operations has set a target of 24 hours or less for downtime of central
services or multiple SPs with direct user impact.
42
Similarly, the target for handling user tickets is a resolution time of 24 hours or less for service
requests. This metric does not apply to Service Provider ticket queues or to internal tickets issued
to XSEDE staff or other XSEDE funded individuals.
6.1 Operations Director’s Office (WBS 2.4.1)
The Operations Director’s Office has been established to provide the necessary oversight to ensure
the greatest efficiency and effectiveness of the Operations area. This oversight includes providing
direction to the L3 management team, coordination of, and participation in, Operations planning
activities and reports through the area’s Project Manager. The Director’s Office also attends and
supports the preparation of project level reviews and activities.
The Operations Director’s Office will continue to manage and set direction for Operations activities
and responsibilities. They will contribute to and attend bi-weekly senior management team calls,
contribute to the project level plan, schedule, and budget, contribute to XSEDE quarterly, annual,
and other reports as required by the NSF and attend XSEDE quarterly & annual meetings. The
Director’s Office will also provide analysis of network traffic and user help ticket data to improve
capabilities, service, and/or metrics.
6.2 Cybersecurity (WBS 2.4.2)
The Cybersecurity Security (Ops-Sec) group protects the confidentiality, integrity and availability of
XSEDE resources and services. The Area Metric for the Cybersecurity group is the percentage of
time that critical XSEDE resources are unavailable due to a security incident.
Downtime resulting from security incidents has a direct impact on the availability of critical XSEDE
resources to users and is the key evaluative measurement of the Cybersecurity group’s efforts.
Users expect XSEDE resources to be reliable and secure, thus the security team’s goal is to minimize
any interruption of services related to a security event.
Table 6-2: Area Metrics for Cybersecurity
Area Metric
Hours of downtime with direct user impacts from
an XSEDE security incident.
PY6 Target
<24
Sub-goal Supported
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
The goal is to have no service interruptions to critical central services and multiple SPs as a result
of a security incident and, should one occur, to restore service quickly. Thus, a less than 24-hour
downtime limit was chosen as the area metric target. The XSEDE security program accomplishes
this through the use of policy, technology, and awareness activities detailed below. Collectively,
each of these activities contribute to XSEDE’s cybersecurity profile and thus each supports the subgoal of providing a reliable and secure infrastructure.
The Ops-Sec group will continue to perform vulnerability scanning and mitigation. It is important
that Service Providers (SPs) identify and remediate any threats that occur from system
vulnerabilities in order to provide reliable and secure infrastructure. A vulnerability that is
exploited at one SP can have direct impact on other SPs. To achieve this goal, the security team will
43
continue this successful activity, which includes weekly scanning and ticket resolution for all
central services. Vulnerability resolution is one of the cybersecurity metrics.
The team will continue to maintain policies, standards and guidelines. In order to ensure a secure
virtual organization, all parties must be aware of a basic set of principles to ensure the security of
the XSEDE project. To achieve this goal, the security team will continue to develop new policies,
standards, and guidelines and maintain existing ones as needed. A measurement of success is how
often these are refreshed and how well they are understood in training quizzes.
They will also update and advance security training. User and staff are instrumental in providing a
secure virtual organization for XSEDE. Training and awareness are required to communicate and
educate individuals for supporting XSEDE’s security goals. An information security training session
will be presented at each XSEDE annual conference. In conjunction with the XSEDE Training group,
training material will be developed for a series of online security courses.
Ops-Sec will perform Incident Response (IR) drills. Rapid response to a security event is a critical
part of providing a reliable and secure infrastructure. The added complexity of using encrypted
communication can cause delays in response if they are not used effectively. To support these
efforts of fast and secure communication for incident response, the use of unannounced response
drills will be conducted. Clearly, the faster the response time to these drills the more effective and
successful they become.
Support will be provided for two-factor authentication development and deployment for users.
Compromised user accounts are a top security issue for the response team and the number one
reason a user’s account may be disabled. Users suffer when either their account is disabled or a
resource is made unavailable due to an exploit. The addition of two-factor authentication can
combat cases of credential reuse, lowers risk, and directly improves Ops-Sec’s ability to provide a
secure infrastructure. A successful measurement of this activity would be the continued rollout of
two-factor to XSEDE services. Ops-Sec will work with the XSEDE Community Infrastructure (XCI)
group to drive service requirements to the two-factor rollout and the External Relations group to
disseminate awareness and training.
Oversight will continue in regards to XSEDE incident response, operational security, and
operational security. The ability to identify and respond to ongoing threats is critical for sustaining
a secure infrastructure and advancing the overall security posture of XSEDE. This involves meeting
preparation and planning to coordinate effectively. It also requires information sharing and
planning for the response and analysis to critical new vulnerabilities. Success can be measured by
effectively and efficiently defending threats so that there is no impact to XSEDE services.
The Ops-Sec team will operate XSEDE’s Certificate Authorities (CA) and CILogon services.
Certificate authorities play a crucial role in the identification process and the CILogon service works
in conjunction with the XSEDE authentication system to connect XSEDE with other user
communities and identity providers. Maintaining each of these security services with an availability
of 99% without incident qualifies as a success. The team will all maintain, approve and securely
distribute the list of trusted CA root certificates that XSEDE accepts. The integrity of this list is key
to XSEDE’s trust fabric.
44
New activities of the Ops-Sec team include federated intelligence sharing, audit security compliance
and documenting lessons learned on cloud computing. Previously, Ops-Sec explored the sharing of
intelligence data in real time between SPs to advance the ability to identify and respond to threats
in order to provide a reliable and secure infrastructure. Ops-Sec is now deploying a federated
intelligence sharing strategy based on these efforts, first within XSEDE and then more broadly with
international partners. The first iteration of XSEDE did well to establish strong security standards
for central services and guidelines for SPs. Beginning this project year, much of the effort previously
spent developing those standards moves to auditing and assessing their effectiveness. Cloud
computing is becoming common for computational science and engineering now and with it comes
many of the challenges of multitenant systems that give root access to users who generally are not
professional system administrators. Keeping these systems secure and from causing harm to other
tenants or netizens is a challenge now being faced by all SPs in one way or another. Bridges and
Jetstream are the first such systems in XSEDE and Ops-Sec will document lessons learned to help
future XSEDE partners and others in the management of these systems.
6.3 Data Transfer Services (WBS 2.4.3)
The Data Transfer Services (DTS) group facilitates data movement and management for the
community by maintaining and continuously evolving XSEDE data services and resources. The Area
Metric for the DTS group is the performance (Gbps) of intra-XSEDE GridFTP transfers > 1GB. The
metric for data transfer performance on large files reflects the ability to consistently provide high
performance data movement facilities.
Table 6-3: Area Metrics for Data Transfer Services
Area Metric
Performance (Gbps) of instrumented, intra-XSEDE
transfers > 1GB
PY6 Target
1 Gbps
Sub-goal Supported
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
The GridFTP performance Area Metric represents the ability of the GridFTP server configurations
at the service providers to take advantage of the XSEDE network capabilities and the performance
of the underlying file systems. Calculating the average performance of all transfers larger than 1GB
between XSEDE endpoints generates the Area Metric. This eliminates the highly variable
performance associated with very small transfers and provides a highly accurate reflection of the
overall user experience in transferring data within XSEDE. The initial target value is 1Gbps, or 1/10
the XSEDE network capacity, based on the expected capabilities of the XSEDE endpoints.
The DTS team will continue to maintain domain name services as this is a critical function for both
Public Key Infrastructure and host name resolution for XSEDE-wide services. When down, critical
XSEDE functions are inaccessible. XSEDE will maintain DNS availability of at least 99% to ensure a
reliable and secure infrastructure. The DTS team will also monitor and maintain data transport
services. The Data Transfer Services team is responsible for various different software and
hardware components that comprise the end-to-end data movement and networking services that
it supports. These services, which include GridFTP and XSEDEnet, must be monitored, maintained,
and updated to provide maximal value to the XSEDE community. The health of XSEDEnet is verified
by monitoring link state, utilization, error rates, and availability; GridFTP services are monitored
45
through analysis of logging and configuration data for the data transfer nodes (DTNs) of XSEDE
resources. Ensuring the health and functionality of these services is key to the DTS team’s effort to
providing a reliable and secure infrastructure.
Providing expert consulting and optimization improves the end-to-end performance of networkbased applications and thus DTS will continue with end-to-end performance tuning. Ensuring that
networking technologies and capabilities are effectively integrated and utilized end-to-end
provides a more efficient and reliable experience for XSEDE users. For example, file transfers larger
than a terabyte can suffer from poor throughput for a variety of reasons. By tuning the networkrelated configuration of hosts, services, and infrastructure, performance and user experience can be
improved. The number of network performance related tickets resolved and end-to-end
performance analyses performed will be monitored to measure success and ensure excellent user
support is being provided.
DTS will maintain data and network services monitoring tools. Regular maintenance of the tools
used to monitor the network ensures that monitoring capabilities are accurate and reliable. The
availability of these tools, including perfSONAR, and their reporting functions will be the measure of
success and will help measure the reliability of XSEDEnet.
DTS staff will regularly engage with Intenet2 administrators and engineers to remain current with
Internet2’s capabilities and to keep them apprised of Use Cases and expectations. The DTS team will
engage with the XSEDE user community to ensure that their interests are represented via XSEDE to
Internet2. This is vital for maintaining the effective utilization of the XSEDEnet infrastructure. Open
communications of XSEDE’s expectations helps to provide a reliable and secure infrastructure for
XSEDE users.
The annual XSEDE Conference is the premiere event for XSEDE staff to engage with the broader
community and a reliable and effective conference network is key to attendees’ satisfaction with
the event. DTS staff provide networking support for the conference, including visits to the
conference hotel and collaborating with hotel staff to configure and optimize the hotel's network
infrastructure to better accommodate the conference's networking needs.
In PY6, the DTS team will investigate and develop new data transport methods and services, engage
the community on data transport challenges, and will design and implement a scalable XSEDEnet
architecture. As data volumes and network speeds increase, more emphasis is placed on protocols
and techniques that can improve both the data transfer efficiency and user experience. The DTS
team will investigate and test new developments in this space, working to incorporate new services
and capabilities that will enhance the user experience and improve the DTS team’s ability to
monitor and maintain its services, thus supporting the XSEDE goal to create and open and evolving
e-infrastructure.
It is not uncommon for problems with data movement to go either unnoticed or unreported by end
users. In general, this is due to a lack of awareness of what the typical user experience should be or
a lack of awareness of the tools and services that are available to the community. The DTS team will
work to engage with the XSEDE community to raise awareness of these tools and services and to
join the national conversation on data transport and raising the bar for researchers in the
community. Doing so supports the goal of raising awareness of the value of advanced digital
services.
46
In order to continually meet the needs of XSEDE and the growing number of participating sites,
XSEDEnet must have an efficient and scalable network architecture that can easily incorporate
additional service provider sites. Designing and implementing a new architecture thus supports the
goal of creating an open and evolving e-infrastructure.
DTS will no longer maintain or improve the XSEDE-Wide File System (XWFS) as the XWFS is being
retired. It will also no longer deploy the Global Federated File System (GFFS) as this has been
discontinued.
6.4 XSEDE Operations Center (WBS 2.4.4)
The XSEDE Operations Center (XOC) serves as user advocates, providing timely and accurate
assistance to the XSEDE community, while simultaneously monitoring and troubleshooting userfacing systems and services.
The mean time to ticket resolution (MTTR) for XSEDE user service requests in the XOC queue
demonstrates how responsive and efficient the XOC is when handling and resolving tickets. This
metric does not apply to WBS ticket queues, Service Provider ticket queues or to internal tickets
issued to XSEDE staff or other XSEDE funded individuals.
Table 6-4: Area Metrics for XSEDE Operations Center
Area Metric
Mean time to resolution of user tickets in XOC
queue
PY6 Target
< 24
Sub-goal Supported
Sustain - Provide excellent user
support (§2.3.2)
The XOC is necessary to monitor and maintain resources and, along with separate monitoring of
WBS ticket queues, provides XSEDE with effective and efficient user support. End-users expect for
their issues to be responded to within a reasonable amount of time. The average MTTR for user
(non-staff) tickets in the XOC ticket queue is the Area Metric for the XOC, and a time of less than 24
hours is a reasonable target, both from feasibility and effective operational standpoints. All XOC
activities support the XSEDE sub-goal of providing excellent user support.
The XOC will continue to maintain the ticket system and call center. The XOC functions as a first
level of support for issues reported by XSEDE users and the staff community via the XSEDE ticket
system, and will attempt to provide a resolution if possible. All other issues are routed
appropriately to service providers and other groups within the WBS. The XOC will field calls from
XSEDE users and staff via a dedicated toll-free number. The call center functions much like the
ticket system: XOC staff will attempt to resolve user issues and answer questions over the phone.
For complicated issues that require expert assistance from service providers, XOC staff will create
tickets and assign them appropriately. Phone assistance is of particular importance around XRAC
submission deadlines and quarterly allocation expiration dates; XOC staff are familiar with many
details within allocation processes and can advise and assist users facing deadlines. The XOC will
maintain and improve its routing documentation that supports ticket system and call center
activities. This includes the incorporation of new systems, software, and user-facing services.
47
The XOC will monitor mission-critical user-facing systems and services using Nagios and other
tools. Incidents will be investigated, documented, and the appropriate support personnel will be
contacted.
The XOC will discontinue providing 24-7 service as it did during the first years of the XSEDE project.
Instead, XOC will now operate during all hours except for the following (all times in Central Time
Zone): Friday 11 p.m. to Saturday 7 a.m., Saturday 11 p.m. to Sunday 7 a.m., and Sunday 11 p.m. to
Monday 7 a.m.
6.5 System Operations Support (WBS 2.4.5)
Systems Operational Support (SysOps) provides enterprise level support and system
administration for all XSEDE central services. The Area Metric for the SysOps group is the
percentage of time that enterprise XSEDE services (a geometric mean) are available.
The availability metric shows the amount of time that the critical enterprise services (Tier 1) are
available for use. Services are grouped into criticality tiers based on their significance and
dependence that other services have on them. Tier 1 systems are systems such as Kerberos, XDCDB
(XSEDE Central Database), and DNS (Domain Name Service). Tier 2 are business-day supported
services such as JIRA. Tier 3 are development supported systems such as the RT test instance. When
calculated as a weighted geometric mean, the availability metric will show the cumulative impact of
the enterprise services to the stakeholders.
Table 6-5: Area Metrics for System Operations Support
Area Metric
Average availability of critical enterprise services
(%) [geometric mean]
PY6 Target
99%
Sub-goal Supported
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
Providing a reliable infrastructure requires that central service resources are functioning and
dependable. The aggregate availability percentage Area Metric of 99% for critical enterprise
services was chosen based on the expectation of providing superior uptime for deployed resources.
SysOps will continue to maintain and deploy XSEDE Enterprise servers and services. Regular
maintenance of enterprise servers/services is critical to providing a reliable and secure
infrastructure. System refreshes and innovation shows growth and intentionality toward providing
a reliable and secure infrastructure. The team will also provide real time solutions to monitor the
health and availability of XSEDE central services using Nagios and other software. This software
will also be used to automate, as much as possible, on demand and quarterly reports. SysOps will
continue to implement XSEDE Staff OTP to ensure proper authentication and protection
mechanisms.
SysOps will operate and further develop the INCA monitoring framework. INCA is an existing
system and needs to be maintained and further developed to ensure that it is keeping pace with the
changing XSEDE landscape. This is in support of providing a reliable and secure infrastructure.
48
An annual XDCDB failover test will be conducted. This activity ensures that the XDCDB failover
processes are audited and familiarized among administrators to ensure continuity if/when there is
a need to failover the XDCDB, which also supports providing a reliable and secure infrastructure.
New activities for SysOps in PY6 include XDCDB future planning and a biannual XSEDE central
services failover audit/test. Planning for the changing landscape and interviewing stakeholders has
shown that the future of the XDCDB may need to physically change locations and/or
implementations. Collaborations will take place between the Accounts and Account Management
and SysOps groups.
Operation of the XSEDE Operations Center was moved out of SysOps and into its own area for PY6.
Sciforma support is being discontinued as the software is being retired.
49
7 Resource Allocation Service (WBS 2.5)
The Resource Allocation Service (RAS) will build on XSEDE’s current allocation processes and
evolve to meet the challenges presented by new types of resources to be allocated via XSEDE, new
computing and data modalities to support increasingly diverse research needs, and large-scale
demands from the user community for limited XSEDE-allocated resources. RAS will accomplish its
objectives through three activities: managing the XSEDE allocations process in coordination with
the XD Service Providers, enhancing and maintaining the RAS infrastructure and services, and
anticipating changing community needs.
The area metrics for RAS focus on support for two of XSEDE’s key sub-goals: providing excellent
user support and providing reliable, efficient and secure infrastructure. An updated quarterly
survey of users who have interacted during the quarter with the allocations request system and the
allocations process more generally will be used to determine the user satisfaction ratings metrics.
The survey will be developed with and coordinated through the XSEDE Evaluation Team. The
availability of XDCDB and XRAS will be measured by tracking the planned and unplanned outages
for the XDCDB server on which XRAS and many other XSEDE services rely.
Table 7-1: Area Metrics for Resource Allocation Service.
Area Metric
PY6 Target
User satisfaction with XRAS system
4 of 5
User satisfaction with allocations process
4 of 5
Availability of XRAS
95%
Sub-goal Supported
Sustain - Provide excellent user support
(§2.3.2)
Sustain - Provide excellent user support
(§2.3.2)
Sustain - Provide reliable, efficient, and secure
infrastructure (§2.3.1)
The targets for user satisfaction represent an average respondent rating of 4 or greater on a scale of
1–5 (very dissatisfied to very satisfied) across a set of questions related to aspects of user
interaction with the allocations process, such as documentation clarity, effort required, and process
transparency, and with the XRAS interfaces, such as responsiveness, clarity of the interface, and
ease of use. The target for XDCDB/XRAS availability of 95% reflects the XSEDE staffing levels
available to address severe, unplanned outages; in most cases it is expected to surpass this target.
7.1 RAS Director’s Office (WBS 2.5.1)
The RAS Director’s Office has been established to provide the necessary oversight to ensure the
greatest efficiency and effectiveness of the RAS area. This oversight includes providing direction to
the L3 management team, coordination of, and participation in, RAS planning activities and reports
through the area’s Project Manager. The Director’s Office also attends and supports the preparation
of project level reviews and activities.
The RAS Director’s Office will continue to manage and set direction for RAS activities and
responsibilities. They will contribute to and attend bi-weekly senior management team calls,
contribute to the project level plan, schedule, and budget, contribute to XSEDE quarterly, annual,
and other reports as required by the NSF and attend XSEDE quarterly and annual meetings.
50
7.2 XSEDE Allocations Process & Policies (WBS 2.5.2)
Allocations enable the national open science community to easily gain access to XSEDE’s advanced
digital resources, allowing them to achieve their research and education goals. In support of the
goal “Sustain – Provide excellent user support,” Allocations has two area metrics: user satisfaction
with the allocations process and the average time to process Startup requests. Allocations also
supports the goal “Provide reliable, efficient, and secure infrastructure” by measuring the percentage
of XRAC-recommended service unit (SUs) allocated, which is a third Area Metric for this group. This
third metric provides insight for RAS, the XSEDE program, NSF, and Service Providers into the
success of the XSEDE ecosystem at meeting users’ scientific objectives.
Table 7-2: Area Metrics for XSEDE Allocations Process & Policies
Area Metric
PY6 Target
User satisfaction with allocations process
4 of 5
Average time to process Startup requests
2 weeks or
less
100%
Percentage of XRAC-recommended SUs allocated
Sub-goal Supported
Sustain - Provide excellent user
support (§2.3.2)
Sustain - Provide excellent user
support (§2.3.2)
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
Average user satisfaction with the allocations process will be measured by a targeted user survey
following each of the quarterly XRAC review meetings and after award notifications have been sent.
This Area Metric is augmented by two questions in the XSEDE annual user survey. The team’s goal
is to achieve an average of 4 on a scale from 1-5; an average of 4 would indicate that the user
community is generally satisfied with the process. The survey also provides the opportunity to
solicit feedback to identify and improve any areas that receive an unsatisfactory rating.
Each quarter, the team will average the time from submission to sending the outcomes to the
relevant Service Providers for the startups completed in the quarter. The times are recorded in the
XRAS database. Two weeks represents the longest expected time for the RAS staff and XSEDE
reviewers to consider and respond to the more than 150 requests received each quarter.
For each quarterly research opportunity, the Allocations team records the allocations
recommended by the XRAC panel and the final allocations after the post-meeting “reconciliation
process” to have the allocations fit within the resources available. From this the percentage from
the total computing SUs awarded and total recommended is calculated.
The L3 manager provides area oversight, reporting and planning, and plans and participates in
quarterly meetings and NSF reviews. The group sets the XRAC review schedule, recruits reviewers
as needed, gathers SP allocation commitments, hosts two request-writing training webcasts each
quarter, plans meeting logistics (selecting venues, reimbursing reviewers’ travel costs, etc.),
conducts the meeting, and processes the requests and awards. In addition, the group fields user
tickets and inquiries, processes startup and educational new/renewal requests, and handles
51
allocation management requests (transfers, supplements, extensions). The group will continue its
support for XSEDE-wide reporting and ad-hoc query support related to allocations request info, etc.
The group will conduct a review and update of allocations documentation, resource information,
and user "on-boarding" process, based on results of the PY5 allocation policy review. The group will
also implement the policy and practice changes resulting from the PY5 allocation policy review. The
group will expand its efforts in providing data analysis and reporting to aid XSEDE decision-making
and enhance the area’s analysis capabilities.
7.3 Allocations, Accounting, & Account Management CI (WBS 2.5.3)
The Allocations, Accounting & Account Management CI (A3M) group maintains and improves the
interfaces, databases and data transfer mechanisms for XSEDE-wide resource allocations,
accounting of resource usage, and user account management. The Area Metrics for A3M are
satisfaction results from surveys on the use of the XSEDE Resource Allocation Service (XRAS)—the
primary user-facing service for RAS—and the aggregated availability of XDCDB and XRAS systems,
and the percentage of approved feature change requests implemented. The activities in A3M
support the XSEDE sub-goals of providing a reliable and secure infrastructure, and creating an open
and evolving e-infrastructure.
Table 7-3: Area Metrics for Allocations, Accounting, & Account Management CI
Area Metric
User satisfaction with XRAS system
PY6 Target
4 out of 5
Availability of the XRAS systems
95%
Percentage of approved feature change requests
implemented
100%
Sub-goal Supported
Sustain - Provide excellent user
support (§2.3.2)
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
Sustain - Provide reliable, efficient,
and secure infrastructure (§2.3.1)
A3M provides critical services to the XSEDE program and Service Providers through the XSEDE
central database (XDCDB), the Account Management Information Exchange (AMIE) service, on
which the XRAS system depends. The aggregate availability of these two critical systems is an
additional Area Metric for the group and has a target of 99%, matching the target Area Metric value
for aggregated availability of the key enterprise central services in XSEDE.
The target for user satisfaction with XRAS represent an average respondent rating of 4 or greater
on a scale of 1–5 (very dissatisfied to very satisfied) across a set of questions related to aspects of
user interaction with the XRAS interfaces, such as responsiveness, clarity of the interface, and ease
of use.
The percentage of approved feature change requests implemented is a new metric which was
intended to reflect the efficiency of the A3M team. Neither a formal definition nor metric calculation
has been finalized. A3M will be examining how to provide relevant data or refine the metric to
reflect appropriate aspects on the performance of the A3M team.
The L3 manager provides area oversight, reporting, participating in quarterlies, planning, NSF
review preparation and attendance. Based on user feedback (tickets, etc.), the A3M team maintains
52
the XRAS submit, review, and admin interfaces and services, as well as reports, fixes, and tests
system issues. The team regularly gathers stakeholder input and user feedback, contributes and
prioritizes requirements and enhances the XRAS submit, review, and admin interfaces and services
and tests new features and document modifications. The team maintains the existing accounting
and account management (A&AM) systems and databases; maintains the AMIE system; maintains
the accounting service API for XRAS and XUP; and maintains relevant documentation for A3M
services.
The A3M team also provides essential database administration services to support and optimize the
more than 20 schemas in XDCDB which provide critical foundations for several XSEDE User Portal
services including user profiles, user publications and user news, key XSEDE-wide metrics
information, and all of the A3M services. The team also supports primary and backup server
hardware and infrastructure integrity (maintenance, UPS, etc.).
The A3M manager will lead RAS engagement in evaluating architectural options, stakeholder
requirements, load testing, performance impacts, and growth planning for the XDCDB, in
coordination with XSEDE Ops and XCI. The A3M manager will also work with stakeholders on
developing an XRAS sustainability and cost model.
The team will take on maintenance and support of the XSEDE Resource Description Repository
(RDR) following its production deployment at the end of PY5. The team will also be responsible for
enhancing the RDR including completion of the integration of the XRAS, A&AM, and XSEDE User
Portal components with the RDR.
In PY6, the team will implement changes to XRAS as required by the outcomes of the allocation
policies review process conducted in PY5. As workload permits, the team will begin work on tasks
to enhance the existing accounting and account management (A&AM) database, enhance the
accounting service API for XRAS and XUP, implement improvements to A&AM processes and
provide enhanced A&AM admin capabilities. Specific examples for PY6 include adding interfaces to
allow Campus Champion, Gateway, NIP and Allocations areas to manage and vet specially tracked
entities in the XDCDB.
53
8 Program Office (WBS 2.6)
The purpose of the Program Office is to ensure the critical project level functions are in place and
operating effectively and efficiently. The oversight, provided via the Project Office, is necessary to
provide consistent guidance and leadership to the L3 managers across the project. A common and
consistent approach managing projects and risks is provided by the Project Management, Reporting
and Risk Management (PM&R) team, while Business Operations manages all financial and subawards. The crucial aspect of communications to all stakeholders is the focus of the External
Relations team. Finally, Strategic Planning, Policy and Evaluation focuses attention in precisely
those areas to ensure the best possible structure continues to exist within XSEDE to allow the
support of all significant project activities and enable efficient and effective performance of all
project responsibilities.
As with other areas of the project, the Program Office has established Area Metrics to track
performance against attaining the project’s strategic goals. The Area Metrics for “Social Media,” and
“Media Hits” have been separated as a way to show the impact of each primary activity with one
number best representing the performance each quarter and year. “Social Media” refers to
“impressions,” which is more comprehensive than just “followers” or “likes,” but includes the
amount of people who may see content based on people sharing XSEDE’s content to their friends
and followers on both Facebook and Twitter. “Media Hits” refers to the number of stories that were
shared or written by external print or digital publications.
To deliver the most advanced and useful digital services for researchers, XSEDE must constantly
evolve. This makes continuous evaluation and improvement of processes and policies critical to the
effectiveness of XSEDE as an organization; therefore, as tracked by the PM&R team, the project’s
KPI for sustaining an effective virtual organization is the number of improvements made. Potential
improvements consist of recommendations based on the staff climate survey, efforts by area
managers to increase efficiency or quality, adaptations in response to an evolving project, or
improvements to overall execution through implementation of widely validated methodologies.
Measuring innovation for an organization like XSEDE (or for organizations in general) is difficult
and represents an area of open research. A partial measure is the number of staff publications
produced since this shows that XSEDE staff is involved in novel activities that achieve peerreviewed publication. Additionally, after much thought and discussion both internally and with
external stakeholders and advisors, XSEDE has identified two additional indicators that strongly
correlate to innovation: 1) the ratio of proactive organizational improvements to those that were
reactive and 2) the number of improvements that are innovative or lead to innovations. The first
indicator is a measurement of organizational maturity and agility; the second measures innovative
actions directly. This will continue to be an open conversation within the organization and with
XSEDE’s stakeholders and advisors as XSEDE assess these measurements and how to best to qualify
innovation.
54
Table 8-1: Area Metrics for Program Office
Area Metric
Number of Social Media impressions
PY6 Target
190,000
Number of media hits
140
Percentage of recommendations
addressed by relevant project areas
Number of strategic or innovative
improvements
Ratio of proactive to reactive
improvements
Number of staff publications
90%
9
3
70
Sub-goal Supported
Deepen/Extend - Raise awareness of the value
of advanced digital services (§2.1.4)
Deepen/Extend - Raise awareness of the value
of advanced digital services (§2.1.4)
Sustain – Operate an effective and productive
virtual organization (§2.3.3)
Sustain – Operate an innovative virtual
organization (§2.3.4)
Sustain – Operate an innovative virtual
organization (§2.3.4)
Sustain – Operate an innovative virtual
organization (§2.3.4)
The “Social Media” and “Media Hits” targets were chosen using results from the previous year as a
baseline. Tools within Facebook and Twitter are used to look at these trends and historical data is
available for each of these. The target of nine strategic or innovative improvements was chosen as a
modest target. This metric was only tracked in PY5 and thus the historical data is not there to
support a trend. The team will continue to refine this target as more data is collected. It is also
expected to see less reactive and general improvements as the project makes use of systematic
processes to identify needed improvements and thus have set the ratio of proactive to reactive ratio
as 3. Over the past five years, there has been a slightly decreasing trend in staff publications and the
target of 70 was selected to try to change this trend. However, the number of staff publications is
often dependent on the researchers we work with. Many of those publications are outcomes of
ECSS projects and, thought we encourage it, various researchers have varying opinions on including
ECSS staff as authors on their papers.
8.1 Project Office (WBS 2.6.1)
The Project Office has been established to provide the necessary oversight to ensure the greatest
efficiency and effectiveness of the Program Office area and to establish responsibility for assuring
advisory activities of the project occur. This oversight includes providing direction to the L3
management team and coordination of and participation in Program Office planning activities and
reports through the area’s Project Manager. The Project Office also attends and supports the
preparation of project level reviews and activities. Importantly, the Project Office is also
responsible for assuring that the XSEDE Advisory Board, the User Advisory Committee, and the SP
Forum are functioning. The Project Office is also responsible for coordination of project-level
meetings such at the bi-weekly Senior Management Team (SMT) teleconference calls and the
project quarterly meetings.
With the transition to a new organizational structure for PY6, the makeup of the Senior
Management Team (SMT) has changed. The XSEDE SMT is the highest-level management body in
the organization. It is chaired by the Program Director (PI Towns), and includes the WBS Level 2
directors of Community Engagement & Enhancement (co-PI Gaither), the Extended Collaborative
55
Support Service (Co-PI Wilkins-Diehr and Co-PI Roskies), XSEDE Community Infrastructure (Lifka),
XSEDE Operations (Peterson), the Resource Allocations Service (Hart) and the Program Office
(Payne). In order to be responsive to both the user community and the set of collaborating SPs, the
chairs of the User Advisory Committee (currently Emre Brookes, University of Texas Health Science
Center at San Antonio) and the XD Service Providers Forum (currently Dan Stanzione, University of
Texas at Austin) are members of the SMT. These ten individuals constitute the voting members of
the SMT. The Senior Project Manager (Gendler) is an ex officio member of the SMT. The cognizant
NSF Program Officer (Eigenmann) is also an ex officio member. The XSEDE Senior Management
team meets on a bi-weekly basis to assess project status, plans, and issues.
8.2 External Relations (WBS 2.6.2)
External Relations’ (ER) mission is to communicate the value and importance of XSEDE to all
stakeholders (including the internal audience) through creative and strategic communications.
The ER team will measure its ability to effectively communicate with internal and external
stakeholders by the number of social media impressions earned through user engagement, the
responsiveness of email communications, and by the frequency and quality of content developed by
the ER team, or successfully placed pitches to targeted media outlets.
Social media impressions are an indicator of the quality of information and content XSEDE is
generating and sharing with current communities, advanced computing enthusiasts, XSEDE users,
and new communities. Effective use of social media tools and the generation of good content allows
XSEDE to raise awareness of the value of advanced digital services.
The number of science stories and announcements produced by media outlets are Area Metrics
because they are an indicator of XSEDE’s ability to identify science success stories and to effectively
work with targeted media in order to reach key audiences.
Monthly open and click-through rates of the external XSEDE newsletter, Impact, are important Area
Metrics because they indicate engagement levels of users and supporters of the project.
Table 8-2: Area Metrics for External Relations
Area Metric
Number of Social Media impressions
PY6 Target
228,000
Number of media hits
147
Number of science success stories and
announcements appearing in media outlets
62
Monthly open and click-through rates of XSEDE’s
newsletter
Open: 35%
Click-through:
20%
Sub-goal Supported
Deepen/Extend - Raise awareness of
the value of advanced digital services
(§2.1.4)
Deepen/Extend - Raise awareness of
the value of advanced digital services
(§2.1.4)
Deepen/Extend - Raise awareness of
the value of advanced digital services
(§2.1.4)
Deepen/Extend - Raise awareness of
the value of advanced digital services
(§2.1.4)
56
This Area Metric goal for “Number of Social Media impressions” has increased from 190,000 to
228,000 based on strong numbers from the first three quarters of PY5.
The Area Metric goal of science success stories and press releases is 62. This is a 5% increase based
on the number of press releases (35) and science highlights (24) distributed in Q4 of PY4 and the
first three quarters of PY5.
The goal open rate of XSEDE’s newsletter for the project year has been set at 35% and the clickthrough rate has been set at 20%. The XSEDE project has consistently out-performed averages in
these areas. Average open rates among organizations in higher education and non-profit services
range from 18.14-21.90%. Average click-through rates for these sectors range from 8.63-8.74%.
The ER Team will continue developing the Strategic Communications and Marketing Plan. In
addition to the ER’s ongoing activities outlined in the project plan, it will assess how marketing
efforts need to evolve in order to raise awareness of the value of advanced digital services. A strong
focus on media relations will help raise awareness for XSEDE science successes among internal and
external stakeholders. This will include building relationships with key journalists and bloggers and
effectively pitching stories and announcements. Developing and executing a strategic social media
communication plan helps XSEDE deliver key messages to a number of constituencies including
media outlets, users, supporters, and future generations of users. ER will maintain a social media
calendar, develop content, and engage users. Science writing supports these efforts, one of the most
important efforts the ER team supports and executes. These stories are used in the creation of
communications pieces, marketing collateral, and news stories for internal and external
stakeholders. Complementing this is the XSEDE Science Highlights book, a staple of the ER program
and a vehicle for raising awareness of XSEDE success stories. The ER team will write, design, print,
and distribute these books.
The monthly electronic communication to our external stakeholders, XSEDE’s Impact newsletter, is
an essential vehicle for communicating successes, events, and other announcements to external
stakeholders and enthusiasts. The internal analog to this, Inside XSEDE, communicates internal
processes and successes along with project status and direction information.
XSEDE ER also coordinates the projects involvement in events throughout the year providing
material design, web support, and publicity support for XSEDE-related events. In particular, ER
provides a strategic and well-executed presence at the XSEDE booth at the annual Supercomputing
conference (SCxy). For both SCxy and the annual XSEDE conference, the ER team provides support,
planning, logistics, and publicity. These activities are essential to meeting our sub-goals of raising
awareness of the value of advanced digital services.
8.3 Project Management, Reporting, & Risk Management (WBS 2.6.3)
Project Management, Reporting & Risk Management (PM&R) enables an effective virtual
organization through application of project management principles, provides visibility to project
progress, successes, and challenges, brings new ideas and management practices into the project
and disseminates lessons learned in XSEDE to other virtual organizations.
To deliver the most advanced and useful digital services for researchers, XSEDE must constantly
evolve. This makes continuous evaluation and improvement of processes and policies critical to the
57
effectiveness of XSEDE as an organization; therefore, the project’s KPI for sustaining an effective
virtual organization is the number of improvements made. Potential improvements consist of
recommendations based on the staff climate survey, efforts by area managers to increase efficiency
or quality, adaptations in response to an evolving project or improvements to overall execution
through implementation of widely validated methodologies.
Process improvements are tracked via a self-reporting method. All areas of the project are asked to
submit the process improvements they have made to their areas on a quarterly basis. From this list,
the improvements are then classified as being either strategic or innovative and proactive or
reactive.
Similarly, communication is critical to success in this highly distributed virtual organization. The
coordination provided by PM&R ensures timely delivery of status reports to NSF, full and timely
consideration of project change requests (PCRs), and awareness and evaluation of risk.
Table 8-3: Area Metrics for Project Management, Reporting, & Risk Management
Area Metric
PY6 Target
Number of strategic or innovative improvements
9
Ratio of proactive to reactive improvements
3
Variance, in days, between relevant report
submission and due date
0
Percentage of risks reviewed
Average number of days to execute PCR
95%
30 calendar
days
Sub-goal Supported
Sustain – Operate an innovative
virtual organization (§2.3.4)
Sustain – Operate an innovative
virtual organization (§2.3.4)
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
The target of nine strategic or innovative improvements was chosen as a modest target. This metric
was only tracked in PY5 and thus historical data is not there to support a trend. The team will
continue to refine this target as more data is collected. It is also expected to see less reactive and
general improvements as the project makes use of systematic processes to identify needed
improvements and thus have set the ratio of proactive to reactive ratio as 3. The team intends to
have zero days of variance between the date a relevant report is submitted and the due date where
due dates are set by the Project Director and the NSF Program Officer and are communicated to the
PM&R. The goal is to review 95% of the risks tracked in the risk register each quarter alongside
continual efforts to identify risks in the project’s activities. Finally, 30 days is the target to execute a
PCR with execution being defined as starting from the time at which all approvals have been
received to the time the PCR is fully implemented.
The planned activities of the PM&R group all support the effectiveness of our virtual organization
(§2.3.3) by fostering and coordinating internal and external communications, developing and
communicating measures of performance in key areas, coordinating and tracking implementation
58
of process improvements, managing project risk, and planning. Our specific planned efforts include
the following support business and finance function, coordinate area management in project-wide
processes, develop and coordinate reporting, plan and execute project processes, develop and
coordinate annual program plan, plan and execute project-wide meetings, assess and manage risk,
develop, track, and review key performance indicators and process and track project change
requests
The PM&R team will manage the transition to the new organizational structure and modified
process for project execution. In particular, PM&R will adopt JIRA (with appropriate extension
modules) to support project wide project management functions. Of particular importance, PM&R
will manage the creation of a dashboard to track KPIs and area metrics.
8.4 Business Operations (WBS 2.6.4)
The Business Operations group, working closely with staff at the University of Illinois’ Grants and
Contracts Office (GCO) and National Center for Supercomputing Applications’ (NCSA) Business
Office, will handle budgetary issues, manage sub-awards and assure timely processing of sub-award
amendments and invoices.
Business Operations will measure its ability to manage the business relationships with all subaward institutions through the processing cycle-time of sub-award amendments and invoices.
Efficient processing of sub-award amendments and invoices is required to avoid possible delays in
the overall execution of the project.
Table 8-4: Area Metrics for Business Operations
Area Metric
PY6 Target
Sub-goal Supported
Processing times for stages of processing subaward amendments
41 calendar
days
Processing times for stages of processing invoices
45 calendar
days
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Total decisions and subsequent actions for
business practice and/or project improvement
5 per quarter
The targets for the processing time of sub-award amendments and invoices are aligned with the
cycle times experienced during XSEDE PY5. The planned activities for XSEDE Business Operations
focus on the necessities of maintaining the business relationship with all sub-award institutions,
including processing of sub-award amendments, processing of sub-award invoices, and ensuring
the appropriate sub-award documentation is provided to the NSF.
8.5 Strategic Planning, Policy & Evaluation (WBS 2.6.5)
XSEDE will dedicate effort to project-wide strategic planning, policy development, evaluation and
assessment, and organizational improvement in support of sustaining an effective and productive
virtual organization.
59
XSEDE will engage an independent Evaluation Team designed to provide XSEDE with information
to guide program improvement and assess the impact of XSEDE services. Evaluations will be based
on five primary data sources: (1) an Annual User Survey that will be part of the XSEDE annual
report and program plan; (2) an Enhanced Longitudinal Study encompassing additional target
groups (e.g., faculty, institutions, disciplines, etc.) and additional measures (e.g., publications,
citations, research funding, promotion and tenure, etc.); (3) an Annual XSEDE Staff Climate Study;
(4) XSEDE KPIs, Area Metrics, and Organizational Improvement efforts, including ensuring that
procedures are in place to assess these data; and (5) Specialized Studies as contracted by Level 2
directors and the Program Office.
The first Area metric was designed to address the extent to which evaluation recommendations are
considered by project leadership. The metric intentionally does not measure “implementation” of
recommendations due to the many factors that may impede implementation (e.g. timelines,
funding, capacity, etc.). The evaluation team intends to track this metric with new tools currently
being implemented in XSEDE, namely JIRA and Confluence. The team intends to upload evaluation
reports to Confluence and copy recommendations to JIRA for tracking. Since this L3 area is also
responsible for strategic planning and policy, the second Area Metric measures how well-prepared
XSEDE staff feel to perform their jobs. This metric is obtained by staff self-report via the Annual
XSEDE Staff Climate Study conducted by the external evaluation team. Staff responses to a set of
Likert Scale items measured on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree) and relating to
staff preparation are combined to determine a mean index score.
Table 8-5: Area Metrics Strategic Planning, Policy & Evaluation
Area Metric
PY6 Target
Percentage of recommendations addressed by
relevant project areas
90%
Average rating of staff regarding how wellprepared they feel to perform their jobs
4 of 5
Sub-goal Supported
Sustain – Operate an effective and
productive virtual organization
(§2.3.3)
Advance - Enhance the Array of
Technical Expertise and Support
Services (§2.2.2)
Due to the time and effort often required to thoughtfully address recommendations, the annual
target for the first Area Metric is set to 90%. Regarding the second metric, Climate Studies
conducted of similar organizations by the External Evaluation Team resulted in mean index ratings
ranging from 2.83 – 3.74 for similar dimensions. XSEDE staff rated this index at 3.41 in 2013, 3.51
in 2014, and 3.57 in 2015.
The External Evaluation Team will continue to provide evaluations of the annual XSEDE Conference
including surveys of conference participants and longitudinal tracking of attendees via the XUP.
These data will continue to inform the conference committee and project leadership of the
effectiveness and impact of the event to address the extent to which it contributes to “sustaining an
effective and productive virtual organization.”
The External Evaluation Team will conduct an enhanced longitudinal study that will include
additional target groups (e.g. faculty, type of institution, disciplines, type of allocation, etc.) and
60
additional measures (e.g. publications, citations, research funding, promotion and tenure, etc.).
These data will inform program managers of the effectiveness and impact of their collective efforts
thus supporting the project sub-goal of “sustaining an effective and productive virtual
organization.”
New processes surrounding the collection and aggregation of XSEDE KPIs and Area Metrics will be
implemented, including consultation with project management and Level 2 managers to assure that
measures and data collection procedures are in place to assess KPIs and Area Metrics. The External
Evaluation Team will work with XSEDE staff to develop data collection procedures, to collect and
analyze data as needed, and assist Level 2 managers in the development of reports and in the use of
findings. This activity will help ensure the project as a whole has a deep understanding of their KPIs
and Area Metrics in order to promote sound interpretation and response to findings.
New studies, as contracted by Level 2 managers and Project Office, will be negotiated annually and
incorporated into the evaluation plan for the following areas: Operations, XCI, ECSS, CEE, RAS and
Project Office. These studies will uncover unique aspects and outcomes attributed to each L2 area
to promote thoughtful response to findings.
Project wide evaluation activities will be integrated, including surveys, interviews, focus groups,
etc. performed by other evaluation groups affiliated with XSEDE to promote efficiency and use of
evaluation data across all project areas thus supporting the sub-goal of “sustaining an effective and
productive virtual organization.”
With the organization change of moving the External Evaluation Team to the Program Office, the
team will be positioned to better serve needs across XSEDE. As a result, in depth evaluation
activities previously conducted with Education and Outreach (and now positioned within CEE) will
be significantly reduced. CEE evaluation activities will be negotiated annually along with all other
Level 2 areas via the Specialized Studies evaluation activity.
61
APPENDICES
A Glossary and List of Acronyms
ACRONYM
DESCRIPTION
A3M
A&AM
A&D
Allocations, Accounting & Account
Management
Accounting & Account Management
Architecture & Design
ADR
AL2S
Architecture Design Review
Advanced Layer 2 Service
AMIE
CB
Account Management Information
Exchange
Application Programming Interface
A quantifiable measure that is used to track
and assess the status of a specific process.
Area metrics can measure performance or
operational status. Area metrics relating to
performance can be used alone or in
combinations as a key performance
indicator (KPI) for the project.
Campus Bridging
CC
CEE
co-Pi
CRI
CRM
CS&E
Campus Champion
Community Enhancement & Engagement
Co-Principal Investigator
Capability & Resource Integration
Customer Relationship Management
Computational Science & Engineering
DNS
DNSKEY
DNSSEC
DTS
E&O
ECSS
Domain Name Service
Domain Name Service Key
DNS Security
Data Transfer Services
Education and Outreach
Extended Collaborative Service
e-infrastructure
The integration of networks, grids, data
centers and collaborative environments,
and are intended to include supporting
operation centers, service registries,
credential delegation services.
External Relations
API
Area Metric
ER
Notes
Enables Internet2 users to
create point-to-point VLANs
Infrastructure to make XSEDE
resources appear to be proximal
to the researcher’s desktop
62
ACRONYM
DESCRIPTION
ESRT
ESSGW
GFFS
GridFTP
HPC
Extended Support for Research Teams
Extended Collaborative Support for Science
Gateways
Extended Support for Training, Education,
& Outreach
Full Technical Equivalent
General Automated Atomic Model
Parameterization
Globus Federated File System
Grid File Transfer Protocol
High Performance Computing
HPCU
HSM
I2
IC
IdM
INCA/Nagios
Hardware Security Models
Internet2
Industry Challenge
Identity Management
an service monitoring tool
ESTEO
FTE
GAAMP
IR
JIRA
KB
KPI
L2
L3
MFC
MS
MSI
NCAR
NCSA
NICS
NIP
OPS-SEC
OSG
OTP
PEP
perfSONAR
Notes
Incident Reports
an activity tracking tool
KB documents
Key Performance Indicators - A metric or
combination of metrics meant to measure
performance in key areas of the program so
that actions and decisions which move the
metrics in the desired direction also move
the program in the direction of the desired
outcomes and goals.
WBS Level 2
WBS Level 3
Minority Faculty Council
Microsoft
Minority Serving Institution
National Center for Atmospheric Research
National Center for Supercomputing
Applications
National Institute of Computational Science
Operations - Cybersecurity
Open Science Grid
One Time Password
Program Execution Plan
PERFormance Service Oriented Network
monitoring Architecture
63
ACRONYM
DESCRIPTION
PI
PM
PM&R
POPS
Principal Investigator
Project Management
Project Management, Reporting & Risk
Management
PACI Online Proposal System
PSC
PY
RAS
Pittsburgh Supercomputing Center
Program Year
Resource Allocations Service
RACD
Requirements Analysis & Capability
Delivery
Representational state transfer
an open source cluster distribution solution
that simplifies the processes of deploying,
managing, upgrading, and scaling highperformance parallel computing clusters.
RESTful
rocks roll
RSIG
RT
SACNAS
SCxy
SD&I
SDIACT
SDSC
SH2
Shodor
Supercomputing Conference (e.g. SC16)
Software Development & Integration
Software Development & Integration
Activity
San Diego Supercomputer Center
SH2 Security
A National Resource for Computational
Science Education
Service Provider
STEM
Science Technology Engineering
Mathematics
Systems Operations
Texas Advanced Computing Center
Training, Education and Outreach Service
an e-Science grid computing infrastructure
combining resources at eleven partner
sites.
Technology Investigation Service
Use Case Canonical Use Case
Campus Bridging Use Case
Data Analytics Use Case
TIS
UCCAN
UCCB
UCDA
this is no longer an acronym and
POPS is just the name for the
allocation submission system;
being supplanted by XRAS
Request Tracker Ticketing System
Society for Advancement of Chicanos and
Native Americans
SP
SysOps
TACC
TEOS
TeraGrid
Notes
64
ACRONYM
DESCRIPTION
UCDM
UCF
UCFC
UCHPC
UCHTC
UCSGW
Data Management Use Case
Federation and Interoperation Use Case
First Connecting Instrumentation Use Case
High Performance Computing Use Case
High Throughput Computing Use Case
Science Gateway Use Case
UCSW
UCVIS
UE
UII
URC
URCE
URM
US
WBS
Scientific Workflow Use Case
Visualization Use Case
User Engagement
User Interfaces and Information
Under-represented communities
Under-Represented Community
Engagement
User Requirement Evaluation and
Prioritization
Under-Represented Minority
United States
Work Breakdown Structure
XCDB
XSEDE Central Database
XDCDB
XCI
XOC
XRAC
XSEDE Central Database
XSEDE Community Infrastructure
XSEDE Operation Center
XSEDE Resource Allocation Committee
XRAS
XSEDE
XSEDE Resource Allocations Service
eXtreme Science and Engineering
Discovery Environment
XSEDE CA
XSEDE Certificate Authorities
XSEDE KDC
XSEDE Kerberos
XSEDE14
XSEDEnet
XSP
XTED
XUP
XSEDE Conference in 2014
an XSEDE-only network
XSEDE Scholars Program
XSEDE Technology Evaluation Database
XSEDE User Portal
XWFS
XSEDE Wide File System
UREP
Notes
Numerical code for each group
within XSEDE
The XDCDB contains 24
schemas, notably the
accounting, resource repository,
portal, and AMIE databases.
Entity responsible for certifying
encryption keys for identity
management
The XSEDE web pages at
http://xsede.org
65