Presentation for the
Advisory Committee for Business & Operations
Effective Practices Research Overview
For Merit Review
Arlington, VA
March 31, 2004
This document is confidential and is intended solely for the use and
information of the client to whom it is addressed.
NSF BUSINESS ANALYSIS
Today, we will provide…
 Brief update on the status of the NSF Business Analysis
 Summary of the key findings from the Effective Practices research for Merit Review
NSF BUSINESS ANALYSIS
1
NSF Business Analysis has recently competed the following
activities and is currently moving into the design phase of the
project
Quick Snapshot of Project Status
Completed as of 3/31/04
“Works in Progress”
 Completed researching effective practices for the
Merit Review and Award Management &
Oversight processes
 Developing process scenarios and preliminary
business cases for the Merit Review and Award
Management & Oversight processes
 Completed workload study that assessed the
amount and distribution of workload across NSF
 Conducting an eJacket study to determine how
the electronic-jacket technology tool impacts
human capital issues
 Completed first full draft of the Target Enterprise
Architecture for Applications, Data, Network and
Security
 Developing an IT Technology Governance
Framework and IT Implementation Plan that
considers the 2-7 year transition plan necessary
to move to the Target Enterprise Architecture
NSF BUSINESS ANALYSIS
2
For the “effective” practices research, the team researched the
practices of grant-making organizations in order to identify
potential alternative practices
Research Objectives:
To understand how other federal and non-federal organizations review proposals
and manage awards, and the relative emphasis that is placed on either activity
To provide NSF with insights into how other organizations address challenges
similar to those faced by the Foundation in the proposal review and award
management processes
To identify practices or principles that could be leveraged or modified by NSF in the
redesign of its core business processes
NSF BUSINESS ANALYSIS
3
The team interviewed representatives of 14 federal and non-federal
organizations
Organizations Interviewed
NSF BUSINESS ANALYSIS
4
Important to note is that few effective practices were identified in
the areas of performance measurement, “working across the
organization,” and technology usage
Few organizations apply performance metrics to their review or award
management practices
– Customer satisfaction surveys are performed ‘ad hoc’
– Processing times tend to be the same or longer than NSF
– Few agencies have monitoring plans in place to address the degree of
award oversight that is applied
Few organizations had a significant need for ‘working across the
organization’
Many agencies were either “developing” or “progressing” in their eBusiness
capabilities with availability of resources cited as the greatest barrier to
development efforts
NSF BUSINESS ANALYSIS
5
Effective practices emerged in each of the following areas
Merit Review Areas
Proposal
Workflow
Management
Review
Methodologies
Reviewer
Community
Management
Description
Process for how an organization receives, processes,
and streamlines activities associated with proposal
intake, review, and decision-making
Various practices an organization applies to review
proposals and make funding decisions with limited
resources
Management practices that an organization applies to
attract, recruit, and reward its reviewers, as well as
how the organization maintains a relationship with its
community
NSF BUSINESS ANALYSIS
6
Many organizations employ different strategies to mitigate
workload demands
Strategy
Description
Apply “filters” to reduce the number
Leveraging Letters of Intent (LOI), pre-applications or
of proposals that receive reviews and
other filters that enable organizations to better manage
to manage future workload
workload
requirements
Identifying early in the process proposals that are
noncompetitive or inappropriate, and begin panel
coordination earlier in the review process
Coordinate the scheduling of
program deadlines to spread the
workload more evenly across the
fiscal year
For organizations with multiple programs, centrally
coordinating the deadlines for programs to reduce the
burden on both internal resources and the reviewer
community
Employ temporary staff during peak
periods or outsource some
administrative functions to mitigate
workloads
Using temporary employees to provide staff with
administrative relief during periods of peak workload.
Services range from minor administrative support (i.e.,
jacket assembly) to outsourcing all proposal support
(i.e., panel coordination, soliciting reviewers, conflicts
of interest, etc.)
Proposal Workflow
Management
Review
Methodologies
Reviewer
Community
Management
NSF BUSINESS ANALYSIS
7
Most organizations structure the review process to optimize the
time of reviewers
Strategy
Description
Structure decision-focused review
process; allow reviewers to spend
more time on competitive proposals
Posting reviews prior to panel or pre-ranking proposals
based on reviewer feedback, then focusing panels on
those proposals that require additional feedback upon
which to base a decision (e.g., proposals “on the
fence”)
Reduce degree of “budget”
negotiations that must occur
throughout the post-review process
Employing strategies to mitigate the degree of postreview budget discussions include:
– Embedding “cost realism” as a review criteria
– Adjusting budgets and issuing ‘conditional awards’
– Employing policies of non-negotiable awards
Enable reviewers to participate
remotely in panels through the use of
technology
Providing tools that: facilitate involvement of panelists
who are unable to physically be present through
remote participation; allow NSF to utilize expert
reviewers for the appropriate panels; and, broaden the
accessible reviewer pool
Proposal Workflow
Management
Review
Methodologies
Reviewer
Community
Management
NSF BUSINESS ANALYSIS
8
Most organizations treat reviewers as “customers” throughout the
process
Strategy
Treat reviewers as “customers”
throughout the review process
Description
Coordinating all panel logistics including travel
arrangements
Providing social/networking opportunities while on-site
Pay them well
Offering attractive honorariums, paying ‘ad hoc’
reviews
Educate reviewers on their duties
and responsibilities
Providing training to new panelists. Training could
include on-line tutorials, mentoring, or “mock reviews”
Structure “terms of service” to ensure Requiring panelists to serve for a period of time -continuity
generally for a 3-year term and not necessarily
consecutive years
Reach out to the full ‘pool’ of possible Providing targeted marketing and outreach efforts to
reviewer candidates
potential reviewers (i.e., soliciting universities,
societies, etc.)
Integrating of databases for applicants, reviewers, and
awardees
Accepting on-line applications
Proposal Workflow
Management
Review
Methodologies
Reviewer
Community
Management
NSF BUSINESS ANALYSIS
9
Based on the findings in the effective practices research, NSF
should consider the following in the scenario design
 Broadening the use of Letters of Intent (LOIs) for determining program fit and begin planning
and coordinating panels earlier in the process
 More pro-actively coordinating program deadlines taking into account the needs of the
research community
 Developing piloting opportunities to broaden the use of temporary help during peak proposal
times or outsourcing certain process functions
 Incorporating process changes that reduce the degree of proposal budget rework that occurs
throughout the Merit Review process
 Compensating off-site reviews, broadening the concept of panel “terms of service”, and
providing training for how best to perform a review
 Developing strategies that “reach out” to a greater pool of reviewers (e.g., integrating
databases for applicants, awardees, reviewers; enabling better remote access; accepting online reviewer applications, etc.)
NSF BUSINESS ANALYSIS
10
Questions & Answers
For additional questions, please contact:
– Abe Zwany, [email protected], 703.902.5342
– Tim Koch, [email protected], 703.377.0389
NSF BUSINESS ANALYSIS
11