Response to Issues Raised by COV
1. Section I.1: “Are the review methods appropriate?”
Comment: In general the use of review methods is deemed appropriate. The COV
compliments the section, in particular, on the rigor and timeliness of the reviews that
were conducted for the NCAR Wyoming Supercomputer Center (NWSC). For observing
facilities, such as CHILL, the merits of a site visit as part of the proposal review process
should be considered.
Response:
FY 2012:
In general we agree. The decision to conduct a site visit is made by the cognizant
Program Officer based on many factors including the maturity of the facility and
its prior performance. It should also be noted that the Program Officer does make
occasional visits to the facility both at its home base and in the field. No site visit
was made to the CSU-CHILL before the current cooperative agreement, but
others have been conducted as part of the review process of prior cooperative
agreements. The entire suite of radar facilities is undergoing an extensive
evaluation by NFS. We would anticipate that site visits will be part of that
process.
FY 2013:
A site visit to the CSU-CHILL is currently scheduled for November 2013. While
not part of the review process of the current award, it is a mid-point site visit that
was made a requirement of the cooperative agreement.
FY2014:
The site visit to the CSU-CHILL radar was held in January 2014, with the site
visit team’s final report received at NSF in April 2014. Discussions held during
the site visit and the final report are available in the award jacket.
2. Section I.5: “Does the documentation in the jacket provide the rationale for the
award/decline decision?”
Comment: While most jacket documentation is clear, the jacket documentation for the
A10 project, including a decline of one relevant proposal, was somewhat confusing.
Based upon discussions with the program officer, the COV notes that the A10 proposal
began as a response to the mid-size infrastructure competition in 2007-8. This
competition included preproposals and invited full proposals, with an ultimate success
rate of about 15%. Moreover, the A10 review process was protracted due to issues
regarding availability of funds and the need to obtain interagency agreements, and,
therefore, the consideration of this proposal extended over the time periods considered
by two COVs. The current COV deems this process appropriate.
1
Response:
FY 2012:
We agree that the documentation could have been clearer. Several factors
contributed to the complexity of this action including: the Section and specifically
the Section Head initially having a conflict with the proposal. This resulted in the
review of the proposal being handled by a Program Officer outside of AGS. The
evaluation was stretched over several years while negotiations were conducted
with DoD to see if an aircraft could be obtained. It is also important to note that
this aircraft is not an NSF owned aircraft, but rather DOD owned. The
negotiations for acquisition and use of the A-10 occurred between the President of
the Naval Post Graduate School and the Office of the Secretary of the Air Force.
For lengthy and complex review procedures, the Section will endeavor in the
future to incorporate a timeline and overview of the review process in the review
analysis.
3. Section I.7: “Additional comments on the quality and effectiveness of the
program’s use of merit review process”
Comment: For the NWSC proposal, there is clear documentation of actions the PIs will
take in response to concerns raised in reviews. In other cases (CHILL and A10), while we
understand from program officers that the PIs were responsive to the reviews, this is not
clearly documented.
Response:
FY 2012:
The Section will endeavor to be more explicit in the documentation. It should be
noted that issues raised in the reviews of both the CSU-CHILL and the A-10 were
considered in the development of the cooperative agreements and subsequent
actions by the Section. For example, the reviewer suggestion of a
community-wide workshop on issues related to the optimal suite of
meteorological research radars will be carried out by NFS in the Fall of 2012.
During the review process of the CSU-CHILL proposal, however, details of the
workshop were not yet developed, so little could be said at that time.
FY 2014:
The PI for the A10 project was asked to provide a formal response and other
documentation following each of the two site visits conducted during FY2014.
This and other related correspondence has been uploaded into the award jacket.
4. Section II.1: “Did the program make use of reviewers having appropriate
expertise and/or qualifications?”
2
Comment: In general the COV believes the program managers do an excellent job in
obtaining reviews from a technically appropriate and diverse pool of reviewers. In the
case of SOARS, while reviews were sought from a broad community, the COV notes that
the reviews obtained were all from people closely associated with geosciences education
and diversity. While the COV believe broader reviewer input for this program would be
useful, it appreciates the difficulty of obtaining such input.
Response:
FY 2012: The program will continue to seek and obtain reviewers from a broad
and diverse community. The program manager agrees with the COV that the
reviews obtained were those associated with geosciences education and diversity.
In the future, efforts will be made to obtain reviewers outside the geosciences.
This will be a challenge since reviewers like to be familiar with the subject and its
relation to their discipline.
5. Section II.2: “Did the program recognize and resolve conflicts of interest when
appropriate?”
Comment: (1) While not formally a conflict of interest, the COV notes that a potential
future user of an observing facility could have a vested interest in the outcome of the
review and recommends that the section be alert to this as a potential source of bias.
(2) More generally, the COV feels that the standard NSF jacket documentation does not
allow us to address this question fully.
Response:
FY 2012:
(1) We are aware of the potential bias of having users of facilities as reviewers.
However, we deem it important to have the perspective of former and potential
users as part, not all, of the review process. The facilities are a service to the
community and it is important for NSF to obtain the viewpoint of those who
receive the service as part of the review.
(2) I discussed this issue with the COV Chair. The point here is that the COV
has no information as to who might have been rejected as a reviewer (i.e. never
even asked) due to a COI. Adding such documentation would have to be a
Foundation wide requirement and it would be open-ended and, in my opinion, not
very feasible and the value gained would be small compared to the additional
burden.
6. Section III.1: “Management of the program.”
3
Comment:
(1) NFS management of NCAR and the observing facilities is greatly enhanced by the
effective engagement of science discipline program officers, especially those in the
Atmospheric Section (AS). The involvement of these programs is especially
important for NCAR’s annual planning and budgeting processes. The COV suggests
that UNIDATA and SOARS would benefit from similar involvement.
Response (1):
FY 2012:
It should be noted that the funding instrument for support of NCAR is a
cooperative agreement between NSF and UCAR. By their nature, cooperative
agreements require greater involvement of NSF in the direction of the project.
The NSF role in the management of NCAR, therefore, is more substantial than
with a grant and, as noted by the COV, the science Program Officers play an
important role in that effort. While previous awards to UNIDATA were via
cooperative agreements, the current funding instruments for both UNIDATA and
the SOARS are grants. While NFS has oversight of these efforts, there is no
significant formal involvement of NSF Program Officers in management of
UNIDATA and SOARS. Of course, the cognizant Program Officer is engaged
with both projects and attends meetings that have strategic implications (e.g. for
UNIDATA the User’s and Policy Committees). The cognizant Program Officer
informs other AGS program directors about meetings and issues that might be
relevant to their research programs or for items for which they might have
specific expertise.
Comment:
(2) The review of UCAR/NCAR was a key management activity in the period covered by
this COV. The COV considers the site-visit format for these reviews appropriate and
applauds the Section on its selection of members of the site-visit team (SVT). While
the COV agrees that NCAR lab directors should continue to provide responses to
SVT reports, we suggest that the NCAR director also provide responses.
Response (2):
FY 2012:
While the Laboratory Directors provided the response, all of these were cleared
by NCAR management and, therefore, NCAR management was involved in
formulating the responses. Also, NCAR management was involved in the
responses to the UCAR Management review, which deals directly with
overarching Center issues. It should be noted that responses and actions resulting
from the reviews are an ongoing process and most actions that require changes to
the center are done under the direct guidance of the NCAR Director.
4
Comment:
(3) Given that the NCAR laboratories provide services to the community, the COV
suggests that user surveys should be a routine part of NFS reviews of NCAR.
Response (3):
FY 2012:
NCAR currently conducts several types of user surveys that range from EOL
surveying participants after each field campaign to a UCAR survey on UCAR
programs, including NCAR, that is executed about once every four years. NFS is
generally informed of these surveys and the results. The existing surveys were
not used uniformly in the recently completed review of UCAR/NCAR and we
will endeavor to make sure they are better utilized during the next review. It
should be noted that some surveys are used by AGS staff on an ongoing basis.
NFS staff and relevant program officers fully participate in the field campaign
debriefs and these debriefs are used as a tool for continual improvements.
FY 2013:
In FY 2013 two post program debriefs were held between NCAR staff, involved
researchers and NSF Program Officers. These were the Southern
Andes-Antarctic Gravity Wave Initiative (SAANGRIA)-Test and the Mesoscale
Predictability Experiment (MPEX).
FY2014:
NSF program officers continue to be involved in post-field campaign debriefs.
During the past year, these were held for the Southeast Atmosphere Study (SAS),
Deep-propagating Gravity Waves (DEEPWAVE-NZ) and the Front Range Air
Pollution and Photochemistry Experiment (FRAPPE).
Comment:
(4) The COV recognizes the unavoidable tension at NCAR between the competing
demands of excellent scientific research and service to the community in the
provision and maintenance of observing facilities and community models. We
believe the review process should explicitly address the balance between these
functions, in terms of the national role of NCAR and the morale of NCAR scientists.
Response (4):
FY 2012:
5
The issue of allocation of resources across all aspects of NCAR was a topic of the
UCAR management review. This balance is also addressed each year during the
annual budget discussions between AGS and UCAR/NCAR management. We
are aware that if facilities are protected to the extent that the science programs
suffer, the morale of the scientific staff will suffer. This is a difficult issue and
one that we take very seriously.
FY 2013:
With the sequester in FY 2013, this tension persisted. NSF/AGS engaged NCAR
management in intense negotiations over the budget including the balance
between research and service functions and came to a resolution acceptable to
both parties.
FY2014:
The balance between the different components of NCAR’s mission continues to
be one of many factors considered during the annual budget discussions between
NSF/AGS and NCAR management.
Comment:
(5) The SVT reports on NCAR laboratories provide generally favorable reviews of their
activities. In some cases, however, the SVT and the program officer noted
potentially serious issues. In such cases, it is not clear to the COV what the followup activities have been; these should be considered by our successors. Where such
issues arise, as per the comment above, it seems reasonable that the NCAR director
should be involved.
Response (5):
FY 2012:
Follow-up actions to the SVTs are an ongoing process. Some reactions were
almost immediate (e.g. scheduling of workshops on issues of climate and
chemistry instrumentation needs, both of which were recently conducted), while
others are still in progress. About one year after the reviews (just after the COV
was completed), NFS asked NCAR for an update on the status of issues raised in
the review. NFS recently received this update and is pursuing resolving any
outstanding issues. As noted above, the NCAR Director approves all responses
and updates and, as appropriate, he is involved in resolution of outstanding issues.
FY 2013:
Consideration of some issues continues. A prominent one was a lack of
coherency in the NCAR educational efforts. This was dealt with by requiring in
6
the new cooperative agreement (which begins 1 October 2013) a comprehensive
strategic educational plan along with an implementation strategy for the plan.
FY 2014:
The cooperative agreement that became effective on October 1, 2013 contains
requirements explicitly intended to address concerns raised by the SVTs and in
the review of UCAR’s 2012 renewal proposal for the management and operation
of NCAR. NFS program staff continues to engage NCAR laboratories on these
and other issues. Program officers attend meetings of laboratory external advisory
committees and generally receive the reports of those committees.
7. Section III.2: “Responsiveness of the program to emerging research and education
opportunities”
Comment:
The COV considers community workshops to be useful tools for setting future directions
for the suite of observing facilities. Because such workshops often have significant and
lasting outcomes, it is important that the attendees represent a broad cross-section of
stakeholders. For example, the upcoming (fall 2012) NSF radar-facilities workshop will
influence critical decisions regarding the future of NSF radars. It is, therefore, important
that the attendees go beyond the direct users of these facilities to include scientists taking
diverse approaches, such as modeling and theory, and working in diverse science areas
(e.g. climate and hydrology).
Response:
FY 2012:
The workshop will be open to any interested attendees and it has been widely
advertised. We agree that diversity (in all aspects) is important and we will
endeavor to accomplish that goal both with the attendees as well as with speakers,
break-out chairs and reporters.
FY 2013:
The workshop was held successfully and we believe that we had representation
from a good cross-section of the atmospheric sciences community.
FY 2014:
A report from this workshop was written by the workshop chair and co-authored
by the invited presenters. This report will appear in the Bulletin of the American
Meteorological Society (it is currently available in the early online release format
at doi: 10.1175/BAMS-D-13-00079.)
7
8. Section III.4: “Responsiveness of program to previous COV comments and
recommendations”
Comment:
At the same time, the previous COV suggested that observing facility cooperative
agreements be regularly competed. While the complexities of conducting such
competitions are acknowledged, this COV believes there may be virtue in holding open
competitions for meeting some of the community’s observational needs. Whether or not
such competitions are useful or appropriate must necessarily be decided on a case-bycase basis.
Response:
FY 2012:
We agree that the appropriateness of any competition must be on a case-by-case
basis. For example, some of the AGS national facilities are not owned by NSF
(e.g. Wyoming King Air, CIRPAS Twin Otter and A-10), so if AGS wishes to
retain those specific facilities, then we must deal with that specific organization.
However, as the award to each facility nears expiration, NFS evaluates not only the
continuing need for such a facility as a national resource, but also the issue of
whether the incumbent facility is the best or only facility that could satisfy
community needs. NFS typically relies on community advice to help guide such
decisions. For example, the radar community workshop to be held in the Fall of
2012, will influence the development of plans by AGS to satisfy community
research radar needs in the near and intermediate future and will likely inform
decisions about whether or not to compete expiring cooperative agreements for
radar facilities.
FY 2013:
AGS has decided to continue to provide mobile Doppler weather radars as a
national facility. A facility manager will be selected through an open competition,
which is now in progress.
FY2014:
The competition (NSF 13-571) for the provider of a mobile Doppler radar facility
noted in the FY2013 entry was successfully concluded, with an award made to the
Center for Severe Weather Research for a five-year term. The performance of the
awardee will be assessed through a formal site visit midway through the award
period.
9. Section IV: “Portfolio Review”
8
Comment:
As the NFS portfolio comprises NCAR, two other UCAR programs, and a suite of
observing facilities, many of the above questions are not applicable. In regard to the
geographical distribution of activities, the strength and success of NCAR leads to a
natural concentration of atmospheric sciences activity along the Front Range. In
recompeting key facilities, NFS should be mindful of community concerns that
atmospheric science research resources are over-concentrated in this region.
Response:
FY 2012:
While we understand the concern, as noted above, AGS is not the owner of some
of these facilities and, therefore, cannot move them at will. We also believe it is
more important that the locations of the deployments of the facilities be
geographically (and, therefore, scientifically) diverse, rather than their home bases
be geographically diverse. Our stance would be that geographical diversity, while
being a consideration, would be of secondary importance compared to how well
matched the facility is to community needs and as to the likelihood the facility
manager will be successful in running a community facility.
10. Other Topics: Question 4: “Please provide comments on any other issues the COV
feels are relevant.”
Comment:
(1) The COV notes that the DOW sets an apparent precedent, in investing a national
facility in a private entity. The COV encourages NFS to give this careful
consideration
Response:
FY 2012:
We agree with the COV’s caution. When the award was first made NSF was
concerned as to whether a small, private, non-profit had both the financial
stability and resources to perform as a national facility. The awardee institution
underwent extra scrutiny by the Division of Grants and Agreements and its
performance was monitored closely by the cognizant Program Officer. Overall,
to date we believe that the performance of the awardee has been more than
adequate. We agree, however, that awards of this sort to small, private companies
warrant special care and oversight.
Comment:
(2) Much of the NFS portfolio comprises activities that serve broad user communities.
Regularly conducted surveys of these communities could prove valuable for
reviewing and managing these facilities and activities.
Response:
9
FY 2012:
We asked NCAR/UCAR to provide a brief description of all periodic community
surveys that they perform (attached). These range from post deployment surveys
for all observational facilities (including non-NCAR) to once every four year
community surveys by UCAR. All of these are used by NCAR and the other
facilities for their and NFS management purposes. Some user surveys also were
incorporated into the NCAR Laboratory reviews, but in an uneven manner. NFS
will investigate incorporating these surveys into the NCAR/UCAR review process
in a more formal and consistent way.
FY 2014:
Post-deployment surveys of facility users are formally required in new
cooperative agreements with facility providers. The facility managers are
expected to report on the survey results in their annual reports to NSF.
10
Attachment
List of Surveys Conducted on NCAR Activities
July 17, 2012
Name
UCAR Community
Survey
Computational User
Survey
Use of Computational
Facilities in Publications
UCAR/NCAR Entity
Frequency
UCAR
2000, 2005, 2009
CISL/USS/DSS
Every 2 to 4 Years
CISL/USS/DSS
Annually (results used
in ABR)
Networking Satisfaction CISL/NETS
Last conducted 2009
EOL Field Programs
EOL
After each field
campaign
CESM Workshop
NESL
CESM Tutorial
NESL
WRF Workshop
NESL, RAL
WRF Tutorials
NESL, RAL
Annually, after the
workshop
Annually, after the
tutorial
Annually, after the
workshop
Twice/year, after each
of the two tutorials
WRF for Hurricane
NESL, RAL
2009, 2010 (none in
2011)
GSI Tutorial
RAL
Annually since 2010
GSI Workshop
MET Tutorial
RAL
RAL
2011, 2013
Annually
Description and Nature of Survey
Web-based survey to obtain feedback from research community on
UCAR programs including NCAR's research and facility programs.
User satisfaction survey of CISL's computational facilities
University user community for list of scientific publications that have
benefitted from CISL computational resources, including the
Research Data Archive (RDA).
Conducts surveys of UCAR community for networking satisfaction
Post-field campaign questionnaires to obtain metrics on field
campaigns and determine satisfaction with field project support and
data management services provided for field programs. Involvement
of PIs in formal field campaign debriefs as a mechanism for providing
direct input to EOL.
Sent to participants to solicit review of the workshop and
suggestions for improvements
Sent to participants to solicit review of the tutorial and suggestions
for improvements
Sent to participants to solicit review of the workshop and
suggestions for improvements
Sent to participants to solicit review of the tutorial and suggestions
for improvements
To obtain feedback on the lectures, written materials (Users Guides),
on the exercises, and on the general tutorial organization. We also
ask if students would like to hear more/less on certain topics in
future tutorials.
To find out how the tutorial or workshop went, as well as some
questions aiming to help improve user support (like what users use
the system for, what they would like to learn in the future etc).
Same as above with the GSI tutorial
Obtain feedback on tutorial
11